Site down: "high CPU usage is due to the large traffic generated from Moz
-
My client's site is down and the web host gives says that Moz is the reason why.
"The fact that your site was limited is because the traffic generated by Moz. This is why I have suggested to block their IP addresses."
Now we have unblocked the IP addresses and as you can see your site was limited again. And again the :
Code:
54.224.139.99 - - [26/Oct/2017:16:00:43 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/ HTTP/1.0" 200 58551 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:02 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/ HTTP/1.0" 200 58521 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:16 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_ HTTP/1.0" 301 - "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:30 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/ HTTP/1.0" 200 58528 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)""Please check with Moz if they can reduce the rate your site is crawled. Only after you confirm that the rate is decreased we will remove the limit imposed on your account."
NOTE: Can you resolve this?
NOTE: I have achieved the campaign at this time in an effort to keep the site live.
-
I received a Moz prompt to check out if this question was answered yet.
Any chance that you have had a response from your development team?
The client's site is still archived. I am looking forward (dare I hope?) to those awesome reports coming again soon.
-
One other question, if I may add it.
I have several additional clients with the same host and at the same server tier.
Why is this one site only affected?
I am glad it is only one. However, if something can be tweaked to prevent a pending occurrence with others, I would be grateful to know.
Thanks in advance for your time and research.
-
Hi Dave,
I appreciate your working on this as we love your weekly reports and find them a huge value.
I already archived the campaign. So we are one very eager client of yours for news of a full solution. If needed, this is for https://www.smiledesigndentistry.com/
Thank you, JH
-
Hey Jeannie, Dave from the Help Team here.
Sorry about the trouble! Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file.
Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files. I'm so sorry about that! Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time.
One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this:
User-agent: rogerbot
Disallow: *I am going to show this conversation to our development team as well. Again, sorry for any inconvenience - please do let me know if there's anything else I can help with!
-
Moz, please advise.
Here is a followup from the web host:
"I also would like to inform you all the CPU is drained from moz bots once again. You can verify thi9s from the below output:
Code:
10123 54.163.0.158 rogerbot/1.2 https://moz.com/help/guides/moz-proc
454 54.144.199.191 rogerbot/1.1 http://moz.com/help/guides/search-ovIn order to resolve the problem I would ask you to contact https://moz.com/ again and ask them to reduce their agent Crawl rate:
rogerbot/1.2
Thank you for your cooperation in advance.
For your convenience, I will leave this ticket with status Awaiting Moz/client reply.
Best Regards, Technical Support Team"
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Solved Site DA and PA
Hi, I'm facing an issue regarding Moz Site Explorer. When I enter my blog for check it's DA and PA. All Moz do is keep loading. I have to reload the page for two or more times for checking DA and PA. Kindly let me know is there any issue with my account. Regards.
Product Support | | BuiLungTen0 -
Campaign Dashboard Error: "Our Connection to Your Google Account Has Been Lost"
I keep getting the following error in my Campaign Dashboard: “Our connection to your Google account was lost. Don’t worry, you won’t lose any data”. Please reauthorize now. I do reauthorize but by the time I log out and log back in, the same message gets displayed. Any ideas? Thanks, Alan
Product Support | | Kingalan10 -
Moz usability
Why is it such a hassle to access Moz campaigns from the frontpage of Moz? How can MOZ be so awesome and at the same time be so annoying? Does Moz have UX designer working with them?
Product Support | | guidetoiceland0 -
Moz pricing plans don't make sense
Hey. I posted a question about this on my other Moz account (one I use for moz local), but it appears to have been removed, so here goes again. Can someone from Moz please explain to me why that in your plans, as the price goes up and more campaigns are allowed, the amount of keywords tracked per campaign goes down? $149 plan - 5 campaigns - 450 keywords = 90 keywords per campaign. $249 plan - 25 campaigns - 1100 keywords = 44 keywords per campaign $599 plan - 100 campaigns - 3700 keywords = 37 keywords per campaign What confuses me here with your pricing plans is two fold Modern SEO requires a lot of quality content, interlinked together to rank well (in most cases). Your SEO guides even talk about this. Blogging is one of the most popular ways to do this. However, in any modest website these days, especially one that runs a blog, how does tracking only 37 keywords per campaign make sense? The fact is, that even your starter plan with only 90 per campaign doesn't make sense. My business website is tracking about 300 (that actually drive traffic). I provide my SEO clients with the same type of work I would do for my self, which again reinforces that 37 keywords is plainly pathetic. Every online pricing model I've ever seen follows one simple rule: When price goes up, you get more for your money, not less. At first glance you might think that is what you're getting when you step up a plan and get more campaigns, but then to have less keywords per campaign simply doesn't make sense at all. What does make sense is to have a more flexible pricing model. Instead of locking us into set tiers which don't really make sense, you should allow a more flexible pricing BEFORE we have to go up to enterprise. Why not have a per project price of 'x' price per month, then have a price per keyword or maybe per 100 keywords. This would allow all of us the flexibility to run campaign depths that make sense for us, not for your counterintuitive views of how we should be tracking them. Again, I'd really like a Moz staff person to respond to this as the last time I commented about this, it was removed.
Product Support | | Multiverse-Media-Group0 -
Is Moz creating a Bunker-Mentality with its Customers?
I’ve been using Moz (and SEOMoz) for almost 3 years. During that time, I’ve found them to be very engaged; however, during the last several months, I’ve found a shift in this philosophy—which is a bit disheartening: SEOMoz used to have a very simple contact-us page that listed an email to the help desk as well as the office. Now, they have the locations of the two Moz offices, with no phone numbers or emails. Moz’s help hub, used to have a form that could be completed when you were having issues. Now, there is only a list of community-managed resources to access. The only way you can find the way to contact their help desk is if something breaks & the website gives you an email help@moz.com. However, this email account seems to get overloaded quite often and you will get a “Delivery Response Failed” message because their inbox is full. Does anyone know how to get a hold of Moz directly? I'm having some issues accessing Moz Analytics and really need help from their support team.
Product Support | | TopFloor4 -
Can you create additional users for Moz Analytics?
I have just begun setting up Moz Analytics for some of my clients. It lets me add 5 client's sites to a single account but I can't find anywhere to create an additional user/login for a specific account that way I can give access to each client only to their campaign. Is this not possible and if not something you plan on integrating in the near future?
Product Support | | Carlson3 -
Moz crawls
Hi all! I'm running two separate campaign crawls at the moment that I'm having a few issues with. The first is http://www.muchbetteradventures.com/. I'm tracking this at root to try an pick any problems with a previous version of the site, which is on the v1.muchbetteradventures subdomain. However, I'm only seeing 100 or so pages as being crawled in Moz, compared to thousands in Google. There's no access blocked alerts in Moz either. The second is http://www.sothebys.com/. I started this crawl at root a few days ago and no pages at all have been processed. Very odd! Any advice would be much appreciated.
Product Support | | neooptic0 -
Duplicate Page Content Report on Moz - Still ranking in Google Results?
Hi, I am experiencing 2 issues with the Duplicate Page Content in Moz. Every week it is notifying me of new duplicate content - so it seems to be missing duplicate content each week and the crawl is never above the 5000-6000 page mark, which means that it is under 10K which is the limit, so presumably everything is crawled in that one go so surely it should detect all of the dupe content on 1 crawl as opposed to having to do various crawls to detect it no? The dupe content report shows me pages that indeed have duplicate content but after checking, these are ranking on Google for their terms... ? Is the duplicate page content report incremental ? Will it add more duplicate content and increase the report every week or does it just show that week's results? If so, it's a bit like chasing my tail... Help!
Product Support | | bjs20100