Site down: "high CPU usage is due to the large traffic generated from Moz
-
My client's site is down and the web host gives says that Moz is the reason why.
"The fact that your site was limited is because the traffic generated by Moz. This is why I have suggested to block their IP addresses."
Now we have unblocked the IP addresses and as you can see your site was limited again. And again the :
Code:
54.224.139.99 - - [26/Oct/2017:16:00:43 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/ HTTP/1.0" 200 58551 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:02 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/ HTTP/1.0" 200 58521 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:16 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_ HTTP/1.0" 301 - "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:30 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/ HTTP/1.0" 200 58528 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)""Please check with Moz if they can reduce the rate your site is crawled. Only after you confirm that the rate is decreased we will remove the limit imposed on your account."
NOTE: Can you resolve this?
NOTE: I have achieved the campaign at this time in an effort to keep the site live.
-
I received a Moz prompt to check out if this question was answered yet.
Any chance that you have had a response from your development team?
The client's site is still archived. I am looking forward (dare I hope?) to those awesome reports coming again soon.
-
One other question, if I may add it.
I have several additional clients with the same host and at the same server tier.
Why is this one site only affected?
I am glad it is only one. However, if something can be tweaked to prevent a pending occurrence with others, I would be grateful to know.
Thanks in advance for your time and research.
-
Hi Dave,
I appreciate your working on this as we love your weekly reports and find them a huge value.
I already archived the campaign. So we are one very eager client of yours for news of a full solution. If needed, this is for https://www.smiledesigndentistry.com/
Thank you, JH
-
Hey Jeannie, Dave from the Help Team here.
Sorry about the trouble! Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file.
Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files. I'm so sorry about that! Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time.
One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this:
User-agent: rogerbot
Disallow: *I am going to show this conversation to our development team as well. Again, sorry for any inconvenience - please do let me know if there's anything else I can help with!
-
Moz, please advise.
Here is a followup from the web host:
"I also would like to inform you all the CPU is drained from moz bots once again. You can verify thi9s from the below output:
Code:
10123 54.163.0.158 rogerbot/1.2 https://moz.com/help/guides/moz-proc
454 54.144.199.191 rogerbot/1.1 http://moz.com/help/guides/search-ovIn order to resolve the problem I would ask you to contact https://moz.com/ again and ask them to reduce their agent Crawl rate:
rogerbot/1.2
Thank you for your cooperation in advance.
For your convenience, I will leave this ticket with status Awaiting Moz/client reply.
Best Regards, Technical Support Team"
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved MOZ API to Google Data Studio
Dopes MOZ have anything like this? https://www.semrush.com/features/google-data-studio-connector/
Product Support | | WebMarkets0 -
Moz not tracking search traffic & not displaying correctly in pdf report
Hi For one of our clients search traffic has stopped tracking and there are no error msgs. For the same client, when you save the report to pdf format, it appears as per below?
Product Support | | Nikki9590 -
Our client changed their domain name. Can we replace without losing the data for the old domain in MOZ ? We are at our campaign limit would the old domain count as a campaign?
Our client changed their domain name. Can we replace it without the new domain without losing the data for the old domain in MOZ ? We are at our campaign limit would the old domain count as a campaign?
Product Support | | ThomasCenterInc1 -
How do I delete a campaign on my Moz Account?
I need to delete one of the campaigns I have currently and cannot figure out how. Anyone want to help? Thanks!
Product Support | | marissaRT1 -
I'm not canceling my MOZ subscription, but just so I know: if I want to cancel my MOZ subscription, can I do this per month, or per year?
If I want to cancel after six months, do I have to pay for the rest of the year or can I cancel per month?
Product Support | | Sophie-Kool0 -
Latest Moz reports (pro) not including competitive link analysis
Hi Pls advise when these have been regenerated once fixed or will they just be emailed out automatically ? My clients all after their New Year monthly & quarterly reports hence this info is needed urgently, pls advise ? Also when will reports start showing social data in the reports and also when will the data that currently shows on the dash start to be accurate ? (or are you leaving this 'as is' on pro since your migrating to Moz analytics ?) Many Thanks Dan
Product Support | | Dan-Lawrence0 -
Can't add "Custom links" in my Moz Profile
Hi, I was trying to add a custom link to my profile but I got an error message: "There was a problem saving your custom URLs" Can somebody look into it for me? Kind Regards
Product Support | | Laurensvda0 -
Duplicate Page Content Report on Moz - Still ranking in Google Results?
Hi, I am experiencing 2 issues with the Duplicate Page Content in Moz. Every week it is notifying me of new duplicate content - so it seems to be missing duplicate content each week and the crawl is never above the 5000-6000 page mark, which means that it is under 10K which is the limit, so presumably everything is crawled in that one go so surely it should detect all of the dupe content on 1 crawl as opposed to having to do various crawls to detect it no? The dupe content report shows me pages that indeed have duplicate content but after checking, these are ranking on Google for their terms... ? Is the duplicate page content report incremental ? Will it add more duplicate content and increase the report every week or does it just show that week's results? If so, it's a bit like chasing my tail... Help!
Product Support | | bjs20100