Site down: "high CPU usage is due to the large traffic generated from Moz
-
My client's site is down and the web host gives says that Moz is the reason why.
"The fact that your site was limited is because the traffic generated by Moz. This is why I have suggested to block their IP addresses."
Now we have unblocked the IP addresses and as you can see your site was limited again. And again the :
Code:
54.224.139.99 - - [26/Oct/2017:16:00:43 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/ HTTP/1.0" 200 58551 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:02 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/ HTTP/1.0" 200 58521 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:16 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_ HTTP/1.0" 301 - "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:30 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/ HTTP/1.0" 200 58528 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)""Please check with Moz if they can reduce the rate your site is crawled. Only after you confirm that the rate is decreased we will remove the limit imposed on your account."
NOTE: Can you resolve this?
NOTE: I have achieved the campaign at this time in an effort to keep the site live.
-
I received a Moz prompt to check out if this question was answered yet.
Any chance that you have had a response from your development team?
The client's site is still archived. I am looking forward (dare I hope?) to those awesome reports coming again soon.
-
One other question, if I may add it.
I have several additional clients with the same host and at the same server tier.
Why is this one site only affected?
I am glad it is only one. However, if something can be tweaked to prevent a pending occurrence with others, I would be grateful to know.
Thanks in advance for your time and research.
-
Hi Dave,
I appreciate your working on this as we love your weekly reports and find them a huge value.
I already archived the campaign. So we are one very eager client of yours for news of a full solution. If needed, this is for https://www.smiledesigndentistry.com/
Thank you, JH
-
Hey Jeannie, Dave from the Help Team here.
Sorry about the trouble! Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file.
Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files. I'm so sorry about that! Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time.
One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this:
User-agent: rogerbot
Disallow: *I am going to show this conversation to our development team as well. Again, sorry for any inconvenience - please do let me know if there's anything else I can help with!
-
Moz, please advise.
Here is a followup from the web host:
"I also would like to inform you all the CPU is drained from moz bots once again. You can verify thi9s from the below output:
Code:
10123 54.163.0.158 rogerbot/1.2 https://moz.com/help/guides/moz-proc
454 54.144.199.191 rogerbot/1.1 http://moz.com/help/guides/search-ovIn order to resolve the problem I would ask you to contact https://moz.com/ again and ask them to reduce their agent Crawl rate:
rogerbot/1.2
Thank you for your cooperation in advance.
For your convenience, I will leave this ticket with status Awaiting Moz/client reply.
Best Regards, Technical Support Team"
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Campaign did not fully update this week
This is not a new campaign. It updates every Thursday but this week did not update some sections like:
Product Support | | alliancehvac
Search Visibility
Desktop vs. Mobile Search Visibility
Top National Keywords
Top Local Keywords
Rankings0 -
Site DA and PA is not increasing
Hy, My site best water softener DA is Still 1. I am Publishing the content regularly, Site age is 6 months But Still DA and PA are 1. Why its not increasing. Is there any way to increase DA and PA Of my site?
Product Support | | ikzayahot0 -
Merging two Sites into One - how is Final Domain Authority Calculated?
Hi, If we merge two sites say, one with DA 35 and another 20 what will be resulting DA of the merged site? Secondly, will the result vary based on which Domain is kept and which one is 301ed? ps :- all content is transferred from one to another using 301. Regards,
Product Support | | Praveenshah10 -
Why have I been charged by Moz for a Pro Subscription?
I recently used the trial free 1 month subscription for Moz Pro as it was suggested in a continued education course I was doing. My email was hacked and I had no access to it for which I had no access to Moz and while getting my email sorted, I got this message that they have charged me and it has come off my account for next month. They also took the money off my account which is extremely difficult for me as I am a student and cannot afford an expense as such. This is creating havoc for me. Currently Moz Pro a service charged in US$ is too expensive for me in India as a student. Any advice on how to get this amount reversed. Today is the day the invoice was sent and immediately afterwards they charged me the amount.
Product Support | | Adilene0 -
Moz usability
Why is it such a hassle to access Moz campaigns from the frontpage of Moz? How can MOZ be so awesome and at the same time be so annoying? Does Moz have UX designer working with them?
Product Support | | guidetoiceland0 -
Is there a way to alphabetize the "Manage your Campaigns" screen?
We just went up to 100 campaigns and can't seem to find a way to alphabetize our list of campaigns. Anybody have a fix for sorting those alphabetically?
Product Support | | BKraff0 -
Moz Analytics Reports Scheduling
Hi, We have been using Moz now for a few weeks now and we are trying to run reports based on various time frames and can't seem to do so? For example on the if we needed a report from say 17/04/2014- 30/04/2014 how would we achieve this? Or if we wanted to schedule reports based on a calendar month- how do we achieve this? I tried running reports on the 1/05/2014 and received reports for Apr 25 - May 2, 2014. Also on the DASHBAORD- the timeframe only appears to display 'WEEKLY' and the last week of the month- 23th April- 30- Apr. Or will this change once we have more historical data compiled? Thanks,
Product Support | | Faxem0 -
Moz Pro Account
How many campaigns can you have before you max out your Moz pro account?
Product Support | | alblaze200