Site down: "high CPU usage is due to the large traffic generated from Moz
-
My client's site is down and the web host gives says that Moz is the reason why.
"The fact that your site was limited is because the traffic generated by Moz. This is why I have suggested to block their IP addresses."
Now we have unblocked the IP addresses and as you can see your site was limited again. And again the :
Code:
54.224.139.99 - - [26/Oct/2017:16:00:43 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/ HTTP/1.0" 200 58551 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:02 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/ HTTP/1.0" 200 58521 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:16 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_ HTTP/1.0" 301 - "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:30 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/ HTTP/1.0" 200 58528 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)""Please check with Moz if they can reduce the rate your site is crawled. Only after you confirm that the rate is decreased we will remove the limit imposed on your account."
NOTE: Can you resolve this?
NOTE: I have achieved the campaign at this time in an effort to keep the site live.
-
I received a Moz prompt to check out if this question was answered yet.
Any chance that you have had a response from your development team?
The client's site is still archived. I am looking forward (dare I hope?) to those awesome reports coming again soon.
-
One other question, if I may add it.
I have several additional clients with the same host and at the same server tier.
Why is this one site only affected?
I am glad it is only one. However, if something can be tweaked to prevent a pending occurrence with others, I would be grateful to know.
Thanks in advance for your time and research.
-
Hi Dave,
I appreciate your working on this as we love your weekly reports and find them a huge value.
I already archived the campaign. So we are one very eager client of yours for news of a full solution. If needed, this is for https://www.smiledesigndentistry.com/
Thank you, JH
-
Hey Jeannie, Dave from the Help Team here.
Sorry about the trouble! Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file.
Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files. I'm so sorry about that! Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time.
One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this:
User-agent: rogerbot
Disallow: *I am going to show this conversation to our development team as well. Again, sorry for any inconvenience - please do let me know if there's anything else I can help with!
-
Moz, please advise.
Here is a followup from the web host:
"I also would like to inform you all the CPU is drained from moz bots once again. You can verify thi9s from the below output:
Code:
10123 54.163.0.158 rogerbot/1.2 https://moz.com/help/guides/moz-proc
454 54.144.199.191 rogerbot/1.1 http://moz.com/help/guides/search-ovIn order to resolve the problem I would ask you to contact https://moz.com/ again and ask them to reduce their agent Crawl rate:
rogerbot/1.2
Thank you for your cooperation in advance.
For your convenience, I will leave this ticket with status Awaiting Moz/client reply.
Best Regards, Technical Support Team"
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
"Branded Reports" not branded email
I am a new Moz customer and specifically chose the $149 plan to get "branded reports" My Logo shows on the pdf report but when that report is emailed to the client there is a big black banner at the top of the email that says MOZ! I chose branded reports so that the customer would see my branding not MOZ. This is very frustrating and there does not seem to be a solution from MOZ, I know that you can send the email with my return email address but why do you have to put the MOZ branding in the email.
Product Support | | GEWEB1 -
Moz Company & Personal Account Conflict
Hey Moz - I just had a question regarding the Moz Pro Account. Sometimes I find myself torn between using my personal Moz account, and the Moz Pro Account associated with my company (and my team). Is there a way to link a personal account WITH a company pro account, so I could post as myself, build my own cred, but still enjoy the benefits of Pro on my company's dime? Thanks!
Product Support | | Blenny0 -
My Whole Moz Account Went Missing - Started From Scratch But Not Working Again
Hi guys my whole moz account went awol - lost all my previous campaign data in terms of search listings - got a bit fed up so started from scratch, the first week was ok but now I login to see where I am on the search engines and all I get is '-' can anyone help - I am paying top dollar for this service but it seems so unstable, I thought I was using this to save me time keying in hundreds of search phrases and manually checking them and as I am sure many of you will appreciate checking daily can drive you crackers so I signed up with the belief that once a week I would get an in depth report and as such I could take a sigh of relief and get away from the obsessive compulsion to check my listings daily - this does not seem to be the case - pretty frustrated to be honest. kind regards Gareth
Product Support | | GarethParkin0 -
Moz Pro Free trial
Hi i got a coupon code from a hosting company for Moz seo 4 month free trial however after registering and entering my information i was charged 99$ even though it said todays charge = $ 0 i would like a refund as it doens't seem to a be a free trial
Product Support | | zemosdesign0 -
Has the Moz website update affected my data?
Hi Moz, I can see you are currently updating your website and regularly get either the new or old moz format when I log into my account. When I am in my account and Moz is in the new format my account info including keyword ranking & external followed links dramatically change. I am just trying to investigate if it is a moz issue with data or something else? Help would be appreciated. Thanks a lot, Beth
Product Support | | cmuknbb0 -
Traffic Data Not Showing
Our traffic data is not being reported but the GA account is linked and correct. The error in the report is "campaign does not have valid GA connection" yet it does. The only thing I did is change the campaign name but Moz is picking that change up.
Product Support | | JessiK1 -
Anyone else receiving several duplicate Moz Weekly Rankings Report today?
Anyone else receiving several duplicate Moz Weekly Rankings Report today? I think I have received about 15-20 email duplicate reports on my weekly rankings. David
Product Support | | David-E-Carey0 -
A problem with the MOZ New response to Question Email
Hi, It seems that the MOZ New response to Question emails are getting my mailbox stuck (using outlook). This does not happen with any other email including your other reports and emails. Maybe its only me but it happens to me on two active questions I have with you (2 different new response emails I got). Thanks
Product Support | | BeytzNet1