Site down: "high CPU usage is due to the large traffic generated from Moz
-
My client's site is down and the web host gives says that Moz is the reason why.
"The fact that your site was limited is because the traffic generated by Moz. This is why I have suggested to block their IP addresses."
Now we have unblocked the IP addresses and as you can see your site was limited again. And again the :
Code:
54.224.139.99 - - [26/Oct/2017:16:00:43 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/ HTTP/1.0" 200 58551 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:02 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/ HTTP/1.0" 200 58521 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:16 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_ HTTP/1.0" 301 - "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:30 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/ HTTP/1.0" 200 58528 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)""Please check with Moz if they can reduce the rate your site is crawled. Only after you confirm that the rate is decreased we will remove the limit imposed on your account."
NOTE: Can you resolve this?
NOTE: I have achieved the campaign at this time in an effort to keep the site live.
-
I received a Moz prompt to check out if this question was answered yet.
Any chance that you have had a response from your development team?
The client's site is still archived. I am looking forward (dare I hope?) to those awesome reports coming again soon.
-
One other question, if I may add it.
I have several additional clients with the same host and at the same server tier.
Why is this one site only affected?
I am glad it is only one. However, if something can be tweaked to prevent a pending occurrence with others, I would be grateful to know.
Thanks in advance for your time and research.
-
Hi Dave,
I appreciate your working on this as we love your weekly reports and find them a huge value.
I already archived the campaign. So we are one very eager client of yours for news of a full solution. If needed, this is for https://www.smiledesigndentistry.com/
Thank you, JH
-
Hey Jeannie, Dave from the Help Team here.
Sorry about the trouble! Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file.
Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files. I'm so sorry about that! Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time.
One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this:
User-agent: rogerbot
Disallow: *I am going to show this conversation to our development team as well. Again, sorry for any inconvenience - please do let me know if there's anything else I can help with!
-
Moz, please advise.
Here is a followup from the web host:
"I also would like to inform you all the CPU is drained from moz bots once again. You can verify thi9s from the below output:
Code:
10123 54.163.0.158 rogerbot/1.2 https://moz.com/help/guides/moz-proc
454 54.144.199.191 rogerbot/1.1 http://moz.com/help/guides/search-ovIn order to resolve the problem I would ask you to contact https://moz.com/ again and ask them to reduce their agent Crawl rate:
rogerbot/1.2
Thank you for your cooperation in advance.
For your convenience, I will leave this ticket with status Awaiting Moz/client reply.
Best Regards, Technical Support Team"
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Issue with MOZ Keyword Tracking: Missing Keywords in Report
Hello, I’m encountering an issue with MOZ’s keyword tracking tool. We have correctly set up our website: https://recomendacionpersonal.com/ to track multiple keywords, but the tool only seems to track a few of them. Some important keywords we’re targeting don’t appear in the report at all. We are in the trial phase and need to ensure the tool functions correctly for our SEO strategy before subscribing to a premium plan. Can someone please help us understand why this might be happening and how to fix it? Thank you!
Product Support | | gabo12120 -
Unsolved Moz lesson question re: Keyword Matrix brainstorm
Hello We are going through the exercise of keyword matrix and want to know if we can get someone to listen in for 5-10 min to make sure we are on the right track. Thank you - Brad
Product Support | | B2daGz0 -
Unsolved Moz doesn't find backlinks for months / link tracking list offline
Hi there
Product Support | | startrek123
I want to share a big problem with you. Moz tracking list seems to be offline or having trouble finding links. I have an example:
The page where the backlink has been placed was created months ago. As you can see on the picture:
Link to the page is http://social.msdn.microsoft.com/profile/tableauxdumonde
Backlink on that page is https://ggongtime.com/ Picture shows clearly that the page of backlink (Microsoft) has been indexed and already has got it's own backlinks in Moz (8) but it is still not linking to the backlink website (ggongtime.com).
I've also attached a picture of page source where you can clearly see that it is a html link and is dofollow.
The problem remains since 3 months and I want to find out what is going on. This Microsoft page is one of example. There are hundreds of other pages being found, or not, and are still not showing up in Link Explorer on website's portfolio.
I want to know if there is a technical problem with link explorer, or anything else, because this is a serious problem for me. moz problem.png0 -
Moz Campaign did not fully update this week
This is not a new campaign. It updates every Thursday but this week did not update some sections like:
Product Support | | alliancehvac
Search Visibility
Desktop vs. Mobile Search Visibility
Top National Keywords
Top Local Keywords
Rankings0 -
Is Google Analytics Connection with Moz broken?
Hi, I keep on getting this error on my moz campaigns "Our connection to your Google Analytics account was lost. Don't worry, you won't lose any data. Please reauthorize now." Everyday i click the reauthorize now and make sure everything is sync properly, but I keep on getting the same error everyday, is there something wrong with GA connection with Moz? I really need urgent help with this problem as I have more than 15 campaigns on moz and this tool is very important for my reporting. Thanks
Product Support | | Josh.B0 -
How do I access my Full SERP Analysis Report in the new Moz?
Another possibly dumb question... I ran a full SERP analysis report and had it emailed to me, but in the email there's no link to the report. I logged back into Moz, but I can't find the report anywhere. Thanks! BiDFIvG
Product Support | | ScottShrum1 -
Moz crawls
Hi all! I'm running two separate campaign crawls at the moment that I'm having a few issues with. The first is http://www.muchbetteradventures.com/. I'm tracking this at root to try an pick any problems with a previous version of the site, which is on the v1.muchbetteradventures subdomain. However, I'm only seeing 100 or so pages as being crawled in Moz, compared to thousands in Google. There's no access blocked alerts in Moz either. The second is http://www.sothebys.com/. I started this crawl at root a few days ago and no pages at all have been processed. Very odd! Any advice would be much appreciated.
Product Support | | neooptic0 -
Moz forum response seems to crash Outlook?
Hi, Like the new look of Moz. I got an automated email the other day saying someone had responded to a question we asked on Moz. However, when opening the email, it really held Outlook 2010 up. So slow that it crashed Outlook a couple of times. When trying to forward the email to a colleague, I got the 'Contacting \d1...cloudfront.net\img_users' dialogue box, which just stood there for ages, and eventually I had to give up and force Outlook to quit. Please see attached screenshot. Looking at the URL Outlook was trying to reach indicates it's trying to load up the avatar of the person who responded. Which would look nice I admit, but don't really need it in the email if it's going to cause Outlook to crash. There is an 'image not found' icon next to the post, so that's why I assume Outlook is searching for that image. It's almost as if the image is incorrectly referencing a local source (network drive) which it can never find. Perhaps there is a missing http:// prefix before the image in the tag? Anyway, was curious to know if anyone else has this problem, or if anyone at Moz can check this out (update template?) Regards, Rich M6AY0GQ
Product Support | | STL2