Site down: "high CPU usage is due to the large traffic generated from Moz
-
My client's site is down and the web host gives says that Moz is the reason why.
"The fact that your site was limited is because the traffic generated by Moz. This is why I have suggested to block their IP addresses."
Now we have unblocked the IP addresses and as you can see your site was limited again. And again the :
Code:
54.224.139.99 - - [26/Oct/2017:16:00:43 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/ HTTP/1.0" 200 58551 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:02 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/ HTTP/1.0" 200 58521 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:16 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_ HTTP/1.0" 301 - "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)"
54.224.139.99 - - [26/Oct/2017:16:01:30 -0500] "GET /amp/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/page/2/@Smile_Design_/@Smile_Design_/page/2/@Smile_Design_/ HTTP/1.0" 200 58528 "-" "rogerbot/1.1 (http://moz.com/help/guides/search-overview/crawl-diagnostics#more-help, rogerbot-crawler@moz.com)""Please check with Moz if they can reduce the rate your site is crawled. Only after you confirm that the rate is decreased we will remove the limit imposed on your account."
NOTE: Can you resolve this?
NOTE: I have achieved the campaign at this time in an effort to keep the site live.
-
I received a Moz prompt to check out if this question was answered yet.
Any chance that you have had a response from your development team?
The client's site is still archived. I am looking forward (dare I hope?) to those awesome reports coming again soon.
-
One other question, if I may add it.
I have several additional clients with the same host and at the same server tier.
Why is this one site only affected?
I am glad it is only one. However, if something can be tweaked to prevent a pending occurrence with others, I would be grateful to know.
Thanks in advance for your time and research.
-
Hi Dave,
I appreciate your working on this as we love your weekly reports and find them a huge value.
I already archived the campaign. So we are one very eager client of yours for news of a full solution. If needed, this is for https://www.smiledesigndentistry.com/
Thank you, JH
-
Hey Jeannie, Dave from the Help Team here.
Sorry about the trouble! Typically to resolve this we'd recommend adding a crawl delay for rogerbot to your robots.txt file.
Unfortunately, at this time, our crawler isn't able to follow crawl delay directives in robots.txt files. I'm so sorry about that! Our engineering team is aware of the issue and are working towards a resolution however I don't have an ETA at this time.
One possible solution in the interim while we work towards a fix here would be to archive the campaign so we stop crawling it, and then when we have the ability to follow crawl delays, you could un-archive your campaign. You could also block rogerbot altogether in your robots.txt file - it would look like this:
User-agent: rogerbot
Disallow: *I am going to show this conversation to our development team as well. Again, sorry for any inconvenience - please do let me know if there's anything else I can help with!
-
Moz, please advise.
Here is a followup from the web host:
"I also would like to inform you all the CPU is drained from moz bots once again. You can verify thi9s from the below output:
Code:
10123 54.163.0.158 rogerbot/1.2 https://moz.com/help/guides/moz-proc
454 54.144.199.191 rogerbot/1.1 http://moz.com/help/guides/search-ovIn order to resolve the problem I would ask you to contact https://moz.com/ again and ask them to reduce their agent Crawl rate:
rogerbot/1.2
Thank you for your cooperation in advance.
For your convenience, I will leave this ticket with status Awaiting Moz/client reply.
Best Regards, Technical Support Team"
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved How can I create a custom gift box using MOZ's products, and what are some creative ideas for designing the packaging?
Creating a custom gift box using MOZ's products can be a fun and thoughtful way to personalize a gift. Here's a step-by-step guide on how to do it, along with some creative packaging ideas: Step 1: Select the Products
Product Support | | MeltonMark8888
Browse MOZ's product catalog and select the items you want to include in the gift box. Consider the recipient's preferences, interests, and the occasion for which you're creating the gift box. Step 2: Choose the Box
Select a Galaxy Gift Box that will fit the products comfortably. It can be a plain cardboard box or a decorative one, depending on the overall aesthetic you're aiming for. Make sure the box is sturdy enough to protect the contents. Step 3: Gather Materials
Besides the products, gather materials for packaging, such as tissue paper, bubble wrap, or shredded paper for cushioning. You'll also need ribbon, twine, or other decorative elements for the finishing touches. Step 4: Arrange the Products
Arrange the chosen products inside the anniversary gifts for men . Place larger items at the back and smaller items at the front for visibility. Ensure that the arrangement looks aesthetically pleasing and well-balanced. Step 5: Add Filler
Add your chosen filler material to protect the products during shipping and to enhance the presentation. You can use colored tissue paper, shredded paper, or bubble wrap.
Gift Box Max0 -
Unsolved No replies from help@moz.com - one of our IPs is blocked by Cloudflare so we cannot access Moz Community from there
Hi all,
Product Support | | DanielDL
I am a bit at my wits end trying to get some acknowledgement from MOZ. Have had no replies, no ticket auto-replies, no updates on any of the messages I have sent via the Moz Help Form on the website. Literally nothing. I wanted to avoid having to post publicly, but does anyone know how to raise a "technical problem" ticket with MOZ? help@moz.com never replies and the Help Form doesn't generate any kind of ticket. From our main office we get an "Access denied" Error (via Cloudflare) specifically for the Moz Community area. This happened to us in February of this year and has been happening again all through May. After testing ourselves with our IT, we determine that MOZ's Cloudflare account has incorrectly blocked the dedicated IP address specific to the internet connection at our head office. This means that none of our Moz User accounts can access anything related to the Community area in our account when working at the studio. We can only do so when working remotely (ie. some other IP address). This is incredibly frustrating, particularly as we've been on a proper paid MOZ account for many years. And I have sent numerous email requests, messages via the Form, etc., and have never heard back from anyone at all. The problem has been on-going for some time and I guess it is my fault because I tried to politely wait a fair amount of time between each follow-up. Only to realize that, actually, I don't think anyone is monitoring help@moz.com or even the Form submissions, or are even looking into the issue for me. Am hoping this message is seen by someone at Moz so they can let me know what is going on please? Guys..... c'mon.....0 -
Moz rank tracking with new subpage address
Hello, I have built a new website and the subpages web address are new. I haven't made a redirect yet but I am about to do it. How do I continue keeping track of my ranking evolution in moz. Do I need to indicate the new web address ? or once the redirect will be done will moz continue the tracking of my keywords showing the old web address but tracking the new address ? Thank you,
Product Support | | seoanalytics0 -
Moz has stopped crawling my site
Hello, We have been using Moz Pro for over a year on our site and in the last month noticed that our site is not being crawled anymore. I took a look in Google Search Console, and everything seems to be fine there, so I think it is just the Moz tool that is not working. Has anyone else experienced this? Are there any tips for troubleshooting it? Thank you, Adam
Product Support | | cwells0 -
How to use moz to find spam backlinks
I am new to MOZ pro. I have lots of spam backlinks from a company that blackhatted me about 5 years ago. Can MOZ list these for me? I don't know how. Thanks
Product Support | | TomHW0 -
Keyword rankings "soon"?
On the insights portion of one of my campaigns, I continually see this message: " You will see rankings for the 100 keywords we found for you very soon. In the meantime, add more keywords for [XXXXX].It has said "very soon" for quite some time. How do I get this section to work?
Product Support | | seo-nicole0 -
On Page Optimization and Page Grader in Moz Needs Some Fixes
Okay, so I've brought this up a couple times to your team and have always been met with resistance and confusion, but I really feel like the fundamental design of your on page grader (the auto one) needs some changes in order to be really useful. Let's start with the basics. An on page grader is supposed to be a tool where you can enter your keyword or keywords and have it scan the URL to see how well the keyword has been implemented on the page. It checks for heading use, anchor text, alt tags, frequency of use, etc. We all should get this if we are here. Now, your manual page grader does just that and does a great job of it. I find it very useful for checking newly created pages, landing pages, competitor pages and the like. However, your auto version found in the Search > On-Page Optimizations is the culprit here. So, according to several staff at Moz, this tool scans your site every week when the rest of your site scan is done. It then takes the rankings of pages for ranked and tracked keywords and assigns a grade for that page. Here's the major problem.... in almost all of my accounts, it very rarely gets the association with keyword and URL correct. Thus, it returns a low page grade even though the grade of the URL would be an A or B for the correct keyword. What seems to be happening is I'm tracking multiple keywords, sometime close variations. This grade tool in a lot of cases is grading a page more than once for several keywords it's ranking for. All the variants it gets a low grade for because the exact keywords isn't in the title, headings, etc. The second thing that seems to be happening is some pages are not being graded using the primary keyword, but instead a variation. In this case it only shows low grades for the page. On one occurrence of trying to discuss this issue with you, I got scoffed at for saying the tool was "guessing at which keyword to use for the grade". Your staff was very adamant in saying "it's not a guess!" My counter point was, call it what you want, it's in accurate to the intended keyword to be optimized for. Now, when one first gets to this tool, all of the above doesn't seem like that big of a deal because there's an "add & manage page grade" section. This section allows you to add and manage page grades. So, one would think that you could go to this tool, add an page grades for keywords that the tool didn't automatically associate and delete ones that it added with an incorrect association and that would be the end of it. However, that's NOT quite the end of it. At your own admission, the tool doesn't function like that. Each week when a new scan is completed, these pages grades essentially reset. So any page grade that you deleted will simply show right back up again. Now, before I go any further, let me explain why this is such an issue for me. First, on page optimization is something my company does and sells as a service and we are pretty good at it. Not saying we're the best, but we're better than most. That said, one of my reasons for choosing Moz was because according to the verbiage on the pages, one can create branded reports to include on page optimization. However, when the tool doesn't get the right keyword to landing page AND there's no way to manually override it, then basically all of my reports make it look like my pages grades suck, when in face they don't. Now, if in my campaigns I only added the exact keyword for each page and didn't do any variations of, then this tools becomes much more accurate. However, I want to track multiple variations. Bottom line is this: I need a page grading and reporting tool I can count on and actually use. Thus far, you solution is riddled with problems and every time I bring it up, you fight me on it saying it's working as it's supposed to. What I'm saying here is "as it's supposed to" is fundamentally wrong. I'd like to discuss this with someone at this point, and preferably someone a bit higher up than a first tier support person because frankly, I'm not getting anywhere. Overall I love your service, but it's things like this that have me questioning wether or not Moz is right. I'm giving you opportunity after opportunity to prove to me that it is. Help me work this out so we can continue that trend. Thanks, Micha
Product Support | | Multiverse-Media-Group4 -
My Whole Moz Account Went Missing - Started From Scratch But Not Working Again
Hi guys my whole moz account went awol - lost all my previous campaign data in terms of search listings - got a bit fed up so started from scratch, the first week was ok but now I login to see where I am on the search engines and all I get is '-' can anyone help - I am paying top dollar for this service but it seems so unstable, I thought I was using this to save me time keying in hundreds of search phrases and manually checking them and as I am sure many of you will appreciate checking daily can drive you crackers so I signed up with the belief that once a week I would get an in depth report and as such I could take a sigh of relief and get away from the obsessive compulsion to check my listings daily - this does not seem to be the case - pretty frustrated to be honest. kind regards Gareth
Product Support | | GarethParkin0
