Unsolved The Moz.com bot is overloading my server
-
. How to solve it?
-
For a step-by-step guide on setting up the tool, check out the solara executor download tutorial.
-
maybe crawl delay will help.
-
@paulavervo
Hi,
We do! The best way to chat with us is via our contact form or direct email. We also have chat within Moz Pro.
Please contact us via help@moz.com or https://moz.com/help/contact
We will be happy to help.
Cheers,
Kerry. -
very nice brother i like it very good keep it up !
-
very nice !
-
does the moz team even monitor this forum?
-
If the Moz.com bot is overloading your server, there are several steps you can take to manage and mitigate the issue effectively. First, you can adjust the crawl rate in your
robots.txt
file by specifying a crawl delay for the Moz bot using directives likeUser-agent: rogerbot
andUser-agent: dotbot
, followed byCrawl-delay: 10
to make the bot wait 10 seconds between requests. If this does not suffice, you can temporarily block the bot by disallowing it in yourrobots.txt
file. Additionally, it's a good idea to contact Moz’s support team to explain the issue, as they may offer solutions to adjust the crawl rate for your site. Implementing server-side rate limiting is another effective strategy. For Apache servers, you can add rules in your.htaccess
file to return a 429 Too Many Requests status code to the Moz bots, while for Nginx servers, you can set up rate limiting in your configuration file to control the number of requests per second from a single user or IP address. Monitoring your server’s performance and log files can help identify specific patterns or peak times, allowing you to fine-tune your settings. Furthermore, using a Content Delivery Network (CDN) can help distribute the load by caching content and serving it from multiple locations, reducing the direct impact on your server caused by crawlers. By taking these steps, you can manage the load from the Moz.com bot and maintain your server’s stability and responsiveness.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Rogerbot directives in robots.txt
I feel like I spend a lot of time setting false positives in my reports to ignore. Can I prevent Rogerbot from crawling pages I don't care about with robots.txt directives? For example., I have some page types with meta noindex and it reports these to me. Theoretically, I can block Rogerbot from these with a robots,txt directive and not have to deal with false positives.
Reporting & Analytics | | awilliams_kingston0 -
Unsolved Why is'nt there an update yet!
0 -
Moz Reporting a brand page as KW stuffed for the the brand KW, not a problem i presume ?
Hi if I'm getting a keyword stuffing warning on Moz for a brand page for the brand name kw that shouldn't really matter should it? Moz just oversimplifying it since seen the brand name says 27 times on a page think it's stuffed, but if its a 2000 word page all about the brand, those occurrences are totally natural & hence Google will think so too generally speaking? Thanks Dan
Moz Pro | | Dan-Lawrence0 -
302 Status Code for .com but 200 Status Code for .com/
I ran a moz crawl report for one of my sites. For mywebsite.com I'm getting a 302 HTTP Status Code but for mywebsite.com/ I'm getting a 200 HTTP Status Code. mywebsite.com -Title tag "blank" - Meta Description "blank" - HTTP Status Code "302" - Referrer "blank" - Content Type Header "blank" - 301/302 target "/" - Rel-Canonical Target "blank" mywebsite.com/ - Title tag "correct tag" - Meta Description "correct tag" - HTTP Status Code "200" - Referrer "mysite.com" - Content Type Header "text/html" - 301/302 target "blank" - Rel-Canonical Target "mysite.com/" Will this cause any issues with the search engines? Is there a way to fix this? I'm on wordpress and use Yoast SEO premium plugin. I have sites ranking in many different niches and have done similar white hat SEO for all of my others. They are ranking in competitive niches but for some reason this site is in the 200-300s for basically all keywords in not a very competitive niche. It has excellent content, links and no penalties. I'm trying to find anything that could be the cause of this. Thanks! <colgroup><col width="64"></colgroup>
Moz Pro | | cccreationsinc
| Rel-Canonical Target |0 -
Possible Crawling Problem with Screaming Frog and Moz Crawlers
So I'm not sure if what I'm seeing is a problem or not. As of about two weeks ago the Moz crawler has only been able to see www.mysite.com, and none of the links, content, title, ect associated with the page. Essentially the report has one line, what should be the homepage, but it's not able to pull any information from the page but does show a 200 http status code. The report shows nothing blocked by robots or any errors. When I use screaming frog to crawl the site about 75% of the time it just reports one line www.mysite.com with a 200 status code, but again the crawler is not able to actually see the html. The other 25% of the time it works perfectly fine, crawls all pages and sees all meta info and content. There are no errors in Google WMT and everything looks ok there. We have seen a traffic drop the last two weeks but I don't know if this is the reason for it. I can't publicly post the page but if someone has an idea of what might be going on I'd be happy to PM them. Thanks
Moz Pro | | CJ50 -
Moz email is freezing Microsoft Outlook
When I try and open an email message from a Moz Q&A response, it hangs Microsoft Outlook for more than 30 seconds. Is anyone else having this problem?
Moz Pro | | ChristopherGlaeser0 -
Seomoz toolbar - Server location
Hello, We target french countries and it seems IP server location is important. So we took some hosting at Planete Hoster (located in Canada) with French IP, but our seomoz toolbar says our website is located in Canada 😞 See: http://www.aupaindantan.fr/ Could you please explain? do we need to change the hoster and to take one in France or this one is ok? Thank you for your answer
Moz Pro | | ahernoux0