Unsolved The Moz.com bot is overloading my server
-
. How to solve it?
-
maybe crawl delay will help.
-
@paulavervo
Hi,
We do! The best way to chat with us is via our contact form or direct email. We also have chat within Moz Pro.
Please contact us via help@moz.com or https://moz.com/help/contact
We will be happy to help.
Cheers,
Kerry. -
very nice brother i like it very good keep it up !
-
very nice !
-
does the moz team even monitor this forum?
-
If the Moz.com bot is overloading your server, there are several steps you can take to manage and mitigate the issue effectively. First, you can adjust the crawl rate in your
robots.txt
file by specifying a crawl delay for the Moz bot using directives likeUser-agent: rogerbot
andUser-agent: dotbot
, followed byCrawl-delay: 10
to make the bot wait 10 seconds between requests. If this does not suffice, you can temporarily block the bot by disallowing it in yourrobots.txt
file. Additionally, it's a good idea to contact Moz’s support team to explain the issue, as they may offer solutions to adjust the crawl rate for your site. Implementing server-side rate limiting is another effective strategy. For Apache servers, you can add rules in your.htaccess
file to return a 429 Too Many Requests status code to the Moz bots, while for Nginx servers, you can set up rate limiting in your configuration file to control the number of requests per second from a single user or IP address. Monitoring your server’s performance and log files can help identify specific patterns or peak times, allowing you to fine-tune your settings. Furthermore, using a Content Delivery Network (CDN) can help distribute the load by caching content and serving it from multiple locations, reducing the direct impact on your server caused by crawlers. By taking these steps, you can manage the load from the Moz.com bot and maintain your server’s stability and responsiveness.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved MOZ Crawler Stalled
MOZ crawl has been waiting since April 2nd. Whats the deal? I am not able to "recrawl", so i have been stuck in limbo.
Product Support | | WebMarkets0 -
Unsolved Why is'nt there an update yet!
0 -
Moz crawler is not able to crawl my website
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
Technical SEO | | JasonTorney
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe0 -
Google results not close to matching MOZ PA and SA - Why?
I'm trying to convince a boss and another client/partner of the value of MOZ, but it's a challenge when Google's results and the MOZ analytics are way off. For example, when I do non-personalized (MOZ toolbar) Google search for "stair lift kansas city" I see some results that make sense. Further down the page I see my prospective client, kclift.com -- it's a new site with poor MOZ grades and almost no backlinks. It has a PA of 1 and a DA of 4. Yet it ranks above a YP site that has a DA of 89, and another site that has a PA of 27 and a DA of 14. What gives? When I used an in-private Bing search for those terms this site didn't display in the 1st 4 pages. That makes sense with these MOZ rankings and the new site. So how does the site get page 1 results on Google for the same terms? (I should note that I've gone to this site a lot, for the client, and done other related searches. I live in Charlotte, which is why I've included Kansas City in the search terms. With this search, I've tried several ways to do a Google search that is truly anonymous, but I'm not sure I've succeeded. I've erased browsing data, used incognito, used other browsers, and used the MOZ non-personalized Google search. All from the same IP though. Does Google track that? If so, how can one do a truly anonymous search on Google?) All suggestions welcome. Thanks! Richard S.
Moz Pro | | RFS550 -
404 error for unknown URL that Moz is finding in our blog
I'm receiving 404 errors on my site crawl for messinastaffing.com. They seem to be generating only from our blog posts which sit on Hubspot. I've searched high and low and can't identify why our site URL is being added at the end - I've tried every link in our blog and cannot repeat the error the crawl is finding. For instance: Referer is: http://blog.messinastaffing.com/take-charge-career-story-compelling-cover-letter/ 404 error is: http://blog.messinastaffing.com/take-charge-career-story-compelling-cover-letter/www.messinastaffing.com I agree that the 404 error URL doesn't exist but I can't identify where Moz is finding it. I have approximately 75 of these errors - one for every blog on our site. Beth Morley Vice President, Operations Messina Group Staffing Solutions
Moz Pro | | MessinaGroup
(847) 692-0613 www.messinastaffing.com0 -
Do we get "Removal of "nofollow" from first custom URL on profile" when we cross 200 Moz Points? I have not received it yet, anything I can do?
Though I have only recently subscribed to Moz Pro, I have been using Moz Blog for quite some time. I recently crossed 200 Moz Points. As per Moz Points, it says "Removal of "nofollow" from first custom URL on profile" for crossing 200 points. I still dont see any links from Moz when I am using OSE. Can anyone suggest what i need to do?
Moz Pro | | vinodh-spintadigital2 -
Keyword Stuffing - MOZ On-Page Grader
We sell a great number of insulation products, many of which are produced by individual manufacturers. On the page identified below the Keyword "Kingspan" is repeated numerous times as these items are included in our online shop. However, the many mentions of Kingspan are recorded in the HTML5 Source Code, rather than an external database. When I used the MOZ On-Page Grader, using the keywords "Kingspan" I was surprised to achieve an "A" Grade! I know I shouldn't be complaining, but I am wondering why the significant repetition of the word "Kingspan" has not negatively impacted my score? http://www.just-insulation.com/001-eshop/buy-kingspan-thermapitch-thermawall-thermafloor-insulation-boards.html
Moz Pro | | JustInsulation0 -
Crawl test. Bot crawled only 200 or so links when it should have crawled thousands
Hi everyone, I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
Moz Pro | | Ev840