Unsolved The Moz.com bot is overloading my server
-
. How to solve it? -
For a step-by-step guide on setting up the tool, check out the solara executor download tutorial.
-
maybe crawl delay will help.
-
@paulavervo
Hi,
We do! The best way to chat with us is via our contact form or direct email. We also have chat within Moz Pro.
Please contact us via help@moz.com or https://moz.com/help/contact
We will be happy to help.
Cheers,
Kerry. -
very nice brother i like it very good keep it up !
-
very nice !
-
does the moz team even monitor this forum?
-
If the Moz.com bot is overloading your server, there are several steps you can take to manage and mitigate the issue effectively. First, you can adjust the crawl rate in your
robots.txtfile by specifying a crawl delay for the Moz bot using directives likeUser-agent: rogerbotandUser-agent: dotbot, followed byCrawl-delay: 10to make the bot wait 10 seconds between requests. If this does not suffice, you can temporarily block the bot by disallowing it in yourrobots.txtfile. Additionally, it's a good idea to contact Moz’s support team to explain the issue, as they may offer solutions to adjust the crawl rate for your site. Implementing server-side rate limiting is another effective strategy. For Apache servers, you can add rules in your.htaccessfile to return a 429 Too Many Requests status code to the Moz bots, while for Nginx servers, you can set up rate limiting in your configuration file to control the number of requests per second from a single user or IP address. Monitoring your server’s performance and log files can help identify specific patterns or peak times, allowing you to fine-tune your settings. Furthermore, using a Content Delivery Network (CDN) can help distribute the load by caching content and serving it from multiple locations, reducing the direct impact on your server caused by crawlers. By taking these steps, you can manage the load from the Moz.com bot and maintain your server’s stability and responsiveness.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
520 Error from crawl report with Cloudflare
I am getting a lot of 520 Server Error in crawl reports. I see this is related to Cloudflare. We know 520 is Cloudflare so maybe the Moz team can change this from "unknown" to "Cloudflare 520". Perhaps the Moz team can update the "how to fix" section in the reporting, if they have some possible suggestions on how to avoid seeing these in the report of if there is a real issue that needs to be addressed. At this point I don't know. There must be a solution that Moz can provide like a setting in Cloudflare that will permit the Rogerbot if Cloudflare is blocking it because it does not like its behavior or something. It could be that Rogerbot is crawling my site on a bad day or at a time when we were deploying a massive site change. If I know when my site will be down can I pause Rogerbot? I found this https://developers.cloudflare.com/support/troubleshooting/general-troubleshooting/troubleshooting-crawl-errors/
Technical SEO | | awilliams_kingston0 -
Moz crawler is not able to crawl my website
Hi, i need help regarding Moz Can't Crawl Your Site i also share screenshot that Moz was unable to crawl your site on Mar 26, 2022. Our crawler was not able to access the robots.txt file on your site. This often occurs because of a server error from the robots.txt. Although this may have been caused by a temporary outage, we recommend making sure your robots.txt file is accessible and that your network and server are working correctly. Typically errors like this should be investigated and fixed by the site webmaster.
Technical SEO | | JasonTorney
my robts.txt also ok i checked it
Here is my website https://whiskcreative.com.au
just check it please as soon as possibe0 -
MOZ Pro - Page Grader Queries
Greetings! I started a new position last week and the extremely supportive director promised to give me anything I required to make my job easier. Of course, my first port of call was MOZ Pro. Having never used MOZ Pro before, I've just been getting to grips with it, fixing any pressing issues and giving the whole site a general SEO health check. A few fairly major issues have been flagged, which I'm in the process of fixing, and I'm currently putting our main landing pages through the MOZ Page Grader. After a little bit of tinkering, our iPhone 6 cases page has been graded B for the term 'iPhone 6 cases', but I have a few queries/concerns regarding some of the suggested fixes: **Avoid keyword stuffing in document **- The term 'iphone 6 cases' only appears thrice in the body, so it can't be that? The term, however, appears 23 times in the page's img alt tags. Could this be the issue? This is an ecommerce site that sells iPhone 6 cases, so the img alt tags are bound to contain that keyword. Each img alt tag is unique, so I don't really know what I can do here? "Show details for YouSave iPhone 6 0.6mm Clear Gel Case" is the example of the img alt tag of one of the products on that page, surely I can't remove the words 'iPhone 6 case'? Avoid too many internal links - MOZ suggests keeping the internal links to below 100 or, at a minimum, less than 100 links on the main navigation menu. I haven't counted, but I'd guess that page has more than 100 links, but not too many on the navigation menu. To me, this looks like a standard ecommerce page, with links to products and different pages via the top and bottom menus. Would I improve visibility if I reduced the amount of links by, say, reducing the number of products on the page? We currently have it set to 36, but can easily be reduced. Only One Canonical URL - We've put a fix in place for this issue and are just waiting for it to go live. For some reason rel=canonical tags have been duplicated on the majority of the pages. Like I say, this is being remedied, but I just wondered whether a duplicate tag negatively affect the page's visibility? The tags are identical and just point to the page they're on. I think that's about it for now! Thanks in advance and keep up the good work! Cheers, Lewis (Andrew is the name of the director) UPDATE Now I've sorted the rel=canonical issue, the pages are being graded A but still with the first two suggestions above.
Moz Pro | | PeaSoupDigital0 -
I have a new website and would like to use the keywords my competitors are using and was wondering the best way to take a look using Moz Pro?
The website is www.fapwrap.com - it's a new adult wall decals website. [editor's note: may not be safe for some workplaces] The wall decals we have are obviously a bit more niche than the bigger competitors, but I want to get into the first page to see how it helps us. I'm slowly taking over some kws like "xbox wraps" and "ps4 wraps" but those have low search volume and I feel like "wall decals" will help - though it's highly competitive. Suggestions? I just kinda want to plug in my competition and pick from their list of existing kws..... Ha.
Moz Pro | | JohnnyRoq0 -
How could I improve my "brand" and "mentions" using the Moz tool?
Hello I am a bit stuck on what to put into this section of MOZ. As you can see from the attached image I have setup to catch any mention of my business name and root url. But beyond that I am uncertain of what to use to monitor who is talking about us. I'm also uncertain of what other abilities or usefulness this function would have. aOq7wVg.png
Moz Pro | | infinart0 -
Moz report says missing info but it appears to be there
Recently SEOmoz erros are saying all of my pages are missing meta and page titles. but when i view source, they appear fine, why?
Moz Pro | | briananglin0 -
Question About Moz Reports
Hey guys Is it possible to change the date/time that SEOmoz runs its ranking reports? I put together ranking reports each Monday which I need to feedback to the business and it would be good if I could schedule these for late Sunday or early Monday mornings for me coming in to the office.
Moz Pro | | EwanFisher0
