Adjusting SEOmoz Crawling Speed
-
How do you adjust the SEOmoz crawling speed?
SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
-
Hello,
You can add a crawl delay to your robots.txt file.
It would look something like this:
User-Agent: rogerbot
Crawl-Delay: 5(time is in seconds between page requests)
Hope that helps!
Kenny
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz and HubSpot SSL - crawl error?
I'm getting an error message when Moz tries to crawl my site, however when I check in Google Search Console, they return no errors. Our site is hosted on HubSpot. Is Moz still having trouble crawling HubSpot sites that have enabled their SSL? I read an article that this should have been corrected in early 2017, but I'm getting an error.
Moz Pro | | jennygriffin0 -
Seomoz shows issue with canconical
Hi When I use the on-page research tool seomoz tells me I have an issue with the rel canconical tag pointing to the wrong url, but I have it set so that on this particular page it points to itself (as per recommendation from seomoz) full url is http://www.growingyourownveg.com/how-to-grow/garlic.html in head section i have <base href="http://www.growingyourownveg.com/"> Have I got this wrong? Google and Bing appear to accept it OK Thanks
Moz Pro | | spes1230 -
Sharing/Transferring SEOMOZ campaigns
What happens when you are no longer working on a campaign, and your client needs the campaign history? I don't pay for my account, my employer does. If I ever have to move on what should I do with the account? I would want to keep my seomoz profile and Q&A history. But the campaigns belong to my employer so I would have to change all of the profile settings into his name… Is there a protocol for this kind of situation?
Moz Pro | | BeTheBoss0 -
SEOmoz campaign - keyword rankings...
Hi there, I have a quick question: Does it make a difference if I add keywords to a campaign with the '[' and ']' brackets included? I just realized that I had copy/pasted many of my keywords from the exported CSV from the Google AdWords Keyword Tool and I fear that the keyword rankings on SEOmoz might factor the brackets as part of the keywords themselves. Thanks in advance! Ben
Moz Pro | | BenHammer0 -
Edit SEOmoz Profile Bug
Hi,
Moz Pro | | rayvensoft
I am trying to complete my SEOmoz profile, but there seems to be a bug. I am able to fill out everything except the 'favorite topics'. It lets me go in and fill it out, but when I click the update button to save it it does not save, and still shows up as missing in bar at the top. I have tried it in Mozilla, IE and Chrome, and it still does not work. Am I missing something? Thanks.0 -
SEOMOZ Crawler unicode bug
for the last couple of weeks the SEOMOZ crawls my homepage only and gets 4xx error for most of the URL's. the crawler have no issues with English url's only with the unicode(Hebrew) ones. this is what is see in the csv export for the crawl (one sample) : http://www.funstuff.co.il/׳ž׳¡׳™׳‘׳×-׳¨׳•׳•׳§׳•׳× 404 text/html; charset=utf-8 you can see that the URL is Gibberish please help.
Moz Pro | | AsafY0 -
Dynamic URL pages in Crawl Diagnostics
The crawl diagnostic has found errors for pages that do not exist within the site. These pages do not appear in the SERPs and are seemingly dynamic URL pages. Most of the URLs that appear are formatted http://mysite.com/keyword,%20_keyword_,%20key_word_/ which appear as dynamic URLs for potential search phrases within the site. The other popular variety among these pages have a URL format of http://mysite.com/tag/keyword/filename.xml?sort=filter which are only generated by a filter utility on the site. These pages comprise about 90% of 401 errors, duplicate page content/title, overly-dynamic URL, missing meta decription tag, etc. Many of the same pages appear for multiple errors/warnings/notices categories. So, why are these pages being received into the crawl test? and how to I stop it to gauge for a better analysis of my site via SEOmoz?
Moz Pro | | Visually0