Adjusting SEOmoz Crawling Speed
-
How do you adjust the SEOmoz crawling speed?
SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
-
Hello,
You can add a crawl delay to your robots.txt file.
It would look something like this:
User-Agent: rogerbot
Crawl-Delay: 5(time is in seconds between page requests)
Hope that helps!
Kenny
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is my MOZ report only crawling 1 page?
just got this weeks MOZ report and it states that it have only crawled: Pages Crawled: 1 | Limit: 10,000 it was over 1000 a couple of weeks ago, we have moved servers recently but is there anything i have done wrong here? indigocarhire.co.uk thanks
Moz Pro | | RGOnline0 -
Setting a campaign in SEOmoz
Hi All, when i set a new campaign i am asked to decide if to use: domin-name.com Or www.domain-name.com Can someone please explain the different in terms of the campaign If i use: domain-name.com - will the campaign run on www.domain-name.com too? Thank you SEOwise
Moz Pro | | iivgi0 -
Why my seomoz keyword ranking are different to google SERPs
I use Cute Rank to check my keywords ranking in Google, but after last update seomoz is showing some of keyword ranked much higher than Cute Rank. I checked them manually and they are on positions showed by Cute Rank and not by Seomoz. Can someone explain me why is that?
Moz Pro | | misterdam0 -
SEOMOZ Crawl Test
Guys I really have an issue that i know have but cannot see if that makes sense. Basically 3 months ago i did a site wide 301 from economyleasinguk.co.uk to www.economy-car-leasing.co.uk Every thing looks good get all the correct header responses , all canonicals work perfectly , Google webmaster tools is updated fetch as google bot shows the old site is 301 I tried the seomoz crawl test today on the old domain and got this message Oh no! Looks like the page you were trying to access is temporarily down which at first thought ok because the site was not there it wont do it on an old 301 domain, however i tried it on a domain i know has just been 301'd and i got this message The URL http://www.site1.com/ redirects to http://site2.com/. Do you want to crawl http://site2.com/ instead?
Moz Pro | | kellymandingo
Would you like to:
Continue with www.site1.com
Continue with site2.com I really do not know what to do, its either the redirect script is missing something however its doing what it should or the server is a problem but again its doing what it should so why would SEOMOZ not be able to crawl the old URL like it example site above. Now the strange thing is Open Site Explorer does see the 301 and asks if i want to check the new URL instead Ps the redirect is done using PHP redirect which i am asking him to change to a htaccess as its now on a apache server and was wondering if this could be an issue, all pages go to correct pages as requested Thanks in Advance1 -
How current is the data reflected in the SEOmoz toolbar?
Been using the SEOmoz toolbar for three weeks, yet nothing on my site seems to have changed in terms of rankings and metrics. How new or "fresh" is the data that the toolbar uses to come up with PA, mR, mT, DA, DmR, and DmT? How often does it update?
Moz Pro | | wlefevre0 -
If there a basic overview of the SEOMoz Pro tools?
I'm surprised there is no simple overview of all the features anywhere, or a guide on how to use the tools? There's a nice intro video, and a good beginners guide, then BAM! - straight into the Pro Tools (it's a bit daunting for someone learning!) Or at least that I can find... and help or pointers would be great 🙂
Moz Pro | | seanuk0 -
FB Shares in SEOmoz Keyword Analysis Tool
Hello everybody, I'm quite confused in how to understand "FB Shares" in SEOmoz Keyword Analysis Tool Report. As You may know, this section is under report's "Social Signals" category. Does this number counts whole FB shares of some website links by ALL FB users? Or is it a number of some website FB account posts? Some clearance would be very helpful.. The same question applies to other social signal reports: Tweets, Google+ shares. Thank You for your answer! Happy New Year!
Moz Pro | | jkundrotas0 -
Errors on my Crawl Diagnostics
I have 51 errors on my Crawl Diagnostics tool.46 are 4xx Client Error.Those 4xx errors are links to products (or categories) that we are not selling them any more so there are inactive on the website but Google still have the links. How can I tell Google not to index them?. Can those errors (and warnings) could be harming my rankings (they went down from position 1 to 4 for the most important keywords) thanks,
Moz Pro | | cardif0