Adjusting SEOmoz Crawling Speed
-
How do you adjust the SEOmoz crawling speed?
SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
-
Hello,
You can add a crawl delay to your robots.txt file.
It would look something like this:
User-Agent: rogerbot
Crawl-Delay: 5(time is in seconds between page requests)
Hope that helps!
Kenny
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Not provided Data in SEOMoz reports
Hi How do SeoMoz reports deal with 'not provided' data I see my total visits from organic search for a month are same as total of both my branded and non branded keyword traffic combined yet GA is reporting 157 visits from non provided data so is SEOMoz being very clever and finding a way to decipher this not provided data and allocate it accordingly in the reports ? Or if not what ? Many Thanks Dan
Moz Pro | | Dan-Lawrence0 -
SEOMoz Private Message Settings
Hi all, Is there a way to be notified (to my personal email address) when I get sent a private message? Thanks,
Moz Pro | | Unity
Davinia0 -
SEOmoz tool Issue?
Hi Mozzers, I am doing a web maintenance task for a client and it's been weeks that Moz is detecting 49 duplicate pages ( contact page). I thought resolving the issue when creating the xml sitemap and excluding those duplicates. The moz tool would still detect them, so I went in making a search with some of these duplicate to check if they were indexed but non of them were indexed. So my question is has anyone recently experienced similar issues? Is the moz tool not 100% accurate? Thanks for sharing your thoughts and answers
Moz Pro | | Ideas-Money-Art0 -
Best way to improve your SEOMOZ avatar status ?
Whats the best way to improve your SEOMOZ avatar status and gain SEOMOZ brownie points? I notice some people have amazing status level such as Guru how is this possible? Surely That becomes a full time job in itself?
Moz Pro | | Bristolweb0 -
How to remove Duplicate content due to url parameters from SEOMoz Crawl Diagnostics
Hello all I'm currently getting back over 8000 crawl errors for duplicate content pages . Its a joomla site with virtuemart and 95% of the errors are for parameters in the url that the customer can use to filter products. Google is handling them fine under webmaster tools parameters but its pretty hard to find the other duplicate content issues in SEOMoz with all of these in the way. All of the problem parameters start with ?product_type_ Should i try and use the robot.txt to stop them from being crawled and if so what would be the best way to include them in the robot.txt Any help greatly appreciated.
Moz Pro | | dfeg0 -
Seomoz Toolbar - Mozilla 6.0
Hi there, when will the toolbar be compatible with Mozilla 6.0? Thank you for your feedback. BR, Ive
Moz Pro | | NeroSolutions0 -
Seomoz Customer Service does anyone feel let down
I would like to know if anyone else feels let down by seomoz and their customer service. I will give you a couple of examples why i feel let down. I contacted customer services over an issue a couple of weeks a go and never got an answer from them, and the second issue is the private question that you can ask. I asked many days ago and it was an important question and it has still not been answered. I am just wondering what we are paying for by using this site and if other customers of this service feel let down like me.
Moz Pro | | ClaireH-1848860 -
SEOMOZ tools are not helping!
So, I have a website www.mobikwik.com which provides mobile phone topup in India. I signed up for SEOMOZ Pro membership a little less than a month ago. For all our competitors , which rank above us in google as of today , I checked out two things a) competitive link analysis and b) on-page optimization. On b) , our home page had the best optimization for targeted keywords and the best score amongst all competitors who ranked above us. On a) , linkscape tools showed our moztrust and mozrank to be the best. Except for 2 competitors, our domain authority was also better than the rest. Inspite of all this, our website is ranked no more than 6 on google for all targeted keywords. We have the best quality links amongst our competitors. I am at a loss to understand this and how to improve our ranking. I have to make a decision whether to renew my SEOMOZ membership in 2 days time. Please help me decide!
Moz Pro | | mobikwik0