Adjusting SEOmoz Crawling Speed
-
How do you adjust the SEOmoz crawling speed?
SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
-
Hello,
You can add a crawl delay to your robots.txt file.
It would look something like this:
User-Agent: rogerbot
Crawl-Delay: 5(time is in seconds between page requests)
Hope that helps!
Kenny
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Diagnostics - Historical Summary
As we've been fixing errors on our website, the crawl diagnostic graphs have been showing great results (top left to bottom right for errors). The problem is the graphs themselves aren't very pretty. I can't use them in my internal reports (all internal reports are standardised colours/formats). Is there anyway of exporting the top level summary with historic data so the graphs can be recreated in company colours? I don't want the detailed CSV breakdown of what errors occurred, but rather than on X date there were Y errors, the next month Z errors and so forth. The data must already be in the SEOMoz system in order to create the graphs themselves - I was hoping this can be made available to us if it isn't already? Does anyone know if there is already a way of doing this? I've tried to 'inspect element' and find the underlying data in the source code but to no avail, and can't see any exports that would do this. Thanks in advance Dean
Moz Pro | | FashionLux0 -
SEOmoz giving duplicate content that does not exist.
My problem is similar, and SEOmoz add campaign is giving me several pag. Duplicate, and he's giving me links pag. That do not exist. Look below. My site has 115 pages and the extent SEMOZ gave me 250. Duplicate Page Content ... pages / Alexandra / Clarisse / Clarisse.html
Moz Pro | | Slash-RJ
... pages / Alexandra / Clarisse / Clarisse / Clarisse.html
... pages / Alexandra / Clarisse / Clarisse / Clarisse / Clarisse.html
.... pages / Alexandra / Clarisse / Clarisse / Clarisse / Lizie / Lizie.html When the verade this link does not exist, there is only. ... pages / Alexandra / Alexandra.html
... pages / Clarisse / Clarissehtml
And so on. How to Solve?0 -
Do you know if there is an problem with the rank Tracker on SEOmoz
My rank tracker is stating that I am not in the top 50 for my keyword which is auto insurance fremont ca. I check on bing and yahoo and i am listed as #2. Also on Google it states that my rank has gone down by 2 place but I acutally increase one from spot 12 to spot 11. Is anyone else having the same issue??
Moz Pro | | Greenpeak0 -
SEOmoz tool Issue?
Hi Mozzers, I am doing a web maintenance task for a client and it's been weeks that Moz is detecting 49 duplicate pages ( contact page). I thought resolving the issue when creating the xml sitemap and excluding those duplicates. The moz tool would still detect them, so I went in making a search with some of these duplicate to check if they were indexed but non of them were indexed. So my question is has anyone recently experienced similar issues? Is the moz tool not 100% accurate? Thanks for sharing your thoughts and answers
Moz Pro | | Ideas-Money-Art0 -
Can Google see all the pages that an seomoz crawl picks up?
Hi there My client's site is showing around 90 pages indexed in Google. The seomoz crawl is returning 1934 pages. Many of the pages in the crawl are duplicates, but there are also pages which are behind the user login. Is it theoretically correct to say that if a seomoz crawl finds all the pages, then Google has the potential to as well, even if they choose not to index? Or would Google not see the pages behind the login? And how come seomoz can see the pages? Many thanks in anticipation! Wendy
Moz Pro | | Chammy0 -
Why is the SEOmoz customer service on this site so awful?
Half the time the tools don't work and there isn't any accountability. Has anyone else had the same experience?
Moz Pro | | wpsoule1 -
Crawl Diagnostics Report
I'm a bit concerned about the results I'm getting from the Crawl Diagnostics Report. I've updated the site with canonical urls to remove duplicate content and when I check the site - it all displays the right values, but the report, which has just finished crawling is still showing a lot of pages as duplicate content. Simple example: http://www.domain.com http://www.domain.com/ Both of them are in the duplicate content section although both have canonical url set as: Does each crawl check the entire site from the beginning or just the pages it didn't have a chance to crawl the last time? This is just one of 333 duplicate content pages, which have canonical url pointing to the right page. Can someone please explain?
Moz Pro | | coremediadesign0
