Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Adjusting SEOmoz Crawling Speed

Moz Pro

867

cappuccino89 last edited by

How do you adjust the SEOmoz crawling speed?

SEOmoz tried to crawl 10,000 pages in 3 hours and crashed our MySQL server.
1 Reply Last reply
Reply Quote 1
kenneth_martin last edited by

Hello,

You can add a crawl delay to your robots.txt file.

It would look something like this:

User-Agent: rogerbot
Crawl-Delay: 5

(time is in seconds between page requests)

Hope that helps!

Kenny
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Tips on speeding up disavow process and still doing it right

Hello, I'm doing a 10-step disavow check using this article, this word document, and this corresponding spreadsheet template. Doing a disavow for a non-manual penguin penalty (paid links, doorway sites) It's taking a long time. Can you give me some pointers of how to speed things up? I want to do it right but I need to know how I can do it faster. Like tricks on domains to automatically disavow, what not to check, whatever tips you have. I'm using Cognitive SEO and manual checking. I'd like to avoid pricy tools. I'm on a mac. Thanks!
Moz Pro | | BobGW

0
Why seomoz crawler does not see my snapshot?

I have a web app that uses angularJS and the content is all dynamic (SPA). I have generated snapshots for the pages and write a rule to redirect ( 301) to the snapshot in case of find escaped_fragment in the URL. E.g http://plure.com/#!/imoveis/venda/rj/rio-de-janeiro Request: http://plure.com/?escaped_fragment=/imoveis/venda/rj/rio-de-janeiro is redirected to: http://plure.com/snapshots/imoveis/venda/rj/rio-de-janeiro/ The snapshot is a headless page generated by PhantomJS. Even following the guideline ( https://developers.google.com/webmasters/ajax-crawling/docs/specification) I still can't see my page crawled and I also in SEOMoz I can only see the 1st page crawled with no dynamic content on it. Am I doing something wrong? SEOMoz was supposed to get the snapshot based on same rules of GoogleBot or SEOMoz does not get snapshots?
Moz Pro | | plure_seo

0
How to exclude a specific subdomain from SEOMoz Campaign?

Forgive me if I've overlooked something obvious, but how can I exclude a subdomain from a campaign? I want to crawl/analyze mywebsite.com, but not subdomain.mywebsite.com Thanks in advance...
Moz Pro | | ahockley

1
How can I see the URL's affected in Seomoz Crawl when Notices increase

Hi, When Seomoz crawled my site, my notices increased by 255. How can I only these affected urls ? thanks Sarah
Moz Pro | | SarahCollins

0
SEOmoz Rank tracking

I think this tool is a bit off : ) This morning I checked my ranking for a keyword and every time I recheck it I see my results in a different place. I checked on word 5 times and got three different results. Is Google brocken or is the tracker doing its own thing? This is not the first time I am seeing this issue only today its really acting up. IMHO I think it would be best if the tool just says out of order 🙂 rather than 3, 10, 6 for one keyword phrase within ten minutes...
Moz Pro | | SEODinosaur

0
Seomoz crawling filtered pages

Hi, I just checked an seo campaign we started last week, so I opened seomoz to see the crawl diagnostics. Lot's of duplicate content & duplicate titles showing up, but that's because Rogerbot is crawling all of the filtered pages as well. How do I exclude these pages from being crawled? /product/brand-x/3969?order=brand&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=ASC
/product/brand-x/3969?order=popular&sortorder=DESC&page=10
/product/brand-x/3969?order=popular&sortorder=DESC&page=11
Moz Pro | | nvs.nim

0
How long does it typically take MOZBOT to crawl a site?

Our site has had "crawl in progress" for over 24 hours now without an update, we're dying for the results since our last changes :).
Moz Pro | | absoauto

0
Seomoz staff help needed

I have this new client new domain new website, www.ard.uk.com when seo moz crawls the site it says it has 24 million links mozrank of over 7, looks amazing but impossible can someone as seomoz have a look please
Moz Pro | | francesco-285016

0