Crawl rate
-
How often does Moz crawl my website ?
(I have a number of issues I believe I have fixed, and wondered if there was a manual request to re-crawl ?)
Thanks.
Austin.
-
Many thanks
-
Perfect, thanks.
-
Thank you, Auston, I was unsure of the number of pages thanks for clearing that up
-
Hi Austin
All campaign updates occur once a week on the same day based on when the campaign was first created. The crawl tests Thomas suggested is helpful as well for instant crawls. Our crawl-test in research tools will limit to 3000 pages.
Cheers!
-
You asked "how often Moz crawls a site" I'm assuming you are asking about a campaign that would be once a week per a campaign & depending on the size/ number of pages of the site
See FAQ
https://moz.com/help/guides/moz-pro-overview/crawl-diagnostics
There are methods you could either create another campaign however my advice would be to use Moz simply crawl test it will let you crawl your site for issues immediately.
https://moz.com/researchtools/crawl-test
That should take care of it, however.
If for whatever reason that does not help some tools are free if your site has less than 500 pages you can use https://www.screamingfrog.co.uk/seo-spider/
Although to get what you want you most likely will have to spend money.
It sounds like crawl test will do what you want. If it does not still needs to be crawled.
Third-party tools like Screaming Frog SEO Spider Pro and deepcrawl.com Are instant and very helpful Additions to the fantastic tools Moz already offers.
I hope this helps,
Tom
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved What would the exact text be for robots.txt to stop Moz crawling a subdomain?
I need Moz to stop crawling a subdomain of my site, and am just checking what the exact text should be in the file to do this. I assume it would be: User-agent: Moz
Getting Started | | Simon-Plan
Disallow: / But just checking so I can tell the agency who will apply it, to avoid paying for their time with the incorrect text! Many thanks.0 -
Crawling issue
Hi, I have to set up a campaign for a webshop. This webshop is a subdomain itself. First question: The two subfolders I need to track are /nl_BE and /fr_BE. What is the best way to handle this? Shall I set up two different campaigns for each subfolder, or shall I just make one campaign and add tags to keywords? **Second question: **it seems like Moz can't crawl enough pages. There are no disallows in the robots.txt. Should I try putting the following at the top into my robots.txt? User-agent: rogerbot
Getting Started | | Mat_C
Disallow: Or is it because I want to crawl only a subdomain that it doesn't work? Thanks0 -
Moz Site Crawl can't index WIX sites
We've been attempting to work on some SEO for a new potential client however they are using a WIX site. We've noticed that Moz SEO tools will not index any WIX sites. e.g. https://www.sharonradisch.com/ (which is one of their case studies). Anyone seen this that can offer any advice? Thanks,
Getting Started | | monkeex
Mark2 -
Standard Syntax in robots.txt doesn't prevent Moz bot from crawling
A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/ To resolve this we have set up a disallow statement in the robots.txt file that says
Getting Started | | btreloar
Disallow: /page/ For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?0 -
Page Count per campaign - Crawl Usage 500,000 Pages
How to you find the page crawl count per campaign? I have 3 campaigns and Moz stats I have used 150,000 pages from 500,000. I want to check this. Thanks
Getting Started | | SJMDT0 -
After fixing Crawl Errors, how long does it take to for Moz or Google to re-crawl a website?
Last night I found out through Moz that my robots.txt file was blocking any crawling of my website. I fixed the issue. Now do I just sit and wait?
Getting Started | | cmc-interactive0 -
MOZ Starter Crawl not happeneing
Hi I added a new site 48hours + ago and the starter crawler has not even begun collecting data. Any help would be appreciated. cheers Isaac
Getting Started | | sodafizz0 -
Is there a way to force re-crawl?
1- Can we force the system to recrawl a website without having to wait for the scheduled date? 2- Please forgive me as I am new to this, but is "Moz Analytics" supposed to replace the "Pro"? Thank you!
Getting Started | | BlackTreeIT0