After fixing Crawl Errors, how long does it take to for Moz or Google to re-crawl a website?
-
Last night I found out through Moz that my robots.txt file was blocking any crawling of my website. I fixed the issue. Now do I just sit and wait?
-
Moz will crawl the site every week so you'll need to wait a few days for that to happen.
In the meantime you could try using the Moz Crawl tool to do a manual crawl. Depending on the size of your site this can take a few days so you might find it's just worth waiting for the weekly crawl. You can find the crawl tool here:
https://moz.com/researchtools/crawl-test
If you site isn't included in Google's index because it was inaccessible due to the robots,txt, then you'll want to go into Google Webmaster Tools and use the Crawl -> Fetch as Googlebot to kickstart Google crawling/indexing your site.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Moz Pro Warning: Redirect Chain
I have just signed up to a Moz Pro account, and after it finished crawling my website it gave me a warning about a redirection chain. http://elementpaints.com >> https://elementpIaints.com >> https://www.elementpaints.com I'm trying to find some more information about how to fix this problem but I'm not having much luck. This article I found even says it is not a problem: https://really-simple-ssl.com/knowledge-base/avoid-landing-page-redirects So now I'm even more confused, do I still need to fix this? If so, how do I do that? FYI: I have Lets Encrypt SSL cert installed on my server, and I'm using Cloudflare with Full SSL option and HSTS enabled, and "Always Use HTTPS" option is turned on. | http://elementpaints.com |
Getting Started | | elementpaints0 -
Standard Syntax in robots.txt doesn't prevent Moz bot from crawling
A client is getting many false positive site crawl errors for things like duplicate titles and duplicate content on pages that include /tag/ in the URL. An example is https://needquest.com/place_tag/autism-spectrum-disorder/page/4/ To resolve this we have set up a disallow statement in the robots.txt file that says
Getting Started | | btreloar
Disallow: /page/ For some reason this appears not to work, as the site crawl errors continue to list pages like this. Does anyone understand why that would be and what we need to do to properly disallow crawling these pages?0 -
Setting up Moz Pro campaign for a subfolder
Hi, I'm just setting up my first campaign and would like some advice please. I have a website and also within it a a number of pages in a sub folder. These are technically a different business to the main website and I would like to track them separately in Moz so they can have their own keyword lists and competitors etc. How do I do this? Do I set up two separate campaigns www.site.com
Getting Started | | craigramsay
www.site.com/subfolder I understand the first one might still report on the subfolder but I guess I could just ignore this. Thanks0 -
Long tail searches
I maintain a site that provides access to bankruptcy records. I want to try an optimize the site for searches involving the case name and generic terms like "bankruptcy filing". So, the keyword would be "(case name) bankruptcy filing". How do I use Moz to help when the number of case names is high (100,000+)? Any other suggestions for what to do when working on long tail matches?
Getting Started | | troller0 -
MOZ custom report is spamming my inbox
Hello, There seems to be a bug regarding MOZ's customized report. I set it up as a weekly report but i received up to 3 emails (4-5 hours interval) with incomplete data. Kindly fix. I'm using gmail. Thanks!
Getting Started | | leecourt0 -
Moz Staff: History + Watch List
Hi All, I have been going crazy trying to find a thread I read last week ... I think!
Getting Started | | Mark_Ch
Several search attempts later, no closer to finding the thread in question. A genius moment or simple frustration got me thinking. How cool would it be to have the following features on MOZ Q&A Forum. Watch List
1] Check off any threads that you would like to monitor over time. Should you stumble upon a thread of value then simply add to watch list and view it's progress over time. No searching for the thread in question. History
2] Keep a history off all threads viewed over a period of time. Thanks Mark0 -
Everything is empty, no stats displayed for 3 days. How long it takes for data to appear?
Everything in my Moz Analytics campaign is empty. Shows either "-" or "0". How long does it take for some data to appear? 3 days ago I have set up the Campaign, connected Google Analytics, Set up keywords, etc. Every day I receive 4 copies of same identical email from Moz, "New Crawl Complete for ..." I go to moz analytics and nothing has changed. Still zeroes and no data. So Moz Analytics doesn't do anything, doesn't show any stats, sends multiple bogus emails. Looks like moz analytics is totally broken? Not even beta quality. Yet to sign up I had to provide my credit card.
Getting Started | | romanr0 -
Lost since the change over to Moz
I was recently lead to a series of videos which looked to be quite helpful here http://moz.com/help/pro/introduction But is seems with the change over from SEO Moz to Moz, nothing is in the same place. Should I bother watching them or wait for new ones? I'm particularly interested in SEO to raise our rank.
Getting Started | | AliciaMarie0