Crawl started on the 18th of December and still hasn't completed!
-
As the title states I started a crawl over 2 weeks ago now and it still hasn't completed. Does anyone know why this could be?
Thanks
-
Thanks I'll give that a go!
-
Hi Laurance,
Thanks for writing in and sorry that your crawl never completed. I'm afraid the issue is that all crawlers are being blocked in the robots.txt file of the site and we aren't able to fully access the site to complete a crawl. If you were to update the robots.txt file to allow our crawler to access the site, we would than be able to get the crawl completed for you. You can use the user-agent rogerbot to specify our crawler or you can remove the disallow: / directive from the robots.txt file to allow us access. Once you've done that, let me know and I will have our engineers complete the crawl for you.
I look forward to hearing back soon.
-Chiaryn
-
Hi Laurence,
What do you mean the crawl still hasn't be completed? Do you mean not all your sites are indexed?
I suggest you checking up on your Google Webmaster Tools and see how your site is doing. Once you log in, click on your website then click on the Health tab on the left hand side. You can see all your crawl stats and errors there. Furthermore, you can use the "Fetch as Google" tool and ask Google to crawl your pages again.
Another thing is that if you have a lot of pages and your pages are not interlinking correctly, it may take weeks for Google to crawl all your pages. There is not set time limit on how long it takes. Maybe submit a sitemap to speed up the process of finding all your pages?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why doesn't the Historical PageRank tool work
example: http://www.seomoz.org/toolbox/pagerank?wurl=www.google.com turns no results for me. what's up? thanks!
Moz Pro | | chiappa0 -
Why does Crawl Diagnostics report this as duplicate content?
Hi guys, we've been addressing a duplicate content problem on our site over the past few weeks. Lately, we've implemented rel canonical tags in various parts of our ecommerce store, over time, and observing the effects by both tracking changes in SEOMoz and Websmater tools. Although our duplicate content errors are definitely decreasing, I can't help but wonder why some URLs are still being flagged with duplicate content by our SEOmoz crawler. Here's an example, taken directly from our Crawl Diagnostics Report: URL with 4 Duplicate Content errors:
Moz Pro | | yacpro13
/safety-lights.html Duplicate content URLs:
/safety-lights.html ?cat=78&price=-100
/safety-lights.html?cat=78&dir=desc&order=position /safety-lights.html?cat=78 /safety-lights.html?manufacturer=514 What I don't understand, is all of the URLS with URL parameters have a rel canonical tag pointing to the 'real' URL
/safety-lights.html So why is SEOMoz crawler still flagging this as duplicate content?0 -
Slowing down SEOmoz Crawl Rate
Is there a way to slow down SEOmoz crawl rate? My site is pretty huge and I'm getting 10k pages crawled every week, which is great. However I sometimes get multiple page requests in one second which slows down my site a bit. If this feature exists I couldn't find it, if it doesn't, it's a great idea to have, in a similar way to how Googlebot do it. Thanks.
Moz Pro | | corwin0 -
Is Rank Tracker still down?
I have been trying to use Rank Tracker but it still appears to be down. The following message is displayed which dates from 5th September and suggests it would take around a week to get it working. "Due to a server failure, we are experiencing a delay in Rank Tracker results this week. Unfortunately, it may take up to a week to get it working properly again. Thanks for your patience and understanding; our engineers are working around the clock to get this issue fixed. - Updated September 5th." It is now 17th September and I am still unable to use it. Just wondering whether this is the same situation for all or whether SEOMoz have an update on when we can expect this to be up and running. Thanks.
Moz Pro | | simon_realbuzz0 -
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Moz Pro | | john_Digino0 -
Why can't I find uipl?
Hi guys, I'm working with the API at the moment. I have most of the information I want, but I cannot find the uipl in any of the returned data using several methods. The methods I have tried are: http://lsapi.seomoz.com/linkscape/links/www.google.co.uk?Scope=page_to_domain&AccessID=xxx&Expires=xxx&Signature=xxx http://lsapi.seomoz.com/linkscape/url-metrics/www.google.co.uk?AccessID=xxx&Expires=xxx&Signature=xxx I can't seem to find how to get the uipl. Any ideas? Thanks in advance!
Moz Pro | | ClickConsult0 -
Crawl Test produced only 1 page
Hi, I recently submitted a crawl for www.cirrato.com using SEOMoz Crawl Test Tool. I have a lot of pages, but the crawl result shows only 1 page, which is the front page and nothing else... Does anyone know what this could mean or what the problem is?
Moz Pro | | yusufcirrato0 -
Crawl test. Bot crawled only 200 or so links when it should have crawled thousands
Hi everyone, I just recieved my crawl test report and its only given me 200 or so URL's when my site has thousands, any thoughts?
Moz Pro | | Ev840