Websites First Crawl - Over 2 Hour Suggested Wait
-
Hello SEOMoz!
We recently signed up for a free trial and on the pro dashboard it states the following.
"To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days."
It's been nearly 24 hours and we see no results under Crawl Diagnostics however we do under rankings.
Is this normal?
Thanks
-
Thanks!
Last time I checked the robots.txt was there. Anyway I have rcreated this.
Will try creating a new campaign and seeing how long it takes.
Update: worked fine after around 10 minutes crawl!
-
Hi Louis,
You are right, the first starter crawl should be complete within the first couple of hours. One reason it might take longer than normal is if the SEOmoz crawler, rogerbot, has trouble crawling your site.
I took a look inside our system and it appears the website associated with your campaign doesn't have a robots.txt file. Because the SEOmoz crawler is extra polite about crawling, whenever it doesn't find a robots.txt file, this can prevent it from crawling at all.
To fix this, create a text file called "robots.txt" and with the following information in it, and place it at the root directory of your site.
User-agent: *
Disallow:Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt
Unfortunately, it may be too late to add this to fix your starter crawl. So here are some things you can try:
- After adding the robots.txt file, try creating another (duplicate) campaign and seeing if that works.
- Or wait until your next crawl
- Contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO.
-
Louis, I think it is just a standard message, so don't worry, I doubt it will take that long.
I just started one on a 30 page site, so I expect it will show results tomorrow.
-
So this means I will not get any data under crawl diagnostics for a week?
Why does it not give results for up to 250 pages as stated? The domain I selected for crawling has only 20 or so pages.
-
The full crawl diagnostics and rank reports are only generated once per week. So, I would give it at least 7 days before you see full data reported in your new campaign.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Only 2 Inbound Links which are the Redirect 301 Links
Hello Everyone, We're having a serious Problem with the ranking of our website so we started testing the website in different Tools. Almost all of the tools say our website works great and in comparision to our competitors we win in almost every test – but we rank behind them. The only thing that makes me wonder is the fact that our website has only 2 Inbound links (Open Site Explorer) when the link source is set to only internal. And the only links shown are 2 links that come from the old http (we're using https) and redirect 301 to our https. No other links – in other tools we have around 80 internal links to our rmain https-webadress. Does anyone have an idea where that comes from? I'm afraid this might be our issue in ranking behind of lots of our competitors. Lokking forward to your ideas and suggestions.
Moz Pro | | Franksen3o3
Greets Frank0 -
Campaign Crawl
I have a site with 8036 pages in my sitemap index. But the MozBot only Crawled 2169 pages. It's been several months and each week it crawls roughly the same number of pages. Any idea why I'm not getting fully crawled?
Moz Pro | | JMFieldMarketing0 -
"link_count" column in Crawl Diagnostics report
On the Crawl Diagnostics report, does "link_count" represent external (links to this URL), internal, both, or what ?
Moz Pro | | GlennFerrell0 -
Why does Crawl Diagnostics report this as duplicate content?
Hi guys, we've been addressing a duplicate content problem on our site over the past few weeks. Lately, we've implemented rel canonical tags in various parts of our ecommerce store, over time, and observing the effects by both tracking changes in SEOMoz and Websmater tools. Although our duplicate content errors are definitely decreasing, I can't help but wonder why some URLs are still being flagged with duplicate content by our SEOmoz crawler. Here's an example, taken directly from our Crawl Diagnostics Report: URL with 4 Duplicate Content errors:
Moz Pro | | yacpro13
/safety-lights.html Duplicate content URLs:
/safety-lights.html ?cat=78&price=-100
/safety-lights.html?cat=78&dir=desc&order=position /safety-lights.html?cat=78 /safety-lights.html?manufacturer=514 What I don't understand, is all of the URLS with URL parameters have a rel canonical tag pointing to the 'real' URL
/safety-lights.html So why is SEOMoz crawler still flagging this as duplicate content?0 -
How to remove URLS from from crawl diagnostics blocked by robots.txt
I suddenly have a huge jump in the number of errors in crawl diagnostics and it all seems to be down to a load of URLs that should be blocked by robots.txt. These have never appeared before, how do I remove them or stop them appearing again?
Moz Pro | | SimonBond0 -
Pages Crawled: 0 ?
I've been with SEO Moz for over a month and a half. Why would this weeks crawl have Pages Crawled: 0? I've made no changes since the crawl last week that had 10k pages crawled...
Moz Pro | | mr_w1 -
404 errors in SEOMoz crawl tool
I currently have several 404 errors in the latest crawls from SEOMoz. Here is an example of the error. http://dealerplatform.com/blog/2011/10/23/videos-for-auto-dealers/www.dealerplatform.com/ In all cases the error is a result of www.dealerplatform.com being added to the real url. Anyone seen this before? The site is a wordpress mutlisite. I don't see where this incorrect link is showing up anywhere on the website. Any advice would be helpful. Thanks
Moz Pro | | Chris_Gregory0