Websites First Crawl - Over 2 Hour Suggested Wait
-
Hello SEOMoz!
We recently signed up for a free trial and on the pro dashboard it states the following.
"To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days."
It's been nearly 24 hours and we see no results under Crawl Diagnostics however we do under rankings.
Is this normal?
Thanks
-
Thanks!
Last time I checked the robots.txt was there. Anyway I have rcreated this.
Will try creating a new campaign and seeing how long it takes.
Update: worked fine after around 10 minutes crawl!
-
Hi Louis,
You are right, the first starter crawl should be complete within the first couple of hours. One reason it might take longer than normal is if the SEOmoz crawler, rogerbot, has trouble crawling your site.
I took a look inside our system and it appears the website associated with your campaign doesn't have a robots.txt file. Because the SEOmoz crawler is extra polite about crawling, whenever it doesn't find a robots.txt file, this can prevent it from crawling at all.
To fix this, create a text file called "robots.txt" and with the following information in it, and place it at the root directory of your site.
User-agent: *
Disallow:Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt
Unfortunately, it may be too late to add this to fix your starter crawl. So here are some things you can try:
- After adding the robots.txt file, try creating another (duplicate) campaign and seeing if that works.
- Or wait until your next crawl
- Contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO.
-
Louis, I think it is just a standard message, so don't worry, I doubt it will take that long.
I just started one on a 30 page site, so I expect it will show results tomorrow.
-
So this means I will not get any data under crawl diagnostics for a week?
Why does it not give results for up to 250 pages as stated? The domain I selected for crawling has only 20 or so pages.
-
The full crawl diagnostics and rank reports are only generated once per week. So, I would give it at least 7 days before you see full data reported in your new campaign.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Could my Crawl Error Report be wrong?
HI there, I am using Yoast SEO plugin on a wordpress website. I am currently showing 70 Med priority crawl errors 'missing meta description' on my Moz pro account. This number of missing meta descriptions has increased over the last 6 weeks. But every single page / post / tag / category has both the title and meta description completed via Yoast. I requested a google bot to crawl the site a few weeks ago as thought it perhaps wasn't been crawled and updated. Any idea what the issue might be? Could the Moz report be incorrect for some reason? Or could something be blocking Moz / Google from seeing the Yoast plugin?
Moz Pro | | skehoe0 -
2 different pages being shown as duplicate content.
I have a small problem with some of the pages on one of my websites.
Moz Pro | | horkans
Pages are shown as duplicate content when they have no content the same apart from the template. But it only happens with a few products and we have well over 100 products for sale. An example would be these which are seen as duplicate content.
http://www.petworlddirect.ie/p/mr-johnsons-supreme-rabbit-food-15kg/106006139
http://www.petworlddirect.ie/p/dreamscape-stone-bridge/187041111 Any help would be appreciated.0 -
Crawl Errors and Notices drop to zero
Hi all, After setting up a campaign in Moz the crawl is successful and it showed the Errors and Warnings in crawl diagnostics (each one had about 40-50), but after a few days the number dropped to zero. Only the "notices" seems to stay normal, with a slight drop since the campaign set up, but not dropping to zero. I set this campaign up in a colleague's account and the same thing happened shortly after set up. I didn't find any Q&A already posted so any insight is appreciated!
Moz Pro | | Vanessa120 -
My Campaign only crawled 3 pages on my site
On my first crawl of a new campaign, the software only crawled 3 pages. XXXaceXXXscholarships.org any ideas?
Moz Pro | | Santaur0 -
Amount of Pages Crawled Dropped Significantly
I am just wondering if something changed with the SEOMoz crawler. I was always getting 10,000 or near 10,000 pages crawled. After the last two crawls I am ending up around 2500 pages. Has anything changed that I would need to look at it see if I am blocking the crawler or something else?
Moz Pro | | jeffmace0 -
How to handle 2 domains pointing to the same server.
My client has 2 domains which point to the same sever. eg a.com and b.com both point to the same place. For now a.com has better seo but the client wants to switch to using b.com more. Im guessing 301 redirects will not work in this case. How would I handle this? If i'm forced to manage both domains, what is the best practice for monitoring in seomoz pro. Should i set up a.com as a competitor?
Moz Pro | | eztrans0 -
MOZ Crawler only crawling one page per campaign
We set up some new campaigns, and now for the last two weekly crawls, the crawler is only accessing one page per campaign. Any ideas why this is happening? PS - two weeks back we did "upgrade" the account. Could this have been an issue?
Moz Pro | | AllaO0 -
Can you help me get started using the crawl diagnostics report?
After getting the crawl diagnostics report for the first time my boss and I looked over it and we have tried to fix the problems but we are stumped.I have tried and watched videos , read books, etc.. but have found nothing to help. I need assistance getting started on improving my website. Can you help?
Moz Pro | | WVInjuryLawyer0