Websites First Crawl - Over 2 Hour Suggested Wait
-
Hello SEOMoz!
We recently signed up for a free trial and on the pro dashboard it states the following.
"To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days."
It's been nearly 24 hours and we see no results under Crawl Diagnostics however we do under rankings.
Is this normal?
Thanks
-
Thanks!
Last time I checked the robots.txt was there. Anyway I have rcreated this.
Will try creating a new campaign and seeing how long it takes.
Update: worked fine after around 10 minutes crawl!
-
Hi Louis,
You are right, the first starter crawl should be complete within the first couple of hours. One reason it might take longer than normal is if the SEOmoz crawler, rogerbot, has trouble crawling your site.
I took a look inside our system and it appears the website associated with your campaign doesn't have a robots.txt file. Because the SEOmoz crawler is extra polite about crawling, whenever it doesn't find a robots.txt file, this can prevent it from crawling at all.
To fix this, create a text file called "robots.txt" and with the following information in it, and place it at the root directory of your site.
User-agent: *
Disallow:Read more about robots.txt here: http://www.seomoz.org/learn-seo/robotstxt
Unfortunately, it may be too late to add this to fix your starter crawl. So here are some things you can try:
- After adding the robots.txt file, try creating another (duplicate) campaign and seeing if that works.
- Or wait until your next crawl
- Contact the help team at help@seomoz.org
Hope this helps! Best of luck with your SEO.
-
Louis, I think it is just a standard message, so don't worry, I doubt it will take that long.
I just started one on a 30 page site, so I expect it will show results tomorrow.
-
So this means I will not get any data under crawl diagnostics for a week?
Why does it not give results for up to 250 pages as stated? The domain I selected for crawling has only 20 or so pages.
-
The full crawl diagnostics and rank reports are only generated once per week. So, I would give it at least 7 days before you see full data reported in your new campaign.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking issues with my local business website.
Hi, I have local business website "asapgaragedoorrepair.com" which im seeing weird ranking issues during last few months . i would like to share it with you guys and get help from you. first days i have indexed the website ,it was ranking good with google and then after 2-3 weeks i have noticed that none of the pages ranking anymore. i have looked for any issues in the optimization and i fixed them and indexed again. right now i have weird issues. 1- the website was ranking for another page in my website other that the one that i wanted for keyword" Garage Door Repair Pasadena " .(it was ranking with garage door installation page). i changed the content ,optimization on the other page and indexed. still the same issue. then i had no ways other that 301 to the right page . 2- when i check the rankings with "ahrefs" website or "Moz" or other tools that i have, it shows that im ranking for 16th position for " Garage door repair Pasadena ". but when i search it by myself im getting no results. (this is bothering) maybe i dont know something . pls help me 3- in Moz or ahref website shows the ranking like the image i have attached. 2 different results for each keywords. why is that? and why i can not see them when i search them by my self? htPQR
Moz Pro | | Mishel2980 -
Website Issues - Duplicate Content
Hello, I'm fairly new to using Moz and I logged on this morning to find Issues have been found in one of the websites - 22 High Priority and 44 Medium. I know it's due to duplicate content in the blog, but i can't figure out what is duplicated? I've only recently come on board this website so I don't know if the content has been plagiarised or what? The link to the site is here: delacyspa.co.uk Any help would be appreciated. Thanks zFxQmmd
Moz Pro | | Cowbang0 -
SEO on-demand crawl
what happened to the on-demand crawl you could do in PRO when they switched to the new MOZ site?
Moz Pro | | Vertz-Marketing0 -
How do i fix the problem of having 2 url's splitting my rankings?
please excuse my noobness. i have a nice site, www.soundsenglish.com which I built from scratch and learned by doing. It has lots of nice content and it does ok, my rankings are woeful mostly cos of all the mistakes i made building it...i'll fix that stuff. This stuff i don't know about. from my adsense i get 2 listings www.soundsenglish.com and soundsenglish.com wierdly the second one gets consistently higher paying ads although most of the visitors come through the first but they are both the same landing page same content -as far as i can tell. when i try to find rankings, use the seo tools etc i get diferent scores, so whatever it is, it is splitting the sites - can't be a good thing. i have no idea why this happens and i have some inkling that maybe i need something to do with cannonical redirects or maybe a 301 redirect. both of which i have little idea how to do. If that isn't enough naive blundering about for you, i have a little more... it occurs to me that this prpoblem is probably happening with every page on my site, i.e. the 'juice ' is not getting credited onto that one page. this surely means cannonical redirects but even afterreading up on them idon't quite get it. or rather ido but idon;t get how to apply it to my context.
Moz Pro | | soundsenglish0 -
Crawled pages are missing and showing just 1 page crawled
One of my campaign has got around 8500 pages crawled(seomoz) and reports are shown, but suddenly it is showing 1 page crawled. Why it is happened like this? How can i get back the previous reports?
Moz Pro | | Sulekha0 -
Crawl Disgnosis only crawling 250 pages not 10,000
My crawl diagnosis has suddenly dropped from 10,000 pages to just 250. I've been tracking and working on an ecommerce website with 102,000 pages (www.heatingreplacementparts.co.uk) and the history for this was showing some great improvements. Suddenly the CD report today is showing only 250 pages! What has happened? Not only is this frustrating to work with as I was chipping away at the errors and warnings, but also my graphs for reporting to my client are now all screwed up. I have a pro plan and nothing has (or should have!) changed.
Moz Pro | | eseyo0 -
Excluding parameters from seomoz crawl?
I'm getting a ton of duplicate content errors because almost all of my pages feature a "print this page" link that adds the parameter "printable=Y" to the URL and displays a plain text version of the same page. Is there any way to exclude these pages from the crawl results?
Moz Pro | | AmericanOutlets0