Crawl started on the 18th of December and still hasn't completed!
-
As the title states I started a crawl over 2 weeks ago now and it still hasn't completed. Does anyone know why this could be?
Thanks
-
Thanks I'll give that a go!
-
Hi Laurance,
Thanks for writing in and sorry that your crawl never completed. I'm afraid the issue is that all crawlers are being blocked in the robots.txt file of the site and we aren't able to fully access the site to complete a crawl. If you were to update the robots.txt file to allow our crawler to access the site, we would than be able to get the crawl completed for you. You can use the user-agent rogerbot to specify our crawler or you can remove the disallow: / directive from the robots.txt file to allow us access. Once you've done that, let me know and I will have our engineers complete the crawl for you.
I look forward to hearing back soon.
-Chiaryn
-
Hi Laurence,
What do you mean the crawl still hasn't be completed? Do you mean not all your sites are indexed?
I suggest you checking up on your Google Webmaster Tools and see how your site is doing. Once you log in, click on your website then click on the Health tab on the left hand side. You can see all your crawl stats and errors there. Furthermore, you can use the "Fetch as Google" tool and ask Google to crawl your pages again.
Another thing is that if you have a lot of pages and your pages are not interlinking correctly, it may take weeks for Google to crawl all your pages. There is not set time limit on how long it takes. Maybe submit a sitemap to speed up the process of finding all your pages?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am using moz pro but ı can't use keyword explorer
hi am buying moz pro but can't please help me. ı want scan this web site https://baliktasarim.com/
Moz Pro | | nurmasumi20020 -
Tips on speeding up disavow process and still doing it right
Hello, I'm doing a 10-step disavow check using this article, this word document, and this corresponding spreadsheet template. Doing a disavow for a non-manual penguin penalty (paid links, doorway sites) It's taking a long time. Can you give me some pointers of how to speed things up? I want to do it right but I need to know how I can do it faster. Like tricks on domains to automatically disavow, what not to check, whatever tips you have. I'm using Cognitive SEO and manual checking. I'd like to avoid pricy tools. I'm on a mac. Thanks!
Moz Pro | | BobGW0 -
I have a duplicate content on my Moz crawler, but google hasn't indexed those pages: do I still need to get rid of the tags?
I received an urgent error from the Moz crawler that I have duplicate content on my site due to the tags I have. For example: http://www.1forjustice.com/graves-amendment/ The real article found here: http://www.1forjustice.com/car-accident-rental-car/ I didn't think this was a big deal, because when I looked at my GWT these pages weren't indexed (picture attached). Question: should I bother fixing this from an SEO perspective? If Google isn't indexing the pages, then am I losing link juice? 6c2kxiZ
Moz Pro | | Perenich0 -
I'm sure you get this all the time but....
Hi New to Moz, I'm finding it all very helpful and daunting at the same time. I've read heaps of Seo the Moz content and watched every Video I can find. It's all going well and I've managed to get A's on some of my pages, I was hoping some one cold tell me why this site (Link to screan grab http://tinypic.com/r/dnj9di/8 ) is way above me in the Google search results? It appears to be under performing me in Moz but in actual fact it is greatly out performing me in Google. Any insights are more than gratefully received! Thanks Ben
Moz Pro | | SussexChef830 -
What should I put in 'Define Branded Keyword Rules' -Starting a Campaign
Hello, I am a new user here (this seems really interesting!), but english level is not very good (I am spanish) and I dont understand what means 'Define Branded Keyword Rules' Hope someone can explain me it in easy words so I can understand Thank you very much! 1362443047.jpeg
Moz Pro | | matiw0 -
'Too many on-page links': doesn't add up...
Hi All, I'm fairly new to the site, have had a full site crawl and am somewhat confused by a large number of pages (82) reported as showing 'too many on page links'. Mine is a new e-commerce site selling a project management methodology. I have about 80 pages which describe individual templates and contain internal links to additional, related content on the site. The crawl warnings define too many on page links as roughly 100 "on any given page" yet almost none of the pages flagged contain more than 20-30 on links. Any thoughts and ideas about (a) the degree to which this will aeffect my ranking in practice and (b) how to resolve it gratefully received! Thanks in advance, Felix
Moz Pro | | RomanCat0 -
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Why are the rankings of my keywords different to rankings from other programmes such as 'Access my Lan' and 'Firefly'? Does SEOMoz count 'Google Places and Maps' in their rank count? Im thinking this could explain the difference
Moz Pro | | john_Digino0
