Crawl Diagnostic | Starter Crawl taken 14hrs.. so far
-
We started a starter crawl 14hrs ago and it's still going, can anyone help on why this is taking so long, when it says '2 hrs' on the interface..
Thanks,
Rory
-
Hi Rory. Most of our help desk is on holiday today, since it's the Fourth of July in the states. We do have a record of your ticket and one other person who is having a slow starter crawl, and a help desk specialist is looking into this now. Sorry for the delays.
Keri
-
I've asked — now heard yet, think i'll wait to hear.
Thanks for your help, appreciate it.
-
Send an email to help (at) seomoz.org for someone to have a look.
-
It's a fairly big site, but it does say:
'To get you started quickly Roger is crawling up to 250 pages on your site. You should see these results within two hours. The full crawl will complete within 7 days.'
There's no option to do anything else, like cancel, reset etc — it just says 'Starter crawl in progress', it's been 16hrs now + bit frustraing as needed to send this through to a client this morning.. Anyone from SeoMoz around to look into this?
-
And here is how you reset the crawl:
1. On your webserver, edit the robots.txt file.
2. Block the seomoz bot from crawling the site by blocking its access to the root.
You can do so by adding the following lines:
User-agent: rogerbot
Disallow: /
This would end the crawl session.
But, before you do this, it may a good idea to check if your site indeed has a lot of content and outgoing links?
-
Rory,
What is the sub-domain that you are crawling? It may just be that there is a lot of content to crawl.
-
How would I reset the crawl? I don't appear to have an option to?
-
Rory,
I would guess that this crawl session has hung-up; it would be a good idea to start a new session. The session could have been left in the middle due to a server side issue on your website or a temporary drop in connection between the API server and your website's server.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Crawl Diagnostics 2261 Issues with Our Blog
I just recently signed up for MOZ, so much information. I've done the walk through and will continue learning how to us the tools. But I need your help. Our first moz crawl indicated 2261 issues (447 404's, 803 duplicate content, 11 502's, etc). I've reviewed all of the crawls issues and they are linked to our Yahoo hosted WordPress blog. Our blog is over 9 years old. The only issue that I'm able to find is our categories are not set up correctly. I've searched for WordPress assistance on this topic and cant find any issues with our current category set up. Every category link that I click returns Nothing Found Apologies, but no results were found for the requested archive. Perhaps searching will help find a related post. http://site.labellaflorachildrensboutique.com/blog/ Any assistance is greatly appreciated.
Moz Pro | | Girlstuff0 -
Crawl Diagnostics - 350 Critical errors? But I used rel-canonical links
Hello Mozzers, We launched a new website on Monday and had our first MOZ crawl on 01/07/15 which came back with 350+ critical errors. The majority of these were for duplicate content. We had a situation like this for each gym class: GLOBAL YOGA CLASS (canonical link / master record) YOGA CLASS BROMLEY YOGA CLASS OXFORD YOGA CLASS GLASGOW etc All of these local Yoga pages had the canonical link deployed. So why is this regarded as an error by MOZ? Should I have added robots NO INDEX instead? Would think help? Very scared our rankings are gonna get effected 😞 Ben
Moz Pro | | Bendall0 -
SEO for e-commerce, linking, brand mentions, insights diagnostics
Hi, I run the seo for www.bilthamber.com It is an e-commerce site selling car care products, and we also manufacture our own products. I was wondering how the seo for an e-commerce site might compare to the seo for a blog or information based site, or other types of site. And also how the seo for a particular product page on my site might compare to a information based page on my site. Are the tactics the same? Should you try and rank for similar terms, the same generic terms, or complelty unique terms for the best seo? Or should every page try and rank for different terms, to ensure that the engines dont rank 2 or more different pages for the same search term? And how many keywords or search terms should I try and rank a particular product page for? Or an information page, or blog page? Is there a minimum or maximum that is recommomended? Does the more terms you try and rank for on a page take the weight out of each of them? So having 4 really good search terms might be better than having 10 ok generic ones? How do I find out what are my best, most highly converting, most profitable keywords overtime for Google? I can find some information like this, but it is showing the stats for Bing UK, not Google, even though Google UK is my main engine in Moz. My domain authority has decreased by 3 this week, this is the first time it has gone down since I started with Moz a few months back, can anyone explain why? And how this will affect the rankings ect, and how I can get it back up higher agian? And for linking and brand mentions, what are the goals of this feature? Should I find my mentions and ask them to talk about us more and link to us properly? I'm not sure if this is right. Should I be asking other websites to make links to our pages, or should it be organic? And for the insights section, I am recieveing tons of issues a lot to do with duplicate content, will having the rel=carnocial tag fix these issues? I know thats a lot for 1 question, but its been building up in my head! Thanks in advance! Max
Moz Pro | | BiltHamber10 -
My Website is not being crawled by Google last 15 days
Hello, Required emergency help, one of client website http://www.mediaxpress.net/ is not being crawled since last 15 days, What can be the reason. I have Checked Google Webmaster Tools, it not showing an Issue and even SEOmoz crawler is crawling it normally. I have updated some data on pages, and i want to get it crawled by Google. But Google is not crawling in anyway.
Moz Pro | | CommercePundit0 -
Campaign web crawl has failed last 4 times
I have 4 websites setup in my pro dashboard. The only site that isn't getting crawled is an HTTPS site. It has worked for over a year, but the past 4 crawls (an entire month now) has returned only one page crawled. Is there something going on with the crawler? I really need to be able to see these stats. Has anyone else experienced this issue?
Moz Pro | | nbyloff0 -
Is there a way in in the pro campaign to initiate an instant crawl?
I have a campaign that's on a very small website at this point. It's only 9 pages, so a crawl would only take a few minutes. Is there a way to start a quick crawl at my request instead of having to wait a week to see if I fixed the issues it found during the last crawl? I've been hunting around, but I don't see a way.
Moz Pro | | mytouchoftech0 -
I have another Duplicate page content Question to ask.Why does my blog tags come up as duplicates when my page gets crawled,how do I fix it?
I have a blog linked to my web page.& when rogerbot crawls my website it considers tags for my blog pages duplicate content.is there any way I can fix this? Thanks for your advice.
Moz Pro | | PCTechGuy20120 -
How long does a crawl take?
A crawl of my site started on the 8th July & is still going on - is there something wrong???
Moz Pro | | Brian_Worger1