Crawl slow again
-
Once again the weekly crawl on my site is very slow. I have around 441 pages in the crawl and this has been running for over 12 hours. This last happened two weeks ago (ran for over 48 hours). Last week's crawl was much quicker (not sure exactly how long but guessing an hour or so).
Is this a known issue and is there anything that can be done to unblock it? Weekends are the best time for me to assess and respond to changes I have made to my site so having this (small) crawl take most of the weekend is really quite problematic.
Thanks.
Mark
-
Your best bet right now is to contact the SEOmoz help desk with an email to help@seomoz.org. They may (not sure) be able to have your crawl start on a different day of the week so it's ready for you by the weekend, and should be able to answer all of your crawl-related questions in general.
-
Thank you for the response. I do not have a robots.txt file. I'm not following how the Google crawl rate effects this (but would love to learn why). I thought that is only used to define how often Google itself crawls my site and has nothing directly to do with SEOMoz. Also, it is not how often the SEOMoz crawl takes (it starts on time each Saturday) but, rather, how long it takes to complete. For two of the last three weeks it has taken almost 48 hours from start to finish.
Thanks again.
Mark
-
Have you tried changing the crawl rate in Google Webmaster Tools? Also make sure your robots.txt isn't specifying a crawl rate delay
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unable to get into top 20 even when pages are optimized and most crawl issues resolved
I have a few keyword phrases I've been trying to rank in the top 20 for (starting place). I have optimized for a few different phrases, ranging in keyword difficulty, but no matter what I do I can't seem to get in. In many cases, the exact same results show up for many different variations of the phrases I'd like to rank for. I've read about how google tries to match user intent and so if it decides those results are more relevant then it will always show them, but does that mean that no matter what I do I will always be behind them? The main question I have is: how should I proceed? Should I stop optimizing pages and focus on link acquisition? Or go through and make sure there isn't a single crawl issue? Or focus on optimizing for longer tail keyword phrases? It just feels like I've done so much of what the moz tools have recommended and I'm seeing very little movement over the past couple of months, if anything I see dips in performance after optimization. Thanks in advance!
Moz Pro | | Dynata_panel_marketing1 -
7,608 High Priority Crawl Diagnostic problems
Hey There, I have an e-commerce site that is showing 7,608 High Priorities to fix - 7,536 are duplicate content. What's the most effective process to start with? I'm open to outsourcing some of the work to an expert - email me on dave@emanbee.com Thanks for your time, Dave
Moz Pro | | emanbee0 -
Lag time between MOZ crawl and report notification?
I did a lot of work to one of my sites last week and eagerly awaited this week's MOZ report to confirm that I had achieved what I was trying to do, but alas I still see the same errors and warnings in the latest report. This was supposedly generated five days AFTER I made the changes, so why are they not apparent in the new report? I am mainly referring to missing metadata, long page titles, duplicate content and duplicate title errors (due to crawl and URL issues). Why would the new crawl not have picked up that these have been corrected? Does it rely on some other crawl having updated (e.g. Google or Bing)?
Moz Pro | | Gavin.Atkinson0 -
404 : Errors in crawl report - all pages are listed with index.html on a WordPress site
Hi Mozers, I have recently submitted a website using moz, which has pulled up a second version of every page on the WordPress site as a 404 error with index.html at the end of the URL. e.g Live page URL - http://www.autostemtechnology.com/applications/civil-blasting/ Report page URL - http://www.autostemtechnology.com/applications/civil-blasting/index.html The permalink structure is set as /%postname%/ For some reason the report has listed every page with index.html at the end of the page URL. I have tried a number of redirects in the .htaccess file but doesn't seem to work. Any suggestions will be strongly appreciated. Thanks
Moz Pro | | AmanziDigital0 -
Should I be running my crawl on our www address or our non-www address?
I currently run our crawl on oursitename.com, but am wondering if it should be run on www.oursitename.com instead.
Moz Pro | | THMCC0 -
Crawled pages are missing and showing just 1 page crawled
One of my campaign has got around 8500 pages crawled(seomoz) and reports are shown, but suddenly it is showing 1 page crawled. Why it is happened like this? How can i get back the previous reports?
Moz Pro | | Sulekha0 -
Is there a way to perform a crawl diagnostics without creating a campaign?
If you wanted to perform a crawl diagnostics but your campaigns are at full capacity are you able to do this and how (or does this mean you will have to remove one campaign to make space for another)?
Moz Pro | | SarahAhmed3790 -
Issue in number of pages crawled
i wanted to figure out how our friend Roger Bot works. On the first crawl of one of my large sites, the number of pages crawled stopped at 10000 (due to the restriction on the pro account). However after a few weeks, the number of pages crawled went down to about 5500. This number seemed to be a more accurate count of the pages on our site. Today, it seems that Roger Bot has completed another crawl and the number is up to 10000 again. I know there has been no downtime on our site, and the items that we fixed on our site did not reduce or increase the number of pages we had. Just making sure there are no known issues with Roger Bot before I look deeper into our site to see if there is an issue. Thanks!
Moz Pro | | cchhita0