Site crawler hasn't crawled my site in 6 days!
-
On 4.23 i requested a site crawl. My site only has about 550 pages. So how can we get faster crawls?
-
Crawl In ProgressSubdomain
www.taxproblem.org
Submitted 5:47pm GMT
Apr 23rd 2011
Is it a good idea to delete this (can't see how in the tool). I was thinking that maybe it got lost or something, it's now 10 days. I wanted to make another request.
I was mistaken before, the site wasn't crawled, it was just a campaign update.
-
I listed my craws in progress with the dates, so yes.
-
That one uses the exact same technology without the pretty report. Have you used it lately? It was upgraded a week ago and I believe intended to replace the one in seomoz labs.
-
Actually that's the link to the worst one that doesn't do anything. Yesterday my site was crawled, so it actually did take a week, but as you can see the results here are lagging...
Crawl In Progress
Subdomain
www.tax-audit.us
Submitted 1:36am GMT
Apr 25th 2011
Crawl In ProgressSubdomain
www.taxproblem.org
Submitted 5:47pm GMT
Apr 23rd 2011
-
That seems long to me. If you run a crawl. You can run it here instead and get results in 24 hours. Up to 2 per day and 3000 pages. It won't give you that nice red, blue and yellow report but it basic info is the same. You just need to manually analyze it. http://pro.seomoz.org/tools/crawl-test
-
I'm not sure to be honest but before it slowed down a little I tended to get a crawl once a week per site, whether a site was large or small.
-
Thanks. When fixed, how long should we have to wait for crawls?
-
That's not actually that long, and there's a queue anyway... I had a small site wait a couple of weeks recently but it did get crawled, was just in the queue for a while first. Just give it a while longer and it'll be fine.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why might my websites crawl rate....explode?
Hi Mozzers, I have a website with approx 110,000 pages. According to search console, Google will usually crawl, on average, anywhere between 500 - 1500 pages per day. However, lately the crawl rate seems to have increased rather drastically: 9/5/16 - 923
Reporting & Analytics | | Silkstream
9/6/16 - 946
9/7/16 - 848
9/8/16 - 11072
9/9/16 - 50923
9/10/16 - 60389
9/11/16 - 17170
9/12/16 - 79809 I was wondering if anyone could offer any insight into why may be happening and if I should be concerned?
Thanks in advance for all advice.0 -
"index.htm" for all url's in google analytics
I don't have this issue with other wordpress websites, only this one website, and I don't know what's causing the issue: Google Analytics is adding an "index.htm" to every single page on the website. So it is tracking the pages, I see no errors - is it tracking the right page? When I click on the page link in a report, I naturally go to a "404 page not found" since the website address isn't "www.example.com/rewards/index.htm" - but instead the actual address would be:
Reporting & Analytics | | cceebar
"www.example.com/rewards/". I have navigated to View Settings in GA to insure "default page" is empty. Although adding anything else to this field does not effect the page url in analytics reports either. Could it be htaccess file - or a plugin effecting the htaccess file?_Cindy0 -
Does subdomain (or sub sub domain) affect analytics data of root site?
We self-host our public website, but over time have also added subdomains onto it that are not public and are for internal or even client portals. I am seeking advice as to whether those subdomains affect the analytics data (self referrals, visits, bounces) of the public site that I am tasked with analyzing. I feel that it does skew the data but need to build a solid case to move the public website to a new domain, so as to leave the existing one in tact with all of its subs.
Reporting & Analytics | | MarketingGroup0 -
Why would our client's site be receiving significant levels of organic traffic for a keyword they do not rank for?
We have a client that has received 100+ organic visits for the keyword 'airport transfers', yet the site does not rank in the top 100 search results for this keyword. We have checked that it is not untagged PPC traffic. Truly baffling. Can anybody help?
Reporting & Analytics | | mrdavidingram0 -
Discovering broken links to my site
How do I find out if other websites are trying to link to my website using a misspelled URL or broken link?
Reporting & Analytics | | AccuraCast-UK0 -
SEOMoz & Google Webmaster Tools crawl error conflicting info
Site im working on has zero crawl errors according to SEOMoz (it did previously have lots since ironed out) but now looking at GWebmaster Tools saying 5000 errors. Date of those are not that recent but Webmaster Tools line graph of errors still showing aprox 5000 up to yesterday There is an option to bulk action/tick them all as fixed so thinking/hoping GWT just keeping a historical record that can now be deleted since no longer applicable. However i'm not confident this is the case since still showing on the line graph. Any ideas re this anomalous info (can i delete and forget in GWT) ? Also side question I take it its not possible to link a GA property with a GWT account if created with different logins/accounts ? Many Thanks Dan
Reporting & Analytics | | Dan-Lawrence0 -
Why did I loose all my product page rankings (e-commerce site)
This friday I noticed that I'd lost pretty much all my product pages in the SERP and also their rankings for the product names. These are products I both have introduced to the market (sweden) and also some that I've been the only one selling. I've analyzed a couple of different ranking-faults. Examples: **"super mario väggdekaler" should rank **http://www.roligaprylar.se/Super-Mario-Vaeggdekaler.html as #1 and has done for several years. Instead this search in my internal search engine ranks #10-#15 with no relevance. www.roligaprylar.se/?q=mario%20v%E4g "jedi morgonrock" should rank www.roligaprylar.se/Jedi-Morgonrock.html as #1 or #2 but instead this url ranks as #12 www.roligaprylar.se/product_detail.php?pid=Jedi-Morgonrock "Charlie sheen bobblehead" (in the swedish serp this should be the most simple term to rank on. previously #1) my internal search engine ranks for #8 with this url <cite>www.roligaprylar.se/?q=Charlie%20Sheen%20Bobblehead</cite>J So I've drawn these conclusions and actions Products that don't rank well longer but still ranks with their alternative non-rewritten url has gotten deep links from affilliates (i track affilliate ids and stuff via this link) and have replaced the original url which is rewritten. Action: Canonical urls for these non-rewritten products to the rewritten version. For example on this product page www.roligaprylar.se/product_detail.php?pid=Jedi-Morgonrock I've placed a canonical for this url www.roligaprylar.se/Jedi-morgonrock.html With the products not ranking at all or when searches in my search engine shows up I suspect some kind of dup content punishment where Google thinks the search result is more important than the product page. Action: All search-pages are now noindex,follow I also increased product name density in terms of keywords on the product page. But I'm still owned and losing tons of money during the holidays (buying adwords at obscene amounts instead hehe). So just wanted to hear with you guys. Are my conclusions and actions correct? What have I missed, what more could I do to reverse this? Thanks Dan
Reporting & Analytics | | nuttinalle0 -
Sort referring sites by visit change over time comparison in GA
I can't believe I've never done this before, so I'm going to assume that I previously must have figured it out via excel, but I'm hoping there's an easier way. So I want to compare the referring sites between April and May and see which have sent (specifically) less traffic. The problem with doing a comparison in GA is that it only sorts by the highest traffic for May, when actually I want to see the largest negative change (by number, not percentage) between April and May. Is there a way to do this via the dashboard or am I just going to have to play about in excel for 10 minutes?
Reporting & Analytics | | StalkerB0