How Often is Site Crawled
-
Good morning-
I saw some errors in my first crawl and immediately removed the pages from my website. I then re-created my XML sitemap and uploaded to Google.
The question I have is will the site be crawled to recognize the changes in the next day or so? The pages were just placed on the site as test pages and never removed. The initial crawl that notified me it was done found the errors and were removed.
Thanks for your help.
Peter
-
Oops sorry I misunderstood - thought you meant Google, but at least Keri jumped in there and gave you the answer
-
SEOmoz will crawl the site once a week. You can go to research tools and select a crawl on demand of up to 3000 URLs from a domain if you'd like a crawl sooner.
-
Thanks Matt. What I meant to ask was how often will SEO Moz crawl the site?
-
There is no set time limit to how often a site is crawled by Google - different sites have different crawl rates depending on factors such as how popular they are. For instance a well ranked, popular news site with content that is regularly being updated and added will have a crawl rate that is higher than a small personal site.
Have a look at this article in regards to ways you can encourage Google to crawl your site more often - http://www.searchenginejournal.com/10-ways-to-increase-your-site-crawl-rate/7159/ (an old article but still applies)
Have you thought about using the URL removal tool in Google Webmaster Tools to quickly remove the test URLs - http://support.google.com/webmasters/bin/answer.py?hl=en&answer=164734
And finally in relation to speeding up the crawling of specific pages on your site you can use the fetch as Googlebot tool > enter a URL > click submit URL to index to speed the crawling of that content up. Here is a great article on this much underused tool - http://searchenginewatch.com/article/2198245/How-to-Force-Google-to-Recrawl-Your-Site
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
White listing a site
A new clients site is blocked by a lot of Firewalls. And I can't work out why, the content is family friendly they sell nursery equipment. I've run it through the Google checker and there is no malicious software found on the site. Can anyone tell me what I need to do to get this site unblocked? The url is http://knuma.co.uk/
Technical SEO | | Marketing_Optimist0 -
Site not loading on Firefox
Hello guys, I can't get my website to be loaded on Firefox, why's that?
Technical SEO | | PremioOscar0 -
Site verification in WMT
Hello all, I have a site and I want to set a preferred domain but when I do it says I need to verify my site but it gives me no ideas how to do that. I know that normally you have to do it when you set the account up but I had an analytics account for this domain first then just logged on with those details and I was in with no verification process. Cheers
Technical SEO | | jwdl0 -
SEO friendldy Site structure?
we are in the process or rewriting all the pages on one of our sites and will be changing some urls around. i was just wondering if dashes or underscores are better in the urls SEO wise? www.site.com/word-word-word/ or
Technical SEO | | 858-SEO
www.site.com/word_word_word/ i personally like the underscores better but some colleagues tell me that dashes are better, any tests out there on this issue?? Thanks0 -
What to do next with my site gamblingsites.co
So I have this site gamblingsites.co, which I launched about a year ago (I think.) This used to be internetgamblingsites.net (a domain I bought, but never managed to get in the index, and it appeared to violate the T/Cs after asking in GWMT) and before that the site used to be casinowarehouse.eu. After moving to gamblingsites.co, the pages were indexed almost instantly. I kept a 301 in place until today as I had some links pointing to internetgamblingsites.net. Now, until a few weeks ago, everything was fine. The site was ranking top 10 for gambling sites (8-10) and I had some traffic everyday. This site wasn't my top priority, so besides adding new unique content, I didn't do much with it. In each case no shady link building or what-so-ever. On February first of this year, however, it lost all of its rankings, and I have no idea why. Much worse site appear in the top 50, where a sub page of my site appears somewhere on the 9th SERP for keyword 'gambling sites.' Last week I started contacting some people and asked them to update my links. I also used my own sites (all on unique hosting accounts) to build some branded links, i.e. 'GamblingSites.co' and similar terms to down tune the exact match. I also decreased the instances of the exact match on the homepage, to avoid over optimization. Finally, I removed the 301 from internetgamblingsites.net, since the better links have been changed (or are about to get changed soon.) Now, couple of days later... no changes, but it's probably to early to judge. My question to you: "What would you do next, to try to save the site and at least get some traffic to it?" Thank you for your help, Giorgio PS: Feel free to ask for more information.
Technical SEO | | VisualSense0 -
Google Webmaster Site Performance
In webmaster tools, under labs/site performance google provides your ave page load time. When google grades a page, does it use how long that specific page loads -or- Does google use the overall ave page load time for the domain as provided in lab/site performance
Technical SEO | | Bucky0 -
On-Site Sitemaps - Guidance Required
Hi, I am looking to find good examples of on-site sitemaps. We already submit our XML sitemap regularly through GWMT but I now wonder if we still need an on-site sitemap, as we have about 30 static pages and 300+ Wordpress blogs which in a sense makes that a spammy page as it has too many links and a higher than average keyword density. The reason I am looking for good examples is that I want to create a basic on-site sitemap that aids navigation but is styled to look ok as well. The Solution I have in mind: mydomain.com/link-example-one.php
Technical SEO | | tdsnet
mydomain.com/link-example-two.php
mydomain.com/liink-example-ten.php mydomain.com/blog then links to my 300 WP blogs, broken down into chunks navigated by using breadcrumbs. Will Google crawl this ok or should I stick to the current format listing ALL posts on one page? Thanks0 -
Page that has no link is being crawled
http://www.povada.com/category/filters/metal:Silver/nstart/1/start/1.htm I have no idea how the above page was even found by google but it seems that it is being crawled and Im not sure where its being found from. Can anyone offer a solution?
Technical SEO | | 13375auc30