Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What shall I call my site?
With all the google changes and social media focus on seo, I am in a bit of a dilemma and any opinions welcome. My husband has a very small horseback riding company based in a tiny mountain town of Costa Rica and I cannot decide whether to call it by the company name or by a more generic name that is likely to be picked up by the search engines ( I do believe the importance of domain name is less than it used to be in terms of search with all the new google changes). The two names are www.horsetrekmonteverde.com (bear in mind that his customers come from the States and horse trekking is not commonly used) and www.horseridingcostarica.com One mentions Costa Rica, the other not as you can see. Also not many people have beard about Monteverde. However I am trying to do some strong branding! What would you do?
Industry News | | Llanero0 -
What's an example of an SEO firm's site that lists their SEO packages excellently
Hello, I'd like to make a packages page on the site I'm still building bobweikelseo.com What's an example of a firm or SEO that lists their packages in a way that is appealing, useful, and all around excellent? I'm thinking of having a first package for analysis at $440, a second package for analysis and on-site SEO for $880, and a third package for analysis, on-site SEO, and link building for $2000-$4000/month depending on how fast they want the link building to go. Looking for some input on how to create a packages page, if my prices are appropriate (I want to be affordable), and an example of a website that does an excellent job at presenting their packages.
Industry News | | BobGW0 -
Manual action penalty by Google
Hello, We have a big well-known brand - www.titanbet.com. This brand is well established and the site has been live for almost 4 years now ranking very well on some very strong KWs. we received a message from Google on Aug 29<sup>th</sup> saying “Google has detected a pattern of artificial or unnatural links pointing to your site” and that “Google has applied a manual spam action to titanbet.com/” The past 2 weeks since the penalty was received we saw some of our major KWs drop in rankings. BUT all brand related KWs were still ranked 1<sup>st</sup> Over the last weekend the penalty has worsen and we no longer rank on any of the brands KWs (we find the site in 5<sup>th</sup> page at best). Moreover, when searching for a sentence from the any of the page on the site in Google, we see other sites ahead of us in the SERPs. Based on the message we originally received from Google we have started cleaning some of the bad links to the site. We found a lot of links from bad sites, some of them are not indexed and probably penalized as well, some are from affiliate websites and some are from some automatic indexation websites based in China and Russia
Industry News | | Tit
we have started reaching out to some of these sites to try and have them remove our links. We are also worried about the duplication of our site. We have found many other sites (mostly affiliate websites) have copied and in some cases completely duplicated our content. Google for some reason has chosen to penalize us for this. Although we do not have control over these other sites. We have run copyscape to try and figure out which pages are the most problematic and we will try to re-write the content on these pages. But what if the other sites copy us again? Any suggestions on the above would be appreciated as we try to understand why Google has penalized us. thank you Titan Bet Team0 -
3724 pages submitted, 3591 indexed
You probably know what I mean, the report in Google Webmaster Tools > Sitemaps. So how do I locate the pages that are NOT indexed? Thanks, Ben
Industry News | | bjs20100 -
Will Google ever begin penalising bad English/grammar in regards to rankings and SEO?
Considering Google seem to be on a great crusade with all their algorithm updates to raise the overall "quality" of content on the Internet, i'm a bit concerned with their seeming lack of action towards penalising sites that contain terrible English. I'm sure you've all noticed this when you attempt to do some proper research via Google and come across an article that "looks" to be what you're after, then you click through and realise it's obviously been either put together in a rush by someone not paying attention or putting much effort in, or been outsourced for cheap labour to another country whose workers aren't (close to being) native speakers. It's getting really old trying to make sense of articles that have completely incorrect grammar, entirely missing words, verb tenses that don't make any sense, randomly over-extravagant adjectives thrown in just as padding, etc. etc. No offense to all those from non-native speaking countries who are attempting to make a few bucks online, but this for me is becoming by far more of an issue in terms of "quality" of information online as opposed to some of the other search issues that are being given higher priority, and it just seems strange that Google have been so blasé about it up to this point - especially given so many of these articles and pages are nothing more than outsourced filler for cheap traffic. I understand it's probably hard to code in something so advanced, but it would go a long way towards making the web a better place in my opinion. Anyone else feeling the same way? Thoughts?
Industry News | | ExperienceOz1 -
"Links To Your Site" In Webmaster Tools
How often does Google update the "links to your site" data. It seems that it has been static for about a month now even though we have made a lot of changes Does anyone have any idea? If you have made changes to your links (i.e removed links, updated anchor text, etc.), do you have to wait for this information to be updated to measure the impact? Or is that whenever Google crawls those pages/sites and sees changes there is a adjustment. Thanks
Industry News | | inhouseseo0 -
Google Product Feeds - New Requirements
We are in the jewelry industry, and for Google product feeds, we list our products under "Apparel & Accessories > Jewelry". As of the new Google feed requirements, they are saying that we have to choose a gender and color for each product that is in the Apparel category. While this makes sense for clothes, it doesn't exactly for jewelry because many items are for both men and women, and there's not always a color associated with each product. I can enter some of these fields manually, but with 5,000+ products, it makes it difficult w/ each update. Anyone have solutions for this? Or a way around it? Can we just include those fields but leave them blank? Any other solutions?
Industry News | | applesofgold1 -
Why has my places ranking dropped off google?
I've recently worked for a company who up untill 3 weeks ago have shown up in the google places ranked at number 1 for the following key phrases. First Aid Training in Cornwall
Industry News | | it-ok
First Aid Courses in Cornwall
Health and Safety Training in Cornwall as well as several more... However over the past 3 weeks they have gradually dropped out of the places listing all together, and I'm struggling to understand why this might be. Yesterday I released a new website for them which has had a lot more SEO input in to the site, is this likely to have an affect on the places ranking? the website is www.insafehandstraining.com Any answers would be very appreciated.0