Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should I buy an expired domain that has already been redirected to some other website? Can I use it or redirect it to my own site if I purchase it?
I was going to purchase an expired domain but then I came to know that it has been redirected to some other website. I have two questions: Can I build a website around this domain if I purchase it? and will the Domain Authority remains the same ? Can I redirect it to my own site? and will all the Link Juice flow to my site?
Industry News | | Kamranktk0 -
Now that Google is no longer publicly displaying Page Rank updates, how will this effect Moz's ability to calculate DA and PA?
Hi, How much more important do you guys think that Moz's Page Authority and Domain Authority metrics are going to become now that Google has stopped giving people public access to a site or pages Page Rank? And how accurate is PA and DA as a measurement in comparison to Page Rank..so for example if I was seeking a guestposting opportunity and saw a site as having a PR of 4....if I now looked to Moz's Page and Domain Authority metrics instead...would that still give me equivalent information on the strength of that domain and thus make a judgement on whether it will be a worthy site for a guestpost.. I guess what I am asking is, how close is now looking at Moz's metrics (ie. a third party company) to the info on PageRank that was being updated by Google themselves? Also will the lack of updated public PR info from Google effect the ability for Moz to calculate PA and DA?? Look forward to your replies on this,
Industry News | | sanj50500 -
Penguin 2.0 Update - Just Hit - Google Messes up again, can anyone on SEOMOZ please tell me why or how some of these websites are ranking?
I am getting a bit tired now writing all of this so please excuse grammar and spelling mistakes, I wanted to post this up quickly tonight so I could possibly get some feedback by morning. So I feel I figured out some of the new update that just hit and I am sure the update will continue to keep coming and rankings will most likely change for a while, but I do have a few questions in the mean time if they stay. Please look below, why would some of those sites be ranking? Some sites utilize no onpage SEO, some no backlinks, one of the sites is a single page site and every page is a broken link even the contact us page and it's rank 1..... I am not 100% sure that Google got this one right. I see many instances below where other sites are much higher quality and have more authority. It's as if Google took terrible sites and said here is a site for you, now deal with it and I hope it works out at the top. I do see several sites that belong on the first page, but I see others that are very questionable. This is how the following is written below. Before the url is the anchor text density % for the site. This indicates the percentage of the amount of times the anchor text was backlined versus other anchor texts they used. Issues with the sites. Most of these sites don't utilize onpage SEO and it's clearly not a factor for onpage density purposes, there is a site with 37% keyword density on it. Some sites even have broken links. Please note: I already know that there is a lot of data that is analyzed to determine rankings more than this, but PA/DA is suppose to be a major factor according to everyone that believes in SEOMOZ.org. So I am taking in account for many of those factors being calculated due to this. These are just some random keywords I picked, because I know you need to analyze other SEO firms after and update to figure out what lasted and what didn't. All of my sites lasted due to ethical tactics, but I had some rankings move down and others go up, very odd. Keyword Analyzed: CHEAP SEO Not Knowng http://searchengineland.com/the-hidden-cost-of-cheap-seo-social-labor-131585 92-DA 75-PA - A lot of content 4715 words on the page including comments. not known http://www.searchenginejournal.com/why-theres-no-such-thing-as-cheap-seo-or-link-building/45932/ 87DA 63 PA Words on page 3673 Not known www.cheap-seo-solutions.com 26 DA -- 38 PA 1112 words on page 9.52% www.cheapseocompany.com 34 DA ---- 41 PA 10 + anchor texts but 623 words on page 10.26% seocheap.net 35 DA --- 45 PA 10 + anchor text 855 words on the page I am not 100% sure why this is ranking lol, the services page isn't even working it errors out. The onpage SEO is sloppy and the writing looks forced. Why is this even ranking? The site also looks low quality. The density is higher than SEOMOZ even and it has less words. In this case DA + words carried this site up, not the anchor ratio being low. 6.17% http://www.cheapseo-services.com cheap seo Page Authority DA 28 -- PA 38 276 Words No onpage, 35 duplicate pages, free template, etc.. etc.. 10% www.seomoz.org/blog/how-to-do-seo-cheap DA 94 --- PA 56 only 5 anchor texts used on this page. 8651 Words I am willing to bet if you diversified this with about 5 more anchor texts this could be number 1 easy. This only has 5 total diverse anchor text backlinks for this page. Keyword Analyzed: Affordable SEO Services Rank 1 47% www.affordableseoservicesx.com/ 24DA PA36 New site It still has broken links all over the page The Contact us page doesn't even work lol. Great going Google on ranking such a high quality 1 page website. 471 words How is this site ranking? How could Google even rank this site? Rank 2 The Term isn't mentioned at all accept in title and header not known mbseoservice.com/ da 20 pa 32 PR 0 Just has affordable seo services in title anchor Possible New Site 627 words Rank 4 12.77% affordableseoservices.net 22 DA 35 DA - Proof Exact match domains still work great with high diversity rates, low word amounts, bad DA, etc..... 348 words Rank 5 4.24% www.howardsemgroup.com DA 39 PA 49 893 words. Rank 6 2.49% www.i4.net/ 59 DA and 66 PA 588 words Rank 7 4.71% www.bluefrogseosolutions.com/ DA 31 PA 42 527 Words This site looks like it was created in 1998 and never updated. Low quality site IMO Rank 8 Not Known www.mainstreethost.com/ 76 DA 81 PA possible co occurrence added in with main domain name /url 281 Words not 1 exact anchor text match Rank 10 2.7% bestcheapseoservices.com/ 18 DA 29PA This is just a blog site, come on Google... 4379 Words
Industry News | | MarketingOfAmerica0 -
Google Alert not working - anyone else have this problem?
I have a Google Alert that has stopped pulling in recent results even though a web search indicates that the pages are being indexed. None of the alert settings have changed. Anyone else have this happen recently and know how to remedy this problem?
Industry News | | BostonWright0 -
Google: 7 Results on First page now (Good or Bad)?
Hi, So what are your thoughts on only 7 results on the first page of a search? Looks like its only in play for brands at the moment with increased sitelinks for the first result. I actually really like it, looks a lot better and its easier to take the overall information in on the page. I hope they role this out onto more generic terms as well. Should mean a higher CTR for those who have 'made it' to the first page. I suppose Google will pickup on a higher CTR on the ads well.
Industry News | | activitysuper0 -
Google Pushed Out Panda Update 3.9 Last Night
Google said it was rolling out a Panda algorithm last night. New data refresh of Panda starts rolling out tonight. ~1% of search results change enough to notice. More context: goo.gl/huekfHas anyone been affected yet ....I seem to see a jump in some newer sites less than 3 months old already for some rankings - I don't like seeing rankings that quick!Do you see anything?
Industry News | | Chenzo1 -
Google to Target Overly SEOd Sites
I just watched the video from Barry Schwartz talking about the new Update to come with the Google Algo. Video of his Friday Post: http://www.youtube.com/watch?v=fJqSPT2NXdA I have also started reading on Webmasterworld on this topic: http://www.webmasterworld.com/google/4429947.htm What do you think Google has in the list of changes?
Industry News | | Ben-HPB0 -
SEO sites were blasted by Panda
Just noticed that lots of SEO sites were blasted by Panda... http://trends.google.com/websites?q=webmasterworld.com&geo=all&date=2011&sort=0 http://trends.google.com/websites?q=seomoz.org&geo=all&date=2011&sort=0 http://trends.google.com/websites?q=digitalpoint.com&geo=all&date=2011&sort=0 http://trends.google.com/websites?q=forums.seochat.com&geo=all&date=2011&sort=0
Industry News | | EGOL3