Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Article marketing sites
Hi, I'm looking for article marketing sites in English. I have searched and analyzed over 50 sites but I have not find any with the specifics I'm searching to: high/average DA, dofollow, and the possibility to have an anchored text, any suggestions? Thanks!
Industry News | | eriksatie0 -
Are there any noob sites that might enlighten me to how everyone earns income here?
I know...very odd beginner question so let me elaborate. I have always worked in the corporate grind. I quit and opened up a couple of small (fairly unsuccessful) businesses since 2007. I enjoyed the SEO parts of building the business but just couldn't get them to stick. (local service businesses) I have fallen into some lead gen business with my old customers and I am trying to build that now. How are the majority of the SEO people earning income here? Are most selling their SEO services to other small businesses? Are most earning ad type income here? (how does that work?) I am quickly being forced back into the corporate grind again and would really prefer to learn and taylor my SEO efforts into staying self employed if possible. Would anyone be able to direct me to sites that might explain how entrepreneurs are earning income around here? A little 'Career Counseling' of sorts? (haha) I sincerely would appreciate some enlightenment. 🙂
Industry News | | MontyTheLab2 -
What is the best way to become a published writer / blogger for bigger industry sites?
I'm a younger SEO manager and wanted to started establishing my name throughout the industry through well written blogs, expert articles, and any advice that is needed. I know a lot of answers will be to start my own blog and establish that and I'll get noticed, but is there a good way to make the first contact with industry sites to get an article or blog post published to their site? Or is it a pretty tight "gotta know someone" inner circle?
Industry News | | MERGE-Chicago0 -
What shall I call my site?
With all the google changes and social media focus on seo, I am in a bit of a dilemma and any opinions welcome. My husband has a very small horseback riding company based in a tiny mountain town of Costa Rica and I cannot decide whether to call it by the company name or by a more generic name that is likely to be picked up by the search engines ( I do believe the importance of domain name is less than it used to be in terms of search with all the new google changes). The two names are www.horsetrekmonteverde.com (bear in mind that his customers come from the States and horse trekking is not commonly used) and www.horseridingcostarica.com One mentions Costa Rica, the other not as you can see. Also not many people have beard about Monteverde. However I am trying to do some strong branding! What would you do?
Industry News | | Llanero0 -
3724 pages submitted, 3591 indexed
You probably know what I mean, the report in Google Webmaster Tools > Sitemaps. So how do I locate the pages that are NOT indexed? Thanks, Ben
Industry News | | bjs20100 -
We noticed that goods offered in our email newsletters used to disappeared from first google search results page!?
We noticed that goods offered in our email newsletters used to disappeared from fisrt google search results page. Goods where in top 5 positions or even higher, but after email newsletters we didn't find them even in top 100. We suspect service provider of email sending is in blacklist? Could it be reason? If yes, how could we check that?
Industry News | | Patogupirkti0 -
Two Industry giant has occupied the Google SE First Page.
I have seen just two websites on Google SE first page one with 4 and second one is with 6 internal pages on Coupon related question/niche . What it means Is this a Google Dance or temporary fluctuation. ? Why Google don't consider other sites to show on first page? Many pages from askvila.amazon, yahoo answers and facebook pages are shown on second and third pages.? Is this meant that there are a lack of quality websites/pages for this industry or query? What do you say? Here is the SERPs from Google US.(as on July 09) http://bitly.com/Md8JFW http://bitly.com/LaT9yD Thanks Alex
Industry News | | alexgray0 -
Bing beats google to disavow links
You can now disavow bad links in Bing WMT, google has stated they will be doing the same, this should shake up the rankings when many sites get penalties lifted http://www.bing.com/community/site_blogs/b/webmaster/archive/2012/06/27/disavow-links-you-don-t-trust.aspx
Industry News | | AlanMosley1