Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My site got hacked and now i have 1000s of 404 pages and backlinks. Should i transfer my site to a new domain name and start again?
My site was hacked and I had 1000s of pages that should not exist created and has had 1000s of backlinks put in. Now i have the same pages and backlinks redirecting to 404 pages. Is this why my site crashed out of google and my SEO fixes since have made no progress to my problem?
Industry News | | KeithWarbyUK0 -
Using Google Reviews for non local business
Hi All, We are deciding on what site is best to capture reviews from customers and I'm just not sure what is the ideal option. We are a SaaS business with multiple offices in different locations but the specific geographies are not really relevant to our customers. Is it worth focusing on google reviews so that when our brand is searched there are plenty of nice shiny stars (plus maybe they can be added into adwords adverts as well...). Search volume for the keywords we don't yet rank for are not massive although still important. Alternatively should we be thinking about something like G2Crowd. None of our competitors are doing anything so there's no real need to our muscle them on a review website and I don't think our end user will visit these sites before buying but we would point to them and say 'hey, look at all these great reviews'. Finally I searched my old company recently who and just under news results were facebook reviews. Maybe that's another option. All advice welcome. All advice appreciated.
Industry News | | jafayeh1 -
Google penalty removal expert questions
We have searched online for a Google penalty “expert” (individual or company) and have located what appear to be “experts”. Please provide feedback on the following 2 individuals/companies we have found that can help with penalty removal. Have you or one of your clients used either of the “experts” below? What were the results? How many disavows and reconsideration requests did you/they have to make? 1.www.penaltypros.com . To give a quote and to see what your links are they use links from Google Webmaster Tools only. Penaltypros.com disavows first and then removes bad links second. This is opposite of what Google and Seo’s recommend but penaltypros.com claims 100% success using this non-traditional approach. See imgur.com link for screenshot. 2.http://www.hiswebmarketing.com/ To give a quote and to see what your links are they use links from https://ahrefs.com/ only. Please provide any and all feedback on the above 2 “experts” and also post the websites, individual names, company names of those that you consider Google penalty removal “experts” so that we may obtain a quote from them. Lp9F3FI
Industry News | | RetractableAwnings.com1 -
Paid links from directory listing and business listing sites are good or bad according to Penguin 2.1 update?
Hi Friends, Recently on October 4<sup>th</sup>, 2013, a new spam filtering algorithm got live named Penguin 21. / Penguin 5. The update goes after sites that may have purchased paid links. I would like to know is it safe, if we submit website details with links in paid directories, eg: https://ecom.yahoo.com/dir/submit/intro/ (yahoo directory) and quality business listing sites provided the categories are related to our website. Our competitor sites having the backlinks from those kind of directories and they are performing (ranking) well in major search engines. May I know how Google treat these kinds of links according to this recent algorithm update?
Industry News | | zco_seo0 -
Google number one search result looks drastically different in firefox compared to chrome
I just noticed this today that some websites and brands look like this on firefox only, and others while still being number one result for their brand name, do not appear like this at all. also, this does not happen over chrome at all. both images provided for comparison are using the same google apps account logged in. It would be nice if someone could shed some light on as to why this happens sporadically and what does it take to be distinguished like this for your own brand if you own the identical domain.com or whatever. Zz7ZkX5.png lpuwheo.png
Industry News | | Raydon0 -
Best ranking Magento sites?
What would be your top 10 Magento sites in terms of SEO and ranking? I'm looking for examples of Magneto sites that have been set up correctly and are ranking well. Especially recently launched sites that have managed to get themselves ranking well in competitive markets! Cheers
Industry News | | OnlineAssetPartners0 -
So, Google is the best site on the internet.. Right? Or is that just what most people tend to think off-face?
LOL woah, put the guns away. I'm not about to rant, I just have a question and wanted to present it well. Then again, I might have actually found some easy fixes to some of Google's tools that they could make. So here's the thing. I noticed how annoyed I always getting when I have to sign in every time I go to the adwords keyword tool, or analytics. Why do you have to sign in a million times? I think it is a problem that can be fixed because if you go to check your webmaster tools, you go straight into your account, where you can then select which site you want to explore. It knows that I am already signed in to Google Accounts when I go to webmaster tools, but it doesn't recognize that fact when I go to my Analytics account, or to use the Adwords Keyword Tool. Now, every site has things that they need to work on, but not necessarily that need to be 'fixed'. Google being so commonly accepted as the best site on the net, I thought it was funny/interesting at the least to point out the problem. Even funnier is the fact that I could submit it as a problem to see if they could fix it or not, but they do such a good job of making it hard for people to contact them, that A) I don't feel like wasting my time trying B) I don't even really know if it is possible to do that. Also, why is there no official Google Analytics App / Mobile site?? Google has been pushing how important mobile is to us webmasters, but then it doesn't seem to be very high on their priority list for the tools that we use. I mean you can't view graphs on phones / tablets (mine at least), in webmaster tools, OR google analytics. Also, its a pain in the but to click the sign in button on Google Analytics when using my phone / tablet, it disapears really fast for me (needs more research from others to see if everyone has the same problem) Thanks for the interest / answers everybody. Look forward to hearing from you guys. Also, tips and help would be nice if anybody knows a solution to my sign in issue
Industry News | | TylerAbernethy0 -
Question on Google's Differentiation
Hi everyone, We have a short question that probably has a very long answer: What are all the ways that Google differentiates one search result from another? This is really important for us to know and, up until now, we can't find a really good and complete source that explains this. Thank you in advance.
Industry News | | jid0