Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a way to get a list (backlink profile) of all tiny url's that point to my site or a competitors site?
I have noticed that most all links you find in all the major back link profile tools such as OSE or GWM, etc... do not show tiny url's. If there is a service that shows all the tiny urls pointing to your site, can someone please share. It has already been proven that tiny url's do pass link juice, so with that being said... if there is no way to find all the tiny urls that point to a site, wouldn't it be a great strategy to create all my back links with tiny url's to mask my profile from competitors? Thanks!
Industry News | | johnd57890 -
What are the click-through rates by position for organic search on Google by industry?
I'm interested in knowing what the average CTRs are by industry (specifically for the credit card industry). Here's an example of what I want in aggregate (not by industry): http://searchengineland.com/organic-click-thru-rates-tumbling-study-97338 Is this information available?
Industry News | | wepayinc0 -
Does Google still have a standard search result? How can I get it?
I have heard a lot from the experts that there are no "Standard" Google search results anymore. They said that most of the SERP's of Google that show up are customized/tailored for each individuals even if they are not logged-in using their Google Custom Search. My questions are, Is there still a way to retrieve the standard Google search result? How? Are these scripts will be helpful when searching on Google? *webhp?
Industry News | | RafaelRanada
*complete=0
*pws=0 watch?v=B8ofWFx525s B8ofWFx525s watch?v=B8ofWFx525s0 -
How elated do you get?
So I was wondering, as from time to time I get a tiny bit excited if I manage to gain a new link that I've tried hard to get or I manage to outrank that **** that has been stuck at number 1 for so long or if I managed to do some A/B testing which has led to more leads and conversions. Does anybody else cheer inside or even have a short outburst of YEAAAH! f*** you Wikipedia! when they do something worthwhile? I may be on my own here..
Industry News | | Hughescov0 -
Google Search Quality Team - Commission Based Reviews
I have been busy this past week writing articles for various sources about the recent update on Google. A number of people contacted me about the analysis I was doing and the report. Some were members of the Google Search Quality Team. I knew manual reports were done before - but after the documents they showed me regarding the reports they do and the compensation for doing the reports - I am left in a state of being pretty shocked. May be I have been naive for all these years but I didn't realize that; Google outsourced the review and reconsideration requests to individual reviewers for a compensation Google's position in terms of checking qualification and experience of these "reviewers" was very insufficient at best, The three contacts I spoke to who had done reports had very little training or experience. I went through the GSQT REVIEWERS PDF (a very long and thorough document) that I was sent - with them. We went together through some sites I wanted them to review and their comments that came back were quite astounding to say the least and would have made many of you Mozzers laugh. Obviously I don't want to post said document online here.... BUT, I wanted to know if: a) any Mozzers had ever been part of such a group - the GSQT b) had any dealings with them - in terms of having your website reviewed and known about it. I knew about this group way back - like in 2005 or 2006 or sometime around then - I was told at time it was stopped and Google had stopped paying these sub contractor reviewers. Please don't get me wrong here... totally on board with manual reviews... I would just prefer them done by a trained team that possibly worked for either a professional company that maintain high quality review testing and standards - or for that matter GOOGLE employees that were trained. I just am a little unsure of them being done by individual subbies that get paid for the amount they do. What if that subbie has got some skin in the game for a particular keyword? What if their knowledge about certain aspects isn't up to par or not tested on a regular basis. This space is always changing and as you guys ./ girls on this forum know - it can change pretty quick. I just would want all websites to be judged fairly and equally by a group trained EQUALLY and to the same standards. I don't care if this is a G team or not - I just want it to be a team that is trained equally and trained continuously as opposed to paying outside people based on numbers of reviews done. When the livelihood of a small business is the balance I don't want a commission hungry toe rag with one years experience being the gate keeper for me or any of our clients. Carlos
Industry News | | CarlosFernandes0 -
Google driving me Nuts - How do you combine 2 accounts?
I know this must be driving a lot of other people mad as I see loads of people who now have 2 registered accounts at google plus due to their seemingly terrible ability to merge or connect accounts. We have a work email address set up through google, then I have a personal Gmail address. In Google Plus now I have 2 profiles - even though I have not signed up to google plus with my work email, I cannot add this email to my Google + account set up on personal email as it just tells me to log into that account taking me to a page to set up a profile for that account. Has anyone managed to solve this problem - it is happening to everyone in the company, driving us all nuts and our IT guys have no idea how to solve it. Were trying to use G+ for the purposes of SEO & Marketing, but if they make it this cumbersome for people to use, then they are going to die a quick death - after a few weeks of use and noticing the huge number of dead accounts, only live accounts being SEO/Internet Marketing related, and huge number of duplicate accounts, I think their user figures are hugely suspect! Rant over - anyone know how to merge accounts?
Industry News | | James770 -
Does anyone have a copy of the 2011 Google Quality Raters Handbook that was recently leaked?
http://searchengineland.com/download-the-latest-google-search-quality-rating-guidelines-97391 Google has been on a conquest taking them down online but I would really like to take a look at it if you have a copy! [moderator note - please use the PM system and exchange email addresses there. We've removed emails from this thread before it gets indexed and exposed to the world]
Industry News | | altecdesign4 -
Did Google's Farmer Update Positively/Negatively Affect Your Search Traffic?
See the attached image, showing a comparison of SEOmoz.org's search traffic from Google over the past 6 days and the prior week. farmer-3x.gif
Industry News | | randfish2