Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Very odd behavior. Google is changing “%20” to “+” in my URLS
I just realized many of the links on my site are BROKEN when entered from a Google SERP. This didn’t used to be this way. I have no idea what’s going but I’m worried. It involves a folder of our site that has a space in it. Google is even displaying the proper “%20” in the SERP but when I click the link it replaces that “%20” with a + which breaks the link!! You can see this in action by typing in “brown jordan sheffield furniture” without quotes into Google. You’ll see our site come up first and displaying this link sheffieldfurniture.com/Other%20Furniture/BrownJordan.html but when you click it the link is broken! This is happening on many of our pages! Anyone know what in the world is going on?
Industry News | | SheffieldMarketing0 -
How can i discover how many of my pages have been indexed by google?
I am currently in the process of trying to produce a report for my corporation and this is a metric that i cannot seem to find on OpenSiteExplorer. Could anyone help?
Industry News | | CF20150 -
Google Changes Up The Search Results Page
Hi Guys, As you Google has made changes on search results page. I have two points two discuss here : 1. Are we going to see more ads on left sidebar in future ? 2. I think it will also affect the CTR of top three ads in SERP ? Waiting for you guys opinion on it ? Reference: http://www.webpronews.com/google-changes-up-the-search-results-page-2012-11
Industry News | | SanketPatel1 -
Google: 7 Results on First page now (Good or Bad)?
Hi, So what are your thoughts on only 7 results on the first page of a search? Looks like its only in play for brands at the moment with increased sitelinks for the first result. I actually really like it, looks a lot better and its easier to take the overall information in on the page. I hope they role this out onto more generic terms as well. Should mean a higher CTR for those who have 'made it' to the first page. I suppose Google will pickup on a higher CTR on the ads well.
Industry News | | activitysuper0 -
How elated do you get?
So I was wondering, as from time to time I get a tiny bit excited if I manage to gain a new link that I've tried hard to get or I manage to outrank that **** that has been stuck at number 1 for so long or if I managed to do some A/B testing which has led to more leads and conversions. Does anybody else cheer inside or even have a short outburst of YEAAAH! f*** you Wikipedia! when they do something worthwhile? I may be on my own here..
Industry News | | Hughescov0 -
Google Webspam Algo Update 24/4/12
Having just checked our clients rankings 95% have not been affected, in fact many have moved up rankings. 1 or 2 have had big drops 😞 Who has been effected by this? The forums are full of people talking about sites being floored from the serp's. it will be interesting to follow the aftermath of this and get some insight into what exactly has changed!
Industry News | | ifluidmedia0 -
Does anyone have a copy of the 2011 Google Quality Raters Handbook that was recently leaked?
http://searchengineland.com/download-the-latest-google-search-quality-rating-guidelines-97391 Google has been on a conquest taking them down online but I would really like to take a look at it if you have a copy! [moderator note - please use the PM system and exchange email addresses there. We've removed emails from this thread before it gets indexed and exposed to the world]
Industry News | | altecdesign4 -
My websites Internal page is not ranking.
Hi, I had "x1", "x2","x3" keywords ranked for my websites internal page that is "www.somesite.com/internal-page.htm" i was fairly doing well for my keywords mostly in top 5, but now the case is from last couple of weeks, my internal page url is gone & replaced with home page. And I don't see internal page anywhere mostly in top 1000, is that penalty to internal page ?If yes how can i troubleshoot this ? I see my other pages in top 20-30 but that specific page is not anywhere. Please let me know. -Sunny.
Industry News | | sunny.popali0