Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can not get the ranking needle to move even after a year with a US based SEO company
Hello community, I have spent the last 16 months spending lots of money on our USA Based SEO company and on content. For some reason. The powers who may be at Google are not giving us any love. I am beginning to become very dissapointed after all of our efforts. We did have a link penalty a year ago however that has long ago been removed. We have been doing everything our SEO company has asked us but the needle is not moving. Any ideas for us? http://tinyurl.com/n2yoeyn Thank you in advance. Chris
Industry News | | scamper0 -
Paid links from directory listing and business listing sites are good or bad according to Penguin 2.1 update?
Hi Friends, Recently on October 4<sup>th</sup>, 2013, a new spam filtering algorithm got live named Penguin 21. / Penguin 5. The update goes after sites that may have purchased paid links. I would like to know is it safe, if we submit website details with links in paid directories, eg: https://ecom.yahoo.com/dir/submit/intro/ (yahoo directory) and quality business listing sites provided the categories are related to our website. Our competitor sites having the backlinks from those kind of directories and they are performing (ranking) well in major search engines. May I know how Google treat these kinds of links according to this recent algorithm update?
Industry News | | zco_seo0 -
Google Analytics (Not Provided) Count will Increase 100% by Oct 2014 ? - Your Advice ?
What will you Do if you cannnot find your Top Keyword in Google Analytics "not provided"
Industry News | | Esaky
Check here for more details: http://www.notprovidedcount.com/0 -
Google Trusted Stores
Hello, So we sell millions of dollars a month in merchandise - most of that comes from eBay transactions. We do have a script that posts to eBay and we do download our transactions from eBay and process the orders from our admin. Now I feel we will do a lot better in the SERPs if we have the trusted stores quality signal. However; it comes down to this. The conversion pixel. Since the don't pay through the site - do you think we can get away of sending a email to a second conversion page for eBay transactions? Have any of you noticed a boost in SERPs once you were approved with the Trusted Stores? Any advise?
Industry News | | joseph.chambers0 -
Google Penguin 2.0 - Coming soon
There is an interesting article on SEW that Google is going to update Penguin to the next major version - http://bit.ly/15Vkr6O So what do you think, what should we expect? And also, is there available updated webmaster guidelines?
Industry News | | ditoroin1 -
So, Google is the best site on the internet.. Right? Or is that just what most people tend to think off-face?
LOL woah, put the guns away. I'm not about to rant, I just have a question and wanted to present it well. Then again, I might have actually found some easy fixes to some of Google's tools that they could make. So here's the thing. I noticed how annoyed I always getting when I have to sign in every time I go to the adwords keyword tool, or analytics. Why do you have to sign in a million times? I think it is a problem that can be fixed because if you go to check your webmaster tools, you go straight into your account, where you can then select which site you want to explore. It knows that I am already signed in to Google Accounts when I go to webmaster tools, but it doesn't recognize that fact when I go to my Analytics account, or to use the Adwords Keyword Tool. Now, every site has things that they need to work on, but not necessarily that need to be 'fixed'. Google being so commonly accepted as the best site on the net, I thought it was funny/interesting at the least to point out the problem. Even funnier is the fact that I could submit it as a problem to see if they could fix it or not, but they do such a good job of making it hard for people to contact them, that A) I don't feel like wasting my time trying B) I don't even really know if it is possible to do that. Also, why is there no official Google Analytics App / Mobile site?? Google has been pushing how important mobile is to us webmasters, but then it doesn't seem to be very high on their priority list for the tools that we use. I mean you can't view graphs on phones / tablets (mine at least), in webmaster tools, OR google analytics. Also, its a pain in the but to click the sign in button on Google Analytics when using my phone / tablet, it disapears really fast for me (needs more research from others to see if everyone has the same problem) Thanks for the interest / answers everybody. Look forward to hearing from you guys. Also, tips and help would be nice if anybody knows a solution to my sign in issue
Industry News | | TylerAbernethy0 -
Anyone else know much about the Google Pirate penalty?
The Google 'Pirate' (no official name) seems to have gone largely undiscussed since it was launched last - Fri 10th August http://insidesearch.blogspot.co.uk/2012/08/an-update-to-our-search-algorithms.html. The idea of it is to ensure those 'Pirating' content or abusing trademarks e.g. fake ugg boot sites and file sharing sites do not appear higher in the search results than the genuine websites. Google is using DMCA take down requests for labeling sites as Pirate and demote their rankings, Im amazed not even seomoz has covered the subject yet as far as I can see, yet it is a hugely important new update, albeit affecting a relatively small number of sites now, and in some cases (at least one I know first hand) seemingly without justification (the example I know is not a file sharing, fake goods, trademark abusive site at all.) Google updating its search algorithm based on DMCA take down requests seems a bit strong - these are takedown requests, not legal proof that a site is infringing a trademark. A real weapon for negative SEO? Anyone else had experience of the pirate update or know much more about it? Outside Danny Sullivan I dont see many SEO folk covering it. Heres my own insights into it and what ive learned about what (only innocently) affected sites should do to appeal http://www.andy-maclean.net/the-google-pirate-dmca-guidance/
Industry News | | AndyMacLean0 -
Chrome blocked sites used by Googles Panda update
Google's Panda update said it used Chrome users blocked sites lists as a benchmark for what they now term poor quality content. They said the Panda update effectively took about 85% of them out of the search results. This got me thinking, it would be very nice to discover what are the exact sites they don't like. Does anyone know if there is an archive of what these sites might be? Or if none exists, maybe if people could share their Chrome blocked sites on here we might get an idea?
Industry News | | SpecialCase0