Get Google To Crawl More Pages Faster on my Site
-
We opened our database of about 10 million businesses to be crawled by Google.
Since Wednesday, Google has crawled and indexed about 2,000 pages. Google is crawling us at about 1,000 pages a day now. We need to substantially increase this amount.
Is it possible to get Google to crawl our sites at a quicker rate?
-
Mark If you have had your category level or city/state level pages, whatever your architecture is, you'd see your detail pages getting crawled pretty quickly. Give it a week. I would also work on getting some deep links to your website. Google has mentioned several times, PR is a direct measure of how deep and how often Google crawls your website. So I would suggest working on building the domain authority and brand in terms of backlinks to the website. Other then that, as bstone81 said, generate sitemaps and sitemap of sitemaps and submit to Google.
-
Two things you can do:
1. Submit a site map with all those pages. They suggest breaking the sitemaps into 50,000 page groups.
2. Use the "Fetch as Google" feature in GWT and fetch your root domain then click the box to submit the root domain along with all the linked pages. This will submit the request to Google to be indexed.
-
Hi Mark,
Google only spends a certain amount of time to crawl and there is no way to tell them to crawl more or faster. If you have a lot of pages, it might take longer.
One way to make them crawl faster may be optimize your internal linking. If you have links pointing to other pages in your website, it makes it easier to crawl.
Other than work arounds, there are no ways to tell Google to crawl faster or more pages. They have their own speed.
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=34439
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why Does OSE (Open Site Explorer) have such little backlink data on russian sites in the google.ru index?
OK this seems v strange, but google.ru are indexing far more BLs in their SERPS for a widget than OSE reports. Very little data is found in OSE for russian based sites. Is this the marketing intention? (I could send raw data if needed!) What is filtering this vast google.ru data list out? Is OSE only catered for US/UK?
Industry News | | Turkey0 -
Google guidlines 2011
Guys I have asked for the leaked SEO guidlines just to fine tune my SEO campaigns and it seems no one wanted to send it to me. Anyone can do it here, please?
Industry News | | SearchOfficeSpace230 -
Any pointers for my site would be greatly apprecieated and rewarded.
We just started changing all the meta data and adding relevant blog content. Anything I'm missing? Any advise or "after panda" link building strategies? inkfarm.com
Industry News | | ibex0 -
Google's New Release "What do you love"
Is this gonna be a game changer for SEO http://www.wdyl.com/ Regards, Shailendra Sial
Industry News | | IM_Learner0 -
Question on Google's Differentiation
Hi everyone, We have a short question that probably has a very long answer: What are all the ways that Google differentiates one search result from another? This is really important for us to know and, up until now, we can't find a really good and complete source that explains this. Thank you in advance.
Industry News | | jid0 -
How do you value a site that generates no income before it's sold?
Hi, I'm currently creating a number of sites which target the wedding market. Each site is localized and specific to a niche within the wedding market. The intention is to create the site, rank it in the top position of Google and then sell it to a local business within that niche and area. The problem that i am having is that I don't know how to place a value on each site. I have factors taken from Google (using Market Samurai) so I know the broad and phrase match traffic details as well as the Google PPC average prices. My initial thought was to use the Google PPC price and multiply that by the number of organic visitors the site would receive if it was ranked #1 on Google. I appreciate that the PPC figures may be higher than what an advertiser would actually pay per click so I conservatively halved the Google PPC figure given in Market Samurai. Is this a good way of valuing the site or do I need to look at the value per customer to the business that I am trying to sell the site to and work from a conversion factor based on the number of visitors that the site would receive. If the site received 500 visitors per month and 5% of those became customers multiplied by the average profit per customer - that would give a monthly value of the website. I am very unsure as to what the best approach is here and look forward to hearing any advice and ideas from fellow members. Thanks, Michael
Industry News | | XSMedia0 -
Anyone else seeing a new google search result look?
Just noticed my google search results page has changed. Anyone else seeing this? Screenshot http://cl.ly/2h0v2i2w191O32082U0T
Industry News | | JohnTurner791 -
How to remove a Google algorithmic penalty
My site has a Google penalty. I seem to be stuck in the 64th position for a Google search for my sites name. All my keywords that I used to rank well for are now well above the 60th search place in Google. I have resolved the issue I recieved the penalty for and I have asked Google for reconsideration. That has been about 3 months ago. The penalty is still firmly in place. I was wondering if anyone else has had a Google algorithmic penalty removed and if so how did they accomplish this?
Industry News | | tadden0