Googlebot encountered extremely large numbers of links on your site??? How Do I resolve this?
-
I am working on a site with over 30 million pages. Every time I get about One Million indexed I get a Message in the Google Webmasters Tools saying "Googlebot encountered extremely large numbers of links on your site"
The indexing then starts dropping like a Rock. I need to get the site indexed. Please Help!
-
Kenneth
I work with extremely large sites quite often. There's no single answer to this because it depends on what's going on as to why the Googlebot is breaking down in its crawl. For example - how many links exist on any single page? Is it 100, 300, 1000 or more? The more links on every page the more likely the bot will choke, though it's a lot better than it used to be.
Does the site validate for markup? Or could there be choke-points due to validation errors?
Is the content organized in an intelligent funnel structure, or is everything one level off the root domain?
Is there only one way for the bot to navigate deep into the site, or are there multiple methods to get down deep?
Is some of the content only linked from within in a way that many of those links are not discovered until the bot has to first go through six or eight other layers, some of which could be timing out just when the bot gets there?
How many quality inbound links point to pages deep within the site?
These are all questions that need to be asked and answered and that's just scratching the surface of the problem potential.
The most important thing is to try and think like the bot - if I go here, will I become overwhelmed? if I go here, will I hit a road-block?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Will noindex pages still get link equity?
We think we get link equity from some large travel domains to white label versions of our main website. These pages are noindex because they're the same URLs and content as our main B2C website and have canonicals to the pages we want indexed. Question is, is there REALLY link equity to pages on our domain which have "noindex,nofollow" on them? Secondly we're looking to put all these white label pages on a separate structure, to better protect our main indexed pages from duplicate content risks. The best bet would be to put them on a sub folder rather than a subdomain, yes? That way, even though the pages are still noindex, we'd get link equity from these big domains to www.ourdomain.com/subfolder where we wouldn't to subdomain.ourdomain.com? Thank you!
Reporting & Analytics | | HTXSEO0 -
Should Google Trends Match Organic Traffic to My Site?
When looking at Google Trends and my Organic Traffic (using GA) as percentages of their total yearly values I have a correlation of .47. This correlation doesn't seem right when you consider that Google Trends (which is showing relative search traffic data) should match up pretty strongly to your Organic Traffic. Any thoughts on what might be going on? Why isn't Google Trends correlating with Organic Traffic? Shouldn't they be pulling from the same data set? Thanks, Jacob
Reporting & Analytics | | jacob.young.cricut0 -
Conflicting numbers in Google Analytics
I am getting 2 completely different numbers in Google Analytics.According to the graph on my Dashboard organic traffic decreased ~40% comparing June 7 -> June 6.However, when I dig a bit deeper and look at the 2 dates specifically compared to each other I get a `3% increase in traffic.When I look at the traffic on just June 7, I get the number indicating the increase.Any ideas or someone getting similar conflicting numbers??
Reporting & Analytics | | theLotter0 -
Verifying Site Ownership & Setting Up Webmaster tools for clients who use Hubspot
We are a Hubspot partner agency. I'm trying to find the best route for managing Google's tools as an extra resource for insight, not the primary basis for marketing effort. I also want to explore adwords in more depth. Finding a lot of our clients don't have one or the other or both Analytics/Webmaster tools in place. Can I verify site ownership to set up webmaster tools simply by having admin access to their analytics account or will that require ownership of the analytics account? With Google merging things together these days I'm not sure of the best approach to take. Usually clients have their site hosted somewhere and built on some platform and ADD a Hubspot blog and the landing pages/cta's, Hubspot tools on a subdomain hosted by Hubspot. Hubspot has tools in it's website settings for adding google analytics (actually it's just a field to add code to the header area). If a client has universal analytics on their primary domain do I still need to go and add a separate analytics property for the subdomain and go through Hubspot's tools to install it on the subdomain? Or just use the same code from their primary domain and add it to the Hubspot header? What is the best route? Any additional thoughts on this subject are welcome - with so much updating and changing coming from Google (and Hubspot as we implement 3.0 - COS) I'm trying to avoid wasted effort, outdated methods, etc. Thanks!
Reporting & Analytics | | rhgraves651 -
New GA interface - Number of conversion by source
Hi Guys, Hope this is an easy one for a GA ninja. In the new GA interface, how can I view the number of conversions by source, instead of the default conversion rate by source. For example, if I want to view the number of conversion from Adwords. I'm looking at Goals > Source / Medium > Google CPC and looking at Goal Set 1. However all data is expressed in percentages. Thanks in advance!
Reporting & Analytics | | David_ODonnell0 -
Internal site referrers
Hi, So I have a segment of my website-let’s call it /examplea, I am trying to figure out how many visits I have to /examplea from all other areas of my website i.e. /exampleb, /examplec etc to /examplea so almost internal site refers to a particular segment of my website, Any thoughts on how to do this within Google analytics ? Marc
Reporting & Analytics | | NRMA0 -
Unique root linking domains - clarification
Hi guys, In SEOMoz Search Ranking Factors, one of the the top ranking factors is number of unique root domains linking to the page: http://www.seomoz.org/article/search-ranking-factors#metrics-5 My question is: do these unique root domains need to be unique root domains liking to my domain also? E.g. www.mydomain.com/landingpage1/ already got a link from www.externaldomain.com If www.externaldomain.com has another link pointing to www.mydomain.com/landingpage2/ will this link be counted? If yes will the value be diluted as www.externaldomain.com has already linked to www.mydomain.com/ Many thanks. David
Reporting & Analytics | | sssrpm0 -
Why is Webmaster Tools not Linking Correctly Google Analytics?
I have a global site with .com as the US site and then /country for all other markets i.e. .com/uk .com/de etc. Each site has it's own WMT profile. Each site has it's own GA profile. Since Google added Search Engine Optimization to the GA interface, I can't track these sites separately under this new feature. It seems to me that it can only be associated with a single WMT profile at a time or rather that WMT can only be associated with a single GA profile at a time. The numbers for the UK and US site are identical, but when I try and link the UK WMT account with GA it removes it from the US profile and vice versa.
Reporting & Analytics | | Red_Mud_Rookie0