Googlebot encountered extremely large numbers of links on your site??? How Do I resolve this?
-
I am working on a site with over 30 million pages. Every time I get about One Million indexed I get a Message in the Google Webmasters Tools saying "Googlebot encountered extremely large numbers of links on your site"
The indexing then starts dropping like a Rock. I need to get the site indexed. Please Help!
-
Kenneth
I work with extremely large sites quite often. There's no single answer to this because it depends on what's going on as to why the Googlebot is breaking down in its crawl. For example - how many links exist on any single page? Is it 100, 300, 1000 or more? The more links on every page the more likely the bot will choke, though it's a lot better than it used to be.
Does the site validate for markup? Or could there be choke-points due to validation errors?
Is the content organized in an intelligent funnel structure, or is everything one level off the root domain?
Is there only one way for the bot to navigate deep into the site, or are there multiple methods to get down deep?
Is some of the content only linked from within in a way that many of those links are not discovered until the bot has to first go through six or eight other layers, some of which could be timing out just when the bot gets there?
How many quality inbound links point to pages deep within the site?
These are all questions that need to be asked and answered and that's just scratching the surface of the problem potential.
The most important thing is to try and think like the bot - if I go here, will I become overwhelmed? if I go here, will I hit a road-block?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site property is verified for new version of search console, but same property is unverified in the old version
Hi all! This is a weird one that I've never encountered before. So basically, an admin granted me search console access as an "owner" to the site in search console, and everything worked fine. I inspected some URL's with the new tool and had access to everything. Then, when I realized I had to remove certain pages from the index it directed me to the old search console, as there's no tool for that yet in the new version. However, the old version doesn't even list the site under the "property" dropdown as either verified or unverified, and if I try to add it it makes me undergo the verification process, which fails (I also have analytics and GTM access, so verification shouldn't fail). Has anyone experienced something similar or have any ideas for a fix? Thanks so much for any help!
Reporting & Analytics | | TukTown1 -
PDF web traffic hitting our site
Hi there, Over the last few months our traffic has spiked due to irrelevant pdf documents sending us crap traffic, our bounce rate is sky high as well as other metrics. I don't want to just filter out this traffic in GA rather try and stop our site from being attacked. Any advice on a way forward would be great. Thanks
Reporting & Analytics | | ICMPmarketing0 -
Moz Crawl shows over 100 times more pages than my site has?
The latest crawl stats are attached. My site has just over 300 pages? Wondering what I have done wrong? RRv3fR0
Reporting & Analytics | | Billboard20120 -
Sudden Increase In Number of Pages Indexed By Google Webmaster When No New Pages Added
Greetings MOZ Community: On June 14th Google Webmaster tools indicated an increase in the number of indexed pages, going from 676 to 851 pages. New pages had been added to the domain in the previous month. The number of pages blocked by robots increased at that time from 332 (June 1st) to 551 June 22nd), yet the number of indexed pages still increased to 851. The following changes occurred between June 5th and June 15th: -A new redesigned version of the site was launched on June 4th, with some links to social media and blog removed on some pages, but with no new URLs added. The design platform was and is Wordpress. -Google GTM code was added to the site. -An exception was made by our hosting company to ModSecurity on our server (for i-frames) to allow GTM to function. In the last ten days my web traffic has decline about 15%, however the quality of traffic has declined enormously and the number of new inquiries we get is off by around 65%. Click through rates have declined from about 2.55 pages to about 2 pages. Obviously this is not a good situation. My SEO provider, a reputable firm endorsed by MOZ, believes the extra 175 pages indexed by Google, pages that do not offer much content, may be causing the ranking decline. My developer is examining the issue. They think there may be some tie in with the installation of GTM. They are noticing an additional issue, the sites Contact Us form will not work if the GTM script is enabled. They find it curious that both issues occurred around the same time. Our domain is www.nyc-officespace-leader. Does anyone have any idea why these extra pages are appearing and how they can be removed? Anyone have experience with GTM causing issues with this? Thanks everyone!!!
Reporting & Analytics | | Kingalan1
Alan1 -
Ok. I'm just going to cut loose with my stupid question. What is internal link equity? What distinguishes an internal link with equity from one without equity?
What distinguishes an internal link with equity from one without equity? Is there a limit to how many of these I want? What's the rule-of-thumb? Cheers, Wes
Reporting & Analytics | | wrconard0 -
Why My Site Got 1000% increase in organic traffic from day to night?
Did Google run any update Monday or recently? My site www.shirts4geek.com, strangely had a 1000% organic traffic increase from day to night. I didn't do anything in this site for long a time... but Monday I had a lot traffic coming from organic and every other day this week the site is doing extremely well on traffic and sales. I'm ranking first page for many keywords relate to my products. I wish I could figure out what happens so I can replicate it. The site has very links and the On Page Optimization is kind of basic. Does any have any idea how it could be possible? Have any one seen something similar lately?
Reporting & Analytics | | Felip30 -
An initial set up question - linking to GA
when viewing campaign it indicates there is a problem linking to our GA account but the details entered are correct. Any ideas what the problem might be - nothing on the campaign settings page gives me a clue. There is a highlighted box saying disconnect from GA but nothing else. The only thing i can think of is that the GA account was set up by a third party and we have access to view it - could it be a permissions problem?
Reporting & Analytics | | GreatGifts4Kids0 -
Problem when searching for "link:www.mysite.com" vs "link: www.mysite.com"
Why does a search for "link:www.mysite.com" show no results, but when there is a space before www.mysite.com it shows results? The same happens for "link:www.mysite.com" (nothing shows up), but when I search for "link:www.mysite.com/index.php" it returns results. Is there a problem I am missing? Thanks so much!
Reporting & Analytics | | EmilyP0