Google indexing is slowing down?
-
I have up to 20 million unique pages, and so far I've only submitted about 30k of them on my sitemap.
We had a few load related errors during googles initial visits, and it thought some were duplicates, but we fixed all that. We haven't gotten a crawl related error for 2 weeks now.
Google appears to be indexing fewer and fewer urls every time it visits. Any ideas why? I am not sure how to get all our pages indexed if its going to operate like this... love some help thanks!
-
Try getting links for important pages. As for google crawling and indexing: i have websites that contain 10k of URL's on which i know, half of that is not relevant for search at all. Google does it completely automaticly and cuts out the most duplicate or non-relevant pages in comparison to what i offer in my sitemap and brings 4k of pages or so in actual search.
It's really no issue. You cant tell me you got 20 million of unique (user) generated content.
-
Share your URL for better responses.
Have you submitted a complete XML sitemap through Google Search Console? And are all the pages free of a no-index meta tag?
20 million unique pages can only be created in some automated fashion based on a much smaller amount of original content, so Google may perceive these pages to be spammy and not worth indexing.
-
Ryan,
G doesn't crawl/index every url it encounters. Its decision to do so is based on it perceived value of the the page/the page the link is one. You can increase that value/PR/PA by sculpting your internal links, revised silo-ing of your content and/or acquiring backlinks to pages deeper within your site.
If you're a brand new site with 20M pages, what reason have you give google to crawl all those pages. Just submitting them or listing them on a sitemap isn't enough of a reason. I mean, 20M pages! google's like damn dude, what's up with all that?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How long does Google/Bing take to index
Hello we have 2-3 new pages being submitted every night to google/bing via our sitemap. Two issues I am noticing. Wondering if anyone else has same issues. a) 22 URL submitted via sitemap but only 1 indexed in two weeks. there are no errors showing b) If i submit manually using "Fetch As Google" and request indexing - the page gets indexed right way but after a day it seems to be unindexed - it will show up when i search (site:domain.com) but then disappear from the results doing the same search a few days later. Is this normal or do i have a problem that needs addressing? thank you
Technical SEO | | sancarlos0 -
Google Indexing Development Site Despite Robots.txt Block
Hi, A development site that has been set-up has the following Robots.txt file: User-agent: * Disallow: / In an attempt to block Google indexing the site, however this isn't the case and the development site has since been indexed. Any clues why this is or what I could do to resolve it? Thanks!
Technical SEO | | CarlWint0 -
Google Webmaster tools Sitemap submitted vs indexed vs Index Status
I'm having an odd error I'm trying to diagnose. Our Index Status is growing and is now up to 1,115. However when I look at Sitemaps we have 763 submitted but only 134 indexed. The submitted and indexed were virtually the same around 750 until 15 days ago when the indexed dipped dramatically. Additionally when I look under HTML improvements I only find 3 duplicate pages, and I ran screaming frog on the site and got similar results, low duplicates. Our actual content should be around 950 pages counting all the category pages. What's going on here?
Technical SEO | | K-WINTER0 -
Does Google Still Pass Anchor Text for Multiple Links to the Same Page When Using a Hashtag? What About Indexation?
Both of these seem a little counter-intuitive to me so I want to make sure I'm on the same page. I'm wondering if I need to add "#s to my internal links when the page I'm linking to is already: a.) in the site's navigation b.) in the sidebar More specifically, in your experience...do the search engines only give credit to (or mostly give credit to) the anchor text used in the navigation and ignore the anchor text used in the body of the article? I've found (in here) a couple of folks mentioning that content after a hashtagged link isn't indexed. Just so I understand this... a.) if I were use a hashtag at the end of a link as the first link in the body of a page, this means that the rest of the article won't be indexed? b.) if I use a table of contents at the top of a page and link to places within the document, then only the areas of the page up to the table of contents will be indexed/crawled? Thanks ahead of time! I really appreciate the help.
Technical SEO | | Spencer_LuminInteractive0 -
Backlinks Indexing
Is there a way of indexing my backlinks?? I have a lot backlinks but Google can't find them
Technical SEO | | CodePlus0 -
Google Penalty?
Hi, I have recently been asked to help www.mycanvas.ie I have a feeling they have a google penalty. All their Google Keywords have literally dropped out of the Google SERP but they are still shown on Yahoo SERP. I recently did a site:www.mycanvas.ie and the pages are still in google index. The only thing that comes to mind is that the site owner submitted to 380 web directories over a period of 2 months with http://www.directorymaximizer.com/ do you think this could be causing the problem with google? Advise and suggestions are welcomed, thank you.
Technical SEO | | Socialdude0 -
Frustration With Google Places
I have been trying to solve this problem with Google Places for quite some time now and just can't figure out where to go from here. I've tried several sent messages explaining the problem and even received several phone calls from Google Places trying to correct the issue with no luck. I have even tried totally deleting the listing and started over from scratch and re-verified the address with a mailed postcard. My site: http://www.captainrichsmith.com has a Google Places account set up and verified http://maps.google.com/maps/place?hl=en&georestrict=input_srcid:1c8fa43cf77e0c93&ie=UTF8&t=h&z=14&vpsrc=0 For some reason when you do a Google search for one of my keywords Miami Fishing Charters On the listings normally under the letter "E" on the Map another website has a placemark at my location Miami Fishing Charters Directory
Technical SEO | | captainrichsmith
www.fishing-charters-miami.com/ - Cached Fishing Charters Miami is a quality directory of the best fishing boats in the Miami area. The top Miami fishing charters are listed on this website.
2550 South Bayshore Drive, Miami
(786) 263-9231
captainrichsmith.com (7) When you view this Google places listing further. I see it is using my images, videos, placemark on map but NOT the address, phone number, or reviews. Any help on this issue would greatly be appreciated0 -
How can I get a listing of just the URLs that are indexed in Google
I know I can use the site: query to see all the pages I have indexed in Google, but I need a listing of just the URLs. We are doing a site re-platform and I want to make sure every URL in Google has a 301. Is there an easy way to just see the URLs that Google has indexed for a domain?
Technical SEO | | EvergladesDirect0