Google has discovered a URL but won't index it?
-
Hey all, have a really strange situation I've never encountered before. I launched a new website about 2 months ago. It took an awfully long time to get index, probably 3 weeks. When it did, only the homepage was indexed.
I completed the site, all it's pages, made and submitted a sitemap...all about a month ago. The coverage report shows that Google has discovered the URL's but not indexed them. Weirdly, 3 of the pages ARE indexed, but the rest are not.
So I have 42 URL's in the coverage report listed as "Excluded" and 39 say "Discovered- currently not indexed." When I inspect any of these URL's, it says "this page is not in the index, but not because of an error." They are listed as crawled - currently not indexed or discovered - currently not indexed.
But 3 of them are, and I updated those pages, and now those changes are reflected in Google's index. I have no idea how those 3 made it in while others didn't, or why the crawler came back and indexed the changes but continues to leave the others out.
Has anyone seen this before and know what to do?
-
Good luck!
-
Thanks Will, appreciate the insight. I'm going to get the Bing and Google wordpress plugins on there to see if that helps, build up a few more links and give it some time to wait and see. Thanks!
-
You're not the only person reporting odd indexation happenings here on Q&A (see for example this question). And, just like I found for that question, your site appears to have more pages indexed in Bing than in Google - which at least seems to point to us not having missed something obvious like meta noindex or similar.
I did also read Google saying that they had issues with the site: command (link) but I don't think that can have anything to do with your situation as they say they have now fixed that issue, and I couldn't find any other pages on your site even with non-site: searches (i.e. it does genuinely appear as though those pages are missing from the index).
While I am loathe to point just at links these days, I do wonder if in this case it is just a case of needing some more authority for the whole site before it is seen as big enough and important enough to justify more pages in the index.
-
Thanks, I've actually submitted request to be indexed multiple times over the last 3 weeks to no avail.
-
Hey Daniel. I agree with Chris. I have also noticed slow indexation recently. Might be a pain in the arse, but maybe you should request each page to be indexed individually in Search Console to add them to the high priority queue.
-
Hi Will, thanks for reaching out! No, not yet resolved. Still struggling to figure this out. I sent you a message on Facebook and Linkedin- would love to connect and try to get this figured out!
-
We keep adding blog posts almost every day, still not getting in the index for some reason. Discovered, yes. Crawled, yes. But not indexed, and no errors or anything.
-
Hi Daniel. Did you get this resolved / did it resolve itself? I'd happily take a look if you'd like if not - just let me know the URL.
-
My advice is, start listing more reviews! It will be picked up by google automaticly. You gotta be a bit more patient. New websites take awefully alot of time to be indexed.
I had a domain of 10+ years of age, replaced it's website, within one day completely reindexed. I have new domains, they can take up to weeks or even a month to be indexed. It's normal.
-
I actually got a quality link 2 weeks ago, but the blog post the link was published in still isn't indexed by Google either. The rest of his site is, just not his newest article for some reason, and it's 2 weeks old now. Another mystery...
-
It's a review website, and only 3 of my 24 reviews are indexed. All are discovered, most even crawled, but only those three in the index. And when I updated them, the search listing in Google results was updated within a few days. So they came back, are aware of the changes, but just not adding the others to the index.
And there are no affiliate links on this site at all. No spam, no links to spam, and I've attached a blog with 500+ word well written articles (about 20 so far) and none of the blog posts are indexed either.
I've never seen anything like this. The content is good, but almost none of it is getting indexed for some reason, despite being discovered and crawled.
-
Get quality links.
-
It's a new domain, no previous ownership, and no issues detected in search console for manual actions or security. There's no robots, noindex or any of that going on. They just won't index a bunch of the pages for some reason and it's very odd.
-
Perhaps the content on those 42 pages or so is alot copy content based? Or pages that really dont matter to be up in search?
-
I'd hold off worrying about it for now. I've heard many people talk about slow indexation lately. In the mean time, aside from the obvious check-for- nofollow- noindex-robots.txt suggestions, have you looked into the history of this domain? By chance was it penalized before you bought it?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google's Knowledge Panel
Hi Moz Community. Has anyone noticed a pattern in the websites that Google pulls in to populate knowledge Panels? For example, for a lot of queries Google keeps pulling data from a specific source over and over again, and the data shown in the Knowledge Panel isn't on the target page. Is it possible that Google simply favors some sites over others and no matter what you do, you'll never make it into the Knowledge box? Thanks.
Intermediate & Advanced SEO | | yaelslater0 -
Should I use noindex or robots to remove pages from the Google index?
I have a Magento site and just realized we have about 800 review pages indexed. The /review directory is disallowed in robots.txt but the pages are still indexed. From my understanding robots means it will not crawl the pages BUT if the pages are still indexed if they are linked from somewhere else. I can add the noindex tag to the review pages but they wont be crawled. https://www.seroundtable.com/google-do-not-use-noindex-in-robots-txt-20873.html Should I remove the robots.txt and add the noindex? Or just add the noindex to what I already have?
Intermediate & Advanced SEO | | Tylerj0 -
Need help in de-indexing URL parameters in my website.
Hi, Need some help.
Intermediate & Advanced SEO | | ImranZafar
So this is my website _https://www.memeraki.com/ _
If you hover over any of the products, there's a quick view option..that opens up a popup window of that product
That popup is triggered by this URL. _https://www.memeraki.com/products/never-alone?view=quick _
In the URL you can see the parameters "view=quick" which is infact responsible for the pop-up. The problem is that the google and even your Moz crawler is picking up this URL as a separate webpage, hence, resulting in crawl issues, like missing tags.
I've already used the webmaster tools to block the "view" parameter URLs in my website from indexing but it's not fixing the issue
Can someone please provide some insights as to how I can fix this?0 -
Help me to understand why this page doesn't rank
Hello everyone. I am trying to understand why most of my website category pages don't show up in the in the first 50 organic results on Google, despite my high website DA and high PA of those pages. We used to rank high a few years ago, not clear why most of those pages have almost completely disappeared. So, just to take one as an example, please, help me to understand why this page doesn't shows up in the first 50 organic search results for the keyword "cello sheet music": http://www.virtualsheetmusic.com/downloads/Indici/Cello.html I really can't explain why, unless we are under some sort of "penalization" or similar (a curse?!)... I have analyzed any possible metric, and can't find a logical explanation. Looking forward for your thoughts guys! All the best, Fab.
Intermediate & Advanced SEO | | fablau0 -
Why isn't the canonical tag on my client's Magento site working?
The reason for this mights be obvious to the right observer, but somehow I'm not able to spot the reason why. The situation:
Intermediate & Advanced SEO | | Inevo
I'm doing an SEO-audit for a client. When I'm checking if the rel=canonical tag is in place correctly, it seems like it: view-source:http://quickplay.no/fotball-mal.html?limit=15) (line nr 15) Anyone seing something wrong with this canonical? When I perform a site:http://quickplay.no/ search, I find that there's many url's indexed that ought to have been picked up by the canonical-tag: (see picture) ..this for example view-source:http://quickplay.no/fotball-mal.html?limit=15 I really can't see why this page is getting indexed, when the canonical-tag is in place. Anybody who can? Sincerely 🙂 GMdWg0K0 -
Monthly Refreshes Aren't Actually Needed, Right?
We get tons of emails from Network Solutions with the following text: To ensure that your website is easily found online it is important that you submit your website to the major search engines and internet directories, including: | Google™ Google Places™ Google Mobile™ Bing™ Yahoo!<sup>®</sup> Twitter<sup>®</sup> | Facebook<sup>®</sup> CitySearch<sup>®</sup> Foursquare™ Angie's List<sup>®</sup> GPS navigation MerchantCircle<sup>®</sup> | To do so, we recommend you go to each search engine and internet directories web page, locate the instructions and then complete a monthly refresh of your listing. If you would like us to complete this process for you please call us at... Everything I've ever read about modern SEO says this isn't necessary and it's just a solicitation to get people to pay them for something they don't even need. We update our social pages regularly and maintain listings on many citation sites using Moz Local (in addition to manually building citations). Can you guys confirm that this is just more spam from Network Solutions?
Intermediate & Advanced SEO | | ScottImageWorks0 -
Why are bit.ly links being indexed and ranked by Google?
I did a quick search for "site:bit.ly" and it returns more than 10 million results. Given that bit.ly links are 301 redirects, why are they being indexed in Google and ranked according to their destination? I'm working on a similar project to bit.ly and I want to make sure I don't run into the same problem.
Intermediate & Advanced SEO | | JDatSB1 -
Google News URL Structure
Hi there folks I am looking for some guidance on Google News URLs. We are restructuring the site. A main traffic driver will be the traffic we get from Google News. Most large publishers use: www.site.com/news/12345/this-is-the-title/ Others use www.example.com/news/celebrity/12345/this-is-the-title/ etc. www.example.com/news/celebrity-news/12345/this-is-the-title/ www.example.com/celebrity-news/12345/this-is-the-title/ (Celebrity is a channel on Google News so should we try and follow that format?) www.example.com/news/celebrity-news/this-is-the-title/12345/ www.example.com/news/celebrity-news/this-is-the-title-12345/ (unique ID no at the end and part of the title URL) www.example.com/news/celebrity-news/celebrity-name/this-is-the-title-12345/ Others include the date. So as you can see there are so many combinations and there doesnt seem to be any unity across news sites for this format. Have you any advice on how to structure these URLs? Particularly if we want to been seen as an authority on the following topics: fashion, hair, beauty, and celebrity news - in particular "celebrity name" So should the celebrity news section be www.example.com/news/celebrity-news/celebrity-name/this-is-the-title-12345/ or what? This is for a completely new site build. Thanks Barry
Intermediate & Advanced SEO | | Deepti_C0