Indexed, though blocked by robots.txt: Need to bother?
-
Hi,
We have intentionally blocked some of the website files which were indexed for years. Now we receive a message "Indexed, though blocked by robots.txt" in GSC. We can ignore as per my knowledge? Are any actions required about this? We thought of blocking them with meta tags but these are PDF files.
Thanks
-
Hi there!
What Google is telling you is that you are indexing URLs that you probably are not wanting to be indexed, or the other way around, that important pages are being blocked but indexed for other reasons.
If I might ask, why did you blocked through robots.txt those files?
There most 2 answers are:
1- Wanted to remove those from search results. If this is your case, you've solved only a part of the problem. What you should have done is (previously allowing robots to crawl those urls) apply noindex rules (keep in mind that can be set up in the HTTP header, as long as not html files cant have meta robots tag), then after a sufficient time block them in robots.txt.
_2- Optimize how GoogleBot (crawiling) time. _Being this case, then you've done it correctly and there is nothing to worry.Hope this help.
Best luck.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Console Not Indexing Pages
Hi there! I have a problem that I was hoping someone could help me with. On google search console, my website does not seem to be indexed well. In fact, even after rectifying problems that Moz's on-demand crawl has pointed out, it still does not become "valid". There are some of the excluded pages that Google has pointed out. I have rectified some of the issues but it doesn't seem to be helping. However, when I submitted the sitemap, it says that the URLs were discoverable, hence I am not sure why they can be discovered but are not deemed "valid". I would sincerely appreciate any suggestions or insights as to how can I go about to solve this issue. Thanks! Screenshot+%28341%29.png Screenshot+%28342%29.png Screenshot+%28343%29.png
Algorithm Updates | | Chowsey0 -
Still no good search results after 2 months of indexation
Hi guys, One of our website (https://www.residentiebosrand.be/) has been online for about two months. It's indexed and Google shows search results. But the website is not ranking on the keywords it's supposed to be ranking: 'residentie bosrand'. How come we still don't find the website on the first pages in the search results, while these are the main keywords on the website's URL, page, ... ? Best regards,
Algorithm Updates | | conversal0 -
Trafic drop after a huge indexation
Hello everyone, My website used to have about 500k indexed pages in Google. After publishing fresh sitemaps and a little local "buzz", it now has about 6 millions indexed pages and the numbers are skyrocketing (GWT says 7 millions and it will probably keep going). My website has a total number of pages of 10 millions. I used to have about 5k organic visite each day, but since the big indexation has started, I now have half less. I read many things about that kind of trafic drop, and it seems to be a normal step when indexing a huge site. I just wanted to know if you guys had any similar experiences and if yes, if there are specific tasks to do in order to recover/develop the organic trafic or if it's just a matter of time. Thanks for your help and share of experiences 😉
Algorithm Updates | | Pureshore0 -
Does articles for SEO purposes have a minimal and maximum word count in ordered to be crawled/indexed by Google and other search engines?
Does articles for SEO purposes have a minimal and maximum word count in ordered to be crawled/indexed by Google and other search engines?
Algorithm Updates | | WebRiverGroup0 -
Google Shopping Blocking All Vitamins and Natural Products - Glitch or Deliberate Censorship?
Hi everyone. We have a client that manufactures and supplies dietary supplements all around the world. We are slightly concerned that a recent Google shopping glitch (or change) is now seemingly excluding products from the shopping search results. This currently appears only to be happening in the US but we are really concerned, as our client ships all over the world and the potential loss of revenue could be quite large. There is already a YouTube video that demonstrates what is going on which is available below: http://www.youtube.com/watch?v=zNDyS0tF4dY Just to clarify, these are products that should not be included in any of Google's “sensitive” categories as they currently stand. Taking Vitamin B12 as an example, it is recognised as a permissible dietary supplement within pretty much every regulatory framework around the world, including those governed by the US FDA, The Euopean Commission and the Australian TGA. Therefore there would be no legal reasons to prevent it's inclusion in shopping results in any country. Has this just slipped under the radar or can anyone point us to a resource that may be able to clarify why this has happened? Thanks in advance guys!
Algorithm Updates | | AduroLabs0 -
Google's not indexing my blog posts anymore! Why?
Google just recently stopped indexing my blog posts immediately after being published, why could this be? I would usually post a blog post and it would be in google results within 45 seconds, now they don't show up until 6 hours later, if at all (a few never even showed up). Also, my home page doesn't even refresh when I make a change to the site. My site is CantStopHipHop [dot] comI have all in one SEO, xml sitemap generator, and webmaster tools and nothing seemed irregular in the settings.I appreciate any thoughts/help/suggestions.
Algorithm Updates | | bb2550 -
How to speed up indexing of my site...
Only 4 out of the 12 pages of my blog/site have been indexed. How can I ensure all the pages get indexed? I'm using a wordpress site, and I also wondered how could I speed the indexing process up (I have submitted a site map) Thanks!
Algorithm Updates | | copywritingbuzz0 -
I need help with drastic SERP difference between Bing and Google
One of our sites that has been around for a couple of years has about 60,000 pages showing on google, however, bing only shows 90 pages for the site. This same phenomenon has been happening across the board for our sites. Any ideas to improve our indexing results for bing?
Algorithm Updates | | atuomala0