Indexed, though blocked by robots.txt: Need to bother?
-
Hi,
We have intentionally blocked some of the website files which were indexed for years. Now we receive a message "Indexed, though blocked by robots.txt" in GSC. We can ignore as per my knowledge? Are any actions required about this? We thought of blocking them with meta tags but these are PDF files.
Thanks
-
Hi there!
What Google is telling you is that you are indexing URLs that you probably are not wanting to be indexed, or the other way around, that important pages are being blocked but indexed for other reasons.
If I might ask, why did you blocked through robots.txt those files?
There most 2 answers are:
1- Wanted to remove those from search results. If this is your case, you've solved only a part of the problem. What you should have done is (previously allowing robots to crawl those urls) apply noindex rules (keep in mind that can be set up in the HTTP header, as long as not html files cant have meta robots tag), then after a sufficient time block them in robots.txt.
_2- Optimize how GoogleBot (crawiling) time. _Being this case, then you've done it correctly and there is nothing to worry.Hope this help.
Best luck.
GR
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
A page will not be indexed if published without linking from anywhere?
Hi all, I have noticed one page from our competitors' website which has been hardly linked from one internal page. I just would like to know if the page not linked anywhere get indexed by Google or not? Will it be found by Google? What if a page not linked internally but go some backlinks from other websites? Thanks
Algorithm Updates | | vtmoz0 -
Do we need to worry about external redirects?
Hi all, We always avoid internal redirects. Just wonder what if many of the out going links are redirecting to new links. I presume there is nothing wrong to host such links. Any ideas? Thanks
Algorithm Updates | | vtmoz0 -
Do back-links to non indexed sub-domains / sub-directories considered by Google as website backlinks and pass Pagerank to website?
Hi, If some noindexed links on our website or sub-domain got some backlinks, will that backlinks pass Pagerank / linkjuice to website? Will they be considered as backlinks to website by Google? Here is a statement from Matt cutts for the question. My question is same as below with answer? Eric Enge: Can a NoIndex page accumulate PageRank? Matt Cutts: A NoIndex page can accumulate PageRank, because the links are still followed outwards from a NoIndex page. Thanks
Algorithm Updates | | vtmoz0 -
Trafic drop after a huge indexation
Hello everyone, My website used to have about 500k indexed pages in Google. After publishing fresh sitemaps and a little local "buzz", it now has about 6 millions indexed pages and the numbers are skyrocketing (GWT says 7 millions and it will probably keep going). My website has a total number of pages of 10 millions. I used to have about 5k organic visite each day, but since the big indexation has started, I now have half less. I read many things about that kind of trafic drop, and it seems to be a normal step when indexing a huge site. I just wanted to know if you guys had any similar experiences and if yes, if there are specific tasks to do in order to recover/develop the organic trafic or if it's just a matter of time. Thanks for your help and share of experiences 😉
Algorithm Updates | | Pureshore0 -
Why am i not ranking in the top 50 for the keyword 'cocktails' even though all my other cocktail related keywords are in the first 2 pages of Google???
I have checked the first 50 pages of google for my website www.socialandcocktail.co.uk using the keyword 'cocktails'. It is NOT to be found. However, if I search for other keyword combinations eg cocktail recipes, cocktail bars etc they are all in the first 2 pages! What is going on????????
Algorithm Updates | | cocktailboss0 -
Will we no longer need Location + Keyword? Do we even need it at all?
Prepare yourselves. This is a long question. With the rise of schema and Google Local+, do you think Google will now have enough data about where a business is located, so that when someone searches for, a keyword such as "Atlanta Hyundai dealers" a business in Atlanta that's website: has been properly marked up with schema (or microdata for business location) has claimed its Google Local+ has done enough downstream work in Local Search listings for its NAP (name, address, phone number) will no longer have to incorporate variations of "Atlanta Hyundai dealers" in the text on the website? Could they just write enough great content about how they're a Hyundai dealership without the abuse of the Atlanta portion? Or if they're in Boston and they're a dentist or lawyer, could the content be just about the services they provided without so much emphasis tied to location? I'm talking about removing the location of the business from the text in all places other than the schema markup or the contact page on the website. Maybe still keep a main location in the title tags or meta description if it would benefit the customer. I work in an industry where location + keywords has reached such a point of saturation, that it makes the text on the website read very poorly, and I'd like to learn more about alternate methods to keep the text more pure, read better and still achieve the same success when it comes to local search. Also, I haven't seen other sites penalized for all the location stuffing on their websites, which is bizarre because it reads so spammy you can't recognize where the geotargeted keywords end and where the regular text begins. I've been working gradually in this general direction (more emphasis on NAP, researching schema, and vastly improving the content on clients' websites so it's not so heavy with geo-targeted keywords). I also ask because though the niche I work in is still pretty hell-bent on using geo-targeted keywords, whenever I check Analytics, the majority of traffic is branded and geo-targeted keywords make up only a small fraction of traffic. Any thoughts? What are other people doing in this regard?
Algorithm Updates | | EEE30 -
Google's not indexing my blog posts anymore! Why?
Google just recently stopped indexing my blog posts immediately after being published, why could this be? I would usually post a blog post and it would be in google results within 45 seconds, now they don't show up until 6 hours later, if at all (a few never even showed up). Also, my home page doesn't even refresh when I make a change to the site. My site is CantStopHipHop [dot] comI have all in one SEO, xml sitemap generator, and webmaster tools and nothing seemed irregular in the settings.I appreciate any thoughts/help/suggestions.
Algorithm Updates | | bb2550
