Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
-
Hi all,
We use robots file and meta robots tags for blocking website or website pages to block bots from crawling. Mostly robots.txt will be used for website and expect all the pages to not getting indexed. But there is a condition here that any page from website can be indexed by Google even the site is blocked from robots.txt; because crawler may find the page link somewhere on internet as stated here at last paragraph. I wonder if this really the case where some webpages have got indexed.
And even we use meta tags at page level; do we need to block from robots.txt file? Can we use both techniques at a time?
Thanks
-
Hi vtmoz,
The most mandatory way to prevent any page to be indexed is by using a meta robots tag with a _noindex _parameter.
Then using robots.txt will help to optimize your server resources and is a way that prevent google to crawl any new page that do not have the meta robots tag.And yeah, its very common to have indexed pages even the robots.txt file blocks the entire website.
If what you are looking for is to remove from index the pages, follow this steps:
- Allow the whole website to be crawable (or at least that specific pages/section) in the robots.txt
- add the robots meta tag with "noindex,follow" parametres
- wait several weeks, 6 to 8 weeks is a fairly good time. Or just do a followup on those pages
- when you got the results (all your desired pages to be de-indexed) re-block with robots.txt those pages
- DO NOT erase the meta robots tag.
Hope it helps.
Best luck.
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Have you ever changed the logo anchor text from "logo" to "keyword"? How Google considers?
Hi all, We know that generally logo with the website homepage link is the first link crawled by Google and other search engines. Can we change the anchor text from "logo" to "keyword"? Have any one tried or seen others doing? How Google considers it? Thanks
Algorithm Updates | | vtmoz1 -
Meta robots at every page rather than using robots.txt for blocking crawlers? How they'll get indexed if we block crawlers?
Hi all, The suggestion to use meta robots tag rather than robots.txt file is to make sure the pages do not get indexed if their hyperlinks are available anywhere on the internet. I don't understand how the pages will be indexed if the entire site is blocked? Even though there are page links are available, will Google really index those pages? One of our site got blocked from robots file but internal links are available on internet for years which are not been indexed. So technically robots.txt file is quite enough right? Please clarify and guide me if I'm wrong. Thanks
Algorithm Updates | | vtmoz0 -
When sub domains take away the traffic from search; will this helps or hurts main website rankings?
Hi all, We have some of the landing pages on our sub domains which are getting ranked for our brand related queries and taking away the traffic as we don't have pages to rank for those search queries. I would like to know if this scenario hurts or helps our main website ranking as the traffic to the main website is getting diverted to sub domain. Thanks
Algorithm Updates | | vtmoz0 -
Do keyword target landing pages increase rankings?
Let's say we create landing pages for targeted keywords in our niche. So like we have landing pages optimised for 80% of the top keywords with decent search volume. If these pages started ranking at first page or around; will this scenario improves the ranking of website? Right now, only few of our top pages are ranking good. Planning to create more of such.
Algorithm Updates | | vtmoz0 -
Is Having Content 'Above The Fold' Still Relevant for Website Design and SEO
Hey there, So I have a client who recently 're-skinned' their website and now there is little to no content above the fold. Likewise, I've noticed that since the transition to this new front-end design there has been a drop in rankings for a number of keywords related to one of the topics we are targeting. Is there any correlation here? Is having content 'above the fold' still a relevant factor in determining a websites' searchability? I appreciate you reading and look forward to hearing from all of you. Have a great day!
Algorithm Updates | | maxcarnage0 -
Is my page footer the reason keyword rankings have dropped?
Hi all, One of my sites http://henstuff.com/ has seen some ranking drops for major keywords over the past few weeks and I was wondering if it was something to do with Penguin not taking a positive view of link-filled footers. It is something we are looking at phasing out but wanted to get the opinions of the SEOMOZ community. Thanks! Rob
Algorithm Updates | | RobertHill0 -
Using Brand Name in Page titles
Is it a good practice to append our brand name at the end of every page title? We have a very strong brand name but it is also long. Right now what we are doing is saying: Product Name | Long brand name here Product Category | Long brand name here Is this the right way to do it or should we just be going with ONLY the product and category names in our page titles? Right now we often exceed the 70 character recommendation limit.
Algorithm Updates | | mlentner1 -
If a page one result for a keyword is mostly directories, do I have a chance to rank for this keyword?
I feel like although directories carry a lot of weight and links, I'd think that my client would be able to gain a top position, since none of the others are competitor pages, nor are the directories engaging.
Algorithm Updates | | randallseo0