Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
-
Hi all,
We use robots file and meta robots tags for blocking website or website pages to block bots from crawling. Mostly robots.txt will be used for website and expect all the pages to not getting indexed. But there is a condition here that any page from website can be indexed by Google even the site is blocked from robots.txt; because crawler may find the page link somewhere on internet as stated here at last paragraph. I wonder if this really the case where some webpages have got indexed.
And even we use meta tags at page level; do we need to block from robots.txt file? Can we use both techniques at a time?
Thanks
-
Hi vtmoz,
The most mandatory way to prevent any page to be indexed is by using a meta robots tag with a _noindex _parameter.
Then using robots.txt will help to optimize your server resources and is a way that prevent google to crawl any new page that do not have the meta robots tag.And yeah, its very common to have indexed pages even the robots.txt file blocks the entire website.
If what you are looking for is to remove from index the pages, follow this steps:
- Allow the whole website to be crawable (or at least that specific pages/section) in the robots.txt
- add the robots meta tag with "noindex,follow" parametres
- wait several weeks, 6 to 8 weeks is a fairly good time. Or just do a followup on those pages
- when you got the results (all your desired pages to be de-indexed) re-block with robots.txt those pages
- DO NOT erase the meta robots tag.
Hope it helps.
Best luck.
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
On page vs Off page vs Technical SEO: Priority, easy to handle, easy to measure.
Hi community, I am just trying to figure out which can be priority in on page, off page and technical SEO. Which one you prefer to go first? Which one is easy to handle? Which one is easy to measure? Your opinions and suggestions please. Expecting more realistic answers rather than usual check list. Thanks
Algorithm Updates | | vtmoz0 -
Content strategy for landing pages: Topics vs Features
Hi all, We are going to create new landing pages and optimise existing pages. We have a confusion on how to employ content on these pages....whether these will be filled with content to rank for "topics" and "keywords" or direclty jump into the features are are providing. If we go with first, users may feel boring about teaching them about that topic, if we go with latter...it's hard to rank being no related content to rank for that topic. I have seen some of the websites are employing multiple landing pages where they fill with topic related content and then link to features pages. I need suggestions here. Thank you
Algorithm Updates | | vtmoz1 -
How long for google to de-index old pages on my site?
I launched my redesigned website 4 days ago. I submitted a new site map, as well as submitted it to index in search console (google webmasters). I see that when I google my site, My new open graph settings are coming up correct. Still, a lot of my old site pages are definitely still indexed within google. How long will it take for google to drop off or "de-index" my old pages? Due to the way I restructured my website, a lot of the items are no longer available on my site. This is on purpose. I'm a graphic designer, and with the new change, I removed many old portfolio items, as well as any references to web design since I will no longer offering that service. My site is the following:
Algorithm Updates | | rubennunez
http://studio35design.com0 -
Indexing of Search Pages
I have a question on indexing search pages of an ecommerce or any website. I read Google doesn't recommend this and sites shouldn't allow indexing of their search pages. I recently attended an SEO event (BrightonSEO) and one of the talks was on search pages and how big players like eBay, Amazon do index their search pages. In fact, it is a core part of the pages that are indexed. eBay has to do it, as their product pages are on a time frame and Amazon only allows certain category search pages to be indexed. Reviewing my competitors, they are indexing search pages and this is why they have thousands and millions of web pages indexed. What are your thoughts? I thought search pages were too dynamic (URL strings) and they wouldn't have a unique page title, meta description or rich content to act as a well optimised page. Am I missing a trick here? Cyto
Algorithm Updates | | Bio-RadAbs0 -
What do media queries have to do with the page layout update?
Who thinks the lack of media queries will have an impact on whether the page layout update affects a site?
Algorithm Updates | | kimmiedawn0 -
GWT: Anyone else seen this weird data trend?
I have a client who came to us after recieving a manual unnatural link warning, and probable Penguin penalty. After a lot of hard work, we had the manual penalty revoked, and have done everything we can to get the Penguin penalty lifted. This all happened after the last Penguin data refresh (early Jan), so we're still waiting to see a recovery; but in the meantime, we're seeing some very weird impression data in GWT....... The website targets UK users, and has very little US traffic, except every thursday, GWT reports a massive spike in search impressions from the US. Could this be Google testing the new Penguin data? See our US impressions for since Feb here - http://imgur.com/2Bl9a3f 2Bl9a3f.png
Algorithm Updates | | BabelPR0 -
Why bing is not indexing our website?
We are up almost a six month already, google indexed 46,900 pages. We have decent traffic and a lot of real external links to us. No single page has been indexed by bing or yahoo. I have submitted sitemap to bing's webmaster tool two weeks ago and still it is in Pending stage. here is our address: www.showme.com and here is site map: http://www.showme.com/sitemapxml.php What can be the reason of that? Thanks for your help. Karen Bdoyan
Algorithm Updates | | showme0 -
If a page one result for a keyword is mostly directories, do I have a chance to rank for this keyword?
I feel like although directories carry a lot of weight and links, I'd think that my client would be able to gain a top position, since none of the others are competitor pages, nor are the directories engaging.
Algorithm Updates | | randallseo0