Have you ever seen or experienced a page indexed which is actually from a website which is blocked by robots.txt?
-
Hi all,
We use robots file and meta robots tags for blocking website or website pages to block bots from crawling. Mostly robots.txt will be used for website and expect all the pages to not getting indexed. But there is a condition here that any page from website can be indexed by Google even the site is blocked from robots.txt; because crawler may find the page link somewhere on internet as stated here at last paragraph. I wonder if this really the case where some webpages have got indexed.
And even we use meta tags at page level; do we need to block from robots.txt file? Can we use both techniques at a time?
Thanks
-
Hi vtmoz,
The most mandatory way to prevent any page to be indexed is by using a meta robots tag with a _noindex _parameter.
Then using robots.txt will help to optimize your server resources and is a way that prevent google to crawl any new page that do not have the meta robots tag.And yeah, its very common to have indexed pages even the robots.txt file blocks the entire website.
If what you are looking for is to remove from index the pages, follow this steps:
- Allow the whole website to be crawable (or at least that specific pages/section) in the robots.txt
- add the robots meta tag with "noindex,follow" parametres
- wait several weeks, 6 to 8 weeks is a fairly good time. Or just do a followup on those pages
- when you got the results (all your desired pages to be de-indexed) re-block with robots.txt those pages
- DO NOT erase the meta robots tag.
Hope it helps.
Best luck.
GR.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking impact: Traffic in website pages vs sub directory vs sub domain
Hi all, I need clarification on this. Not every time website main pages rank, some times even pages from sub directories or sub domains like blogs or guides; especially for branded keywords. I just wonder what happens when so much traffic is generating in sub directories and sub domains just because of limited landing pages in main website. Will this traffic be counted as traffic in main website as per Google? Traffic increase in main website really an ranking factor? Will the "brand + topic" related keywords' traffic is more for a website; will it ranking improves even for "topic keywords"? Thanks
Algorithm Updates | | vtmoz0 -
Bing not indexing pages
We have taken all recommended steps to index our site sitegeek.com pages to Bing Bot but failed to index them. Bing bot crawled more than 5,000 pages every day but strange why pages are not getting index ? if we query site:sitegeek.com in Bing Bing Search Engine shows only 1,200 pages got indexed. but we query site:sitegeek.com in Google Google Search Engine show more 546,000 pages got indexed. For example : https://www.sitegeek.com/000webhost Above page crawled by Google but Bing. Can anyone suggest what we are missing on this page? what need to change to index such pages? Thanks! Rajiv
Algorithm Updates | | gamesecure0 -
Trafic drop after a huge indexation
Hello everyone, My website used to have about 500k indexed pages in Google. After publishing fresh sitemaps and a little local "buzz", it now has about 6 millions indexed pages and the numbers are skyrocketing (GWT says 7 millions and it will probably keep going). My website has a total number of pages of 10 millions. I used to have about 5k organic visite each day, but since the big indexation has started, I now have half less. I read many things about that kind of trafic drop, and it seems to be a normal step when indexing a huge site. I just wanted to know if you guys had any similar experiences and if yes, if there are specific tasks to do in order to recover/develop the organic trafic or if it's just a matter of time. Thanks for your help and share of experiences 😉
Algorithm Updates | | Pureshore0 -
How to get indexed in yahoo and bing?
Hello all I have uploaded sitemap to bing webmaster on January 21st, 2014. However, the site has not been indexed yet. I see few pages crawled and some crawl page error but it does not show what pages have an error. Can anyone help me please on how to get this done right so i can can our website indexed in Bing and Yahoo quickly. By the way our website address is : http://www.eduniche.com
Algorithm Updates | | Eva20140 -
Why Is The Wrong Page Ranking?
In the past two weeks, I've seen some movement in ranking for "Tampa Personal Injury Attorney." The problem is that this page: http://www.kempruge.com/personal-injury/ is the one that's ranking and not this page: http://www.kempruge.com/location/tampa/tampa-personal-injury-legal-attorneys/ which is the one I've been working on. Also, the former page has made it to page 4 (not great) but better than 7, which is what the latter page was. In addition, the latter page now doesn't rank at all (or at least not in the first 16 pages). Finally, according to Moz, the latter page (the one that no longer ranks) is my second best page after my homepage. I just don't understand this at all. Is this a fluke? Should I just try to work on the page that's ranking higher over the page I've put the time into? Thanks, Ruben
Algorithm Updates | | KempRugeLawGroup0 -
Pages fluctuating +/- 70 positions in Google SERPs?
I've got some pages that appear somewhere around #25 in Google. Every now and then, it just goes away from the top 100 results for a few days (even up to a week) and then it comes back. I've got other pages that rank around #8 which falls down to about #75 for a while and then it comes back. But while a page may be gone from the top 100 results in the US, it still ranks at about the same place everywhere else in the world (+/- 10 positions). I've seen this happen in the past but never it happened so often. What gives?!?
Algorithm Updates | | sbrault740 -
In the body of index page i want to be able to add text that can be picked up by crawlers but I do not want these text to be visible? How can I code this?
in the body of index page i want to be able to add text that can be picked up by crawlers but I do not want these text to be visible? How can I code this?
Algorithm Updates | | FinindDesign0 -
Google site links on sub pages
Hi all Had a look for info on this one but couldn't find much. I know these days that if you have a decent domain good will often automatically put site links on for your home if someone searches for your company name, however has anyone seen these links appear for sub pages? For example, lets say I had a .com domain with /en /fr /de sub folders, each seoed for their location. If I were to then have domain.com/en/ as no1 in Google for my company in the UK would I be able to get site links under this or does it only work on the 'proper' homepage domain.com/ A client of mine wants to reorganise their website so they have different location sections ranking in different markets but they also want to keep having sitewide links as they like the look of it Thanks Carl
Algorithm Updates | | Grumpy_Carl0