Will disallowing in robots.txt noindex a page?
-
Google has indexed a page I wish to remove. I would like to meta noindex but the CMS isn't allowing me too right now. A suggestion o disallow in robots.txt would simply stop them crawling I expect or is it also an instruction to noindex? Thanks
-
Hi
I would also suggest to make sure to change internal links if possible and make them point to the page you want to see in the index
-
Good answer we both submitted this at the same time.
-
Check out this article, it's the Google Requirements for Removing content
There are multiple ways to do this:
Ideally, robots.txt is the way to go, you can also noindex the page. There is no need to do both. If your looking for the page to disappear as well, i'd let it 404.
Another option is to request the URL to be removal request in Google Webmaster Tools along with these things it's under optimization>Remove URLs.
Hope this helps
-
You'd have to add the disallow in robots.txt and then go into the Google Webmaster Console, use the Remove URLs option to get Google to deindex that page.
From Google:
Use **robots.txt **to specify how search engines should crawl your site, or request **removal **of URLs from Google's search results ( have you read our removal requirements? ). Only site owners and users with full permissions can request removals.
-
It the page has already been indexed and you block via robots.txt file what you will most likely see in SERPs is a listing with no meta description (so just one line). I do believe that ranking placements will drop. If you want to remove from SERPs completely you need to use the noindex tag.
I would block by robots.txt file now while you work out how to add the noindex tag.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Wrong Pages Ranking
Good Afternoon We had an issue a while ago with the incorrect pages ranking in Google for some of our key terms. For example the page ranking for the term in Hotels in Spain was an individual information page for one particular hotel in Spain rather than the top level page which is optimised for "Hotels in Spain" The individual property page was ranking around 36-40 so we tightened up all the internal linking structure to ensure the term "Hotels in Spain" was pointing to the correct page and de-optimised the individual property page for the term. After a few weeks, everything seemed to be working and we were ranking top of second page for correct page however, ranking report today has reversed our good fortune and the incorrect page is ranking in a low position Any further suggestions or advise would be very much appreciated. Ideally, I don't want to remove the page that is ranking as it's still relevant for a search for that particular hotel
Intermediate & Advanced SEO | | Ham19790 -
Will this affect
Hi there! I've got a question that I'm having trouble answering. My client has one site url essentially has two sites within it. The homepage (name-photography.com) content focuses on her fashion photography services with a specific design and look with almost no mention of weddings unless you click the weddings icon. When you click on weddings, it takes you to a "new" site on the same url (name-photography.com/weddings) that has entirely different look and feel. The client would like to improve her visibility for her wedding services and not the fashion photography side. Would it be more beneficial to house the wedding services on an entirely new URL so that homepage content can be wedding focused. Again, with the current homepage, it's all fashion photography focused and not easy to redo. Or could one implement a 301 redirect from the fashion homepage (name-photographer.com) to the wedding homepage (name-photographer.com/weddings)? Thanks for your advice! Jessica
Intermediate & Advanced SEO | | zigzen0 -
Should I disallow all URL query strings/parameters in Robots.txt?
Webmaster Tools correctly identifies the query strings/parameters used in my URLs, but still reports duplicate title tags and meta descriptions for the original URL and the versions with parameters. For example, Webmaster Tools would report duplicates for the following URLs, despite it correctly identifying the "cat_id" and "kw" parameters: /Mulligan-Practitioner-CD-ROM
Intermediate & Advanced SEO | | jmorehouse
/Mulligan-Practitioner-CD-ROM?cat_id=87
/Mulligan-Practitioner-CD-ROM?kw=CROM Additionally, theses pages have self-referential canonical tags, so I would think I'd be covered, but I recently read that another Mozzer saw a great improvement after disallowing all query/parameter URLs, despite Webmaster Tools not reporting any errors. As I see it, I have two options: Manually tell Google that these parameters have no effect on page content via the URL Parameters section in Webmaster Tools (in case Google is unable to automatically detect this, and I am being penalized as a result). Add "Disallow: *?" to hide all query/parameter URLs from Google. My concern here is that most backlinks include the parameters, and in some cases these parameter URLs outrank the original. Any thoughts?0 -
SSL and robots.txt question - confused by Google guidelines
I noticed "Don’t block your HTTPS site from crawling using robots.txt" here: http://googlewebmastercentral.blogspot.co.uk/2014/08/https-as-ranking-signal.html Does this mean you can't use robots.txt anywhere on the site - even parts of a site you want to noindex, for example?
Intermediate & Advanced SEO | | McTaggart0 -
Interlinking from unique content page to limited content page
I have a page (page 1) with a lot of unique content which may rank for "Example for sale". On this page I Interlink to a page (page 2) with very limited unique content, but a page I believe is better for the user with anchor "See all Example for sale". In other words, the 1st page is more like a guide with items for sale mixed, whereas the 2nd page is purely a "for sale" page with almost no unique content, but very engaging for users. Questions: Is it risky that I interlink with "Example for sale" to a page with limited unique content, as I risk not being able to rank for either of these 2 pages Would it make sense to "no index, follow" page 2 as there is limited unique content, and is actually a page that exist across the web on other websites in different formats (it is real estate MLS listings), but I can still keep the "Example for sale" link leading to page 2 without risking losing ranking of page 1 for "Example for sale"keyword phrase I am basically trying to work out best solution to rank for "Keyword for sale" and dilemma is page 2 is best for users, but is not a very unique page and page 2 is very unique and OK for users but mixed up writing, pictures and more with properties for sale.
Intermediate & Advanced SEO | | khi50 -
SEO structure question: Better to add similar (but distinct) content to multiple unique pages or make one unique page?
Not sure which approach would be more SEO ranking friendly? As we are a music store, we do instrument repairs on all instruments. Currently, I don't have much of any content about our repairs on our website... so I'm considering a couple different approaches of adding this content: Let's take Trumpet Repair for example: 1. I can auto write to the HTML body (say, at the end of the body) of our 20 Trumpets (each having their own page) we have for sale on our site, the verbiage of all repairs, services, rates, and other repair related detail. In my mind, the effect of this may be that: This added information does uniquely pertain to Trumpets only (excludes all other instrument repair info), which Google likes... but it would be duplicate Trumpet repair information over 20 pages.... which Google may not like? 2. Or I could auto write the repair details to the Trumpet's Category Page - either in the Body, Header, or Footer. This definitely reduces the redundancy of the repeating Trumpet repair info per Trumpet page, but it also reduces each Trumpet pages content depth... so I'm not sure which out weighs the other? 3. Write it to both category page & individual pages? Possibly valuable because the information is anchoring all around itself and supporting... or is that super duplication? 4. Of course, create a category dedicated to repairs then add a subcategory for each instrument and have the repair info there be completely unique to that page...- then in the body of each 20 Trumpets, tag an internal link to Trumpet Repair? Any suggestions greatly appreciated? Thanks, Kevin
Intermediate & Advanced SEO | | Kevin_McLeish0 -
How can you indexed pages or content on pages that are behind a pay wall or subscription login.
I have a client that has a boat of awesome content they provide to their client that's behind a pay wall ( ie: paid subscribers can only access ) Any suggestions mozzers? How do I get those pages index? Without completely giving away the contents in the front end.
Intermediate & Advanced SEO | | BizDetox0 -
Google swapped our website's long standing ranking home page for a less authoritative product page?
Our website has ranked for two variations of a keyword, one singular & the other plural in Google at #1 & #2 (for over a year). Keep in mind both links in serps were pointed to our home page. This year we targeted both variations of the keyword in PPC to a products landing page(still relevant to the keywords) within our website. After about 6 weeks, Google swapped out the long standing ranked home page links (p.a. 55) rank #1,2 with the ppc directed product page links (p.a. 01) and dropped us to #2 & #8 respectively in search results for the singular and plural version of the keyword. Would you consider this swapping of pages temporary, if the volume of traffic slowed on our product page?
Intermediate & Advanced SEO | | JingShack0