How to Remove /feed URLs from Google's Index
-
Hey everyone, I have an issue with RSS /feed URLs being indexed by Google for some of our Wordpress sites. Have a look at this Google query, and click to show omitted search results. You'll see we have 500+ /feed URLs indexed by Google, for our many category pages/etc. Here is one of the example URLs: http://www.howdesign.com/design-creativity/fonts-typography/letterforms/attachment/gilhelveticatrade/feed/. Based on this content/code of the XML page, it looks like Wordpress is generating these:
<generator>http://wordpress.org/?v=3.5.2</generator>
Any idea how to get them out of Google's index without 301 redirecting them? We need the Wordpress-generated RSS feeds to work for various uses.
My first two thoughts are trying to work with our Development team to see if we can get a "noindex" meta robots tag on the pages, by they are dynamically-generated pages...so I'm not sure if that will be possible. Or, perhaps we can add a "feed" paramater to GWT "URL Parameters" section...but I don't want to limit Google from crawling these again...I figure I need Google to crawl them and see some code that says to get the pages out of their index...and THEN not crawl the pages anymore.
I don't think the "Remove URL" feature in GWT will work, since that tool only removes URLs from the search results, not the actual Google index.
FWIW, this site is using the Yoast plugin. We set every page type to "noindex" except for the homepage, Posts, Pages and Categories. We have other sites on Yoast that do not have any /feed URLs indexed by Google at all.
Side note, the /robots.txt file was previously blocking crawling of the /feed URLs on this site, which is why you'll see that note in the Google SERPs when you click on the query link given in the first paragraph.
-
I tried many different htaccess file codings (such as recommended here), but they didn't work. Had to succumb to using the outdated Meta Robots plugin by Yoast, which can add the "noindex" code to the http header of /feed/ URLs. But, at least it's a solution: http://wordpress.org/plugins/robots-meta/. Hopefully this helps someone else.
-
I believe I found the solution: implement an x-robots-tag into the HTTP header of the various feed URLs. But, I need some help with creating the code to place in my .htaccess file. Any takers?
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
After you remove a 301 redirect that Google has processed, will the new URL retain any of the link equity from the old URL?
Lets say you 301 redirect URL A to URL B, and URL A has some backlinks from other sites. Say you left the 301 redirect in place for a year, and Google had already replaced the old URL with the new URL in the SERPs, would the new URL (B) retain some of the link equity from URL A after the 301 redirect was removed, or does the redirect have to remain in place forever?
Technical SEO | | johnwalkersmith0 -
Keep getting "/feed" broken links in Google Search Console
Hey guys, I'm having an issue for the past few months. I keep getting "/feed" broken links in Google Search Console (screenshot attached). The site is a WordPress site using the YoastSEO plugin for on-page SEO and sitemap. Has anyone else experienced this issue? Did you fix it? How should I redirect these links? s7elXMy
Technical SEO | | Extima-Christian0 -
URL Indexing with Keyword
Hi, My webpage url is indexed in Google but don't show when searching the Main Keyword. How can i index it with keyword. It should show on any SERP when the keyword is searched. Any suggestions.
Technical SEO | | green.h1 -
Should you use google url remover if older indexed pages are still being kept?
Hello, A client recently did a redesign a few months ago, resulting in 700 pages being reduced to 60, mostly due to panda penalty and just low interest in products on those pages. Now google is still indexing a good number of them ( around 650 ) when we only have 70 on our sitemap. Thing is google indexes our site on average now for 115 urls when we only have 60 urls that need indexing and only 70 on our sitemap. I would of thought these urls would be crawled and not found, but is taking a very long period of time. Our rankings haven't recovered as much as we'd hope, and we believe that the indexed older pages are causes this. Would you agree and also would you think removing those old urls via the remover tool would be best option? It would mean using the url remover tool for 650 pages. Thank you in advance
Technical SEO | | Deacyde0 -
How to remove the specific link from Google Listed Index?
I am working on SEO for an e-commerce client. When I search for brand name in the Google it displays the top link with tabular index of categories. Whereas I want to remove the category called Coffee from the tabular index because it redirecting to the Home page which is not relevant. For your ref. attached is the screenshot. pXjdaCH.png
Technical SEO | | mountain.penguine0 -
Should you change Temporary redirects 302's to a 301 even if page is not important/intended for ranking ?
Hi Whilst i appreciate its best practice to 301 redirect permanently moved pages, what if the page is say a login page or other page you not really interested in ranking or transferring juice to ? is it still important/best practice to do so simply because the page has permanently moved hence should still be a 301 even though you don't really want it to rank ? cheers dan
Technical SEO | | Dan-Lawrence1 -
De-indexed from Google
Hi Search Experts! We are just launching a new site for a client with a completely new URL. The client can not provide any access details for their existing site. Any ideas how can we get the existing site de-indexed from Google? Thanks guys!
Technical SEO | | rikmon0 -
Removing some of the indexed pages from my website
I am planning to remove some of the webpages from my website and these webpages are already indexed with search engine. Is there any way by which I need to inform search engine that these pages are no more available.
Technical SEO | | ArtiKalra0