Is 404'ing a page enough to remove it from Google's index?
-
We set some pages to 404 status about 7 months ago, but they are still showing in Google's index (as 404's). Is there anything else I need to do to remove these?
-
Nice information John. I hadn't thought of adding a temporary page with a noindex tag but that sounds like a way to go for faster results.
I know Google has automatically removed 404 pages in the past. I noticed the issue Michelle is sharing and the information you shared offers great details on the process.
-
Setting pages to 404 should be enough to remove them after Google indexes your page enough times. Google has to be careful about this, because when many sites crash or have site maintenance, they return 404 instead of 503, so Google wouldn't want to remove pages from their index until they're sure the page is gone.
Google talks about removing pages from there index here. The Google Webmaster Tools URL removal tool is only intended for pages that urgently need to be removed, so I wouldn't recommend that. Google recommends:
- If the page no longer exists, make sure that the server returns a 404 (Not Found) or 410 (Gone) HTTP status code. This will tell Google that the page is gone and that it should no longer appear in search results.
- If the page still exists but you don't want it to appear in search results, use robots.txt to prevent Google from crawling it. Note that in general, even if a URL is disallowed by robots.txt we may still index the page if we find its URL on another site. However, Google won't index the page if it's blocked in robots.txt and there's an active removal request for the page.
- Alternatively, you can use a noindex meta tag. When we see this tag on a page, Google will completely drop the page from our search results, even if other pages link to it. This is a good solution if you don't have direct access to the site server. (You will need to be able to edit the HTML source of the page).
Is there a reason you are 404'ing these pages rather than redirecting them? If these pages have new pages with similar content, you should do a 301 redirect to keep the link juice flowing and to take advantage of these pages being linked to. If you do continue returning 404 for these pages (or even if you don't...), make sure your 404 page is a useful one, that helps users find the page they're looking for (Google help article).
Also, Ryan, I'd be interested in hearing the results of using the 410 status code. I would imagine that status code would do the trick! I'm surprised I haven't read about this more, or why it's not mentioned in the help file linked to above.
-
I have experienced this same issue with Google.
I just began a test by making a change on my site to one of the URLs. I am bookmarking this Q&A and will try to remember to update it if I see a change. It can take Google some time to check any individual link so it could take weeks.
In case you are curious, I have added a 410 status code for one of the pages involved. 410 means the resource is gone, while 404 is simply not found. Perhaps the 410 header code will send the right message to Google.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Drop in Indexed pages
Hope everyone is having an Awesome December! I first noticed a drop in my index in the beginnings of November. My site drop in indexed pages from 1400 to 600 in the past 3-4 weeks. I don't know the cause of it, and would like the community to help me figure out why my indexing has dropped. Thank you for taking time out of your schedule to read this.
Intermediate & Advanced SEO | | BSC0 -
Removing pages from index
My client is running 4 websites on ModX CMS and using the same database for all the sites. Roger has discovered that one of the sites has 2050 302 redirects pointing to the clients other sites. The Sitemap for the site in question includes 860 pages. Google Webmaster Tools has indexed 540 pages. Roger has discovered 5200 pages and a Site: query of Google reveals 7200 pages. Diving into the SERP results many of the pages indexed are pointing to the other 3 sites. I believe there is a configuration problem with the site because the other sites when crawled do not have a huge volume of redirects. My concern is how can we remove from Google's index the 2050 pages that are redirecting to the other sites via a 302 redirect?
Intermediate & Advanced SEO | | tinbum0 -
Putting "noindex" on a page that's in an iframe... what will that mean for the parent page?
If I've got a page that is being called in an iframe, on my homepage, and I don't want that called page to be indexed.... so I put a noindex tag on the called page (but not on the homepage) what might that mean for the homepage? Nothing? Will Google, Bing, Yahoo, or anyone else, potentially see that as a noindex tag on my homepage?
Intermediate & Advanced SEO | | Philip-DiPatrizio0 -
Google's form for "Small sites that should rank better" | Any experiences or results?
Back in August of 2013 Google created a form that allowed people to submit small websites that "should be ranking better in Google". There is more info about it in this article http://www.seroundtable.com/google-small-site-survey-17295.html Has anybody used it? Any experiences or results you can share? *private message if you do not want to share publicly...
Intermediate & Advanced SEO | | GregB1230 -
Indexing non-indexed content and Google crawlers
On a news website we have a system where articles are given a publish date which is often in the future. The articles were showing up in Google before the publish date despite us not being able to find them linked from anywhere on the website. I've added a 'noindex' meta tag to articles that shouldn't be live until a future date. When the date comes for them to appear on the website, the noindex disappears. Is anyone aware of any issues doing this - say Google crawls a page that is noindex, then 2 hours later it finds out it should now be indexed? Should it still appear in Google search, News etc. as normal, as a new page? Thanks. 🙂
Intermediate & Advanced SEO | | Alex-Harford0 -
There's a website I'm working with that has a .php extension. All the pages do. What's the best practice to remove the .php extension across all pages?
Client wishes to drop the .php extension on all their pages (they've got around 2k pages). I assured them that wasn't necessary. However, in the event that I do end up doing this what's the best practices way (and easiest way) to do this? This is also a WordPress site. Thanks.
Intermediate & Advanced SEO | | digisavvy0 -
Best solution to get mass URl's out the SE's index
Hi, I've got an issue where our web developers have made a mistake on our website by messing up some URL's . Because our site works dynamically IE the URL's generated on a page are relevant to the current URL it ment the problem URL linked out to more problem URL's - effectively replicating an entire website directory under problem URL's - this has caused tens of thousands of URL's in SE's indexes which shouldn't be there. So say for example the problem URL's are like www.mysite.com/incorrect-directory/folder1/page1/ It seems I can correct this by doing the following: 1/. Use Robots.txt to disallow access to /incorrect-directory/* 2/. 301 the urls like this:
Intermediate & Advanced SEO | | James77
www.mysite.com/incorrect-directory/folder1/page1/
301 to:
www.mysite.com/correct-directory/folder1/page1/ 3/. 301 URL's to the root correct directory like this:
www.mysite.com/incorrect-directory/folder1/page1/
www.mysite.com/incorrect-directory/folder1/page2/
www.mysite.com/incorrect-directory/folder2/ 301 to:
www.mysite.com/correct-directory/ Which method do you think is the best solution? - I doubt there is any link juice benifit from 301'ing URL's as there shouldn't be any external links pointing to the wrong URL's.0 -
Why does this page not show in google at all?
www.lavenderblue-flowers.co.uk Sorry for formatting, below is the source. There are alot of blocks from robots.txt but is there anything easily rectified to get this site SOME visibility? Duplicate content maybe PANDA had it? No backlink profile too which isnt helping but even still, surprising to see a domain auth of 1. Thanks in advance for any responses. DOCTYPE HTML PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html><head><meta content="text/html; charset=UTF-8" http-equiv="Content-Type"><meta http-equiv="expires" content="Fri, 17 Jun 2011 12:06:27 GMT"><title>Bridport Interflora Florist, Lavender Blue, Dorset, DT16 3XDtitle><meta name="description" content="Lavender Blue in Bridport, Dorset, DT16 3XD delivers to Interflora florist based in Bridport is a well established family run business with a dedicated team of florists. We specialise in beautiful wedding flowers and take great pride in our funeral tributes, floral arrangements designed for any occasion for local, national and worldwide delivery."><meta name="keywords" content="Bridport,Interflora Florist,Lavender Blue,Dorset,DT16 3XD"><meta name="abstract" content="Interflora florist based in Bridport is a well established family run business with a dedicated team of florists. We specialise in beautiful wedding flowers and take great pride in our funeral tributes, floral arrangements designed for any occasion for local, national and worldwide delivery."><meta name="robots" content="index,nofollow"><link rel="stylesheet" type="text/css" href="/kernel/styles/print.css?new=new" media="print"><link rel="stylesheet" href="/kernel/styles/d4.css?designtype=d4;theme=blue;" type="text/css"><style type="text/css">style><script language="JavaScript1.2" src="/kernel/utils.js?new" type="text/javascript">script><script language="JavaScript1.2" type="text/javascript" src="/kernel/interflora.js?head=1;si=1000343;">script><script language="JavaScript1.2" type="text/javascript">script><script language="javascript"> var b_site_url = getcookie('b_site_url');if (b_site_url != "" && !getcookie('referral_id') && location.protocol == 'http:' && b_site_url != location.host && location.pathname.indexOf('catalog2') == -1) location.href = location.protocol + "//" + b_site_url + location.pathname + location.search;script>head><body><img border="0" src="/kernel/images/speck.gif" width="1" height="1" alt class="nospace"><div id="page-body"><table class="page-topbanner" border="0" cellpadding="0" cellspacing="0"><tr><td background="/kernel/images/d4/border-blue_03.gif" align="left" valign="top"><img src="/kernel/images/d4/border-blue_01.gif" alt>td><td colspan="2" style="background-image: url(/kernel/images/d4/border-blue_03.gif); background-position: top; background-repeat: repeat-x;"><img src="/kernel/images/speck.gif" width="300" height="50" alt>td><td align="right" valign="top" background="/kernel/images/d4/border-blue_03.gif"><img src="/kernel/images/d4/border-blue_04.gif" alt>td>tr><tr><td style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-repeat: repeat-y;" align="left" valign="top"><img src="/kernel/images/d4/border-blue_01b.gif" alt>td><td valign="top" class="sd-image_only" id="sd-logo_store" colspan="1" rowspan="1"><img src="/kernel/imageload?ttl2=15;table=content_images;key1=fd_img_2606422_1" alt="" title="">td><td class="logo-if" align="right"><img src="/kernel/images/logo-if.png" alt="interflora.co.uk the flower experts™">td><td style="background-image: url(/kernel/images/d4/border-blue_07.gif); background-position: right; background-repeat: repeat-y;"> td>tr><tr><td style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-position: left; background-repeat: repeat-y;" colspan="3" align="center"><table id="website" cellspacing="0" border="0" align="center"><tr><td colspan="3" id="fol_address">1 Lilliput Lane, Bridport, Dorset, DT16 3XDtd>tr><tr><td id="email" colspan="3"><b>Email:b> lavenderblueflowers@hotmail.co.uktd>tr><tr><td style="padding-right:10px;"><b>Phone:b> 01308 459145td><td style="padding-right:10px;"><b>Fax:b> 01308 458417td>tr>table>td><td style="background-image: url(/kernel/images/d4/border-blue_07.gif); background-position: right; background-repeat: repeat-y;"> td>tr><tr><td style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-position: left; background-repeat: repeat-y;" colspan="3" align="center"><div class="page-topmenu"><table class="page-topmenu" cellspacing="0"><tr><td id="account"><a href="/myaccount/"><img src="/kernel/images/d4/icon-account.gif" style="margin: 3px 3px 4px 3px; vertical-align: middle;" width="15" height="13" alt="My Account">My Accounta>td><td id="menu"><a href="/">Homea><img class="bullet" src="/kernel/images/speck.gif" width="2" height="2" alt style="margin: 10px 4px 10px 4px;"><a href="/page.xml?page_name=about">About Usa><img class="bullet" src="/kernel/images/speck.gif" width="2" height="2" alt style="margin: 10px 4px 10px 4px;"><a href="/page.xml?page_name=delivery">Delivery Infoa><img class="bullet" src="/kernel/images/speck.gif" width="2" height="2" alt style="margin: 10px 4px 10px 4px;"><a href="/page.xml?page_name=contactus">Contact Usa>td><td id="cart"><a href="/shopcart/"><img src="/kernel/images/d4/icon-shopcart.gif" style="margin: 3px; vertical-align: middle;" width="14" hieght="14" alt="Shopping Basket">Shopping Basketa>td>tr>table>div>td><td style="background-image: url(/kernel/images/d4/border-blue_07.gif); background-position: right; background-repeat: repeat-y;"> td>tr>table><p id="browser-warning" style="display: block; padding: 2px; border: 2px solid #FC9F85; margin: 0px; background-color: #FDFFC4;"><b>For your information:b> This message has appeared because we've noticed your browser doesn't fully support all functions of this site. For further information please <a href="/page.xml?page_name=faq">click herea>.p><script language="JavaScript1.2" type="text/javascript">var theBrowser = navigator.userAgent.toLowerCase();if(is_nav7up || (parseInt(is_moz_ver) >= 1) || is_ie5_5up || theBrowser.indexOf("safari") != -1) {hideElement('browser-warning',0);}script><table class="body" border="0" cellspacing="0" cellpadding="0"><tr><td align="left" valign="bottom" style="background-image: url(/kernel/images/d4/border-blue_05.gif); background-position: left; background-repeat: repeat-y;"><img src="/kernel/images/d4/border-blue_05.gif" alt>td><td class="menu" valign="top"><img src="/kernel/images/speck.gif" width="150" height="1" border="0" alt><br><form method="get" action="/search/index.xml" id="leftnav_search"><table border="0" cellspacing="0" class="global-search"><tr><th colspan="2">SEARCHth>tr><tr><td width="50%"><input class="text" type="text" name="keywords1" id="search" value maxlength="50" size="15">td><td align="left"><input type="submit" class="button" name="search" id="search" value="GO">td>tr><tr><td colspan="2" align="left"><a href="/search/advanced_search.xml">Advanced Searcha>td>tr>table>form><div class="menusection"><a class="menuParent_off" id="parentcat_2003443" href="/catalog/category.xml?category_id=2003443"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Anniversaryspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003443">div><a class="menuParent_off" id="parentcat_2003453" href="/catalog/category.xml?category_id=2003453"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Congratulationsspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003453">div><a class="menuParent_off" id="parentcat_4" href="/category/flower-arrangements/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">All Flower Bouquetsspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_4">div><a class="menuParent_off" id="parentcat_2003493" href="/catalog/category.xml?category_id=2003493"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Sympathy & Funeralspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003493">div><a class="menuParent_off" id="parentcat_2003463" href="/catalog/category.xml?category_id=2003463"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Thank Youspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003463">div><a class="menuParent_off" id="parentcat_2001478" href="/category/same-day-flowers/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Same Day Flower Deliveryspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2001478">div><a class="menuParent_off" id="parentcat_2124203" href="/category/summer_flowers/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Summer Flowersspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2124203">div><a class="menuParent_off" id="parentcat_2003403" href="/category/luxury-flowers/"><div class="spacer">div><span class="menu-bullet"><img src="/kernel/images/arrow.gif" class="menu-bullet">Luxury Flowersspan><div class="spacer">div>a><div class="menuChildren" id="menuChildrencat_2003403">div><a class="menuParent_off" id="parentcat_1000343" href="/catalo
Intermediate & Advanced SEO | | ewanstevenson0