Quickest way to deindex a large number of pages
-
Our site was recently hacked by spammers posting fake content and bringing down our servers, etc. After a few months, we finally figured out what was going on and fixed the issue. However, it turns out that Google has indexed 26K+ spammy pages and we've lost page rank and search engine rankings as a result.
What is the best and fastest way to get these pages out of Google's index?
-
Given that I'm sure you've removed these pages from your site, there will be no page to which to add a meta-noindex tag.
Disallowing these pages in robots.txt in no way signals to the search engines that they should be removed from the index, just that they should no longer be crawled. Given that they're already indexed, blocking in robots.txt would potentially save some "crawl budget" but wouldn't do anything to remove them from the index.
So submitting them to the URL Removal Tool would be by far the most effective, along with an explanation.
You'll also want to keep a very close watch on your penalty warnings within Webmaster Tools. If you get flagged, you'll want a complete history of the issue and the steps you've taken to address it in order to prepare a reinclusion request.
Lastly, don't forget to submit these same URLs to the Bing Webmaster Tools Block URLs tool. You may not get a massive amount of traffic from Bing, but there's no sense throwing it away, since you've already prepared the URL removal list anyway.
Hope that helps?
Paul
-
Yup. Just wanted to add as well that if these pages are in a particular directory, then you can deindex the entire directory in one command using the URL removal tool.
-
Disallow in robots.txt
Add a noindex meta tag to these pages
Request Google to remove the URLs from their index via WMT URL removal request
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Non-indexed or indexed top hierarchy pages get high PageRank at Google?
Hi, We are creating some pages just to capture leads from blog-posts. We created few pages at top hierarchy like website.com/new-page/. I'm just wondering if these pages will take away more PageRank. Do we need to create these pages at low hierarchy like website.com/folder/new-page to avoid passing more PageRank? Is this is how PR distributed even now and it's same for indexed or non-indexed pages? Thanks
Algorithm Updates | | vtmoz0 -
Blog-posts pages are dominating in search console "Internal Links". Only home-page at top!
Hi all, Ours is WordPress website and we have a blog...website.com/blog/. All the important pages in the website are well linked from top and footer menu. But in our webmasters...internal links section, only homepage is at the top. Blog-posts are others followed by homepage. I wonder why blog pages are dominating our website pages. Please give your suggestions on this. Do you think Google will give more priority for the blog-posts than website pages as they are more linked technically? Thanks
Algorithm Updates | | vtmoz1 -
Why is Page Authority dropping?
Hi I'm trying to review pages which have previously ranked, but in March have dropped out completely. Some of these pages I can see have dropped to having a Page Authority of 1, we haven't changed anything on these pages, so is there a reason why the authority has dropped? These pages only had around 8 - 10 Page Authority to begin with. I'm trying to identify why we have lost keywords, and if it has anything to do with the Google Updates in March Here are examples of the pages with drops: http://www.key.co.uk/en/key/heavy-duty-shelving-1830x1830mm-blue-orange
Algorithm Updates | | BeckyKey
http://www.key.co.uk/en/key/metal-feet-for-heavy-duty-steel-shelving
http://www.key.co.uk/en/key/health-and-safety-law-poster-a2 Thank you!0 -
Schema Mark up - Product Listing Pages
Hi I know you can add product schema to a product page, but can you add mark up to a product listing/category page? If so, which one would you use? I saw the item list mark up but didn't think this was relevant. Thank you
Algorithm Updates | | BeckyKey0 -
Our company is mentioned on some high-traffic, authoritative sites and some of our products are linked as well. If we link to those pages, does it affect our SEO? How can we take advantage of those mentions?
I heard that if you link to another site, when Google indexes your site, they crawl that page that is referenced. By whatever metrics they use, if that site has your name or a link to your site, Google would rank it higher. I am not sure how true that is, but what value does another site mentioned our site have on our SEO?
Algorithm Updates | | JonathonOhayon1 -
Does the page title keyword count in anchor text when link is web address?
If someone links to my plumbing site with this link as the anchor text: http://www.plumbers.com/austin-plumbers.html does the key phrase "austin plumbers" get counted in the anchor text by google or is this a sample of anchor text that google ignores? Thanks mozzers! Ron
Algorithm Updates | | Ron100 -
How long can your header be in number of characters?
hey all, have hear conflicting information on how many characters in the header you are allowed? i've heard 60, 65, and 70?? what are your thoughts? thanks!!
Algorithm Updates | | tm46150 -
Trying to figure out why one of my popular pages was de-indexed from Google.
I wanted to share this with everyone for two reasons. 1. To try to figure out why this happened, and 2 Let everyone be aware of this so you can check some of your pages if needed. Someone on Facebook asked me a question that I knew I had answered in this post. I couldn't remember what the url was, so I googled some of the terms I knew was in the page, and the page didn't show up. I did some more searches and found out that the entire page was missing from Google. This page has a good number of shares, comments, Facebook likes, etc (ie: social signals) and there is certainly no black / gray hat techniques being used on my site. This page received a decent amount of organic traffic as well. I'm not sure when the page was de-indexed, and wouldn't have even known if I had't tried to search for it via google; which makes me concerned that perhaps other pages are being de-indexed. It also concerns me that I have done something wrong (without knowing) and perhaps other pages on my site are going to be penalized as well. Does anyone have any idea why this page would be de-indexed? It sure seems like all the signals are there to show Google this page is unique and valuable. Interested to hear some of your thoughts on this. Thanks
Algorithm Updates | | NoahsDad0