Quickest way to deindex a large number of pages
-
Our site was recently hacked by spammers posting fake content and bringing down our servers, etc. After a few months, we finally figured out what was going on and fixed the issue. However, it turns out that Google has indexed 26K+ spammy pages and we've lost page rank and search engine rankings as a result.
What is the best and fastest way to get these pages out of Google's index?
-
Given that I'm sure you've removed these pages from your site, there will be no page to which to add a meta-noindex tag.
Disallowing these pages in robots.txt in no way signals to the search engines that they should be removed from the index, just that they should no longer be crawled. Given that they're already indexed, blocking in robots.txt would potentially save some "crawl budget" but wouldn't do anything to remove them from the index.
So submitting them to the URL Removal Tool would be by far the most effective, along with an explanation.
You'll also want to keep a very close watch on your penalty warnings within Webmaster Tools. If you get flagged, you'll want a complete history of the issue and the steps you've taken to address it in order to prepare a reinclusion request.
Lastly, don't forget to submit these same URLs to the Bing Webmaster Tools Block URLs tool. You may not get a massive amount of traffic from Bing, but there's no sense throwing it away, since you've already prepared the URL removal list anyway.
Hope that helps?
Paul
-
Yup. Just wanted to add as well that if these pages are in a particular directory, then you can deindex the entire directory in one command using the URL removal tool.
-
Disallow in robots.txt
Add a noindex meta tag to these pages
Request Google to remove the URLs from their index via WMT URL removal request
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301'ing old (2000), high PR, high pages indexed domain
Hi, I have an old (2000), very high PR, 20M+ pages indexed by goog domain which... got adsense banned. The domain has taken a few hits over the years from penguin/panda, but come out pretty well compared to many competitors. The problem is it was adsense banned in the big adsense acct ban of 2012 for invalid activity. No, I still have no idea what the issue was. I'd like to start using a new domain if I can safely get goog to pass the PR & indexing love so I can run adsense & Adx. What are your initial thoughts? Am I out of my mind to try?
Algorithm Updates | | comfortsteve1 -
Best Moz article on landing pages?
From what I understand, building landing pages to link back to sites is a thing of the past. I am looking for a good article that explains best current landing page practices (post Panda and Penquin). Any suggestions?
Algorithm Updates | | cschwartzel0 -
Rankings fluctuating by around 10 pages between night and day
Hi all, I'm experiencing something very odd with my website ranking at the moment. My homepage is fluctuating in rank for my main keyword by 10 pages every day and night. So, during the day i am on page 14, 15 or 16 for my main keyword yet by night i am on page 5 or 6. This trend has continued for the past 7 days now and i can't quite understand why this is. I'm using pagewash dot net to carry out manual searches and a ranking tool - both of which produce exactly the same result. Does anyone have any experience of this or why this is happening? My domain is around 8 years old and has around 50,000 pages. Any pointers would be greatly appreciated.
Algorithm Updates | | MarkHincks0 -
A Serious drop in Pages crawled per day
On 21st April ,I spotted a sudden decrease in pages crawled per day.Previously it was about 5,000 bust after the drop it reached to 225.From the crawl rate never spiked. Here is my website url - http://www.wpstuffs.com/ 8fQHW2G.png
Algorithm Updates | | vividvilla0 -
Is it ok to repeat part of a meta-description across multiple pages?
For example, what if I was to conclude each meta-description tag with the line... "Free shipping for orders over $90." The rest of the meta-description tag on every page is unique, but the last sentence would be the same or at least similar. Thoughts?
Algorithm Updates | | B-man0 -
Is anybody else seeing large scale rankings drops in Bing this week?
I track around 1000 keywords for this site, and my rankings in Bing dropped for about half of them on Wednesday. No major changes have been made to the site, rankings are maintaining or improving in Google for a majority of these same terms. The average drop seems to be around 9-12 places, which to me signals more than just standard fluctuation. Anyone else seeing anything strange with Bing this week? Or does anyone have any ideas? I looked for posts about an algorithm change but haven't found anything. Thanks.
Algorithm Updates | | BrianCC0 -
Title of home page is changed to domain name in SERPs
Hi, We have a unique problem, we are getting a totally different title in Google serps for a large site. When we search with domain name with space in google.com. We are getting title as domain name with space. We don't have any Open Directory listing. We don't have any cannonical issues and other pages with title as domain name. Can you please tell us what we have to do get our original title back in SERP ? Thanks, With Regards,
Algorithm Updates | | semshah1430 -
If a page one result for a keyword is mostly directories, do I have a chance to rank for this keyword?
I feel like although directories carry a lot of weight and links, I'd think that my client would be able to gain a top position, since none of the others are competitor pages, nor are the directories engaging.
Algorithm Updates | | randallseo0