Quickest way to deindex a large number of pages
-
Our site was recently hacked by spammers posting fake content and bringing down our servers, etc. After a few months, we finally figured out what was going on and fixed the issue. However, it turns out that Google has indexed 26K+ spammy pages and we've lost page rank and search engine rankings as a result.
What is the best and fastest way to get these pages out of Google's index?
-
Given that I'm sure you've removed these pages from your site, there will be no page to which to add a meta-noindex tag.
Disallowing these pages in robots.txt in no way signals to the search engines that they should be removed from the index, just that they should no longer be crawled. Given that they're already indexed, blocking in robots.txt would potentially save some "crawl budget" but wouldn't do anything to remove them from the index.
So submitting them to the URL Removal Tool would be by far the most effective, along with an explanation.
You'll also want to keep a very close watch on your penalty warnings within Webmaster Tools. If you get flagged, you'll want a complete history of the issue and the steps you've taken to address it in order to prepare a reinclusion request.
Lastly, don't forget to submit these same URLs to the Bing Webmaster Tools Block URLs tool. You may not get a massive amount of traffic from Bing, but there's no sense throwing it away, since you've already prepared the URL removal list anyway.
Hope that helps?
Paul
-
Yup. Just wanted to add as well that if these pages are in a particular directory, then you can deindex the entire directory in one command using the URL removal tool.
-
Disallow in robots.txt
Add a noindex meta tag to these pages
Request Google to remove the URLs from their index via WMT URL removal request
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does Google giving more important to internal pages than homepage recently? Especially after the recent Major algo update?
Hi everybody, I can see the change Google brought in the SERP. Previously website homepages will be shown for primary keywords, now it's slowly and almost switched to showing most related internal pages in a website. You can check same for keyword "SEO", Most or all the results are internal pages. I can see this change for our primary keyword from last one month. So basically Google is trying to show a page explaining about the primary keywords rather than website, that's how "what is seo" pages are ranking than homepages. If there is no such pages existed or not well written, Google is just showing the website homepage. But I noticed that websites ranking with homepages are dropped compared to the websites with dedicated page about that primary keyword. Please share your thoughts. Thanks
Algorithm Updates | | vtmoz0 -
Clicks are the ultimate factor to stick the page on position?
Hi all, We know many factors contribute to make a page rank at (top) position like somewhere in top 5 results. I have seen some of our pages suddenly spike to that positions and locked there. They been receiving clicks too. Will they be dropped if they don't get estimated clicks? I think many factors contribute to make a page rank higher but clicks are the one factor which makes the page consistently rank at its best position. What do you say? Thanks
Algorithm Updates | | vtmoz0 -
Google Search Analytics desktop site to losing page position compared to the mobile version of the site
Looking at Google Search Analytics page position by device. The desktop version has seen a dramatic drop in the last 60 days compared to the mobile site. Could this be caused by mobile first indexing? Has Google had any releases that might have caused this?
Algorithm Updates | | merch_zzounds0 -
Landing page redirect along with complete content
Hi Moz community, We have a page with "keyword" we are targeting in slug like website.com/keyword/. This page doesn't have much back-links or visits like homepage. So we decided to redirect homepage to /keyword page along with complete content. Will this going to hurt? Only change anybody can notice is URL. Are there any risks involved. I think this is the best way to highlight the page we been thinking about. Thanks
Algorithm Updates | | vtmoz0 -
Numbers vs #'s For Blog Titles
For your blog post titles, is it "better" to use numbers or write them out? For example, 3 Things I love About People Answering My Constant Questions or Three Things I Love About People Answering My Constant Questions? I could see this being like the attorney/lawyer, ecommerce/e-commerce and therefore not a big deal. But, I also thought you should avoid using #'s in your url's. Any thoughts, Ruben
Algorithm Updates | | KempRugeLawGroup0 -
Should you include Website Title in all page title tags?
We recently spent analyzing some of the best SEO software companies on the U.S. market fishing for the best practices in SEO and I saw one thing in common : They all had website titles in all the page title tags separated by " | " Is that the best practice for SEO or is it just for Branding? Interestingly enough, the website titles were completely unrelated to the pages' content or keywords. (Here's my personal opinion on what it looked like: "riding on a bicycle" | Ferrari ) But when I looked up the keywords ... ranked #1 or #2 spots, in some serious competition. (So in the example above, "bicycle" would be in the top spot)
Algorithm Updates | | HMCOE0 -
How could Penguin kill my top ten rank and promote this garbage page to a #5 spot
Hey, Before penguin, I had a #9 rank for the term "yoga poses". So as many of us are doing, I started looking at my link profile... and yes, there were around 300 links from an old yoga news website (anchor: yoga poses)... that lead to the page on my site optimized for this term. The problem is they took the site down, but not properly... I.E. they generate a "not available" message for browsers, but underneath, I guess the bots can still index all the pages... so I guess they were interpreting these links as coming from a cloaked site. So, I was able to get them to remove the links... webmaster tools reports half of them gone now. What I don't get though... is how Google can give this garbage page a #5 spot for a competitive term like "yoga poses"... Check out http://www.ebmyoga.com/beginyoga.html and compare it to my page... http://www.yogaclassplan.com/yoga-poses/ This page leads to highly quality 100% unique yoga pose articles... in my mind we deliver so much more value than the site with a #5 rank. I don't understand. Any insight? Thanks,
Algorithm Updates | | biomat0 -
Google +1 link on Domain or Page?
Since its release, I've seen Google +1 being used across an entire domain but only reference the root href in the code snippet. At the same time, you see other sites use +1 more naturally with the button being specific to the page you're on. What's your take on this? To clarfiy, do you add: or .. on each page.
Algorithm Updates | | noeltock0