Quickest way to deindex large parts of a website
-
Hey there,
my clients website was set up with subdirectories for almost every country in the world plus multiple languages in each country. The content in each subfolder is (almost) identical. So no surprise: They have a big problem with duplicate content and ranking fluctuations.
Since they don't want to change the site's structure I recommended limiting the languages available in each subfolder with robots.txt. However before doing this we marked the contents to be exluded with noindex, nofollow. It's only been 2 days now but I hardly notice any decline in the number of indexed pages.
I was therefore wondering if it would speed up things if I marked the pages with just noindex instead of noindex and nofollow.
It would be great if you could share your thoughts on that.
Cheers,
Jochen
Hey there,
my clients website was set up with subdirectories for almost every country in the world plus multiple languages in each country. The content in each subfolder is (almost) identical. So no surprise: They have a big problem with duplicate content and ranking fluctuations.
Since they don't want to change the site's structure I recommended limiting the languages available in each subfolder with robots.txt. However before doing this we marked the contents to be exluded wiht noindex, nofollow. It's only been 2 days now but I hardly notice any decline in the number of indexed pages.
I was therefore wondering if it would speed up things if I marked the pages with just noindex instead of noindex and nofollow.
It would be great if you could share your thoughts on that.
Cheers,Jochen -
Thanks for the hint Dirk! I've used the tool and it works great. I even found a handy chrome extension ("WebMaster Tools - Bulk URL removal") that made the removal of my 3,000 subdirectories very smooth and saved me about 25 hours of manual work!
WebMaster Tools - Bulk URL removal
-
Hi,
There was a similar question a few days ago: https://moz.com/community/q/is-there-a-limit-to-how-many-urls-you-can-put-in-a-robots-txt-file
Quote: Google Webmaster Tools has a great tool for this. If you go into WMT and select "Google index", then "remove URLs". You can use regex to remove a large batch of URLs then block them in robots.txt to make sure they stay out of the index.
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
HTTP → HTTPS Migration - Both Websites Live Simultaneously
We have a situation where a vendor, who manages a great deal of our websites, is migrating their platform to HTTPS. The problem is that the HTTP & new HTTPS versions will be live simultaneously (in order to give clients time to audit both sites before the hard switch). I know this isn't the way that it should be done, but this is the problem we are facing. My concern was that we would have two websites in the index, so I suggested that they noindex the new HTTPS website until we are ready for the switch. They told me that they would just add cannonicals to the HTTPS that points to the HTTP and when it's time for the switch reverse the cannonicals. Is this a viable approach?
Intermediate & Advanced SEO | | AMSI-SEO0 -
How to find the redirects on website
I want to find the complete internal redirects on website. Just internally linked. How to find such?
Intermediate & Advanced SEO | | vtmoz0 -
Website Ranks and gets de indexed ??
Hi My website is almost 3-4 months old . Whats strange is that as soon as it get Crawled it ranks for few terms for 1-2 days and all of a sudden gets de Indexed for these same terms or Rank drops like drops from page 5 to page 10 . Nothing shows up in Webmater tools under Manual Action . Assuming its a Algorithmic penalty, How to deal with this kind of stuff. Should I stop working on this site all together ? Or assuming its a New website, google does not want it to rank for medium or high volume keywords ? What keywords I am after have 300 -2k searches per month .
Intermediate & Advanced SEO | | aus00070 -
What is the best way to take advantage of this keyword?
Hi SEO's! I've been checking out webmaster tools (screenshot attached) and noticed that we're getting loads of long tail searches around a search query 'arterial and venous leg ulcers' - on a side note we're a nursing organisation so excuse the content of the search!!! The trouble is that google is indexing a PDF page which we give out as a freebie:
Intermediate & Advanced SEO | | 9868john
http://www.nursesfornurses.com.au/admin/uploads/5DifferencesBetweenVenousAndArterialLegUlcers1.pdf This PDF is a couple of years old and needs updating but its got a few links pointing to it. Ok so down to the nitty gritty, we've just launched a blog:
http://news.nursesfornurses.com.au/Nursing-news/ We have a whole wound care category in which this content belongs, and i'm trying to find the best way to take advantage of the search, so I was thinking: Create an article of about 1000 words Update the PDF and re-upload it to the main domain (not the sub domain news.nursesfornurses.com.au) Attach the PDF to the article on the blog OR would it be better to host this on the blog, and setup a 301 redirect to this page? I just need some advice on how best to take advantage of this opportunity, our blog isn't getting much search traffic at the moment (despite having 300+ articles!!) and i'm looking into how we can change that. I look forward to your response and suggestions. Thanks! qtY64B10 -
60 countries, 1 website. How to develop this?
I've been asked to rank the website of an international association under the same keywords but in multiple languages (they rank very highly for English speaking countries), however they only have the one website with 1 .com domain. Question: Is the better approach to translate the site in multiple languages and then allow people to select the language they desire? OR Buy ccTLD and run the site multiple times in different languages as separate entities?
Intermediate & Advanced SEO | | MassivePrime0 -
We're currently not using schemas on our website. How important is it? And are websites across the globe using it?
Schemas looks like an important thing when it comes to structuring your website and ensuring the crawl bots get all the details. I've been reading a lot of articles around the web and most of them are saying that schemas are important but very few websites are using it. Why so? Are the schemas on schema.org there to stay or am I wasting my time?
Intermediate & Advanced SEO | | Shreyans920 -
API to power all websites
I spoke to one of my lead web developers and we are planning on powering all of our desktop, mobile and tablet sites with our new API. Everything will be populated through javascript and is cloud-based rather than through actual HTML. How do we incorporate all of our SEO?
Intermediate & Advanced SEO | | recbrands0 -
Website architecture and sitewide link
Hello, I was reading this article about website architecture http://www.seomoz.org/blog/site-architecture-for-seo and I have a question about site wide link... I don't think site wide link are good if you want your homepage to have the most " juice " can someone confirm that to me. What I mean by site wide link is let's say I have a page about golf in califorina and I create a link to another page of my website which is about golf in florida that a good practicee to do that or not ? I just used this calculator http://www.webworkshop.net/pagerank_calculator.php and and if I do site wide links the juice doesn't go to my most important page ( the homepage ) which is the one I currently want to have the most juice. However, I noticed that website like amazon do site wide link let's say you are looking at a book and they have lots of links for other books or categories and they still rank very well for the word "textbooks" so I am kind of lost and don't understand why they still rank even though to me their juice goes away by doing site wide links to other books or categories ( even though it is related ) Can someone tell me more about site wide links are they good , bad etc... and is this calculator I use right or wrong ? Thank you
Intermediate & Advanced SEO | | seoanalytics0