Quickest way to deindex large parts of a website
-
Hey there,
my clients website was set up with subdirectories for almost every country in the world plus multiple languages in each country. The content in each subfolder is (almost) identical. So no surprise: They have a big problem with duplicate content and ranking fluctuations.
Since they don't want to change the site's structure I recommended limiting the languages available in each subfolder with robots.txt. However before doing this we marked the contents to be exluded with noindex, nofollow. It's only been 2 days now but I hardly notice any decline in the number of indexed pages.
I was therefore wondering if it would speed up things if I marked the pages with just noindex instead of noindex and nofollow.
It would be great if you could share your thoughts on that.
Cheers,
Jochen
Hey there,
my clients website was set up with subdirectories for almost every country in the world plus multiple languages in each country. The content in each subfolder is (almost) identical. So no surprise: They have a big problem with duplicate content and ranking fluctuations.
Since they don't want to change the site's structure I recommended limiting the languages available in each subfolder with robots.txt. However before doing this we marked the contents to be exluded wiht noindex, nofollow. It's only been 2 days now but I hardly notice any decline in the number of indexed pages.
I was therefore wondering if it would speed up things if I marked the pages with just noindex instead of noindex and nofollow.
It would be great if you could share your thoughts on that.
Cheers,Jochen -
Thanks for the hint Dirk! I've used the tool and it works great. I even found a handy chrome extension ("WebMaster Tools - Bulk URL removal") that made the removal of my 3,000 subdirectories very smooth and saved me about 25 hours of manual work!
WebMaster Tools - Bulk URL removal
-
Hi,
There was a similar question a few days ago: https://moz.com/community/q/is-there-a-limit-to-how-many-urls-you-can-put-in-a-robots-txt-file
Quote: Google Webmaster Tools has a great tool for this. If you go into WMT and select "Google index", then "remove URLs". You can use regex to remove a large batch of URLs then block them in robots.txt to make sure they stay out of the index.
Dirk
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best way to structure website URLs ?
Hi, can anyone help me to understand if having category folder in URL matters or not? how to google treat a URL? for example, I have the URL www.protoexpress.com/pcb/certification but not sure google will treat it a whole or in separate parts? if in separate parts, is it safe to use pcb/pcb-certification? or it will be considered as keyword stuffing? Thank you in anticipation,
Intermediate & Advanced SEO | | SierraPCB1 -
Conundrum with brand new website keywords...
I'm working with on a website for an app called BetterRX. There's a prescription card called BetterRX Card. Our domain is Better RX.com and the card is BetterRXCard.com. "Better RX" as a brand search is dominated by prescription discount cards, with Good RX being the most dominant. Any suggestions on how to go about mixing optimization for brand as well as the app?
Intermediate & Advanced SEO | | sickle3110 -
Temporarily redirecting a small website to a specific url of another website
Hi, I would like to redirect a small website that contains info about a specific project temporarily to a specific url about this project on my main website. Reason for this is that the small website doesn't contain accurate info anymore. We will adapt the content in the next few weeks and then remove the redirect again. Should I set up a 301 or a 302? Thanks
Intermediate & Advanced SEO | | Mat_C1 -
Homepage is deindexed in Google
Please help for some reason my website home page has disappeared, we have been working on the site but nothing that I can think of which would block it. There are no warnings in google console? Can anyone lend a hand in understanding what has gone wrong, I would really appreciate it. The site is: http://www.discountstickerprinting.co.uk/ Seems to be working again but I had to fetch the home page in google console, any idea why this has happened cannot afford a heat op at this age lol?
Intermediate & Advanced SEO | | BobAnderson0 -
SEO transfer to new website
My website currently has some strong SEO and i will be re-developing the website on a wordpress platform...which will change many of the existing URL. Will this affect the current pages that are well indexed in Google? Does using Wordpress or changing the URL extension (.html to .php) make a difference? If i want to make a clean transition without effecting our existing SEO...what are some essential steps i need to take? Example. Current page is www.mydomain.com/name.html .... and the new URL would be www.mydomain.com/product/name.php Thanks
Intermediate & Advanced SEO | | Souk0 -
Penguin hit Website - Moving to new domain
Hey! I am working on a Penguin hit Website. Still ranking for all brand keywords and blog articles are still being returned in Google SERPs, but the website is showing up for only 3 or 4 money keywords. It is clearly a penguin hit as it was ranked 1st page for all money keywords before latest update (3.0). We already did a link cleanup and disavowed all bad backlinks. Still, the recovery process could take over 2 years from previous experience, and in 2 years, the site will suffer a slow death. Solution: We own the .com version of the domain, currently being served on the .net. We bought the .com version about 6 years ago, it is clean and NOT redirected to the .net (actual site). We were thinking about moving the whole Website to the .com version to start over. However, we need to make sure Google doesn't connect the 2 sites (no pagerank flow). Of course Google will notice is the same content, but there won't be any pagerank flowing from the old site to the new one. For this, we thought about the following steps: Block Googlebot (and only googlebot) for the .net version via robots.txt. Wait until Google removes all URLs from the index. Move content to the .com version. Set a 301 redirect from .net to .com (without EVER removing the block on googlebot). Thoughts? Has anyone went over this before? Other ideas? Thanks!
Intermediate & Advanced SEO | | FedeEinhorn0 -
Best way to merge 2 ecommerce sites
Our Client owns two ecommerce websites. Website A sells 20 related brands. Website has improving search rank, but not normally on the second to fourth page of google. Website B was purchased from a competitor. It has 1 brand (also sold on site A). Search results are normally high on the first page of google. Client wants to consider merging the two sites. We are looking at options. Option 1: Do nothing, site B dominates it’s brand, but this will not do anything to boost site A. Option 2: keep both sites running, but put lots of canonical tags on site B pointing to site A Option 3: close down site B and make a lot of 301 redirects to site A Option 4: ??? Any thoughts on this would be great. We want to do this in a way that boosts site A as much as possible without losing sales on the one brand that site B sells.
Intermediate & Advanced SEO | | EugeneF0 -
Major Website Migration Recovery Ideas?
Since starting our business back in 2006 we've gone through alot of branding, and as a result URL and architectual migrations. This has always something that has been driven by usability, brand awareness and technical efficiency reasons, while knowing that there would be SEO hits to take from it....but ultimately hoping to have a much stronger foundation from an SEO perspective in the long run. Having just gone through our most recent (and hopefully final) migration, we are now about 15% down on traffic (although more like 35% - 40% in real terms when seasonality is stripped out). Below is a timeline to our structural history: 2007 - 2009 = We operated as a network of inidividual websites which started as 1, www.marbellainfo.com, but grew to 40, with the likes of www.thealgarveinfo.com, www.mymallorcainfo.com, www.mytenerifeinfo.com, www.mymaltainfo.com etc.. 2009 - 2010 = We decided to consolitdate everything onto 1 single domain, using a sub-domain structure. We used the domain www.mydestinationinfo.com and the subdomains http://marbella.mydestinationinfo.com, http://algarve.mydestinationinfo.com etc.. All old pages were 301 redirected to like for like pages on the new subdomains. We took a 70% drop in traffic and SERPS disappeared for over 6 months. After 9 months we had recovered back to traffic levels and similar rankings to what we had pre-migration. Using this new URL structure, we expanded to 100 destinations and therefore 100 sub-domains. 2011 = In April 2011, having not learnt our lesson from before :(, we undwent another migration. We had secured the domain name www.mydestination.com and had developed a whole new logo and branding. With 100 sub-domains we underwent a migration to the new URL and used a sub-directory folder. So this time www.myalgarveinfo.com had gone to <a></a>http://algarve.mydestinationinfo.com and was now www.mydestination.com/algarve. No content or designs were changed, and again we 301 re-directed pages to like for like pages and with this we even made efforts to ask those linking to us to update their links to use our new URL's. The problem: The situation we fine ourselves in now is no where near as bad as what happend with our migration in 2009/2010, however, we are still down on traffic and SERPS and it's now been 3 months since the migration. One thing we had identified was that our re-directs where going through a chain of re-directs, rather than pointing straight to the final urls (something which has just been rectified). I fear that our constant changing of URL's has meant we have lost out in terms of the passing over of link juice from all the old URL's and loss of trust with Google for changing so much. Throughout this period we have grown the content on our site by almost 2x - 3x each year and now have around 100,000 quality pages of unique content (which is produced by locals on the ground in each destination). I'm hoping that someone in the SEOmoz Community might have some ideas on things we may have slipped up on, or ways in which we can try and recover a little faster and actually get some growth, as opposed to working hard and waiting a while just for another recovery. Thanks Neil
Intermediate & Advanced SEO | | Neil-MyDestination0