Removing URLs in bulk when directory exclusion isn't an option?
-
I had a bunch of URLs on my site that followed the form:
http://www.example.com/abcdefg?q=&site_id=0000000048zfkf&l=
There were several million pages, each associated with a different site_id. They weren't very useful, so we've removed them entirely and now return a 404.The problem is, they're still stuck in Google's index. I'd like to remove them manually, but how? There's no proper directory (i.e. /abcdefg/) to remove, since there's no trailing /, and removing them one by one isn't an option. Is there any other way to approach the problem or specify URLs in bulk?
Any insights are much appreciated.
Kurus
-
I'd go into Google Webmaster Tools and their parameter settings and tell them to ignore this parameter.
I would need to look up the exact syntax, but Google does accept some dynamic exclusions and parameters in robots.txt, and you may be able to put that into robots and then use the URL removal tools.
-
There are no links to these pages, so no juice. There are also no 'new' replacement pages. We just want them out of the index ASAP by any means necessary.
-
You should have 301 your most important pages to the new urls, so that you would keep your juice.
-
Thanks, but the goal is to expedite the removal process via the URL removal tool. We've already 404'd the pages, so they'll be removed from the index. It's a question of timing, since the pages in question are low quality and hurting us in the context of Panda.
-
try 301 redirect for most important links. http://www.seomoz.org/learn-seo/redirection
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ranking for keyword I don't optimize for & Other oddities
Hi Moz Community! I've been working with a clients website for about a year now. They were hit with the original Panda update because of some spammy links from a shady SEO firm. We've made a decent climb back but not a full recovery. There are some weird things happening that I would love some insight into. 1. Ranking for keywords we don't optimize for: I noticed some low keyword volume for a keyword term that is close to our main term, but is slightly different. We don't optimize for this term at all on our website. We rank third for this term, and actually show site links in the result, which doesn't happen for any of our other pages. 2. Index not found when doing site: search: Other oddity is that when you search site:www.mywebsite.com, I see all the pages within the site except the homepage. Not sure whats going on here, but when I fetch the homepage in GWMT, it returns the homepage. When you query the homepage by itself, it also ranks. Any help would be appreciated! Regards, J
Intermediate & Advanced SEO | | artscienceweb0 -
Are clean mobile URL's necessary?
Adding code to redirect/clean up ugly URL's slows down mobile site performance, so it is necessary if we are already using rel=alternate tags on our desktop/www pages?
Intermediate & Advanced SEO | | recbrands0 -
New my domain.com/blog option vs. my blog.mydomain.com option
Our e-commerce site has been on Big Commerce for about a year now. One thing many SEO folks had told us is that having a blog located at /blog was going to help more than a subdomain blog. option. BC has never had the option to have a blog hosted on their platform (/blog) until now. I am now wondering, since we have lost traffic in the past and are trying everything we can to regain it, if we should purchase the Wordpress Site Redirect upgrade and move the subdomain blog (blog.) to the new site option /blog. Any help or feedback from you is very much appreciated. I have attached a screenshot of our main website vs. our blog from Open Site Explorer in case it helps anything. I29Tw5P
Intermediate & Advanced SEO | | josh3300 -
Can't find X-Robots tag!
Hi all. I've been checking out http://www.unthankbooks.com/ as it seems to have some indexing problems. I ran a server header check, and got a 200 response. However, it also shows the following: X-Robots-Tag:
Intermediate & Advanced SEO | | Blink-SEO
noindex, nofollow It's not in the page HTML though. Could it be being picked up from somewhere else?0 -
Whats the best way to revive a directory that was 301'd and now I want to remove that?
Last year i 301'd one of my directories on my site, pointing everything to a different directory. Long story short I am going to sell this product line again and would like to just remove the 301 to that original directory, but I am reading that the 301s are also cached in most browsers for a long time. Has anyone successfully done this and if you did what was it that you had to do? Thanks Mike
Intermediate & Advanced SEO | | SandyEggo0 -
Directory backlink
Hello everyone, I know that this question has been asked millions of time, but I am really not getting a straight answer for it. Well the question will be divided in few other questions : Google changed, I get that, but I am reading everywhere, come up with a great content and the rest will follow, stop creating your own backlink and let user link to you ... But I don't know if this is apply for every site on the web, let take the example of a flash gaming site that we manage, we are creating games every day, coming up with great (unique) text for each of them, we are active on social media and stopped backlink from directories. But now we can see our sites losing ranking and seeing some websites that are not having much content on their sites or even active on social medias that are ranking better than us. We always used white hat techniques, this is why we were so well ranked for so long, but now we see our ranking change on a daily basis but can't explained why. So my question is, should we totally stop directories backlink (even the good directories)? Or we should keep on going and try PR at the same time? For a site that just started how on earth will he be able to get backlinks if it's not using directories in the first place? So I feel that I am going in circle here and I don't know what else we could do to improve our site. We even recast the site to bring better experience to the user to see if this will help on us on getting our ranking back. And this help, as the page views and time on the site improved with it, but the ranking is still unchanged (that has been done 3 months ago). Just to let you know we are aware about the panda and penguin updates 🙂 Thanks for your help on this, and I hope the answers will help others 🙂 Thanks, Mounir
Intermediate & Advanced SEO | | drimlike0 -
No longer showing for 'money' phrases but long tail combinations rank high?
I hope someone can shed some light on this as I've been pulling my hair out so much there's hardly any left! Background: 12 year old website that for about 10 years had Top 3 rankings for 100's of phrases but rankings first dropped off August 2011. Panda seemed to be the cause but finding the exact issue is hard. We are an online travel agent and every hotel page has duplicate content copied from other websites. This has not been changed although lots of sections in the site still rank well, so do the hotel pages themselves. Lots of internal duplicate issues have been resolved but with no effect. Our old style link, link, link all day long with our 2-word main key phrase as anchor text has given us an unnatural backlink profile but no message has been left by G about this in WMT (yet). Internal link structure is poor with all pages linking back to the homepage with our 'money' 2-word phrase in 3 places. Penguin wiped two thirds of all our backlinks back in May 2012. Why then, do we still rank for our 'money' phrase on the homepage when it has some extra words included and becomes long tail? e.g. CityName Apartments (money phrase) - Now ranks page 2-3 CityName Apartments to rent for the night - Ranks #2 on Google in all countries To make things more confusing other pages rank really well for similar money phrase e.g. CityName Apartments Offers - Ranks 2nd on 185,000,000 results (not homepage) It seems only the homepage is effected (where 95% of inbound links point) but if the site wide duplicates or unnatural link profile was flagged it would effect more than one page of the site. Wouldn't it?
Intermediate & Advanced SEO | | lchoice0 -
Best solution to get mass URl's out the SE's index
Hi, I've got an issue where our web developers have made a mistake on our website by messing up some URL's . Because our site works dynamically IE the URL's generated on a page are relevant to the current URL it ment the problem URL linked out to more problem URL's - effectively replicating an entire website directory under problem URL's - this has caused tens of thousands of URL's in SE's indexes which shouldn't be there. So say for example the problem URL's are like www.mysite.com/incorrect-directory/folder1/page1/ It seems I can correct this by doing the following: 1/. Use Robots.txt to disallow access to /incorrect-directory/* 2/. 301 the urls like this:
Intermediate & Advanced SEO | | James77
www.mysite.com/incorrect-directory/folder1/page1/
301 to:
www.mysite.com/correct-directory/folder1/page1/ 3/. 301 URL's to the root correct directory like this:
www.mysite.com/incorrect-directory/folder1/page1/
www.mysite.com/incorrect-directory/folder1/page2/
www.mysite.com/incorrect-directory/folder2/ 301 to:
www.mysite.com/correct-directory/ Which method do you think is the best solution? - I doubt there is any link juice benifit from 301'ing URL's as there shouldn't be any external links pointing to the wrong URL's.0