Most Painless way of getting Duff Pages out of SE's Index
-
Hi,
I've had a few issues that have been caused by our developers on our website. Basically we have a pretty complex method of automatically generating URL's and web pages on our website, and they have stuffed up the URL's at some point and managed to get 10's of thousands of duff URL's and pages indexed by the search engines.
I've now got to get these pages out of the SE's indexes as painlessly as possible as I think they are causing a Panda penalty.
All these URL's have an addition directory level in them called "home" which should not be there, so I have:
www.mysite.com/home/page123 instead of the correct URL www.mysite.com/page123
All these are totally duff URL's with no links going to them, so I'm gaining nothing by 301 redirects, so I was wondering if there was a more painless less risky way of getting them all out the indexes (IE after the stuff up by our developers in the first place I'm wary of letting them loose on 301 redirects incase they cause another issue!)
Thanks
-
If all /home directories should be removed, and they all correlate directly to the same URL without the /home then the best means to correct the issue would be a single regex expression to redirect the URLs.
Once you add the redirect Google's index should be cleaned up in about 30 days.
-
Add robots.txt to stop Google crawling the subdirectories you don't want shown such as /home
More information is available on http://www.seomoz.org/learn-seo/robotstxt
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
After hack and remediation, thousands of URL's still appearing as 'Valid' in google search console. How to remedy?
I'm working on a site that was hacked in March 2019 and in the process, nearly 900,000 spam links were generated and indexed. After remediation of the hack in April 2019, the spammy URLs began dropping out of the index until last week, when Search Console showed around 8,000 as "Indexed, not submitted in sitemap" but listed as "Valid" in the coverage report and many of them are still hack-related URLs that are listed as being indexed in March 2019, despite the fact that clicking on them leads to a 404. As of this Saturday, the number jumped up to 18,000, but I have no way of finding out using the search console reports why the jump happened or what are the new URLs that were added, the only sort mechanism is last crawled and they don't show up there. How long can I expect it to take for these remaining urls to also be removed from the index? Is there any way to expedite the process? I've submitted a 'new' sitemap several times, which (so far) has not helped. Is there any way to see inside the new GSC view why/how the number of valid URLs in the indexed doubled over one weekend?
Intermediate & Advanced SEO | | rickyporco0 -
Facets Being Indexed - What's the Impact?
Hi Our facets are from what I can see crawled by search engines, I think they use javascript - see here http://www.key.co.uk/en/key/lockers I want to get this fixed for SEO with an ajax solution - I'm not sure how big this job is for developers, but they will want to know the positive impact this could have & whether it's worth doing. Does anyone have any opinions on this? I haven't encountered this before so any help is welcome 🙂
Intermediate & Advanced SEO | | BeckyKey0 -
When Mobile and Desktop sites have the same page URLs, how should I handle the 'View Desktop Site' link on a mobile site to ensure a smooth crawl?
We're about to roll out a mobile site. The mobile and desktop URLs are the same. User Agent determines whether you see the desktop or mobile version of the site. At the bottom of the page is a 'View Desktop Site' link that will present the desktop version of the site to mobile user agents when clicked. I'm concerned that when the mobile crawler crawls our site it will crawl both our entire mobile site, then click 'View Desktop Site' and crawl our entire desktop site as well. Since mobile and desktop URLs are the same, the mobile crawler will end up crawling both mobile and desktop versions of each URL. Any tips on what we can do to make sure the mobile crawler either doesn't access the desktop site, or that we can let it know what is the mobile version of the page? We could simply not show the 'View Desktop Site' to the mobile crawler, but I'm interested to hear if others have encountered this issue and have any other recommended ways for handling it. Thanks!
Intermediate & Advanced SEO | | merch_zzounds0 -
Google indexed wrong pages of my website.
When I google site:www.ayurjeewan.com, after 8 pages, google shows Slider and shop pages. Which I don't want to be indexed. How can I get rid of these pages?
Intermediate & Advanced SEO | | bondhoward0 -
Best tips for getting a video page to rank?
We have a video for our company, located here: http://www.imageworkscreative.com/imageworks-creative-video It's an overview of our company and the services we offer. We'd like to get this page ranking, but we haven't had much luck so far. Our Youtube account does better, but I'm looking for some things we can do on or offsite to get this page to rank. Any tips would be appreciated!
Intermediate & Advanced SEO | | ScottImageWorks0 -
Will these 301's get me penalized?
Hey everyone, We're redesigning parts of our site and I have a tricky question that I was hoping to get some sound advice about. We have a blog (magazine) with subcategory pages that are quite thin. We are going to restructure the blog (magazine) and feature different concert and have new subcategories. So we are trying to decide where to redirect the existing subcategory pages, e.g. Entertainment, Music, Sports, etc. www.charged.fm/magazine Our new ticket category pages ( Concert Tickets, NY Yankees Tickets, OKC Thunder Tickets, etc) are going to feature a tab called 'Latest News' where we are thinking of 301 redirecting the old magazine subcategory pages. So Sports News from the blog would 301 to Sports Tickets (# Latest News tab). See screenshot below for example. So my question is: Will this look bad in the eyes of the GOOG? Are these closely related enough to redirect? Are there any blatant pitfalls that I'm not seeing? It seems like a win/win because we are making a rich Performer page with News, Bio, Tickets and Schedule and getting to reallocate the link juice that was being wasted in an pretty much useless page that was allowed to become to powerful. Gotta keep those pages in check! Thoughts appreciated. Luke Cn6HPpH.jpg
Intermediate & Advanced SEO | | keL.A.xT.o0 -
Wrong page getting ranked
Hi all, we have product category pages on our ecommerce web site and we also produce blog content (such as buyers guides, setup guides etc) to help with ranking and give our site some good quality, unique content. However we are sometimes finding that the buyers guide / blog content gets ranked by Google over our product category page. I'm hoping, if I give an example or two, some one smart out there may be able to point me in the right direction as to how we can avoid this and get the product category page ranked instead? You will see from my examples we are linking internally using the keywords from the buyers guides to the product category pages in order to show the most important page to Google for these keywords and are trying to structure the product category pages as well as possible to make it the most optimized page for the term. Example: Keyword "twin dvd player"... product category page: http://www.3wisemonkeys.co.uk/dvd/portable-dvd-player-car/twin-dvd-player/ ... blog page actually getting ranked for this keyword: http://www.3wisemonkeys.co.uk/advice-center/dual-screen-and-twin-dvd-player-explained/ Keyword "site radio".... product category page: http://www.3wisemonkeys.co.uk/audio/radio/site-radio/ .... blog buyer guide page actually getting ranked for keyword: http://www.3wisemonkeys.co.uk/advice-center/Site-radio-buying-guide/ Any help / pointers appreciated. Thanks.
Intermediate & Advanced SEO | | jasef0 -
Will blocking google and SE's from indexing images hurt SEO?
Hi, We have a bit of a problem where on a website we are managing, there are thousands of "Dynamically" re-sized images. These are stressing out the server as on any page there could be upto 100 dynamically re-sized images. Google alone is indexing 50,000 pages a day, so multiply that by the number of images and it is a huge drag on the server. I was wondering if it maybe an idea to blog Robots (in robots.txt) from indexing all the images in the image file, to reduce the server load until we have a proper fix in place. We don't get any real value from having our website images in "Google Images" so I am wondering if this could be a safe way of reducing server load? Are there any other potential SEO issues this could cause?? Thanks
Intermediate & Advanced SEO | | James770