How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I am temporarily moving a site to a new domain. Which redirect is best?
A client is having their site redeveloped on a new platform in sections and are moving the sections that are on the new platform to a temporary subdomain until the entire site is migrated. This is happening over the course of 2-3 months. During this time, is it best for the site to use 302 temporary redirects during this time (URL path not changing), or is it best to 301 to the temp. domain, then 301 back to the original once the new platform is completely migrated? Thanks!
Intermediate & Advanced SEO | | Matt3120 -
Temporary Domain Changes
Hi All, Our development team needs to do a temporary site name change from www.sitename.com to new.sitename.com and then wants to return to www.sitename.com. They need to do this for the whole site due to how it's built with single sign on (SSO) and how certain post login pages utilize pre login pages and need to keep people logged in. This process is changing with a CMS upgrade and website and post login pages will be independent of the pre login pages moving forward. My question is what is the best way to manage this transition? Right now it seems like the best solution I've been able to work out with development is to reduce the domain shift down to one week and use 302 Redirects, don't index the new.sitename.com site, and for that week and take my lumps as they come from search. Looking for any other suggestion that may help marketing work with dev without casting blame on any teams for drops in organic traffic.
Intermediate & Advanced SEO | | dapacifi0 -
Change domain.com to www.domain.com - influence on linkbuilding, seo, etc.
Hello, Do you know what can happen when i change domain.com to www.domain.com? Will it have an influence to my link-building portfolio (external links to domain.com), position on google search, etc. Thank you for help.
Intermediate & Advanced SEO | | Reyzer0 -
Five best practice for SEO
Hello Everyone, I am new to SEO and need some help. I have 6 sites and I need to know what are the top 5 strategies for Off page seo. I mean the most important ones. Thanks Abie
Intermediate & Advanced SEO | | signsny0 -
How to know the real history of a domain
Dear mates, I'm looking for register a new domain, and I'm looking for its history on http://web.archive.org/, my question is how to know if this domain has been penalized by Google, banned etc?. Please let me know your ideas. Thank you Claudio
Intermediate & Advanced SEO | | SharewarePros0 -
Moving low ranking domain
I have a website, that I rewrote great content for, but I recently found that there are many, many links going to the subdomain that may be pulling it down. Has anyone had experience taking down a site and then moving the content to a new site? Will it be considered duplicate content if you completely take the old site down and use rel="canonical" on new site pages? I don't want to lose the good content, but I cannot have it on the current URL with all the bad backlinking (it's a complicated situation, as I need to keep those backlinks which are affiliates). Thank you.
Intermediate & Advanced SEO | | RoxBrock0 -
I run an (unusual) clothing company. And I'm about to set up a version of our existing site for kids. Should I use a different domain? Or keep the current root domain?
Hello. I have a burning question which I have been trying to answer for a while. I keep getting conflicting answers and I could really do with your help. I currently run an animal fancy dress (onesie) company in the UK called Kigu through the domain www.kigu.co.uk. We're the exclusive distributor for a supplier of Japanese animal costumes and we've been selling directly through this domain for about 3 years. We rank well across most of our key words and get about 2000 hits each day. We're about to start selling a Kids range - miniature versions of the same costumes. We're planning on doing this through a different domain which is currently live - www.kigu-kids.co.uk. It' been live for about 3-4 weeks. The idea behind keeping them on separate domains is that it is a different target market and we could promote the Kids site separately without having to bring people through the adult site. We want to keep the adult site (or at least the homepage) relatively free from anything kiddy as we promote fancy dress events in nightclubs and at festivals for over 18s (don't worry, nothing kinky) and we wouldn't want to confuse that message. I've since been advised by an expert in the field that that we should set up a redirect from www.kigu-kids.co.uk and house the kids website under www.kigu.co.uk/kids as this will be better from an SEO perspective and if we don't we'll only be competing with ourselves. Are we making a big mistake by not using the same root domain for both thus getting the most of the link juice for the kids site? And if we do decide to switch to have the domain as www.kigu.co.uk/kids, is it a mistake to still promote the www.kigu-kids.co.uk (redirecting) as our domain online? Would these be wasted links? Or would we still see the benefit? Is it better to combine or is two websites better than one? Any help and advice would be much appreciated. Tom.
Intermediate & Advanced SEO | | KIGUCREW0 -
Consolidating 3 regional domains
We recently took the decision to consolidate 3 domains for .com.au, .eu and .us. This decision was made before I arrived here and I'm not sure it's the right call. The proposal is to use a brand new .co (not .com isn't available) domain. The main reason is in terms of trying to build domain strength towards one domain instead or trying to grow 3 domains. We re-sell stock simlar to hotel rooms (different industry) and our site is heavily search based. So duplicate content is an issue that we hope improve on with this approach. One driver was we found for example that our Autralian site was outranking out european site in european searches. We don't want to only hold certain inventory on certain sites either because this doesn't work with our business rules. Anyway if we are to go about this, what would be the best practise in terms of going about this. Should we suddenly just close one of the domain and to a * 301 redirect or should we redirect each page individually? Someone has proposed using robots text to use a phased approach, but to my knowledge this isn't possible with robots.txt, thought a phased individual page 301 using htaccess may be possible? In terms of SEO is 1 domain generally better that 3? Is this a good strategy? What's the best 301 approach? Any other advice? Thanks J
Intermediate & Advanced SEO | | Solas0