How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to build Domain Authority?
My site: https://www.fishingspots.com.au/ has started to drop Domain Authority in the past weeks, however less quality sites like http://silverstories.com.au/ are rising... I am not sure why? Is there someway I can understand why my site would suddenly start dropping authority?
Intermediate & Advanced SEO | | thinkLukeSEO0 -
New domain purchase 301 and 404 issues. Please help!
We recently purchased www.carwow.com and 301 redirected the site to www.carwow.co.uk (our main domain). The problem is that carwow.com had URLs indexed like www.carwow.com/a-b-c the 301 sends them to carwow.co.uk/a-b-c which obviously doesn't exist so is a 404! What should be done in this situation? Should it be ignored and not re-directed at all, or is there a way to delete/disavow these dead pages? An SEO has advised we redirect all pages to the homepage, but won't that mess up the link profile? Any advice would be great!
Intermediate & Advanced SEO | | JamesPursey0 -
Title Tag Best Practices
In light of all the Google updates in 2013, have you updated/changed your title tag best practices? Is the format of (Keyword | Brand) still working well for your optimization efforts or have you started incorporating an approach similar to this format . (Keyword in a Sentence | Brand) Thanks in advance for your opinions.
Intermediate & Advanced SEO | | SEO5Team0 -
We are moving one website to a different domain and would like to know what is the best way to do it without hurting SEO
The website we want to move, let's say www.olddomain.com has a low quality back links profile, in fact it received a manual notification from google of unnatural links detected; but the home page has a PR 3. We want to move it to a different domain let's say www.newdomain.com. We would like to know if it's better to do a 301 redirect to the new domain, in order to transfer the link juice or if it would be better to do a 302, taking into account that this redirect won't pass any link juice, so it would be like start from scratch with this new domain. Thanks for your help.
Intermediate & Advanced SEO | | DoitWiser0 -
Should I Roll Back Domain Change?
A couple years ago I changed domain names and switched platforms for my site. The traffic dropped dramatically (80-90%). I've tried to get inbound links changed, clean up on-page stuff, but nothing is making a big change. I think most of the problem is loss of link juice with the 301 redirects from the old domain to the new one. Would I be risking bigger losses by switching back to the old domain name?
Intermediate & Advanced SEO | | iJeep0 -
Redirecting a Page from Domain A to Domain B
We have a page on Domain A, an established and well-ranking website, that would be more appropriate on Domain B, a site that we launched about two years ago. This page ranks well, pulls nice search traffic and has traffic from external links. We would like to move the page and its traffic from Domain A to Domain B using a 301 redirect. Have you ever done this or have you heard of how it has worked for someone else? Thanks!
Intermediate & Advanced SEO | | EGOL0 -
Domain authority enhancement
Dear all, what is the best way to increase your domain authority in a couple of weeks? Thanks for your reply! Best regards, Ben
Intermediate & Advanced SEO | | HMK-NL0 -
Are sub domains considered completely different than the root domain?
We have a project that is going to generate duplicate content. If we move the new content to a sub-domain (E.g. product.domain.com) will it still be considered duplicate content to the root domain? Or is it like having two completely different domains? Thanks!
Intermediate & Advanced SEO | | tripled5110