How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Domain authority a better metric then referring domain count?
Hi Guys, When reviewing competitors what would be a better metric - Referring domain count OR domain authority. From my understanding DA is a indication of the quality of the link profile. So if a site has a high DA this is a better metric for comparison then referring domain count. What are your thoughts on this? Cheers/
Intermediate & Advanced SEO | | cathywix0 -
Preserve domain on 301 redirect?
We have a domain solely used for print advertising that does a 301 redirect to a landing page (a department home page) on our "real" domain that is indexed on Google. Example: www.bmwrepairs.com redirects to www.repairshop.com/bmwrepairs. Is there a way to do a 301 redirect so that when they get redirected, the URL in the browser address bar remains www.bmwrepairs.com?
Intermediate & Advanced SEO | | Jazee1 -
No Domain Authority after 6 weeks
Hi, We have just launched a new e-commerce site and 301'd a lot of the products and categories to the new site. I have also added a link or 2 from other domains. We launched it on 10th of June - but still using site explorer, there is no domain authority showing - just a 1. Why is that? Any ideas?
Intermediate & Advanced SEO | | bjs20100 -
When should you redirect a domain completely?
We moved a website over to a new domain name. We used 301 redirects to redirect all the pages individually (around 150 redirects). So my question is, when should we just kill the old site completely and just redirect (forward/point) the old domain over to the new one?
Intermediate & Advanced SEO | | co.mc0 -
Impact of Domain Authority on Keyword ranking
Experts Please advise if there is any impact on Keyword rankings due to high or low Domain authority?
Intermediate & Advanced SEO | | INN0 -
Best links to gain?
Hi, Just a quick one to see what peoples thoughts are regarding links. I have just gained a free link on a .gov website in the UK. In one of their offers pages, will this provide any link value or domain trust to me and what can the benefits be SEO wise from having a link on a government domain? The link is just the website url with a a few lines of text detailing our address etc... so its not got any anchor text regarding targetting one of our brands. It is equallly, less or more important to target anchor text links to specific brands or to get good high quality links from trusted sites such as the .gov one I have linking to my root domain? The website is a local council website in the UK. And was listed by a member of their staff, who only list in the offer page if your offering discount to council members etc..., so its not a spammed page or anything like that. What are peoples views on anchor text links vs domain url links? Cheers Will
Intermediate & Advanced SEO | | YNWA0 -
Should we move to a new domain??
Working with a company that is considering moving to a new domain and wanted some advice.They have 2 domains (they own both) and one is a shorter version of the other. The current domain is over 10 years old and has a domain authority score of 80. It is 9 characters long. The new domain (which they also own) has a DA score of 43 and is only 3 characters long. The debate is as much about branding as anything but there is a concern on how it will affect search. I was hoping for so advice and any pitfalls others may have experienced. Thanks!!
Intermediate & Advanced SEO | | 2comarketing0 -
What's the best way to hold newly purchased domains over 2 years?
Hi, A friend has just bought 3 domains and is not planning to build websites with them for around 2 years. He asked me what the best thing to do with these domains was...I have 2 ways of look ing at it: a) Putting a holding page on these and submit to Google Webmaster Tools - this way they are indexed by Google and hold search engine trust when the site finally goes up - HOWEVER, if they are not updated with fresh content would that work against them in 2 years time? b) Simply redirect them to their existing site and don't do anything else. Let me know your thoughts. Adido.
Intermediate & Advanced SEO | | Adido-1053990