How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Sitemaps: Best Practice
What should and what shouldn't go in the sitemap? In particular, pages like subscribe to our newsletter/ unsubscribe to our newsletter? Is there really any benefit in highlighting those pages to the SEs? Thanks for any advice/ anecdotes 🙂
Intermediate & Advanced SEO | | Fubra0 -
Lower quality new domain link vs higher quality repeat domain link
First time poster here with a dilemma that head scratching and spreadsheets can't solve! I'm trying to work out whether to focus on getting links from new domains or to nurture relationships with the bigger sites in our business and get more links. Of the two links below which does the community here think would be more valuable a signal to Google? Both would be links from within relevant text/post copy. Link 1. Site DA 30. No links currently from this domain. Link 2. Site DA 60. Many links over last 12 months already from this domain. I suspect link 1 but given the enormous disparity in ranking power am I correct?! Thanks for any considered opinions out there! Matthew
Intermediate & Advanced SEO | | mat20150 -
Best Approach to Redirect One Domain to Another
So I'm about to migrate one domain to another. Lets say I'm migrating boo.com to foo.com. Boo.com has good organic traffic & has some really well ranked pages. For this reason (I think) I want to send that traffic to some where other than the foo.com homepage. Perhaps a catered landing page. My question is can I redirect some of the specific pages on boo.com to a landing page on foo.com & then redirect the delta to foo.com's homepage? Or am a risking not fully transferring the full credit of one domain to another if I take that approach & therefore I should just redirect one domain to the other in its entirety? Thanks, Rich
Intermediate & Advanced SEO | | RPD0 -
SEO for a redirected domain name
Our client is a law firm with a name that is challenging to spell. We have procured a domain name for them that is catchy, easy to spell, and plays well into their brand, or at least the current campaign. We're using the campaign domain to direct traffic to their website with a 301 redirect. We have placed the campaign domain in a variety of offline mediums including print and outdoor. The client is currently in the number 1 spot for a good number of our highest priority keywords, so I do not want to do anything to jeopardize that. I'm also not sure this campaign will be their "brand" long-term so I don't want to risk making a switch and making it back. So for now, I'm most comfortable leaving the campaign domain as a redirect to their primary domain. Recently, the client approached me complaining (legitimately) that when people google the campaign domain, they are brought to search results for an entirely different domain because Google "corrects" the domain name for them. This is obviously a bad thing, with many users defaulting to entering urls into Google instead of the address bar. If you tell Google that it was wrong about the autocorrection, our site is in the number 1 position. I liken the situation to Overstock.com using O.co as their offline domain, but overstock.com as their online domain. But imagine if you googled o.co and google brought you to a list of results for "on.co" because it assumed you fat-fingered it. Is there anything I can do to prevent the domain name from getting corrected by Google?
Intermediate & Advanced SEO | | steverobinson0 -
Is a .co.uk domain better?
Hi I have www.example.net however trying to judge if it is better having www.example**.**co.uk instead (as it will be targeted for UK people). I could use Webmaster tool to geographically target UK - however perhaps best to use a .co.uk domain instead. Any views on using a .net ending domain? Many thanks Nigel
Intermediate & Advanced SEO | | Richard5550 -
Purchased a high ranking domain and want to transfer my site to the new domain
I recently purchased a highly ranked domain. The link profile is very good and I don't want to lose to many links however I want to transfer my site to that domain. What is the best way of doing this without losing the current rank on search engines? Also how much does the transfer of registrar information impact my SEO?
Intermediate & Advanced SEO | | ydop0 -
Could Sub domains damage our SEO?
Hi there, We're currently looking into integrating a new internal search function to our site which will involve housing the search results on a sub domain of our site. We have no intention of these search result pages becoming landing pages for organic traffic but would the inclusion of a sub domain affect the optimization of the main domain? i.e. could it effect our authority? Nige
Intermediate & Advanced SEO | | NigelJ0 -
Setting a 404, best practices
Is it enough to just delete a page, or is it necessary to do something else to 404 a page correctly? Is there a great link to explain how to set http status codes?
Intermediate & Advanced SEO | | nicole.healthline0