How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Purchasing an existing domain + redirecting to company's domain
Let's pretend that competitor.com ranks well for certain search terms and generates some traffic from organic search. If a company were to acquire the competitor (or their domain), what's the smartest way to redirect that SEO value to the acquiring company's website? Does a 301 redirect work between different root domains? Even if it does work, is that the smartest approach? Thanks for your help!
Intermediate & Advanced SEO | | Raleigh0 -
Do links to a domain that re-directs to my domain pass link equity?
Hi guys. We've recently taken control of a third-party site and we're going to set up a domain re-direct so any traffic comes to our site. With any existing links that the third-party site has, will these pass link equity to our main site through the redirect? Thanks, Paul
Intermediate & Advanced SEO | | kevinliao0 -
Why domain authority increase
Hi all Our domian authority has increased from 39 to 42 last week. We have been improving our metadata and removing bad backlinks recently. Is there any other reason or updates last week that would have resulted in this increase? Thanks
Intermediate & Advanced SEO | | gavinr
Gavin0 -
Is my domain scorched earth from Penguin?
http://pisoftware.com was never a huge leader of traffic, but it ranked top 5 for my money keyphrases, and was bringing consistent quality visitors. As traction went up, that traffic just became more valuable. I was happy. Then Penguin came along, and made me sad. 60% loss in traffic, I stayed calm. I disavowed. I sent emails asking for links to come down. I atoned for my sins (of the distant, distant past - I know better now) - and waited. Never a hard penalty, never an email from Google - just rankings that got hammered. From #3 for my best keyphrase for #25 today. I write content, and I try and write it better all the time. I try to make it valuable. I leverage social media to the extent that I can. I do outreach. I'm trying to be patient, but it's hard when the software is awesome, and so few people see it. I'm considering starting over - or maybe even just creating another domain to use if this one never comes back. I wonder on the thoughts of experts. At MozCon I talked to a lot of people in the same boat, and it seems we are all taking similar steps. So the questions: 1. Should I start over? Or stay the course? 2. What has worked for others - what seems to have been the most valuable in getting back on the rise? 3. Thoughts on the site as it is now? I've worked lately on speed, mobile rendering, etc - and it seems responsive and solid to me. Thanks in advance, you crazy bunch of Mozzers you. Kelly
Intermediate & Advanced SEO | | Kellster0 -
URL or Domain length
Hi All, I am wondering if google still does give importance to the length of the domain or url. If yes then how much is the acceptable length of a domain and URL. Many Thanks!
Intermediate & Advanced SEO | | HiteshBharucha0 -
What's the best way to hold newly purchased domains over 2 years?
Hi, A friend has just bought 3 domains and is not planning to build websites with them for around 2 years. He asked me what the best thing to do with these domains was...I have 2 ways of look ing at it: a) Putting a holding page on these and submit to Google Webmaster Tools - this way they are indexed by Google and hold search engine trust when the site finally goes up - HOWEVER, if they are not updated with fresh content would that work against them in 2 years time? b) Simply redirect them to their existing site and don't do anything else. Let me know your thoughts. Adido.
Intermediate & Advanced SEO | | Adido-1053990 -
Raise my Domain Authority and MozRank
I have a budget of about $10,000 over the next 2 months and I would like to raise my Domain Authority from 28 to past 50. What should I do with this money? I am looking for suggestions and ideas. If anybody wants to work on this, please PM me. I also need an On Page SEO expert to make corrections and suggestions. I can PM the actual domain to anybody interested.
Intermediate & Advanced SEO | | trickyzoo0 -
Domain certificate
Hello, I would like to know if there is any certificate we can buy to increase the Seo of my website. Thanks very much for your time
Intermediate & Advanced SEO | | Cartageno0