How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Referring domain issues
Our website (blahblah).org has 32 other domains pointing to it all from the same I.P address. These domains including the one in question, were all purchased by the website owner, who has inadvertently created duplicate content and on most of these domains. Some of these referring domains have 301's, some don't - but it appears they have all been de-indexed by Google. I'm somewhat out of my depth here (most of what I've said above has come from an agency who said we should address this before being slapped by Google). However I need to explain to my line manage the actual issues in more detail and the repercussions - any anyone please offer advice please? I'm happy to use the agency, or another - but would like some second opinions if possible?
Intermediate & Advanced SEO | | LJHopkins0 -
Domain Authority
Hi Our website Domain Authority isn't as high and was wondering why it's not increasing. Compared to 1 or 2 competitors we're not scoring as high as them. Are rankings are good for all chosen keywords. Just trying to get a better handle where our site is falling short on.
Intermediate & Advanced SEO | | Cocoonfxmedia1 -
Domain Mapping, WordPress MultiSite
Hello, With WordPress MultiSite, does Domain Mapping negatively impact search rankings? I am wondering if the search engines can tell if the Domain is part of a MultiSite Network. Or does it just see the site as a regular website? I understand the issue of IP Address and C Blocks but I'm wondering if the search engines will treat a Mapped Domain Name as it would any other website that is on a shared hosting account. Thanks
Intermediate & Advanced SEO | | bronxpad0 -
Duplicate Sub-domains Being Indexed
Hi all, I have this site that has a sub-domain that is meant to be a "support" for clients. Some sort of FAQ pages, if you will. A lot of them are dynamic URLs, hence, the title and most of the content are duplicated. Crawl Diagnostics found 52 duplicate content, 138 duplicate title and a lot other errors. My question is, what would be the best practice to fix this issue? Should I noindex and nofollow all of its subdomains? Thanks in advance.
Intermediate & Advanced SEO | | EdwardDennis0 -
How to handle the 301 of a complete domain on URL level
We will be shutting down an old website with many (good) links, since the site has no strategic relevance anymore. We do have many other sites, but none of them has exactly the same content/topic. Nonetheless, I would like to keep the juice and redirect the site to another newer project. However, I want to redirect certain URLs of the old site to probably even different domains, depending on which content matches best with the alternative newer site. Does this make sense? Or would youjust redirect the whole domain to one other domain although they don't really have the same topic And how would you handle the URL redirects if the old site has more than 50k URLs? Because that is the case. Thanks for any advice
Intermediate & Advanced SEO | | Windex0 -
Multi domain redirect to single domain
Hello, all SEOers. Today, I would like to get some ideas about handling multiple domains. I have a client who bought numerous domains under purpose of prevent abuse of their brand name and at the same time for future uses. This client bought more than 100 domains. Some domains are paused, parked, lived and redirected to other site. I don't worry too much of parked domains and paused domains. However, what I am worrying is that there are about 40 different domains are now redirected to single domain and meta refresh was used for redirections. As far as I know, this can raise red flag for Google. I asked clients to clean up unnecessary domains, yet they want to keep them all. So now I have to figure out how to handle all domains which are redirect to single domain. So far, I came up with following ideas. 1. Build gateway page which shows lists of my client sites and redirect all domains to gateway page. 2. Implement robots.txt file to all different domains 3. Delete the redirects and leave it as parked domains. Could anyone can share other ideas in order to handling current status? Please people, share your ideas for me.
Intermediate & Advanced SEO | | Artience0 -
How to handle web server downtime?
We have a client who is taking their web server down Saturday morning from 1am - 7am for planned maintenance. Initially, we thought to have all requests return a 503 (service unavailable) response but the web server itself will be down so we are not able to have it return any response codes. Updating the DNS on the registrar will have too much lag time while it propogates out so we aren't sure exactly how to handle this. I had thought possibly of using a second DNS, or a service like DynDNS but that seems like a large amount of effort to set up just for some planned downtime. I have to imagine that Google understands planned website/server downtime every once in a great while. This client has pretty good rankings for some incredibly competitive terms so we want to do all that we can to make sure those rankings are preserved. What are some other potential solutions? We could totally just be overthinking this but we'd rather be safe than sorry... Thanks in advance!
Intermediate & Advanced SEO | | MichaelWeisbaum0 -
Do any of you regularly use expired domains?
I know there has been discussion on using expired domains in the past. This is not so much a question as to how to do it or whether it works, but rather I would love to see how many of you use this in your backlink strategy. I have a domain in a low to moderately competitive niche that ranks really well, mostly on the power of a couple of expired domains. I bought the domains, created a quick wordpress site and pointed some anchor texted links to the site. It took some time for the expired domains to regain their PR, but when they did, the benefit was great. I'm considering whether I want to do this with another domain of mine. On one hand, it's a relatively inexpensive way to get some good quality anchor texted links. But, on the other hand, something in it feels "immoral" or "sneaky" to me. What do you think?
Intermediate & Advanced SEO | | MarieHaynes0