How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 redirecting staff Domain to Company Domain
My colleague owns a domain (A) for about 10 years that he does not use. The domain's content is the same as my company's website (B) content.
Intermediate & Advanced SEO | | khi5
Question: Can I 301 redirect domain A to domain B's homepage or is it better he just closes down his website since this would not be SEO best practices? thank you0 -
Domain Mapping, WordPress MultiSite
Hello, With WordPress MultiSite, does Domain Mapping negatively impact search rankings? I am wondering if the search engines can tell if the Domain is part of a MultiSite Network. Or does it just see the site as a regular website? I understand the issue of IP Address and C Blocks but I'm wondering if the search engines will treat a Mapped Domain Name as it would any other website that is on a shared hosting account. Thanks
Intermediate & Advanced SEO | | bronxpad0 -
Redirection Effects on Sub Domain
Hi, I would try to summarize my query through an example. Lets say site A (www.siteA.com) have two sub domain (subdomain1.siteA.com & subdomain2.siteA.com) and another site B ( www.siteB.com ) have no sub domain. Due to some obvious reason we need re direct the site site A (www.siteA.com) to site B ( www.siteB.com ) and one of the sub domain (subdomain1.siteA.com) to site B (subdomain1.siteB.com). Now the question is that in case of ( subdomain2.siteA.com ) can we keep the sub domain to site A even though site A has been re directed to site B ? Reasons for keeping this can be traffic, earnings etc. Is it possible to keep it like that or provision for further optimization? Plz help.
Intermediate & Advanced SEO | | ITRIX0 -
Brand Name in Domain name
I am thinking to make a site for amazon affiliate ,but hesitate to get a domain name related to brand name like samsungxxxxxxxxx.com .getting such domain name is good or bad ? advice me to proceed
Intermediate & Advanced SEO | | innofidelity0 -
How do I optimise my products for best results?
Hi We have a number of products we want to optimise for example. Barbeque Boss Double Oven Glove, Black When performing keyword research, there are a number of generic terms such as, black oven gloves, barbeque boss and so on. Now i can write the page and optimise for these keyword phrases but I am not sure this is the right way about going for it, particularly if i have several products in the range that are "black oven gloves" or " double oven gloves" How so i best structure my meta tags, meta description and descriptions? Should i just use the product title and optimise around this in the hope Google displays our page in any search queries containing words in this product title? Any advice would be appreciated. Thanks Craig
Intermediate & Advanced SEO | | Towelsrus0 -
How do you build domain trust and authority ?
I am doing some link building and was hoping to get some clear information about the quality of the links and what I need to build a higher domain trust and authority. My sites page I wish to build for has: DA 22 PA 33 MR 4.6 I am looking at back links with these figures Considering outbound links are equal, which pages can help build my page ? page1 DA 66 PA 48 MR 4.8 page2 DA 66 PA 35 MR 3.75 page 3 DA 66 PA 28 MR 3.5 page 4 DA 41 PA 18 MR 2.73 Page 4 DA 41 PA 18 MR 1.87 Do I need back links with a Domain Authority Higher than my own ? PA higher than my own ? or MR higher than my own ? or would any of these pages be able to improve my own ? Thank you in advance for helping with this as I do not want to spend 6 months building the wrong links.
Intermediate & Advanced SEO | | jlane90 -
Affects of vanity domains?
Hi there! My client's domain name is his name, let's say (www,myname.com) because he is well-known in his industry. He is a physician with a very specific specialty and organic competition is fierce for the most relevant keyword to his specialty. A domain has just become available that includes the keyword. If we bought the domain, how could we use it to our advantage? I'm confused about redirecting, etc., with this type of situation. Am i making any sense here? Help! Thanks. 🙂
Intermediate & Advanced SEO | | Mills0 -
How to move website to new domain?
We have a website that has run under the same domain name for the past 10 years. We have built up a decent amount of SEO "mojo" (and traffic) over time, however, the original domain name no longer applies to the business model. A little over 1 year ago we started using a new brand name for the website and created a landing page for that domain name. Everything on that landing page links over to pages on the original domain name (to preserve the SEO value that we have built up over the years). We would like to move all (or most) of the pages/content to the new domain name. Would using 301 redirects be the safest, most effective way of doing this? I have heard of other people doing it this way, and often they will see their traffic drop for a few weeks before it eventually comes back. Anyone else had experience with this? What worked? What didn't? Thanks!
Intermediate & Advanced SEO | | seo-mojo0