How Best to Handle Inherited 404s on Purchased Domain
-
We purchased a domain from another company and migrated our site over to it very successfully. However, we have one artifact of the original domain in that there was a page that was exploited by other sites on the web. This page allowed you to pass any URL to it and redirect to that URL (e.g. http://example.com/go/to/offsite_link.asp?GoURL=http://badactor.com/explicit_content).
This page does not exist on our site so the results always go to a 404 on our site. However, we find that crawlers are still attempting to access these invalid pages.
We have disavowed as many of the explicit sites as we can, but still some crawlers come looking for those links. We are considering blocking the redirect page in our robots.txt but we are concerned that the links will remain indexed but uncrawlable.
What's the best way to pull these pages from search engines and never have them crawled again?
UPDATE: Clarifying that what we're trying to do it get search engines to just never try to get to these pages. We feel the fact they're even wasting their time on getting a 404 is what we're trying to avoid. Is there any reason we shouldn't just block these in our robots.txt?
-
@gastonriera calm down mate. We have actually tested this at not seen any negative effect on any site we have done it on. It is the "easiest" option, but it won't cause the death and destruction your comment implies. Good day sir.
-
Hi there,
I'm considering that you have over 500k URLs, to be worrying about crawl efficiency. If you have less than that, please don't worry.
Having 404s is completely fine, and google will eventually lower their crawl frequency to those pages.
Blocking them in robots.txt will cause to google stop crawling them, but never to never remove them from the index.
My advice here: don't block them in robots.txtAs Rajesh pointed out, you could force those 404s into 410 to tell Google that they are gone forever. Yet, Google said that they treat 404s and 410s as the same.
John Mueller said over a year ago that 4xx status codes don't incur in crawl wastage. You can check it our in these Webmasters hangout notes - DeepcrawlHope it helps,
Best luck.
Gaston -
FOR THE LOVE OF GOD DONT REDIRECT 404s TO THE HOME!
This is terrible advice. Doing that you'll turn those 404s into soft 404s, making them more problematic than ever.
-
I would actually recommend redirecting it to the homepage. If you have a Wordpress website and a bunch of 404 pages, you can install a free plugin called "All 404 to Homepage" and this will solve the problem. I would, however, recommend that if you have replacement pages or pages covering similar content, that you redirect those to the corresponding replacement page.
-
You need to do one thing with those 404 pages. Move them as 410 status code. Redirection is not good practice for the same.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does redirecting from a "bad" domain "infect" the new domain?
Hi all, So a complicated question that requires a little background. I bought unseenjapan.com to serve as a legitimate news site about a year ago. Social media and content growth has been good. Unfortunately, one thing I didn't realize when I bought this domain was that it used to be a porn site. I've managed to muck out some of the damage already - primarily, I got major vendors like Macafee and OpenDNS to remove the "porn" categorization, which has unblocked the site at most schools & locations w/ public wifi. The sticky bit, however, is Google. Google has the domain filtered under SafeSearch, which means we're losing - and will continue to lose - a ton of organic traffic. I'm trying to figure out how to deal with this, and appeal the decision. Unfortunately, Google's Reconsideration Request form currently doesn't work unless your site has an existing manual action against it (mine does not). I've also heard such requests, even if I did figure out how to make them, often just get ignored for months on end. Now, I have a back up plan. I've registered unseen-japan.com, and I could just move my domain over to the new domain if I can't get this issue resolved. It would allow me to be on a domain with a clean history while not having to change my brand. But if I do that, and I set up 301 redirects from the former domain, will it simply cause the new domain to be perceived as an "adult" domain by Google? I.e., will the former URL's bad reputation carry over to the new one? I haven't made a decision one way or the other yet, so any insights are appreciated.
Intermediate & Advanced SEO | | gaiaslastlaugh0 -
Can’t put a finger on, what is causing 12 year domain, SEO optimized and decent link profile to rank lower than other less superior domains.
Can’t put a finger on, what is causing 12 year domain, SEO optimized and decent link profile to rank lower than other less superior domains. I have dissected the site and link, content, etc profile using ahrefs tools, still no luck, and unfortunately they do not have a community to ask anyone opinion. Hoping someone on Moz will be able to provide me with a secondary opinion or something I obviously missing here. Looking for any constructive feedback/professional opinion with fresh look on what maybe the cause of our down rankings and what may be a cause of it. Any feedback is very much appreciated. Search Term: 3030 aventura condos / One of our link samples (SE Position #6): https://goo.gl/FbYj4V Competing Domains (SE Position #1): https://goo.gl/fLPKX5 Competing Domains (SE Position #2): https://goo.gl/GqXGse
Intermediate & Advanced SEO | | Im_Jake0 -
Page Authority inherited by domain
Hi there, We (blog.fust.ch) created our blog at the same time as (blog.interdiscount.ch). We're both blogs of big brands in switzerland. But there is one big different. The interdiscount blog ranks much better than we do. But they not even optimize the blog for standard SEO points. Now i found the reason i guess. They get the complete page authority of the brand interdiscount.ch inherited. How is that possible, because we don't..? (Screens in attachment) Best regards Sandro K7zxz ndODO
Intermediate & Advanced SEO | | Sandro_Haag0 -
Domains and Domain Authority
Looking for some advice 🙂 I have a domain that has been registered since 1999 and currently hosts my website - the problem is that my business has moved in a different direction and my URL is no longer associated with my main product offering. For example in the past I was xyzgarden.com however now something like xyzhomedecor.com is much more appropriate. How should I handle this so that I am not at a disadvantage for SEO. thanks!
Intermediate & Advanced SEO | | MainstreamMktg0 -
Best way to block a sub-domain from being indexed
Hello, The search engines have indexed a sub-domain I did not want indexed its on old.domain.com and dev.domain.com - I was going to password them but is there a best practice way to block them. My main domain default robots.txt says :- Sitemap: http://www.domain.com/sitemap.xml global User-agent: *
Intermediate & Advanced SEO | | JohnW-UK
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /category//
Disallow: */trackback/
Disallow: */feed/
Disallow: /comments/
Disallow: /?0 -
Renaming your domain from an existing live domain and SEO implications - Please Help *shudder*
Please see the details below. Site A: http://south-african-holiday.mobi is an existing site that is our best site. It is Joomla 3.1 and runs all our ecommerce. Site B: http//www.southerncircle.com/ is our original and has the best DA but is out of date and pretty clunky. joomla 1.5 and all bookings (tour site) are redirected to Site A for processing. Instead of redesigning the Site A I'd like to change the domain name of http://south-african-holiday.mobi -> http://southerncircle.com So far my reading and research (Thanks MOZ for awesome forum!) has provided me with: 1. Do the SEO groundwork. i.e. remove dead links from both sites. Delete useless content and generally tidy up both sites. 2. Map all pages from site a: http://southerncircle.com -> http://south-africa-holiday/ so that the existing pages that have good ranking will have a home on the new site. 3. When ready do a small sample 301 redirect from: http://southerncircle.com to http://south-africa-holiday.mobi. 4. arghhhh now I'm stuck ..... If I redirect to this site then I lose my http://southerncircle.com domain which is what I want to keep....I just want the .mobi site to move to the southerncircle.com site.... I don't consider myself totally thick but this is really confuseing the *$%# out of me PLEASE could you give me some insight here. I'm sure it has been done before without completely losing the sites seo ranking and sending my site into SEO oblivion. If there are any JOOMLA gurus that have done this I'd love to hear from you as well. Many thanks in advance.
Intermediate & Advanced SEO | | SoutherlySwell0 -
How do you build domain trust and authority ?
I am doing some link building and was hoping to get some clear information about the quality of the links and what I need to build a higher domain trust and authority. My sites page I wish to build for has: DA 22 PA 33 MR 4.6 I am looking at back links with these figures Considering outbound links are equal, which pages can help build my page ? page1 DA 66 PA 48 MR 4.8 page2 DA 66 PA 35 MR 3.75 page 3 DA 66 PA 28 MR 3.5 page 4 DA 41 PA 18 MR 2.73 Page 4 DA 41 PA 18 MR 1.87 Do I need back links with a Domain Authority Higher than my own ? PA higher than my own ? or MR higher than my own ? or would any of these pages be able to improve my own ? Thank you in advance for helping with this as I do not want to spend 6 months building the wrong links.
Intermediate & Advanced SEO | | jlane90 -
Keywords in domains losing power
One of my clients received a letter from another seo company touting for business. I would be interested in hearing what your opinion is on this fellow SEO providers. Letter attached - > letter.jpg
Intermediate & Advanced SEO | | onlinemediadirect0