Old domain still being crawled despite 301s to new domain
-
Hi there,
We switched from the domain X.com to Y.com in late 2013 and for the most part, the transition was successful. We were able to 301 most of our content over without too much trouble.
But when when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
So, Google has freshly crawled the page. It does know of the 301 to Y and is showing that page's content. But the X.com home page still shows up on site:X.com. How is the domain for X showing rather than Y when even Google's cache is showing the page content and URL for Y?
There are some other similar examples. For instance, you would see a deep URL for X, but just looking at the <title>in the SERP, you can see it has crawled the Y equivalent. Clicking on the link gives you a 301 to the Y equivalent. The cached version of the deep URL to X also shows the content of Y.</p> <p>Any suggestions on how to fix this or if it's a problem. I'm concerned that some SEO equity is still being sequestered in the old domain.</p> <p>Thanks,</p> <p>Stephen</p></title>
-
Hey Stephen,
Did you ever solve this? I'm experiencing the exact same issues you've described above, so I'm curious if you ever figured it out.
Thanks.
-
If using a PHP CMS you will have a page # eg ?P123 this will make it so you can collect all of the old pages and have them redirect to the new ones perfectly.
Turn off all catching frist
A close friend of mine and I did this on his site which is quite large he got 30,000 URLs that were not pointing from his old domain to his new domain simply by adding the code below. If running WordPress and Nginx follow the steps below if you need to modify them so they correspond with your page format use
http://danielmiessler.com/blog/redirect-archives-pages/
Check out Yoast
https://yoast.com/wp-content/permalink-helper.php
&
https://yoast.com/change-wordpress-permalink-structure/
Sorry for being MIA,
Tom
-
Hi Matt, thanks for the reply. I can give this a test.
The puzzling thing here is that Google has re-crawled those old page URLs at the old domain. It is getting 301 re-directed to the new URL. The content that Google has cached for the old URL is from the new URL and has the new URL listed as the source of the cached page.
Stephen
-
Force a recrawl by pinging the pages.
Get as complete a list as possible of the pages that are indexed. Then submit them to PingFarm, 247pinger and RankonTop.
We had 2500+ pages indexed from an old development server and this removed almost all of them in 2-3 weeks.
-
Hi David, you can see my responses to these questions in my reply to Thomas. But the URLs from old X.com are being 301d correctly to y.com. For the vast majority of duplicate listings shown for site:x.com, you can click on them and they get 301d to y.com and the cached version of a page in site:x.com shows y.com content and even the y.com URL in the cached description.
-
I am going to assume the 301 redirects are working when you click on them?
Hi Thomas,
Yes, the 301s work and confirmed with http headers
_You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. _
The migration was before my time, but I think that this was done. If I try to use the change of address on the old site, X.com, I just get a "There is no change of address pending for your site." message with no control options (conversely, if I tried to do the same thing for the current Y.com, it would give me the full list of instructions on how to migrate my site.)
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
The 301s are at a page level at X.com to the same equivalent page at Y.com. The majority of pages have migrated over without obvious problems. But it's a little disturbing to see new pages of y.com which have never been part of x.com somehow make it into a site:x.com query and listed with an x.com domain to start the URL.
So just to recap, site:x.com shows long-tail pages that clearly belong to site:y.com and were never part of x.com. The <title>of some of the site:x.com pages. for instance, are definitely from y.com pages. For some reason, Google is associating these pages with the x.com domain.</p> <p>If you click on the cached version of a listing for site:x.com, the cached version will show the content and URL of Y.com/foo in the cached description header. Clicking on the actual link gets you 301d from X.com/foo to Y.com/foo. Both events indicate that the 301 is working and that Google is recognizing the 301.</p> <p>I don't know if this impacting our SERPs or not. If I do a very page-specific search for "blue widgets A, B, and C in Montana" for a page that is indexed in both site:x.com and site:y.com, you only see y.com's page which is expected behavior. You don't see x.com in the SERPs for that specific query. It's only if you do site:x.com "blue widgets A, B, and C in Montana" do you see the duplicate listing. But again, clicking on the URL that is shown results in a 301 to the proper y.com page.</p> <p>I can dig deeper with my developers, check logs, etc. But it's weird. It's almost like Google sees a URL for y.com and indexes that URL for x.com and y.com even though it knows that y.com is the dominant or real page (evidenced by cached data info). Everytime I click on an site:x.com link, I get 301 redirected properly to the y.com equivalent.</p></title>
-
Did you keep the page URL structure the same on the new site? If so, you can do a simple htaccess rule that will forwards all traffic from one domain, to the corresponding pages on the other domain. If not, then you will have to go through and check each one manually.
Have all the steps been covered? Did you resubmit a new sitemap? Specify in "fetch as Google" that all the new pages on the new site be crawled? If you are using a CMS, did you check to see if multiple versions of a certain page exist, and forget to redirect those?
Lastly, even though you still see references to the old domain, do the 301 redirects work? Once clicked do they send a user to the new domain?
-
I am going to assume the 301 redirects are working when you click on them?
Going on that assumption and please correct me if I am wrong. You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. Basically any changes that have occurred when the first crawl is completed after approximately 90 days tell Google to crawl it again. You have a fairly large site based on the amount of links you are discussing. You need to get Google to find everything. That is the best way to tell Google that you are changing domains.
https://support.google.com/webmasters/answer/55281
Two references I would use. Each contains a lot of good information.
https://support.google.com/webmasters/topic/6033102?hl=en&ref_topic=6029673
&
http://moz.com/community/q/how-to-keep-old-url-juice-during-site-switch
"But when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
I hope this helps,
Thomas
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
301 redirected all my pages to my new domain, now I have a problem with Google Search Console
Hi guys! I bought a new domain name and redirected all my URLs from the old domain to the new one. Everything worked perfectly but now I have a little problem. I want to use the option 'Address Change' in google search console. Step 1 Works (Select new website in the list) Step 2 Works (Confirm that the 301 are working) Step 3 Asks me to Verify the old domain (huh!?) in order to complete the request. Obviously that doesn't work because my 301s WORKS! So if I try to verify the old website by putting a google file in the root of my domain Google tries to access it and it automatically redirects to the new domain. I must be missing something lol help!
Technical SEO | | benoit_20180 -
How to get google to forget my old but still working page and list my new fully optimized page for a keyword?
Hi There! (i am beginner in seo) I have dynamic and static pages on our site. I created a static page for a specific keyword. Fully optimized it, (h1, alt, metas, etc.....maybe too optimized). My problem is that this page is alive for weeks, checked it in GWT and it is in robots.txt, google sees it, and indexed it. BUT whenewer i do a search for that keyword, we still appear with the dynamically created link in the google listings. How could i "redirect" google, if sy make a search for that keyword than shows our optimized page? Is there a tool for that? I cant delete the dynamic page... Any ideas? Thx Andrew
Technical SEO | | Neckermann0 -
Domain vs Sub Domain and Rankings
Hi All Wanting some advice. I have a client which has a number of individual centres that are part of an umbrella organisation. Each individual centre has its own web site and some of these sites have similar (not duplicate content) products and services. Currently the individual centres are sub domains of the umbrella organisation. i.e. Umbrella organisation www.organisation.org.au Individual centres are sub domains i.e. www.centre1.organisation.org.au, www.centre2.organisation.org.au etc. I'm feeling that perhaps this setup might be affecting the rankings of the individual sites because they are sub domains. Would love to hear some thoughts or experience on this and whether its worth going through the process of migrating the individual centre domains. Thanks Ian
Technical SEO | | iragless0 -
Will Redirection of Unnatural Links to the New Domain will work?
Hi All, If a website www.mywebsite.com has received 100 unnatural links and because of that rankings are dropped. I want to follow the below strategy to protect my website from these 100 unnatural links. Please let me know if it will work: Buy a new domain www.newdomain.com with new whois record (Not related to www.mywebsite.com ) Do 301 redirection of all 100 unnatural links (Those are pointing to www.mywebsite.com) to www.newdomain (New domain) Do 302 redirection from www.newdomain (New domain) to www.mywebsite.com (Old Domain) After applying the above strategy, will google still consider those 100 unnatural links for Old domain??
Technical SEO | | RuchiPardal0 -
Is it a good idea to use an old domain name for a new product
Hi guys, I have a domain name XYZ.com which hosts the site of a technology service company as of now. The company however didn't do well and shut down a few years ago. Now, that company wants to launch a new set of technology products and wants to use the same domain name. Is it a good idea. The issues that I can see here are: 1. Google has previous pages indexed 2. There are a couple of subdomains totally irrelevant to the business. like employees.xyz.com there. 3. Can the previous indexing be completely undone. Regards, Mayank
Technical SEO | | mayanksaxena0 -
301'ing domain to an addon domain
My googlefu failed me in finding this... How to 301 a domain to an addon domain? Domain structure is as follows: http://addondomain.maindomain.com/ http://www.maindomain.com/addondomain/ http://www.addondomain.com/ <--(addon domain has its own domain as well) I want main domain to all point to the addon domain like so: http://www.maindomain.com/ --> http://www.addondomain.com/
Technical SEO | | JasonJackson0 -
Selling Domain,,,,, OOOOPPPPPPsssss!
Hey Everyone, I think I may have screwed up big time. I have an old 4 letter domain that I've been sitting on for many years. I had no idea what it was worth and ended up putting it up on ebay. I have since been told that it's quite valuable. I just tried to end the auction and ebay won't let me end it. My thinking is that someone here may want to take advantage of my mistake. Even better, let me know if there are any tricks to ending the auction. The domain is http://proa.com If for some reason it doesn't sell, where is a good place to sell it at a fair price? Have a good one. David
Technical SEO | | dmac0 -
Domain taken. Which is better? Using hypens or longer domain.
I am wanting to set up an e commerce site and the domain name that I want is taken. I am considering using a domain that has the main keyword I want to rank for as the domain. I have heard chatter of google penalizing these types of sites and it seems that it hasn't come about. This is something that I would like to test out. So if "electricscooters.com" is taken, should I use "electric-scooters.com" or "electricscooters4less.com" Just wondering if the hyphenated or the longer domain will rank higher. The site won't be spammy at all, I will carry a few different companies that offer similar products. So for this case, I would only sell scooters from a few different manufacturers. Feedback would be appreciated!
Technical SEO | | Dave_Whitty0