Old domain still being crawled despite 301s to new domain
-
Hi there,
We switched from the domain X.com to Y.com in late 2013 and for the most part, the transition was successful. We were able to 301 most of our content over without too much trouble.
But when when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
So, Google has freshly crawled the page. It does know of the 301 to Y and is showing that page's content. But the X.com home page still shows up on site:X.com. How is the domain for X showing rather than Y when even Google's cache is showing the page content and URL for Y?
There are some other similar examples. For instance, you would see a deep URL for X, but just looking at the <title>in the SERP, you can see it has crawled the Y equivalent. Clicking on the link gives you a 301 to the Y equivalent. The cached version of the deep URL to X also shows the content of Y.</p> <p>Any suggestions on how to fix this or if it's a problem. I'm concerned that some SEO equity is still being sequestered in the old domain.</p> <p>Thanks,</p> <p>Stephen</p></title>
-
Hey Stephen,
Did you ever solve this? I'm experiencing the exact same issues you've described above, so I'm curious if you ever figured it out.
Thanks.
-
If using a PHP CMS you will have a page # eg ?P123 this will make it so you can collect all of the old pages and have them redirect to the new ones perfectly.
Turn off all catching frist
A close friend of mine and I did this on his site which is quite large he got 30,000 URLs that were not pointing from his old domain to his new domain simply by adding the code below. If running WordPress and Nginx follow the steps below if you need to modify them so they correspond with your page format use
http://danielmiessler.com/blog/redirect-archives-pages/
Check out Yoast
https://yoast.com/wp-content/permalink-helper.php
&
https://yoast.com/change-wordpress-permalink-structure/
Sorry for being MIA,
Tom
-
Hi Matt, thanks for the reply. I can give this a test.
The puzzling thing here is that Google has re-crawled those old page URLs at the old domain. It is getting 301 re-directed to the new URL. The content that Google has cached for the old URL is from the new URL and has the new URL listed as the source of the cached page.
Stephen
-
Force a recrawl by pinging the pages.
Get as complete a list as possible of the pages that are indexed. Then submit them to PingFarm, 247pinger and RankonTop.
We had 2500+ pages indexed from an old development server and this removed almost all of them in 2-3 weeks.
-
Hi David, you can see my responses to these questions in my reply to Thomas. But the URLs from old X.com are being 301d correctly to y.com. For the vast majority of duplicate listings shown for site:x.com, you can click on them and they get 301d to y.com and the cached version of a page in site:x.com shows y.com content and even the y.com URL in the cached description.
-
I am going to assume the 301 redirects are working when you click on them?
Hi Thomas,
Yes, the 301s work and confirmed with http headers
_You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. _
The migration was before my time, but I think that this was done. If I try to use the change of address on the old site, X.com, I just get a "There is no change of address pending for your site." message with no control options (conversely, if I tried to do the same thing for the current Y.com, it would give me the full list of instructions on how to migrate my site.)
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
The 301s are at a page level at X.com to the same equivalent page at Y.com. The majority of pages have migrated over without obvious problems. But it's a little disturbing to see new pages of y.com which have never been part of x.com somehow make it into a site:x.com query and listed with an x.com domain to start the URL.
So just to recap, site:x.com shows long-tail pages that clearly belong to site:y.com and were never part of x.com. The <title>of some of the site:x.com pages. for instance, are definitely from y.com pages. For some reason, Google is associating these pages with the x.com domain.</p> <p>If you click on the cached version of a listing for site:x.com, the cached version will show the content and URL of Y.com/foo in the cached description header. Clicking on the actual link gets you 301d from X.com/foo to Y.com/foo. Both events indicate that the 301 is working and that Google is recognizing the 301.</p> <p>I don't know if this impacting our SERPs or not. If I do a very page-specific search for "blue widgets A, B, and C in Montana" for a page that is indexed in both site:x.com and site:y.com, you only see y.com's page which is expected behavior. You don't see x.com in the SERPs for that specific query. It's only if you do site:x.com "blue widgets A, B, and C in Montana" do you see the duplicate listing. But again, clicking on the URL that is shown results in a 301 to the proper y.com page.</p> <p>I can dig deeper with my developers, check logs, etc. But it's weird. It's almost like Google sees a URL for y.com and indexes that URL for x.com and y.com even though it knows that y.com is the dominant or real page (evidenced by cached data info). Everytime I click on an site:x.com link, I get 301 redirected properly to the y.com equivalent.</p></title>
-
Did you keep the page URL structure the same on the new site? If so, you can do a simple htaccess rule that will forwards all traffic from one domain, to the corresponding pages on the other domain. If not, then you will have to go through and check each one manually.
Have all the steps been covered? Did you resubmit a new sitemap? Specify in "fetch as Google" that all the new pages on the new site be crawled? If you are using a CMS, did you check to see if multiple versions of a certain page exist, and forget to redirect those?
Lastly, even though you still see references to the old domain, do the 301 redirects work? Once clicked do they send a user to the new domain?
-
I am going to assume the 301 redirects are working when you click on them?
Going on that assumption and please correct me if I am wrong. You need to go into Google Webmaster tools and tell Google that you have moved from domain A to domain B this will make Google bot crawl both sites vigorously looking for 301 redirects. Basically any changes that have occurred when the first crawl is completed after approximately 90 days tell Google to crawl it again. You have a fairly large site based on the amount of links you are discussing. You need to get Google to find everything. That is the best way to tell Google that you are changing domains.
https://support.google.com/webmasters/answer/55281
Two references I would use. Each contains a lot of good information.
https://support.google.com/webmasters/topic/6033102?hl=en&ref_topic=6029673
&
http://moz.com/community/q/how-to-keep-old-url-juice-during-site-switch
"But when I do a site:X.com in Google, I still see about 6240 URLs of X listed. But if you click on a link, you get 301d to Y. Maybe Google has not re-crawled those X pages to know of the 301 to Y, right? The home page of X.com is shown in the site:X.com results. But if I look at the cached version, the cached description will say :This is Google's cache of Y.com. It is a snapshot of the page as it appeared on July 31, 2014."
I want to be certain that you did not 301 redirect domain to domain and not page to page. So if site X has homepage, about page, whatever page, and site Y would contain the exact same pages or equivalent pages that you would have already 301 redirected page 2 page not just point the domain at the other domain is that right?
I hope this helps,
Thomas
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
HTTPS & 301s
Hi We have like most set up a redirect from HTTP to HTTPS. We also changed our website and set up redirects from .ASP pages to PHP pages We are now seeing 2 redirects in place for the whole of the website.
Technical SEO | | Direct_Ram
http.www.domain.com > https.www.domain.com (1) >> oldwebpage.asp >> new webpage.php (2) The question is: Is there anyway of making the redirect 1 and not 2? thanks
Enver0 -
Why does my 301 show the old urls with new descriptions and titles?
Hi all, We've just rebranded. The 301 appears to have worked well and moved the results and rankings onto the new domain. However a site:olddomain.com search in Google brings up about a hundred pages that have the new titles and descriptions but show the old urls - does anyone have any idea how to make the old domain disappear from the SERPS? Many thanks, Richard
Technical SEO | | panini0 -
An EMD with top level domain of another country. Still useful ?
Normally, EMDs (with couple of keywords in it) has more advantage over other domain names ( I understand other factors matters too & I am aware of EMD update). But what if there is an EMD with couple of keywords but top level domain is of another country. Confused? see ex - shoesinsydney.co.uk Will it STILL have a natural advantage over other domains (with no keywords in them)?
Technical SEO | | Personnel_Concept0 -
.ca and. com domains
Hello, currently the main site im working on is a .com, but have the .ca version purchased from register.com. should i have this setup to redirect to the .com site. will google see these as dup content. We have the .ca for our canadian customers but both sites are identical. Thank you
Technical SEO | | TP_Marketing0 -
New Website and Domain Question
Hi all, I am launching a new website around the end of October and I have purchased a great domain to use for it. My question is should I put some kind of holding page up to try and start building up some domain authority in preperation for launch? Or maybe a blog at www.domain.com/blog and then keep all the blog content at the same location when the full site goes up? Or is it best to wait and just launch the site when the first version is complete? Thanks, Ben
Technical SEO | | BenInder0 -
Redirecting a old aged site to a new exact match site?
Hi All, I have a question. I have 2 sites with me in the same sector and want some help. site 1 is a old site started back in 2003 and has some amount of links to it and has a pr 3 with some good links to it but doesn't rank much for any keywords for the timing. site 2 is a aged domain but newly developed with unique content and has a good amount of exact match with a .com version. so will there be any benefit by redirecting site 1 to site 2 to get the seo benefits and a start for link bulding? or is it best to develop and work on each site? the sector is health insurance. Thanks
Technical SEO | | macky71 -
Selecting a new domain name
If a two word domain is already taken (e.g. onetwo.com), which of the following is a better alternative? 1. one-two.com 2. onetwo.org Best,
Technical SEO | | ChristopherGlaeser
Christopher0 -
Too many 301s?
Hi there, If there is a website that has accidently generated say 1,000 pages of duplicate content, would the seo be hurt if all those pages were re-directed to the origional source of the content? There are no plans to re-write the 1,000 duplicate pages, they are already cached and indexed by Google. I thought about canonical tags but as they have some traffic and a little seo value i thought 301 re-direct would be more appropiate to the relevant pages? I am also right in thinking you would be able to remove the 301 in the .htaccess file once the index has updated? Also once removed the 301 - i could use those urls later from scratch if i wanted? Any info much appreciated.
Technical SEO | | pauledwards0