Sitemaps during a migration - which is the best way of dealing with them?
-
Many SEOs I know simply upload the new sitemap once the new site is launched - some keep the old site's URLs on the new sitemap (for a while) to facilitate the migration - others upload both the old and the new website together, to support the migration. Which is the best way to proceed? Thanks, Luke
-
Very much appreciated CleverPhD!
-
Found this while looking for a answer for another question could not find this the other day- right from the mouth of Google to not include pages that do not exist in XML sitemaps.
http://googlewebmastercentral.blogspot.com/2014/10/best-practices-for-xml-sitemaps-rssatom.html
URLs
URLs in XML sitemaps and RSS/Atom feeds should adhere to the following guidelines:
- Only include URLs that can be fetched by Googlebot. A common mistake is including URLs disallowed by robots.txt — which cannot be fetched by Googlebot, or including URLs of pages that don't exist.
-
Mate nailed it completely!
-
I would say make sure that your new sitemap has all the latest URLs. The reason people say that you should have old URLs in the sitemap is so that Google can quickly crawl the old URLs to find the 301s to the new URLs.
I am not convinced that this helps. Why?
Google already has all your old URLs in its systems. You would be shocked how far back Google has data on your site with old URLs. I have a site that is over 10 years old and I still see URL structures referenced in Google from 7 years ago that have a 301 in place. Why is this?
Google will assume that, "Well, I know that this URL is a 301 or 404, but I am going to crawl it every once in a while just to make sure the webmaster did not do this by mistake." You can notice this in Search Console error or link reports when you setup 301s or 404s, they may stay in there for months and even come back once they fall out of the error list. I had an occurrence where I had some old URLs showing up in the SERPs and various Search Console reports for a site for 2 years following proper 301 setups. Why was this happening?
This is a large site and we still had some old content still linking to the old URLs. The solution was to delete the links in that old content and setup a canonical to self on all the pages to help give a definitive directive to Google. Google then finally replaced the old URLs with the new URLs in the SERPs and in the Search Console reports. The point here being that previously our site was giving signals (links) that told Google that some of the old URLs were still valid and Google was giving us the benefit of the doubt.
If you want to have the new URLs seen by Google, show them in your sitemap. Google already has all the old URLs and will check them and find the 301s and fix everything. I would also recommend the canonical to self on the new pages. Don't give any signals to Google that your old URLs are still valid by linking to them in any way, especially your sitemap. I would even go so far as to reach out to any important sites that link to old URLs to ask for an updated link to your site.
As I mentioned above, I do not think there is an "advantage" of getting the new URLs indexed quicker by putting old URLs in the sitemap that 301 to the new URLs. Just watch your Google Search Console crawl stats. Once you do a major overhaul, you will see Google really crawl your site like crazy and they will update things pretty quick. Putting the old URLs in the sitemap is a conflicting signal in that process and has the potential to slow Google down IMHO.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Consolidating product pages during website migration
Hello, We are an e-commerce & content site undergoing a website migration and redesign in the coming months. We will be getting an entirely new website. Many of our URLs will be changing: Current URL setup: www.mysite.com/catalog/SKU12345/product-title-here
Intermediate & Advanced SEO | | katelynroberts
Future URL setup: www.mysite.com/catalog/product-title-here So we're aware we will be using plenty of 301 redirects to achieve this. Further to this though, we currently have a product page for each configuration of a product - for example, a single-sided bookmark has its own page and URL, and the double-sided version of the same bookmark has its own page and URL. In our site redesign, we are hoping to consolidate each of these instances into one product page where users can select single or double-sided and the price will update accordingly. The bookmark URLs would then go from:
_www.mysite.com/catalog/SKU12345/bookmark-single-sided _(call this URL A for simplicity)www.mysite.com/catalog/SKU67890/bookmark-double-sided (call this URL B) To (after migrating to the new URL structure for the new site, and the now-consolidated single- & double-sided product pages):
www.mysite.com/catalog/bookmark (call this URL C) What is the best way to make this transition without losing too much of our SEO value? I understand there is nearly always traffic loss with URL changes but I'd like to at least minimize the damage as best I can. We have backlinks and ranks for many product pages so I want to make sure we pass as much of this as we can. (And is this at all further complicated by the fact that URL A & B won't exist on the new site, and URL C doesn't exist on the current site? Does this impact the use of the 301 redirects and if so, how?) Are we better off to approach this page consolidation after the site migration and treat it as a separate project? This is something that is important to our user experience, and is definitely a change we want to make. Any advice is appreciated - thank you! I'm a fairly beginner-intermediate SEO so this is all somewhat new but I want to be able to at least convey some understanding to our developer of what we need to do. I was able to find this discussion (https://moz.com/community/q/merging-pages-and-seo) which describes a similar situation and solutions if we were just consolidating the pages but doesn't quite have the complicating factor of the entire site migration happening at the same time. Thanks so much!0 -
Best SEO Strategy
Hi fellow Mozers: I have a question about strategy. I have a client who is a major real estate developer in our region. They build and sell condominiums and also built and manage several major rental apartments. All rental properties have their own websites and there is also a corporate website, which has been around for many years and has decent domain authority (+/- 40). The original intent of the corporate website was to communicate central brand positioning points, attract investors and offer individual profiles of all major properties. My client is interested in developing an organic search strategy which will reach consumers looking to rent apartments. Typical search strings would include the family whose core string would be 'apartments in Baltimore.' (Currently, the client runs PPC for each one of their properties. This is expensive and highly competitive.) In doing research, we've found that there are two local competitors who are able to break on to Page 1 and appear beside the National 'apartment search guides' who dominate the Page 1 SERPS (like apartments.com). The two local competitors have websites of either the same or lower authority than our client's; one has a better link profile, the other is comparable. Here's our problem: our local competitors only build and manage apartments. So, then, the home pages and all the content of their sites ONLY talk about apartment rental related information. Our client's apartment business is actually larger in scope than either local competitor but is only one of their major real estate verticals. So my question is this: if we want to build out a bunch of content which will rank competitively with our local competition, are we better off creating a new area of the corporate site, creating targeted content and resources appropriate for apartment seekers OR would we be better off creating an entirely new site, just devoted to the same? I'm wondering if a new section will ever rank well against competitors whose root domains actually feature content which is only rental related? Likewise, I'm wondering whether we'd be giving up too much, in terms of authority, by creating an entirely new site? I've also only found examples in the industry where an entirely new site was created, so it makes me question the strategy of building out a rental-specific section of a site which also contains information about their condo business. For instance, the Related Companies are a huge builder in the East; they have a corporate site and a site called https//relatedrentals.com . Any feedback would be greatly appreciated!
Intermediate & Advanced SEO | | Daaveey0 -
Best Way To Go About Fixing "HTML Improvements"
So I have a site and I was creating dynamic pages for a while, what happened was some of them accidentally had lots of similar meta tags and titles. I then changed up my site but left those duplicate tags for a while, not knowing what had happened. Recently I began my SEO campaign once again and noticed that these errors were there. So i did the following. Removed the pages. Removed directories that had these dynamic pages with the remove tool in google webmasters. Blocked google from scanning those pages with the robots.txt. I have verified that the robots.txt works, the pages are longer in google search...however it still shows up in in the html improvements section after a week. (It has updated a few times). So I decided to remove the robots.txt file and now add 301 redirects. Does anyone have any experience with this and am I going about this the right away? Any additional info is greatly appreciated thanks.
Intermediate & Advanced SEO | | tarafaraz0 -
Dealing with Canonical tag in volusion
Hi We have an ecommerce site where we have some returns/scratch /dented products identical to the original one. The onpage content of the damaged/original is pretty much identical with the damaged just having a describing the damage. I had wanted to make a canonical tag on the damaged product to the original so it would not be a problem of duplicate content but as it is a volusion site we dont have that option - it only canonicalizes back to itself! Any ideas what else I can do - cant really change the content much and I dont really want to deindex it so people find it? Thanks!
Intermediate & Advanced SEO | | henya0 -
Best way of connecting with key influencers
Hi, I've identified a list of key influencers within my niche. I now want to connect with them, specifically to ask them to provide regular content for our site (they'll hopefully promote it via their own channels as well as us doing some promotion). The reason I want to do this is because it's easier to move Mohammed than the mountain! - I want to piggy-back on their success! How would you go about doing this? Some of the influencers are follwing us on Twitter and I've had some nice discussions with them over the last few months or so (our Twitter has only been going since about March this year). Would you reach out via Twitter, or call them up? I really don't want to mess this up! Thanks, Amelia
Intermediate & Advanced SEO | | CommT1 -
Recovering from a site migration
Hi. I've been working on http://www.alwayshobbies.com/ for a number of months. All was fine, but then we had a site migration which involved a huge number of redirects. There's been a couple of similar moves in the past. As a result, rankings have plummeted. To resolve this, we're considering letting all the old pages 404 by turning of the redirects, and removing all links to them where we can. Some key pages could have canonicals added, but basically we're looking to purge as much as possible. Does this sound like a reasonable tactic?
Intermediate & Advanced SEO | | neooptic0 -
Canonical URLs and Sitemaps
We are using canonical link tags for product pages in a scenario where the URLs on the site contain category names, and the canonical URL points to a URL which does not contain the category names. So, the product page on the site is like www.example.com/clothes/skirts/skater-skirt-12345, and also like www.example.com/sale/clearance/skater-skirt-12345 in another category. And on both of these pages, the canonical link tag references a 3rd URL like www.example.com/skater-skirt-12345. This 3rd URL, used in the canonical link tag is a valid page, and displays the same content as the other two versions, but there are no actual links to this generic version anywhere on the site (nor external). Questions: 1. Does the generic URL referenced in the canonical link also need to be included as on-page links somewhere in the crawled navigation of the site, or is it okay to be just a valid URL not linked anywhere except for the canonical tags? 2. In our sitemap, is it okay to reference the non-canonical URLs, or does the sitemap have to reference only the canonical URL? In our case, the sitemap points to yet a 3rd variation of the URL, like www.example.com/product.jsp?productID=12345. This page retrieves the same content as the others, and includes a canonical link tag back to www.example.com/skater-skirt-12345. Is this a valid approach, or should we revise the sitemap to point to either the category-specific links or the canonical links?
Intermediate & Advanced SEO | | 379seo0 -
Best tools for exploring links?
and not just every single link, but ones you know that Google is actually indexing. I find seomoz to be super easy, but there is no way to distinguish links that are actually counting "juice", or am i missing something. What about majesticseo - any other similar tools you use when trying to find linking sites that pass juice?
Intermediate & Advanced SEO | | imageworks-2612900