Sitemaps during a migration - which is the best way of dealing with them?
-
Many SEOs I know simply upload the new sitemap once the new site is launched - some keep the old site's URLs on the new sitemap (for a while) to facilitate the migration - others upload both the old and the new website together, to support the migration. Which is the best way to proceed? Thanks, Luke
-
Very much appreciated CleverPhD!
-
Found this while looking for a answer for another question could not find this the other day- right from the mouth of Google to not include pages that do not exist in XML sitemaps.
http://googlewebmastercentral.blogspot.com/2014/10/best-practices-for-xml-sitemaps-rssatom.html
URLs
URLs in XML sitemaps and RSS/Atom feeds should adhere to the following guidelines:
- Only include URLs that can be fetched by Googlebot. A common mistake is including URLs disallowed by robots.txt — which cannot be fetched by Googlebot, or including URLs of pages that don't exist.
-
Mate nailed it completely!
-
I would say make sure that your new sitemap has all the latest URLs. The reason people say that you should have old URLs in the sitemap is so that Google can quickly crawl the old URLs to find the 301s to the new URLs.
I am not convinced that this helps. Why?
Google already has all your old URLs in its systems. You would be shocked how far back Google has data on your site with old URLs. I have a site that is over 10 years old and I still see URL structures referenced in Google from 7 years ago that have a 301 in place. Why is this?
Google will assume that, "Well, I know that this URL is a 301 or 404, but I am going to crawl it every once in a while just to make sure the webmaster did not do this by mistake." You can notice this in Search Console error or link reports when you setup 301s or 404s, they may stay in there for months and even come back once they fall out of the error list. I had an occurrence where I had some old URLs showing up in the SERPs and various Search Console reports for a site for 2 years following proper 301 setups. Why was this happening?
This is a large site and we still had some old content still linking to the old URLs. The solution was to delete the links in that old content and setup a canonical to self on all the pages to help give a definitive directive to Google. Google then finally replaced the old URLs with the new URLs in the SERPs and in the Search Console reports. The point here being that previously our site was giving signals (links) that told Google that some of the old URLs were still valid and Google was giving us the benefit of the doubt.
If you want to have the new URLs seen by Google, show them in your sitemap. Google already has all the old URLs and will check them and find the 301s and fix everything. I would also recommend the canonical to self on the new pages. Don't give any signals to Google that your old URLs are still valid by linking to them in any way, especially your sitemap. I would even go so far as to reach out to any important sites that link to old URLs to ask for an updated link to your site.
As I mentioned above, I do not think there is an "advantage" of getting the new URLs indexed quicker by putting old URLs in the sitemap that 301 to the new URLs. Just watch your Google Search Console crawl stats. Once you do a major overhaul, you will see Google really crawl your site like crazy and they will update things pretty quick. Putting the old URLs in the sitemap is a conflicting signal in that process and has the potential to slow Google down IMHO.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
XML Sitemap Question!
Hi All, I know that the sitemaps.xml URL must be findable but what about the sitemaps/pageinstructions.xml URL? Can we safely noindex the sitemaps/pageinstructions.xml URL? Thanks! Yael
Intermediate & Advanced SEO | | yaelslater0 -
Sitemaps: Best Practice
What should and what shouldn't go in the sitemap? In particular, pages like subscribe to our newsletter/ unsubscribe to our newsletter? Is there really any benefit in highlighting those pages to the SEs? Thanks for any advice/ anecdotes 🙂
Intermediate & Advanced SEO | | Fubra0 -
Best Sitemap for Large Website
i have more than 3500 pages on my website. Please let me know the best sitemap plugin for my website.
Intermediate & Advanced SEO | | Michael.Leonard1 -
Worth it to redirect .swf during a website migration?
Recently one of our clients switched web providers. We've been implementing 301 redirects to the new URL structure for the most important pages. We noticed upon doing a site search, Google has indexed a .swf URL on page 3 of search results. Please see attached image. 1. Should I implement a redirect for this? There are no links to this filetype.
Intermediate & Advanced SEO | | EEE3
2. There are three others (.swf) that are indexed I would think the best thing to do is to have the new website return a 404 for this page, as it's currently doing, and it will eventually drop out of the index. Any other suggestions? Thanks! g2tur4r0 -
Sitemap for SmartPhone site
Hello I have a smartphone site (e.g.m.abc.com). To my understanding we do not need a mobile sitemap as its not a traditional mobile site. Shall I add those mobile site links in my regular www XML sitemap or not bother to add the links as we already have rel = canonical (on m.abc.com ) and rel= alternate in place (on www site) to respective pages. Please suggests a solution. I really look forward to an answer as I haven't found the "official" answer to this question anywhere.
Intermediate & Advanced SEO | | AdobeVAS0 -
Best way to find broken links on a large site?
I've tried using Xenu, but this is a bit time consuming because it only tells you if the link sin't found & doesn't tell you which pages link to the 404'd page. Webmaster tools seems a bit dated & unreliable. Several of the links it lists as broken aren't. Does anyone have any other suggestions for compiling a list of broken links on a large site>
Intermediate & Advanced SEO | | nicole.healthline1 -
Are there any disadvantages of switching from xml sitemaps to .asp sitemaps in GWT
I have been using multiple xml sitemaps for products for over 6 months and they are indexing well with GMT. I have been having this manually amended when a product becomes obsolete or we no longer stock it. I now have the option to automate the sitemaps from a SQL feed but using .asp sitemaps that I would submit the same way in GWT. I'd like your thoughts on the Pro's and cons of this, pluses for me is realtime updates, con's I percieve GMT to prefer xml files. what do you think?
Intermediate & Advanced SEO | | robertrRSwalters0 -
Major Website Migration Recovery Ideas?
Since starting our business back in 2006 we've gone through alot of branding, and as a result URL and architectual migrations. This has always something that has been driven by usability, brand awareness and technical efficiency reasons, while knowing that there would be SEO hits to take from it....but ultimately hoping to have a much stronger foundation from an SEO perspective in the long run. Having just gone through our most recent (and hopefully final) migration, we are now about 15% down on traffic (although more like 35% - 40% in real terms when seasonality is stripped out). Below is a timeline to our structural history: 2007 - 2009 = We operated as a network of inidividual websites which started as 1, www.marbellainfo.com, but grew to 40, with the likes of www.thealgarveinfo.com, www.mymallorcainfo.com, www.mytenerifeinfo.com, www.mymaltainfo.com etc.. 2009 - 2010 = We decided to consolitdate everything onto 1 single domain, using a sub-domain structure. We used the domain www.mydestinationinfo.com and the subdomains http://marbella.mydestinationinfo.com, http://algarve.mydestinationinfo.com etc.. All old pages were 301 redirected to like for like pages on the new subdomains. We took a 70% drop in traffic and SERPS disappeared for over 6 months. After 9 months we had recovered back to traffic levels and similar rankings to what we had pre-migration. Using this new URL structure, we expanded to 100 destinations and therefore 100 sub-domains. 2011 = In April 2011, having not learnt our lesson from before :(, we undwent another migration. We had secured the domain name www.mydestination.com and had developed a whole new logo and branding. With 100 sub-domains we underwent a migration to the new URL and used a sub-directory folder. So this time www.myalgarveinfo.com had gone to <a></a>http://algarve.mydestinationinfo.com and was now www.mydestination.com/algarve. No content or designs were changed, and again we 301 re-directed pages to like for like pages and with this we even made efforts to ask those linking to us to update their links to use our new URL's. The problem: The situation we fine ourselves in now is no where near as bad as what happend with our migration in 2009/2010, however, we are still down on traffic and SERPS and it's now been 3 months since the migration. One thing we had identified was that our re-directs where going through a chain of re-directs, rather than pointing straight to the final urls (something which has just been rectified). I fear that our constant changing of URL's has meant we have lost out in terms of the passing over of link juice from all the old URL's and loss of trust with Google for changing so much. Throughout this period we have grown the content on our site by almost 2x - 3x each year and now have around 100,000 quality pages of unique content (which is produced by locals on the ground in each destination). I'm hoping that someone in the SEOmoz Community might have some ideas on things we may have slipped up on, or ways in which we can try and recover a little faster and actually get some growth, as opposed to working hard and waiting a while just for another recovery. Thanks Neil
Intermediate & Advanced SEO | | Neil-MyDestination0