Domain restructure, sitemaps and indexing
-
I've got a handcoded site with around 1500 unique articles and a handcoded sitemap. Very old school.
The url structure is a bit of a mess, so to make things easier for a developer who'll be making the site database-driven, I thought I'd recategorise the content. Same content, but with new url structure (I thought I'd juice up the urls for SEO purposes while I was at it)
To this end, I took categories like:
/body/amazing-big-shoes/
/style/red-boots/
/technology/cyber-boots/And rehoused all the content like so, doing it all manually with ftp:
/boots/amazing-boots/
/boots/red-boots/
/boots/cyber-boots/I placed 301 redirects in the .htaccess file like so:
redirect 301 /body/amazing-boots/ http://www.site.co.uk/boots/amazing-boots/
(not doing redirects for each article, just for categories which seemed to make the articles redirect nicely.)
Then I went into sitemap.xml and manually overwrote all the entries to reflect the new url structure, but keeping the old dates of the original entries, like so:
<url><loc>http://www.site.co.uk/boots/amazing-boots/index.php</loc>
<lastmod>2008-07-08</lastmod>
<changefreq>monthly</changefreq>
<priority>0.5</priority></url>And resubmitted the sitemap to Google Webmasters.
This was done 4 days ago. Webmaster said that the 1400 of 1500 articles indexed had dropped to 860, and today it's climbed to 939.
Did I adopt correct procedure? Am I going about things the right way? Given a little time, can I expect Google to re-index the new pages nicely?
I appreciate I've made a lot of changes in one fell swoop which could be a bit of a no-no... ?
PS Apologies if this question appears twice on Q&A - hopefully I haven't double-posted
-
If your developer will be making the website dynamic via a system like WordPress there will be automated ways to keep your sitemap up to date every time you publish a new page to your system and then it will even ping the search engines that the sitemap is updated It will be a "set it and forget it" type of thing with sitemaps if you are moving in that direction
Good luck!
-
Oh, no what you did is perfect! I guess I meant the site architecture/navigation, but you answered it in your original post when you said "Same content" so disregard that question. Congrats.
-
Sadly I did change the internal linking structure, so that internal links now point to new urls not the old ones. The good news is that even with changing the internal linking structure, Google seems to be keeping abreast of it all. The number of urls indexed has now jumped - in a day - from 939 to 1024, so good old Google is clearly keeping up with the changes. Looks like my fears were ungrounded. Yay
-
Looks perfect to me too. Did the internal linking structure change at all or is that still the same? If it's all the same you should be right back where you were in no time. And you should see some benefits from having a more common sense, easy to understand URL structure. Cheers!
-
That's fair. I get that you're not recommending it personally - but it does seem popular with consistently good feedback from people, so I'll give it a go
-
Just to clarify, I know the sitemap tool I mentioned is very popular. Many small sites use it because it is online, fast and free. I have used it a few times myself. I can't necessarily say I recommend it because I have never personally purchased the software. I would say that if I was looking to obtain a sitemap for your site, I would start with that tool but may take a look at some others.
-
Thanks Ryan, that's a weight off my mind. I'll definitely take up your advice on the sitemap generator, too. Thanks for the recommendation - I'd seen a few around, but wasn't sure - it's great to be pointed in the right direction!
-
Did I adopt correct procedure? Am I going about things the right way? Given a little time, can I expect Google to re-index the new pages nicely?
I would say Yes to all three, but clarify with details below.
When you submit a sitemap to Google with 1500 pages, there is no guarantee they will index all of your pages. It sounds like you have done a lot of intensive, manual work. Fortunately, you have done things the correct way in properly redirecting each page to the new URL. If Google indexed 1400 pages before, they should index around that same number once again. It may take several weeks depending on a few factors such as your site's Domain Authority, navigation and how many links each page has received.
With respect to the sitemap, I would highly recommend using sitemap generation software. It is simply not reasonable to manually update a sitemap with 1500 entries. I would have updated the lastmod date on the sitemap but it may not make any difference.
A popular sitemap tool: http://www.xml-sitemaps.com/. The free version only generates 500 pages, but for $20 you can buy the full version and automate it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
New SEO manager needs help! Currently only about 15% of our live sitemap (~4 million url e-commerce site) is actually indexed in Google. What are best practices sitemaps for big sites with a lot of changing content?
In Google Search console 4,218,017 URLs submitted 402,035 URLs indexed what is the best way to troubleshoot? What is best guidance for sitemap indexation of large sites with a lot of changing content? view?usp=sharing
Technical SEO | | Hamish_TM1 -
Will doing a 301 redirect for one domain to another give the latter domain the formers links?
I have some websites that I built a few years ago that are still in existence, but I no longer have access to the sites as they weren't hosted by myself. These sites all carry a "Designed by Me" text on the footer with a link to my (now old) website. I have since done 301 redirects on the domain names that are used in the footers of these sites so they link directly to my new site. However, will these websites now show up on Google Webmasters for example as external links to my site?
Technical SEO | | mickburkesnr0 -
Search results indexed
Hi there, is is bad practice in seo to have search results for products indexed? For example a search result of holidays to Ibiza, with lots of deals coming up? its a search query url that would be indexed, with just an image and price per product on the page, with about 10 per page? Any advice appreciated.
Technical SEO | | pauledwards0 -
Do we need to manually submit a sitemap every time, or can we host it on our site as /sitemap and Google will see & crawl it?
I realized we don't have a sitemap in place, so we're going to get one built. Once we do, I'll submit it manually to Google via Webmaster tools. However, we have a very dynamic site with content constantly being added. Will I need to keep manually re-submitting the sitemap to Google? Or could we have the continually updating sitemap live on our site at /sitemap and the crawlers will just pick it up from there? I noticed this is what SEOmoz does at http://www.seomoz.org/sitemap.
Technical SEO | | askotzko0 -
Changing preferred domain
My company has an international website, and because of a technical issue visitors in one of our main countries cannot visits the "www" version of our site. Currently, the www version is our preferred domain - and the non www redirects to that page. To solve this problem, I was thinking of proposing the following and would greatly appreciate any feedback! (Note: If you answered my www vs. non www question, thanks - this is a follow up) 1. Set non www site as the preferred version 2. Redirect from www to non www 3. Contact our current links and ask them to change to without “www” 4. Change canonical URLs to without “www”
Technical SEO | | theLotter0 -
Partial mobile sitemap
Hi, We have a main www website with a standard sitemap. We also have a m. site for mobile content (but m. is only for our top pages and doesn't include the entire site). If a mobile client accesses one of our www pages we redirect to the m. page. If we don't have a m. version we keep them on the www site. Currently we block robots from the mobile site. Since our m. site only contains the top pages, I'm trying to determine the boost we might get from creating a mobile sitemap. I don't want to create the "partial" mobile sitemap and somehow have it hurt our traffic. Here is my plan update m. pages to point rel canonical to appropriate www page (makes sure we don't dilute SEO across m. and www.) create mobile sitemap and allow all robots to access site. Our www pages already rank fairly highly so just want to verify if there are any concerns since m. is not a complete version of www?
Technical SEO | | NicB10 -
Sitemaps - Format Issue
Hi, I have a little issue with a client site whose programmer seems kind of unwilling to change things that he has been doing a long time. So, he has had this dynamic site set up for a few years and active in google webmaster tools and others, but is not happy with the traffic it is getting. When I looked at webmaster tools I see that he has a sitemap registered, but it is /sitemap.php When I said that we should be offering the SE's /sitemap.xml his response is that sitemap.php checks the site every day and generates /sitemap.xml, but there is no /sitemap.xml registered in webmaster tools. My gut is telling me that he should just register /sitemap.xml in webmaster tools, but it is a hard sell 🙂 Anyone have any definitive experience of people doing this before and whether it is an issue? My feeling is that it doesn't need to be rocket science... Any input appreciated, Sha
Technical SEO | | ShaMenz0 -
Exact match domains
I know buying new websites hoping to 301 redirect them to achieve higher rankings before they’re established is not a good idea, but what about uploading a info pages + on-page SEO + some link-building and when the site is established (it ranks) direct visitors to a landing page? Buying a new domain for the SINGLE purpose of 301 redirecting won’t boost the rankings (don't think it would get penalties either, unless it’s the only link building activity). Not planning to redirect to the homepage, but to the related sub page on our main site (specific landing page). Will this pass the appropriate anchor text and link authority to the right page on our website and help those pages to rank for their keywords? Only thinking of a few pages (no more than 3).
Technical SEO | | Diana.varbanescu0