Domain restructure, sitemaps and indexing
-
I've got a handcoded site with around 1500 unique articles and a handcoded sitemap. Very old school.
The url structure is a bit of a mess, so to make things easier for a developer who'll be making the site database-driven, I thought I'd recategorise the content. Same content, but with new url structure (I thought I'd juice up the urls for SEO purposes while I was at it)
To this end, I took categories like:
/body/amazing-big-shoes/
/style/red-boots/
/technology/cyber-boots/And rehoused all the content like so, doing it all manually with ftp:
/boots/amazing-boots/
/boots/red-boots/
/boots/cyber-boots/I placed 301 redirects in the .htaccess file like so:
redirect 301 /body/amazing-boots/ http://www.site.co.uk/boots/amazing-boots/
(not doing redirects for each article, just for categories which seemed to make the articles redirect nicely.)
Then I went into sitemap.xml and manually overwrote all the entries to reflect the new url structure, but keeping the old dates of the original entries, like so:
<url><loc>http://www.site.co.uk/boots/amazing-boots/index.php</loc>
<lastmod>2008-07-08</lastmod>
<changefreq>monthly</changefreq>
<priority>0.5</priority></url>And resubmitted the sitemap to Google Webmasters.
This was done 4 days ago. Webmaster said that the 1400 of 1500 articles indexed had dropped to 860, and today it's climbed to 939.
Did I adopt correct procedure? Am I going about things the right way? Given a little time, can I expect Google to re-index the new pages nicely?
I appreciate I've made a lot of changes in one fell swoop which could be a bit of a no-no... ?
PS Apologies if this question appears twice on Q&A - hopefully I haven't double-posted
-
If your developer will be making the website dynamic via a system like WordPress there will be automated ways to keep your sitemap up to date every time you publish a new page to your system and then it will even ping the search engines that the sitemap is updated It will be a "set it and forget it" type of thing with sitemaps if you are moving in that direction
Good luck!
-
Oh, no what you did is perfect! I guess I meant the site architecture/navigation, but you answered it in your original post when you said "Same content" so disregard that question. Congrats.
-
Sadly I did change the internal linking structure, so that internal links now point to new urls not the old ones. The good news is that even with changing the internal linking structure, Google seems to be keeping abreast of it all. The number of urls indexed has now jumped - in a day - from 939 to 1024, so good old Google is clearly keeping up with the changes. Looks like my fears were ungrounded. Yay
-
Looks perfect to me too. Did the internal linking structure change at all or is that still the same? If it's all the same you should be right back where you were in no time. And you should see some benefits from having a more common sense, easy to understand URL structure. Cheers!
-
That's fair. I get that you're not recommending it personally - but it does seem popular with consistently good feedback from people, so I'll give it a go
-
Just to clarify, I know the sitemap tool I mentioned is very popular. Many small sites use it because it is online, fast and free. I have used it a few times myself. I can't necessarily say I recommend it because I have never personally purchased the software. I would say that if I was looking to obtain a sitemap for your site, I would start with that tool but may take a look at some others.
-
Thanks Ryan, that's a weight off my mind. I'll definitely take up your advice on the sitemap generator, too. Thanks for the recommendation - I'd seen a few around, but wasn't sure - it's great to be pointed in the right direction!
-
Did I adopt correct procedure? Am I going about things the right way? Given a little time, can I expect Google to re-index the new pages nicely?
I would say Yes to all three, but clarify with details below.
When you submit a sitemap to Google with 1500 pages, there is no guarantee they will index all of your pages. It sounds like you have done a lot of intensive, manual work. Fortunately, you have done things the correct way in properly redirecting each page to the new URL. If Google indexed 1400 pages before, they should index around that same number once again. It may take several weeks depending on a few factors such as your site's Domain Authority, navigation and how many links each page has received.
With respect to the sitemap, I would highly recommend using sitemap generation software. It is simply not reasonable to manually update a sitemap with 1500 entries. I would have updated the lastmod date on the sitemap but it may not make any difference.
A popular sitemap tool: http://www.xml-sitemaps.com/. The free version only generates 500 pages, but for $20 you can buy the full version and automate it.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Goals for Sub Domain
I have created separate account for my sub domain and created goal for that, we have sales force data base to capture the leads.The problem which i'am facing is comparing to the goals and Sales force leads is very much difference.Goals leads are showing huge than sales force leads. 2) When lead is creating from organic source but in goals it's showing referral lead.
Technical SEO | | Anshul.S0 -
Is there a way to index important pages manually or to make sure a certain page will get indexed in a short period of time??
Hi There! The problem I'm having is that certain pages are waiting already three months to be indexed. They even have several backlinks. Is it normal to have to wait more than three months before these pages get an indexation? Is there anything i can do to make sure these page will get an indexation soon? Greetings Bob
Technical SEO | | rijwielcashencarry0400 -
Why are only a few of our pages being indexed
Recently rebuilt a site for an auctioneers, however it has a problem in that none of the lots and auctions are being indexed by Google on the new site, only the pages like About, FAQ, home, contact. Checking WMT shows that Google has crawled all the pages, and I've done a "Fetch as Google" on them and it loads up fine, so there's no crawling issues that is standing out. I've set the "URL Parameters" to no effect too. Also built a sitemap with all the lots in, pushed to Google which then crawled them all (massive spike in Crawl rate for a couple days), and still just indexing a handful of pages. Any clues to look into would be greatly appreciated. https://www.wilkinsons-auctioneers.co.uk/auctions/
Technical SEO | | Blue-shark0 -
Domain forwarding
Hi Is it ok or bad practice to domain forward shorter more memorable snappier domains used for promoting a website to a longer domain where the website actually lives, such as: Promoting in social media profiles, emails and offline literature a domain with forwarding set up like: www.brand.com To the main website: www.brandincludingprimaryproductrelatedkeyword.com And if ok (not bad practice), since its the forwarded domains that are being promoted they are hence the links most likely to be shared on social media and other websites so will they be treated like 301's and 'link building' for those will pretty much equate to link building for the main domain (or not) ? Many Thanks Dan
Technical SEO | | Dan-Lawrence0 -
What is the advantage of using sub domains instead of pages on the root domain?
Have a look at this example http://bannerad.designcrowd.com/ For each category of design, they have a landing page on the sub domain. Wouldn't it be better to have them as part of the same domain? What is the strategy behind using sub domains?
Technical SEO | | designquotes0 -
AJAX and Bing Indexation
Hello. I've been going back and forth with Bing technical support regarding a crawling issue on our website (which I have to say is pretty helpful - you do get a personal, thoughtful response pretty quickly from Bing). Currently our website is set with a java redirect to send users/crawlers to an AJAX version of our website. For example, they come into - mysite.com/category..and get redirected to mysite.com/category#!category. This is to provide an AJAX search overlay which improves UEx. We are finding that Bing gets 'hung up' on these AJAX pages, despite AJAX protocol being in place. They say that if the AJAX redirect is removed, they would index and crawl the non-AJAX url correctly - at which point our indexation would (theoretically) improve. I'm wondering if it's possible (or advisable) to direct the robots to crawl the non-AJAX version, while users get the AJAX version. I'm assuming that it's the classic - the bots want to see exactly what the users see - but I wanted to post here for some feedback. The reality of the situation is the AJAX overlay is in place and our rankings in Bing have plummeted as a result.
Technical SEO | | Blenny0 -
What to do if my site was De-indexed?
Hello fellow SEOs, I have been doing SEO for about a year now, I'm not expert, but I know enough to get the job done. I'm learning everyday about better techniques. So enough about that... Tonight I noticed that my site has, I believe, been de-indexed. Its a fairly new site, as we just launched it a few days ago and I went in and did all the title tags and meta. I still have to go in to do the h1 and h2 tags...plus add some alt tags and anchor text. Well anyways, after a couple of days after the title tags were implemented. I was propagating all over the place. Using my keyword tool here...I was number on the first page in Google for 71 or the 88 keywords. My new site was just indexed yesterday and thats when i noticed all my keywords. Well today I noticed that I am no where to be found, even if i type in my company's name. PLEASE help me out...any advice would be appreciated. Thank you. p.s. could my competitors could have done something to my site? just wondering... The website is www.eggheadconsultants.com
Technical SEO | | Jegghead1 -
Delete old site but redirect domain to a new domain and site
I just have a quick query and I have a feeling about what the answer is so just wanted to see what you guys thought... Basically I am working on a client site. This client has a few other websites that are divisions of their company. However these divisions/websites are no longer used. They are wanting to delete the websites but redirect the domains to their name main website. They believe this will pass on SEO benefits as these old division sites are old and have a good PR and history. I'm unsure for DEFINITE, which way is correct?
Technical SEO | | Weerdboil0