Submitting XML Sitemap for large website: how big?
-
Hi there,
I’m currently researching how I can generate an XML sitemap for a large website we run. We think that Google is having problems indexing the URLs based on some of the messages we have been receiving in Webmaster tools, which also shows a large drop in the total number of indexed pages.
Content on this site can be accessed in two ways. On the home page, the content appears as a list of posts. Users can search for previous posts and can search all the way back to the first posts that were submitted.
Posts are also categorised using tags, and these tags can also currently be crawled by search engines. Users can then click on tags to see articles covering similar subjects. A post could have multiple tags (e.g. SEO, inbound marketing, Technical SEO) and so can be reached in multiple ways by users, creating a large number of URLs to index.
Finally, my questions are:
- How big should a sitemap be? What proportion of the URLs of a website should it cover?
- What are the best tools for creating the sitemaps of large websites?
- How often should a sitemap be updated?
Thanks
-
Thanks Matt, that's really useful
-
Yeah, it's better to have one than not - but I have always aimed to make it as complete as I can. Why? I'm not sure - mostly because I figure Google is GREAT at crawling my main structure - it's those far-reaching pages that I'm hoping they find in the sitemap.
-
Thanks for both your replies - I will check out the tools and recommendations you suggested.
I'm sure I remember somewhere reading a recommendation that it was only necessary to submit the basic site structure in a sitemap. It sounds like this is not the case and that a site map should , if possible, be comprehensive.
Would it be better to have a basic sitemap giving the main navigational URLs than having nothing at all?
-
I've created sitemaps with the paid version of Screaming Frog that were almost 80,000 pages. That's what I'd use. No point asking what % unless you can't get it all. If you're crawling Microsoft, break it up. Otherwise, organize it if you can (category sitemap, month by month, something.) or just make one big finger to Google type sitemap. lol
-
Hi!
First off, since your content can be accessed in multiple ways, I'd make sure that you're applying means to indicate duplicate pages as such to search engines. Easy access to great content is fantastic, but you can devaluate your own pages a lot when you're not careful. If you're not using it yet, I recommend implementing the rel="canonical" tag in your website.
To answer your questions:
- It should cover all URLs that want indexed. Ideally, that would be every URL
- I'm not sure what 'the best' tools would be, but I used http://www.xml-sitemaps.com a lot a few years back. Their sitemaps are free up to 500 URLs. There are payment plans for bigger ones.
- I wouldn't update an XML sitemap for every new page you make once a month. Instead, let the search engine find their own way in that case. Should your entire site structure change, an XML sitemap can be a great way to help search engine understand your new site setup better.
I hope this helps!
- It should cover all URLs that want indexed. Ideally, that would be every URL
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Any Tips for Reviving Old Websites?
Hi, I have a series of websites that have been offline for seven years. Do you guys have any tips that might help restore them to their former SERPs glory? Nothing about the sites themselves has changes since they went offline. Same domains, same content, and only a different server. What has changed is the SERPs landscape. I've noticed competitive terms that these sites used to rank on the first page for with far more results now. I have also noticed some terms result in what seems like a thesaurus similar language results from traditionally more authoritative websites instead of the exact phrase searched for. This concerns me because I could see a less relevant page outranking me just because it is on a .gov domain with similar vocabulary even though the result is not what people searching for the term are most likely searching for. The sites have also lost numerous backlinks but still have some really good ones.
Intermediate & Advanced SEO | | CopBlaster.com1 -
Xml sitemap Issue... Xml sitemap generator facilitating only few pages for indexing
Help me I have a website earlier 10,000 WebPages were facilitated in xml sitemap for indexation, but from last few days xml sitemap generator facilitating only 3300 WebPages for indexing. Please help me to resolve the issue. I have checked Google webmaster indexed pages, its showing 8,141. I have tried 2-3 paid tools, but all are facilitating 3300 pages for indexing. I am not getting what is the exact problem, whether the server not allowing or the problem with xml sitemap generator. Please please help me…
Intermediate & Advanced SEO | | udistm0 -
SEO transfer to new website
My website currently has some strong SEO and i will be re-developing the website on a wordpress platform...which will change many of the existing URL. Will this affect the current pages that are well indexed in Google? Does using Wordpress or changing the URL extension (.html to .php) make a difference? If i want to make a clean transition without effecting our existing SEO...what are some essential steps i need to take? Example. Current page is www.mydomain.com/name.html .... and the new URL would be www.mydomain.com/product/name.php Thanks
Intermediate & Advanced SEO | | Souk0 -
Training Website Improvements...
Hi Folks, I'm in the process of going over our corporate website with a view to improving on-page optimisation, layout, design and user experience and I would like your feedback on what you think I should improve or change with respect to SEO. Some of my ideas include: Restructure Home Page to Better Show Our Services Possibly Add a Slider to the Home Page (I know engagement rates with these are generally low) Restructure the Course Pages Completely (https://purplegriffon.com/courses/itil-training/itil-foundation-training/itil-foundation) Restructure the Events Pages Completely (https://purplegriffon.com/event/2028/itil-foundation) Improve & Streamline the Booking Process AJAXIFY the Booking Process Improve Responsive Elements I'm also interested in conducting user testing before I go ahead and make any changes. What are your thoughts? What would you change? Thanks. Gaz
Intermediate & Advanced SEO | | PurpleGriffon0 -
How To Internationalize - Big Question
Hi all, Here is a big question. We have a long-established good content website with a .co.uk domain. The site is UK focussed. However, we are planning a new feature which will be UK and worldwide. So do we: 1. Keep it all on our .co.uk ? 2. Put the non-UK parts on a .com domain ? We don't have any content as such for a separate domain, and are not planning any. But, we are not sure if for example US users would be unimpressed with a UK domain. We could fudge it with "co.uk/us" etc. (Notice how we have not mentioned Google. Fed-up chasing big G the whole time. We just want to concentrate on our users and the service we provide to them. But G remains the elephant crapping in the corner of the room.) Also, we are asking this question before we let our developers and designers get to work. Basically we value Moz community opinions over and above theirs. Realise this is a big question, but you have big brains. Please chip in.
Intermediate & Advanced SEO | | dexm100 -
Unable to Crawl my Website
Hi all, I have a website that I am trying to promote, but tried to add it here in SEOMoz and got the following message: We have detected that the root domain evolving-networks.co.uk does not respond to web requests. Using this domain, we will be unable to crawl your site or present accurate SERP information. Does anyone know why this website cannot be crawled? Please help. Thank you in advance!
Intermediate & Advanced SEO | | LSDigital0 -
Changing website providers
After increasing suffering down time from my current website provider, I am seriously considering finding a new one. My only concern is the effect on SERP. Does anyone have any experience with this and what to do and avoid?
Intermediate & Advanced SEO | | casper4340 -
SEO Ideas for B to B Website?
Hi, I am running SEO for a medium-sized B-to-B website. We sell cloud-based software to other businesses/organizations. Our site currently consists of static product-related pages that already have been highly optimized (correct tags, rich content etc) and a blog that we updated regularly. For 2012, I've been tasked with coming up with ideas to significantly increase SEO. Assuming no budget constrains, what are some other general content ideas / other strategies to increase SEO for a B-to-B website? Some of my current ideas include: starting a keyword-targeted YouTube video series and acquiring more links. What are other tactics that could work for a b-to-b site? Thanks!
Intermediate & Advanced SEO | | mindflash0