Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
About DA of the Website
Hi There, I would like to know why one of my website - https://patnalocalguide.com Domain Authority is not improving. It is not going up and down but it is maintaining its score between 10 since last 5 months. Not going above 10 or below. Why is it so? I'm conitnuously working on creating backlinks for the website and the DR score on Ahref improving but the DA is not. What steps to follow to increase DA? How to find out whats going wrong with our present strategy?
Intermediate & Advanced SEO | | MozByAnuj0 -
Penalty for duplicate content on the same website?
Is it possible to get a penalty for duplicate content on the same website? I have a old custom-built site with a large number of filters that are pre-generated for speed. Basically the only difference is the meta title and H1 tag, with a few text differences here and there. Obviously I could no-follow all the filter links but it would take an enormous amount of work. The site is performing well in the search. I'm trying to decide whether if there is a risk of a penalty, if not I'm loath to do anything in case it causes other issues.
Intermediate & Advanced SEO | | seoman100 -
Check website update frequency?
Is the tools out there that can check our frequently website is updated with new content products? I'm trying to do an SEO analysis between two websites. Thanks in advance Richard
Intermediate & Advanced SEO | | seoman100 -
What is optimal sitemap for large website
My website is having more than 3500 posts. Please let me know what sitemap plugin I need to use for the website and what is the best practice for it?
Intermediate & Advanced SEO | | Michael.Leonard0 -
Merging websites
My company (A) is about the merge with another company (B). The long-term plan is not to keep their brand or website. In terms of the merge process, I have been doing a bit of research and this is how I'm thinking about doing it so far., which is open minded about changing... On the homepage of company B, do a 302 redirect to an information page on the same website which details the merger. This will only be for a year. After a year has passed, do a 301 redirect to the homepage of company A Do 301 redirects from all other pages to similar pages on company A. For pages that don't correspond, either do a 302 to the 'merger detail page', or do a 301 to the homepage of company A. Bring across any content that is effective at driving traffic. Contact all high authority websites that have links to company B and request for them to be updated. Any tips/corrections appreciated. Stu
Intermediate & Advanced SEO | | Stuart260 -
How important is the optional <priority>tag in an XML sitemap of your website? Can this help search engines understand the hierarchy of a website?</priority>
Can the <priority>tag be used to tell search engines the hierarchy of a site or should it be used to let search engines know which priority to we want pages to be indexed in?</priority>
Intermediate & Advanced SEO | | mycity4kids0 -
Are there any SEO Tips before killing a website?
Hey guys, My company acquired another company, and after a couple of months we decided to completely kill their website. I'm not finding any info about SEO best practices for this type of situation. From the "switching domains" and "new sites" articles and blog posts I can extrapolate that I should: 301 redirect their home page to ours Look at specific pages with good authority that relate to our pages and 301 them. Look at the strongest backlinks to their site and try to change them to point to our site. Create a 404 page for the rest of their webpages that tells them that we acquired the company (hopefully with a main menu and search bar) Any other suggestions?
Intermediate & Advanced SEO | | nrv0 -
Domains for regional websites
Please take a look at 7city.com This landing page contains links to: www.7city.co.uk www.7city.ae www.7city.com.sg and our US website which is also www.7city.com It is programmed so: If you are a first time user and type www.7city.com you go to the landing page above. If you then click on AMERICAS, it sets a cookie and directs you to http://www.7city.com/home . When you revisit www.7city.com in the future as the cookie is set you will be automatically sent to the AMERICAS website i.e http://www.7city.com/home. Our US websites is nor performing well on organic ranking compared to other regional website. Is the above technique hindering our organic ranking in the US. Also, I have been led to believe that you get a higher ranking if the domain is specific to a country. Is this true? Does 7city.com receive higher ranking than if I created it as 7city.us for example? Many Thanks Mark
Intermediate & Advanced SEO | | markc-1971830