Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Someone redirected his website to ours
Hi all, I have strange issue as someone redirected website http://bukmachers.pl to ours https://legalnibukmacherzy.pl We don't know exactly what to do with it. I checked backlinks and the website had some links which now redirect to us. I also checked this website on wayback machine and back in 2017 this website had some low quality content but in 2018 they made similar redirection to current one but to different website (our competitor). Can such redirection be harmful for us? Should we do something with this or leave it, as google stop encouraging to disavow low quality links.
Intermediate & Advanced SEO | | Kahuna_Charles1 -
Would you redirect Website A to Website B, when Website B is in the middle of a HTTP=>HTTPS migration?
Hey guys, I'm curious on your thoughts around this scenario... Website A: 35,000 monthly pageviews 1,000 pages 375 root linking domains currently HTTPS focused on one topic weak rankings for competitive keywords Website B: 3M monthly pageviews 32,500 pages 3,500 root linking domains started HTTP to HTTPS migration 1 week ago. 1/3 of pages indexed as HTTPS. focused on many topics strong rankings for competitive keywords Requirement: I want to have a reliable read on how Website A's keyword rankings change after redirecting it's pages to Website A. This post-migration analysis will be used as a basis to assess the risk of redirecting another website we own that is similar to Website A into Website B. My question: Would you wait until most of the pages on Website B are indexed as HTTPS before doing a 301 of Website A to Website B? Please back up your answer with reasons why or why not 🙂
Intermediate & Advanced SEO | | jeremycabral0 -
Client rebranded with a new website but can't migrate now defunct franchise website to new website.
Hi everyone, My client is a chain of franchised restaurants with a local domain website named after the franchise. The franchise exited the market while the client stayed and built its own brand with a separate website. The franchise website (which is extremely popular) will be shut down soon but the client will not be able to redirect the franchise website to the new website for legal reasons. What can I do to ensure that we start ranking immediately for the franchise keyphrase as soon as the franchise website is shutdown. We currently have the new website and access to the old website (which we can't redirect) Thanks, T
Intermediate & Advanced SEO | | Tarek_Lel0 -
XML sitemaps questions
Hi All, My developer has asked me some questions that I do not know the answer to. We have both searched for an answer but can't find one.... So, I was hoping that the clever folk on Moz can help!!! Here is couple questions that would be nice to clarify on. What is the actual address/name of file for news xml. Can xml site maps be generated on request? Consider following scenario: spider requests http://mypage.com/sitemap.xml which permanently redirects to extensionless MVC 4 page http://mypage.com/sitemapxml/ . This page generates xml. Thank you, Amelia
Intermediate & Advanced SEO | | CommT0 -
XML Sitemap index within a XML sitemaps index
We have a similar problem to http://www.seomoz.org/q/can-a-xml-sitemap-index-point-to-other-sitemaps-indexes Can a XML sitemap index point to other sitemaps indexes? According to the "Unique Doll Clothing" example on this link, it seems possible http://www.seomoz.org/blog/multiple-xml-sitemaps-increased-indexation-and-traffic Can someone share an XML Sitemap index within a XML sitemaps index example? We are looking for the format to implement the same on our website.
Intermediate & Advanced SEO | | Lakshdeep0 -
Merging two websites to one...
Hi all. Could do with a second opinion on this please... At present a client of ours owns two shops (both doing the same but in towns about 20 miles apart - they sell flooring, but using different names) and has a website for each. The plan is to rebrand both of these stores the same and merge both websites into one. The problem comes that both of the individual websites rank very well in their respective Google Local search results and I fear that killing one of the sites will mean that one store will vanish from the local listings. One domain is a DA 45 and the other a DA 11 so the plan is to use the stronger of the two domains. The question I would like to ponder with people wiser than myself is how can we ensure that the new single domain ranks for both locations in the local? Would the easiest solution be to have pages such as domain.com/store1 and domain.com/store2 with full listings for that store inc name, address, phone number, customer reviews etc? At present the DA 45 domain ranks very well in it's Google local so we need to find a way to change the homepage of that to have both the stores phone numbers but without affecting the local listing. I was considering adding the second phone number as a text based image so that it's visible for people but not for bots Finally, would 301 redirecting the now unused store to domain.com/store2 help with ensuring that we do not lose any local listing for that keyword? If not, are there any suggestions people could offer up Many thanks for any help and sorry for the very long question Carl
Intermediate & Advanced SEO | | GrumpyCarl0 -
XML Sitemaps for Message Boards / Forums - Best Practices?
I'm working with a message board that has been around for 10+ years and never taken SEO best practices into consideration. They recently started seeing mobile URLs show up in regular results, which they don't want. I'm recommending they implement multiple sitemaps to properly indicate to Google how to crawl the site and what to index. I've never dealt with a site this large so I'm not sure best practices. They have a HUGE community and new URLs are created every second. Doing a site: search returns "About 12,100,000" URLs. What are some best practices / the best way to approach sitemaps for a site of this size?
Intermediate & Advanced SEO | | MichaelWeisbaum0 -
Domains for regional websites
Please take a look at 7city.com This landing page contains links to: www.7city.co.uk www.7city.ae www.7city.com.sg and our US website which is also www.7city.com It is programmed so: If you are a first time user and type www.7city.com you go to the landing page above. If you then click on AMERICAS, it sets a cookie and directs you to http://www.7city.com/home . When you revisit www.7city.com in the future as the cookie is set you will be automatically sent to the AMERICAS website i.e http://www.7city.com/home. Our US websites is nor performing well on organic ranking compared to other regional website. Is the above technique hindering our organic ranking in the US. Also, I have been led to believe that you get a higher ranking if the domain is specific to a country. Is this true? Does 7city.com receive higher ranking than if I created it as 7city.us for example? Many Thanks Mark
Intermediate & Advanced SEO | | markc-1971830