Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Penalty for duplicate content on the same website?
Is it possible to get a penalty for duplicate content on the same website? I have a old custom-built site with a large number of filters that are pre-generated for speed. Basically the only difference is the meta title and H1 tag, with a few text differences here and there. Obviously I could no-follow all the filter links but it would take an enormous amount of work. The site is performing well in the search. I'm trying to decide whether if there is a risk of a penalty, if not I'm loath to do anything in case it causes other issues.
Intermediate & Advanced SEO | | seoman100 -
Would you redirect Website A to Website B, when Website B is in the middle of a HTTP=>HTTPS migration?
Hey guys, I'm curious on your thoughts around this scenario... Website A: 35,000 monthly pageviews 1,000 pages 375 root linking domains currently HTTPS focused on one topic weak rankings for competitive keywords Website B: 3M monthly pageviews 32,500 pages 3,500 root linking domains started HTTP to HTTPS migration 1 week ago. 1/3 of pages indexed as HTTPS. focused on many topics strong rankings for competitive keywords Requirement: I want to have a reliable read on how Website A's keyword rankings change after redirecting it's pages to Website A. This post-migration analysis will be used as a basis to assess the risk of redirecting another website we own that is similar to Website A into Website B. My question: Would you wait until most of the pages on Website B are indexed as HTTPS before doing a 301 of Website A to Website B? Please back up your answer with reasons why or why not 🙂
Intermediate & Advanced SEO | | jeremycabral0 -
Subdomain optimization - advices
Hi, I need some specific advices on which is the best way to optimize the subdomain of a main domain. Besides meta title, description, etc. Br.
Intermediate & Advanced SEO | | Tormar0 -
Advice on Link Removal Services
Hello everyone, Due to the Penguin update my site unfortunately took a bit of a hit. A little while ago I submitted all of our questionable/bad links to the disavow tool, however I still wante to go back and delete any and all problematic links that are still out there. Ive looked into many services, however I haven't been too impressed. Removeem - The email addresses they provided weren't always valid, and their email tool didn't always deploy correctly - a lot of cross referencing and was not saving me any time. Link Detox - Free trial was a bust. They show you 10 links on the free trial, however for me, 9 of the 10 were all the same. Couldn't get a good feel of their system. Rmoov - Their tool is one where you upload your own links, and they help manage everything, however they DONT allow you to email through their system, so Im not sure how this helps my process if I have to do everthing manaully anyway. A lot of sites I see are also a full service approach that charge you based on how many links they remove, and this can get quite costly. I have also contacted:
Intermediate & Advanced SEO | | Lukin
Link Delete - No reponse from multiple email requests
Linkquidator - No response
Infatex - No response My questions to all of you are: Is there any company out there that you recommend that provide a self service tool [online or desktop driven]? Is this even an avenue I should explore, or should I compile my own list [as 3rd party algorithms are not always accurate] and reach out to sites manually? Is disavowing good enough and Im just spinning my wheeles trying to now get them all removed? Thanks!0 -
Website having same business and IP address
Hi All, How Google will react with the websites having condition as mentioned below: Two websites, Same owner, same business, same IP, Interlinking with each other ? Two websites, Same owner, same business, different IP, Interlinking with each other ? Also please elaborate best practices(Such as IP address, Physical address, look and feel etc.) if someone wants to run same business through more than one website?
Intermediate & Advanced SEO | | RuchiPardal0 -
Does Google index more than three levels down if the XML sitemap is submitted via Google webmaster Tools?
We are building a very big ecommerce site. The site has 1000 products and has many categories/levels. The site is still in construccion so you cannot see it online. My objective is to get Google to rank the products (level 5) Here is an example level 1 - Homepage - http://vulcano.moldear.com.ar/ Level 2 - http://vulcano.moldear.com.ar/piscinas/ Level 3 - http://vulcano.moldear.com.ar/piscinas/electrobombas-para-piscinas/ Level 4 - http://vulcano.moldear.com.ar/piscinas/electrobombas-para-piscinas/autocebantes.html/ Level 5 - Product is on this level - http://vulcano.moldear.com.ar/piscinas/electrobombas-para-piscinas/autocebantes/autocebante-recomendada-para-filtros-vc-10.html Thanks
Intermediate & Advanced SEO | | Carla_Dawson0 -
Ecommerce website consolidation
I have a large ecommerce site and several smaller nitche ecommerce sites. All have the same products, but the smaller sites are loosing traffic. I want to combine all the sites to the larger site so it will be easier to manage, but I don't want to loose any rank on the smaller sites. Example: www.yourpromopeople.com - This is the large site I want to use. www.logocoolies.com www.fourcolormagnets.com - These are a couple of the smaller sites I want to combine with the larger one. Questions: What are the pros and cons in doing this? What would be the best way to do this? Would redirecting the URL's to the larger site's product pages do the trick or is there a better option? Thanks for the help.
Intermediate & Advanced SEO | | JHSpecialty0 -
Redirecting Canonical 301s and Magento Website
I have an issue with a client's website where it has 3700+ pages, but roughly half of them are duplicates. Thankfully, the only difference between the original and the duplictes is the "?print" at the end of each URL (I suppose this is Magento's way of making a printable page version of the same page. I don't know, I didn't build it.) My questions is, how can I get all the pages like this http://www.mycompany.com/blah.html?print to redirect to pages like this... http://www.mycompany.com/blah.html Also, do they NEED to be Canonical, or will a 301 redirect be sufficient. Also, after having done this, if anybody knows, is there a way I can turn that feature off in Magento, because we're expanding our product line, and I don't want to have to keep chasing after these "?print" pages after the fact.
Intermediate & Advanced SEO | | ClifThompson0