Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is possible to submit a XML sitemap to Google without using Google Search Console?
We have a client that will not grant us access to their Google Search Console (don't ask us why). Is there anyway possible to submit a XML sitemap to Google without using GSC? Thanks
Intermediate & Advanced SEO | | RosemaryB0 -
Acquired a company, what should be done with their website?
Hi, I work for a furniture company that essentially purchased another furniture store some time back a few years ago. However, this furniture store that was acquired had a website. The website has no existing pages anymore, only the homepage. The homepage has a message on it describing how it's been taken over and then links to our website. The spam score for the website is 8. I was wondering if there was something else we should do instead of the link, whether that be a straight 301 redirect or if we should have the link at all considering its score. I can provide more information and links if needed. Thanks in advance, Adam
Intermediate & Advanced SEO | | AdamEgarr0 -
Proper sitemap update frequency
I have 12 sitemaps submitted to Google. After about a week, Google is about 50% of the way through crawling each one. In the past week I've created many more pages. Should I wait until Google is 100% complete with my original sitemaps or can I just go ahead and refresh them? When I refresh the original files will have different URLs.
Intermediate & Advanced SEO | | jcgoodrich0 -
Changing the XML Sitemap address
For technical reason we are having to change our XML sitemap URL's from domain.com/sitemap.xml to domain.com/sitemaps/sitemap.xml - What checklist do I need to do to make sure this transition goes smoothly and is there any problems that I might come across?
Intermediate & Advanced SEO | | JohnW-UK0 -
Sitemap Submission
I was wondering if anyone has any insight into Sitemap submission with Google. I submitted a XML Sitemap for my new site at the end of October. Since then GWT says it is pending. l have made a few changes to the site and added some new pages so l decided to submit an updated XML sitemap. This was about a week ago and is also still pending. Does anybody know how long this process should take and if it is the reason why the site hasn't started ranking for any of our targeted search terms as yet? The site is www.theremovalistsguide.com.au
Intermediate & Advanced SEO | | RobSchofield0 -
Adding a Directory to Successful Article Website
We are considering adding roughly 1,300 pages to a 2,300 page website within the drug rehab niche. Our website is generating roughly 10,000 uniques from Search / month. **Is there a way to estimate the change in traffic to the existing content on the site when we add 30-40% pages in the form of a directory? ** **Is there a way to estimate the effect of the existing traffic and links to our newly added part of the site (the directory)? **
Intermediate & Advanced SEO | | alltreatment0 -
Domains for regional websites
Please take a look at 7city.com This landing page contains links to: www.7city.co.uk www.7city.ae www.7city.com.sg and our US website which is also www.7city.com It is programmed so: If you are a first time user and type www.7city.com you go to the landing page above. If you then click on AMERICAS, it sets a cookie and directs you to http://www.7city.com/home . When you revisit www.7city.com in the future as the cookie is set you will be automatically sent to the AMERICAS website i.e http://www.7city.com/home. Our US websites is nor performing well on organic ranking compared to other regional website. Is the above technique hindering our organic ranking in the US. Also, I have been led to believe that you get a higher ranking if the domain is specific to a country. Is this true? Does 7city.com receive higher ranking than if I created it as 7city.us for example? Many Thanks Mark
Intermediate & Advanced SEO | | markc-1971830 -
Should I Combine 30 websites into one?
I have a Private health care company that I have just begun consulting for. Currently in addition to the main website serving the whole group, 30 individual sites which are for each of the hospitals in their group. Each has it's own domain. Each site, has practically identical content: something that will be addressed in my initial audits. But should I suggest that they combine all the sites into one domain, providing individual category pages for each hosptial, or am I really going to suggest that each of the 30 sites, create unique content of their own. This means thirty pages of content on "hip replacements" thirty different versions of "our treatement" etc, and bearing in mind they all run off the same CMS, even with different body text, the pages are going to be practically identical. It's a big call either way! The reason they started out with all these sites, is that each hospital is it's own cost centre and whilst the web development team is a centralized resource. They each have their own sites to try and rank indivdually for local searches, naturally as they will each tend to get customers from their own local area. Not every hospital provides the full range of treatments.
Intermediate & Advanced SEO | | Ultramod0