Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What are the Best SEO Website which you read daily
Hai Moz memebers, Can you pls suggest me some best seo websites that you people read articles everyday a part from MOZ
Intermediate & Advanced SEO | | SEO_GB1 -
Subdomain optimization - advices
Hi, I need some specific advices on which is the best way to optimize the subdomain of a main domain. Besides meta title, description, etc. Br.
Intermediate & Advanced SEO | | Tormar0 -
Editing A Sitemap
Would there be any positive effect from editing a site map down to a more curated list of pages that perform, or that we hope they begin to perform, in organic search? A site I work with has a sitemap with about 20,000 pages that is automatically created out of a Drupal plugin. Of those pages, only about 10% really produce out of search. There are old sections of the site that are thin, obsolete, discontinued and/or noindexed that are still on the sitemap. For instance, would it focus Google's crawl budget more efficiently or have some other effect? Your thoughts? Thanks! Best... Darcy
Intermediate & Advanced SEO | | 945010 -
Interlinking multiple websites
Most websites (for example) Zalando intern link there other CC tld domains to the root. For example: On http://www.zalando.nl/damesschoenen-pumps/ the links in the footer go to the other CC tld's: http://www.zalando.es , zalando.co.uk etc. Does anyone have experience with the fact if you would interlink to the relevant page on the other CC tld;s.
Intermediate & Advanced SEO | | TjeerdvZ
For example: http://www.zalando.nl/damesschoenen-pumps/ links to http://www.zalando.co.uk/womens-shoes-heels/ in stead of linking to the homepage ?
In theory this would give more relevance intern linking. Looking forward to hear if anyone tried or experienced this and what the results where?0 -
Link from archived article.
A strong news site has an "archived.domainname" folder, where they have older articles listed. I can get a link on a page where there is a 4 year old article, which will be in this archived sub-domain. My questions: Will Google view a link from a 4 year old article as less valuable. Will Google notice the article is 4 years old and find it odd why the page all of a sudden has a link to my site, and thus devalue such link the sub-domain "archived" does that tell Google it is old and a link will be less valuable thank you
Intermediate & Advanced SEO | | knielsen0 -
Should i remove sitemap from the mainsite at a webshop (footer link) and only submit .XML in Webmaster tools?
Case: Webshop with over 2000 products. I want to make a logical sitemap for Google to follow. What is best practice at this field? Should i remove the on-page sitemap there is in html with links (is shown as a footer link called "sitemap") and only have the domain.com/sitemap.xml ? Links for great articles about making sitemaps are appreciated to. The system is Magento, if that changes anything.
Intermediate & Advanced SEO | | Mickelp0 -
Someone To Review My Website
I want someone to review My Website as I'm tired of doing everything but its not getting ranked, one day its in top 50 and on the very next day it jut gets disappear, Website URL: http://goo.gl/duJPf I'm trying to get my inner pages rank in google, http://goo.gl/Ha1Vm http://goo.gl/913DR Can someone please review it and tell me if m doing something wrong or if m missing something! Please suggest Thanks
Intermediate & Advanced SEO | | AnkitRawat0 -
I need some blogging advice please!
My name is Matthew and I am a new PRO member and founder of my own Internet marketing company in KS. So far I love the interaction and tools and functionality of seomoz. I am a true student of seo and love the subject. My dilemma is I know a blog is an important piece of any good seo campaign but I know very little about HOW to blog well......this is my new site and blog page. I only have a couple articles so far but many more planned. http://sawwebmarketing.com/seo-blog/ When I read an article that would be particuarly beneficial for my visitors can i post or share that on MY blog (giving the author the credit of course) without google thinking its duplicate content? is there anything specific I need to do with my blog for google to "see" the new, fresh content that is being added to the site? I have seen "tagged" items at the bottom of some blogs. Is this important? Some blogs will have a word or string of 2-3 words that are a link to a specific website. Does this help me or just them or just people reading the blog? **All I know is articles I write need to be relevant to my site and interesting and ORIGINAL and of benefit to my site visitors. ** Any advice that would help insure my blog articles get me all the juice they can would be GREATLY appreciated! Thank you in advance! Matthew ps - my site only went live a couple days ago so I am still working on a few onpage items but ANY feedback about the site itself would be spectacular! Have a GREAT weekend!
Intermediate & Advanced SEO | | Mrupp440