Xml sitemap advice for website with over 100,000 articles
-
Hi,
I have read numerous articles that support submitting multiple XML sitemaps for websites that have thousands of articles... in our case we have over 100,000. So, I was thinking I should submit one sitemap for each news category.
My question is how many page levels should each sitemap instruct the spiders to go? Would it not be enough to just submit the top level URL for each category and then let the spiders follow the rest of the links organically?
So, if I have 12 categories the total number of URL´s will be 12???
If this is true, how do you suggest handling or home page, where the latest articles are displayed regardless of their category... so I.E. the spiders will find l links to a given article both on the home page and in the category it belongs to. We are using canonical tags.
Thanks,
Jarrett
-
It's really a process of experimenting over time to find out the method that results in the most URLs indexed that in turn brings the most relevant traffic. Personally I wouldn't have one for each category, yet without tests there's no conclusive reasoning either way.
-
Thanks for the tip... I will do that.
I´m still unsure if I really need to submit a sitemap with thousands of URL´s I was thinking I should create an sitemap index file the points to individual top level category sitemaps and leave it at that. If I do this though, I suppose I don´t need individual sitemaps per category as I will just insert the category URL´s in the root sitemap. What do you think?
-
To add to Corey's response, I'll repeat what I just provided another question here on Pro Q&A. Sitemap.xml files can handle a maximum of 50,000 URLs, however I've seen them choke with as few as 10,000. Its important to run them through a tool like tools.pingdom.com to ensure they load within just a couple seconds.
Then submit them through Google/Bing webmaster systems and then see if they succeed in crawling all of them.
-
We break up our sitemap files into several different site maps, and then use a sitemap index file to make sure Google finds them all.
At the bottom of this post they talk about using an index file to combine multiple sitemaps, and they also specifically say it is fine to have one time sensitive site map (ie: front page items) and several other less time sensitive ones (categories in your case).
http://googlewebmastercentral.blogspot.com/2006/10/multiple-sitemaps-in-same-directory.html
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Priority Attribute in XML Sitemaps - Still Valid?
Is the priority value (scale of 0-1) used for each URL in an XML sitemap still a valid way of communicating to search engines which content you (the webmaster) believe is more important relative to other content on your site? I recall hearing that this was no longer used, but can't find a source. If it is no longer used, what are the easiest ways to communicate our preferences to search engines? Specifically, I'm looking to preference the most version version of a product's documentation (version 9) over the previous version (version 8). Thanks!
Intermediate & Advanced SEO | | Allie_Williams0 -
Invest in a Image Sitemap - Yes or No?
Hey Mozers, 2 part question I'm reaching out to see if you all think Image Sitemaps are totally worth it for a big company. I can totally understand its value for a smaller mom & pop company. With a larger company they would have way more products so is it worth it having an image site map? I cant find examples of image sitemaps online. Would you be able to provide a website that is doing it? I can only find video sitemaps.
Intermediate & Advanced SEO | | rpaiva0 -
Backlink for Old publised Article
Hi There, Suppose if a website abc.com published an article in 2014 and that article got indexed after few days as Google normally do. If we give backlink from abc.com article published in 2014 to recently published website, Is there issue if backlink given? or If the abc.com article contented updated first and then backlink is given, that would be the correct way? because updated content will re-index and at the time of re-indexing backlink will be fount by Google bot. Rajiv
Intermediate & Advanced SEO | | gamesecure0 -
Any issue? Redirect 100's of domains into one website's internal pages
Hi all, Imagine if you will I was the owner of many domains, say 100 demographically rich kwd domains & my plan was to redirect these into one website - each into a different relevant subfolder. e.g. www.dewsburytilers..com > www.brandname.com/dewsbury/tilers.html www.hammersmith-tilers.com > www.brandname.com/hammersmith/tilers.html www.tilers-horsforth.com > www.brandname.com/horsforth/tilers.html another hundred or so 301 redirects...the backlinks to these domains were slim but relevant (the majority of the domains do not have any backlinks at all - can anyone see a problem with this practice? If so, what would your recommendations be?
Intermediate & Advanced SEO | | Fergclaw0 -
My websites position has dropped, any ideas why?
Hi, First off im new here, so hello to everyone. Now to the reason why I have joined. I am currently trying to rank for 2 terms: **UK Bank Holidays 2013 (Term 1) **and Bank Holidays 2013 (Term 2) The page which im trying to rank these terms on is: http://www.followuk.co.uk/bank-holidays Now some background history: On the 29th Dec 2013, term 1 was 5th and term 2 was 7th - rankings achieved through guest blogging. Last night I changed the h1 tag from 'Bank Holidays 2013' to 'UK Bank Holidays 2013'. Re-worded the meta description to try and increase the CTR. And removed the term 'Bank Holiday' from the end of each sub-heading - Ex: 'New Year's Day Bank Holiday' to 'New Year's Day' - I did this because I felt it was to much so in total 'Bank Holiday' term had been removed from 5 sub-headings. Ok, so I went into WMT and resubmitted for indexing, over night the page got reindexed - the term 'UK Bank Holidays 2013' stayed at the same position (5) BUT the 'Bank Holidays 2013' term dropped into hell at roughly position 250. I'm thinking of changing everything back and crossing my fingers that term which dropped comes back BUT maybe im being to rash and it might jump back as the page stands. I did a grade test using SEOMOZ and both terms generate a grade of 'A'. Has anyone got any ideas? Sorry if the thread is a bit messy im currently crying all over the keyboard as im typing. Thanks
Intermediate & Advanced SEO | | followuk0 -
Advice on Getting this site ranking?
Hi there I'm looking to optimise this site for SEO -> Gets about 3,000 visits per day but all from branded searches. Gets virtually no 'keyword searches' It's just a landing page at the moment. Would you recommend I integrate a blog with it, so we can start targeting more long tail keywords (free football game etc) Any thoughts/advice appreciated 🙂 Thanks Howard
Intermediate & Advanced SEO | | HowardK0 -
I need some blogging advice please!
My name is Matthew and I am a new PRO member and founder of my own Internet marketing company in KS. So far I love the interaction and tools and functionality of seomoz. I am a true student of seo and love the subject. My dilemma is I know a blog is an important piece of any good seo campaign but I know very little about HOW to blog well......this is my new site and blog page. I only have a couple articles so far but many more planned. http://sawwebmarketing.com/seo-blog/ When I read an article that would be particuarly beneficial for my visitors can i post or share that on MY blog (giving the author the credit of course) without google thinking its duplicate content? is there anything specific I need to do with my blog for google to "see" the new, fresh content that is being added to the site? I have seen "tagged" items at the bottom of some blogs. Is this important? Some blogs will have a word or string of 2-3 words that are a link to a specific website. Does this help me or just them or just people reading the blog? **All I know is articles I write need to be relevant to my site and interesting and ORIGINAL and of benefit to my site visitors. ** Any advice that would help insure my blog articles get me all the juice they can would be GREATLY appreciated! Thank you in advance! Matthew ps - my site only went live a couple days ago so I am still working on a few onpage items but ANY feedback about the site itself would be spectacular! Have a GREAT weekend!
Intermediate & Advanced SEO | | Mrupp440 -
Redirecting my new Website URL to my old Website URL
Hi! OK, I am semi - new to SEO Moz but have been self-teaching for 3 years. However I am stuck.. I have been operating my e-commerce site from www.shopadornonline.com for the past 3 years. I just purchased www.shopadorn.com Right now Shopadorn.com re-directs to www.shopadornonline.com because all my products and links go to shopadornonline.com/productblahblahblah I guess I am stuck. Not sure what to tell my web designer to do? Do I give up on having shopadorn.com OR do I start re-directing customers and doing 301 re-directs? I think from what i have read that it is bad to have traffic going to both shopadorn and shopadornonline as they compete for rankings? Where should I start?
Intermediate & Advanced SEO | | Shopadorn0