Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Invest in a Image Sitemap - Yes or No?
Hey Mozers, 2 part question I'm reaching out to see if you all think Image Sitemaps are totally worth it for a big company. I can totally understand its value for a smaller mom & pop company. With a larger company they would have way more products so is it worth it having an image site map? I cant find examples of image sitemaps online. Would you be able to provide a website that is doing it? I can only find video sitemaps.
Intermediate & Advanced SEO | | rpaiva0 -
Changing domains - best process to use?
I am about to move my Thailand-focused travel website into a new, broader Asia-focused travel website. The Thailand site has had a sad history with Google (algorithmic, not penalties) so I don't want that history to carry over into the new site. At the same time though, I want to capture the traffic that Google is sending me right now and I would like my search positions on Bing and Yahoo to carry through if possible. Is there a way to make all that happen? At the moment I have migrated all the posts over to the new domain but I have it blocked to search engines. I am about to start redirecting post for post using meta-refresh redirects with a no-follow for safety. But at the point where I open the new site up to indexing, should I at the same time block the old site from being indexed to prevent duplicate content penalties? Also, is there a method I can use to selectively 301 redirect posts only if the referrer is Bing or Yahoo, but not Google, before the meta-refresh fires? Or alternatively, a way to meta-refresh redirect if the referrer is Google but 301 redirect otherwise? Or is there a way to "noindex, nofollow" the redirect only if the referrer is Google? Is there a danger of being penalised for doing any of these things? Late Edit: It occurs to me that if my penalties are algorithmic (e.g. due to bad backlinks), does 301 redirection even carry that issue through to the new website? Or is it left behind on the old site?
Intermediate & Advanced SEO | | Gavin.Atkinson0 -
What are the best ways to fix 404 errors?
I recently changed the url of my main blog and now have about 100 404 errors. I did a redirect from the old url to the new one however still have errors. 1. Should I do a 301 redirect from each old blog post url to the new blog post url? 2. Should I just delete the old blog post (url) and rewrite the blog post? I"m not concerned about links to the old posts as a lot of them do not have many links.
Intermediate & Advanced SEO | | webestate0 -
Should I create a separate sitemap.xml for paginated categories?
For example: http://www.site.com/category/sub-category http://www.site.com/category/sub-category/1 http://www.site.com/category/sub-category/2 http://www.site.com/category/sub-category/3 Thanks in advance! 🙂
Intermediate & Advanced SEO | | esiow20130 -
Best Practices for Pagination on E-commerce Site
One of my e-commerce clients has a script enabled on their category pages that allows more products to automatically be displayed as you scroll down. They use this instead of page 1, 2, and a view all. I'm trying to decide if I want to insist that they change back to the traditional method of multiple pages with a view all button, and then implement rel="next", rel="prev", etc. I think the current auto method is disorienting for the user, but I can't figure out if it's the same for the spiders. Does anyone have any experience with this, or thoughts? Thanks!
Intermediate & Advanced SEO | | smallbox0 -
Website Siloing..best practice?
Hi all I am doing some research this week on the effects of siloing a Magento site. We have about 1,654 pages with approx 1,400 products. We want to silo the website in order to address the internal linking issues and to also focus the customer journey in a more organised way. I need to report all of the possible angles and effects that this will have on the site, prior to implementing it. Does anyone have info on best practice for siloing? I'd appreciate any help... Thanks Nick
Intermediate & Advanced SEO | | Total_Displays0 -
High Ranking Site with tons of junk on the server
Hey All, So to make a long story short, we own a site that has been passed through many hand and many strategies. We are in the financial field and rank high for many relevant search terms. My job is now to audit/optimize and purge out site of the garbage that has collected over the years (since 2002). During the audit I have found many issues, fized them, but I am not sure own how to proceed with the follwing issues. Any advice to solve the following would be greatly appreciated! 9932 orphan files - does just removing them affect my SEO.. I like a clean house, can I somehow use them to my benefit? Hundreds of 404s with many external "follow" links that we are no longer getting juice from 8 Sitelinks in webmaster tools, but only 4 show in our search I am straight n00b so sorry if this is 101 for anyone you but your input would be greatly appreciated!! Thanks!
Intermediate & Advanced SEO | | deuce1s0 -
Best Service for optimizing google product feed?
We're looking for a company that can help us optimize our google product feed. Does anyone have any recommendations or suggestions? Thanks!
Intermediate & Advanced SEO | | eric_since1910.com0