Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is a Dedicated Server a Worthwhile Investment?
Greeting MOZ Community: Our site is hosted on a virtual private server. Apparently there are dozens of other web sites hosted on the same server. The performance is usually pretty fast, with the site downloading in 1-3 seconds. However, a few times per month, the performance slows down, to say 5-6 seconds. Please see the attached image. I suspect this may have something to do with the other web sites on the server. Currently we pay about $60/month. A dedicated server would cost about $120/month. Would site performance be more consistent on a dedicated server? Could we enjoy potential SEO benefits by having our own server, i.e. could we rank slightly higher if the speed was more consistent and the performance slightly faster? Thanks, Alan 6D7kK61
Intermediate & Advanced SEO | | Kingalan10 -
Changing the XML Sitemap address
For technical reason we are having to change our XML sitemap URL's from domain.com/sitemap.xml to domain.com/sitemaps/sitemap.xml - What checklist do I need to do to make sure this transition goes smoothly and is there any problems that I might come across?
Intermediate & Advanced SEO | | JohnW-UK0 -
What are partial urls and why this is causing a sitemap error?
Hi mozzers, I have a client that recorded 7 errors when generating Xml sitemap. One of the errors appear to be coming from partial urls and apparently I would need to exclude them from sitemap. What are they exactly and why would they cause an error in the sitemap. Thanks!
Intermediate & Advanced SEO | | Ideas-Money-Art0 -
Best way to move a page without 301
I have a page that currently ranks high for its term. That page is going away for the main website users, meaning all internal site links pointing to that page are going away and point to a new page. Normally you would just do a 301 redirect to the new URL however the old URL will still need to remain as a landing page since we send paid media traffic to that URL. My question is what is the best way to deal with that? One thought was set up a canonical tag, however my understanding is that the pages need to be identical or very close to the same and the landing page will be light on content and different from the new main page. Not topically different but not identical copy or design, etc.
Intermediate & Advanced SEO | | IrvCo_Interactive0 -
Sitemap Folders on Search Results
Hello! We are managing SEO campaign of a video website. We have an issue about sitemap folders. I have sitemaps like ** /xml/sitemap-name.xml .** But Google is indexing my /xml/ folder and also sitemaps and they appear in search results. If i will add Disallow: /xml/ to my robots.txt and remove /xml/ folder from webmaster tools, Google could see my sitemaps? or it ignores them? Will my site effect negatively after remove /xml/ folder completely from search results? What should i do?
Intermediate & Advanced SEO | | roipublic0 -
Canonical Meta Tag Best Practices
I've noticed that some website owners use canonical tags even when there may be no duplicate issues.For examplewww.examplesite.com has a canonical tag.......rel="canonical" href="http://www.examplesite.com/" />www.examplesite.com/bluewidget has a canonical tag.......rel="canonical" href="http://www.examplesite.com/bluewidget/" />Is this recommended or helpful to do this?
Intermediate & Advanced SEO | | webestate0 -
4 sites on same server with similar theme
Hi, We're building 4 e-commerce stores at the moment which are all very similar. They all sell sofas but to different markets. Product content will be different on each site so the content won't be duplicated but the overall theme of the sites will be the same, all on the same server, being launched at the same time. Should I be concerned that Google may not index all of them? Thanks, Andy
Intermediate & Advanced SEO | | AndyMediaLounge0 -
How long until Sitemap pages index
I recently submitted an XML sitemap on Webmaster tools: http://www.uncommongoods.com/sitemap.xml Once Webmaster tools downloads it, how long do you typically have to wait until the pages index ?
Intermediate & Advanced SEO | | znotes0