Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
For a sitemap.html page, does the URL slug have to be /sitemap?
Also, do you have to have anchors in your sitemap.html? or are naked URLs that link okay?
Intermediate & Advanced SEO | | imjonny1230 -
What would the Impact of having a sitemap be?
Hi, some more general question here: How important would you rate it to have a sitemap? Would you rate it fundamentally important or just something you can add as bonus? Thanks in advance
Intermediate & Advanced SEO | | brainfruit0 -
Best SEO url woocommerce, what to do?
Hi! Today we have our product categories indexed (by misstake) and for one of our desired keywords, a category have the nr 1 rank. By misstake, we didnt set nofollow, noindex on our categories, just tags, archives etc. We are now migrating to from Ithemes Exchange to Woocommerce and ime looking on improving our SEO urls for the categories. For keyword "Key1" we rank with this url: http://site/product-category/Key1. The seo meta title and description where untouched when we launched the site last spring so it doesnt look so good.. The plan is to stripe out product-category and instead ad some description ( i have a newly written text of 95 words, 519 letters without space with they keyword precent 5 times in a natural way ) to that particular category and have the url as following: http://site/key1 and then have a 301 redirect for the old http://site/product-category/Key1. What do you think of this? What shall i consider? on the right track? Grateful for any help! // Jonas
Intermediate & Advanced SEO | | knubbz0 -
XML sitemaps questions
Hi All, My developer has asked me some questions that I do not know the answer to. We have both searched for an answer but can't find one.... So, I was hoping that the clever folk on Moz can help!!! Here is couple questions that would be nice to clarify on. What is the actual address/name of file for news xml. Can xml site maps be generated on request? Consider following scenario: spider requests http://mypage.com/sitemap.xml which permanently redirects to extensionless MVC 4 page http://mypage.com/sitemapxml/ . This page generates xml. Thank you, Amelia
Intermediate & Advanced SEO | | CommT0 -
What are the best ways to fix 404 errors?
I recently changed the url of my main blog and now have about 100 404 errors. I did a redirect from the old url to the new one however still have errors. 1. Should I do a 301 redirect from each old blog post url to the new blog post url? 2. Should I just delete the old blog post (url) and rewrite the blog post? I"m not concerned about links to the old posts as a lot of them do not have many links.
Intermediate & Advanced SEO | | webestate0 -
Best way to handle old re-directs?
What happens if you go back and change old 301 re-directs? So instead of it re-directing from A to B then C, we write a new redirect for A to C. What does Google see this as next time it crawls the site?
Intermediate & Advanced SEO | | anchorwave0 -
Sitemap in SERPS
What's up guys, Having some troubles with SERP rankings. My sitemap (navigation) is appearing instead of my actual keywords. I have tried a few methods to fix this; setting a preferred domain, using a 301 redirects, deleting out of date pages via Google webmaster tools. Nothing seems to work. My next step was to refresh the cache for my entire site - does anyone know how to do this? Can't see any tools... Any help would be great. Cheers, Jon.
Intermediate & Advanced SEO | | jamesjk240 -
High Ranking Site with tons of junk on the server
Hey All, So to make a long story short, we own a site that has been passed through many hand and many strategies. We are in the financial field and rank high for many relevant search terms. My job is now to audit/optimize and purge out site of the garbage that has collected over the years (since 2002). During the audit I have found many issues, fized them, but I am not sure own how to proceed with the follwing issues. Any advice to solve the following would be greatly appreciated! 9932 orphan files - does just removing them affect my SEO.. I like a clean house, can I somehow use them to my benefit? Hundreds of 404s with many external "follow" links that we are no longer getting juice from 8 Sitelinks in webmaster tools, but only 4 show in our search I am straight n00b so sorry if this is 101 for anyone you but your input would be greatly appreciated!! Thanks!
Intermediate & Advanced SEO | | deuce1s0