Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can I add external links to my sitemap?
Hi, I'm integrating with a service that adds 3rd-party images/videos (owned by them, hosted on their server) to my site. For instance, the service might have tons of pictures/videos of cars; and then when I integrate, I can show my users these pictures/videos about cars I might be selling. But I'm wondering how to build out the sitemap--I would like to include reference to these images/videos, so Google knows I'm using lots of multimedia. How's the most white-hat way to do that? Can I add external links to my sitemap pointing to these images/videos hosted on a different server, or is that frowned upon? Thanks in advance.
Intermediate & Advanced SEO | | SEOdub0 -
What are best page titles for sub-domain pages?
Hi Moz communtity, Let's say a website has multiple sub-domains with hundreds and thousands of pages. Generally we will be mentioning "primary keyword & "brand name" on every page of website. Can we do same on all pages of sub-domains to increase the authority of website for this primary keyword in Google? Or it gonna end up as negative impact if Google consider as duplicate content being mentioned same keyword and brand name on every page even on website and all pages of sub domains? Thanks
Intermediate & Advanced SEO | | vtmoz0 -
Help FORUM ( User generated content ) SEO best practices
Hello Moz folks ! For the very first time im dealing with a massive community who rely on UGC ( user generated content ). Their forum is finding a great deal of duplicate content/broken link/ duplicate title and on-site issue. I have Advance SEO knowledge related to ecommerce or blogging but new to forum and UGC. I would really love to learn or get ressources links that would allow me to see/understand the best practices in term of SEO. Any help is greatly appreciated. Best, Yan
Intermediate & Advanced SEO | | ydesjardins2000 -
Best option for Affiliate links on your website?
Hello! I have a website which is completely affiliate based. What is the best option for the links on-page? Examples would be: affiliate.website.com/12901730?2=3532523=user12342901730?2=3532523=user?Whittie www.website.com/affiliate=user?Whittie=load-of-tracking=date=blah=blaH?blah And So on... Which look ugly as sin when you hover over the Anchor Text. Ideally I would like a 301 redirect to mysite.com/goto/affiliatename, which would then have a rel nofollow. This way I could also track the exit pages via Analytics too guess, which I've not currently got set up and i'm desperate for it to be done. Does this method effect anything on search engines though? I've seen mixed report, but going back to 2011 which is too long ago in the SEO world. Another option is to use the likes of "Bit.ly" or use another domain and host 301s on there? The new bit.ly integration from moz might come in handy here. Please advise on the subject, I really appreciate any help on this, as i'm at a brick wall. Thanks
Intermediate & Advanced SEO | | Whittie0 -
How to seo websites on another server
We have partners that want us to build and manage a co-branded white label for them. We will have unique content on the white label, however the white label will be located on our server. I was planning to put it on a subdomain and mask the URL, however was told that google will see through that and not give any credit to the white label. Our partners all have high PR and we are a new company with low PR. We want the white labels to get the credit from the partner websites. Should we do it through url masking or by changing the A Record in the other website to point to our server?
Intermediate & Advanced SEO | | TravelerVIP0 -
Server requests: 302 followed by a 200
Hi, On an IIS system clicking a particular link the following response codes are returned: GET /nl/nl/process?Someparameter1=1&Someparameter2=2 302 found GET /nl/nl/SomeOtherPage.cms 200 OK What concerns me, besides the obvious 302 and the cAmeLcAse canonical issues is the 200 response without a redirect.
Intermediate & Advanced SEO | | Muffin
What page will then be indexed, ranked and what effect does this have on the pagerank flow, if the 302 was to be changed into a 301?
Also would extention .cms be an issue? Thanks for any answers. Edit. I contacted the developer. He says it's a rewrite, not a meta redirect.
I still think, this rewrite is an issue? Canonical maybe?0 -
Submitting URLs multiple times in different sitemaps
We have a very dynamic site, with a large number of pages. We use a sitemap index file, that points to several smaller sitemap files. The question is: Would there be any issue if we include the same URL in multiple sitemap files? Scenario: URL1 appears on sitemap1. 2 weeks later, the page at URL1 changes and we'd like to update it on a sitemap. Would it be acceptable to add URL1 as an entry in sitemap2? Would there be any issues with the same URL appearing multiple times? Thanks.
Intermediate & Advanced SEO | | msquare0 -
Generating 404 Errors but the Pages Exist
Hey I have recently come across an issue with several of a sites urls being seen as a 404 by bots such as Xenu, SEOMoz, Google Web Tools etc. The funny thing is, the pages exist and display fine. This happens on many of the pages which use the Modx CMS, but the index is fine. The wordpress blog in /blog/ all works fine. The only thing I can think of is that I have a conflict in the htaccess, but troubleshooting this is difficult, any tool I have found online seem useless. Have tried to rollback to previous versions but still does not work. Anyone had any experience of similar issues? Many thanks K.
Intermediate & Advanced SEO | | Found0