Best server-side sitemap generators
-
I've been looking into sitemap generators recently and have got a good knowledge of what creating a sitemap for a small website of below 500 URLs involves. I have successfully generated a sitemap for a very small site, but I’m trying to work out the best way of crawling a large site with millions of URLs.
I’ve decided that the best way to crawl such a large number of URLs is to use a server side sitemap, but this is an area that doesn’t seem to be covered in detail on SEO blogs / forums. Could anyone recommend a good server side sitemap generator? What do you think of the automated offerings from Google and Bing? I’ve found a list of server side sitemap generators from Google, but I can’t see any way to choose between them. I realise that a lot will depend on the type of technologies we use server side, but I'm afraid that I don't know them at this time.
-
Unless they have fixed it in recent months, xml-sitemaps does not generate correct video sitemaps.
-
Yeah, they offer free and paid hosted versions too. But I found the server side version much simpler to setup and control.
-
-
Excellent advice Federico. My first reaction was, "but that's not a server-side sitemap generator". I just looked at their website though and it turns out that it is! Looks like I need to read things more carefully!
I'll look into that as an option but if anyone else has any server side sitemap generators that they'd recommend then I'd be really interested to hear about them
-
I have been using xml-sitemaps (paid version) for all my sites over 5 years and they work like a charm, scraping and indexing what it needs to be indexed ans scraped, plus it consumes really low resources. 100% recommended (they have nice plugins too for extra sitempas (video, news, images, etc).
Hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What is the best SEO way for a shop
Hi there ! A client want to sell some products on its future website but just a small range (the most part of this website will not be an online shop). The idea is to add a "shop" button in the menu to redirect clients in this shop. I would like your opinion about how should I construct this shop, what do you think is the best for SEO : "www.website.com/shop" or "shop.website.com" thank you in advance for your answers !
Intermediate & Advanced SEO | | EnjinFrance0 -
Sitemap into SE
Hi Moz community experts, I have a question about the sitemap into search engine like here : http://i.imgur.com/gQ0JhuH.jpg. Do you know what I need to do to get the same structure or do decide which pages we want to present into our result. We created a new page and we would like to see it into the resultat when the visitor is searching for our branded keywords. Thank in advance for your support. gQ0JhuH.jpg.
Intermediate & Advanced SEO | | johncurlee0 -
installed PageSpeed Module on our server but no difference to site
Hi
Intermediate & Advanced SEO | | Direct_Ram
I have been searching for an answer for a while now and couldnt find it so maybe someone has had a similar problem. We have installed PageSpeed Module on our server. The administrator has said it is active and has run a test below: [root@mydomain ~]# curl -D- https://www.mydomain.com/ | head -10
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
14 102k 14 15029 0 0 40506 0 0:00:02 --:--:-- 0:00:02 64780HTTP/1.1 200 OK
Server: nginx/1.6.0
Date: Fri, 10 Apr 2015 11:28:43 GMT
Content-Type: text/html
Content-Length: 104885
Connection: keep-alive
Set-Cookie: ci_session=BGANYlg8VmsPLgN1AWABMldkAGUGLVZwVmhQdQd0CGIEaFI6VgkEOQdmUSYHbQZyXz9TZVE4Vm4CIwxnB2hYbAZrAGUHZQg%2BUjUFOgRlUWAEYg05WDxWMg82A2ABOQEzV2IAaQZsVjBWPFA2BzEIaAQ%2FUjBWNwRmBztRJgdtBnJfP1NnUTpWbgIjDDoHflhSBjwAMgdjCHlSNAVwBHdRJwQ6DStYM1ZgD2YDPAF4ATJXZABmBiFWMVY%2FUD4HKQg5BDRSelZnBGAHIFE%2FByUGO180U2ZRMFZ2AnQMIAdrWH8GAgA3B2AIblI%2FBXcEJlE%2BBHINYlg4VmAPZwM8AXgBYFchAC0GY1YsVjpQKAc2CDIEKVJjVnYEeAd6UTwHYAZeXzNTYlEnViYCZAw3B2ZYbAYpAHsHawhiUj8FdgR8USgEZg02WHxWeA91A2oBMwFhVzcAKgZ9Vm9WIlAxBykIOgQ%2BUnpWYQRwB0xRVwcFBi5fNlN4UTtWYgIvDGEHIFg%2BBn0AFAdmCHhSOAVgBCRRQARCDRtYKVYrDzkDbwE4ASxXZQBxBj1WLVY%2BUCYHawhiBGVSPVYyBD4HLVE1B3gGMF89U3ZRZlY9AmMMIAd9WGUGbwB5BzYIJVJlBS0ENlEnBDoNK1gzVmAPZgM8AXgBb1c1ACwGe1ZcVmxQZQdzCGIEcVI9ViIEKQcgUT8HPwY7XzRTYlE4VmwCNwxlBztYPgZvAGUHPAh4UmsFOgQ%2BUScEdA0rWGxWIw8KA2IBOwF3VzUAfQY0VnBWN1A2Bz0IKQQlUm9WKw%3D%3D; expires=Fri, 10-Apr-2015 13:28:43 GMT; path=/
Set-Cookie: ci_session=a%3A0%3A%7B%7D; expires=Thu, 10-Apr-2014 21:28:43 GMT; path=/
Set-Cookie: ci_session=BWEFalk4UWwJKFIq; expires=Fri, 10-Apr-2015 13:28:43 GMT; path=/
X-Mod-Pagespeed: 1.9.32.3-4448 But there doesn't seem to be any difference to the sites speed or change in google speed test recommendations. I do not have much knowledge on servers but the server company has assured me it is active and all the filters are on - so not sure why I am not seeing anything different. if anyone has any advise on this it would be great. thanks E0 -
Pagination, Canonical Tag & Best Practices
I have an eCommerce site that dynamically creates category pages, which produce canonical tags in the header. For multiple page categories, it adds the page number to the URL. For example, this category has 3 pages.... Because most categories have too many products, I can't follow Googles suggestion of creating a "view all" page. Furthermore since all these pages use the same template, I'm unable to insert a NOINDEX tag in all the pages after the first page. Also, in this scenario, I'm unable to insert the discreet code for Next/Previous, which is also suggested by Google. My only option for maintaining these dynamically generated category pages would be to hardcode the first conical tag in the template, which would then be produced on all subsequent paginated pages. Consequently, every paginated page in this category would have the same canonical tag pointing to the first page. Would this incur the wrath of Google and would I'd be better off leaving the pagination they way it is?
Intermediate & Advanced SEO | | alrockn0 -
Best practice with duplicate content. Cd
Our website has recently been updated, now it seems that all of our products pages look like this cdnorigin.companyname.com/catagory/product Google is showing these pages within the search. rather then companyname.com/catagory/product Each product page does have a canaonacal tag on that points to the cdnorigin page. Is this best practice? i dont think that cdnorigin.companyname etc looks very goon in the search. is there any reason why my designer would set the canonical tags up this way?
Intermediate & Advanced SEO | | Alexogilvie0 -
Getting a Sitemap for a Subdomain into Webmaster Tools
We have a subdomain that is a Wordpress blog, and it takes days, sometimes weeks for most posts to be indexed. We are using the Yoast plugin for SEO, which creates the sitemap.xml file. The problem is that the sitemap.xml file is located at blog.gallerydirect.com/sitemap.xml, and Webmaster Tools will only allow the insertion of the sitemap as a directory under the gallerydirect.com account. Right now, we have the sitemap listed in the robots.txt file, but I really don't know if Google is finding and parsing the sitemap. As far as I can tell, I have three options, and I'd like to get thoughts on which of the three options is the best choice (that is, unless there's an option I haven't thought of): 1. Create a separate Webmaster Tools account for the blog 2. Copy the blog's sitemap.xml file from blog.gallerydirect.com/sitemap.xml to the main web server and list it as something like gallerydirect.com/blogsitemap.xml, then notify Webmaster Tools of the new sitemap on the galllerydirect.com account 3. Do an .htaccess redirect on the blog server, such as RewriteRule ^sitemap.xml http://gallerydirect.com/blogsitemap_index.xml Then notify Webmaster Tools of the new blog sitemap in the gallerydirect.com account. Suggestions on what would be the best approach to be sure that Google is finding and indexing the blog ASAP?
Intermediate & Advanced SEO | | sbaylor0 -
What is the best way to consolidate two websites into one?
Someone within our company's IT department just sent me some SEO advice that I believe is bogus. Can someone let me know if my initial gut-check is correct? We have two websites selling two identical catalogs of products but branded differently (color scheme, wording, etc.) like this: www.one.com
Intermediate & Advanced SEO | | Ryan-Ricketts
www.two.com We want to shut down the second website. I think we should set up 301 redirects from all pages on the second site to corresponding (relevant) pages on the first. In theory, this would pass over 90% of the earned link juice from one to the other. Here is what my IT peer said: "We could keep www.two.com set up indefinitely and just have it as the same web site as www.one.com (so two URLs but one site). This would help alleviate any issues with search engine results, etc. (Although I believe Ryan would agree this does impact www.one.com's rankings a bit, but shouldn't be a problem as long as we don't advertise both.) Google doesn't know they are on the same site, so you could technically get away with it. And it helps in indexing multiple pages on our sites." ... but wouldn't this be a big no-no because of the massive amounts of duplicate content it would create?0 -
Dynamically generated page issues
Hello All! Our site uses dynamically generated pages. I was about to begin the process of optimising our product category pages www.pitchcare.com/shop I was going to use internal anchor text from some high ranking pages within our site but each of the product category pages already have 1745 links! Am I correct in saying that internal anchor text links works to a certain point? (maybe 10 or so links) So any new internal anchor text links will count for nothing? Thanks Todd
Intermediate & Advanced SEO | | toddyC0