Tool for Generating Sitemap/ URL List
-
HI,
I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator.
However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well.
Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing?
Thanks.
-
Try DYNO Mapper. If you are looking for a tool that will generate a URL list, I think it will work really well for you. It also will give you a full content inventory that you can filter, search, and export content types. This particular Sitemap Generator is also integrated with Google Analytics and has collaboration capability for multiple users.
-
Just spent the last couple hours checking out Xenu. Found this article here on SEOmoz:
http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder.
Turns out Xenu is an awesome tool - can't wait to make more use of it.
I also took Joyce's advice which was more suited to the task I was trying to do. Thanks for the simple solution!
-
Yes Xenu will do it, it checks all the links and then offers you a sitemap as a report at the end
I like Xenu, it's quick too
-
If you're using SEOMoz Pro, you can enter the domain into Open Site Explorer and then click the Top Pages tab. This will give you an exportable list of up to 10,000 pages on the site. Alternatively, you can go to Google Analytics and click Content > Top Content. This will give you an even bigger list, though you may have to filter out any attributes that Google picks up.
Xenu can also be a good option, but you still have to do a lot of filtering.
I hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
410 /demandware.store/ pages
I have a client that just migrated to salesforce commerce cloud. They have a large amount of 410 pages showing up in site crawls. They are all of the /on/demandware.store/ variety. Is there any reason to redirect thousands of these pages? Any value? Or should they be allowed to drop off?
On-Page Optimization | | bdcseo1 -
Url permalink structure question!
Hello! I just read the "15 SEO Best Practices for Structuring URLs" but I have still a question: My A version bellow "20 accessoires" has no meaning in french. If I add "voyage" (version B), then is it considered as spammy? I mean the "voyage" keyword repetition? A) http://www.lytchee.com/preparer-son-voyage/20-accessoires/ B) http://www.lytchee.com/preparer-son-voyage/20-accessoires-voyage Thanks for advices! Sylvain
On-Page Optimization | | lytcheetv0 -
Paginated URLs are getting Indexed
Hi, For ex: - My site is www.abc.com and Its paginated URLs for www.abc.com/jobs-in-delhi are in the format of : www.abc.com/jobs-in-delhi-1, www.abc.com/jobs-in-delhi-2 and vice versa also i have used pagination tags rel=next and rel=prev. My concern is all the paginated URLs are getting indexed so is their any disadvantage if these URLs are getting indexed as somewhere i have read that link juice may get distributed in case of pagination. isn't it good to use Noindex, Follow so that we can make the Google to understand that paginated page are not so much important and that should not be ranked.
On-Page Optimization | | vivekrathore0 -
How to fix this URL?
Moz crawl test only show one URL indexed for the website - http://ravisharrma.com/ Site:http://ravisharrma.com/ shows 3 URLs and how to fix URLs for the website?
On-Page Optimization | | Dan_Brown10 -
Schema for Web/SEO Service Pages
I was thinking about setting up my services pages for SEO and Web design with Schema code for a product. Is this a good idea? Has anyone done it?
On-Page Optimization | | NeilBelliveau0 -
Lists of Product Links: What is good, what is bad?
I am a web designer but a bit of an SEO noob (trying to get better at both). I am working with one particular client on a site I inherited with existing structure. This client has about 10 products on 2 pages. On every page there is a product list that is basically the same list sorted in 2 ways: 1st by product, 2nd by usage. These all link to internal anchors so this might be an example on www.site.com Cleaner X1 - links to www.site.com/cleaners.php#x1
On-Page Optimization | | mparry9
Cleaner X2 - links to www.site.com/cleaners.php#x2
Cleaner X3 - links to www.site.com/cleaners.php#x3
...
Cleaner For Brick - links to www.site.com/cleaners.php#x1
Cleaner For Marble - links to www.site.com/cleaners.php#x2
Cleaner For Stone - links to www.site.com/cleaners.php#x3 Obviously this adds about 20 links on every page on the site (including the actual pages these products are on). What are your thoughts on this? Good idea or bad to have on the site? Should I remove the redundant links on the actual page that product falls on...or is this bad and should be removed altogether?0 -
No follow for html sitemap?
Hi, I have been working on an e-commerce site and have been wondering if i should add the meta robot tag with no follows, on pages like delivery , terms, returns and the html sitemap? From what I have read this seems a good idea, but i am a little confused as what to do with the html sitemap. I can understand that having this on the homepage will allow crawlers quick access to the deeper pages within the site. But would it be better to guide them down the natural route of the category navigation instead? Thanks
On-Page Optimization | | Bmeisterali0 -
Close URL owned by competitors.
The following example is exactly analogous to our situation (site names slightly altered😞 We own www.business-skills.com. It's our main site. We don't own, and would rather avoid paying for, www.businessskills.com. It's a parked domain and the owners want a very large sum for it. We own www.business-skills.co.uk and point it to our main site. We don't own www.businessskills.co.uk. This is owned by our biggest competitor. We also own www.[ourbrand].com and .co.uk, and point them to the main site. My question is - how much traffic do you think we may be missing due to these nearly-but-not-quite URL matches? Does it matter in terms of lost revenue? What sort of things should I be looking at to get a very rough estimate?
On-Page Optimization | | JacobFunnell0