Tool for Generating Sitemap/ URL List
-
HI,
I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator.
However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well.
Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing?
Thanks.
-
Try DYNO Mapper. If you are looking for a tool that will generate a URL list, I think it will work really well for you. It also will give you a full content inventory that you can filter, search, and export content types. This particular Sitemap Generator is also integrated with Google Analytics and has collaboration capability for multiple users.
-
Just spent the last couple hours checking out Xenu. Found this article here on SEOmoz:
http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder.
Turns out Xenu is an awesome tool - can't wait to make more use of it.
I also took Joyce's advice which was more suited to the task I was trying to do. Thanks for the simple solution!
-
Yes Xenu will do it, it checks all the links and then offers you a sitemap as a report at the end
I like Xenu, it's quick too
-
If you're using SEOMoz Pro, you can enter the domain into Open Site Explorer and then click the Top Pages tab. This will give you an exportable list of up to 10,000 pages on the site. Alternatively, you can go to Google Analytics and click Content > Top Content. This will give you an even bigger list, though you may have to filter out any attributes that Google picks up.
Xenu can also be a good option, but you still have to do a lot of filtering.
I hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
URL Structure on Category Pages
Hi, Currently, we having the following URL Structure o our product pages: All Products Pages: www.viatrading.com/wholesale/283/All_Products.html Category Page: www.viatrading.com/wholesale/4/Clothing.html Product Page: www.viatrading.com/wholesale/product/LOAD-HE-WOM/Assorted-High-End-Women-Clothing-Lots.html?cid=4 Since we are going to use another frontend system, we are thinking about re-working on this URL Structure, using something like this: All Products Pages: www.viatrading.com/wholesale-products/ Category Page: www.viatrading.com/wholesale-products/category/ Product Page: www.viatrading.com/wholesale-products/category/product-title/ I understand this is better for SEO and user experience. However, we already have good traffic on the current URL Structure. Should we use same left-side filters on Category Pages as in All Products Page? Since we are using Faceted Navigation, when users filter the Category (e.g. Clothing) they will see same page as Clothing Category Page. Is that an issue for Duplicate Content? Since we are a wholesale company - I understand is using "/wholesale/products/" in URL for all product pages a good idea? If so, should we avoid word "wholesale" in product-title to avoid repeated word in URL? For us, SKU in URL helps the company employees and maybe some clients identify the link. However, what do you think of using the SEO-friendly product-title, and 301 redirect it to www.viatrading.com/BRTA-LN-DISHRACKS/, so 1st link is only used by company members and Canonicalized 2nd is the only one seen by general public? Thank you,
On-Page Optimization | | viatrading10 -
Changing URL Structure From Flat to Pyramid Theme
Hello Mozzers, I have an on-page SEO question regarding URL structure. A few months back we hired a full-time SEO person who is working on-page right now and she really wants us to completely re-due our URL restructure from a flat to pyramid style (example below). Current URL structure / page title is: Dog training Collars - K9electronics.com
On-Page Optimization | | k9byron
http://www.k9electronics.com/dog-training-collars/ Small Dog training Collars - K9electronics.com
http://www.k9electronics.com/small-dog-training-collars/ Einstein ET-300TS Mini Dog Training Collar - K9electronics.com
http://www.k9electronics.com/einstein-et-300ts-mini-remote-dog-trainer.html Suggested URL structure / page title change: Quality Dog Training Collars - Lowest Price Guarantee - K9electronics
http://www.k9electronics.com/dog-training-collars/ Dog Training Collars - Small Dog - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/ Einstein ET-300TS Mini - Dog Training Collars - K9electronics.com
http://www.k9electronics.com/dog-training-collars/small-dog/einstein-et-300ts-mini I guess you could say we are the poster-boys for Google penalties and have received just about every penalty in the book. Panda, penguin algo penalties and a partial manual action for unnatural links. Several months ago we removed 1000's of in-bound links and had our manual action lifted a few months back and are now hovering around the top / mid 2nd page for all our big terms ...we used to be top 3 for everything. As we were removing bad links, we also completely redesigned the site and removed lots of categories and products and 95% of all our old, low quality content and replaced it with new, high quality content. The site was really slooooow, so we optimized it and moved it to a big dedicated server and tripled page load time. Added rich snippets, Google authorship, increased our FB and other social presences and much more ... I had also considered this URL structure change during the redesign because I had heard and read that it was good to do, but it required redirecting practically all our URL's which I know can hurt the site even more so then it already has been ... Our SEO says that as it sits now, our pages are competing with each other and really seems to think this is going to improve our rankings a lot ...after several weeks. My question is, at this stage in the game, is it really going to help a lot and give us more benefit compared to the 301 redirect link juice loss? Any comments and/or suggestions are very much appreciated!1 -
URL best practices
Hi, I have a problem here, I used http://www.vietnamvisacorp.com/faqs.html instead of http://www.vietnamvisacorp.com/faqs. Hence, http://www.vietnamvisacorp.com/faqs will be caused 404 page. My question is should I change from faqs.html to faqs (no .html)? Thanks in advance any advice?
On-Page Optimization | | JohnHuynh0 -
URL parameters
Hello, Currently, I paginated a content to 5 pages eg: http://abc.com/faqs.html?&page=2 Is it right? and how to check it is correct or not?
On-Page Optimization | | JohnHuynh0 -
Slash at the end of a url
I keep reading contradicting information, so I figured I'll ask here. What's the best practice for slash '/' at the end of a URL? Should it be idealchooser.com/search/laptop/ or idealchooser.com/search/laptop (no trailing slash)? The options: 1. Accept both equally 2. Accept 1 and redirect the other with 301 3. Accept 1 and treat the other as a wrong URL returning 404 Which would be the best for SEO? Thank you.
On-Page Optimization | | corwin0 -
Bing - how come we are not shown in listing?
Hi When looking for our site 'mpn windows' on Bing www.mpnwindows.com it brings back no results for the site? It has been registered in bing webmaster center with pages starting being indexed on the 3rd October as we only recently included it in Bing. In Google it is showing up ok however?
On-Page Optimization | | idv0 -
URL structure for a new WordPress site
Hi I'm building a new next big thing website from scratch (for a translation agency) and I encountered an issue with the URL structure. I need to chose the URL for important targeted keyword pages and I have a conflict between two tools I'm using. Please read below the situation: domain: mashtranslation.com target keyword: french translation services which URL you think is better from a SEO point of view (and possibly for users): mashtranslation.com/services/french/ OR mashtranslation.com/french-translation-services/ I'm asking this because one WordPress plugin (Wordpress SEO by Yoast) says the URL structure is not optimised while another tool (Market Samurai) says the URL is optimised.
On-Page Optimization | | flo20 -
URL question
Hi guys, the pro campaign thing you got going is wicked, love it. I'm recieving good results with my keywords and have noticed that categories that go beyond sub/sub/sub don't do to well. So I wanna move those that do one step up which makes it go from: http://spytunes.com/practice-guitar/advanced-routine/scales/aeolian to here http://spytunes.com/practice-guitar/advanced-routine/aeolian The existing menu system that follow all these categories across the site will soon go so it won't be a user friendly problem, I will have other type of menus. But, and here is the question: Would I greatly benefit from taking the non existent menu away and just go for: http://spytunes.com/practice-guitar/aeolian while i'm at it? Or do I stick with my current structure? I guess my real question is; how much is there to flat URLs? Cheers -dan lundholm spytunes.com
On-Page Optimization | | spytunes0