Tool for Generating Sitemap/ URL List
-
HI,
I'm looking for a tool that'll generate a URL list for a site. I looked at this thread here http://www.seomoz.org/q/online-sitemap-generator which came up when I searched for sitemap generator.
However, I don't need a sitemap per se, and I don't need to submit it to Google - just a list of pages is what I need.If it updated automatically, that would be useful as well.
Anyone know of a tool, on or offline? Or anyone used Xenu and know if it's what I'm looking for? Or is there a simple solution that I'm missing?
Thanks.
-
Try DYNO Mapper. If you are looking for a tool that will generate a URL list, I think it will work really well for you. It also will give you a full content inventory that you can filter, search, and export content types. This particular Sitemap Generator is also integrated with Google Analytics and has collaboration capability for multiple users.
-
Just spent the last couple hours checking out Xenu. Found this article here on SEOmoz:
http://www.seomoz.org/blog/xenu-link-sleuth-more-than-just-a-broken-links-finder.
Turns out Xenu is an awesome tool - can't wait to make more use of it.
I also took Joyce's advice which was more suited to the task I was trying to do. Thanks for the simple solution!
-
Yes Xenu will do it, it checks all the links and then offers you a sitemap as a report at the end
I like Xenu, it's quick too
-
If you're using SEOMoz Pro, you can enter the domain into Open Site Explorer and then click the Top Pages tab. This will give you an exportable list of up to 10,000 pages on the site. Alternatively, you can go to Google Analytics and click Content > Top Content. This will give you an even bigger list, though you may have to filter out any attributes that Google picks up.
Xenu can also be a good option, but you still have to do a lot of filtering.
I hope that helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Url permalink structure question!
Hello! I just read the "15 SEO Best Practices for Structuring URLs" but I have still a question: My A version bellow "20 accessoires" has no meaning in french. If I add "voyage" (version B), then is it considered as spammy? I mean the "voyage" keyword repetition? A) http://www.lytchee.com/preparer-son-voyage/20-accessoires/ B) http://www.lytchee.com/preparer-son-voyage/20-accessoires-voyage Thanks for advices! Sylvain
On-Page Optimization | | lytcheetv0 -
URL Structure on Category Pages
Hi, Currently, we having the following URL Structure o our product pages: All Products Pages: www.viatrading.com/wholesale/283/All_Products.html Category Page: www.viatrading.com/wholesale/4/Clothing.html Product Page: www.viatrading.com/wholesale/product/LOAD-HE-WOM/Assorted-High-End-Women-Clothing-Lots.html?cid=4 Since we are going to use another frontend system, we are thinking about re-working on this URL Structure, using something like this: All Products Pages: www.viatrading.com/wholesale-products/ Category Page: www.viatrading.com/wholesale-products/category/ Product Page: www.viatrading.com/wholesale-products/category/product-title/ I understand this is better for SEO and user experience. However, we already have good traffic on the current URL Structure. Should we use same left-side filters on Category Pages as in All Products Page? Since we are using Faceted Navigation, when users filter the Category (e.g. Clothing) they will see same page as Clothing Category Page. Is that an issue for Duplicate Content? Since we are a wholesale company - I understand is using "/wholesale/products/" in URL for all product pages a good idea? If so, should we avoid word "wholesale" in product-title to avoid repeated word in URL? For us, SKU in URL helps the company employees and maybe some clients identify the link. However, what do you think of using the SEO-friendly product-title, and 301 redirect it to www.viatrading.com/BRTA-LN-DISHRACKS/, so 1st link is only used by company members and Canonicalized 2nd is the only one seen by general public? Thank you,
On-Page Optimization | | viatrading10 -
Home page keyword in url
I have been looking into SEO for a few weeks now trying to perfect a homepage. Going through various sources on MOZ, and other examples out there on the internet, I keep seeing that you should have your keyword in the URL of the page. The homepage is the page most people want to rank the highest in google searches, however, you cannot put the keyword in the URL as most home page URLs are simply /. Should I actually make the home like this: www.example.com/key-word-example? I would imagine this would not be the normal for many users and would seem like it's not the home page.
On-Page Optimization | | Matthew_smart0 -
How do you handle URLs with slashes?
I asked this question before, but with a different scenario. I upgraded my plan to a more advanced cart and all of my URLs changed about 1.5 years ago. I knew nothing about redirects and such, so none of that was done. Basically, let's say my site was: http://www.abc.com, but when people actually visit my site, they are directed to https://www.abc.com/. I have asked my host about redirecting and she that it is not possible. In the past, the link shared has been just www.abc.com . Will this hurt my ranking? My second question is ...let's say I have a link http://www.abc.com/blog , but now, the link is http://www.abc.com/blog/ . Will I be affected, since all my old links omit the slash?
On-Page Optimization | | tiffany11030 -
Www.colourbanners.co.uk/ & colourbanners.co.uk showing up as two seperate URLs - is this going to be dupliacte content issue?
Hi Guys, I have just created a report in Moz and there appears to be 91 duplicate content issues with the site which i need to fix as i think it could be the reason why we are suffering from a penalty. One of the main questions i have is these 3 variations of the URL http://www.colourbanners.co.uk/ http://colourbanners.co.ukhttp://colourbanners.co.uk/Each have links pointing to them. My question is, could this be causing a dupe issue?regardsGerry
On-Page Optimization | | gezzagregz0 -
URL Domain Used in Meta Description
Today I was asked if using a domain url in your meta description can have a negative impact on your website. This description includes a list of the homepage url, but directs visitors to a different internal page of the website. My concern fell with directing visitors to a different page of the site, but promoting the homepage in the description/snippet. With Penguin 2.1 release on the 4th, I'm very cautious of my links/urls. What are your thoughts behind this? What are the possible, if any negative impacts this could have on a site? This URL does have a brand name as so the Title.
On-Page Optimization | | flcity150 -
Left Nav / SEO
We are in the middle of a redesign and would love to get rid of our left navigation. It's currently full of SEO rich categories and has been the same for 10+ years. Our programmers think removing the left navigation will have no impact on our SEO. As SEO's, do you agree with this?
On-Page Optimization | | EileenCleary0 -
Can Sitemap Be Used to Manage Canonical URLs?
We have a duplicate content challenge that likely has contributed to us loosing SERPs especially for generic keywords such as "audiobook," "audiobooks," "audio book," and "audio books." Our duplicate content is on two levels. 1. The first level is at our web store, www.audiobooksonline.com. Audiobooks are sometimes published in abridged, unabridged, on compact discs, on MP3 CD by the same publisher. In this case we use the publisher description of the story for each "flavor" = duplicate content. Can we use our sitemap to identify only one "flavor" so that a spider doesn't index the others? 2. The second level is that most online merchants of the same publisher's audio book use the same description of the story = lots of duplicate content on the Web. In that we have 11,000+ audio book titles offered at our Web store, I expect Google sees us as having lots of duplicated (on the Web) content and devalues our site. Some of our competitors who rank very high for our generic keywords use the same publisher's description. Any suggestions on how we could make our individual audio book title pages unique will be greatly appreciated.
On-Page Optimization | | lbohen0