Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
What tools produce a complete list of all URLs for 301 redirects?
-
I am project managing the rebuild of a major corporate website and need to set up 301 redirects from the old pages to the new ones. The problem is that the old site sits on multiple CMS platforms so there is no way I can get a list of pages from the old CMS. Is there a good tool out there that will crawl through all the sites and produce a nice spreadsheet with all the URLs on it? Somebody mentioned Xenu but I have never used it. Any recommendations? Thanks -Adrian
-
And the winner is... LinkExaminer! Pretty amazing tool, simple to use, but comprehensive - just the thing we need. Thanks for your help Marcin!
Adrian
-
Thanks Marcin and Robert. I will definitely give Xenu Link Sleuth a try first. The Search Engine Optimization Toolkit does look quite awesome though. Thanks for your help. - Adrian
-
There actually many tools which could do the job, both free and paid.
-
Xenu Link Sleuth - free, available at http://home.snafu.de/tilman/xenulink.html
-
Screaming Frog - free (limited)/paid, available at http://www.screamingfrog.co.uk/seo-spider/
-
A1 Website Analyzer - paid, but with a fully functional trial, available at http://www.pendragonexpress.com/hop/a1
-
LinkExaminer - free, available at http://www.analogx.com/contents/download/Network/lnkexam/Freeware.htm
-
Search Engine Optimization Toolkit - free, available at http://www.iis.net/download/seotoolkit
If you just need a crawl, I would recommend 1, 4 or 5. They are both free and work fast. If you want to optimize your content, 5 is a powerhouse.
Hope it helps.
Marcin
-
-
xenu linksleuth is the answer to your question. Good luck. Nothing better I am aware of.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Redirection chain and Javascript Redirect
Hi, A redirection chain is usually defined as a page redirecting to another page which itself is another redirection. URL1 ---(301/302)---> URL2 ---(301/302)---> URL3 But what about Javascript redirect? They seem to be a different beast: URL1 ---(301/302)---> URL2 ---(200 then Javascript redirect)---> URL3 From what I know if the javascript redirect is instant Google counts it as a 301 permanent redirection, but I'm still not sure about if this counts as a redirection chain. Most of the tools (such as moz) only see the first redirection. So is that scenario a redirection chain or no?
Technical SEO | | LouisPortier0 -
301 Redirect for multiple links
I just relaunched my website and changed a permalink structure for several pages where only a subdirectory name changed. What 301 Redirect code do I use to redirect the following? I have dozens of these where I need to change just the directory name from "urban-living" to "urban", and want it to catch the following all in one redirect command. Here is an example of the structure that needs to change. Old
Technical SEO | | shawnbeaird
domain.com/urban-living (single page w/ content)
domain.com/urban-living/tempe (single page w/ content)
domain.com/urban-living/tempe/the-vale (single page w/ content) New
domain.com/urban
domain.com/urban/tempe
domain.com/urban/tempe/the-vale0 -
Meta descriptions and h1 tags during a 301 redirect
My employer is shifting to a new domain and i am in the midst of doing URL mapping. I realize that many of the meta descriptions and H1 tags are different on the new pages - is this a problem ? Thank you.
Technical SEO | | ptapley0 -
Tool to Generate All the URLs on a Domain
Hi all, I've been using xml-sitemaps.com for a while to generate a list of all the URLs that exist on a domain. However, this tool only works for websites with under 500 URLs on a domain. The paid tool doesn't offer what we are looking for either. I'm hoping someone can help with a recommendation. We're looking for a tool that can: Crawl, and list, all the indexed URLs on a domain, including .pdf and .doc files (ideally in a .xls or .txt file) Crawl multiple domains with unlimited URLs (we have 5 websites with 500+ URLs on them) Seems pretty simple, but we haven't been able to find something that isn't tailored toward management of a single domain or that can crawl a huge volume of content.
Technical SEO | | timfrick0 -
301 Redirects Relating to Your XML Sitemap
Lets say you've got a website and it had quite a few pages that for lack of a better term were like an infomercial, 6-8 pages of slightly different topics all essentially saying the same thing. You could all but call it spam. www.site.com/page-1 www.site.com/page-2 www.site.com/page-3 www.site.com/page-4 www.site.com/page-5 www.site.com/page-6 Now you decided to consolidate all of that information into one well written page, and while the previous pages may have been a bit spammy they did indeed have SOME juice to pass through. Your new page is: www.site.com/not-spammy-page You then 301 redirect the previous 'spammy' pages to the new page. Now the question, do I immediately re-submit an updated xml sitemap to Google, which would NOT contain all of the old URL's, thus making me assume Google would miss the 301 redirect/seo juice. Or do I wait a week or two, allow Google to re-crawl the site and see the existing 301's and once they've taken notice of the changes submit an updated sitemap? Probably a stupid question I understand, but I want to ensure I'm following the best practices given the situation, thanks guys and girls!
Technical SEO | | Emory_Peterson0 -
Increase 404 errors or 301 redirects?
Hi all, I'm working on an e-commerce site that sells products that may only be available for a certain period of time. Eg. A product may only be selling for 1 year and then be permanently out of stock. When a product goes out of stock, the page is removed from the site regardless of any links it may have gotten over time. I am trying to figure out the best way to handle these permanently out of stock pages. At the moment, the site is set up to return a 404 page for each of these products. There are currently 600 (and increasing) instances of this appearing on Google Webmasters. I have read that too many 404 errors may have a negative impact on your site, and so thought I might 301 redirect these URLs to a more appropriate page. However I've also read that too many 301 redirects may have a negative impact on your site. I foresee this to be an issue several years down the road when the site has thousands of expired products which will result in thousands of 404 errors or 301 redirects depending on which route I take. Which would be the better route? Is there a better solution?
Technical SEO | | Oxfordcomma0 -
How long will Google take to stop crawling an old URL once it has been 301 redirected
I need to do a clean-up old urls that have been redirected in sitemap and was wondering about this.
Technical SEO | | Ant-8080 -
If a redirecting URL has more value than the website should I move it?
Client has two website addresses: Website A is a redirect to Website B. It has one indexed page. But this is the URL being used in collateral. It has the majority of back links, and citations everywhere list Website A as the URL. Website B is where the actual website lives. Google recognizes and indexes the 80+ pages. This website has very few backlinks going to it. This setup does not seem good for SEO. Moreover, the analytics data is completely messed up because Website B shows that the biggest referral source is... you guessed it Website A. I'm thinking going forward, I should: Move all the content from Website B to Website A. Setup Website B to permanently 301 Redirect to Website A. Is that the best course of action?
Technical SEO | | flowsimple0