Tool to Generate All the URLs on a Domain
-
Hi all,
I've been using xml-sitemaps.com for a while to generate a list of all the URLs that exist on a domain. However, this tool only works for websites with under 500 URLs on a domain. The paid tool doesn't offer what we are looking for either. I'm hoping someone can help with a recommendation.
We're looking for a tool that can:
- Crawl, and list, all the indexed URLs on a domain, including .pdf and .doc files (ideally in a .xls or .txt file)
- Crawl multiple domains with unlimited URLs (we have 5 websites with 500+ URLs on them)
Seems pretty simple, but we haven't been able to find something that isn't tailored toward management of a single domain or that can crawl a huge volume of content.
-
@PatrickDelehanty The tool mentioned in the statement not only excels in the two areas mentioned earlier but also offers a wide range of additional capabilities. I recommend that you explore it for yourself! Best of luck!
-
@PatrickDelehanty The tool mentioned in the statement not only excels in the two areas ```
mentioned -
It seems to crawl all the wordpress folders and media files.
Is there not a tool that will tell you just your live website URLs, I'm after creating a site map and a mass re-organising content exercise, so want a list in excel of URLs.Any tips welcome
Thanks
Sarah
-
2nd Vote for Screaming Frog. Tried a lot of tools to pull info on all the URL's and this tool is by far the best one for the job.
-
Hi Felicia
Try ScreamingFrog - they crawl the entire site (you can configure how you want it to crawl your site) and have ways of creating a XML Sitemap for you.
The tool goes above and beyond those two areas as well and can do so much. I suggest you check it out! Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google selecting incorrect URL as canonical: 'Duplicate, submitted URL not selected as canonical'
Hi there, A number of our URLs are being de-indexed by Google. When looking into this using Google Search Console the same message is appearing on multiple pages across our sites: 'Duplicate, submitted URL not selected as canonical' 'IndexingIndexing allowed? YesUser-declared canonical - https://www.mrisoftware.com/ie/products/real-estate-financial-software/Google-selected canonical - https://www.mrisoftware.com/uk/products/real-estate-financial-software/'Has anyone else experienced this problem?How can I get Google to select the correct, user-declared canoncial? Thanks.
Technical SEO | | nfrank0 -
Redirect Chain Domain
MozPro is highlighting some redirect chain issues with our domain that I do not recall ever setting up in our redirect list. In our Moz Pro Campaign I see the Site Crawl has flagged 36 Redirect Chain Issues. I understand how the redirect chain errors can happen but I do not recall ever manually redirecting our domain, yet I have http://stickylife.com, https://stickylife.com & https://www.stickylife.com all associated in one of our redirect chain errors. When looking at our redirect files I do not see any of these domain redirects and wonder how this has happened and how to fix it. It appears as though our HTTP and HTTPS is causing some redirection. I wonder if this is coming from our DNS settings?
Technical SEO | | StickyLife0 -
URL Structure
Hi, Hope you are all well. On our website we have a 'blog' and a 'news' section. The blog is located on "/blog" - but when you click on a post the url structure changes to /name-of-article and the blog subdomain isn't included. Would it be better to have "blog/name-of-article as this would then make the blog perform better in search results? Also, if our news page is under /news - but when you click on an article it changes to /news-article/name-of-article Wouldn't it be better to have /news/name-of-article Thanks a lot!! 🙂
Technical SEO | | National-Homebuyers0 -
Magento URL change
We have a Magento website parked at HostGator. The site is comprised of both a PC and a mobile version. We changed the URL to a new one ... We made the domain changes in the ‘core_config_data’ (phpMyAdmin) ... We flushed the cache in the ‘File Manager’ part of cPanel (regular and mobile version) Currently we can access the http://newsite.com (on a desktop) with no problem ... We can also access http://m.newsite.com BUT… only from a desktop PC. When we try http://newsite.com from a MOBILE device, it routes to: http://m.OLDsite.com (it keeps going to the old URL) Need some help please. Thanks in advance!
Technical SEO | | Prime850 -
Should I change or redirect this URL?
Happy Friday everyone! I just noticed that one of our Attorney Profile's url's is wrong. We used to have someone named "Dana Fortugno" as our Family Law attorney, but when he left, (over two years ago) we hired "Scott Finelli." The person who setup the site, just changed the information on the page not url. So instead of it saying "http://www.kempruge.com/scott-finelli-jd-llm/;" it says "http://www.kempruge.com/dana-fortugno-jd-llm/." I'm considering taking all the content on the page with the wrong url, copying it to a new page with the correct URL and 301 redirecting (what would now be a blank page) to the new page with the correct URL. Is this the best way to handle this? Also, I don't believe there are many SEO concerns regarding the pages specifically. The profile pages aren't what we rank for in any of our Family Law related keywords. I am worried about having a completely blank page that just 301 redirects as looking bad to google, but not sure if it would? As always, thank you for your time and any assistance you can provide. Ruben
Technical SEO | | KempRugeLawGroup0 -
Content and url duplication?
One of the campaign tools flags one of my clients sites as having lots of duplicates. This is true in the sense the content is sort of boiler plate but with the different countries wording changed. The is same with the urls but they are different in the sense a couple of words have changed in the url`s. So its not the case of a cms or server issue as this seomoz advises. It doesnt need 301`s! Thing is in the niche, freight, transport operators, shipping, I can see many other sites doing the same thing and those sites have lots of similar pages ranking very well. In fact one site has over 300 keywords ranked on page 1-2, but it is a large site with an 12yo domain, which clearly helps. Of course having every page content unique is important, however, i suppose it is better than copy n paste from other sites. So its unique in that sense. Im hoping to convince the site owner to change the content over time for every country. A long process. My biggest problem for understanding duplication issues is that every tabloid or broadsheet media website would be canned from google as quite often they scrape Reuters or re-publish standard press releases on their sites as newsworthy content. So i have great doubt that there is a penalty for it. You only have to look and you can see media sites duplication everywhere, everyday, but they get ranked. I just think that google dont rank the worst cases of spammy duplication. They still index though I notice. So considering the business niche has very much the same content layout replicated content, which rank well, is this duplicate flag such a great worry? Many businesses sell the same service to many locations and its virtually impossible to re write the services in a dozen or so different ways.
Technical SEO | | xtopher660 -
Domains and subdomains
When I started a campaign for my message, I got the message: "We have detected that the domain www.vamospaella.com and the domain vamospaella.com both respond to web requests and do not redirect. Having two "twin" domains that both resolve forces them to battle for SERP positions, making your SEO efforts less effective. We suggest redirecting one, then entering the other here." I wasn't sure whether I had said it was a subdomain when in fact it was a domain (or the other way round), so I started another campaign for the same website using the other option and the message didn't come up. However, I still don't understand what you meant by this and whether it's an issue. When I search for my website in Google, it shows as vamospaella.com when other websites come up as www. and then their domain name. If it is a problem, is it to do with my hosting package and how it's set up or is it to do with my local site on my computer? I did ring my web host, 1&1, but they said they couldn't see a problem. Please can you let me know how I can resolve this as my ranking is still quite low in Google and I'm not sure why. If it is because of "twin domains", then will Google see my content as duplicated and keep me low in their rankings? I'm new to SEO and not a website novice, so please answer in lay terms! Thanks Melissa
Technical SEO | | melissa10 -
Domain with more Languages
Hey folks! I was wondering what you would do. I do have a Website. The website is provided in 8 other languages. Right now every language has it's own Domain name. The domain name is always the country in the language. I'm thinking about combine everything to one domain and hope to get some great linkjuice from the other 7 domains. So it would be www.example.com/en/ www.example.com/fr/ and so on. How do you handle that. Would this have a big positive impact on that one domain I'm forwarding to?
Technical SEO | | leitpix
I really think so!0