Tool to Generate All the URLs on a Domain
-
Hi all,
I've been using xml-sitemaps.com for a while to generate a list of all the URLs that exist on a domain. However, this tool only works for websites with under 500 URLs on a domain. The paid tool doesn't offer what we are looking for either. I'm hoping someone can help with a recommendation.
We're looking for a tool that can:
- Crawl, and list, all the indexed URLs on a domain, including .pdf and .doc files (ideally in a .xls or .txt file)
- Crawl multiple domains with unlimited URLs (we have 5 websites with 500+ URLs on them)
Seems pretty simple, but we haven't been able to find something that isn't tailored toward management of a single domain or that can crawl a huge volume of content.
-
@PatrickDelehanty The tool mentioned in the statement not only excels in the two areas mentioned earlier but also offers a wide range of additional capabilities. I recommend that you explore it for yourself! Best of luck!
-
@PatrickDelehanty The tool mentioned in the statement not only excels in the two areas ```
mentioned -
It seems to crawl all the wordpress folders and media files.
Is there not a tool that will tell you just your live website URLs, I'm after creating a site map and a mass re-organising content exercise, so want a list in excel of URLs.Any tips welcome
Thanks
Sarah
-
2nd Vote for Screaming Frog. Tried a lot of tools to pull info on all the URL's and this tool is by far the best one for the job.
-
Hi Felicia
Try ScreamingFrog - they crawl the entire site (you can configure how you want it to crawl your site) and have ways of creating a XML Sitemap for you.
The tool goes above and beyond those two areas as well and can do so much. I suggest you check it out! Good luck!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a tool out there to check any domain that might be pointing to my existing domain?
Is there a tool out there to check any domain that might be pointing to my existing domain?
Technical SEO | | adlev0 -
Acquired domains for SEO
hi there, For one of our insurance websites we acquired a domain, this domain is going to be redirected to our domain. After some research we discovered the domain we've bought also includes other domains which 301 redirect to specific 'insurance products'. These domains are also included in the deal. But what is the best technical solution for redirecting these specific redirected product domains? They already redirect to the product pages of the domain we've bought, so after redirect this domain, the sub (product) redirected domains will also link to us. It would be like this: A) www.sub-carinsurancesite.nl (301) -> www.domain-we-bought.com/car-insurance -> www.ourdomain.com/car-insurance
Technical SEO | | remkoallertz
B) www.sub-carinsurancesite.nl (301) -> www.ourdomain.com/car-insurance & www.domain-we-bought.com/car-insurance -> www.ourdomain.com/car-insurance etc0 -
Old Redirected Domain is replacing my current domain on SERPs
Hello everyone, All of a sudden a 2 year old redirected domain is replacing my current domain for 2 weeks now, my site is apitus.com and my old domain is aptitus.pe (the redirect is still working), however this only happens on my country google results (google.com.pe), if you check my site on google.com, everything looks ok even with a sitelink, which I no longer have on my country search results. Back to the issue, the first thing I thought was go to Search Console and take it out from the index, so I asked for access by uploading a file but since everything on that old site redirects to my current site I can't make such action. While still waiting for such access, is there anything else I could do?. Thanks in advance. PD: I'm adding the images of my SERPs CmzN8kY G3zZwwj
Technical SEO | | JoaoCJ0 -
Does the domain extension effect domain authority and ranking
We have produces a website on a .company domain extension. We have produced a good sized website with unique content. However the DA remains at 1. There are no high priority issues in the page crawl. We suspect that the domain extension may be causing a lower DA. Is this a ranking factor?
Technical SEO | | easydomains0 -
Different Domains on Same IP
Hello I'm just wondering how much of a difference it makes having links to a site from 2 separate domains that are on the same IP, compared to if the domains were on separate IPs? Thank you! Sam
Technical SEO | | wearehappymedia0 -
Mobile Domain Setup
Hi, If I want to serve a subset of pages on my mobile set from my desktop site or the content is significantly different, i.e. it is not one to one or pages are a summarised version of the desktop, should I use m.site.com or is it still better to use site.com? Many thanks any help appreciated.
Technical SEO | | MarkChambers0 -
How to 301 multiple domain names to a single domain
Hey, I tried to find and answer to this seemingly simple question, but no luck. So, I have one domain name with a website attached to it. I also registered all the other domain names that are similar to it or have different extensions - I want to redirect all the other domain names to my one main domain name without getting penalised by the big G. It looks like this: www.mainsite.com - this is my main domain I also have www.mainsite.com.au, www.mainsite.org, and www.mainsite.org.au which I all want to just redirect to www.mainsite.com I have been told that the best way to do this is a 301 redirect, but to do that you need to make a CNAME for all the other domains that points to www.mainsite.com. My problem is that I cannot seem to create a CNAME record for http://mainsite.com - I have it working for http://www.mainsite.com but not the non www record. What should I be doing differently? Is it just my DNS provider is useless? Thanks, Anthony
Technical SEO | | Grenadi0 -
When URL rewrite can lead to un pretty URLs
Hi Mozzers. I've a client that has done a little bit of mess rewriting the URLs of its site. In fact, also the data base driven URLs are rewritten, but the dev forgot to change the space with "-", so that now the 95% of the URLs are like this one: http://www.portalesardegna.com/search/Appartamenti e Residence/ Obviously not really a pretty URL. I am not so sure if this issue has an SEO consecuences (in fact, the site ranks pretty well also with those kind of url), but I am thinking more on usability issue. Could you suggest me any easy fix to this rewrite problem?
Technical SEO | | gfiorelli12