Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Large Global Site Structure
Hi, I have a question about the advised structure for a website that I'm currently building. It's a large international brand with it's main office in the UK. The main website is the .com but there is a growing international franchisee network. I've built the .com site on Wordpress but I'm not sure if I the best way forward would be to create each international website on a separate hosted site or just include it in the .com Wordpress structure using the The WordPress Multilingual Plugin. So to sum up... should I build the entire global network on one domain and then use WPML plugin or should I build separate websites for each International franchisee? Hope some one can educate me on the best route to take. Thanks Moz Community
Web Design | | SeoSheikh0 -
Is having copy above the fold still beneficial for SEO
Many sites nowadays are approaching a section based approach with copy further down the page. Is it still valid to keep copy above the fold now for SEO. Will it impact on SEO if you have to scroll down the page?
Web Design | | vortexuk0 -
Help needed on URL structures
I am busy structuring URL's for a client and an issue i have come across is as follows: i have a URL that is a long one, we cant remove words in it so the question is which one is better structurally: root/courses/businessmanagementandadministration.aspx or root/courses/business-management-and-administration.aspx please help.
Web Design | | nick_pageone0 -
Live website is an addon domain - Need site old development url inaccessable from live domain
Hi everyone, I have a website which is built in Joomla 2.5. The development site is located at www,abc.com/subdomain/. We have set the site live using an addon domain which is www.xyz.com. The problem is, www.abc.com/subdomain/ is still accessible and being crawled by Google. How is the best way to make the development url inaccessible? Any help would be appreciated!
Web Design | | DougHosmer0 -
Does DNS location affect international SEO?
Hi All Smart SEOmozers! I have another dumb question =] I have almost no knowledge on how DNS works and all the website background work. I understand that DNS is the server that translates a domain name to the IP address. Furthermore, I also know that IP Address location or web host location plays a small factor in international SEO. Webhosts usually provide the DNS service as well but for this case ABC Company uses a different domain service, diferent DNS service and different webhost service so things get complicated. So the question, does the location of DNS service we use affect International SEO like how the location of the webhost does. Thank you in advance for your help!
Web Design | | TommyTan0 -
Does changing nameservers and a new site design affect SEO dramatically
We are about to change nameservers and upload a new website design design, completely rebuilt website to that new hosting, will this effect our seo efforts previously and have an effect on our SEO rankings?
Web Design | | CompleteOffice0 -
Page Title or Search Friendly Urls?
We are currently auditing our website as part of our SEO strategy. One item which hascome up is the importance of search friendly urls against the search engine friendly page titles. Do url's or page titles carry more relevance than the other in search engines? Obviously the ideal would be to have both to maximise search impact but do either carry more importance. Thanks
Web Design | | bwfc770 -
Changing the domain - To do or not to do - that is the question
Hello, I am in the process of updating my website (hopefully to improve SEO). It is a major update as we are going from 20,000 product line to 200,000 product lines and hiring two marketing people to work on more content. Unfortunately, I think my domain isn't the best i.e. www.direct-tractorparts.com Some of the parts have long part numbers and I am worried that my URLs will be too long i.e. I will lose link juice from the search engines. typical url www.direct-tractorparts.com/6XY41210TTTAZCAz2 I am thinking of launching the new site with a shorter domain name rather than updating the old site. My site doesn't have that many backlinks but generates a decent level of business for my company. Not sure if it would be worth setting up another brand or not Anyone got any advice?
Web Design | | DavidLenehan0