Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I increase my website speed?
Hi, I hope you are doing good. My website speed is too much slow. Mobile speed is 12 and Desktop speed is 39. Please check my website speed.
Web Design | | Bigbrand
&
What can I do for my website to increase speed?
What is best way to increase website speed. Here is my website: https://www.myqurantutor.com/0 -
Shortened URL is breaking when URL is in Upper Case
Hi there, Currently I'm having some troubling mitigating an odd occurrence with some redirected shortened URLs being in upper case. Here is how they should be behaving - www.rhinosec.com/webapp -> https://rhinosecuritylabs.com/landing/sample-report-webapp-pentest/
Web Design | | BCaudill
www.rhinosec.com/network -> https://rhinosecuritylabs.com/landing/sample-report-network-pentest/
www.rhinosec.com/se -> https://rhinosecuritylabs.com/landing/social-engineering-example-report/ but when the /______ is capitalized - for example - WEBAPP, NETWORK, SE; WordPress either gives me a 404 or guesses the pages and lands on: NETWORK = https://rhinosecuritylabs.com/assessment-services/network-penetration-testing/
SE = https://rhinosecuritylabs.com/assessment-services/secure-code-review/
WEBAPP = 404 I was wondering if this discrepancy should be taken care of in the Htaccess file, Cloudflare, or WordPress redirect plug-in?0 -
Thoughts on our Agency Site
Hi all, We'd all welcome opinions on our digital agency site http://www.newbrandvision.com/. We are in the planning stage of launching a new site; and we'd welcome any UX or SEO thoughts. It's a strange one but our agency has been around since 2002, and we operate in the heart of London; however we don't rank anywhere near the first page for "digital agency in London" or any long-tail /semantics around this. We feel that it's pretty clear from search and when landing on the site what we are; but would welcome any general thoughts as to why we aren't ranked that highly. Much appreciated!
Web Design | | Tangent0 -
Domain Consolidation & Proper Linking Strategy?
We have a client that operates 5 gyms in 5 different part of Miami, and each gym has its own website. All sites rank well and have a a good pagerank. For the purpose of their marketing and brand they would like 1 website developed which includes all of their gyms which we are launching later today. Each gym will have it's own landing page within the website Should we redirect the URL's of the different websites to the individual landing pages on the new site or how should that work to minimize any penalties on our SEO. For example (these are fake url's): www.gymA.com, www.gymB.com, www.gymC.com, www.gymD.com The new url: www.gym.com New landing pages:
Web Design | | POPCreative
www.gym.com/gymA, www.gym.com/gymB, www.gym.com/gymC, www.gym.com/gymD Should we do a redirect from: www.gymA.com to www.gym.com/gymA www.gymB.com to www.gym.com/gymB www.gymC.com to www.gym.com/gymC www.gymD.com to www.gym.com/gymD Thank you in advanced. If there is a better way to do this, or anything extra I need to know, that would be great. Thanks!0 -
How will a sites ranking be affected??
Hey guys, My MD has just put this to me, we have a site the is currently ranked in top 3 for all there chosen search terms. The company is undergoing an overhaul of their design and want a new site to match this. They have asked how changing the site design will affect their rankings. More content is going to be added with this there will be new pages links etc Any information would be greatly appreciated.. Thanks Anthony
Web Design | | Anthonykal-group0 -
What are some good places to read about SEO with Joomla CMS?
I recently was asked by a potential client to work with them on Joomla based website. I very rarely come across people with Joomla based websites and am struggling to find good resources outside of Joomla's own community site. At this point, I am not very comfortable working with them on Joomla. Any tips or advice?
Web Design | | GCSMasone0 -
Effect of Off-Site Images
I'm getting to start work with a new client, and I've run across something I've never had to deal with before, off-site images. The site I'll be working on is for an appliance retailer, both online and physical. The way they've had their site built (not something I was part of) a third party company maintains the product inventory side of things. They're sourcing from about 35 different manufacturers, and this third party has direct access to the product information streams. They push the weekly updated information to my clients site. What this means, though, is that the product images don't live on the client's site. They're hotlinked from the third party's inventory doohickey. I've never seen something quite like this before. Has anyone else? Any ideas as to what problems I may face when it comes to on-site SEO?
Web Design | | MRCSearch0 -
What is the optimal URL Structure for Internal Pages
Is it more SEO friendly to have an internal page URL structure that reads like www.smithlawfirm.com/personal-injury/car-accidents or www.smithlawfirm.com/personal-injury-car-accidents? The former structure has the benefit of showing Google all the sub-categories under personal injury; the later the benefit of a flatter structure. Thanks
Web Design | | rarbel0