Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is having copy above the fold still beneficial for SEO
Many sites nowadays are approaching a section based approach with copy further down the page. Is it still valid to keep copy above the fold now for SEO. Will it impact on SEO if you have to scroll down the page?
Web Design | | vortexuk0 -
URL Help
Will the following urls will be considered as two different urls? 1. www.example.com/key=value1& key2=value2 2. www.example.com/key2=value2 & key=value1
Web Design | | prsntsnh0 -
What is the best way to point newly built website on new domain name to the original more well known domain?
Live website on abc.com domain is being totally redone and moved to a new platform. to facilitate full testing and compliance, the new look and content was built on a different url - xyz.com for example. Now that all content is approved and testing, we want people visiting the abc.com domain to see the xyz.com website without necessarily redirecting abc.com to xyz.com What is the best to do this? Thanks all
Web Design | | wkismb0 -
What are some of the best Word Press themes for SEO?
Does anyone have any suggestions on the best wordpress themes to use? Here is one that has been recommended to me before: http://www.smallbiztheme.com/
Web Design | | webestate0 -
Site Re-Design - Running old XML site map for 301's
Hi all, We are going to launch a new site design for our current e-commerce site. I have taken this opportunity to change some categories due to keyword research and all old categories will be 301ed to best fitting new category. So I have 2 questions about moving stuff over; 1. I read that leaving the old xml site map running for the first week, would help, because this would give crawlers the chance to run through the site and follow the 301s, which would help pass the juice. How true does this sound? 2. I was thinking of re-writing all category and sub category titles, meta descriptions and on page content. The positive of this is loads of fresh content - but doing this all at the same time with the new site launch might see some major dropping in search ranking. I've identified our top traffic keyword terms/pages, would it be more wise to leave these pages, and change the others, or would the total new fresh burst have a better impact? Cheers
Web Design | | ToxicFox0 -
Development site accidentally crawled - Will this cause problems?
We are currently developing a new version of our website and to make it easy to access for all team members, we just set it up on a server accessible via a publicly accessible domain name (ie devsite.com). There has been no SEO and no links created to this site, or so I thought. Recently, I found out that Google somehow found its way to this development site and has been indexing the pages! I was a little alarmed, as there are no links to the domain and we'll soon be transitioning all the content over to our primary production domain. I immediately created a robots.txt file to disallow access to the entire development domain. My fear is that there may be some duplicate content penalty if Google sees that the content that is on our new site (once it goes live and is pushed to our REAL domain name) was previously indexed on our test domain. We're slated to launch in 2-3 weeks. Is there anything else I should do? Should I even be worried? I'm probably a bit paranoid, but given the amount of time and effort that has gone into this new site, I love any advice or thoughts. Thank You!
Web Design | | AndrewY0 -
Which ecommerce platform is best for SEO?
We currently run our eyewear store on osCommerce. However, for various reasons we are considering a redevelopment onto another platform, the most obvious choice being Magento. What might the advantage and improvement in SEO with such a change and is the pain worth the gain???
Web Design | | seanmccauley0 -
SEO to Drive Leads
Dear friends, I have clients asking for SEO to drive leads. I have no experience on it, how could I do that? Is there any web layout I can suggest to my client? Another question is how do I implemented the layout on the web structure? I usually structuring my website based on keywords. For example: Homepage niche: plumbing Category: london plumbing, plumbing service, etc Content: Finding London plumbing service in your area, what kind of plumbing service you should pay attention? So, based on these layout, my SEO efforts goes to the content, instead of the homepage. Now, the question is how do I increase conversion with these strategy? Is placing a form above the fold would be wise, then place the content below it. Please advice 🙂 Thanks for your kind support, Albertus
Web Design | | toeplly0