Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Site Migration due to Corporate Acquisition
Hey everyone, Wanted to check-in on something that I've been thinking way too much about lately. I'll do my best to provide background, but due to some poor planning, it is rather confusing to wrap your head around. There are currently three companies involved, Holding Corp (H Corp) and two operating companies, both in the same vertical but one B2B and the other is B2C. B2C corp has been pushed down the line and we're focusing primarily on H Corp and B2B brand. Due to an acquisition of H Corp and all of it's holdings, things are getting shuffled and Ive been brought in to ensure things are done correctly. What's bizarre is H Corp and it's web property are the dominant authority in SERPs for the B2B brand. As in B2B brand loses on brand searches to H Corp, let alone any product/service related terms. As such, they want to effectively migrate all related content from H Corp site to B2B brand site and handover authority as effectively as possible. Summary: Domain Migration from H Corp site to B2B Brand site. Ive done a few migrations in my past and been brought in to recover a few post-launch so I have decent experience and a trusted process. One of my primary objectives initially is change as little as possible with content, url structure (outside the root) etc so 301s are easy but also so it doesn't look like we're trying to play any games. Here's the thing, the URL structure for H Corp is downright bad from both a UX perspective and a general organizational perspective. So Im feeling conflicted and wanted to get a few other opinions. Here are my two paths as I see and Id love opinions on both: stick with a similar URL structure to H Corp through the migration (my normal process) but deviate from pretty much every best practice for structuring URLs with keywords, common sense and logic. Pro: follow my process (which has always worked in the past) Con: don't implement SEO/On-page best practices at this stage and wait for the site redesign to implement best practices (more work) Implement new URL structure now and deviate from my trusted process. Do you see a third option? Am I overthinking it? Other important details: B2B brand is under-going a site redesign, mostly aesthetic but their a big corporation and will likely take 6-9 months to get up. Any input greatly appreciated. Cheers, Brent
Web Design | | pastcatch1 -
Changing top level navigation between site sections
We've got an internal proposal to change our top level nav depending on the section of the site. For example, on our homepage it might read: Products, Library, About with relevant links dropping down below. As we have varied products, the drop down underneath it would include the various families. When arriving on the product family page the top-level nav would change to represent more specific offerings. For example: xxx.com 1. Products; 2. Library; 3. About xxx.com/xxx 1. Product family 1; 2. Product family 2; 3. Product family 3; 4. Library; 5. About What are the SEO/UX implications of this? It seems confusing but allows more specific navigation via the main nav depending on the section of the site. Also it seems that an alternating TLN might not be too Google-friendly.
Web Design | | gwelch0 -
Multi region stores, one domain
hi all Wondering if I could get some options on the pros and cons of using one domain for two stores targeting different regions. My website is a fashion retailer, targted at the female market. In addition to the stores the site has a number of blogs, articles etc on. At present we have a co.uk domain and a .com which targets the US market. The trouble with this split approach is having seo two sites at once, in addition to adding content to two sites etc. we are considering combining the stores into one domain and the having the U.S. Shop at /us and UK store at /UK - in wmt we will specifiy the directories as targeted to a specific location, the hotel language etc will be showing UK and U.S. English to further help geo targeting. we are thinking that, in theory, managing just one site will mean it's easier to build the authority and brand name. Pretty much all of the blog and article content is non region specific so it is relevant to both markets, it will also reduce the need to generate unique content for two sites at once. Is there any major downside to merging the sites like this. At present the UK site is da 4 and U.S. site da 0 - they are both pretty new and one of the problems we have at the moment is building up two sites at once. i welcome any opinions. thanks. Carl
Web Design | | WonkyDog0 -
Does interlinking on mobile site helps in seo & improvement in rankings
Hi, Does interlinking on mobile site helps in seo & improvement in rankings. Our desktop site & mobile site has same urls. Regards
Web Design | | vivekrathore0 -
2 or more wordpress installs on 1 domain
Hi ....I need to know if anyone has any concerns of running 2 or more WordPress installs on one domain and if so what would the concerns be. The situation is that we have a static html site that has a WordPress blog....and it is installed in example.com/blog/ . We will upgrade the static site to WordPress and this WordPress install will be in example.com/monkeys/ and then we will point its address to the root and make all other needed changes when ready to go live...but the install will stay in its own directory. So bottom-line is we have one domain with two WordPress installs that are each installed in their own sub directory and run from their own independent databases. There are quite a number of reasons for us to want to do this but I wont get into that......I just want to know if anyone has any concerns with this setup? The research we have done says its OK as well as our hosting provider....but before we get cracking we thought maybe one more "educated" input would make us feel more comfortable. Thanks
Web Design | | nomad-2023230 -
How does a Responsive Site kill SEO?
How does a Responsive Site poentially kill SEO? I've seen a few feeds on twitter how a website took a rankings dive after implementing a Responsive theme; yet, it's not clear to me what is actually going on within a Responsive site that would cause the SEO rank to tank? I can only speculate that it introduces a bunch of 404 errors, or that it changes all of the URLs into gibberish, so you loose all of the links coming into your website if not 301'ed? Can someone clarify, what are the actual mechanical issues on a Responsive website that becomes a concern to SEO? Thanks.
Web Design | | ExploreConsulting1 -
Responsive Vs Mobile Sites
I know this is some cutting edge technology, but I think that this will be a very important topic in the coming months, as html5/css3 becomses more and more the standard, or at least standardized, I think the topic of this in relation to SEO will also arise much more. My question is simple, is it better to code a responsive site, or a completely mobile site for a small company with no special needs (mobile ordering, ecommerce, etc...) I obviously know the visuall differences, and, personally, I think respomsive websites look better. From an seo perspective, my big thing is for the resizing, for example, with WordPress, when you reach the tablet size you can set the sidebar to basically display:none, can that impact your website? I would really appreciate any feedback
Web Design | | ZacharyRussell0 -
Two URLs with same content
We recently had a client who own multiple brands switch from having multiple urls to having a single domain with multiple sub domains. I've posted an example below to better explain. My question is the original url is still functional, so there are two urls with identical content, yet I haven't been getting a duplicate content error. Also, would a rel canonical link be beneficial in this case since the duplicate content is on two separate domains? My thoughts were to put a 301 redirect on the original pages so they permanently forward to the new sub-domain format. Is this the best course of action? If not, what would you recommend? Example: Original URLs
Web Design | | BluespaceCreative
www.example1.com
www.example2.com
www.example3.com
www.parentcompany.com New URLs
example1.parentcompany.com
example2.parentcompany.com
example3.parentcompany.com
www.parentcompany.com Let me know if this I need to clarify anything in better detail.
Thanks in advance!0