Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Can the design still be considered adaptive if the URL is different?
I was under the impression our site had a mobile dedicated design, but my developers are telling me we have an adaptive design. The mobile site is set up different and has different content and the url is as follows: www.site.com/MobileView/MobileHome.aspx Can it still be considered adaptive if the URL is not the exact same? Hopefully this make sense and I appreciate anyone's input!
Web Design | | AliMac260 -
Can forwarding users from one domain to a different domain damage rank and authority of first domain?
Preliminary Explanation: We launched a new website a couple months back but haven't had much luck in Google taking notice. One of the main attractions to our site is an old flash app that was made nearly a decade ago. As the original developer has long ago moved on and we are unable to figure out how to integrate it with our new site, we've been stuck hosting the flash app on a different domain. As such, users who come to our site and want to use the app must immediately navigate away from our site to this other domain. This has caused our primary domain's bounce rate and average site time to plummet while raising it for the other domain. My question: is this damaging our search rank and page authority with Google for this primary domain/site and counter-acting any other positive SEO changes we can make? How much weight does Google give towards bounce rate/average site time spent by users in its overall calculations for search rank and page authority? Our average site time for this primary domain is resting currently at 50-60 seconds, while for the secondary domain that hosts the old flash app it is 4-5 minutes.
Web Design | | Closetstogo0 -
Sitemap created on client's Joomla site but it is not showing up on site reports as existing? (Thumbs Up To Answers)
I am working with a web developer who built our client's site in Joomla. I seem to have a lot of issues with Joomla based sites. Any how, the site is www.pitgearusa.com and when we run site reports it is showing there is no xml sitemap. However he used a popular Joomla plugin for sitemaps called Xmap. Here is their url: http://www.jooxmap.com/ Can anyone provide any advice on what the website developer needs to do in order for the xml sitemap to function and "show up" on reports? Thanks Mashed Up
Web Design | | Atlanta-SMO0 -
Footer link back to developers domain
I have read a lot about where it is suggested to either not put an attribute link in the footer of a clients site or to no follow it. But I have a little bit different take on the question. How does this work on a large scale? Are these manual penalties, or are they automatic? By large scale, I am talking about big cms programs such as Wordpress, Joomla, and the likes of those. They all have links back to their site in the footer of the default templates. Is this bad? Does it not rally matter on the scale of companies such as this?
Web Design | | LesleyPaone0 -
Totally flat URL structure
Hi Mozzers! I've just been viewing a website with a flat URL structure - the site has a definite structure - with various sections - and yet the URL structure doesn't reflect this... The developer tells me this is purely for SEO purposes! Would be interested in your thoughts...
Web Design | | McTaggart0 -
Hey on some of my report cards its saying im not using rel canonical correctly how do i change this on my site?
on some of my report cards its saying certain things featured on my services page are actually linking to my blog or something. and its saying im not using rel canonical correctly. can you help me out?
Web Design | | ClearVisionDesign0 -
Preserve Rankings When Switching to a New Site
Hello community! I have a quick question for you regarding preserving my SERPs once I switch a development site to replace the current production site. Basically, we are switching to a new CMS and will be featuring the same content, architectural layout, URL structure, etc. Again, the only difference is that it's going to be on a new CMS. Upon switching to the new version of the site, what would be the best strategy for making sure we preserve our rankings for content already ranking highly within the search engines? Likewise, is there anything additional we may be able to do right-off-the-bat in order to assist content that may not be ranking highly in the SERPs, rank more highly?
Web Design | | NiallSmith0 -
Dynamic pages and code within content
Hi all, I'm considering creating a dynamic table on my site that highlights rows / columns and cells depending on buttons that users can click. Each cell in the table links to a separate page that is created dynamically pulling information from a database. Now I'm aware of the google guidelines: "If you decide to use dynamic pages (i.e., the URL contains a "?" character), be aware that not every search engine spider crawls dynamic pages as well as static pages. It helps to keep the parameters short and the number of them few." So we wondered whether we could put the dynamic pages in our sitemap so that google could index them - the pages can be seen with javascript off which is how the pages are manipulated to make them dynamic. Could anyone give us a overview of the dangers here? I also wondered if you still need to separate content from code on a page? My developer still seems very keen to use inline CSS and javascript! Thanks a bundle.
Web Design | | tgraham0