Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Does adding new pages, new slugs, new URLS in a site affects rankings and visibility?
hi reader, i have decided to add new pages to my site. if i add new urls, i feel like i have to submit the sitemap again. my question is, does submitting sitemap again with new slugs or urls affects visibility is serps, if yes, how do i minimize the impact?
Web Design | | SIMON-CULL0 -
How much is the site architecture impacting my site?
Hi there, I'm interested to learn how much the site archecture of griffith.ie (higher education) maybe impacting our rankings. In recent months there was some changes to the the faculty landing pages but not to the site archecture. The rankings in the last 6 months have dipped a little. There are two main path ways to get to the course. 1. Course finder - https://www.griffith.ie**/find-a-course** => 2. Faculties - https://www.griffith.ie**/faculties** Most of the SEO authority is coming through the Faculties pages as this is where all the courses are found in term of the url structure. For example; https://www.griffith.ie**/faculties/**business/courses/ba-hons-accounting-finance The UX on the site tells a different story and directs people to the course finder. /find-a-course Ideally, I feel the site would benefit much more if all the traffic was directed through the course finder however this would require (I think) a big redevelopment of the search tool and I feel we are diluting our efforts as when somebody arrives to the site through the homepage they go through the course finder and if they come through specific searches they get taken to the specific course page under the faculty section. the site has has this archecture for the best part of 4 years and I'm considering recommend a change if it would greatly improve SEO and UX. Any feedback on this would be great. Many Thanks Rob
Web Design | | robhough9091 -
Can anyone recommend a great programming company?
I have had terrible luck with programmers who seem to live in their own little world and never get things done on time. Can anyone recommend a great company here in the usa that you have used before that has done great work? I am looking at the nerdery. Anyone use them?
Web Design | | netviper0 -
Confluence and SEO
I think this is a difficult question so apologies in advance and any help would be appreciated! We currently have a large amount of support center content sitting on our main pages which we don’t think is very effective (mainly basic how to guides). We think it is difficult for visitors to understand and the UI is very poor. In order to solve this we’re currently moving this content onto a subdomain using Confluence, a wiki based team collaboration tool (from a company called Atlassian). What we’re planning on doing is very much like what Atlassian themselves have done on this page: https://confluence.atlassian.com/display/ALLDOC/Atlassian+Documentation What are the SEO issues / dangers that I need to consider before moving this content? I’m assuming that as this content will still be on the same domain then we can minimise link equity / authority loss by setting up re-directs to the new content. Also, has anyone had any experience of using Confluence and whether individual pages can be optimised for SEO? I notice that there are lots of add-ins that can be used, one of which is an SEO add-on which allows you to customise things like meta description tags.
Web Design | | RG_SEO0 -
Does DNS location affect international SEO?
Hi All Smart SEOmozers! I have another dumb question =] I have almost no knowledge on how DNS works and all the website background work. I understand that DNS is the server that translates a domain name to the IP address. Furthermore, I also know that IP Address location or web host location plays a small factor in international SEO. Webhosts usually provide the DNS service as well but for this case ABC Company uses a different domain service, diferent DNS service and different webhost service so things get complicated. So the question, does the location of DNS service we use affect International SEO like how the location of the webhost does. Thank you in advance for your help!
Web Design | | TommyTan0 -
Two sites in same industry and which shopping cart
Right. So I suspect I am going to sound paranoid here - but you'll all forgive me right?? I am sure I saw a reply to a question on the Q&A suggesting that it was a bad idea to have two sites in one industry as Google may see it as trying to get two bites of the SERP cherry... is this accurate? I have an existing asp.net site in the maternity wear industry here in Australia and am wanting to start another site to appeal to a different customer base... the market is quite broad. There will be a core list of products that are the same between the sites, but also some quite different products. Content, product descriptions and categorys will be different. I have another website that I bought with reasonable age and links in the industry that I was going to 301 to the new site to give it a kick in the juice. So, not wanting to deceive my customers in anyway, I was thinking I would call it a "division of" or "sister site to" the existing ecommerce site, with a single link back and forward between the two sites. Would there be anything wrong with this in googles eyes? Even with same contact details? They would be run on totally different platforms and hosted by totally different providers. Or would you keep them totally seperate and only have contact details in images? Or a step further and have totally different phone numbers etc? Then the shopping cart - I would love some suggestions on which opensourse cart to use, preferrably one that I can set up myself, and that has a good framework for seo. I want to use schema.org, authorship, seo friendly urls all of which I am having trouble getting out of the developer of my asp.net site.... I don't want the new site to be asp.net Thanks in advance!!
Web Design | | catfree0 -
How will a sites ranking be affected??
Hey guys, My MD has just put this to me, we have a site the is currently ranked in top 3 for all there chosen search terms. The company is undergoing an overhaul of their design and want a new site to match this. They have asked how changing the site design will affect their rankings. More content is going to be added with this there will be new pages links etc Any information would be greatly appreciated.. Thanks Anthony
Web Design | | Anthonykal-group0 -
Location in domain name but want to expand
my blog is ocpatentlawyer.com. The domain name has the OC letters in it. When los angeles internet users land on my blog, they don't click through to the contact page or pick up the phone to call me. Should i start a new blog with a LA focus then link to this new LA patent blog from my main OC blog? I'm hoping that the LA internet users who land on my OC blog will click through to the LA blog and feel as if I'm in their neighborhood. Should I leave my domain name the same but change the blog to something neutral such as XYZ patent attorney? The basic thing that I want to do is to convert the LA users from my oc blog. I'm unsure how to do it.
Web Design | | jamesjd70