Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How can I increase my website speed?
Hi, I hope you are doing good. My website speed is too much slow. Mobile speed is 12 and Desktop speed is 39. Please check my website speed.
Web Design | | Bigbrand
&
What can I do for my website to increase speed?
What is best way to increase website speed. Here is my website: https://www.myqurantutor.com/0 -
Best URL when adding an SSL certificate . . .
Our (small) company is a little late to the party on this, and we've only just realised that we're better off with an SSL certificate for our website. (Yes I know, I know, but we dropped SEO some time ago after getting severely bitten by a certain Penguin, and are only just making tentative step back to it after those intervening years, so we're running to get back up to date with these things.) This has now been implemented, but our web guy has dropped the 'www' element during the process. Our http://domain.com address has always historically been redicrected to our main http://www.domain.com address. Now our web guy has implemented the SSL cert, our website URL is appearing as https://domain.com, and he has redirected the http://www.domain.com to that new URL. Obviously all our historic (and more recent) link building has been to the http://www.domain.com address. Is this an issue, should the new Https URL keep the 'www', or does it make no difference what so ever? Conversely could it actually be of benefit dropping the 'www.' because our keyword specific product URL's are now 4 characters closer to the http and 4 digits shorter? Finally, on the links we have control of (professional trade associations etc) do we need to ask them to change the links to the new Https address, or does the transition from Http to Https make no difference?
Web Design | | Wookii0 -
Does blogger.com stink for SEO purposes?
Three different SEO guys suggest moving my blog to another platform. " I think migrating to a more robust CMS platform like Drupal or Wordpress would be a wise decision." But, they never say why moving would benefit SEO. My blog is on a custom domain, has lots of original content and has decent organic traffic to begin with. I think I have other SEO issues to deal with before bothering with a new platform. Does blogger stink for SEO? Why?
Web Design | | Eric_haney0 -
Manufacturer, New Direct-to-Consumer Site (Separate Site, or Sub-Domain?)
Hi All! Working with an established manufacturer, been around for many years, it's an internationally known brand, and their products are sold by thousands on distributors. They recently started a new website (separate from their old established B2B manufacturer site) which will be used to sell direct to customer. The new site is great, with a nice responsive design, clean look, flexible, etc. The problem is, it's a new site with low Domain Authority. The manufacturer's B2B site has been around a while, very high Domain Authority. So, I'd like to be able to harness all the link equity they've build instead of trying to optimize a brand new site. The problem with this old established site is that it IS in fact old. The design is terrible, it's not responsive, old code, bad look and feel, etc. We could incorporate the new B2C site (which has its own CMS) into a sub-domain, like store.site.com. But, I'd worry that site.com's crapiness will limit growth potential for the new pages at store.site.com. Same issue were we to add the new site into a sub-folder, like site.com/store/. On the other side, we could just keep the new site, with it's own domain, sitestore.com, and have product pages and/or category pages from the manufacturer's B2B site link to the relevant pages on the new B2C site. Thanks!
Web Design | | fiberglass0 -
Show new mobile site to 60% users & old mobile site to 40% users
Hi, We are planning to show new mobile site to 60% users & old mobile site to 40% users. We will show the old site to google crawler. Our old site has some interlinking through footer & content whereas the new site does not has it. We wanted to do this since our new site does not supports some browsers. Will there be an issue with Google on showing the site like this. The mobile site & desktop site will have same url across devices & browsers. Regards
Web Design | | vivekrathore0 -
Just How Bad is Adobe MUSE for SEO?
Adobe's new website builder "Adobe Muse" has a reputation for creating terrible code. I want to know if anyone has experience with the software and what your opinion is on just how bad the code really is for SEO. I'm currently using "Weebly", which is a similar, but more basic website building software. My results for SEO have been going well using that software, however it's limited in terms of building an aesthetically pleasing website design. On the other hand, you can build gorgeous websites with Adobe Muse, but I don't want to use it if it's going to prevent me from ranking. What are your thoughts?
Web Design | | Alchemist230 -
SEO list for creating the *perfect* website
If you could build your website from scratch and have your developers do anything you want (within reason), what list of SEO requirements would you send them? Does anyone know of any good articles on the perfect SEO wish list? Happy Holidays!
Web Design | | MirandaP1 -
Local SEO - Title Tag?
For www.bluedotlandscaping.com/fencing.htm in which we mostly only care about Greenvile and Spartanburg counties in SC. Is <title>Patio designs - Water features - Brick patio by Blue Dot Landscaping</title>
Web Design | | SCyardman
good... or or do you prefer... <title>Patio designs - Water features - Brick patio - Greenville, Spartanburg, Simpsonville</title> Thanks for your help, Rich0