Can SEO Moz perform a full site crawl and provide a report showing all URLs within an existing domain?
-
We are conducting a site redesign and need to get an idea of all pages that are out there on our domain (in some report fashion). This would help for discovery and cleanup as we re-work the site and move to a new CMS.
Thanks
-
Hey Stephen,
THanks - I have heard of Xenu many times before. I'll try that out.
I appreciate the input!
-
just going to throw in an alternative for this type of thing Xenu Link sleuth - free, easy and powerful.
Yes that site looks like ass but the tool is great. Dr Pete and others have done "how to use" posts on seomoz - just search xenu on seomoz
-
Yes - that's exactly it. Can I just do an export of the campaign crawl? or is that found somewhere else? Forgive my ignorance...still new to the tool and just can't seem to find.
-
Are you looking at your crawl report? It is available in an Excel file which provides the exact information you desire.
-
Hey Ryan,
Thanks for the response.
It's not a huge site. I would say there's probably around 600 total pages. It definitely looks like the SEO crawler has crawled the entire site, but I can't seem to find a way to just get a one-shot view of all the pages it's accessed.
Any ideas on where this can be found?
Thanks again!
-
How many pages are on your site?
The SEOmoz crawler will do exactly what you want but there is a URL limit. You should also understand it will experience the same limitations of any crawler. It wont list pages which behind a user log-in nor pages which are blocked by robots.txt. I am pretty sure the SEOmoz crawler doesn't list secure pages either but last I heard they were working on including them.
If your site has any island pages they wont be discovered either.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Are provincial third-level domains bad for SEO?
My prospect's domain ends in ".on.ca" (Ontario, Canada). The structure of their site is companyname.on.ca (main page) and all other pages are sub-folders (companyname.on.ca/page-name-1 All pages are no more than two levels deep. I'm wondering if anyone knows if the provincial sub-domain (.on.ca) presents an SEO challenge?
Web Design | | 22Eighteen1 -
Domain Authority
I have a website which has been up and running for around 7 years. It contains relevant content, receives somewhat timely content updates and has not been involved in any poor marketing activities. http://www.aleutianadventures.com/ That said, it has a very low domain authority and bounces around quite a lot in search results. Any thoughts on how to improve the sites current state. Thanks a bunch!
Web Design | | APICDA0 -
How do you influence the default site title?
Hi, We have noticed that on brand searches, a site's page title is replaced with the name of the site or the business, we can understand that this is due to the fact that a CTR enticing title is not as important when the customer is looking for a certain brand. What tells Google what company name to display in this instance? We're having trouble with our French site displaying the page title, we are moving the position of the title code earlier in the page, but can't see why a) Telefleurs is not displaying the title chosen and b) why it is displaying EuroFlorist when our French brand is Telefleurs. Any advice on this would be much appreciated! Thanks, Sam JgLwnGV.png
Web Design | | seoeuroflorist0 -
Migrating a site to Wordpress
I've recently been converting our old website to a wordpress based website and been working on the new version of the site on a subdomain. Now at the stage when I am getting ready to let the site go live and just wondering exactly how to do this so I have minimal downtime? Looking in the wordpress control panel there is the setting to enter the address of the site if you want it to be different from the directory it has been installed within - is this a good idea (i.e. is it stable if I do this? good for seo, bad for seo or makes no difference?)? or should I manually install everything in the root myself (if I do this is there a way to direct people to the temp version of the site on the subdomain? Any tips, do and don't s would be appreciated as I want to do this right!
Web Design | | Jon-C0 -
Ecommerce SEO - product sort order
Hi, I've been trying to find the answer to this in google but having no luck. In the current era, is it damaging to have products ordered randomly in an ecommerce website? Also, how long would you suggest is a good length of time to establish your natural rank? Ive launched and still work on several succesful ecommerce sites, but have recently launched a completely new venture - brand new url, brand new site and it has been live for around 5 weeks now, and although it is being found in search, it isnt doing as well as i'd like using the moz pro tools ive picked up some issues and have in the last few days tweaked page titles, added 'nofollow' to all my filters, added content etc, so I feel as though ive reset the clock. the site (it's an adult site by the way) is www.lovesauce.co.uk - would appreciate some feedback from the pro's
Web Design | | tom.dollar0 -
What site do you admire/like for its SEO - technical, content, whatever - and why?
I am gathering examples of great SEO'd sites and would appreciate your examples. The rationale can be anything - great SEO structure, great linking, solid content - you think stands out. Thank you!
Web Design | | josh-riley0 -
Google search issue with exact domain
We had a site from Feb-2011 to Nov-2011 at the domain amcoexterminating.com. The site was pure HTML/CSS and the daily unique visitors steadily increased over that time. So all was fine. We then moved the site to a CMS (Joomla) on Dec. 6th. From that day forward, the daily visitors went into the tank. Before the move, if you typed "amcoexterminating.com" or "amco exterminating" into Google search, the site would be the first result (as you'd expect since those are the words that make up the actua domain). But we tried this yesterday and the site did not come up at all. NOT GOOD. It would work in Yahoo or Bing, but not in Google. So obviously, the problem with Google search directly affected the daily visitors. We just checked Webmaster tools yesterday (yes, this should have been done sooner, lesson learned) and it said "Site has severe health issues - Important page blocked by robots.txt". It listed the "important" page URL and it was just a link to an image. Regardless, I wiped out the Joomla created robots.txt file and added a new one and made it just say... User-agent: *Allow: / About 14 hours later, after the new robots.txt file was recognized by Google, the "severe health" message went away. However if I search in Google for "amcoexterminating.com", it still doesn't show up and the client is concerned (as they should be). Do you think the search engines just need more time to refresh? If so, once it refreshes, should the site show up first again right away? Or is it possible the robots.txt file had nothing to do with the issue? If so, what other things could I check into that might cause Google search to not find a site even if you search for exact domain name? Please share any and all things I should look into as I need to get this site showing in Google search again (as it was before moving to the CMS). Thanks!
Web Design | | MarathonMS0 -
Are tables OK in e-commerce sites?
I know that for best results with search engines, tables should not be used. However as I look to redesign our older e-commerce site, I find that not only our website but many other e-commerce sites use tables. Is it acceptable to use tables in e-commerce or am I just viewing outdated e-commerce sites. Could you recommend a good shopping cart solution? We currently use volusion. Thanks for your time and see you in Boston next week!
Web Design | | Lael0