Crawl Test is now On-Demand Crawl!
-
If you've been with Moz a while, you may have used our old Crawl Test tool. A year ago we launched an all new, campaign-based Site Crawl (with an entirely rebuild crawl engine), but Crawl Test fell into disrepair and we haven't had a solid tool for crawling non-campaign domains.
I'm happy to announce that we've just launched an all new On-Demand Crawl, built on the new Site Crawl engine, with a UI that's focused on quick insights. Moz Pro Standard tier customers can run up to 5 crawls per month at 3,000 page per crawl (crawls are saved for 90 days), with per-month limits increasing at higher levels.
Most On-Demand Crawls should run in a few minutes, making the tool perfect to get quick insights for sales meetings, vetting prospects, or analyzing competitors. We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl.
Try it out -- we'd love to hear your use cases (either here or in the blog post comments).
-
That's a big help thank you!
-
It's been a while since I tried Moz's crawler. This new "major refresh" looks great Dr Pete and Team Moz. Cheers.
-
thank you
-
On-Demand Crawl, built on the new Site Crawl engine, with a UI that's focused on quick insights
That is a huge plus!
Thanks, Dr. Pete hope you're doing well man.
Tom Zickell
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is on-demand crawl reporting numbers that are very different than the regualr site crawl by Moz?
When I look at the site crawl available through my dashboard, it reports 22.9K pages crawled. I just ran a on-demand crawl and it is only reporting 6K pages crawled. Why is the number so much lower?
Moz Bar | | ctripp10100 -
Moz Site Crawl Test 404
Crawled site a number of times using Crawl Test. Its reporting 404's from files that are actually present. What do you make of this? Justin
Moz Bar | | GrouchyKids0 -
Http:// https:// google search console crawl errors
How to direct http:// to https:// to get rid of 404 errors in google webmaster search console (http:// crawl errors)
Moz Bar | | O.D.0 -
Possible bug in Crawl Issues report?
Hi all - My crawl issues report shows 3 pages with missing titles. These are just google verification files and the robot.txt file - shouldn't these be excluded? Pages with Title Missing or Emptyas of May 11
Moz Bar | | A-Drive
URL Page Authority Linking Root Domains
https://www.mysite.com/googlea87e28121c071983.html
1 0
https://www.mysite.com/robots.txt
1 0
https://www.mysite.com/google9b9dc57478f61677.html0 -
Odd crawl test issues
Hi all, first post, be gentle... Just signed up for moz with the hope that it, and the learning will help me improve my web traffic. Have managed to get a bit of woe already with one of the sites we have added to the tool. I cannot get the crawl test to do any actual crawling. Ive tried to add the domain three times now but the initial of a few pages (the auto one when you add a domain to pro) will not work for me. Instead of getting a list of problems with the site, i have a list of 18 pages where it says 'Error Code 902: Network Errors Prevented Crawler from Contacting Server'. Being a little puzzled by this, i checked the site myself...no problems. I asked several people in different locations (and countries) to have a go, and no problems for them either. I ran the same site through Raven Tool site auditor and got some results. it crawled a few thousand pages. I ran the site through screaming frog as google bot user agent, and again no issues. I just tried the fetch as Gbot in WMT and all was fine there. I'm very puzzled then as to why moz is having issues with the site but everyone is happy with it. I know the homepage takes 7 seconds to load - caching is off at the moment while we tweak the design - but all the other pages (according to SF) take average of 0.72 seconds to load. The site is a magento one so we have a lengthy robots.txt but that is not causing problems for any of the other services. The robots txt is below. Google Image Crawler Setup User-agent: Googlebot-Image
Moz Bar | | Arropa
Disallow: Crawlers Setup User-agent: * Directories Disallow: /ajax/
Disallow: /404/
Disallow: /app/
Disallow: /cgi-bin/
Disallow: /downloader/
Disallow: /errors/
Disallow: /includes/
#Disallow: /js/
#Disallow: /lib/
Disallow: /magento/
#Disallow: /media/
Disallow: /pkginfo/
Disallow: /report/
Disallow: /scripts/
Disallow: /shell/
Disallow: /skin/
Disallow: /stats/
Disallow: /var/
Disallow: /catalog/product
Disallow: /index.php/
Disallow: /catalog/product_compare/
Disallow: /catalog/category/view/
Disallow: /catalog/product/view/
Disallow: /catalogsearch/
#Disallow: /checkout/
Disallow: /control/
Disallow: /contacts/
Disallow: /customer/
Disallow: /customize/
Disallow: /newsletter/
Disallow: /poll/
Disallow: /review/
Disallow: /sendfriend/
Disallow: /tag/
Disallow: /wishlist/
Disallow: /catalog/product/gallery/ Files Disallow: /cron.php
Disallow: /cron.sh
Disallow: /error_log
Disallow: /install.php
Disallow: /LICENSE.html
Disallow: /LICENSE.txt
Disallow: /LICENSE_AFL.txt
Disallow: /STATUS.txt Paths (no clean URLs) #Disallow: /.js$
#Disallow: /.css$
Disallow: /.php$
Disallow: /?SID= Pagnation Disallow: /?dir=
Disallow: /&dir=
Disallow: /?mode=
Disallow: /&mode=
Disallow: /?order=
Disallow: /&order=
Disallow: /?p=
Disallow: /&p= If anyone has any suggestions then please i would welcome them, be it with the tool or my robots. As a side note, im aware that we are blocking the individual product pages. Too many products on the site at the moment (250k plus) which manufacturer default descriptions so we have blocked them and are working on getting the category pages and guides listed. In time we will rewrite the most popular products and unblock them as we go Many thanks Carl0 -
Moz Crawl Report showing non-existent Duplicate Errors since new reporting layout
Hi Moz Community, Since Moz changed to the new style of Crawl report, we've seen a jump in duplicate errors for our site. These duplicate errors do not exist and were not present on the Crawl reports before the report change and also we have not made any changes to the flagged pages on our site since then either. When you download the report data in csv it appears that the Moz report is mixing up data for two or more pages on the site. e.g.in csv for 'Page1' data, it will show the meta description for 'Page2' and 'Page2' shows that for 'Page1', so this then gets flagged as duplicate, however looking at the actual Meta description assigned onsite, both Page 1 and Page 2 are completely unique. Has anyone else experienced this and Moz Team - are you looking into this? Thanks, V
Moz Bar | | WWTeam1 -
Moz Crawl Test: Referrer is sitemap.gz?
Hi,
Moz Bar | | Titan552
I'm looking at a crawl test report, and I'm seeing that most of the pages have the sitemamp.gz file listed as the referrer. As I recall in my other reports the referrer is usually the root domain - unless of course there's a redirect. Does having sitemap.gz as the referrer indicate a problem? If so, what problem does it indicate? Thanks!0 -
Is it possible to extend my crawling date in SEO Moz?
My web site was crawled by MOZ before week, next crawling date is tomorrow. Because of some reason I am not able to take any action on last week MOZ report.I want to extend MOZ next crawling date, Can I ?
Moz Bar | | ankit.rahevar0