Crawl test
-
I used to use the crawl test tool to crawl websites and it presented the information in a really useful hierarchy of pages. The new on-demand crawl test doesn't seem to do this. Is there another tool I should be using to get the data?
-
Perfect thank you!
-
Hey, thanks for reaching out to us!
With On Demand Crawl, you can sort by crawl depth, either in the UI or in the CSV export. This should give you the hierarchy that you are looking for. The higher the depth number the deeper the crawler was on your site. I hope that helps to answer your question, let us know if you need anything else.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Moz crawler not crawling on my site
Hi all, im facing an issue where moz crawler is unable to crawl my site. The following error keeps showing Our crawler was banned by a page on your site, either through your robots.txt, the X-Robots-Tag HTTP header, or the meta robots tag. This is my robots.txt file : https://www.wearefutureheads.com/robots.txt I'm not sure what else am I missing.. can anyone help
Product Support | | teikh0 -
Why are only two pages of my site being crawled by MOZ?
MOZ is only crawling two pages of my site even though there are links on the homepage, and the robot.txt file allows for crawling. Our site address is www.hrparts.com.
Product Support | | hrparts0 -
Website can't be crawled
Hi there, One of our website can't be crawled. We did get the error emails from you (Moz) but we can't find the solution. Can you please help me? Thanks, Tamara
Product Support | | Yenlo0 -
Crawl still in process for 3 days. Not sure why the site isn't being crawled
I added a new site to the crawl, but it seems to be stalled. It was supposed to crawl Feb 19, but it is still in process Feb 22. It tried to crawl the site and there was a robots.txt issue, but that issue was resolved way before the 19th. Not sure what is going on. this is for the clear lake campaign.
Product Support | | dpsoftware0 -
Is is possible to revert reporting to a past crawl date?
Site Crawl report defaults to the last crawl. Is there a way to get data from a previous crawl for comparison?
Product Support | | JThibode1 -
How to block Rogerbot From Crawling UTM URLs
I am trying to block roger from crawling some UTM urls we have created, but having no luck. My robots.txt file looks like: User-agent: rogerbot Disallow: /?utm_source* This does not seem to be working. Any ideas?
Product Support | | Firestarter-SEO0 -
SEO Campaign for testing site?
We are in the process of a website redesign and have initiated an SEO campaign on the current site to set benchmarks. Does it make sense for us to run a separate SEO campaign on the testing site as well? Michelle
Product Support | | Michelle-Meyer0 -
What is the difference between the "Crawl Issues" report and the "Crawl Test" report?
I've downloaded the CSV of the Crawl Diagnositcs report (which downloads as the "Crawl Issues" report) and the CSV from the Crawl Test Report, and pulled out the pages for a specific subdomain. The Crawl Test report gave me about 150 pages, where the Crawl Issues report gave 500 pages. Why would there be that difference in results? I've checked for duplicate URLs and there are none within the Crawl Issues report.
Product Support | | SBowen-Jive0