Is there a tool that works like crawl test that allows more than 3000 pages?
-
I enjoy using crawl test inside moz but I need to find a way to crawl all the pages on a site. It would probably be in the neighborhood of 10,000 urls. Does anyone know of a free tool and if not is there a paid tool that will do this?
-
Just a note that we've discontinued the old Crawl Test tool and have launched an entirely new On-Demand Crawl tool based on our upgraded Site Crawl engine (launched last year). The new tool has an enhanced UI, entirely rebuilt back-end, full export capability, and will save your old crawls for up to 90 days.
We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl.
If you're looking to crawl more than 3,000 pages, you can also use campaign-based Site Crawl in Moz Pro.
-
Hi Brad!
Donna is spot-on; I'm especially a fan of Screaming Frog.
That said, it looks like you're a Moz Pro subscriber, so you can use your Moz Analytics campaigns to crawl any site up to 50,000 pages.Whenever you set up a new campaign, it automatically crawls the site it's set to track. It then updates every week, with the crawl results surfaced in Crawl Diagnostics. It looks like your account can hold up to 5 active campaigns, but they can always be deleted and more can be added.
Does that help at all?
-
Hi Brad,
Yes there are a couple. The two that come to mind are Screaming Frog and Xenu. Screaming frog is very popular among SEOs. The free version of will allow you to crawl up to 500 pages. A licensed version will allow you to crawl 10,000 pages or more and costs 99 pounds / year. (It's an Oxfordshire, UK company.) Highly recommended.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is the Content Suggestions section under Page Optimization a TF-IDF Analysis?
If you start a campaign in Moz, go to page optimization, enter a URL and keyword, and go to the bottom where it says "Content Suggestions" is that basically do a TF-IDF analysis? I want to make sure I understand how that works. Thanks!
Moz Bar | | brettmandoes0 -
Moz Pro: Redirect Chain warning given to pages that don't have redirects
When I look up crawl errors for a page, I'm always told the page suffers from redirect chaining. However, when I do a redirect check (in this case, using the Redirect Path Chrome extension), it indicates that my page does not use a redirect. Why would Moz detect redirects, while no other redirect checker resource does? For example, this URL gets Moz's redirect chain warning: https://www.aem.org/news/january-2018/5-reasons-iot-projects-fail/ But there is no redirect associated with this URL.
Moz Bar | | jrichter0 -
MOZ Tool not reading my xml sitemap.
I have submitted an xml sitemap to Search Console back in December, and it's been approved, but the moz tool says there's no xml sitemap. Please help!!!
Moz Bar | | SEO_Tap220 -
605 : Page banned by robots.txt
Hello everyone, I need experts help here, Please suggest, I am receiving crawl errors for my site that is , X-Robots-Tag: header, or tag. my robots.txt file is: User-agent: * Disallow:
Moz Bar | | bhomes0 -
Canonical in Moz crawl report
I'm wondering if the moz bot is seeing my rel="canonical" on my pages. There are 2 notices that are bothering me: Overly Dynamic URL Rel Canonical Overly Dynamic URL - This notice is being generated by urls with query strings. On the main page I have the rel="canonical" tag in the header. So every page with the query string has the canonical tag that points to the page that should be indexed. So my question...Why the notice? Isn't this being handled properly with the canonical tag? I know I can use my robots.txt or the tool in Google search console but is it really necessary when I have the canonical on every page? Here is one of the links that has the "Overly Dynamic URL" notice, as you can see the the canonical in the header points to the page without the query string: https://www.vistex.com/services/training/traditional-classroom/registration-form/?values=true&course-title=DMP101 – Data Maintenance Pricing – Business Processes&date=March 14, 2016 Rel Canonical - Every page in my report has this notice "Using rel=canonical suggests to search engines which URL should be seen as canonical". I'm using the rel="canonical" tag on all of my pages by default. Is the report suggesting that I don't do this? Or is it suggesting that I should? Again...why the notice?
Moz Bar | | Brando160 -
Tool for seeing targeted keywords?
Hello Friends, Is there a tool that provides a list including targeted keywords and there ranking for those keywords? For example if I type in a website url it shows me which keywords they targeted, on which page and their ranking position. Thanks for helping out a foreign noobie;)
Moz Bar | | grobro0 -
I requested a new crawl, this was done but my dashboard only shows the crawl done last week?
We recently moved our old website to a new CMS and structure. there have been some configuration errors and I needed to make some changes with things like canonical url's etc. However I need to check if these changes have made a difference and requested a new crawl through the crawl test page. I was emailed each time that a new crawl had been done but my reporting and dashboards still only show data from the last scheduled crawl. Regards Chris
Moz Bar | | LRQA-Marketing0 -
Is there any tool that checks the rankings at any time I want?
My problem is that in Moz when I check ranking it shows me only on a weekly basis. Acording to MOZ I was on rank 11 yesterday. This is not what my SEO guy has sent me in his report. And usually he is right. Would be nice to have a ranking report for each day. What is the sense of a ranking tool if I don't know what happened between 2 ranking dates? How can I check that? Is there any rank history tool? Or a tool that will check in real time. The way I do it sometimes is to go on google.com/ncr to see the us version of google because I am in Europe. Rank tracker does not work for me at the moment. Get ranking button is inactive. My old keyword list has not been imported from SEOMOZ
Moz Bar | | Valentinopolus0