Is there a tool that works like crawl test that allows more than 3000 pages?
-
I enjoy using crawl test inside moz but I need to find a way to crawl all the pages on a site. It would probably be in the neighborhood of 10,000 urls. Does anyone know of a free tool and if not is there a paid tool that will do this?
-
Just a note that we've discontinued the old Crawl Test tool and have launched an entirely new On-Demand Crawl tool based on our upgraded Site Crawl engine (launched last year). The new tool has an enhanced UI, entirely rebuilt back-end, full export capability, and will save your old crawls for up to 90 days.
We've written up a sample case study or logged-in customers can go directly to On-Demand Crawl.
If you're looking to crawl more than 3,000 pages, you can also use campaign-based Site Crawl in Moz Pro.
-
Hi Brad!
Donna is spot-on; I'm especially a fan of Screaming Frog.
That said, it looks like you're a Moz Pro subscriber, so you can use your Moz Analytics campaigns to crawl any site up to 50,000 pages.
Whenever you set up a new campaign, it automatically crawls the site it's set to track. It then updates every week, with the crawl results surfaced in Crawl Diagnostics. It looks like your account can hold up to 5 active campaigns, but they can always be deleted and more can be added.
Does that help at all?
-
Hi Brad,
Yes there are a couple. The two that come to mind are Screaming Frog and Xenu. Screaming frog is very popular among SEOs. The free version of will allow you to crawl up to 500 pages. A licensed version will allow you to crawl 10,000 pages or more and costs 99 pounds / year. (It's an Oxfordshire, UK company.) Highly recommended.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How do I see amount of traffic to a particular page on a website?
I am trying to figure out how much traffic goes to a particular page on a website. How would I go about finding this?
Moz Bar | | jbcorcoran6 -
How Can I Batch Upload URLs to get PA for many pages?
Howdy folks, I'm using advanced search operators to generate lists of long tail queries related to my niche and I'd like to take the batch of URLs I've gathered and upload them in a batch so I can see what the PA is for each URL. This would help me determine which long tail query is receiving the most love and links and help inform my content strategy moving forward. But I can't seem to find a way to do this. I went to check out the Moz API but it's a little confusing. It says there's a free version, but then it looks like it's actually not free, then I try to use it and it says I've gone over my limit even though I haven't used it yet. Anyone that can help me with this, I'd really appreciate it. If you're familiar with SEMRush, they have a batch analysis tool that works well, but I ideally want to upload these URLs to Moz because it's better for this kind of research. Thanks!
Moz Bar | | brettmandoes2 -
Title tag not registering on page optimization
I added a link to the "page optimization" section at moz. When I submit the url to be reviewed (click "optimize") it doesn't find the title tag, although it is definitely there e.g this page
Moz Bar | | digitalalchemy
http://naamanewman.co.il/photosGalleryID.aspx?id=34 Anybody as any Idea why is that? Maybe the tag placement on the DOM (it's not directly under the opening head tag).0 -
How can I find duplicate pages from a Moz Crawl?
We have many duplicate pages that show up on the Moz Crawl, and we're trying to fix these but it's very difficult because I can't see a way to isolate the code where the duplicate is found. For instance, http://experiencemission.org/immersion/ is one of our main pages, and the crawl shows one duplicate of http://experiencemission.org/immersion. It appears that one of our staff manually edited the source code in one of our pages but forgot the trailing slash. This would be an easy fix but the problem is that this page is linked to internally on our website 2423 times, so it's next to impossible to find the code that is incorrect. We have many other pages with this same basic problem. We know we have duplicates, but it's next to impossible to isolate them. So my question is this: When viewing the Moz Crawl data is there any way to see where a specific duplicate page link is located on our website? Thanks for any and all help!
Moz Bar | | expmission0 -
804 : HTTPS (SSL) Error in Crawl Test
So I am getting this 804 Error but I have checked our Security Certificate and it looks to be just fine. In fact we have another 156 days before renewal on it. We did have some issues with this a couple months ago but it has been fixed. Now, there is a 301 from http to https and I did not start the crawl on https so I am curious if that is the issue? Just wanted to know if anybody else has seen this and if you were able to remedy it? Thanks,
Moz Bar | | DRSearchEngOpt
Chris Birkholm0 -
Use of Keyword Analysis Tool (espcially regarding search volume)
Hello! I am fairly new to Moz and just starting to explore the available Research Tools. At first glance, I was very excited about the Keyword Analysis tool, but I quickly realized that (a) I wasn't receiving the information I needed and (b) I didn't know what to do with the information it gave me. So... Two specific questions: Is there any way to see the Google Search Volume instead of the Bing Search Volume? The overwhelming majority of our organic site traffic comes from Google, so it seems silly to look at the Bing search volume. How do you deal with the fact that this tool only returns search volume based on an exact match? As we know, there are some many ways the same general query could be entered, but it seems ridiculous for me to sit there and run the tool against every iteration I can think of just to get a general sense of the search volume. Two general questions: What do you find helpful about the Keyword Analysis tool and how does it inform your SEO work? What other tools do you recommend to help determine Google search volume for keywords? Thanks in advance for your help and advice!
Moz Bar | | Marketing_Dept0 -
KW difficulty tool?
I can't get the keyword difficulty tool to work in AU at the moment. Can you get AU KWs to work? Any tips on us getting it to work? We've: deleted KWs and started over. logged out & logged back in. tried from multiple computers. waited hours. US search works fine for us - AU not at all. Thanks!
Moz Bar | | WME0