Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Title tag and user intent
I am just wondering if I create a page that present different e-bike kits and my title tag tag is "the best e-bike kits in 2019", will I rank on "e-bike kits" and "best e-bike kits" or on just "best e-bike kits" ? It seems that user intent can be tricky and sometimes a title tag can make all the difference. How about if I write "Explore Burgundy on a bike tour "to rank on "Burgundy bike tour", will I rank or is the user intent different when I write explore (meaning I am looking for something self guided instead of guided) Thank you,
Intermediate & Advanced SEO | | seoanalytics0 -
Onsite Videos- Multiple on a Single Page- How to Optimize?
I have a specific page which needs multiple videos. A primary video of the client (Youtube Video) and two secondary videos with patient testimonials (Wistia Videos). Here is the actual page: https://www.johnbarrasdds.com/houston-tmj-dentist/ My understanding is Google only values the first video on a page. Is this accurate and either way what is the best practice for how to post the second group of videos and gain SEO value? Thanks!
Intermediate & Advanced SEO | | mgordon0 -
Multiple product hierarchies (creation of refurbished products section) - best solution?
Hi all, I'm in discussion with a client who wishes to introduce a 'refurbished' products section to their website. This section will effectively replicate the structure of the 'brand new' products section. Unusually the key difference will be the fact that the 'refurbished' products section will feature significantly more products than the 'brand new' section, in the region of four times as many. As a guide the website currently stocks approximately 200 products across 8 core product areas. We have recommended that the two sections should be combined in order to prevent the creation of two separate product hierarchies. With 'brand new' / 'refurbished' products segmented via filter functionality. However the client is set on having two separate product hierarchies, i.e. a 'refurbished' section within a completely separate directory. Just wanted to crowd source opinion, in additionally to gaining insight if anyone has experience of a similar request. What solution did you implement? My feeling is that there is a high likelihood over time of the 'refurbished' section growing in authority and starting to outrank the 'brand new' products section. Not to mention a key missed opportunity to group and build authority / content within one product hierarchy. All thoughts and opinions much appreciated!
Intermediate & Advanced SEO | | 26ryan0 -
Webmaster Tools Content Keywords & Meta Tagging
In Webmaster tools , Content keywords give an indication of what Google thinks a site is about. This site is a health site ( online shopping - health supplements ) - but one of the terms it thinks the site is about is "Dollar" . I'm guessing this is because on every page there is Currency Selection from multiple currencies. How do I tell Google that this part of the page is nothing to do with what my site is about? Thanks for your reply in advance!
Intermediate & Advanced SEO | | s_EOgi_Bear0 -
What is value in a back-link from article with multiple links pointing to various other sites?
In a standard article with 400-500 words my site got a back-link. However, within the article there are 4 other links pointing to other external content as well (so total 5 links within articles all pointing to external sites, and 1 of the links is to my site). All links are to relevant external content that is. Question: wouldn't it be much more valuable for my site if only my site got a back-link from the article, as less link juice is now passed to my site, since there are 4 other links pointing to various sites from this same article? Or, is the case that given the other links are pointing to quality material it actually makes the link to my site look more credible and at the end of the day have more value. Conclusion: is it that on one hand less links in same article is better from a link juice perspective, however, from a credibility perspective it looks more convincing there are other links pointing to quality content?
Intermediate & Advanced SEO | | knielsen0 -
Multiple country site versions and hosting
If I intend to have 5 versions of an e-commerce site, one for each countries marketplace, would it be best to: a) Host each site in its own country b) Host all sites from 1 server/1 country I'm presuming that hosting in each country is better? But how quantifiable is this, and is it worth the extra hosting and management costs? Any advice gratefully received..
Intermediate & Advanced SEO | | cottamg0 -
Multiple sitemaps for one site?
Excuse my sitemap ignorance here. I've got a site and it's got a blog in a sub-folder. The blog gets updated frequently, the main site does not. Is it best to; a) Have 2 sitemaps.. one in the root and one in the /blog folder. b) Have 1 sitemap that is regularly updated The reason being, I know there's various plugins that create blog sitemaps on the fly, so that would be much easier than updating the main sitemap every time a change was made. If the answer is 2 sitemaps; Would you stop the root sitemap from detailing the contents of the blog folder or just update it every so often with the contents of the blog folder?
Intermediate & Advanced SEO | | PeterAlexLeigh0 -
Get-targeted homepage for users vs crawlers
Hello there! This is my first post here on SEOmoz. I'll get right into it then... My website is housingblock.com, and the homepage runs entirely off of geo-targeting the user's IP address to display the most relevant results immediately to them. Can potentially save them a search or three. That works great. However, when crawlers frequent the site, they are obviously being geo-targeted for their IP address, too. Google has come to the site via several different IP addresses, resulting in several different locations being displayed for it on the homepage (Mountain View, CA or Clearwater, MI are a couple). Now, this poses an issue because I'm worried that crawlers will not be able to properly index the homepage because the location, and ultimately all the content, keeps changing. And/or, we will be indexed for a specific location when we are in fact a national website (I do not want to have my homepage indexed/ranked under Mountain View, CA, or even worse, Clearwater, MI [no offence to any Clearwaterians out there]). Of course, my initial instinct is to create a separate landing page for the crawlers, but for obvious reasons, I am not going to do that (I did at one point, but quickly reverted back because I figured that was definitely not the route to go, long-term). Any ideas on the best way to approach this, while maintaining the geo-targeted approach for my users? I mean, isn't that what we're supposed to do? Give our users the most relevant content in the least amount of time? Seems that in doing so, I am improperly ranking my website in the eyes of the search engines. Thanks everybody! Marc
Intermediate & Advanced SEO | | THB0