Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is there a difference in SEO ranking value between a single backlink from a domain and multiple backlinks
Hi all I see that Moz gives data on Linking Domains and also External Links. The former being the number of domains which have one or more links pointing at your site and the latter the total number of links, including multiple links on the same domain. Apart from the potential benefit of people clicking the links and coming to your site and so increasing traffic, is there any SEO ranking benefit from multiple links? The only one I can think of is that you MAY get benefit from different anchor text for each link...? I'd be interested to hear any comment or experience on this. Bob
Intermediate & Advanced SEO | | BobBawden10 -
What can we do to optimize / be mobile-friendly for PDFs?
I'm getting a "Your page is not mobile-friendly." notice in the SERPs for all of our PDFs. I check the pdf on the phone and it appears just fine. rFtLq
Intermediate & Advanced SEO | | johnnybgunn0 -
Bypassing Google, Data Highlighter and Webmaster tools
eLLo! Has anyone used Data Highlighter? I've had colleagues mentioning a jump in CTR after using the data highlighter on pages. Thought I'll do the same and went into my webmaster tools but I've hit a brick wall. Whenever I highlight a product page, my country selector pops up and I'm unable to highlight a product page. A colleague of mine mentioned to bypass google by basing it on user agent, this will allow you to avoid the country selector. But if I bypass Google, wouldn't it affect Google Analytics, Indexing etc?
Intermediate & Advanced SEO | | Bio-RadAbs0 -
Question about multiple websites in same field
I know what most people say that it is best to only have the 1 website for focus but if we can put this to the back of our minds, if we create 2 different websites that are totally different designs (one upmarket one and one targeting the cheaper market) but in the same fields (printing) and go after 80% of the same keywords is this ok (could we be penalized). Please note we will not be interlinking the websites, the website .will be on different servers and the names will be registered under different people (2 partners in the business). We will however be accessing webmaster tools from the same location.
Intermediate & Advanced SEO | | BobAnderson0 -
Spellcheck necessary for user generated content?
We have a lot of user generated reviews on our key landing pages. Matt Cutts recommended using correctly spelled content. Would you perform a spellcheck of all already published user reviews or would you leave already published reviews rather intact and only perform spellcheck for new reviews before they are published? Since reviews have been marked up using schema.org, I am not sure whether posterior editing of lots of reviews may raise a flag with google regarding manipulating reviews. Thanks.
Intermediate & Advanced SEO | | lcourse0 -
Multiple domain level redirects to unique sub-folder on one domain...
Hi, I have a restaurant menu directory listing website (for example www.menus.com). Restaurant can have there menu listed on this site along with other details such as opening hours, photos ect. An example of a restaurant url might be www.menus.com/london/bobs-pizza. A feature i would like to offer is the ability for Bob's pizza to use the menus.com website listing as his own website (let assume he has no website currently). I would like to purchase www.bobspizza.com and 301 redirect to www.menus.com/london/bobs-pizza Why?
Intermediate & Advanced SEO | | blackrails
So bob can then list bobspizza.com on his advertising material (business cards etc, rather than www.menus.com/london/bobs-pizza). I was considering using a 301 redirect for this though have been told that too many domain level redirects to one single domain can be flagged as spam by Google. Is there any other way to achieve this outcome without being penalised? Rel canonical url, url masking? Other things to note: It is fine if www.bobspizza.com is NOT listed in search results. I would ideally like any link juice pointing to www.bobspizza.com to pass onto www.menus.com though this is a nice to have. If it comes at the cost of being penalised i can live without the link juice from this. Thanks0 -
How to solve a PHP problem to increase user experience ?
Hi all, I am having a peculiar PHP problem that is affecting the user experience of the site and thus, the SEO. The site is www.atlantiksurf.com and as you can see I use the QTranslate Plugin to manage the 3 different languages: www.atlantiksurf.com (Spanish) www.atlantiksurf.com/en www.atlantiksurf.com/de If you enter the german version: http://www.atlantiksurf.com/de/ , and you scroll down to the bottom of the page, you will notice there are a couple of posts that have (Español) in their Titles. This means that this particular post is written in Spanish and should only appear in the Spanish version of the site. Every single post is published on the three versions even when they are language orientated. The result is that when you press that specific Spanish title on the German version, you get nothing but this: http://www.atlantiksurf.com/de/aritz-aranburu-tendra-que-estar-un-mes-sin-competir/ Because obviously there is nothing there. If I go to the wordpress admin panel and search for the post manually, I can solve the problem by erasing all the default code of the different language versions that should not appear. But this is only a manual and non practical solution. The problem, I think, might be in the PHP orders that the Wordpress Theme is receiving from somewhere. I come to all of you SEomoz users as my last chance, because I've been months in discussions with Qtranslate users and php amateurs that cannot solve the issue. I know that this question isn't strictly about SEO, but in a way it is, because it must be affecting the way that Google look at us. Please, some help or orientation would be highly appreciated. rTqZI.png
Intermediate & Advanced SEO | | Tintanus0 -
Multiple blogs for seo
I have signed up for some rather expensive lawyer directories that have very high domain PR, 's of 6 or 7 . Some of these allow you to make blog posts or articles on their site which should be good for SEO because of the high domain PR. I understand that if I do a lot of posts on one of these blogs with links back to my site, I should rapidly reach the point of diminishing returns because they are all coming from the same domain. Therefore, I plan to mix up my blo posts betwee several of these sites and also rewrite them and post them on my own site's blog. My question is this, if I post on any of these sites and I link back to internal pages of my site, and not to the home page, does this offset the "diminishing returns" factor? Paul
Intermediate & Advanced SEO | | diogenes0