Mac-Friendly, DOM-Rendering Spidering Tool for Multiple Users
-
Hello!
I am looking for a spidering tool that:
- Is Mac-friendly
- Can render the DOM and find JS links
- Can spider password-protected sites (prompts for password and then continues spider, etc.)
- Has competitive pricing for 8+ users.
Screaming Frog is amazing - and maybe we're just going to have to bite the bullet there. But if anyone has any other ideas, I've love to hear them. Thanks!
-
So - after digging around a lot and reading and re-reading every article that popped up for "screaming frog alternative", I've come to the conclusion that for the price, there really is nothing better than Screaming Frog right now.
I was impressed, however, with the incredibly helpful team from Deep Crawl. This enterprise tool is designed for larger websites - whereas Screaming Frog can crap out of your local machine runs out of memory. Because it's a more powerful tool, it's more expensive than Screaming Frog - but if you need an enterprise solution, it's definitely worth looking into. Another big differentiator is that Deep Crawl has no limit to the number of users, which is our primary pain point with Screaming Frog.
-
Right now we're updating SEOSpyder ( http://www.mobiliodevelopment.com/seospyder/ ) for rendering pages but i can't give you timeframe when will be done.
So far memory requirements isn't too high and was crawl 250k site with 8G ram machine.
-
Oh actually something I just realized is that potentially ScreamingFrog can do what you want and it will provide you with access to 8 users, but the setup is complicated. You would need to run it in a big virtual machine on AWS or Google Cloud Platform. That way you can scale the machine so it won't time out and everybody will still have access to it.
Back to your question: I've worked with Deepcrawl, a bit with Ryte and more with Botify. They're all great tools that are able to crawl your site. But you probably already looked into some of them.
-
Oh, interesting - can you help me understand about more about the cloud solution are you using...? Thanks!
-
Going to follow this, as I've been looking for something too. But we went the cloud service, as there is nothing that I acme across that can otherwise fulfill all these needs.
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Multiple h1 tags on this html 5 page a issue?
Hi Guys, I have a html5 page located here: https://tinyurl.com/yc6s3xs2 I know from some online discussions having multiple h1 tags on HTML 5 pages like this, shouldn't be an issue. Any thoughts on this? Cheers,
Intermediate & Advanced SEO | | bridhard80 -
OK to have multiple local business structured data on one website?
Hello there, I'm working on implementing local business structured data for a website but we have multiple offices. Is it okay from a Google perspective to add different local business data on different pages of the website, or can I only use one set of local business data site wide? Many thanks, Gill.
Intermediate & Advanced SEO | | Cannetastic0 -
Decreased organic traffic but increased Webmaster Tool Queries
We have a client who has had a significant decrease in organic traffic this last month (about 20%) but in Webmaster tools it tells me there was an increase in impressions and clicks. How can these both be accurate?
Intermediate & Advanced SEO | | jfeitlinger0 -
Google Mobile Friendly designation in Search results
We have recently deployed a mobile (http://m.pssl.com) version of our desktop website (http://www.pssl.com). We've followed the guidelines in their documentation (https://support.google.com/webmasters/answer/6101188) & (http://googlewebmastercentral.blogspot.com/2015/04/rolling-out-mobile-friendly-update.html), added the appropriate rel=alternate/rel=canonical tags updated site maps and robots.txt files, etc. A mobile search for our company shows the "mobile-friendly" flag in the search results for our home page, but for some reason other pages such as category and brand are not showing showing as "mobile-friendly". I can submit the pages using the mobile-friendly tester (https://www.google.com/webmasters/tools/mobile-friendly/) and all of the pages I test come back as mobile friendly. Does anyone have any experience or advice they'd be willing to share that might help us resolve this issue?
Intermediate & Advanced SEO | | ovenbird0 -
Bypassing Google, Data Highlighter and Webmaster tools
eLLo! Has anyone used Data Highlighter? I've had colleagues mentioning a jump in CTR after using the data highlighter on pages. Thought I'll do the same and went into my webmaster tools but I've hit a brick wall. Whenever I highlight a product page, my country selector pops up and I'm unable to highlight a product page. A colleague of mine mentioned to bypass google by basing it on user agent, this will allow you to avoid the country selector. But if I bypass Google, wouldn't it affect Google Analytics, Indexing etc?
Intermediate & Advanced SEO | | Bio-RadAbs0 -
E-commerce site, one product multiple categories best practice
Hi there, We have an e-commerce shopping site with over 8000 products and over 100 categories. Some sub categories belong to multiple categories - for example, A Christmas trees can be under "Gardening > Plants > Trees" and under "Gifts > Holidays > Christmas > Trees" The product itself (example: Scandinavian Xmas Tree) can naturally belong to both these categories as well. Naturally these two (or more) categories have different breadcrumbs, different navigation bars, etc. From an SEO point of view, to avoid duplicate content issues, I see the following options: Use the same URL and change the content of the page (breadcrumbs and menus) based on the referral path. Kind of cloaking. Use the same URL and display only one "main" version of breadcrumbs and menus. Possibly add the other "not main" categories as links to the category / product page. Use a different URL based on where we came from and do nothing (will create essentially the same content on different urls except breadcrumbs and menus - there's a possibiliy to change the category text and page title as well) Use a different URL based on where we came from with different menus and breadcrumbs and use rel=canonical that points to the "main" category / product pages This is a very interesting issue and I would love to hear what you guys think as we are finalizing plans for a new website and would like to get the most out of it. Thank you all!
Intermediate & Advanced SEO | | arikbar0 -
Showing Duplicate Content in Webmaster Tools.
About 6 weeks ago we completely redid our entire site. The developer put in 302 redirects. We were showing thousands of duplicate meta descriptions and titles. I had the redirects changed to 301. For a few weeks the duplicates slowly went down and now they are right back to where they started. Isn't the point of 301 redirects to show Google that content has permanently been moved? Why is it not picking this up? I knew it would take some time but I am right where I started after a month.
Intermediate & Advanced SEO | | EcommerceSite0 -
Managing Multiple Websites via Add-On Domain
Hey SEOMOZ community, I've always been curious about whether or not hosting multiple websites through an 'add-on' domain has positive/negative effects on SEO for websites. Currently, I'm hosting 5 sites through an add-on domain at Hostgator.com. Is this a poor way to set-up my sites or is this OK?
Intermediate & Advanced SEO | | NiallSmith0