Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How much should I analyze my competitors?
Hi, Recently I've started a competitor research (aka competitive analysis). A few hours after I began, I've started to feel that maybe I'm doing too much unnecessary work. What I've been doing so far is analyzing the competitor's links (using the Dr Pete's Link Profiling excel and some other things), and their site optimization. The on-site optimization analysis has been very light, just to see if the competitors are doing something really well, or not. But still, I think that maybe I'm doing too much work. So, I'd like to see your thoughs: How much work is necessary when doing competitor analysis? What things do you analyse? And how much do you report to the clients? Thank you all in advance! Ivan
Competitive Research | | ivankrm0 -
Anyone want to test out my keyword research theory?
Hi all, I'm relatively new here but not new to the world of SEO / SEM. Over the years I've loved using SEOmoz and other tools but of course have found certain limitations with respect to how I like to work. That's the case with any tool / service. So over the years I've put together a keyword research / competitor analysis process that has worked well for me and I'm wondering if it might also work for others. I've spent the last 15 years of my life as a director of a range of companies, mainly in printing but also in systems development, marketing, etc. I spent a large percentage of my time developing systems and tools to help me with my search engine marketing. I've now sold all my companies and I'm semi-retired, somewhat bored, and would love it if I can assist others with the process I've used over the years. I'm curious to know whether SEM professionals agree with the way my system ranks search terms from "best" to "worst". If you're interested in testing this process and telling me if you think the resulting list of search terms that I come up with for your website is "spot on", "not bad" or "horrible!", then please read on. My key motivation here is to educate myself as well as others. I'm not charging for any of this... If you give me your website URL, your top 5 competitors and your top 5 search terms, I will return to you: a complete list of search terms including "niche" and longtail search terms you can then... really easily filter out irrelevant search terms, thus creating a list of negative keywords, ready to import into your Adwords campaign. easily group your search terms in "education" and "purchase" search terms so I can analyse these two groups of keywords separately see which words are used most often across all the search terms so you can easily create keyword specific Ad Groups in your Adwords campaign. You can tell me which options you prefer: broad match, "phrase match" or [exact match] when getting search engine results specify any country you want the SERP results for, or even any city. I will then: do all the keyword research, getting the latest (live) Google SERP results combine all competitor metrics (page rank, domain age, juice links, etc) and search term information (search volume, CPC, search term length, etc) together to give you a list of search terms ranked from "best" to "worst" do the same for both organic search terms and paid (Adwords) search terms do the same for both direct search terms and niche search terms take into account "word count" (number of words in a search term) as longtail keywords generally higher-converting search terms And you can adjust things to change how the keywords are ranked: specify "thresholds". eg; you can ignore the really competitive search terms, or ignore the really short search terms specify "weightings". eg; you can put a greater emphasis on search term length, or a lesser emphasis on cost per click As a result of the keyword analysis, it'll also show you who your organic and adwords competitors are based on all keywords, or just your top ranked ("best") keywords. In that competitor data you can see: which paid ads appear at the top, side AND bottom of the results page which organic results are shopping, image, video, and local results all the metrics for each competitor (page rank, domain age, juice links, etc) All of this is is handled in a simple web interface that I threw together recently. It's really simple, merely asking for your site and preferences and then an interface to view / sort the results. Interested?
Competitive Research | | eatyourveggies
I'd like to hear from any SEM professionals who want to test this process. Once I have your basic details, I can get a keyword list together simply (using my internal process / software) and then you need to do some basic sorting, particularly if your search terms are in an industry that I know nothing about. Your input will be required. From there, give me 24-48 hours and I'll return 2 lists of search terms: "organic", and "Adwords". I'd love to hear your opinion about the relevance of the search term lists. I hope it will also spark some interesting discussion and hopefully help people learn a bit more about keyword / competitor research. If you're interested, please shoot me a private message letting me know why you'd be a good candidate to test this system. I really do want people who are well versed in search engine marketing. So please include a basic "resume" about who you are. If you have an SEM company and that's your main career focus then I definitely want to hear from you. Adam0 -
What keywords & phrases are my competitors targeting?
I wanted to compare our keyword focus, to my competitors. What is the best way to audit what type of content / keywords - my competitors are using?
Competitive Research | | jwochna0 -
What's the best way to discover my business and search competitors?
I am trying to figure out who the real competitors are for the domain that I have been recently handed. Other than the client's references, which I don't think are the real competition (they are his benchmarks), how do I go about discovering the true competition? What the simplest, most effective way to go about discovering my business and search competitors? Given that this is a web portal, aren't both of the latter the same?
Competitive Research | | amit20760 -
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period?
Is it possible to see how many visitors a competitor has received for any given keyword over a set time period via either Organic / Paid means? Thanks
Competitive Research | | jaycfc1230070 -
Competior Research & Analyse
Hi, I like to know the inbound marketing sources of my competitors, and like to know how many traffic are receiving from Organic Ranking and Adwords or Search Engine Paid Marketing. ANd it would be nice to know the social media traffic too. I am searching, but i see here compete.com, but at the moment it's too expensive for me, like to ask you if you can find me to find any software cheaper than compete. Thanks.
Competitive Research | | leadsprofi0 -
Link analysis competitors
Am I missing something here or does SEO Moz only support three competitor sites to do comparisons against in each campaign? If not where do I add another website to the list to compare against? Thanks in advance!
Competitive Research | | AdenBrands0 -
Competitor analysis
Hi This is obviously a well worn question (I've read the great blog by Sam Crocker last year - see below), but I'm looking for an updated set of opinions on sources of competitor data. The competitor data I'm interested in is traffic volumes either aggregated or disaggregated by referral source (i.e. organic search, PPC, email, social etc.). The types of data I've checked so far are Google Trends for websites (no disaggregation), Alexa (some good sources of free data) and Experian's Hitwise (too expensive for my current needs). In addition I'm aware that Compete will be providing an interface for such data in the UK in January at reasonable rates for what I'm after. I'm interested in views on these and other sources in relation to experience of their relative accuracy, costs, pros, con etc. Ideally I'm looking for something that is free, comprehensive, and is not being used by the competition - is that too much to ask!!! 😉 Any inputs would be greatly appeciated. Thanks in advance! Neil Sam's great blog: http://www.seomoz.org/blog/monitor-competitor-traffic
Competitive Research | | mccormackmorrison0