Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitor ranks high, but how?
On our campaign, I have three local competitors which I like to keep an eye on. Two have been around for years but the other is pretty new. According to Moz, he has a Domain Authority of 1, has no backlinks and ranks lower on every keyword I keep track of. How then is he getting ahead of us for certain search terms? Can Moz not give any more information?
Competitive Research | | robandsarahgillespie0 -
Methods for estimating competitor website traffic from natural search
Hi all, I'm currently working on some competitor analysis and estimating website traffic based on rankings and estimated search volume for approximately 500 keywords. I'm then estimating based on click through rate from Webmaster tools. I think this will give me a relatively accurate estimation but was wondering how everyone else does this? Any other methods out there? Thanks, Elias
Competitive Research | | A_Q0 -
Anyone want to test out my keyword research theory?
Hi all, I'm relatively new here but not new to the world of SEO / SEM. Over the years I've loved using SEOmoz and other tools but of course have found certain limitations with respect to how I like to work. That's the case with any tool / service. So over the years I've put together a keyword research / competitor analysis process that has worked well for me and I'm wondering if it might also work for others. I've spent the last 15 years of my life as a director of a range of companies, mainly in printing but also in systems development, marketing, etc. I spent a large percentage of my time developing systems and tools to help me with my search engine marketing. I've now sold all my companies and I'm semi-retired, somewhat bored, and would love it if I can assist others with the process I've used over the years. I'm curious to know whether SEM professionals agree with the way my system ranks search terms from "best" to "worst". If you're interested in testing this process and telling me if you think the resulting list of search terms that I come up with for your website is "spot on", "not bad" or "horrible!", then please read on. My key motivation here is to educate myself as well as others. I'm not charging for any of this... If you give me your website URL, your top 5 competitors and your top 5 search terms, I will return to you: a complete list of search terms including "niche" and longtail search terms you can then... really easily filter out irrelevant search terms, thus creating a list of negative keywords, ready to import into your Adwords campaign. easily group your search terms in "education" and "purchase" search terms so I can analyse these two groups of keywords separately see which words are used most often across all the search terms so you can easily create keyword specific Ad Groups in your Adwords campaign. You can tell me which options you prefer: broad match, "phrase match" or [exact match] when getting search engine results specify any country you want the SERP results for, or even any city. I will then: do all the keyword research, getting the latest (live) Google SERP results combine all competitor metrics (page rank, domain age, juice links, etc) and search term information (search volume, CPC, search term length, etc) together to give you a list of search terms ranked from "best" to "worst" do the same for both organic search terms and paid (Adwords) search terms do the same for both direct search terms and niche search terms take into account "word count" (number of words in a search term) as longtail keywords generally higher-converting search terms And you can adjust things to change how the keywords are ranked: specify "thresholds". eg; you can ignore the really competitive search terms, or ignore the really short search terms specify "weightings". eg; you can put a greater emphasis on search term length, or a lesser emphasis on cost per click As a result of the keyword analysis, it'll also show you who your organic and adwords competitors are based on all keywords, or just your top ranked ("best") keywords. In that competitor data you can see: which paid ads appear at the top, side AND bottom of the results page which organic results are shopping, image, video, and local results all the metrics for each competitor (page rank, domain age, juice links, etc) All of this is is handled in a simple web interface that I threw together recently. It's really simple, merely asking for your site and preferences and then an interface to view / sort the results. Interested?
Competitive Research | | eatyourveggies
I'd like to hear from any SEM professionals who want to test this process. Once I have your basic details, I can get a keyword list together simply (using my internal process / software) and then you need to do some basic sorting, particularly if your search terms are in an industry that I know nothing about. Your input will be required. From there, give me 24-48 hours and I'll return 2 lists of search terms: "organic", and "Adwords". I'd love to hear your opinion about the relevance of the search term lists. I hope it will also spark some interesting discussion and hopefully help people learn a bit more about keyword / competitor research. If you're interested, please shoot me a private message letting me know why you'd be a good candidate to test this system. I really do want people who are well versed in search engine marketing. So please include a basic "resume" about who you are. If you have an SEM company and that's your main career focus then I definitely want to hear from you. Adam0 -
Competitor's Ranking's Reports are Showing Blank Values for Rankings
When selecting competitors to see how they rank for specific terms in the ranking's report page here, it loads the page normally, but without any actual rankings. Where values should be, it lists all values as our competition not ranking whatsoever (for all competitors). I'm not sure if there is something we're doing wrong, or perhaps we just need to wait for a refresh, but if anybody could help out, that would be wonderful!
Competitive Research | | ACLens0 -
Tool for finding what keywords a competitor ranks for?
Does anyone know of any good tools that display what keywords a competitor ranks for? I have many competitors that I know get a lot of traffic, but I'm not entirely sure where the traffic comes from so it would be nice to plug in their url and get a general overview of what keywords they rank for and what positions.
Competitive Research | | shawn810 -
How to track competitor who uses another website on the url?
Hi everyone, The thing is pretty simple: I have a competitor who uses a different format of url, very similar to this: www.example.com - but he started to use a thing like this: example.mtv.com The problem is that every analysis that I make tell me something about mtv.com instead of the site I want. The url is redirected. Sorry about my english, but I think it is very clear. I want to know how to track something like this, because even in competitor analysis I'm getting info about the big site, and not the one I need. Thank you very much.
Competitive Research | | bluehelmet0 -
Whats the best way to see why your competitor is out ranking you?
I am trying to rank in the number on position for Part Time CFO. Currently we are in 2nd and 3rd. We have way more quality inbound links then the site in first position, more content etc. How would you go about investigating why they are ahead of us?
Competitive Research | | b2bcfo0
