Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's a good way to get started with competitive research?
Hi all, SEO noob here. I'm doing an audit for a firm that makes specialized accounting software. It's a relatively new firm, with a barebones website. My client has identified three direct business competitors. In addition, I see indirect competitors (such as product reviews) on the SERP for a relevant keyword phrase. I want to provide actionable advice for my client. What information should I present? I'd like to help my client understand: Why my client's competitors are outperforming them on the SERPs What my client needs to do to overtake their competitors What information should I present to my client? Thanks, all.
Competitive Research | | AndyKubrin0 -
Competitor ranks high, but how?
On our campaign, I have three local competitors which I like to keep an eye on. Two have been around for years but the other is pretty new. According to Moz, he has a Domain Authority of 1, has no backlinks and ranks lower on every keyword I keep track of. How then is he getting ahead of us for certain search terms? Can Moz not give any more information?
Competitive Research | | robandsarahgillespie0 -
How to top competitor's article
Hello, For bobweikel.com I'm going to work on getting some natural backlinks by making a better article than http://www.medicinenet.com/weight_loss/article.htm even though they're 30 times more of an authority than me. If you think this is a good idea, how do you recommend I top this thing? I'm just brainstorming
Competitive Research | | BobGW0 -
Keyword Research - tools
Hello all, I would like to find better synonyms for my keywords, and dig deeper to bid / place strategy into place for them. I am currently using the adwords too but it only gives me closely related keyword ideas. Is there something "free" which can give me a better co-relation data to work with? Thanks Aditya
Competitive Research | | shanky10 -
What metrics are you using to build profiles of competitors Google Places pages?
Hi, I am attempting to put together a set of procedures to undertake when optimizing Google Places pages. Does anyone have a good list of metrics and methods for determining them in order to build profiles of competitors' Places pages? Thanks
Competitive Research | | networkelites0 -
Whats the best way to see why your competitor is out ranking you?
I am trying to rank in the number on position for Part Time CFO. Currently we are in 2nd and 3rd. We have way more quality inbound links then the site in first position, more content etc. How would you go about investigating why they are ahead of us?
Competitive Research | | b2bcfo0 -
My client has shown me a similar site, though not a competitor. He wants to know what sites they are linked from that give them such a good Google rank for certain kewords. Can SEOMoz tell me this?
When using google.com.au and searching for "travel to france", www.frenchtravel.com.au is the 3rd organic result. (the 1st two are not travel businesses, they are non profit travel guides) My client, who runs www.visituk.com.au, an Australian site that organises tours of the UK, said "so we just need to add these sort of words to the site?" I said, yes, but it doesn't end there. The real task is to have a link to your site on other sites surrounded with the words "travel" and "UK". He asked if he could see a list of the sites the french site was being referred by relevant to the search phrase. Is there an SEOmoz tool for this? Or is there another way I can generate that list? Thanks Simon
Competitive Research | | electrik0 -
How do you perform competitive research for SEO?
What metrics tell you the most when you're looking at your competitors across the search landscape? PageRank/MozRank Inbound links Keyword rankings Alexa/QuantCast/etc. Pages indexed Something else entirely? What numbers speak volumes to you when you want to get an idea of how you benchmark against your competitors? And how do you communicate these results?
Competitive Research | | jcolman2