Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitors' paid search analysis tool
Hi, What tool will do the following: will tell me paid keywords our competitors are using. Ideally, I could narrow down by country. Thanks. Katarina
Competitive Research | | Katarina-Borovska0 -
Competitors SERP's
One of our competitors is routinely achieving excellent SERP for brand related keywords within only a couple of months of adding the respective brand to their website. Full details are explained in the attached image to avoid the competitor being notified of this post via Google Alerts. Can anyone identify how they are achieving this and point me in the right direction? The other thing is that I was also achieving good SERPs for these and other brand related keywords but I seem to have dropped off the face of google for them, where my competitor has not and who continues to achieve the seemingly impossible. Thanks TOfVHD8
Competitive Research | | UnderMe0 -
What keywords & phrases are my competitors targeting?
I wanted to compare our keyword focus, to my competitors. What is the best way to audit what type of content / keywords - my competitors are using?
Competitive Research | | jwochna0 -
Competitor's Ranking's Reports are Showing Blank Values for Rankings
When selecting competitors to see how they rank for specific terms in the ranking's report page here, it loads the page normally, but without any actual rankings. Where values should be, it lists all values as our competition not ranking whatsoever (for all competitors). I'm not sure if there is something we're doing wrong, or perhaps we just need to wait for a refresh, but if anybody could help out, that would be wonderful!
Competitive Research | | ACLens0 -
What's the best way to discover my business and search competitors?
I am trying to figure out who the real competitors are for the domain that I have been recently handed. Other than the client's references, which I don't think are the real competition (they are his benchmarks), how do I go about discovering the true competition? What the simplest, most effective way to go about discovering my business and search competitors? Given that this is a web portal, aren't both of the latter the same?
Competitive Research | | amit20760 -
SEOMoz bar shows competitors ranking higher in SERPS with less Moztrust and Mozrank?
We are targeting the term "wholesale fireworks" we are currently on page two and just completed some on page optimization. I noticed using the Mozbar that the SERP results on the first page show a site with near zero page rank, Moztrust, domain trust, etc.. It is steadily number two in SERPS. Our site is just as strong or stronger on these ratings as anyone on the first page. How is it possible that we are still hanging out on page 2?
Competitive Research | | devonkrusich0 -
Link analysis competitors
Am I missing something here or does SEO Moz only support three competitor sites to do comparisons against in each campaign? If not where do I add another website to the list to compare against? Thanks in advance!
Competitive Research | | AdenBrands0 -
Competitor Ranking High has 2 Domains, But Duplicate Website ?
I was using OSE and noticed all the backlinks to one of our competitors is there other domain name, which is the EXACT SAME website. You can enter both url's and they display the same content. They are not useing any canonical tags either. Why are they not penalized for duplicate content? And for using there own website for backlinks ? We try to do everything right, but still cannot beat them. Any thoughts on this?
Competitive Research | | hfranz0