Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Visibility Score Against Top Search Competitors
Top Search Competitors
Competitive Research | | PhoebeK
The competitors that compete for the same keywords as this domain, ranked by visibility. Other websites that compete with this domain for the same keywords, ranked by visibility. Visibility (What does the visibility rating mean on this report)?
For example: If you have a visibility of 8.33 on this report and your competitor has a rating of 6.21 or 4.64, is your visibility better than theirs?0 -
Missing tab in Competitor research Template
The template provided in https://moz.com/blog/competitor-analysis-for-seo is missing the 'Audit Data' Tab. Can you please update it or share the Original Template
Competitive Research | | AnkitaDwivedi3 -
Competitor Keywords
Is there any way to find out what keywords my competitors are ranking for? I would like ideas for blogs, website content, etc. to drive traffic and am curious as to what topics my competitors are focusing on. I am able to see my competitors' rankings for keywords which I am currently tracking but I would like to see keywords they are ranking for that I am not tracking if possible. Any suggestions? Josh
Competitive Research | | jbeals0 -
Competitors ranking in top three with worse SEO
Ok I am at a loss. Our client is trying to rank for keywords like "dental assistant programs utah", but they are still on the bottom of page 1/top of page 2. The guys in the top 3 have terrible SEO. No links, newer domains, etc. I looked into it extensively and learned about the "honeymoon period" people talk about but we are long past that. I've looked through webmaster tools to see if there are any issues, and I can't find a thing. Anyone have an idea of what we could be doing wrong that is making us be put down so far on the page? Before I got here the SEO guys were doing older tactics which I think was starting to hurt them with penguin and what not so I tried my best to delete the crap content and clean everything up on the website. thanks for all your help! I really appreciate it.
Competitive Research | | boyackp0 -
Keyword Research Competition
Hello everyone. Was talking to my "competition" the other day and they mentioned the amount of traffic they are getting from google. It's double what I currently am receiving. I feel like I am not targeting the correct keywords. What is the best way to research competitions keywords? Anyway to find out what is driving traffic their direction?
Competitive Research | | Jasonalanmagic0 -
Moz Rank Moz Trust and Authority higher than my competitor but still getting outranked
Hi There, I'm new to SeoMoz but ive been using it for the past month and I've been using the tools to try to optimize my site but for some reason even though my metrics are showing that my rank and authority are rising past my competitor they still rank higher in the searches. my website is the site on the far left with the domain authority @27. the competitor that keeps outranking me is the third from the left with a domain authority of 12. If anyone can advise as to why I'm being outranked that would be very helpful. I'm stumped here. Thanks guys! eYtMRr9.png
Competitive Research | | SamRaskin1 -
Would sharing the same IP address with competitors in the same market hurt SEO?
I work for the Chevy dealership in New York who is trying to rank against other Chevy dealerships in the same area. All the dealerships, including my client, are using the same Chevy endorsed CMS (Cobalt). I just noticed that all of these competitors, including my client, are using the SAME IP address. Would it be beneficial to SEO if we were to ditch the Cobalt platform and choose another one that gave us a unique IP address? Have any of you run into this before? Any help would be appreciated.
Competitive Research | | tjkirgin0 -
Competitor Ranking High has 2 Domains, But Duplicate Website ?
I was using OSE and noticed all the backlinks to one of our competitors is there other domain name, which is the EXACT SAME website. You can enter both url's and they display the same content. They are not useing any canonical tags either. Why are they not penalized for duplicate content? And for using there own website for backlinks ? We try to do everything right, but still cannot beat them. Any thoughts on this?
Competitive Research | | hfranz0