Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why is my competitor's site ranking #1?
I'm about to work for a local business website that offers cleaning services and products. The keyword they want the most is ruled by a very odd site; My client's competitor's site has been around for 7 years. (Less than the average of it's competitors. Less than my client's) Has 1 backlink. Lower PA MR MT & DA than any other in the SERP. It's a 1 page site made with Flash. They do not have FB or Twitter accounts. So I thought maybe they were ranking so well because of their traffic. But neither my client, me or my coworkers have ever heard of this company. And yet, they are ranking #1. And the only thing I notice that might have helped is that the title of their page is the exact keyword and nothing more. Any ideas?
Competitive Research | | Eblan0 -
Competitor research: No data / results displaying on Keyword tools, Aexa
Hi there! I'm trying to research a few competitors using various Keyword tools (SEM Rush, Compete, Keyword Spy -- even Alexa for high level insight). While the bulk of the competitors generate expected results through these tools (a smattering of their top organic and paid search keywords, some traffic estimations through Alexa), ONE of these competitors lists "No results" across all categories and all tools: http://www.bgstar.ca Despite this, we know that they invest heavily in search -- and my SEM Rush toolbar indicates that they have a Google PR of 5 (though I recognize that that should be taken with a grain of salt). So I'm stumped! Has anyone encountered this before? Is there something structural that they might be doing, that's blocking not only Google-based platforms, but Alexa too? Thanks for your help!
Competitive Research | | MACJ0 -
Is there a SEO Moz Competitor Analasys tool that will show me how many Vistors their site receives?
I'm trying to find out how much traffic a competitor gets. Is there an SEOmoz tool for this?
Competitive Research | | starkSEO0 -
Keyword Research Tool Problem
Hi, I had been in SEO for around 2 to 3 years, I am looking for your suggestion if there is any best keyword research tool. I tried almost most of the keyword research tool (both paid or free) but I found the following problem: -Worktracker: I think it is too difficult to use and understand as a non technical person. I always wonder the data accuracy because when the same keyword appear in Google for few thousand traffic, it only appear in Wordtracker for few dozen or hundred of search. Also I don't think they suggest any long trail keywords in my view. The long trail keyword you can also find in free keyword search tool. -Google Keywords Tool- Good, but it can only show 100 results. I cannot get a better pictures of related or niches keywords. I always find some competitive keywords can rank easily but some non competitive keywords usually very difficult to rank in some case. It will depend on how you interpret the data results. -Keyword Discovery and Wordstream-Although I really want to subscribe, but they are a way too expensive (you have to subscribe for 12 months). I check with their free keywords tool but only shown very limited results. Although there are some other like spyfu.com or keywords spy...... Is there any one can suggest me which is the best tool to start with keyword search for the long trail and niche keywords ? or is there any way or trick to find the best keywords ? Thank you. Tom
Competitive Research | | Stevejobs20110 -
What if your content is way better than competitors?
Do you have any suggestions if you are being outranked by competitors even though your page has way better content and better links?
Competitive Research | | Charlessipe0 -
Better tactics for keyword research
I am paying for monthly Adwords Google. I am planning to build out and optimize my website content with the findings from the Adwords campaign. What is frustrating me is the Google list of keywords recommended. Is there a better tool for determining excellent, "long-tail" keywords specific to my industry, products and services? -Feeling Left Out
Competitive Research | | natearistotle0 -
Whats the best way to see why your competitor is out ranking you?
I am trying to rank in the number on position for Part Time CFO. Currently we are in 2nd and 3rd. We have way more quality inbound links then the site in first position, more content etc. How would you go about investigating why they are ahead of us?
Competitive Research | | b2bcfo0 -
How do you perform competitive research for SEO?
What metrics tell you the most when you're looking at your competitors across the search landscape? PageRank/MozRank Inbound links Keyword rankings Alexa/QuantCast/etc. Pages indexed Something else entirely? What numbers speak volumes to you when you want to get an idea of how you benchmark against your competitors? And how do you communicate these results?
Competitive Research | | jcolman2