Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Intent and Relevance
Hello SEO gurus 🤓 I’m looking for the most efficient ways to analyze the search intent and relevance of competitors who are ranking for the keywords we’re targeting. While I know Google excels at assessing search intent and relevance, I’m interested in learning how we can evaluate these factors as metrics for our competitors. The goal is to understand their strategy better and find ways to outrank them. Do you have any tools or methodologies that you recommend for assessing competitor content to determine its alignment with search intent and user needs/relevance? I’d love to hear your thoughts and suggestions on this!
Competitive Research | | Cricket931 -
Is it allowed to buy an old website with external links en redirect it to your own site??
Hello, We noticed at MOZ that one of our competitor's websites has increased the number of external backlinks from 5k to 25k within a month. Upon investigating the links, we found that the competitor had acquired an old, unknown website with completely different content and external links in that month. The competitor then redirected that purchased website to their own website, instantly gaining 20k external links. This seems to be against Google's guidelines as it is an extremely unnatural way of link building and should be penalized by search engines. However, the website's Domain Authority (DA) has increased by 10 points in that month, and its rankings have greatly improved on Google. So it appears that acquiring an old website with many external links unrelated to your own website is highly profitable. We try to obtain links honestly, but we cannot reach 25k links naturally. Is there any way to report these practices to Google? Does anyone know how to do this?
Competitive Research | | Femamedia0 -
What is the relationship between inbound links and linking domains?
In doing competitor research, there's a specific competitor that has fewer linking domains, but dramatically more inbound links. How do these two relate to each other? I do have a higher domain authority and more internal followed links with them having more internal nofollow and external (both follow and nofollow) links. I'm trying to understand how they can have 3.7k linking domains and 4.8 million inbound links while I have 4.1k linking domains and only 601k inbound links. What really defines an inbound link and what's there relationship between inbound links and linking domains?
Competitive Research | | grayloon0 -
Rank online sex shop
I have been trying for some time to take the online store to the front page, but it is very difficult to go up in google searches for the keyword ( sex shop ).
Competitive Research | | Constantin2021
Can anyone help me with some tips. Thank you.
https://www.voluptas.ro/0 -
How do you select which keywords to push in SEO?
Hi Guys Selecting the right keywords that a website can realistically rank for is a key to gain top rankings relatively quick. I am just curious to hear how you guys do it (the methodology) when selecting which keywords to push? I mean you need to check the competition for each keyword as well so how to check this quickly to see what we realistically can rank for? Cheers John
Competitive Research | | igniterman751 -
I am looking to find the top pages based on traffic volume on my competitors websites, does anyone know of any good resources?
I want to know how which pages on my competitors websites are the most popular based on the traffic volume. I do not care how many links or directed to that page or any other metric. Only thing I am looking for is the traffic volume. It would also be nice to know the length of time spent on that page.
Competitive Research | | kanteenboy0 -
Traffic Data for Competitor Subdomain
Does anyone know of a way to get monthly visitor data for a subdomain that I do not manage? I would normally use SEMRush or Compete, but they only provide domain level data.
Competitive Research | | dsinger0 -
Can I write guest blogs on competitor's blogs?
I have been doing some guest blogging and its hard to find guest blogs opportunities. I have found some but they are direct competitors. Is it safe to write guest post? by the way the are not local competitors. Any suggestion please. thanks
Competitive Research | | conversiontactics0