Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Competitor using our product descriptions
Hi, We have a competitor (with a higher DA) using some content from our product pages word for word on a few of their product pages. It's only a couple paragraphs per product page (including the main product description), not the entirety of the content on the page. It also appears they have scraped some content from another authoritative competitor (and even left their brand name mentioned on it!) I've read mixed thoughts on whether plagiarized content will harm the original site, but am concerned since it is an authoritative site. Should I be concerned since it's only parts of our product page? Thank you all!
Competitive Research | | ATShock0 -
My competitor's sneaky link building technique
Hi, I can't figure out how my competitor seems to fit four invisible nav menus into her home page. If you view her page source and type "/about" for example, it shows five examples of the text linking to her "about" page. But four of them can't be located on the web page (they are hidden). How does she do this? She basically has four duplicate links that are counted for SEO but can't be seen on the webpage. address: http:// camilla peffer.com. au/ Any ideas?
Competitive Research | | SleekCopy0 -
How does my competitor website jump from a natural ranking of 10 to 1 in a matter of days?
OK this has really got me baffled, and I really hope the MOZ community can give me the answers I am seeking.... My statements and queries below are my own opinions and thoughts. I am not an experienced SEOer and I am learning all the time, so if I have written something incorrectly, please feel free to correct me 🙂 I have been told that natural rankings take time, and I have read with the recent algorithm changes from Google, that purchased links do not have an effect like they used to. I have also read somewhere that google has clamped down on bogus article writing (MOZ please clarify) But what I am seeking some clarification on is, how does my competitors website, that has had a natural rank of #1 in the past, suddenly drop off; for a lengthy period of time, then suddenly re-appear at #10 on page #1, then some three/four days later, hit #1? I have read that natural rankings are purely natural. I.e. a domain owner cannot pay google to boost their natural ranking. But is this really true? Why is it when I speak to a close colleague of mine in the same industry, he mentions that his maps listing has dropped off, and after making a couple of phone calls to his SEO guys in India, maybe a day or two later, magically his maps re-appear? I really am starting to question is a "natural ranking" really "natural?" I have heard that some of these SEO agencies do have some tools that dramatically effect the natural ranking of a domain. What are these tools?
Competitive Research | | Giorgio680 -
SEO Keyword Research
Hi, We are SEO beginners so please bear with us! We are trying to promote "Web based Invoicing Software". The SEO company we have signed with offer us 5 keywords for the package we are on with them. They have suggested\offered us: 1. Invoicing Software - Fine
Competitive Research | | Studio33
2. Online Invoicing - Fine
3. Online Invoicing Software - Covered by 1 and 2
4. Small Business Invoicing Software - Covered by 1
5. Invoice Template - Fine. Will Invoice templateS be covered on this one too? My question is does number 3 cover number 1,2 & 4 anyway? If so I am thinking to not go for 1,2, & 4 just keep 3 and choose three other new keywords. Would this be a better strategy and "more for our money?" Or, keep 1 and 2 and lose 3 and 4, would that be a good option. So, in summary options are (all assuming keeping number 5) 1. Keep all
2. Keep 1 & 2 - Lose 3 & 4
3. Keep 3 - Lose 1,2 & 4 4. Any other combo you can suggest? Any advice welcomed Thanks nutnut0 -
Keyword Research - tools
Hello all, I would like to find better synonyms for my keywords, and dig deeper to bid / place strategy into place for them. I am currently using the adwords too but it only gives me closely related keyword ideas. Is there something "free" which can give me a better co-relation data to work with? Thanks Aditya
Competitive Research | | shanky10 -
Tracking competitors in search engines
Hi Mozzers, I'm using Advanced Web Rankings (AWR) to track my site and a number of competitors in Google - seeing how rankings and brand visibility change from week to week. I didn't set this up from scratch and I'm worrying with all of the recent algorithm changes that I might well be tracking the wrong competitors. Is there a tool or methodology I can use to find the biggest players in the market? I'm in the travel market so there's lots of choice and I track the large sites but want to be sure I'm aware of smaller/mid-sized sites gaining visibility without me tracking them via a platform like AWR. Many thanks!
Competitive Research | | panini0 -
Link analysis competitors
Am I missing something here or does SEO Moz only support three competitor sites to do comparisons against in each campaign? If not where do I add another website to the list to compare against? Thanks in advance!
Competitive Research | | AdenBrands0 -
Whats the best way to see why your competitor is out ranking you?
I am trying to rank in the number on position for Part Time CFO. Currently we are in 2nd and 3rd. We have way more quality inbound links then the site in first position, more content etc. How would you go about investigating why they are ahead of us?
Competitive Research | | b2bcfo0
