Screaming Frog for competitor research
-
Hi guys,
Does somebody experienced an issue with Screaming Frog spider while trying to crawl a website? It's weird since no error message appears and at the same time the crawler cannot crawl anything. There is no restriction in the robots.txt. Has someone encountered such an issue?
Thank you.
Cheers
-
Thank you, guys!
@ Dan, I will get in touch! Thanks for your quick response
Cheers,
Ani
-
Thanks for helping out with the above query guys, all good answers.
Hey @Margarita,
If you'd like to drop an email through to support@screamingfrog.co.uk, I can tell you what the exact issue is
Cheers,
Dan
-
Hi Margarita,
I have encountered similar issues when you need to enter a password or where the site might require cookies.
Here is what Screaming Frog has to say on the topic:
"13) Why Won’t The SEO Spider Start? Top ↑
This is nearly always due to an out of date version of Java. If you are running the PC version, please make sure you have the latest version of Java. If you are running the Mac version, please make sure you have the most up to date version of the OS which will update Java. Please uninstall, then reinstall the spider and try again.
14) Why Won’t The SEO Spider Crawl My Website? Top ↑This could be for a number of reasons. Before contacting us, please check your robots.txt to see if the website is blocking the SEO spider. Please also ensure the website is html (the SEO spider does not crawl framesets), that it can be crawled without JavaScript or requiring cookies. Please also check for the ‘nofollow’ attribute if certain links are not being crawled. There is an option in the configuration to crawl ‘nofollow’ links."
Also, SEER has a great article about how you can use Screaming Frog 55 different ways - just some good info.
Hope this helps.
Mike
-
Hey Margarita,
Try using the Googlebot User-Agent for Screaming Frog, since other bots might be blocked via .htaccess.
Hope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My competitor's sneaky link building technique
Hi, I can't figure out how my competitor seems to fit four invisible nav menus into her home page. If you view her page source and type "/about" for example, it shows five examples of the text linking to her "about" page. But four of them can't be located on the web page (they are hidden). How does she do this? She basically has four duplicate links that are counted for SEO but can't be seen on the webpage. address: http:// camilla peffer.com. au/ Any ideas?
Competitive Research | | SleekCopy0 -
How better can we compare with a competitor ?
We were ranking 1st to 5th spot for the keyword, "PSD to Joomla Template" for this page, http://www.qubesys.com/psd-to-joomla-template/ for last 2 years! But since last 6 months, we are getting like 0 traffic to this page and absolutely no leads. We just tried to compare with a ranking competitor who has been on top for last 7-8 months already, www.joomla6teen.com/Convert-PSD-to-Joomla-Template.html I just compared it with my page, http://www.opensiteexplorer.org/comparisons?site=www.joomla6teen.com%2FConvert-PSD-to-Joomla-Template.html&comparisons[0]=http://www.qubesys.com/psd-to-joomla-template/ I see our page wins in PAGE METRICS , but falls in SUB DOMAIN AND ROOT DOMAIN Metrics . Whats that ? How do we know, what all things we need to improve at ?
Competitive Research | | monali1230 -
My (properly optimised) webpage outscores page#1 ranked competitors on page/domain authority ... but I'm only on page#2\. Huh?
I'm puzzled. I've optimised a particular page for a particular search term, and the SEOMoz tool gives me an A for on-page optimisation. So no problem there. I can understand why my webpage/site is being outranked by pages from (for example) the Guardian and Oxford University, but there are several sites that Google is ranking on page #1 though their page and domain scores are well below ours. Specifically: my page/domain authority scores are 46/52, compared with 22/46 for the competitor that Google is ranking #5 - yet we only rank a lowly #12. And it's not as though the particular page in question isn't an obvious and appropriate part of our site. We work with new writers and the page in question offers a selection of creative writing courses. It's not like we're a writing-related site that suddenly has a page advertising fake rolexes. It's not a timing issue either, as most of our links have been in place for a couple of years at least. So I'm puzzled. And concerned. This page of ours was a reliable revenue generator for us and it's dying out there on the page#2 wilderness. If anyone can help, I'd be massively grateful. I don't know if this is helpful, but the page in question is http://www.writersworkshop.co.uk/Creative-Writing-Courses.html and the search term is ... well, heck, you take a wild guess. We're a British firm, so the only search engine that really matters to us is google.co.uk
Competitive Research | | harrybingham0 -
What keywords & phrases are my competitors targeting?
I wanted to compare our keyword focus, to my competitors. What is the best way to audit what type of content / keywords - my competitors are using?
Competitive Research | | jwochna0 -
Problem Adding a competitor
I am having a problem adding a competitor at one of my campaigns... I am not sure if I should add the link of my competitor so I wont 🙂 the link is with a /path competitorlink.gr/magazine Anyone knows what should I do?
Competitive Research | | Angelos_Savvaidis0 -
Manta inbound link in OSE - competitor site yes, my site no, why?
Our site has had a link back to our site from Manta for a long time, but it's not appearing in our OSE results. One of our competitors similarly has an inbound link from Manta and that link DOES appear in their OSE results. I'm not certain whether that matters for us. Does it? But in any event, why would the one link appear but not the other? Thanks! Tim
Competitive Research | | tcolling0 -
Better tactics for keyword research
I am paying for monthly Adwords Google. I am planning to build out and optimize my website content with the findings from the Adwords campaign. What is frustrating me is the Google list of keywords recommended. Is there a better tool for determining excellent, "long-tail" keywords specific to my industry, products and services? -Feeling Left Out
Competitive Research | | natearistotle0 -
PA and PD better than my competitors is a way to rank better?
I got a site that stuck in last position of google's first page. My on-page is grade A for this particulary page, i got links for that page too. But My competitors PA and PD is better than mine, not too much, but it is. If i build more quality backlinks and start increasing my PA and PD probaly i will rank better than my competitors? Thats the factor that i have do cara abour now?
Competitive Research | | Ex20