Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What's a good way to get started with competitive research?
Hi all, SEO noob here. I'm doing an audit for a firm that makes specialized accounting software. It's a relatively new firm, with a barebones website. My client has identified three direct business competitors. In addition, I see indirect competitors (such as product reviews) on the SERP for a relevant keyword phrase. I want to provide actionable advice for my client. What information should I present? I'd like to help my client understand: Why my client's competitors are outperforming them on the SERPs What my client needs to do to overtake their competitors What information should I present to my client? Thanks, all.
Competitive Research | | AndyKubrin0 -
What tool can I use to find the top traffic-driving keyword for a batch of pages from multiple sites?
I thought I might be able to use Ahref's Batch Analysis for this, but that just gives stats on backlinks. I have a several lists of thousands of pages scraped for particular sets of keywords, but what I need is some way to automate fetching the biggest keyword that each page ranks for - biggest being the keyword that is estimated to drive the most organic traffic. Is there a tool out there that does something like this?
Competitive Research | | helenlorettahasan0 -
2 sites with low DA and PA and unoptimised <title>outranking everyone for competitive search term - How ?</title>
Im doing some keyword research for a new client, Ive found 2 anomalies for competitive search terms where sites with unoptimised <title>tags, poor backlinks and other stats, (compared to the other sites listed in the SERP) are out ranking everyone and claiming the #1 spot</p> <p> The 2 search terms are :</p> <ul> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8">Interior designers London</a>- #1 new-id.co.uk - <a href="https://moz.com/researchtools/ose/links?site=new-id.co.uk&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> <li dir="ltr"> <p dir="ltr"><a href="https://www.google.co.uk/search?q=interior+designers+london&oq=interior+designers+london&aqs=chrome..69i57j69i65j69i60j69i65l2j69i60.2784j0j1&sourceid=chrome&ie=UTF-8#q=interior+designer+london">Interior designer London</a>- #1 nstudio.co.uk - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnstudio.com%2F&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a></p> </li> </ul> <p> (interestingly they hardly rank at all for any modification of those search terms)</p> <p> The only thing i could think of is that they both had some really good links using that exact anchor text, but ive been checking OSE for the past month both in the backlinks and the just discovered but couldnt see anything like that.</p> <p>I conducted the SERP checking from a laptop with a broadband connection based in London, although im not very local to either of these locations im about an 8th of the cities distance away from one of those location and about half a cities distance away from the other. So i dont think its a local skew thats causing the SERP rankings.</p> <p>For both sites i can see they had old domains which now redirect to the them :</p> <ul> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">New-id.co.uk = </span><span data-mce-mark="1"><a href="http://newidinteriors.com/">newidinteriors.com</a> - <a href="https://moz.com/researchtools/ose/links?site=http%3A%2F%2Fnewidinteriors.com&no_redirects=1&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - PA1, DA1</span></p> </li> <li dir="ltr"> <p dir="ltr"><span data-mce-mark="1">Nstudio.co.uk = intarya.com - <a href="https://moz.com/researchtools/ose/links?site=www.intarya.com&filter=&source=external&target=page&group=0&page=1&sort=page_authority&anchor_id=&anchor_type=&anchor_text=&from_site=">OSE info</a> - Da 32, PA43</span></p> </li> </ul> <p> Although the intarya domain was quite strong with good backlinks (but nothing above par comapred to the other sites the nstudio.co.uk site is now outranking, the anchor text was mostly natural with only some keyword anchor text from PA1 pages. </p> <p>The <a href="http://newidinteriors.com/">newidinteriors.com</a> didnt seem to have a backlink profile, both PA1 and DA1.</p> <p> Ive taken screenshots of the SERP pages below, but feel free to dig further</p> <p>Any idea what's going on here ?</p> <a download="HWkjmDf.png" class="imported-anchor-tag" href="http://i.imgur.com/HWkjmDf.png" target="_blank">HWkjmDf.png</a> <a download="A70jXse.png" class="imported-anchor-tag" href="http://i.imgur.com/A70jXse.png" target="_blank">A70jXse.png</a></title>
Competitive Research | | mike8780 -
Tool to scrape data from Homedepot,ebay,amazon.
Does anybody know about a tool that you can use to aggregate data about the best seller products in categories for major retailers in USA? I did find a couple not so good tools for amazon and even ebay but retailers like walmart and homedepot are left out.
Competitive Research | | Harveyspecter0 -
Good tools to compare site speed on different sites?
Google tells me that my site is slower than 81% of the internet.Now this is a very large news website, so I would like to compare this against other very large news websites. Are there any tools that will be able to tell me the speed of a list of websites? I am looking for something a little more detailed than 'this site loaded in 1 second'. Also, do any of you fellow SEOs know of any good tools to analyse what is causing your site to slow down? Kind regards
Competitive Research | | MirandaP0 -
Your Favorite Tools for Competitive Analysis?
When you review your competitors online presence, what are some of your favorite tools? I like Open Site Explorer's domain comparison tool. SEMRush is great too but the data is often inaccurate.
Competitive Research | | qlkasdjfw0 -
Free SEO tools appearing in SERPS for a site
Hi there, I've just taken over a site and it doesn't have a huge amount of shelfspace on the SERPs for it's brand name. Looking at the results that are there are a lot of free SEO tools that are coming up. They haven't had an SEO before so it's not coming from their side. There's such a lot of them I was wondering if anyone had come across this before - is it simply a competitor looking at the seo, could it be one of the seo analytics packages using free sites or is there anything else to be aware of. I can start knocking the sites down the SERPs but just wondering if anyone has a good solution for this. thanks Shiv
Competitive Research | | Shivvyt0 -
Link Analysis Tool
Does anyone have/use any really good link analysis tools? I have been using SEOmoz open site explorer but don't find it that indepth or find that many links. I don't mind paying once the link analysis tool is good and I can export links to excell. I was thinking of buying this software http://www.link-assistant.com/features-and-editions.html Cheers,
Competitive Research | | Socialdude0