Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Keyword Research - tools
Hello all, I would like to find better synonyms for my keywords, and dig deeper to bid / place strategy into place for them. I am currently using the adwords too but it only gives me closely related keyword ideas. Is there something "free" which can give me a better co-relation data to work with? Thanks Aditya
Competitive Research | | shanky10 -
Effort for "moderate competition" keywords
I'm rather new to this, and while I'm getting some sense of everything I'm trying to figure out what kind of scope of work lays ahead of me. The keywords I'm looking to rank for are "moderate difficulty" -- somewhere between the 45%-55% "difficulty scale" on seomoz's keyword difficulty report. Assuming I have a number of "A-grade" (according to SEOmoz's reports) optimized pages for these keywords, how many links of a given quality level should I be looking at building up? I mean, of course, the more the better, but if I'm gunning for high DA/PA pages, am I looking at dozens here or hundreds of such links? I can imagine that any answer isn't going to come with much specificity, but if there was just an "idea" of the scale of backlinking involved here, that'd be great!
Competitive Research | | yoni450 -
Tool for finding what keywords a competitor ranks for?
Does anyone know of any good tools that display what keywords a competitor ranks for? I have many competitors that I know get a lot of traffic, but I'm not entirely sure where the traffic comes from so it would be nice to plug in their url and get a general overview of what keywords they rank for and what positions.
Competitive Research | | shawn810 -
Free SEO tools appearing in SERPS for a site
Hi there, I've just taken over a site and it doesn't have a huge amount of shelfspace on the SERPs for it's brand name. Looking at the results that are there are a lot of free SEO tools that are coming up. They haven't had an SEO before so it's not coming from their side. There's such a lot of them I was wondering if anyone had come across this before - is it simply a competitor looking at the seo, could it be one of the seo analytics packages using free sites or is there anything else to be aware of. I can start knocking the sites down the SERPs but just wondering if anyone has a good solution for this. thanks Shiv
Competitive Research | | Shivvyt0 -
Discrepancy in Competitive Link Analysis?
I currently have a campaign set-up that monitors a few competitors compared to our site. When I look at the Root Domain Metrics I see that Company A has 2,053 total external links, and Company B has 1,632 total external links. I wanted to view these links in more detail so I ran Advanced Inbound Links reports. The reports returned 432 results for Company A and 693 results for Company B. Shouldn't these numbers match up with each other, or am I missing something? Also, I noticed that in the Inbound Links analysis for my competitors there are a number of results listed that are not actually inbound links, but are internal links from there own site. Is there a reason these are appearing? I am having problems because I am trying to use this campaign to show the improvements in our site vs our competitors. However, OSE shows dramatically fewer total external links for our site (about 260 vs 3200 in Webmaster Tools) and I cannot tell if the competitor data is accurate? Thanks
Competitive Research | | zazo0 -
What is a Competitive Domain Analysis In Laimens Terms?
Hello all, I know what a Competitive Domain Analysis is, essentially, as a Developer, aspiring SEO master like Danny Dover, or Rand; LOL, I wish, those dudes a serious geniuses! I was looking for the Trifecta Tool, in SEOmoz, tools, but it seems to be gone??? So it looks like I will be using the "Competitive Domain Analysis", to assist my sales pitch. My Question is how would you explain it to a Client, or my Head Sales guy, for a pitch to a prospective career making client. Many thanks to your thoughts.
Competitive Research | | smstv0 -
Best Traffic Estimation Tool
Hello, What is the best system/tool to estimate traffic to a site, something more accurate than Alexa rank. This is of course assuming that one does not have access to the site's Google Analytics data, Thanks, Carl
Competitive Research | | CarlLarson0 -
Looking Up C-Block Links
I have been trying to find the tool that provides the campaign data the linking C-block data. Does anyone know which tool it is? Or is that just part of the campaigns?
Competitive Research | | purch0