Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Tool to scrape data from Homedepot,ebay,amazon.
Does anybody know about a tool that you can use to aggregate data about the best seller products in categories for major retailers in USA? I did find a couple not so good tools for amazon and even ebay but retailers like walmart and homedepot are left out.
Competitive Research | | Harveyspecter0 -
Keyword Research Competition
Hello everyone. Was talking to my "competition" the other day and they mentioned the amount of traffic they are getting from google. It's double what I currently am receiving. I feel like I am not targeting the correct keywords. What is the best way to research competitions keywords? Anyway to find out what is driving traffic their direction?
Competitive Research | | Jasonalanmagic0 -
Advice on using google analytics and webmaster tools....for noobs?
I was wondering if anyone could suggest any resources on how to use GA and webmaster tools? I just sort of look and see my stats, etc...but I'd like to learn how to find my low hanging fruit, and try to help it rank better. I don't really use those tools as well as I should. I know there various spreadsheet's you can use, etc, but I haven't found a good (and easy to understand) resource that helps you learn how to use those tools together well. Thanks
Competitive Research | | NoahsDad0 -
Understanding Competitive Domain Analysis
I'm finding this site great for getting at the information that I need but I am having trouble understanding what I see as conflicting Competitive Domain Analysis and keyword research results for companies in my sector. My company is new to the sector we are in and I am tracking two really well established leaders (Moma and Poppa bear) and one company that is new to the sector (baby Bear) too. When looking at the competitive domain analysis the Momma and Poppa Bear wipe the floor with baby bear (and of course me!) on every single criteria; but when I see who is actually ranking best for the most competitive keywords for our sector who wins...baby bear; I just dont understand it. What is the reason for the disparity between the domain analysis and actual performance? Many thanks, Sinead
Competitive Research | | YGF0 -
Best Link Analysis Tool?
Now that the Yahoo Site Explorer is not usable on sites unless they are in your Bing Webmaster Tools, what is the best tool out there to see an accurate list of inbound links to a site? Google tools are still pretty unreliable from what I can see...
Competitive Research | | Bandicoot0 -
What is a Competitive Domain Analysis In Laimens Terms?
Hello all, I know what a Competitive Domain Analysis is, essentially, as a Developer, aspiring SEO master like Danny Dover, or Rand; LOL, I wish, those dudes a serious geniuses! I was looking for the Trifecta Tool, in SEOmoz, tools, but it seems to be gone??? So it looks like I will be using the "Competitive Domain Analysis", to assist my sales pitch. My Question is how would you explain it to a Client, or my Head Sales guy, for a pitch to a prospective career making client. Many thanks to your thoughts.
Competitive Research | | smstv0 -
Competitive Link Research Tool is not working
I receive a busy message each time I try. This has been out of commission the last several times I've tried it.
Competitive Research | | cmiller0 -
What tools do you use for competitive link analysis?
I have been using SEO Spyglass lately but it can be very slow at times. Wondering if there are any other accurate/robust tools out there.
Competitive Research | | rmteamseo0