Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Which is the best tool to check number of css, js, images and it's size?
Hello Team, Can you please suggest which is the best tool to check number of css, js, images and it's size of my competitors? Also what if they are using CDN network and bundling? Will that tool consider that also? Thanks! Wrights
Competitive Research | | wright3350 -
Keyword strategy for wordpress category base blog in competitive industry
Hi friends, We have a client in the movie review blog industry, which we are aware is highly competitive, but we have tons of content (over 10,000 unique posts) and a good team of reviewers for fresh content. I was wondering about any input on keyword variations for our SEO Title of our wordpress main categories, like "Movie Reviews". I was thinking a modification to "Latest Movie Reviews", but the competition is only slightly less, and still seems questionable for attaining eventual rankings. "Best Movie Reviews" is slightly higher than "Latest Movie Reviews" but the competition still seems tough. What would be a good long term strategy with these category pages, any suggestions?
Competitive Research | | JustinMurray0 -
Ranking for Competitive Keywords vs. Less Competitive Keyword Variations
I'm curious about situations where a website ranks very well for query variations, but doesn't rank for the query itself (or the reverse of that). For Redfin (where I work), here is the situation with regard to keyword rankings on Google (searched today from USA, incognito)... real estate search - #4 real estate online - #4 real estate site - #5 find real estate - #9 get real estate - #16 real estate - #163 It stands to reason that a site ranking well for a competitive query should also rank well for less competitive query variations - especially query variations that are non-limiting and do not demand a custom landing page (for example, I would consider 'board games' to dramatically limit the query 'games' and be best targeted with a targeted page...not so with 'real estate site' and 'real estate'). So, my question is, what are some theories regarding situations like this? Why do some sites rank so well for competitive queries but not for non-limiting query variations? Why aren't the sites that are crushing us for 'real estate' also crushing us for 'real estate' variations (to be clear...the top sites are crushing us for both)? Is it anchor text? Is it social signals? Is it offline signals, co-occurrence, or citations? What about internal linking and site structure? I realize it's likely a mix of all this, but I'm hoping we can drum up some new ideas here. FYI, on Bing we also rank very well for 'real estate' variations, but leap up to 31st for 'real estate'. Thoughts?
Competitive Research | | RyanOD0 -
Which analytics tools do you use?
If you are interested in web analytics, could you tell me, which tools (software) do you use? No just Google Analytics, but also some other tools (e.x: for heatmaps, facebook, internal statistics, ... etc) I´m interested also in testing (A/B, ..) - so, also these tools you can mention thanks 🙂
Competitive Research | | mysho0 -
Keyword Research Tool Problem
Hi, I had been in SEO for around 2 to 3 years, I am looking for your suggestion if there is any best keyword research tool. I tried almost most of the keyword research tool (both paid or free) but I found the following problem: -Worktracker: I think it is too difficult to use and understand as a non technical person. I always wonder the data accuracy because when the same keyword appear in Google for few thousand traffic, it only appear in Wordtracker for few dozen or hundred of search. Also I don't think they suggest any long trail keywords in my view. The long trail keyword you can also find in free keyword search tool. -Google Keywords Tool- Good, but it can only show 100 results. I cannot get a better pictures of related or niches keywords. I always find some competitive keywords can rank easily but some non competitive keywords usually very difficult to rank in some case. It will depend on how you interpret the data results. -Keyword Discovery and Wordstream-Although I really want to subscribe, but they are a way too expensive (you have to subscribe for 12 months). I check with their free keywords tool but only shown very limited results. Although there are some other like spyfu.com or keywords spy...... Is there any one can suggest me which is the best tool to start with keyword search for the long trail and niche keywords ? or is there any way or trick to find the best keywords ? Thank you. Tom
Competitive Research | | Stevejobs20110 -
Free SEO tools appearing in SERPS for a site
Hi there, I've just taken over a site and it doesn't have a huge amount of shelfspace on the SERPs for it's brand name. Looking at the results that are there are a lot of free SEO tools that are coming up. They haven't had an SEO before so it's not coming from their side. There's such a lot of them I was wondering if anyone had come across this before - is it simply a competitor looking at the seo, could it be one of the seo analytics packages using free sites or is there anything else to be aware of. I can start knocking the sites down the SERPs but just wondering if anyone has a good solution for this. thanks Shiv
Competitive Research | | Shivvyt0 -
Link analysis competitors
Am I missing something here or does SEO Moz only support three competitor sites to do comparisons against in each campaign? If not where do I add another website to the list to compare against? Thanks in advance!
Competitive Research | | AdenBrands0 -
Gathering SERP analysis data
Hello, Im gathering data to complete the spreadsheet found here http://seomoz.box.net/shared/g3vz0kxbbp (using Ben's). A lot of the data can be gathered from open site explorer, the moz tool bar or via the key previded but where is the quickest place to gather the below info or is a lot of it manual gathering? Lk RDs KW Usage LDA Score Xct Anc. Txt Links Xct Anc. Txt RDs Part Anc. Txt Links Part. Anc Txt RDs Domain # Linking RDs Tweets FB Shares Blog Links
Competitive Research | | RikkiD220