Moz Q&A is closed.
After more than 13 years, and tens of thousands of questions, Moz Q&A closed on 12th December 2024. Whilst we’re not completely removing the content - many posts will still be possible to view - we have locked both new posts and new replies. More details here.
Blocking Competitive Analysis Tools in robots.txt.... Worth it?
-
I've been considering blocking third party crawlers for a while – specifically those crawling my website for the sake of competitive analysis, such as SEMrush and Ahrefs. I'm familiar with how to do so, but when researching the question I found practically no one asking the same question. The guides I've found on what to put in your robots.txt make no mention of whether to block competitive analysis crawlers. Which makes me wonder whether this is a good idea after all.
My chief concern here is rival sites going after the same search terms we target – one of our competitors in particular has an uncanny way of going after the same searches we are. I know blocking crawlers won't prevent competitors from watching our content, but it will make it slightly harder for them. Is there any major drawback I'm missing? Any big reason not to go ahead and block SEO analysis crawlers?
-
Hi David,
Well, if you use a tool that can change the name of the bot that comes after a site, then you can't do anything about it. However, I do know that SEMrush is causing problems for some hosts and the likes of Heart Internet, they block access to the SEMrush bot on all of their servers.
To be honest with you, it is quite difficult to block everything and time consuming and unless you have some secret formula to keywords that are magically drawing in more traffic, then I wouldn't worry about it too much.
Take Accuranker. I use this for tracking and part of setting up, is it will find new phrases that I perhaps haven't been using or looking at before. It doesn't get these from either SEMrush or Ahrefs. All of this information is available from many sources.
I would honestly spend the time making your pages as good as they can be. Look at your cometitors. Are they winning? How are they doing it? Fight back against them rather trying to hide yourself from view.
-Andy
-
Issa is correct and to be honest, he's pointed out what I see as a pretty major problem - not being able to see these metrics (particularly ahrefs) really isn't ideal for analysis and/or troubleshooting.
Also, it may very well be that the competitor that keeps targeting the same terms as you is doing so via other means. There are countless tools available to all of us that allow for analysis of strong and relevant terms so your site may have little to do with their decisions.
Beyond this, one glance at your page titles will often be enough to figure out what you're targeting
-
Hi David,
This is completely your choice. Your website's SEO and overall ranking wont be affected by block SEMRush for example.
The only drawback I can think of is that you wont be able to use this tool for your own domain research, which can be helpful if you're looking for ranking stats, or click share from search engines. I would hate to not be able to research my client's websites regularly, as this is the kind of intelligence I use to a) understand Google's and Users behaviour to find my website. b) discover technical and SEO issues with my websites.
So although you can go a head and do it, I would say you should always be ready to unblock them when needed. Unfortunately SEM Rush does not update their database so regularly, so you wouldn't be able to predict when you should unblock their robot for information.
I hope I could help
Issa
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Search Intent and Relevance
Hello SEO gurus 🤓 I’m looking for the most efficient ways to analyze the search intent and relevance of competitors who are ranking for the keywords we’re targeting. While I know Google excels at assessing search intent and relevance, I’m interested in learning how we can evaluate these factors as metrics for our competitors. The goal is to understand their strategy better and find ways to outrank them. Do you have any tools or methodologies that you recommend for assessing competitor content to determine its alignment with search intent and user needs/relevance? I’d love to hear your thoughts and suggestions on this!
Competitive Research | | Cricket931 -
How to set up a competitor URL with a language slug for a campaign
Hello, I am trying to set up a competitor with language slug for my (subfolder) website with a language slug. Let's say my website is something like: websiteholding.com/de/website
Competitive Research | | Siir
My competitor is: competitor.com/de When I go to Campaign Settings > Comptetitor Sites > type in competitor.com/de > Hit Save Competitor > Then it shows the saved competitor without the language slug as competitor.com I am not sure if this is a correct method of tracking since for my DE website I would like to track the DE page of the competitor, not their global page. Please correct me if I am wrong and help me out on possible solutions? I am quite new to SEO & Moz , so any help on the topic would be appreciated.0 -
Is it allowed to buy an old website with external links en redirect it to your own site??
Hello, We noticed at MOZ that one of our competitor's websites has increased the number of external backlinks from 5k to 25k within a month. Upon investigating the links, we found that the competitor had acquired an old, unknown website with completely different content and external links in that month. The competitor then redirected that purchased website to their own website, instantly gaining 20k external links. This seems to be against Google's guidelines as it is an extremely unnatural way of link building and should be penalized by search engines. However, the website's Domain Authority (DA) has increased by 10 points in that month, and its rankings have greatly improved on Google. So it appears that acquiring an old website with many external links unrelated to your own website is highly profitable. We try to obtain links honestly, but we cannot reach 25k links naturally. Is there any way to report these practices to Google? Does anyone know how to do this?
Competitive Research | | Femamedia0 -
Interal linking
Re: I'm stuck by internal linking. What structure should a football website follow? Silo or Topic Cluster?
Competitive Research | | thanhdung0906
I need advice on my website. My website: https://tipbongda247.net/
I hope there are answers!
Thanks0 -
Rank online sex shop
I have been trying for some time to take the online store to the front page, but it is very difficult to go up in google searches for the keyword ( sex shop ).
Competitive Research | | Constantin2021
Can anyone help me with some tips. Thank you.
https://www.voluptas.ro/0 -
MOZ vs Ahrefs vs SEMRush vs Spyfu and so on
Hello, fellas. I've been trying to find some type of comparison analysis for online SEO related tools and I couldn't find any, which are fresh and overview more than three. I'm interested to see if anybody saw good comparison reviews of such tools, as well as your own thoughts (back your thoughts up with more than just "I like it", please 🙂 The tools I'm interested in comparing are: MOZ Ahrefs Majestic Spyfu SEMRush Raven Wordstream whatever else you use or have in mind
Competitive Research | | DmitriiK2 -
Why do some sites perform great but show no data on the tool bar?
Hey everyone, Been a paid member for 3 days and this is my first time posting 🙂 I have a quick question about competitive analysis with the MozBar. I do not want to point out the exact website (unless that is something cool to do on this forum), but there is a site ranking for a single keyword in the local 7 pack in slot one, which is also slot one for the SERP entirely. When I go to look at the page in more detail, the MozBar is showing no DA, PA, anything at all, and the Open Site Explorer is showing no links internal or external. I know that the page has to have at least some of these elements populated in order to be ranking for a term this competitive, so why is the bar not showing anything? The bar works GREAT on other sites below this one. Thanks for any help!
Competitive Research | | tonyflorespsa0 -
My competition is using 1px images and links without anchor text. Good or bad idea?
I've been scoping out some of my competition and I noticed the number 1 ranking site is doing a lot of "interesting" stuff. I saw a bunch of 1px blank images that they must be using for the alt tag keywords. I also noticed several links to internal pages with different keywords without anchor text so they don't actually appear on the page. Is this considered black hat? Should I be doing something similar? It feels like the work I'm doing to try to rank #1 with a clean/professional looking site are wasted when the #1 sites are doing things like this.
Competitive Research | | technota0