Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why MOZ just index some of the links?
hello everyone i've been using moz pro for a while and found a lot of backlink oppertunites as checking my competitor's backlink profile.
Link Building | | seogod123234
i'm doing the same way as my competitors but moz does not see and index lots of them, maybe just index 10% of them. though my backlinks are commenly from sites with +80 and +90 DA like Github, Pinterest, Tripadvisor and .... and the strange point is that 10% are almost from EDU sites with high DA. i go to EDU sites and place a comment and in lots of case, MOZ index them in just 2-3 days!! with maybe just 10 links like this, my DA is incresead from 15 to 19 in less than one month! so, how does this "SEO TOOL" work?? is there anyway to force it to crawl a page?0 -
I'm stuck by internal linking.
What structure should a football website follow? Silo or Topic Cluster?
Link Building | | gogoanimetp
I need advice on my website. My website: https://tructiepbongda.site
I hope there are answers!
Thanks0 -
My Backlinks are indexed in Ahrefs But Not Indexed in MOZ. Why?
I Create backlinks for my website to Increase DA But Backlinks are indexed and Ahrefs Show that backlinks but MOZ is showing backlinks. I am confuse Can you explain?
Link Building | | Seogamesokay1a1 -
Paid Subscription Directory with Low Moz Open Site Explorer Spam Score
Looking at different backlinks of my competitors and came across a few directories that require a 1 time payment for the year. Is this considered the same as paying for a link or would this be considered something like getting a listing on your local Chamber of Commerce site? Also, I put the site through Moz's Open Site Explorer and it had PA:22 and DA:20, which is nothing incredible, but also had a spam score of 2. So would a site like this hurt my rankings? I know that's a good spam score, so I am a little confused with what to do. Here is the site: http://scl-online.net/en.htm Thanks for any help! Love this site.
Link Building | | aua0 -
How important are 'anchor' text links now
We have started building some good links but I'm just wandering how important anchor based text ones are now.
Link Building | | nick-name123
I'm not talking about spamming/going too heavy but a few here and there. What's your recent experience?0 -
Does anyone have a list of bad site's. I should not be on or submit to?
My Gatlinburg Cabin Rental site is losing ranking on the keyword "gatlinburg cabins" I was wondering if anyone had a list of site's it should not be linked on so I can demote them.
Link Building | | GatlinburgMan0 -
OSE shows links on sites but can't find links
Hi mozzers, I'm cleaning up our backlink profile and looking up anchortexts in OSE. I downloaded and selected one anchortext. However when I go to the sites OSE found, I can't find the links. I look in the source code and onpage keywords. Is it because of my lack of skills that I can't find the links 🙂 or isn't OSE working properly.
Link Building | | StephWeigert0 -
I'm interested in knowing link building strategies for regional businesses.
I'm not just interested in sites to target, but also how to manage anchor text when you are targeting phrases that include a keyword + a geo modifier. Thanks!
Link Building | | medtouch0