Questions created by astronaute
-
Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such. A simple solution for those bots would be to mimic Google by using its user agent string for example.
Link Building | | astronaute
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking. This should not be a big deal IMHO, am I missing something obvious ?0