Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My links are not Getting indexed in Moz
Hi Moz family i have a website relate to air purifier and i purchased 5 to 10 gigs for different sort of backlinks but 4 weeks gone still i have not seen any backlinks are indexed in moz but 40% are indexed in ahref Any body can please explain how can they will indexed? Any best indexing method? Thanks
Link Building | | bradyknaus1 -
Why Google Search Console Data is different from Moz Data for my website
Hi All I am running a website, I have been using Moz since Feb 2021. Kindly go through these pics My question is why Moz is showing 2K plus backlinks while Google search console is showing just 1253 backlinks. Why fewer links in the google search console is less? How can I increase Google search console backlinks? Also, Moz is showing 90+ DA backlinks but those websites are not showing by the Google search console. What should I do to let google consider them? m0JGjBQ.jpg Cmp8ei3.png
Link Building | | ssubodhsingh0 -
Moz bot not discovering important links (high DA sites link)
Moz bot is unable to crawl and discover my links on the high authority websites like microsoft, linkedin, pinterest, etc. Where is the problem?
Link Building | | TechG0 -
What's the difference between these 2 url's in wordpress
https://www.yourdomain.be/blog/-test-article/
Link Building | | conversal
https://www.yourdomain.be/blog/test-article/ What is the difference in wordpress with this "-" in the url? Both url's show the page like it's supposed to. Is this normal?0 -
Spammy Links in MOZ but when I go to the external link I can't find a link to us
I was going to try to contact webmasters to see if they would remove some of our spammy links. I see alot of them in MOZ but when you go to the site our anchor text is not there. Is this good? How often does MOZ refresh external links. Please see: http://www.opensiteexplorer.org/anchors?site=www.totalvac.com None of the links for the anchor text <a class="clickable title link-pivot" title="See top linking pages that use this anchor text." data-text="vacuum cleaner parts vacuum parts vacuum bags vacuum cleaner bags" data-id="46391436859">vacuum cleaner parts vacuum part...</a> in MOZ exist? We got hit extremely hard by Penguin in May
Link Building | | totalvac0 -
Why aren't my backlinks showing up?
I've recently switched hosting my site on www.vamospaella.com to www.vamospaella.co.uk (using a 301 redirect) and I've been building backlinks. According to Majesticseo, vamospaella.co.uk has 38 backlinks with 8 referrring domains and vamospaella.com has 15 external backlinks from 11 referring domains. Some of these backlinks date back a month, while others have shown up in the last week. However, I have seen no improvement in my Google rankings for my keywords. Why is this?
Link Building | | RMelly0 -
Backlink reports in OSE, the good and the bad!
Hi all Mozers, I have a couple of questions re the backlink reports in Open Site Explorer. In the introductory video Rand suggests that you can indentify backlinks that are a) Having a positive effect, and b) Having a negative effect on SEO campaigns. Do you identify such links using the domain/page authority of the linking page? Also, we know we have more links than OSE is reporting, does this mean that the links that are not reported are not helping our SEO campaign? Many thanks in advance, much appreciated. Lee
Link Building | | Webpresence0 -
What is the best way to make sure competitors or others aren't buying links on my sites behalf to penalize us?
Is there a good way to do this? Does the Open Site Explorer have an ability to screen by when the link was found, or help by picking up on potentially shady links? Thanks much..
Link Building | | jim_shook0