Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Ahrefs Backlinks VS Moz Backlinks
HI Team and members We have website related to jobs in Pakistan and we purchased 5 niche edits backlinks to a specific page " MES Jobs 2021" In ahrefs it show only 2 backlinks to that page but in moz 0 backlinks but in actually there are 5 links. Why this happen in moz? We actually love this tool used for Keywords research for jobs in Pakistan find some cool keywords using moz but we facing backlinks issues in it.
Link Building | | AliHassanbinali1 -
My Domain has a couple of badlinks decreasing my rankings, will disavowing them reduce my Domain Authority on Moz?
Good day Every Body, I have a heart aching issue, my site (nightwatchng.com) amassed a number carnivorous backlinks, I have lost rankings, i studied my search traffic and discovered that I have been hit by Google Penguin Algorithm Penalty, I was forced to believe that those backlinks were built to my site on purpose just so my rankings will drop, I know the importance of link building and thats why i follow the white hat technique. Now the big question is, IF I DISAVOW THESE TERRIBLE LINKS FROM GOOGLE SEARCH, WILL MY MOZ DOMAIN AUTHORITY DROP FROM WHAT IT CURRENTLY HAS?? I also want to know if the Algorithm Penalty will affect the subdomain (news nightwatchng.com) of my site?
Link Building | | Newswatchng0 -
Eric Ward's urlwire.com - worth a $500 investment?
I'm after a turbo boost (direct and indirect) to my site's SERPs. How beneficial is URLwire.com? Anyone here used it? Worth $495? Given that Matt Cutts has said links from PR sites don't pass link pop, would URLwire fall into this category? (I'm aware the idea is also to generate backlinks from other referring sites). Thanks
Link Building | | Jeepster0 -
Is anybody else noticing a dramatic change to their 'links to your site' section in Google Webmaster Tools?
Hey,
Link Building | | ChrisHolgate
Over the last six months or so we've been going through our backlink profile and cleaning up links from poor quality sources. Week by week there have been small changes in our Google Webmaster Tools 'links to your site' section to reflect this. I logged on this morning however and there has been a dramatic shift in the information displayed. Pretty much every bad link has been removed from the list including sites I know for a fact are still linking to us as they didn't communicate at all to our removal requests. Additionally, rather than showing the top 1000 links to our site as it used to, WMT is only showing 73 linking domains. The remaining 73 domains are good natural links from high quality sources. I'm guessing Google are just in the middle of an update and that the remaining linking domains (including the bad ones) will reappear shortly. This isn’t a request for advice or help but I’m just curious as to whether anybody else is seeing anything similar?0 -
Multiple Links from High Ranking Site Vs. Links from Multiple Domains - What's More Important?
I understand it is important to get links from many quality domains. Currently, I do have links from top domains (PR, Trust) and it I can get more from (high rank) pages on these same domains. Would it be better to focus on expanding my reach (find additional domains to link from) or to continue to build links from the current domains I have a connection with? What is weighted more? I realize doing both is important, but trying to figure out how to best use my time. Thanks! David
Link Building | | DWill0 -
Why Can't My Ecommerce Site Rank?
I've already done the on-site SEO things, URL's are all SEO-friendly, etc AND I've already done a few rounds of link building to some of the interior pages on my site, and I still can't get my site to rank. What other strategies can I employ to get my site to rank? I don't believe my keywords are that competitive (fleece blankets, baby blankets, etc.)
Link Building | | locallyrank0 -
I wish to know how can I track users via what keywords they are searching and coming to my site exactly. These are non paid keywords.
There is a list of non paid keywords which is showing up but is that all ? I wish to know all the keywords people are searching and coming to my site? How can I accomplish the same.
Link Building | | shanky10 -
Any benefit to others' links in my blog comments?
I recently started a blog for my client. So far the response has been decent but, as should probably be expected, most of the commenters want to put a link to their own site in the comment. Do I have anything to gain by allowing these comments to stay on the page?
Link Building | | AmericanOutlets1