Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I just want to ask Moz DA help anymore in Rank keywords
Hello Folks, i want to ask i have website related to kullu manali tour package and digital marketing institute, where tour package website DA is 2 and digital marketing institute DA is 26, my institute website rank on almost 1st page on most of keywords where travel website still doing struggle after much good links. please suggest me i need to boost my website DA.
Link Building | | ashux526g0 -
Give a Solution my DA is not increasing and not index my backlinks in MOZ??
DA of my website online is no longer increasing. It seems like Moz is not indexing my backlinks. Multiple backlinks of my website online BasketballShoes are now not indexing in Moz that is why DA is not increasing.
Link Building | | fbowable
Backlinks of my site have indexed in both Ahref and semrush each now not showing in Moz. DR has elevated to 51 however DA is still 7. If all and sundry have the solution please let me know.0 -
Domain Change, loss of inbound links ...
We're strongly considering a domain name change; this is purely for marketing reasons. We think in the long term, this will be a good thing. I believe we can mitigate the page redirection, branding changes, etc. My concern are the inbound links: from 200+ domains, 3,000+ links. So, I guess we can contact each of the top sites linking to us and hope they update our links. I'm not hopeful. I believe we'll loose must of the links. Has anyone been down this road and have experience to share?
Link Building | | jmueller0823
What should I expect, worst case? Is there a way to mitigate? Thanks much.
Jim0 -
Help! Someone's inserted my link in the footer of another site!
There's a site in my industry where (I'm guessing on flawed SEO advice) all of their pages contain footer links to rival sites – with each page containing a different set of links. One page has a link to my site, which I've just found out from GWT as the link now produces a 404.
Link Building | | Jeepster
Should I
a) ignore it?
b) ask them to replace it with a live link from my site (their site's highly relevant to mine)? or
c) ask them to remove it altogether as no-one wants footer links?0 -
Spammy Links in MOZ but when I go to the external link I can't find a link to us
I was going to try to contact webmasters to see if they would remove some of our spammy links. I see alot of them in MOZ but when you go to the site our anchor text is not there. Is this good? How often does MOZ refresh external links. Please see: http://www.opensiteexplorer.org/anchors?site=www.totalvac.com None of the links for the anchor text <a class="clickable title link-pivot" title="See top linking pages that use this anchor text." data-text="vacuum cleaner parts vacuum parts vacuum bags vacuum cleaner bags" data-id="46391436859">vacuum cleaner parts vacuum part...</a> in MOZ exist? We got hit extremely hard by Penguin in May
Link Building | | totalvac0 -
Should I contact webmasters to remove URL's to my site from domains with a low DA?
Should I be contacting webmasters to remove URL's from domains with a low DA - to remove low quality links? Thanks, -Mike
Link Building | | naturalsociety0 -
Back linking to t foreign sites
One of our major competitors seems to be linking to an Asian speaking website that has a blog/product review formant. I was wondering how they are achieving this. Are there any non English /Asian sites worth submitting to and what is the best way to go about this if English is the only language you speak! And of course is it worth while doing this from an SEO perspective.
Link Building | | Hardley10 -
SEOmoz's ranking of links on competitor question..
I am Perplexed about SEOmoz crawl for new links on a competitors website. When I looked at a company who received very high ranks for the links...on SEOMoz...such as. 95, 96, 97 etc..... I couldn't figure out how they ranked so high. One company bought 20 links for a couple hundred dollars and all of them were ranked very high on SEOmoz but when I used the page rank tool on them individually they were either "0" or "1" and none were over a 2. I was debating submitting for those since I am just starting out and wanted to get our name listed in the internet world. Are the high rankings by SEOmoz related to something else? Would I be better off buying a listing in some of the premium directories instead. Like Yahoo or BBB or Manta. ( After I get my site optimized first) Thank you, Greg
Link Building | | Boodreaux0