Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Do i have too many 'follow' backlinks and am i being penalised by Google for it?
Hi all. I read on Moz recently that if a website has too large a percentage of 'follow' backlinks, that Google penalise the website because that is unnatural. IS this correct please? I ask because i have recently found that our own website, according to Moz, has 16,500 inbound links and they are ALL 'follow' links. These are all from independent 3rd parties and we havent commissioned any of them, so it is completely natural. URL if anyone cares is www.themosquito.co.uk Any advice would be appreciated. Cheers
Link Building | | TheMozzy0 -
One month later, MOZ has not scanned my web
It's been more than a month since we published our new website, you can see the link in my profile, and MOZ has not scanned or taken out the links to my page.
Link Building | | Expansyon
Do you know any way to tell MOZ that the page is published?
I have checked my robots.txt and everything seems to be ok. Google search console, takes out all the links correctly but MOZ is not able to.
Thanks to all of you for helping me1 -
Hi I changed my site to https://www.cocaineteskit.co.uk - now unsure
Hi ALL, just changed over to https//.www.cocainetestkit.co.uk with the money being in wholesaling. However I am having my links poorly indexed - any suggestions?
Link Building | | AndreavanEugen0 -
How much 'ranking power' would a link from a privacy policy page of a Big Brand have if the site content is totally unrelated?
Let's say that Site A is a large brand with high authority ( Moz shows it at 90 DA). Site A is about “Blue Shoes”. Site B is about “fishing”. If Site A links from their privacy policy page to Site B … would that inbound link carry enough weight in a way that could impact rankings for Site B given that the content is totally unrelated and it's a link from a privacy policy? I’m asking if the work to get clients/partners do this help. I have a hypothetical question for you all! Site A is a large brand with high authority ( Moz shows it at 90 DA). Site A is about “Blue Shoes”. Site B is about “fishing”. If site A links from their privacy policy to Site B … would that inbound link carry enough weight in a way that could impact rankings? I’m asking if the work to get clients/partners do this help.
Link Building | | RosemaryB0 -
Linkbuild on 'new' or 'old' url?
Hi, I'm trying to rank for a keyword. For the last year and 4 months the rank hasn't changed much. It stays on the 3th page. Within this period the rank has first gone down and then up. Three months ago I started doing linkbuilding for the url the keyword was linked to. This is a main category page. Lately, I don't know when exactly I've discovered that the keyword is ranking for a different url, a product category of these items for 1 brand. I'm wondering if it's wise to shift my focus and start using the 'new' url for linkbuilding? The 'old' url isn't in the top 100 for the keyword. However, if you search for the url in Google, it shows up. And, if I'm advised to shift my focus to the 'new' url, is it advisable to go back and change the backlinks to the 'new' url? A question related to the above one, is there another tool like Open Site Explorer which shows you an overview of the backlinks directing to a deeplink? Like MajestickSEO. I'm trying to get the most extensive overview possible. Thanks in advance.
Link Building | | anubis20 -
Would it be a valid "link building' strategy to pay youtube video owners, to link to our company website in the decription of a certain video. ( For popular video's that are relevant )
I was wondering if it would it be a valid "link building' strategy to pay / work out a deal with youtube video owners, to link to our company website in the decription of a certain video they posted? ( For popular video's that are relevant to our business. ) Anyone have any thoughts on this? Thanks in advance! Steven
Link Building | | RockyMountainFlyboard0 -
Root Domain Link for Affiliate's Link
It seems my affiliate link: http://www.hrmsplugins.com?partners=21 is not being considered as a "root domain" backlink when this link is used on their website. Is there a reason for this?
Link Building | | delphia0 -
Fresh set of eyes on this page please. Why isn't it ranking?
Morning all, I'd really appreciate it if you could take a quick look at this page and see if I'm missing something here. The targeted keyword (wedding favours) is pretty competitive and the rank had been slowly improving until recently and we've now slipped to 25th on Google UK. I've added a "Pay with a tweet" button for our eBook which has been pretty well received (around 100 downloads) so far so the social side is better than our competition. I've also written a few guest blogs with links back to the page from a variety of sources. Here is the page analysis on OSE. If you could take a quick look and let me know if I'm missing anything here, it would be most appreciated! Thanks in advance.
Link Building | | Confetti_Wedding0