Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I'm stuck by internal linking.
What structure should a football website follow? Silo or Topic Cluster?
Link Building | | gogoanimetp
I need advice on my website. My website: https://tructiepbongda.site
I hope there are answers!
Thanks0 -
Grr SEO linking.. I am not understanding why I wouldn't have lots more links.. Please help. Thanks
I have done the whole moz open exploer and I am not understanding why my site wouldn’t have more links registering to my website.. I have lots of sites(directorys and 3rd party) with my website domain in them. The only one that is linking to my site is BBB.com and my advertsing with saint paul press. www.somerersetautodealer.com But if I have links with all kinds of automotive directories why wouldn’t they register? I am sure this a simply answer or that I am not understanding something. Thanks for your help! Scott
Link Building | | Scott12340 -
Google Don't allow to publish duplicate content for other website?
Hi All, How can share other website content in our website and same for other user, how can share our content on his website? Everyone is saying, sharing content will be good but Google saying you can add duplicate content so i want to know process for content sharing to earning natural links. Thanks, Akhilesh
Link Building | | dotlineseo0 -
Why Links from Top Level Domains doesn't Pass Link Equity?
Hello, I have a doubt about Equity-Passing links report of OpensiteExplorer.org According to Sam Weber http://moz.com/community/users/432678 links which pass value from one page to another including followed 301 and Meta refresh links are under Equity-Passing links. I am surprising after looking at the links report generated by opensiteexplorer.org. Most of the article and directory links which have low DA and PA are under Equity-Passing links. Whereas websites like EzineArticle or Articlebase which have good DA and PA are not found in either Equity-passing links or under only nofollow category. Please suggest me if the report of Opensiteexplorer is not good enough or the links from the site like Ezine Article doesn’t pass the link equity. Thanks.
Link Building | | TopLeagueTechnologies0 -
Eric Ward's urlwire.com - worth a $500 investment?
I'm after a turbo boost (direct and indirect) to my site's SERPs. How beneficial is URLwire.com? Anyone here used it? Worth $495? Given that Matt Cutts has said links from PR sites don't pass link pop, would URLwire fall into this category? (I'm aware the idea is also to generate backlinks from other referring sites). Thanks
Link Building | | Jeepster0 -
How do I help an author of family histories and biography's reach her niche's
Please have a look at this site for me http://www.louisewilson.com.au/ louise also has about six blogs on different books she has written. She has sold a few thousand copies which is great and has help people find out where they came from. Not many of these people have linked to her site's and she is not getting the traffic she deserve's for lots of long tail keywords and some broader ones. What simple on page changes can she make and what would be the best broader keyword's to go after. What would a good strategy be she has a very small ( tiny ) budget but can write and is enthusiastic , It's also a great way for her to be involved with people who are interested in simalar thing's. Oh yeah she has never used facebook of social networks how could she effectivly market her books and engage with potential reader's. Thanks in advance Oh and by the way Im not getting paid to help her Just incase you think i'm trying to get you to do my work for me. I just think this is an interesting case. I do do sales for an IT and SEO company but she is not one of our clients. So far I have just explained to her about keywords in titles meta tags and internal linking and just explained to her a bit about link building. But really need some help thanks very much. PS was this question too long?
Link Building | | duncan2740 -
What percentage of an old post can I change without lose rank?
A few months ago I updated some old posts of my site (Wordpress), to improve old content, with new pictures (better quality), better content, new info about product, and links to our review of this product (newer posts). Most part of the content is this old posts were new, so I "deleted" the old text but replaced the new one with another of better quality. I suppose it will help new post to improve rank, but I los thousands of daily visits, because old posts ranks worse since the change. What do you think? What happened here? Maybe I shouldn't update the whole text, only add a link to new content? Thank you!
Link Building | | DSG0 -
Do you think it's a good idea to try to find synergy between clients for blog posts/citations/links, or should you keep clients away from each other?
Say you have for example three (in this case) clients, and: Client A sells red widgets Client B is a doctor Client C sellls blue widgets With some research, you find that: Red widgets (A) can make the process of blue widget creation (C) even more effective. Red widgets (A) can protect you from harmful things that doctors (B) are qualified to recommend that you stay away from. Furthermore, there are things that doctors (B) recommend that you do in order to maximize the benefits of red widgets (A) Blue widgets (C) carry with them certain potential health risks, which according to doctors (B) can be minimized using the following means Sometimes blue widgets (C) can be used to effectively repair red widget (A) factories ...and so forth. Sure you're really writing these articles to generate links and exchange authority, and frankly you started with "how can I find synergy between these clients?" rather than a with a great article subject that needed a citation which luckily happened to be another client, but the citations are legitimate and the clients are qualified to speak on the subjects where their expertise and interests overlap. Would you consider going ahead with this? Does anyone have any experience doing it? I could see potential pitfalls if clients were to interact with each other, but keeping yourself as the intermediary might well work and overall it seems like a decent way to grab low-hanging fruit as they say. What do you guys think?
Link Building | | PathMarketing0