Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What happens to your domain authority if you are forced to change your website name?
Out of curiosity if for legal reason you or a client were forced to change their domain name, what happens to all of your domain authority once you lose the old domain? While you still own the domain you can easily redirect any old links to your new website, but what happens if you no longer own that domain and someone else purchases it? Do they steal all of your domain authority? Or is it possible to keep those old links pointing toward your website?
Link Building | | SteveKettig0 -
Links still showing in MOZ Reports
Hi, we have some links that have definitely been removed from a website but are still showing in our moz reports, and are not showing a date lost... They were removed months ago Why would be still be showing in Moz reports?
Link Building | | Antdxb0 -
Give a Solution my DA is not increasing and not index my backlinks in MOZ??
DA of my website online is no longer increasing. It seems like Moz is not indexing my backlinks. Multiple backlinks of my website online BasketballShoes are now not indexing in Moz that is why DA is not increasing.
Link Building | | fbowable
Backlinks of my site have indexed in both Ahref and semrush each now not showing in Moz. DR has elevated to 51 however DA is still 7. If all and sundry have the solution please let me know.0 -
301 redirect of PR 6 domain to PR 1 site, Will it improve my site's seo?
I have a pagerank 1 website but i recently purchased one old pagerank 6 domain, if I 301 redirect this pr6 domain to my currrent pr1 site, will it will increase my site's pr and improve seo ?
Link Building | | IndiaFPS0 -
What's the best seo link building company after May 2013?
Which is the best and most trustworthy seo company if I want to outsource my link building after being severely hit by the google penguin 4 update?
Link Building | | naimelhajj0 -
Should you ever change your anchor text ?
Hello I have a question about anchors. I have done all my own seo over the last 3 years, with tools from various sites. I had an seo audit done about 1 month ago and was told my link profile was very natural. They had one recommendation. To go back over my link profile and ask some webmasters if they would change the anchor from the name of my site or my url to a more seo friendly phrase. This seemed logical. I never did a lot of anchor text just name or site or url. Anyway, over last 4 weeks I have messaged several webmasters and asked to have anchor text changed to something along the lines of the keywords Im targeting. Tedious task to go through all the links but I changed several anchors to what was recommended. I was also out link building at same time. These last links and I got several of them all natural links after 16 hours work days. Are all will seo friendly anchors, because as Ive gotten more experienced my links have gotten more in lines of what is "seo friendly" or at least I hoped. I asked one webmaster to change my anchor and he warned me I would be slapped with a penguin penalty and wouldn't recommend I do this. I have already done this to several of my links. Then today the new seomoz update came up and I was down on DA and PA by 3-4 points. Do these have anything to do with one another and have I been given bad advice and can I fix it if I have ? Sorry about long post just a little confused. I don't want to step into penalty land and not know I did.
Link Building | | New1000ad0 -
What percentage of an old post can I change without lose rank?
A few months ago I updated some old posts of my site (Wordpress), to improve old content, with new pictures (better quality), better content, new info about product, and links to our review of this product (newer posts). Most part of the content is this old posts were new, so I "deleted" the old text but replaced the new one with another of better quality. I suppose it will help new post to improve rank, but I los thousands of daily visits, because old posts ranks worse since the change. What do you think? What happened here? Maybe I shouldn't update the whole text, only add a link to new content? Thank you!
Link Building | | DSG0 -
SEO MOZ LINK BUILDING TOOLS
So I learnt that in link building you need to send an email to someone from where you want the inbound link coming in from. In this case, are there any tools withing SEO moz that allow you toactually search for links and also provide you the contact email? Thank you, Vijay
Link Building | | vijayvasu0