Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Backlinks are indexed in Ahrefs But Not Indexed in MOZ. Why?
I Create backlinks for my website to Increase DA But Backlinks are indexed and Ahrefs Show that backlinks but MOZ is showing backlinks. I am confuse Can you explain?
Link Building | | Seogamesokay1a1 -
'spammy' domains redirecting to website
Hi Everyone, I hope that someone will be able to help us with this one as we have trawled the internet looking for a solution! We have multiple domains (.com/.co.uk/.net versions) which all point to the one website, however, some of these domains have a high spam score - 9-11. Our first initial reaction would be to remove the auto redirects, but, the other domains have been a source of conversion in previous months (or so analytics tells us). So what I'm wondering, is do we remove the 'spammy' links from redirecting to our site, or do we leave them there? We certainly don't want to risk a penalty. Thanks for reading!
Link Building | | hydra_creative0 -
I've published my forst infographic and started outreach to get links. Should I use a canonical URL and if so how?
I've published my first infographic and started outreach to get links. I've also submitted it to several infographic directories. Should I use a canonical URL and if so how?
Link Building | | roadhaulageservices0 -
Does having '?search' in a URL affect the page quality?
In our Costume Themes category on our website, http://www.costumedirect.com.au/themes/, the links direct to search pages with URLs containing the term '?search'. For example, the 'Edward Scissorhands' link directs to http://www.costumedirect.com.au/search.php?search_query=edward. Will the question mark and 'search' in our URL affect the page quality and rankings? Thank you.
Link Building | | CostumeD0 -
How long until links 'fall off'?
If I have site A linking to site B, and take down the links - does anyone have any experience in about how long they take to 'fall off', that is stop appearing in Webmaster Tools or Moz? I'm going on three weeks currently. Perhaps this takes months?
Link Building | | GFujioka0 -
Changing Anchor Text and Domain Name on external sites
Hey guys, I was hoping somebody might be help with my current dilema. We have a international website due to go live soon which has changed its brand name. It is International educational website funded by the government is all I can tell you I'm afraid. They have over 40,000 inbound links many of which are images. I'm wondering what i site best approach. To contact the web master of the top PR sites and ask them to change the listing to the new brand? I was also thinking if I was to leave most of them there as they would be redirected anyhow. Could I be clever and add the new brand link to some of these sites without removing the old and reap the benefits of having two links, the old site url and the new site url? Here is the main dilema though, the commission wish to keep the old site live for 6 months before we can redirect. Thanks, Rob
Link Building | | daracreative0 -
Links aren't showing up in SEOMOZ resports
Hi, I've been building links to my client's website for the past 3 weeks. I know that there are several sites that link to my client's website now but SEOMOZ's link analyses says there aren't any sites linking to my client's website. Anybody know what's up with that? Sincerely, Rex
Link Building | | Rex0 -
Think I'm ready to do some link building. Couple questions.
Getting ready to do some link building. I've got several lists of competitors' links, including a bunch of sites with broken links that would be a great fit to link to us. I've got a capable VA to get started work on reaching out to people. Just curious if this is the right game plan, seems a little simple: For this round of link building I'm thinking all the links would point to my root domain. -Find quality sites/links to go after -Find an email to the owner/webmaster -Have the VA send them a value proposition email(i.e. why it's good fit for all)...or tell them about broken links etc. -Follow up myself when a response is generated. -Hope/verify they link to us. Thanks for the help with the newbie questions.
Link Building | | astahl110