Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Why are there less backlink domains in Moz vs. Semrush?
For our domain studyville.com, Semrush is reporting 46 linking domains, and Moz is reporting 7. Does anyone know where there is such a large discrepancy?
Link Building | | shelbythomas0 -
How can I get my wesbite on Dmoz if it isn't accepting? Am I S.O.L.?
Each of the categories I would certainly fall under aren't accepting sites at this time because of it isn't showing the icon you need next to it to submit a website. I have been checking this for months, but it isn't changing. Is there another way in or am I just out of luck here? See step 3: http://www.dmoz.org/docs/en/add.html
Link Building | | pmull0 -
I am getting links on people's wordpress blogs but are not showing up on the just discovered tool. Is it true that wordpress links are no-follow links that do give off any link-juice?
A blogger told me that "wordpress.com does not allow blogs to show advertisements or use links to sites that sell merchandise of any kind" is this true? Am i wasting my outreach time trying to get links from wordpress operated blogs?
Link Building | | odegi0 -
Why don't some external links "count"?
A car-dealer client advertises through DexKnows, and the entry includes a link to the client's website. That link is not listed as a linking domain through OpenSite Explorer for the client's site. A competing car dealer also advertises through DexKnows, but that link is counted as a linking domain for the competitor. Why the difference? (I'm new and still learning -- linkbuilding appears to be my weakness.) Thank you!
Link Building | | TheOptimizer690 -
Weird change in amount of links
We just went from 50.000 external followed links to more than 150.000 ext followed links within a week. At the same time we went from just below 200.000 total links (internal/external) to more than 650.000 links and linking root domains dropped from around 750 to below 500. We don't do linkbuilding. We don't use a seo-agency. We do all stuff on our own. So why this major change and what impact will it have?
Link Building | | alsvik0 -
Remove links or change anchor text?
I am currently in the process of cleaning up the link profile for a website that has been hit by Penguin thanks to loads of links from free directories with exact match keyword anchor texts (about 200 root domains from total of 300 root domains). I was wondering whether it's best to remove these un-natrual keyword anchor text links altogether, or change the anchor texts to brand (domain name, domainname.com, www.domainname.com, http://www.domainname.com)? I am currently trying to remove these links but was thinking it would be quicker to get to a healthier link profile (in terms of brand/commercial anchor text split) by altering the anchor texts and not removing them. Some of these directories are the worst of the worst on the other hand. Also note that I'm only really getting about a 30% response rate from the owners of these directories. Any thoughts? Many thanks in advance.
Link Building | | ec9awp0 -
Should I Just Copy A Competitor's Backlinks?
Forgive the newbie question, but now that I have found SeoMoz and OpenSiteExplorer, should I just piggy back on my competitors backlinks? What would be the downside? By way of explanation, I've never had the need to explore SEO before. Our site, Widgets.com has always ranked highly for all Widgets keywords because we have the keyword in our domain and our site has been around since 1998. But out of the blue this summer, a site, let's call them WidgetsCircus.com suddenly began outranking us on widgets keywords, and pretty much every keyword we can imagine in our little widget universe. Now that I have run OpenSiteExplorer, I can see how they've done it. They've pretty much spent the last year commenting on blog posts all over the place, editing wiki pages, etc., and built thousands of links for all these widget keywords. So, I'm wondering: why shouldn't I just go down the list of links and do exactly what they've done? Where they commented on a blog, why don't I just comment right along side them. Obviously, this has worked for them! Wouldn't it work for us too? Or is that too simple?
Link Building | | brianmcc0 -
Why doesn't the Better Business Bureau show up in my link analysis
I've been working on SEO for one of the companies I've designed a website for and I'm confused by the company's lack of Better Business Bureau backlinks. The Company in question does have a BBB account and that account links back to the company's website. However, when I check in the link analysis for the site, the BBB link doesn't appear. My competitors, on the other hand, do have BBB links in their analyses. So, I'm wondering if I somehow don't have the right type of BBB account. The BBB seems to be a pretty good place to have a link from, and the company pays $300.00 per year for the membership, so I'd like to get the most out of it. Here's a link to the BBB page for the company http://www.bbb.org/utah/business-reviews/plumbers/platinum-plumbing-services-in-west-jordan-ut-22199778#bbblogo And here's the company's website www.slcplumbing.com Now, the company site I've just listed is 301 redirected to www.platinumplumbinginc.com, but even when www.slcplumbing.com was the main site, the BBB backlink didn't show up. Thank you Blake
Link Building | | BlakeMcGillis0