Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Backlinks not indexed on MOZ
Why our backlinks do not show up? Even GSC shows a good amount of our backlinks and other applications show too. But in the case of MOZ, it still shows 0 backlinks and 0 organic visibility. Don't you guys think this puts a user of your software way behind in terms of competition and real data? If my backlinks will be considered by MOZ, when will it be❓
Link Building | | MariApps0 -
JPG's listed as the Top Pages in Moz
In Moz, under Links, my Top Pages are all .jpg's. Is this unusual? Should I change the names of those jpg's and redirect the jpgs listed on the Top Pages to the html page it can be found on?
Link Building | | Boweb
Like instead of people linking to this page: https://www.freequilt.com/cat-and-the-fiddle.html it looks like from the Top Pages that people are linking directly to this image: https://www.freequilt.com/images/nursery-rhyme-series/cat-and-the-fiddle/moona.jpg
My thoughts are to rename that image so no one is linking to it, and redirect the image to the html page. Would I be hurting my Google ranking any by doing so?1 -
Paid Subscription Directory with Low Moz Open Site Explorer Spam Score
Looking at different backlinks of my competitors and came across a few directories that require a 1 time payment for the year. Is this considered the same as paying for a link or would this be considered something like getting a listing on your local Chamber of Commerce site? Also, I put the site through Moz's Open Site Explorer and it had PA:22 and DA:20, which is nothing incredible, but also had a spam score of 2. So would a site like this hurt my rankings? I know that's a good spam score, so I am a little confused with what to do. Here is the site: http://scl-online.net/en.htm Thanks for any help! Love this site.
Link Building | | aua0 -
Would it be a valid "link building' strategy to pay youtube video owners, to link to our company website in the decription of a certain video. ( For popular video's that are relevant )
I was wondering if it would it be a valid "link building' strategy to pay / work out a deal with youtube video owners, to link to our company website in the decription of a certain video they posted? ( For popular video's that are relevant to our business. ) Anyone have any thoughts on this? Thanks in advance! Steven
Link Building | | RockyMountainFlyboard0 -
Easy JOB - Ranking a Website For Easy Keywords (Moz Keyword Difficulty below 25)
Hello Moz Members,
Link Building | | Sebastian21
Easy JOB - Ranking a Website For Easy Keywords (Moz Keyword Difficulty below 25)
Ok so idea is simple here but I need some help because lack of time. I have 4 pages that I want to rank for these keywords in the next 1-2 months. Please contact me if you Done SEO in the past 3-5 years successfully and use White Hat Tecniques ! For more details contact me directly via email at sebastyan62@gmail.com or Contact@smartwebsiteflipping.com Thanks in Advance ! Keywords:
Sophia Hills
Mount Sophia Condo
sophia hills mount sophia
www.thesophiahills.com.sg high rental yield
dhoby ghaut rental yield
http://thesophiahills.com.sg/sophiahills/ Sophia Hills location
location map
sophia hills site plan
sophia hills location map
http://thesophiahills.com.sg/location-map/ Sophia hills floorplans
floor plans
floorplans
http://thesophiahills.com.sg/floor-plans/ THANK YOU !0 -
Can you recommend a link builder from the Moz list?
Does anyone have any experience with some link building companies on the Moz recommended list? There are a lot of them on there so I would like to hear from you if you have used them and what type of results you got. Thanks
Link Building | | netviper0 -
New article algorithm changes -- links allowed
hey guys, i understand that google has made algorithm changes for article writing in that only one link is allowed now per 500 words. would this include the "author resource" box if you don't place links in the body? any suggestions would be great: Due to the latest changes in Google's algorithms, regarding the proportions between the length of the content to the outbound links, ArticleSlash's requirements are changing effective immediately. • - For articles with up to 500 words, we will allow only one hyperlink • - For Articles up to 900 words - two hyperlinks • - For articles with 900 words and above - three hyperlinks Short articles with many outbound links are running the chance to get into Google's spam filters, which is why we are implementing this change, effective immediately.
Link Building | | tm46150 -
Should a site run it's own affiliate program to get inbound links?
I've recently signed an ecommerce client who runs their affiliate program through Commission Junction. If they were to bring their program in-house and eventually get all those affiliate links pointed to their domain, would those links be counted?
Link Building | | HunterW0