Why Moz OSE, Ahrefs, Majestic and so on, don't change their user agent while crawling?
-
Some blackhat websites, PBNs and other "cheaters" are using various methods to effectively block third party backlink checker bots (OSE, Ahrefs, Majestic...) : robot.txt, IP and such.
A simple solution for those bots would be to mimic Google by using its user agent string for example.
Or if not legally permitted (which I doubt) use some kind of randomness in user agent strings, urls, and IPs in order to prevent blocking.This should not be a big deal IMHO, am I missing something obvious ?
-
The ethics of the Internet dictate that you
- crawl politely,
- obey robots.txt and
- properly identify yourself
This isn't a new issue. Link networks and sites have blocked crawlers and manipulated Google for years. Fortuneatly, it's only a small fraction of the web. Also, it unlikely links from those networks have much value, so crawl priority would be super low anyway.
Actually, it could be viewed as beneficial when blackhat sites block OSE and aHrefs, because those sites often get penalized by Google, but 3rd party crawlers have no way to know this, so blocking effectively keeps them out of the indexes.
-
Well, I think bot blocking is an obvious problem even now, and will be more important tomorrow with all private networks as you can imagine.
MOZ (and others) should find and implement the best possible solution, I see no problem with TAGFEE as soon as you are transparent with regards to the fact that your bots are undetectable.
I understand that what I'm proposing is maybe not best nor wanted solution, but the problem must be addressed or OSE will soon have no value at all
What do you propose ?
-
I agree with George here -- we'd hear a huge outcry if we pretended to be Googlebot or a different bot. We'd also likely get blocked, as sometimes people only let in a certain few known bots/IPs to crawl their site. If we changed user agents and IPs regularly, it would not be cool or TAGFEE.
-
What about using different user agents and IPs regurarly in order to avoid detection ?
Is there any acceptable other solution ?
-
The reputation and integrity of the major players would be at stake here. If they changed their user agent identification (to spoof Googlebot or Bing or whatever) that could be detected, and they would be castigated. The crawler IP address and its user agent ID would be out of sync...
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Confused over PRWeb and it's Moz PA/DA
Hi I was looking at using PRWeb for some link building, however I have read quite a few articles bemoaning it and saying it carries very little, if any weight with Google. I am therefore confused that in Moz it has a DA of 90 (!) and a PA of 38, with a spam score of 1%. Are these negative articles wrong...is Moz ranking it wrong...or have I got the wrong end of the stick somewhere along the line?! Thanks Bob
Link Building | | BobBawden10 -
When buying used domains, how do i see the links pointing to that domain? OSE not showing links
when buying used domains, how do i see the links pointing to that domain? Sometimes the open site explorer doesn't show any links to the domain, especially if the domain is parked. Obviously a domain for sale with 1000 domains linking to it has lots of SEO Value right? Thanks mozzers!
Link Building | | Ron100 -
What's Your Most Jaw-Droppingly Spammy Link ( or Promo) Request?
Sheesh. I get them every day. But this one takes the cake. It's from a woman I met once in my life on a long run with my running club. That was two years ago. Now she wants me to endorse her -- even though I know nothing about her professionally. But wait, there's more. She's not much of a runner. As it happens, I know a thing or two about fitness -- having edited a fitness magazine and worked on websites for many top fitness and nutrition pros. The idea of an MSW doing fitness programs for people with medical problems raises all kinds of red flags. She sent this Facebook IM: Hi Daniel - hope all is well! I was wondering if you can share my page with your friends, with the following message (or something similar). Thank you so much in advance! My friend XXXXXX, MSW, has been running XXXX Personal Training for the past 18 years. She specializes in exercise programming for people who have medical problems, everything from heart problems to chronic pain conditions. She works on site in her clients' homes in midtown and downtown XXXXXX. Please contact her if you know anyone who would benefit from her expertise.
Link Building | | DanielFreedman
http://www.facebook.com/pagesXXXX
XXXX Personal Training
We specialize in helping people who are experiencing medical issues incorporate exercise into their treatment plan. We work as a team with all other professionals involved in your care, whether that is your medical doctor, naturopath, physiotherapist, osteopath, chiropractor, massage therapist, or...
Page: 141 like this [note from SEOmoz staff: we ask that you keep this post TAGFEE and not include specific names, URLs, or phone numbers in your response. Thanks!]2 -
Should you ever change your anchor text ?
Hello I have a question about anchors. I have done all my own seo over the last 3 years, with tools from various sites. I had an seo audit done about 1 month ago and was told my link profile was very natural. They had one recommendation. To go back over my link profile and ask some webmasters if they would change the anchor from the name of my site or my url to a more seo friendly phrase. This seemed logical. I never did a lot of anchor text just name or site or url. Anyway, over last 4 weeks I have messaged several webmasters and asked to have anchor text changed to something along the lines of the keywords Im targeting. Tedious task to go through all the links but I changed several anchors to what was recommended. I was also out link building at same time. These last links and I got several of them all natural links after 16 hours work days. Are all will seo friendly anchors, because as Ive gotten more experienced my links have gotten more in lines of what is "seo friendly" or at least I hoped. I asked one webmaster to change my anchor and he warned me I would be slapped with a penguin penalty and wouldn't recommend I do this. I have already done this to several of my links. Then today the new seomoz update came up and I was down on DA and PA by 3-4 points. Do these have anything to do with one another and have I been given bad advice and can I fix it if I have ? Sorry about long post just a little confused. I don't want to step into penalty land and not know I did.
Link Building | | New1000ad0 -
Whoa 1000's of links from Industrial Interface?
Hello all! I just took over an account, and in webmater tools the site has thousands of links to its homepage from a site named http://www.industrialinterface.com. Not sure if this is a good or bad thing. (thinking bad) Tried to contact the webmaster, and the contact form does not work, so that right there is a bad sign. Does anyone have an opinion on industrial interface? Anyone have luck in reaching them? Appreciate the feedback! Dorian
Link Building | | drufast10 -
Changing Anchor Text and Domain Name on external sites
Hey guys, I was hoping somebody might be help with my current dilema. We have a international website due to go live soon which has changed its brand name. It is International educational website funded by the government is all I can tell you I'm afraid. They have over 40,000 inbound links many of which are images. I'm wondering what i site best approach. To contact the web master of the top PR sites and ask them to change the listing to the new brand? I was also thinking if I was to leave most of them there as they would be redirected anyhow. Could I be clever and add the new brand link to some of these sites without removing the old and reap the benefits of having two links, the old site url and the new site url? Here is the main dilema though, the commission wish to keep the old site live for 6 months before we can redirect. Thanks, Rob
Link Building | | daracreative0 -
Curating Content. used to avoid, but now i'm having second thoughts
It's really hitting me now, because content curating is extremely common in a lot of other forms of media. I always avoided it because I didn't know the extent of any negatives that I might see from the search engines (duplicate content). Does anybody else curate content for their blogs? The main problem that I am having is that I just don't have enough time to publish the amount of content that I need. Outsourcing is the best alternative, and quite frankly, unless your high school English teacher needs extra money, it really isn't a very good one. Basically, i'm looking at content curating as a really good way to publish a lot of content, on more topics, a lot easier. What extent do you consider reasonable? How do you go about creating your content? What would you say is the easiest way to create content efficiently?
Link Building | | TylerAbernethy0 -
Multiple KW's , on-page and anchor text
Hello, For each page on my site, I've targeted one primary keyword and three to four secondary keywords. All of the keywords variants are tightly themed. With some on-page, I've ranked page two or three for all of the keywords and many are starting to convert based on Analytics data. Each page scores an "A" using the SEOmoz KW targeting tool for the "primary keyword only". For secondary keywords, I've only included words but not the complete keyword. For example, if the primary keyword is "blue green widgets" and the secondary keyword is "get blue green widgets", I've included the word "get" throughout the copy to target the secondary keyword. My questions are... Should I include each secondary keyword once in the copy and not just the word "get" for example? Just wondering if there is a better approach to target all of the keywords via on-page. When getting links to each page, how would you vary the anchor text to target all of the keywords, primary and secondary? Thanks!
Link Building | | ShaneO0