Captcha wall to access content and cloaking sanction
-
Hello, to protect our website against scrapping, visitor are redirect to a recaptcha page after 2 pages visited.
But for a SEO purpose Google bot is not included in that restriction so it could be seen as cloaking.
What is the best practice in SEO to avoid a penalty for cloaking in that case ?
I think about adding a paywall Json shema NewsArticle but the content is acceccible for free so it's not a paywall but more a captcha protection wall.What do you recommend ?
Thanks,Describe your question in detail. The more information you give, the better! It helps give context for a great answer.
-
In general, Google cares only about cloaking in the sense of treating their crawler differently to human visitors - it's not a problem to treat them differently to other crawlers.
So: if you are tracking the "2 pages visited" using cookies (which I assume you must be? there is no other reliable way to know the 2nd request is from the same user without cookies?) then you can treat googlebot exactly the same as human users - every request is stateless (without cookies) and so googlebot will be able to crawl. You can then treat non-googlebot scrapers more strictly, and rate limit / throttle / deny them as you wish.
I think that if real human users get at least one "free" visit, then you are probably OK - but you may want to consider not showing the recaptcha to real human users coming from google (but you could find yourself in an arms race with the scrapers pretending to be human visitors from google).
In general, I would expect that if it's a recaptcha ("prove you are human") step rather than a paywall / registration wall, you will likely be OK in the situation where:
- Googlebot is never shown the recaptcha
- Other scrapers are aggressively blocked
- Human visitors get at least one page without a recaptcha wall
- Human visitors can visit more pages after completing a recaptcha (but without paying / registering)
Hope that all helps. Good luck!
-
Well I'm not saying that there's no risk in what you are doing, just that I perceive the risk to be less risky than the alternatives. I think such a fundamental change like pay-walling could be moderately to highly likely to have a high impact on results (maybe a 65% likelihood of a 50% impact). Being incorrectly accused of cloaking would be a much lower chance (IMO) but with potentially higher impact (maybe a 5% or less chance of an 85% impact). When weighing these two things up, I subjectively conclude that I'd rather make the cloaking less 'cloaky' in and way I could, and leave everything outside of a paywall. That's how I'd personally weigh it up
Personally I'd treat Google as a paid user. If you DID have a 'full' paywall, this would be really sketchy but since it's only partial and indeed data can continue to be accessed for FREE via recaptcha entry, that's the one I'd go for
Again I'm not saying there is no risk, just that each set of dice you have at your disposal are ... not great? And this is the set of dice I'd personally choose to roll with
The only thing to keep in mind is that, the algorithms which Googlebot return data to are pretty smart. But they're not human smart, a quirk in an algo could cause a big problem. Really though, the chances of that IMO (if all you have said is accurate) are minimal. It's the lesser of two evils from my current perspective
-
Yes our DA is good and we got lot of gouv, edu and medias backlinks.
Paid user did not go through recaptcha, indeed treat Google as a paid user could be a good solution.
So you did not recommend using a paywall ?
Today recaptcha is only used for decision pages
But we need thoses pages to be indexed for our business because all or our paid user find us while searching a justice decision on Google.So we have 2 solutions :
- Change nothing and treat Google as a paid user
- Use hard paywall and inform Google that we use json shema markup but we risk to seen lot of page deindexed
In addition we could go from 2 pages visited then captcha to something less intrusive like 6 pages then captcha
Also in the captcha page there is also a form to start a free trial, so visitor can check captcha and keep navigate or create a free account and get an unlimited access for 7 days.To conclude, if I well understand your opinion, we don't have to stress about being penalized for cloaking because Gbot is smart and understand why we use captcha and our DA help us being trustable by gbot. So I think the best solution is the 1, Change nothing and treat Google as a paid user.
Thank a lot for your time and your help !
It's a complicated subject and it's hard to find people able to answer my question, but you did it -
Well if you have a partnership with the Court of Justice I'd assume your trust and authority metrics would be pretty high with them linking to you on occasion. If that is true then I think in this instance Google would give you the benefit of the doubt, as you're not just some random tech start-up (maybe a start-up, but one which matters and is trusted)
It makes sense that in your scenario your data protection has to be iron-clad. Do paid users have to go through the recaptcha? If they don't, would there be a way to treat Google as a paid user rather than a free user?
Yeah putting down a hard paywall could have significant consequences for you. Some huge publishers manage to still get indexed (pay-walled news sites), but not many and their performance deteriorates over time IMO
Here's a question for you. So you have some pages you really want indexed, and you have a load of data you don't want scraped or taken / stolen - right? Is it possible to ONLY apply the recaptcha for the pages which contain the data that you don't want stolen, and never trigger the recaptcha (at all) in other areas? Just trying to think if there is a wiggle way in the middle, to make it obvious to Google you are doing all you possibly can to do keep Google's view and the user view the same
-
Hi effectdigital, thanks a lot for that answer. I agreed with you captcha is not the best UX idea but our content is sensitive, we are a legal tech indexing french justice decision. We get unique partnership with Court of Justice because we got a unique technology to anonymize data in justice decision so we don't want our competitor to scrap our date (and trust me they try, every day..). This is why we use recaptcha protection. For Gbot we use Google reverse DNS and user agent so even a great scrapper can't bypass our security.
Then we have a paid option, people can create an account and paid a monthly subscription to access content in unlimited. This is why I think about paywall. We could replace captcha page by a paywall page (with a freetrial of course) but I'm not sur Google will index millions of page hiding behing a metered paywall
As you said, I think there is no good answer..
And again, thank a lot to having take time to answer my question -
Unless you have previously experienced heavy scraping which you cannot solve any other way, this seems a little excessive. Most websites don't have such strong anti-spam measures and they cope just fine without them
I would say that it would be better to embed the recaptcha on the page and just block users from proceeding further (or accessing the content), until the recaptcha were filled. Unfortunately this would be a bad solution as scrapers would still be able to scrape the page, so I guess redirecting to the captcha is your only option. Remember that if you are letting Googlebot through (probably with a user agent toggle) then as long as scrape-builders program their scripts to serve the Googlebot UA, they can penetrate your recaptcha redirects and just refuse to do them. Even users can alter their browser's UA to avoid the redirects
There are a number of situations where Google don't consider redirect penetration to be cloaking. One big one is regional redirects, as Google needs to crawl a whole multilingual site instead of being redirected. I would think that in this situation Google wouldn't take too much of an issue with what you are doing, but you can never be certain (algorithms work in weird and wonderful ways)
I don't think any schema can really help you. Google will want to know that you are using technology that could annoy users so they can lower your UX score(s) accordingly, but unfortunately letting them see this will stop your site being properly crawled so I don't know what the right answer is. Surely there must be some less nuclear, obstructive technology you could integrate instead? Or just keep on top of your block lists (IP ranges, user agents) and monitor your site (don't make users suffer)
If you are already letting Googlebot through your redirects, why not just have a user-agent based allow list instead of a black list which is harder to manage? Find the UAs of most common mobile / desktop browsers (Chrome, Safari, Firefox, Edge, Opera, whatever) and allow those UAs plus Googlebot. Anyone who does penetrate for scraping, deal with them on a case-by-case basis
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Duplicate Content Question With New Domain
Hey Everyone, I hope your day is going well. I have a question regarding duplicate content. Let's say that we have Website A and Website B. Website A is a directory for multiple stores & brands. Website B is a new domain that will satisfy the delivery niche for these multiple stores & brands (where they can click on a "Delivery" anchor on Website A and it'll redirect them to Website B). We want Website B to rank organically when someone types in " <brand>delivery" in Google. Website B has NOT been created yet. The Issue Website B has to be a separate domain than Website A (no getting around this). Website B will also pull all of the content from Website A (menus, reviews, about, etc). Will we face any duplicate content issues on either Website A or Website B in the future? Should we rel=canonical to the main website even though we want Website B to rank organically?</brand>
Intermediate & Advanced SEO | | imjonny0 -
Tabbed content impact
Hi all, I know historically tabbed content has been devalued, what's the situation currently? I've heard a lot about mobile first changing this. This is a design that has been produced by our designers: https://i.gyazo.com/35f655c7ba2bc89a87b9476e4a14534d.png Each tab contains approx 1000 words and previously has been a unique article. Would love to know your thoughts on this design and the benefits/losses of doing it like this. Thanks, Tom
Intermediate & Advanced SEO | | ThomasHarvey0 -
Block lightbox content
I'm working on a new website with aggregator of content.
Intermediate & Advanced SEO | | JohnPalmer
i'll show to my users content from another website in my website in LIGHTBOX windows when they'll click on the title of the items. ** I don't have specific url for these items.
What is the best way to say for SE "Don't index these pages"?0 -
What is Best Way to Scale RCS Content?
SEO has really moved away from the nitty gritty analysis of backlinking factors, link wheels, and the like and has shifted to a more holistic marketing approach. That approach is best described around MOZ as “Real Company S#it”. RCS is a great way to think about what we really do because it is so much more than just SEO or just Social Media. However, our clients and business owners do want to see results and want it quantified in some way. The way most of our clients understand SEO is by ranking high on specific terms or online avenues they have a better possibility of generating traffic/sales/revenue. They understand this more from the light of traditional marketing, where you pay for a TV ad and then measure to see how much revenue that ad generated. In the light of RCS and the need to target a large number of keywords for a given client, how do most PROs handle this situation; where you have a large number of keywords to target but with RCS? Many I’ve asked tend to use the traditional approach of creating a single content piece that is geared towards a given target keyword. However, that approach can get daunting if you have say 25 keywords that a small business wants to target. In this case is not really a case of scaling down the client expectations? What if the client wants all of the keywords and has the budget? Do you just ramp your RCS content creation efforts? It seems that you can do overkill and quickly run out of RCS content to produce.
Intermediate & Advanced SEO | | AaronHenry0 -
CPanel Redirect not allowing login access.
Using the redirect function in cPanel I am able to create the 301 redirect that I need to not have duplicate content issues in Moz. However, the issue now is that when I try to login to domain.com/login it redirects to domain.com/index.php?q=admin, which is not a page on the site and I can no longer login. I have checked the htaccess file and it appears that the entry is correct ( I originally thought that the cPanel redirect was not writing access correctly ). I am not sure if there is a small detail that I am missing with this or not. So my main question is how do I redirect my site to remove dup content errors while retaining the login at domain.com/admin and not be redirected to domain.com/index.php?q=admin? Thank you ahead of time for your assistance.
Intermediate & Advanced SEO | | Highline_Ideas0 -
Aggregators outranking me for my own content
WARNING : The follow question is for an adult website. If you are at work, near children or are offended by such material, DO NOT CLICK Hey guys, This one has had me stumped for awhile. I operate www.deviantclip.com. Its a very old and trusted domain by google with loads of history. However, in the past year, Google has been giving me the cold shoulder. One major problem I've noticed is that I've lost all longtail traffic. Its even gotten to the point where aggregators are outranking me in google for my own custom titles and content. **Example A : ** Google Link 1 This search has my own sitename in the title and my site ranks somewhere on page 2 or further. **Example B : ** Google Link 2 This content originated from our site and has a unique title, yet we're dead last in the serps. I submitted my site for reconsideration a few times, and the outcome everytime is that Google tells me they have not applied any manual penalty. There are a TON of issues to adress with this site, but obviously, getting my own content to rank first is the primary problem I would like to fix. Your time and advice is greatly appreciated. If you need furter info, don't be afraid to ask.
Intermediate & Advanced SEO | | CrakJason0 -
What is the best way to allow content to be used on other sites for syndication without taking the chance of duplicate content filters
Cookstr appears to be syndicating content to shape.com and mensfitness.com a) They integrate their data into partner sites with an attribution back to their site and skinned it with the partners look. b) they link the image back to their image hosted on cookstr c) The page does not have microformats or as much data as their own page does so their own page is better SEO. Is this the best strategy or is there something better they could be doing to safely allow others to use our content, we don't want to share the content if we're going to get hit for a duplicate content filter or have another site out rank us with our own data. Thanks for your help in advance! their original content page: http://www.cookstr.com/recipes/sauteacuteed-escarole-with-pancetta their syndicated content pages: http://www.shape.com/healthy-eating/healthy-recipes/recipe/sauteacuteed-escarole-with-pancetta
Intermediate & Advanced SEO | | irvingw
http://www.mensfitness.com/nutrition/healthy-recipes/recipe/sauteacuteed-escarole-with-pancetta0 -
301 redirect for duplicate content
Hey, I have just started working on a site which is a video based city guide, with promotional videos for restaurants, bars, activities,etc. The first thing that I have noticed is that every video on the site has two possible urls:- http://www.domain.com/venue.php?url=rosemarino
Intermediate & Advanced SEO | | AdeLewis
http://www.domain.com/venue/rosemarino I know that I can write a .htaccess line to redirect one to the other:- redirect 301 /venue.php?url=rosemarino http://www.domain.com/venue/rosemarino but this would involve creating a .htaccess line for every video on the site and new videos that get added may get missed. Does anyone know a way of creating a rule to rewrite these urls? Any help would be most gratefully received. Thanks. Ade.0