Google Analytics Content Experiments

[David_ODonnell]

Has anyone else found that Google Analytics Content Experiments seems to quite quickly favor the best performing variant in an experiment, and then show that variant many times more often than other/s - not split the traffic evenly?

What is Google's thinking behind 'optimizing' during an experiment? It seems odd to me.

[eminent]

Hi

do you know how google treats the content?

I assume the test page would need:

robots noindex
meta no index
Rel=canonical

or am i missing something?

Thanks
Paul

[David_ODonnell]

Gah, answered my own question (I swear I trawled the Google product forums for 20mins before asking!) source if anyone else has the same question in the future;

http://analytics.blogspot.com.au/2013/01/multi-armed-bandit-experiments.html

Summary:

Google Analytics uses a multi-armed bandit approach to managing online experiments. A multi-armed bandit is a type of experiment where:

• The goal is to find the best or most profitable action
• The randomization distribution can be updated as the experiment progresses

Twice per day, we take a fresh look at your experiment to see how each of the variations has performed, and we adjust the fraction of traffic that each variation will receive going forward. A variation that appears to be doing well gets more traffic, and a variation that is clearly underperforming gets less. The adjustments we make are based on a statistical formula (see the appendix if you want details) that considers sample size and performance metrics together, so we can be confident that we’re adjusting for real performance differences and not just random chance. As the experiment progresses, we learn more and more about the relative payoffs, and so do a better job in choosing good variations.

---

I'd still prefer the option of a true A/B test, but at least I have an official answer.