Split test content experiment - Why can't GA identify a winner here?
-
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.
Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?
Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?
Image 1 - Showing e-commerce performance (objective of split test was transactions)
Image 2 - Showing conversions (same split test, same objective, just different report)
Your thoughts and comments will be appreciated.
-
Hey Emeka,
Short Answer:
You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.Long Answer:
The math behind this statistical significance calculation is:
Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.
Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates
- Conversion Rate is: Conversions/Sessions.
- Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)
Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.- Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
- For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.
Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.
Hope this helps! Let me know if you have any further questions on this!
Trenton
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Individual statistics (not aggregated) for 'local pack' and 'organic search results' when both on the same page
Hi, For a particular keyword combination we show up on the local pack but also on the organic search results on the 1st page. I have never been able to find out what the CTR, #clicks, conversions for this two different kinds of positions are individually. The aggregate figures I can find out in Google Search Console / Analytics, but I would like to know on an individual level as I am testing out different things. With the statistics on Google My Business I can't get along actually...no CTR as far as i know. Any hint? Cheers, Cesare
Conversion Rate Optimization | | Cesare.Marchetti1 -
What are the best practices for server-side A/B testing?
When performing a full-site server-side A/B test, what best practices should you follow to ensure that you're: not cloaking or at least not being shady only showing Google (and other search engines) the version you want them to see (i.e. not the test pages) Thanks in advance!
Conversion Rate Optimization | | vsem0 -
Meta Descriptions - Does Cutt's comment still hold true?
When looking through my old research on Meta's came across this article on Search Engine Land where Matt Cutt's stated... "In fact, Matt said for his own blog, he doesn’t bother to make meta descriptions for his own site. In short, it is better to let Google auto-create snippets for your pages versus having duplicate meta descriptions."(http://searchengineland.com/googles-matt-cutts-dont-duplicate-your-meta-descriptions-177706) Recently, I have seen Google continuously play around with my metas, pulling from some really weird places on the page, discrediting the meta that I have there..and more-or-less really f them up.. Does the community think that maybe what Matt said in 2013 still holds weight and we are just wasting time with metas since Google does what they want in that space anyways?
Conversion Rate Optimization | | GoAbroadKP2 -
Can using the Google Keywords tool and/or SEOmoz cause problems for my company's IP address?
I just started with a company a few weeks back, working on-site doing SEO work. I've been using SEOmoz as well as Google Adwords' Keywords tool. I just got a message from one of our reps saying that people at the office (and this is a huge office) have reported tons of CAPTCHAs being thrown up frequently. I told them I had been running searches with Google Keywords tool and SEOmoz, but would that really cause that much of a problem? Is there anything I can do in order to fix this, some keyword finders that won't get me in trouble, etc.?
Conversion Rate Optimization | | SmokewagonKen0 -
Analytics (Other Than Google) & A/B Testing Tools
Hey, I have Google Analytics on my website, like most people, but I was looking at some other analytics and testing tools like heat maps, click maps, A/B, etc. I was hoping that some of you could recommend a few that you use to see where pages are going wrong, track user behaviour and to how improve conversion rate. I have had a look at a few websites like: https://www.clicktale.com/
Conversion Rate Optimization | | JonathanRolande
https://www.optimizely.com/ It would also be interesting to know how acurate these tools actually are, they seem quite legit. Cheers0 -
What's the Best Method of Content Distribution
This question is both specific and open ended. I have a new web project I've been working on content for. I paid to have a really sweet research article written and I am in the process of turning it into an info-graphic as well. Being that the the website is so new it doesn't have many customers and few to no blog readership. Now that I will soon have great content for the site should I : A. Post on the Blog ? ( Its a Blogger on a blog. domain.com ) B. Post as a static page on the website ? (domain.com/content) C. Something Else ? Whats the easiest to promote and push traffic to?
Conversion Rate Optimization | | CDUBP
Which will rank best?
Which will rank longer?
Which will rank faster? Any other suggestions to get the content in front of the niche market?0 -
Does the word next to domain gives more relevancy to the page of it's URL?
Whether putting a (category or brand) word next to domain is better than on the end of an URL? (i.e. domain.com/sony/tvs or domain.com/tv/sony) Which one would get higher result on SERP "tv" or "sony" in both cases? Or maybe they both serves the same?
Conversion Rate Optimization | | komeksimas0 -
Can anyone recommend software for recording phone calls through Adwords?
Hey Mozzers, I want to start recording phone calls as a result of PPC campaigns. Can anyone recommend an easy to implement system for setting this in place which will allow me to track calls placed as a result of PPC advertising. The ability to record the calls for training purposes would be a bonus. Thanks in advance Geoff
Conversion Rate Optimization | | SEM-Freak0