Split test content experiment - Why can't GA identify a winner here?
-
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.
Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?
Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?
Image 1 - Showing e-commerce performance (objective of split test was transactions)
Image 2 - Showing conversions (same split test, same objective, just different report)
Your thoughts and comments will be appreciated.
-
Hey Emeka,
Short Answer:
You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.Long Answer:
The math behind this statistical significance calculation is:
Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.
Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates
- Conversion Rate is: Conversions/Sessions.
- Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)
Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.- Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
- For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.
Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.
Hope this helps! Let me know if you have any further questions on this!
Trenton
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Adwords shows clicks but no conversions in GA
Hi Folks, I was wondering if anyone has come across a similar situation, I have setup adwords campaigns and included these in my Google Tag Manager setup and deployed the GTM snippet to a clients site. In adwords I can see clicks (2,000+) however in Google Analytics I can see only a handful ot conversions. Is there anyway I can test/track a conversion to ensure that it is being processed and report correctly? Also within AdWords Conversion Tracking I have noticed that under the "Webpages" tab that pages other than the pages that should have the conversion code on them are listed. Have I setup my Adwords camapign or GTM tags incorrectly? Any and all feedback or insight would be greatly welcomed and appreciated.
Conversion Rate Optimization | | icanseeu
Regards,
Dave0 -
Which eCommerce site you consider using best practices? Site we can learn from
Hi, I'm looking to hear thoughts and suggestions as per sites that you consider to have great practices in the eCommerce world. Almost none of the sites do everything good so you can split your suggestions by any criteria -
Conversion Rate Optimization | | BeytzNet
the site structure
conversion funnel
Converting product pages (good design)
content creation and blog management / structure
content marketing
SEO guidelines / practices
... Thanks0 -
Identifying crawler clicks
I have a website where i register outbound clicks on some affiliate links. But when crawlers and search engines pass my site i get clicks registered which are not customer clicks. How can I identify these clicks and not save records of them? -Rasmus
Conversion Rate Optimization | | rasmusbang0 -
Can using the Google Keywords tool and/or SEOmoz cause problems for my company's IP address?
I just started with a company a few weeks back, working on-site doing SEO work. I've been using SEOmoz as well as Google Adwords' Keywords tool. I just got a message from one of our reps saying that people at the office (and this is a huge office) have reported tons of CAPTCHAs being thrown up frequently. I told them I had been running searches with Google Keywords tool and SEOmoz, but would that really cause that much of a problem? Is there anything I can do in order to fix this, some keyword finders that won't get me in trouble, etc.?
Conversion Rate Optimization | | SmokewagonKen0 -
What's the most effective web marketing tactic you've seen or used that very few people know about?
I wanted to start a thread to share some of the really cool marketing tactics I've seen on the web that I think few folks are using, AND ask the community here what you've seen, too! Some of my favorite undiscovered or less-used tactics include: Making smart use of bios for conferences, events, interviews, etc. where folks ask you or your team members for a "bio" and you get to control the links, link targets, and anchor text. This is super powerful in my experience, so long as you have a moderately strong profile or regular participation in this type of stuff. Price anchoring on conversion pages, e.g. http://www.trackur.com/options - note how they start with the highest price to help "anchor" the audience to bigger numbers. A great principle of psychology in action. Using re-marketing to draw people to content rather than just purchase/conversion pages. The effectiveness of these is, I've heard, dramatically higher than the usual re-marketing campaigns that take you to a squeeze or purchase page. I can't share the example I'm thinking of, unfortunately, but I'd urge you to try it! Get more social shares and clicks by SHARING MORE THAN ONCE! A lot of folks feel like they are burdening their audience on Twitter/Facebook/G+ or frustrating them if they post multiple times, when in fact, very, very few of your followers are online at any given time. I've tested this myself and I get almost no negative feedback but can triple or more the number of shares/+1s/likes/visits/etc I get just by sharing 2-3X! The key is not to be too repetitive or annoying, and to acknowledge past shares (at least for me). e.g. I'll say "my blog post from last night on XYZ" and get a ton more clicks. What are your favorites? Please share!
Conversion Rate Optimization | | randfish16 -
Shorter checkout form converts better? Or can it be harmful?
I've frequently read that the shorter the checkout form the better. My checkout form has the following fields: First name Last name Email Username Password Retype password Card number Card expiration date CVV code Billing street Billing city Billing state Billing zip Here's the thing. Since it's a web-app, I don't need the "billing address" as I'm not physically shipping them anything. Should I remove it? The no-brainer answer seems to be "yes", but I'm wondering if folks don't see any billing address fields, it may look suspicious. Conversions for having it and not don't seem to make much difference, so I suppose I'm looking for some tie-breaker opinions. The "password" and "retype password" fields could be eliminated by emailing the user's a system-generated password. But once again, could a user see this as odd or suspicious and then abandon? Even if I tell them I'll be emailing them a password? They could be sensitive thinking we'd email the wrong email address due to system error or their own typo. I could also eliminate the CVV and not validate against that. But once again, could a user seeing the CVV gone become wary? As much as I'd like to have "guest checkout' it's not feasible. The app is tied to a logged in account, which would also make eliminating the "username" impossible. Based on all the above, I could trim down the form considerably, but would I be doing more harm than good? I could A/B test it, but I don't believe I have a sufficient number of users to test against. Everything I buy, physical or online app, has an address field, so perhaps folks are accustomed to filling out this stuff and I should just keep it to align with user expectations? Thanks.
Conversion Rate Optimization | | bluekite770 -
Keeping pages indexed but making sure they fill out a form before access - confusing Q's
OK so let me break down this little scenario we have going on. I work for a b2b company so we have a lot of gated content that is behind a form fill out - this is how we get a lot of our lead generation. Some pages that we have behind the form are showing up in search which allows people to view the documents bypassing the form. At first I thought, well why dont we just no index that page so that it does not appear in search. But then I thought it would be smart to keep the pages indexed to keep the SEO value, Is there a way to keep these pages indexed but make sure that when they click the link in the SERPS that they need to fill out the form in order to gain access to the document? Something on the backend that checks to make sure that the referral URL was completed or something like that? Anybody deal with this before?
Conversion Rate Optimization | | PatBausemer0 -
Can Adwords campaigns affect other campaigns performance?
I understand that keywords and adgroups within a campaign can affect the performance of other keywords within that parent campaign. I was interested to understand whether campaigns in my account can affect one another. I have an adwords account with around 30 campaigns. Some are heavy hitters and others get minimal to 0 traffic. Will these low quality campaigns drive up the costs of my high end campaigns?
Conversion Rate Optimization | | peigenesis0