Split test content experiment - Why can't GA identify a winner here?
-
I have been running a content experiment for a short while now and GA has just ended it saying it cannot determine a winner.
Looking at the images (links below), without any form of analysis I can already see a pattern of greater success in Variation 1. It ended with a 93% probability of outperforming the original yet the content experiment ended with no winner. Does this mean the 95% confidence threshold I set should've been lowered?
Ultimately I'm going to choose this as my winner but why didn't GA push it as the winner? Is there something I am missing?
Image 1 - Showing e-commerce performance (objective of split test was transactions)
Image 2 - Showing conversions (same split test, same objective, just different report)
Your thoughts and comments will be appreciated.
-
Hey Emeka,
Short Answer:
You're correct. Effectively what Google is saying here is that they don't have enough statistical confidence to definitely tell you that the variation is outperforming the original at a 95% confidence level, but they do at a 93.8% confidence level.
Quick Note: 95% is the lowest setting in GA Experiments.Long Answer:
The math behind this statistical significance calculation is:
Full credit to vwo.com for their A/B Testing Significance Calculator & doing all the work here.
Link to Image One - This is simply the data of the Control vs Variation & the Conversion Rate & Standard Error Rates
- Conversion Rate is: Conversions/Sessions.
- Standard Error is: √(Conversion Rate*(1-Conversion Rate)/Visitors)
Link to Image Two - Confidence Levels, Z-score, & P-value
To find if something is truly significant at a specific confidence level, we need to calculate the Z-score then use that value to find the P-value and from there we can determine the confidence level.- Z-score is: (Control Conversion Rate-Variation Conversion Rate)/√((Control Standard Error^2)+(Variation Standard Error^2))
- For the P-value, we need to calculate the normal distribution of the z-score with a mean of 0 and a standard deviation of 1. The easiest way to do this is to use an online tool, here's a link to your specific example.
Finally, we take the Confidence % expressed as a decimal (i.e. 0.90, 0.95, 0.99) and 1 minus these values (i.e. 0.1, 0.05, 0.01)
If the P-value is greater than the Confidence % or less than 1 minus the Confidence %, then it is significant, otherwise it is not. Let me explain that using our example:
at 95% confidence, our P-value needs to be <0.05 or >0.95. Since our P-value is .05799, it doesn't fit either of those requirements and such is not significant at that confidence level.I know that's a lot of math, but this is why Google Experiments is saying that the result is not statistically significant.
Hope this helps! Let me know if you have any further questions on this!
Trenton
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
What are the best practices for server-side A/B testing?
When performing a full-site server-side A/B test, what best practices should you follow to ensure that you're: not cloaking or at least not being shady only showing Google (and other search engines) the version you want them to see (i.e. not the test pages) Thanks in advance!
Conversion Rate Optimization | | vsem0 -
GA expert for publishers
Hi. I'm looking for a GA expert who has experience working with magazine publishers. Anyone have a lead?
Conversion Rate Optimization | | RicardoSalcedo0 -
Have My SEO Cake & Eat It - Mix Viral Content & Business Content?
Hello, I optimize for a fashion, eCommerce site which has catalogue pages and product pages. http://www.zando.co.za/ We're currently working on a large "blog" which tons of great content designed to attract readers, links and tons of long tail traffic. We'll be doing how-to videos, celeb interviews, top-tens, etc. etc. We'll even have a UGC community section eventually. Really great. Now, I want to use the links generated by this section to rank for generics like this: For example: I want to rank for "Lingerie" and I have this page set up as the target: http://www.zando.co.za/women/clothing/lingerie/ I create some amazing content that generates lots of links containing the word "lingerie" . That awesome content lives in:(hypothetical URL:) http://www.zando.co.za/fashion/magazine/how-lingerie-has-changed-over-50-years/ I want to (and this is where you come in 😉 get people linking to my catalogue page (http://www.zando.co.za/women/clothing/lingerie/) where they can find a banner or something alluding to that piece of link bait (the /fashion/magazine/ url). Perhaps similar to the Wordpress style <read more="">page break on posts?</read> Reason is, I want my "Money-Page" to get the links with "Lingerie" in it. Problem is, it'll be another click to the thing they want to find. Also, say I decide a few months later that I want to do another piece, like "lingerie in films" and do the same. My 'banner' idea would get very crowded or, it would mean that I bump the 50-years piece, which then makes for bad UX for readers clicking through looking for the old piece. Or perhaps this can all be solved with good design? Here's a WBF talking about mixing content: http://www.seomoz.org/blog/mixing-viral-content-with-business-content-whiteboard-friday Does anyone have an example of someone doing this well? PS. Was also thinking that on my "archive'' url, in between the content, I'll query for the top products that, that piece speaks to and make it possible for people to add to cart as they're reading the link bait content that is relevant to the product. Cheers Paul.
Conversion Rate Optimization | | RocketZando0 -
How much time is enough to test?
I am not an adwords expert but this is is the strategy I am following: 1. I create an adgroup for each set of keywords that will target the same ad. 2, I set the bid high to appear first. 3. After 3 to 5 days I remove the ones with low CTR over the impressions, anything below 1%. For example: for this url: http://www.theprinterdepo.com/hp-laser-4250-printer-q5400a-r I created an AD, and initially I put 10 keywords, after one week I had wasted a lot of money, after 4 days I deleted 7 keywords, and after 8 days, I left only one keyword with a very good CTR of almost 3% and conversion rate of 4%. The question is: how much is enough time to test and remove keywords? is one day of data good enough? Should I make this test with the highest bid possible? Should I keep the high bid forever?? 11 dollars for one click seems pretty expensive for one printer of 200 dollars when converted. Does position 1 guarantee more conversions? Maybe as long as it appears on the first 5 or 6 then its enough so I can decrease bid?
Conversion Rate Optimization | | levalencia10 -
Analytics (Other Than Google) & A/B Testing Tools
Hey, I have Google Analytics on my website, like most people, but I was looking at some other analytics and testing tools like heat maps, click maps, A/B, etc. I was hoping that some of you could recommend a few that you use to see where pages are going wrong, track user behaviour and to how improve conversion rate. I have had a look at a few websites like: https://www.clicktale.com/
Conversion Rate Optimization | | JonathanRolande
https://www.optimizely.com/ It would also be interesting to know how acurate these tools actually are, they seem quite legit. Cheers0 -
Shorter checkout form converts better? Or can it be harmful?
I've frequently read that the shorter the checkout form the better. My checkout form has the following fields: First name Last name Email Username Password Retype password Card number Card expiration date CVV code Billing street Billing city Billing state Billing zip Here's the thing. Since it's a web-app, I don't need the "billing address" as I'm not physically shipping them anything. Should I remove it? The no-brainer answer seems to be "yes", but I'm wondering if folks don't see any billing address fields, it may look suspicious. Conversions for having it and not don't seem to make much difference, so I suppose I'm looking for some tie-breaker opinions. The "password" and "retype password" fields could be eliminated by emailing the user's a system-generated password. But once again, could a user see this as odd or suspicious and then abandon? Even if I tell them I'll be emailing them a password? They could be sensitive thinking we'd email the wrong email address due to system error or their own typo. I could also eliminate the CVV and not validate against that. But once again, could a user seeing the CVV gone become wary? As much as I'd like to have "guest checkout' it's not feasible. The app is tied to a logged in account, which would also make eliminating the "username" impossible. Based on all the above, I could trim down the form considerably, but would I be doing more harm than good? I could A/B test it, but I don't believe I have a sufficient number of users to test against. Everything I buy, physical or online app, has an address field, so perhaps folks are accustomed to filling out this stuff and I should just keep it to align with user expectations? Thanks.
Conversion Rate Optimization | | bluekite770 -
How Do I Create a Google Analytics Dashboard for My Designer To Monitor Landing Page A/B Testing
Hello All, We recently started doing some "AdWords Experiments" using A/B testing of our search landing pages. My web design team does not have access to our AdWords account, but they do have "user" access to our Google Analytics account. What I need to figure out how to do is setup an easy dashboard (or custom report) that will show them at a quick glance how the two versions of their page are performing in terms of: Goal Completions (Conversions) where the specific page is the entrance/landing page. Bounce Rate Time spent on site where the specific page is the entrance/landing page. Pages viewed where the specific page is the entrance/landing page. Possibly a way to see the most popular page visited 'next' after starting on the specific entrance/landing page Anything else that might be useful The two URLs would be like: http://www.domain.com/search/testa/
Conversion Rate Optimization | | Robert-B
http://www.domain.com/search/testb/ Any insight about the best way to do this is greatly appreciated! Cheers!0 -
SEO For Personalized Content Pages
Hi, We are building a site that has content that personalizes based each visitor. This is being done to serve the most relevant content for conversion purposes and NOT for SEO purposes. So, there are many diferent versions of each page based on the visitor's location, device, browser, keyword-used, traffic source, etc. Obviously, we don't want this to appear like cloaking (it is genuine content personalization) and wondered how Google would view this? What is the safeguard for protecting the website from potential penalties? Any help would be greatly appreciated! Thanks in advance.
Conversion Rate Optimization | | Clickthroo0