When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google showing in snippets last individual rating instead of aggregate rating
the product pages of my ecommerce site contain schema markup.
Conversion Rate Optimization | | lcourse
according to rich snippet tool from google all looks fine and properly formatted
however in the SERP google shows the last rating in the snippet instead of the aggregate rating. any idea how to show the aggregate ratings? URL: https://www.humidordiscount.co.uk/adorini-triest-deluxe-rosewood-humidor below extract of schema code as recognized by the rich snippet tool: Product name:adorini Triest Deluxe Rosewood Humidor image:https://www.humidordiscount.co.uk/952-large_atch/adorini-triest-deluxe-rosewood-humidor.jpg description:High-quality multiple ..... brand [Organization]: name:Adorini offers [Offer]: price:141 priceCurrency:GBP availability:http://schema.org/InStock aggregateRating [AggregateRating]: worstRating:1 ratingValue:4.5 bestRating:5 ratingCount:460 review [Review]: description:It 's my first .... reviewRating [Rating]: ratingValue:4 worstRating:1 bestRating:5 author [Thing]: name:Alessandro M review [Review]: description:excellent ... reviewRating [Rating]: ratingValue:5 worstRating:1 bestRating:50 -
How to track conversions from a third party email campaign?
Heyy.. New to the Moz community, first question so here goes! I'm going to be running an email campaign with an e-magazine and I want to know how to track conversions from the email they're sending out so I can monitor success/failure. I've found Google's URL builder:
Conversion Rate Optimization | | 9868john
https://support.google.com/analytics/answer/1033867?hl=en
But this doesn't really explain what i'm trying to do. Can anyone of you in this fine community shed some light on this please?
Johnny1 -
Sudden drop in conversions!
Over the past week, I have noticed a drop in conversions. By drop, I mean NONE at all. The traffic to the site seems steady from different sources, so I don't think we've been hit with a penalty. I have looked through the site, and there are no problems with the functionality. Is it possible that you lose conversions because of the type of keywords that are driving traffic? In other words, the impressions and queries remain steady, but you are not getting traffic that converts (if that makes any sense). Thanks in advance for thoughts and advice.
Conversion Rate Optimization | | inhouseseo0 -
How to Improve Conversions?
One of our two prime goals is to increase visitor --> shopping cart --> order conversions. I would appreciate any advice on how we could improve our conversions from product pages like this one: http://www.audiobooksonline.com/Alex-Cross-Run-James-Patterson.html All our product pages are presented in the same format.
Conversion Rate Optimization | | lbohen0 -
Adwords and Seller Ratings
We are in the process of having an agency take over an adwords account. Will we lose our seller ratings? The website we are advertising will not change, just the adwords account. Based on what I've read google pulls data from display URL. We were concerned with losing them as we've had a CTR% since they started showing up in our search ads. They will NOT be using our account, but they will be adding us to their MCC & granting us access. Thank You!!
Conversion Rate Optimization | | TP_Marketing0 -
Is it better to get a new site instead of maintaining a category? I'm stuck.
If you had a website "Food for Dogs" and: Food for German shepherds was improving in search traffic & conversions Snack Food for German shepherds had no change and never resulted in conversions but resulted in many leads for #1 Food for Poodles was unchanged but improving conversions Here's the kicker... Food Container for Dogs was super high traffic but with no conversions or any activity to other categories Would you redirect "food container for dogs" to a different site and delete that tab from the original site completely? It's still related but does not offer any $ value or leads to any of the other categories. Just note that we have a different interest in this category that will support the other categories at one time, but not any time soon. We also have a major group on LinkedIN that's grown to thousands of members for the "food container for dogs" category. On top of it all, the "food container for dogs" has been increasing in rank on Google for several keyphrases to positions 2-5 (nationally) Other important question: Would keeping "food container for dogs" influence the ranking of the other keywords? (by dilution of the site's PR or any other reason?)
Conversion Rate Optimization | | HMCOE0 -
Improving Bounce rate, Time on Site & Pageviews per visit
Based on what seems like a general agreement that the Panda update has modified Google's algorithm and on page metrics such as Bounce rate, Time on Site & Page views per visit are now vital to a websites rankings. What methods are you using to improve these metrics? Have they worked?
Conversion Rate Optimization | | donthe0
