When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Individual statistics (not aggregated) for 'local pack' and 'organic search results' when both on the same page
Hi, For a particular keyword combination we show up on the local pack but also on the organic search results on the 1st page. I have never been able to find out what the CTR, #clicks, conversions for this two different kinds of positions are individually. The aggregate figures I can find out in Google Search Console / Analytics, but I would like to know on an individual level as I am testing out different things. With the statistics on Google My Business I can't get along actually...no CTR as far as i know. Any hint? Cheers, Cesare
Conversion Rate Optimization | | Cesare.Marchetti1 -
Vat conversion rates (UK)
Does anyone have any evidence about the impact of displaying or not displaying prices including VAT on your website on conversions? A client wants to show prices excluding VAT so the prices look competitive with their competition who also display excluding VAT (20% UK sales tax). I have explained that the sudden 20% increase to the basket cost will put off a lot of customers (even if all prices are marked as excluding tax) but they will not listen. We have previously displayed both prices but this looked messy and customers got confused. What I really need is some research that shows what the impact of doing this is (ignoring the fact that it is not even allowed by HMRC). Any thoughts welcome!
Conversion Rate Optimization | | MrFrisbee0 -
Looking for UK based Conversion Rate Optimization Expert for PPC Landing Pages
Hi, Could anyone recommend a very good conversion rate optimization expert that can: -redesign existing landing pages using responsive twitter bootstrap -split test using optimizely or other tool Thanks, Dwayne
Conversion Rate Optimization | | deelo5550 -
Adwords conversion data is contradicting
I am looking at my Adwords data from a segment of last year. It shows my total for all ad groups in this campaign at 18. I only have one ad and it shows 7 conversions. Where are the other 11 coming from?
Conversion Rate Optimization | | EcommerceSite0 -
Can someone give me some good articles about conversion?
I finally bought the seo moz pro version. I consider myself pretty knowledgeable in seo, content/social media marketing, but I am a rookie at conversion optimization. HELP PLEASE! I am trying to convert traffic into phone calls and/or contact forms.
Conversion Rate Optimization | | SwanJob0 -
Need Advice on Home page Conversion Rate Improvement - not much $$s
Anyone out there have some suggestions for how I can improve my home page and increase my lousy conversion rate? I've spent years on the content and SEO optimization of my site and very little time on making a home page that sells stuff. Caveat - I don't have the money to go to one of those fancy companies who do a complete makeover and charge in the tens of thousands. Any suggestions would be greatly appreciated! Jean http://www.northwest-wine.com
Conversion Rate Optimization | | JeanYates0 -
Adwords Conversion Optimiser
OK Mozers! Here's the situation (my parents went away on a weeks vacation...) • I am managing an online furniture retail website Infurn .com • Our budget is 500 euro per day and we need basically 20% cost per sale and im hovering at 30 - 35% • I recently was given the opportunity to participate in The Conversion Optimiser which allows you to set a desired CPA and Google works its magic geting you that target by adjusting your bids accordingly using historical campaign data. • Is this something I should use? Or should I continue to hone my ad groups, bids, products, landing pages, and ad copy until I reach my 15 - 20% CPA goal? Thanks!
Conversion Rate Optimization | | TheGrid0 -
Conversion Rate - site feedback
Hi We gave a website a bit of a facelift last nov with the aim of increasing its position within Search Engines and getting it to begin making sales. The site: www.funkyfootstools.com is now moving in the right direction for its keyphrases, footstools and ottomans although it still needs some work. The issue I have is that from the visitors that come to the site, the stats prove they have a look around, they are just not purchasing - I am not sure if this is because of the layout, ie. usability or if there is something else. Any ideas & feedback would be greatly appreciated. Thanks Carl
Conversion Rate Optimization | | keane-1145170