When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Weighted number of conversions - any explanation?
Do any of you know meaning of "weighted number of conversions" in attribution models? How google analytics calculate it? Any example? Thank you.
Conversion Rate Optimization | | Tormar0 -
Track bounce rate per PPC keyword
Hi all Wondering if anyone could give me a nudge in the right direction with this please. When running an adwords campaign is there a way in which one can track the bounce rate on the landing page per keyword sending paid traffic? I've had a play about in adwords and GA and couldn't find the right option. The site I am sending traffic to is not a ecommerce site so cannot set up conversion tracking so we are going to monitor the bounce rates of different keywords and drop the ones which are bouncing much higher than others as it seems better to invest the budget in those words which are keeping people on the page for longer. Many thanks Carl
Conversion Rate Optimization | | carl_daedricdigital0 -
Any tips and tricks to increase Conversion Rates on Google PLA?
I have recently organized my Adwords (Product Listing Ads) campaign according to this link: http://certifiedknowledge.org/blog/how-to-create-a-state-of-the-art-pla-campaign While I like the organization and control this provides, my fundamental concern is to increase conversions. I understand conversions are dependent on other factors such as landing pages etc. Are there any Google Adwords tips someone has had success with to increase CTR and increase conversion rates? Pating $10/day to Google Adwords while only getting a sporadic business every 3 days is getting a bit frustrating. Are there any tools in MOZ that can help find faults and generate simple reports on the Google Adwords campaign? Any help is much appreciated.
Conversion Rate Optimization | | amitramani1 -
What's the Best Method of Content Distribution
This question is both specific and open ended. I have a new web project I've been working on content for. I paid to have a really sweet research article written and I am in the process of turning it into an info-graphic as well. Being that the the website is so new it doesn't have many customers and few to no blog readership. Now that I will soon have great content for the site should I : A. Post on the Blog ? ( Its a Blogger on a blog. domain.com ) B. Post as a static page on the website ? (domain.com/content) C. Something Else ? Whats the easiest to promote and push traffic to?
Conversion Rate Optimization | | CDUBP
Which will rank best?
Which will rank longer?
Which will rank faster? Any other suggestions to get the content in front of the niche market?0 -
Need Advice on Home page Conversion Rate Improvement - not much $$s
Anyone out there have some suggestions for how I can improve my home page and increase my lousy conversion rate? I've spent years on the content and SEO optimization of my site and very little time on making a home page that sells stuff. Caveat - I don't have the money to go to one of those fancy companies who do a complete makeover and charge in the tens of thousands. Any suggestions would be greatly appreciated! Jean http://www.northwest-wine.com
Conversion Rate Optimization | | JeanYates0 -
Clicks versus Conversions
Have had a significant amount of success getting clicks to my site through a number of online marketing methods. Still falling a bit short on conversions. Any thoughts on the types of things that are successful to get people to contact you?
Conversion Rate Optimization | | casper4340 -
61.45% Shopping Cart Abandon Rate - How to troubleshoot?
Hello MozPeeps! I wanted to see if I could get some help on the best method for troubleshooting our abandoned shopping cart funnel. Maybe point me in the best areas to research, or what are the best tools to use, or an improved method for analyzing and interpreting the existing data for more clarity about what is happening. I have attached the screen shot that shows what's going on. This account is using a Magento shopping cart with a custom built checkout page that is trying to copy Overstock.com's checkout page. Here are my thoughts: 1. Use Clicktale to monitor the checkout pages in question to try and get a feel for what people are doing or why they are clicking off? 2. Use another tool (not sure which one) that allows us to monitor page load speeds for the checkout process to see if orders are being hung up by slow loading pages. 3. Or, maybe 62% is not an uncommon abandoned cart rate and therefore we should just focus on automating abandoned cart follow up emails for the customers. Thanks in advance for any help! funnel-goal-visualization.jpg
Conversion Rate Optimization | | JerDoggMckoy0 -
Lookiing for examples of local "small biz" sites with phone or contact form conversion points
Hi Guys, I'm looking for any good examples of small business sites that you've come across that do a good job at converting potential visitors to clients. I'm primarily thinking about service oriented business such as Doctors, Dentists, Salons, Clothing stores, small restuarants, car detailing services, local photographers, etc. I'm just trying to digest other examples and look for new inspirations.
Conversion Rate Optimization | | DotCar0
