When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How does Shopify attribute conversions?
I'm trying to resolve a number of conversion discrepancies found in Google Analytics. I discovered that Shopify also provides its own 'conversion details'. In some cases they correlate, which is awesome! But in many cases they do not. Anyone know why this is? And which is more reliable (remembering that Shopify is an actual, real order)? Does anybody know how Shopify attributes conversions (ie. last click, first click, or some hybrid of the two)?
Conversion Rate Optimization | | muzzmoz0 -
B2B average conversion rates?
Hi guys, I've been struggling trying to find average conversion rates for qualified traffic (e.g. PPC) for lead generation/phone calls in the B2B services(non- SaaS) environment. I know that it varies depending on the industry, but having at least an idea would help us a lot. Any information you can provide would be very valuable. Thanks a lot.
Conversion Rate Optimization | | anagentile0 -
Tracking Facebook Mobile Visits to Desktop Conversions?
Hi. If I set an ad in Facebook, I can obviously track visits, conversions and so on from that ad and see if they where mobile or desktop. However, how do I, if possible track a mobile visitor who the converts on a desktop? Eg. I go on XXX Facebook Ad on my mobile whilst in work, look around. Like what I see. Then I go home from work, open my laptopn and go on XXX website and send an email enquiry form in. Is this possible to track? Thanks
Conversion Rate Optimization | | YNWA0 -
Facebook Connect login preventing traffic source conversion data. Any solutions?
We're buying traffic to the website via Adwords, and when they hit the website and go to sign up for an application (enter into our conversion funnel), the session concludes as a bounced session, a new session starts and that session now has a referral from Facebook rather than the original referring source. This means we can't monitor adwords conversions properly (or any other traffic source conversions) this seems to be quite a common problem with Facebook/Google account logins, in the case of Google, people block account.google.com but we can't afford to block Facebook outright as we're wanting to see other Facebook referrals. Has anyone come across this problem? Any suggestions of how to fix this?
Conversion Rate Optimization | | Startupfactory0 -
Large changes in conversions with consistent traffic
Hi Everyone, I am looking for some help, I have just taken over a campaign for a client and we are seeing huge changes in conversion rates overnight. The client is selling a health product and is consistently getting around 1-2k visitors per day. We are getting about 6% of the traffic to the buy page consistently. However, we have found that one day the order process may convert 1/20 of these visitors and another day it could be 1/80. We are only talking about 5-20 sales per day, but I am really confused why one day it can convert so well and another so poorly. The traffic being sent is the same and all other factors are the same (no split testing). There seems to be no logic, the day of the week does not seem to impact, it almost seems random. If anybody has any recommended places to look that would be very helpful as at the moment I am at a loss. Example: Sunday 15th Visitors: 1,279
Conversion Rate Optimization | | Twist360
Order Page: 240
Sales: 3
Declines: 3 Monday 16th Visitors: 1,407
Order Page: 323
Sales: 17
Declines: 51 -
Google Analytics: Do 'Goals' actually work?
Hi, Hoping someone can clarify this for me. I've set a goal up whereby the rule is to pick up all pages with "Thank-you" in the destination URL (regular expression). So, when someone fills out a form, it will count as a lead. I'm about 8 weeks in to the campaign now and measuring the last 4 weeks I've had a total of 30 goals completed. I have two issues with this and I really hope someone can help clarify: Using a custom report I can see that I've actually had 52 Visits to the Thank-You pages, 49 Unique Visits and 60 pageviews. Where is Google getting the goals from? I have 51 leads in my inbox from the campaign so far. So, much closer to the above stats in the custom report than in the Goal report. Why is this? Thanking you in advance!
Conversion Rate Optimization | | IndustrySteve0 -
Need Advice on Home page Conversion Rate Improvement - not much $$s
Anyone out there have some suggestions for how I can improve my home page and increase my lousy conversion rate? I've spent years on the content and SEO optimization of my site and very little time on making a home page that sells stuff. Caveat - I don't have the money to go to one of those fancy companies who do a complete makeover and charge in the tens of thousands. Any suggestions would be greatly appreciated! Jean http://www.northwest-wine.com
Conversion Rate Optimization | | JeanYates0 -
Adwords Conversion Optimiser
OK Mozers! Here's the situation (my parents went away on a weeks vacation...) • I am managing an online furniture retail website Infurn .com • Our budget is 500 euro per day and we need basically 20% cost per sale and im hovering at 30 - 35% • I recently was given the opportunity to participate in The Conversion Optimiser which allows you to set a desired CPA and Google works its magic geting you that target by adjusting your bids accordingly using historical campaign data. • Is this something I should use? Or should I continue to hone my ad groups, bids, products, landing pages, and ad copy until I reach my 15 - 20% CPA goal? Thanks!
Conversion Rate Optimization | | TheGrid0