When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Should we change branded keywords based on audience knowledge?
Good Morning Moz, Wanted to pose a question to the community that our company has been wrestling with internally for a while now. I think your insights would be tremendously valuable. Our company effectively has two business units, one that is online and one that is offline. Each appeals to a different customer base as well. In one business unit, the larger, more offline and better established of the two, we refer to ourselves as a "manufacturer of high end automobiles" (not true, but using as an example). We've carried this through to the other business unit which is very small, web-based, and growing, but have quickly realized that the majority of would-be customers do not search for "automobiles", they search for "cars." And the keyword volume is night and day. On the order of 5000 searches/mo for automobiles, and 1,000,000 searches/mo for cars (again, not true for these example keywords but this is the actual ratio against our real keywords). So, do we optimize heavily against a lesser known and used keyword, or do we try to target "cars" which sees higher search volume even though it's significantly more competitive? We have this problem with a few different branded keywords, actually, and our search performance reflects as much. If someone were to search for these lesser known keywords they would find us, and we rank well. But the volume isn't there. Thoughts? Thanks in advance.
Conversion Rate Optimization | | J-Me0 -
Suggestions to Improve Worst Conversion Form Ever Designed?
My commercial real estate web site has about the worst conversion form ever designed (http://www.nyc-officespace-leader.com/visitor-details). The form appears whenever a visitor does a property search. If they do not release their phone number and email address they don't get results. The form only gets completed 25% of the time and often the contact details are wrong. The visitors either exit the site at that page or go to the listing or neighborhood pages to view listings without doing a filtered search. In an attempt to increase conversions we are softening the form and not making completion mandatory. After entering an inquiry the visitor will be shown three property listings before the form pops up, The idea being that once they have seen product they will be more inclined to release contact info. Also, the form will appear once they have viewed three listings anywhere on the site, not just for search results. Unlike the existing conversion form (http://www.nyc-officespace-leader.com/visitor-details) we are trying to keep the text very concise. The new form will be Java script rather than a URL. Can anyone offer some ideas on how to make the text of these new forms more enticing? The forms are listed below. We will probably show the first one after the visitor views 3 listings and the variation of it after the visitor views another 3 listings. ** FORM #1** Quit wasting time searching for listings. (watch icon) Call me to discuss available space. Yes No --------------------------------------------------------------------------------------------------------- FORM #2 Don’t search for listings, we will do it for you. We have off market listings that no one knows about. Contact us now about off market listings. Yes No ----------------------------------------------------------------------------------------------------------- Thanks, Alan
Conversion Rate Optimization | | Kingalan11 -
Google Product Ads - Improving our Cost Per Conversion?
Hi all, We have been running Google Product Ads (paid for) ever since it began in the UK. We are still struggling to get to grips with the best way to make it worthwhile for us. Our data for the last 14 days:
Conversion Rate Optimization | | complete_outdoors
Clicks: 1827
Impressions: 65789
CTR: 2.78%
Average CPC: £0.22
Cost Per Conversion: £15.13 We've added some auto targets for our most popular brands, with a higher CPC than "all products". We've also added negative keywords so we only come up with relevant searches. Basically, what we'd like to know is: how we can best set-up our product ads to reduce our Cost Per Conversion (I guess everyone would like to do that!), as it is barely worth us using it currently. If you require any further data to help answer my question, please ask. Any help would be much appreciated. Thanks.0 -
Conversion Report in Adwords
I would like to create a report that shows Campaign-Ad Group-Ad-Keyword but only for items that have conversions. Anyone know a way to do this? Thanks
Conversion Rate Optimization | | EcommerceSite0 -
Conversion Rate Optimization (CRO): Best Approach/Plan Documentation
Hey Mozzers - First off, I am well aware of what CRO is, its benefits, and why it is essential for any inbound or performance marketing campaigns. However i am a total newbie at the actual planning and implementation of CRO, so any advice/feedback on the questions below are much appreciated! When planning a CRO experiment/test what is the best way to document it? Should i develop the full plan ahead of time laying out each element? If so, what elements should i include in this plan? What are the KPIs i should report on? What tools should i use for reporting (i was thinking a blend of Google and Clicktale)? Any lessons learned through out your campaigns? Thank you all so much for your input! Regards - Kyle
Conversion Rate Optimization | | kchandler0 -
Conversion and CTR best books and tips?
So there are two things a site owner needs, 1. To rank on page one and 2. To have a high conversion and clock through rate. I'm still learning both but need more help on the second. Any tips? What are the best books to study on CTR and Conversions?
Conversion Rate Optimization | | bronxpad0 -
Conversion Software
Can someone please suggest a conversion software other than omniture that I can used to track the amount of revenue a specific keyword is bringing into my company? Is there a way that I can do this using Google Analytics? I know in omniture there is a feature that shows you what the specific keyword is bringing in x amount of dollars. Can you guys recommend any other software that can do this?
Conversion Rate Optimization | | TSpike10 -
Landing Page Conversion Rates
I had a client claiming his industry (replacement home windows) conversion rate was around 15% based on his research. In my research i could find nothing of the sort, and really couldn't find a single place that provided this type of information. Anyone out there have some advice on where to research industry conversion rates (averages) or tools/thought processes to explore? Thanks! (these stats would primarily be used for goal setting and competitive analysis)
Conversion Rate Optimization | | kchandler0