When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Is being in position 1-3 without a featured snippet better for conversions?
I'm a cosmetic dental practice so we're local and want people to visit our website, look around and make a booking. I've followed very closely the featured snippet debate on WBF and in the blogs here and everywhere else on the web. I've got many 1-3 positions and have been optimising for featured snippets with great success. I have noticed though that once I get a featured snippet the CTR of the page that the snippet is taken from drops significantly. This can often put me from position one into two or three. The same thing happens sometimes on the local listings. You get in the pack and people start visiting your homepage rather than the particular services page in the SERP. At a conference I was at, here https://www.youtube.com/watch?v=1iXlVcbpvd0 (11:14) there is a suggestion that Vodafone de-optimise because a position one or two is better for their sales than a featured snippet. After all my interests are not the same as google. They want to answer questions and I want to sell cosmetic dentistry. So is answering a question without bringing a user to my site a good idea? I've certainly seen the amount of enquiries drop once the category has a number of featured snippets. Are they as good as everyone says they are? I think possibly not from my data and this Vodafone talk is the first time i've heard anyone agree. I've scoured the web for answers and it's all just "GET FEATURED SNIPPETS", "HERES HOW TO GET THEM" etc. But my business is not about getting more traffic it's about selling. Could someone please point me to research on conversion rate and featured snippets. or ROI and featured snippets? Thanks.
Conversion Rate Optimization | | Smileworks_Liverpool0 -
Experiences Tracking Phone Call Conversions?
Tracking conversions via phone call can be challenging. The Adwords phone ads only register a conversion if someone clicks on the phone number from their mobile phone. In many cases a landing page on a website has a lead/contact form AND a phone number displayed. The user calling that number would not be registered as a conversion unless you setup multiple virtual forwarding phone numbers for prominent keywords, ad groups and/or campaigns. This can be costly for smaller advertisers. Has anyone done or know of any empirical data on the average percentage of leads generated via phone call versus online form on a landing page. I know it can vary from business type to business type (and the form layout itself will have a modest effect.) More specifically I'm interested in the professional business services industry such as Engineering, Surveying, Commercial Real Estate, Accounting, etc. So for example if you conversion rate for online form submission on a landing page is 3%, and your phone number is prominently displayed, what would be a realistic ballpark estimate for the actual conversion rate including phone calls? 25% more? 50% more? Double? "Ballpark" is the keyword here. Just seeing if anyone has measured this and what their results were.
Conversion Rate Optimization | | JCCMoz1 -
Bounce rate vs main domain and subdomain
So there is a website www.domainname.com that is based on wordpress. Basically this site has 8-10 pages but it is not a blog. And it has a subdomain mystore.domainname.com which is based on magento. While developing the site the circumstances were such that we had to use a subdomain for ecommerce based on magento. So the wp site is ranking and people are landing on that site. It makes sense when it has a bounce rate of around 72% because people are actually going to the subdomain ( store ) after they land on the wp site. Because they are actually looking to buy. My question is will this bounce rate affect in ranking of the wp site ? Because for now only the wp site is ranking for most of the terms. Should i consider removing the wp site and have the whole site based on magento ? I appreciate any kind of feedback and suggestion 🙂
Conversion Rate Optimization | | MindlessWizard0 -
What is the Average Pogo Rate?
I've created a custom Google Segment which you can find here. Segment Settings: Organic Traffic Sessions with 1 visit. Sessions with less than 15 seconds time on site. To calculate Pogo rate then I looked at Pogo vs. Total organic and got the percentage of Pogo sessions. I tested a group of 20 sites and got 53% on average with a high of 87% and a low of 33%. What do you get? Any interesting insights?
Conversion Rate Optimization | | eyeflow0 -
Large changes in conversions with consistent traffic
Hi Everyone, I am looking for some help, I have just taken over a campaign for a client and we are seeing huge changes in conversion rates overnight. The client is selling a health product and is consistently getting around 1-2k visitors per day. We are getting about 6% of the traffic to the buy page consistently. However, we have found that one day the order process may convert 1/20 of these visitors and another day it could be 1/80. We are only talking about 5-20 sales per day, but I am really confused why one day it can convert so well and another so poorly. The traffic being sent is the same and all other factors are the same (no split testing). There seems to be no logic, the day of the week does not seem to impact, it almost seems random. If anybody has any recommended places to look that would be very helpful as at the moment I am at a loss. Example: Sunday 15th Visitors: 1,279
Conversion Rate Optimization | | Twist360
Order Page: 240
Sales: 3
Declines: 3 Monday 16th Visitors: 1,407
Order Page: 323
Sales: 17
Declines: 51 -
How ppc affects organic and direct convesions and traffic.
Hi, I was wondering what percentage of organic conversions is related to ppc previous traffic. I mean, when we evaluate the results of a campaign, we take a look at the cost per conversion in analytics for one specific word. Lets say that we know that for PPC campaigns we have the following data Adwords / PPC Visits: 4500 %conversions rate = 1,15% Cost per conversion = $10 ORGANIC Visits: 1300 % conversions rate = 2,10% DIRECT Visitis: 1000 %conversions rate= 2,54% It is obvious that if I cancel the adwords campaign part of the organic and direct traffic will drop too. So, I was wondering if have you ever done this calculation and if you know what percentage of the organic a direct traffic is related with de ppc campaigns. I mean... Let's say... "20% of the ppc traffic will come back to the website as organic traffic or direct traffic", or something like that... Anyone??? Thanks 😄
Conversion Rate Optimization | | teconsite0 -
Google Product Ads - Improving our Cost Per Conversion?
Hi all, We have been running Google Product Ads (paid for) ever since it began in the UK. We are still struggling to get to grips with the best way to make it worthwhile for us. Our data for the last 14 days:
Conversion Rate Optimization | | complete_outdoors
Clicks: 1827
Impressions: 65789
CTR: 2.78%
Average CPC: £0.22
Cost Per Conversion: £15.13 We've added some auto targets for our most popular brands, with a higher CPC than "all products". We've also added negative keywords so we only come up with relevant searches. Basically, what we'd like to know is: how we can best set-up our product ads to reduce our Cost Per Conversion (I guess everyone would like to do that!), as it is barely worth us using it currently. If you require any further data to help answer my question, please ask. Any help would be much appreciated. Thanks.0 -
Conversion Software
Can someone please suggest a conversion software other than omniture that I can used to track the amount of revenue a specific keyword is bringing into my company? Is there a way that I can do this using Google Analytics? I know in omniture there is a feature that shows you what the specific keyword is bringing in x amount of dollars. Can you guys recommend any other software that can do this?
Conversion Rate Optimization | | TSpike10