When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Experiences Tracking Phone Call Conversions?
Tracking conversions via phone call can be challenging. The Adwords phone ads only register a conversion if someone clicks on the phone number from their mobile phone. In many cases a landing page on a website has a lead/contact form AND a phone number displayed. The user calling that number would not be registered as a conversion unless you setup multiple virtual forwarding phone numbers for prominent keywords, ad groups and/or campaigns. This can be costly for smaller advertisers. Has anyone done or know of any empirical data on the average percentage of leads generated via phone call versus online form on a landing page. I know it can vary from business type to business type (and the form layout itself will have a modest effect.) More specifically I'm interested in the professional business services industry such as Engineering, Surveying, Commercial Real Estate, Accounting, etc. So for example if you conversion rate for online form submission on a landing page is 3%, and your phone number is prominently displayed, what would be a realistic ballpark estimate for the actual conversion rate including phone calls? 25% more? 50% more? Double? "Ballpark" is the keyword here. Just seeing if anyone has measured this and what their results were.
Conversion Rate Optimization | | JCCMoz1 -
Suggestions to Improve Worst Conversion Form Ever Designed?
My commercial real estate web site has about the worst conversion form ever designed (http://www.nyc-officespace-leader.com/visitor-details). The form appears whenever a visitor does a property search. If they do not release their phone number and email address they don't get results. The form only gets completed 25% of the time and often the contact details are wrong. The visitors either exit the site at that page or go to the listing or neighborhood pages to view listings without doing a filtered search. In an attempt to increase conversions we are softening the form and not making completion mandatory. After entering an inquiry the visitor will be shown three property listings before the form pops up, The idea being that once they have seen product they will be more inclined to release contact info. Also, the form will appear once they have viewed three listings anywhere on the site, not just for search results. Unlike the existing conversion form (http://www.nyc-officespace-leader.com/visitor-details) we are trying to keep the text very concise. The new form will be Java script rather than a URL. Can anyone offer some ideas on how to make the text of these new forms more enticing? The forms are listed below. We will probably show the first one after the visitor views 3 listings and the variation of it after the visitor views another 3 listings. ** FORM #1** Quit wasting time searching for listings. (watch icon) Call me to discuss available space. Yes No --------------------------------------------------------------------------------------------------------- FORM #2 Don’t search for listings, we will do it for you. We have off market listings that no one knows about. Contact us now about off market listings. Yes No ----------------------------------------------------------------------------------------------------------- Thanks, Alan
Conversion Rate Optimization | | Kingalan11 -
Brand Name: Conversion Rate Optimizing: Eponymous VS Generic
Good day all, We are entertaining the idea of changing the Domain Name & Brand Name for straplessfashion.com. While still in development, the website sells women undergarments. We did some survey research and it seems that women (age 18-40) really liked either "Demi's Dream" or "Almost Skin" as a good name. More specifically, Almost Skin seems to be the more popular choice over "Demi's Dream". I think "Almost Skin" catches attention quicker and may work best in PPC. However, when personalizing a brand name is considered, "Demi's Dream" might be the best choice. Would you please offer your advise on which of these two names might be the best for a new Domain and Brand name for the website mentioned above? Which would convert better? Thank you for your help. Carlos
Conversion Rate Optimization | | 90miLLA0 -
How Would You Market A Photo Of Jay-Z Wearing Your Client's Apparel?
If Jay-Z was spotted wearing your client's sweatshirt, how would you maximize the marketing potential of this photo and opportunity to sell more sweatshirts?
Conversion Rate Optimization | | farmiloe0 -
Adwords conversion data is contradicting
I am looking at my Adwords data from a segment of last year. It shows my total for all ad groups in this campaign at 18. I only have one ad and it shows 7 conversions. Where are the other 11 coming from?
Conversion Rate Optimization | | EcommerceSite0 -
Has anyone heard about Facebook's Revenue Problems? Duh right. Well have you seen how they are losing money yet?!?!
So about a year ago now (maybe more/less), Facebook changed the way their ads are displayed. On the desktop version of Facebook (for me at least), the Sponsored ads in the right sidebar display anywhere between 3 and 7 ads at a time. Well in the case of 7 ads, that is a lot of ads to look at and read through. All seems pretty ok right?? Well, here is the problem. I cannot tell you how many times I have seen an ad that interests me, and then read through the others below them, then before I can go back to click on the one that interested me, the ads automatically refresh, and I don't get to click on the ad. Maybe they figure that by refreshing ads, it gets them more chances to catch your interest, and that the value of that is higher than the value of me getting to click what I wanted. (It kinda upset me honestly. I thumbs down things that tease me.) Either way, they could have both. My solution? A small button that you can click to view the ads that were previously displayed. (imagine how much money that could bring them. Especially if I saw the ad yesterday, left the window open for 24 hours with the computer on, and just sat there clicking until i found it. You are BOUND to find MORE ads that interest you, and click on those too.) ((ok, ok, ok, maybe 24 hours is a little ridiculous, but at least SOME amount of time and a button is necessary)) If we were really going to build a marketing playground, then we could also throw in a "More Like This" button. It would open the doors to some AMAZING data. Full to the brim with advertisement related feedback coming happily from your users. When you are talking about positioning your brand, it doesn't get much more important than to talk to them while they are enjoying themselves. (i.e. they click button, and next page has 10 links. you show the links, based on how many times people are clicking them. Not to mention, when you are that big, you can look at the value of what they do immediately after, the next day, and even the next week, and really tell what is the most valueable add, in terms of egagement, and further CTRs and Conversions for later in your relationship ) Anyways, I thought everyone would find that interesting, and hopefully it would inspire some people to get analytical with their thoughts today. Do you think that Facebook could use some conversion rate optimization? Do you think that Facebook could be doing more than they are to make more money? Do you think that Facebook (being such a powerful site), could make a lot of money by implementing an SEO/Content strategy?
Conversion Rate Optimization | | TylerAbernethy0 -
Query: Analytics Goal Conversion
Please see the attached image. Can anyone please explain the attached image. Goal Overview tab says 8 Goals. but in the attached image says, 3 visitors finished, I've even set the required step as well, so it is also not possible to goal might have completed from any other page. Thanks llSj0
Conversion Rate Optimization | | Visiblics0 -
Conversion optimization with Optimizely, impact on SEO
I am looking into signing up with www.optimizely.com for conversion testing. They put scripting on my site which will then redirect half of my site visitors to an alternate home page. The site I want to test on is ranking quite well with Google and I do not want to hurt my rankings. And with this set up,what Google will think is my home page is not and so I am essentially sending visitors to a different page than Google reads as my home page in source code. So, my concern is whether this will have a negative impact on my SEO rankings to redirect 50% of site visitors to a different page using this testing tool? I would use Google Web Site Optimizer, but many of my sites are in Wordpress and it seems that Wordpress and web site optimizer are not so compatible. Advice would be appreciated. Thank you, Robert
Conversion Rate Optimization | | Robertnweil11