When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
We changed our domain, I used the move tool in Google Search Console and I am having our site redirected and go daddy, and now I spoke with someone who suggest we do a 301 redirect for all pages on our site and I’m not sure that’s the correct move.
We just changed our domain name after 15 years. when I bought the new domain name I called Go Daddy and they instructed me to contact my google G sweet admin account and change all of our emails over which I did and then I went into Shopify who is my host and changed my primary domain there and then I went back to Go Daddy and had my old website forwarded to my new site. since then there has been nothing but problems with Google. my product feed from my merchant center account has been suspended three or four times now, I tried to rename and move all of my Google accounts from my old domain to my new one, but I am not an SEO person... after making the changes I have started google chats with analytics department with the merchant center with Google as they all keep saying that it looks fine but I’m not convinced because the product feed keeps getting disapproved. So I posted an ad for help and the Guy I spoke with suggested I do a 301 redirect for every single page on my old site, But I’m concerned that might confuse things further? I’ve already started the move in Google Search console And in Shopify I added the old domain back into the domains section and am having it redirectEd that way too... I guess I’m just looking to know which way I should proceed, any and all advice is warmly welcome thank you in advance Maureen
Conversion Rate Optimization | | TooFast130 -
What happens when i import analytics conversions into adwords if I already have conversions set up?
Hey everyone, I have a question. We already have conversion tracking set up for our landing pages, however Adwords has the option to import goals from anaytics, one of which is for the landing pages. What will happen if I do? Will it erase all the conversion data I already have? Will it mess things up? Or am I not understanding this correctly... Any help would be appreciated 🙂
Conversion Rate Optimization | | jhinchcliffe0 -
Does adding a well scripted and produced video help significantly in conversions from visits to my divorce lawyer website to calls for my services?
Should the addition of well scripted and produced video to my divorce lawyer website increase my conversions and, if so, to what extent?
Conversion Rate Optimization | | hildebrandlaw0 -
Is it better to get a new site instead of maintaining a category? I'm stuck.
If you had a website "Food for Dogs" and: Food for German shepherds was improving in search traffic & conversions Snack Food for German shepherds had no change and never resulted in conversions but resulted in many leads for #1 Food for Poodles was unchanged but improving conversions Here's the kicker... Food Container for Dogs was super high traffic but with no conversions or any activity to other categories Would you redirect "food container for dogs" to a different site and delete that tab from the original site completely? It's still related but does not offer any $ value or leads to any of the other categories. Just note that we have a different interest in this category that will support the other categories at one time, but not any time soon. We also have a major group on LinkedIN that's grown to thousands of members for the "food container for dogs" category. On top of it all, the "food container for dogs" has been increasing in rank on Google for several keyphrases to positions 2-5 (nationally) Other important question: Would keeping "food container for dogs" influence the ranking of the other keywords? (by dilution of the site's PR or any other reason?)
Conversion Rate Optimization | | HMCOE0 -
Query: Analytics Goal Conversion
Please see the attached image. Can anyone please explain the attached image. Goal Overview tab says 8 Goals. but in the attached image says, 3 visitors finished, I've even set the required step as well, so it is also not possible to goal might have completed from any other page. Thanks llSj0
Conversion Rate Optimization | | Visiblics0 -
Conversion and CTR best books and tips?
So there are two things a site owner needs, 1. To rank on page one and 2. To have a high conversion and clock through rate. I'm still learning both but need more help on the second. Any tips? What are the best books to study on CTR and Conversions?
Conversion Rate Optimization | | bronxpad0 -
Any idea why my bounce rate could be so high?
Hello Everyone, I'm stumped. My primary website has a horrendous bounce rate of around 75% My domain is www.radicalmarketingsolutions.com My home page has a bounce rate of around 68%. The funny thing is, I have a page on Google Analytics basics for small business owners that is my 3rd top page for content, I have an average time on page of over 3 minutes yet my bounce rate is 90%. I have another page on how to depersonalize your google searches in Chrome and Firefox that is a top page with a time on page over 5 minutes and this page has a bounce rate of 96%! I felt that my old design was really hurting me so I changed the design in the beginning of this month and it does not look like it's helped much. I really simplified the site, created a lot more white space and added some links to my other important pages in hopes to get more click throughs. I would really love some honest "constructive" feed back on my site as to why the bounce rate could be so high. I have a ton of content on my blog and I get a lot of positive feedback. One thing I did not do was exclude my IP from Analytics so I could be influencing the numbers. I've added this filter and hoping this will help. Another issue is I do email my list with each new blog post. This could mean those folks visit the post and then leave without reading anything else on my blog. So this could be hurting me as well. Would love your suggestions. Thanks, Bill Parlaman
Conversion Rate Optimization | | wparlaman0 -
Reviews not shown on google search & seller rating extensions (adwords)
Hello, we have 2 questions, regarding reviews from rating websites. 1. Reviews on google Search We have several reviews for our website www.theprintspace.co.uk from various rating sites (e.g. qype, yelp, remotegoat, etc.). When searching for theprintspace on google, only some reviews and only some rating website are shown, most are not. For qype, for instance, we have 8 reviews, but google only indicates that we have 3. The yelp reviews (and many others) are not shown at all. According to google, it takes a maximum of 2 months for the correct number of reviews to be shown on google, but the reviews have been online for over 2 months now. Is there any way this problem can be solved, so google shows the correct number of reviews? 2. Seller rating extensions on google adwords We would like the reviews to be displayed in our google adwords as a seller rating extension. These will include the merchant star rating for advertisers that are highly rated on Google Product Search. We are setting up a google merchant account to do so. Google says: "Ratings are based on the user reviews collected by Google Product Search, which aggregates reviews from a broad base of sources across the web, including Reseller Ratings, Bizrate, ReviewCentre.com and Viewpoints." Do you know if the following rating website are in google's "base of sources" and will thus be recognised and used in google adwords as rating extensions? Yelp Qype Scoot Remotegoat viewlondon tipped Or does google only source from big e-commerce rating websites? How do we get the ratings into our adwords? Thank you for your help!
Conversion Rate Optimization | | Waplington0