When to determine that a change DIDN'T affect conversion rates
-
Hi everyone,
Description of test: We're a lead gen site trying to add more value by providing users with real, live quotes after they submit a lead. However, we don't want showing the quotes to tank our lead conversion rates. So we're running a test where 50% of leads see quote results and 50% don't, and we compare the lead conversion rates for each. The best possible outcome is to show that showing the quotes DIDN'T negatively affect conversion rates.
My issue: When do we conclude the test? In the end, we're hoping to see that the change didn't cause a statistically significant difference between the control and version B, which is the opposite of every other test I've ever run. So, at what point do we conclude that the changes in version B didn't have a significant effect on lead conversions? Currently the control is doing 5% better than the variation with a p-value of .379
-
I suggest using Google Analytics Experiments, with multi-armed bandit.
Doing so you can set the significance level you want and the experiment will stop when that value is reached. By default the p-value there is 95%, I usually left it set to 95%.
And I usually repeat the experiment a second time to confirm results.
-
Hi Benjamin,
Since you’re conducting this experiment with care I think you might see my answer coming. It depends on the amount of confidence you want the results to have. In the little experience I have with A/B testing this normally depends on the amount of traffic and conversions you have. If you’re really big the more important reliable data is to your company. If you’re smaller you might not have the time / traffic / budget to get the most trustworthy data so you’ll (kind of automatically) settle for less.
In my case, we based most of our decisions with early prototypes of services on data that wasn’t flawless. But at the end, it’s up to you what is and isn’t acceptable for your company.
I hope this helps, although I think you would have come to this point without my comment
Best regards,
Bob
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Better use Google Analytics? This is for Conversion Rate Optimization purposes
Hi Moz-ers, I've had much help from the Moz community and hoping you can help me out on this one too. I would like to track everyone who visits our town listing pages. We operate much like ratings and reviews websites like tripadvisor and booking.com. What is the best way to track visitors coming to our listings pages only (for example tripadvisor shows top 10 hotels in London.) I would like Google Analytics on London and all other towns including Birmingham, Southampton ect... Is it possible to set up Google analytics to track this? These are our highest volume pages so having an idea of the traffic behaviour is crucial for SEO and Conversion Rate Optimization. As usual, looking forward to the advice we receive! Thanks,
Conversion Rate Optimization | | Eric_S
Eric0 -
Individual statistics (not aggregated) for 'local pack' and 'organic search results' when both on the same page
Hi, For a particular keyword combination we show up on the local pack but also on the organic search results on the 1st page. I have never been able to find out what the CTR, #clicks, conversions for this two different kinds of positions are individually. The aggregate figures I can find out in Google Search Console / Analytics, but I would like to know on an individual level as I am testing out different things. With the statistics on Google My Business I can't get along actually...no CTR as far as i know. Any hint? Cheers, Cesare
Conversion Rate Optimization | | Cesare.Marchetti1 -
Should we change branded keywords based on audience knowledge?
Good Morning Moz, Wanted to pose a question to the community that our company has been wrestling with internally for a while now. I think your insights would be tremendously valuable. Our company effectively has two business units, one that is online and one that is offline. Each appeals to a different customer base as well. In one business unit, the larger, more offline and better established of the two, we refer to ourselves as a "manufacturer of high end automobiles" (not true, but using as an example). We've carried this through to the other business unit which is very small, web-based, and growing, but have quickly realized that the majority of would-be customers do not search for "automobiles", they search for "cars." And the keyword volume is night and day. On the order of 5000 searches/mo for automobiles, and 1,000,000 searches/mo for cars (again, not true for these example keywords but this is the actual ratio against our real keywords). So, do we optimize heavily against a lesser known and used keyword, or do we try to target "cars" which sees higher search volume even though it's significantly more competitive? We have this problem with a few different branded keywords, actually, and our search performance reflects as much. If someone were to search for these lesser known keywords they would find us, and we rank well. But the volume isn't there. Thoughts? Thanks in advance.
Conversion Rate Optimization | | J-Me0 -
Track bounce rate per PPC keyword
Hi all Wondering if anyone could give me a nudge in the right direction with this please. When running an adwords campaign is there a way in which one can track the bounce rate on the landing page per keyword sending paid traffic? I've had a play about in adwords and GA and couldn't find the right option. The site I am sending traffic to is not a ecommerce site so cannot set up conversion tracking so we are going to monitor the bounce rates of different keywords and drop the ones which are bouncing much higher than others as it seems better to invest the budget in those words which are keeping people on the page for longer. Many thanks Carl
Conversion Rate Optimization | | carl_daedricdigital0 -
Is my destination url affecting QS?
Hello, I'm running a PPC campaing targeting "Storenvy App" using the ads below
Conversion Rate Optimization | | JustinButlionand the following DESTINATION LINK: http://www.yotpo.com/landing_page?utm_source=google&utm_medium=cpc&utm_campaign=s-storenvy&redirect=http://lp.yotpo.com/storenvy-app.html As you can see there is a 301 redirect from http://www.yotpo.com/landing_page?utm_source=google&utm_medium=cpc&utm_campaign=s-storenvy TO http://lp.yotpo.com/storenvy-app.html How this will affect QS? Thanks in advance for all your answers! AdDzQi0.jpg
0 -
Create better conversion at checkout?
I'm getting extremely poor conversion rates at checkout..would like to hear some ideas to increase checkout. Last month we had 60 users who added to cart, and only 7 who converted. We offer merchant CC, PayPal, and GCO, so I don't believe payment options is the reason for the exit. Any ideas?
Conversion Rate Optimization | | William.Lau0 -
Huge Increase in Search Traffic - Expected Drop in Conversions?
We've recently implemented a successful strategy whereby we've increased non-branded organic search traffic by over 50%. We started with our niche products and secured those rankings and then expanded the program to include local results. Rev is up, trans# is up, but conversion rate dropped. Here is the question...is there an acceptable ratio between the percentage growth in visits and the percentage decrease in conversions?
Conversion Rate Optimization | | NaHoku1 -
Reviews not shown on google search & seller rating extensions (adwords)
Hello, we have 2 questions, regarding reviews from rating websites. 1. Reviews on google Search We have several reviews for our website www.theprintspace.co.uk from various rating sites (e.g. qype, yelp, remotegoat, etc.). When searching for theprintspace on google, only some reviews and only some rating website are shown, most are not. For qype, for instance, we have 8 reviews, but google only indicates that we have 3. The yelp reviews (and many others) are not shown at all. According to google, it takes a maximum of 2 months for the correct number of reviews to be shown on google, but the reviews have been online for over 2 months now. Is there any way this problem can be solved, so google shows the correct number of reviews? 2. Seller rating extensions on google adwords We would like the reviews to be displayed in our google adwords as a seller rating extension. These will include the merchant star rating for advertisers that are highly rated on Google Product Search. We are setting up a google merchant account to do so. Google says: "Ratings are based on the user reviews collected by Google Product Search, which aggregates reviews from a broad base of sources across the web, including Reseller Ratings, Bizrate, ReviewCentre.com and Viewpoints." Do you know if the following rating website are in google's "base of sources" and will thus be recognised and used in google adwords as rating extensions? Yelp Qype Scoot Remotegoat viewlondon tipped Or does google only source from big e-commerce rating websites? How do we get the ratings into our adwords? Thank you for your help!
Conversion Rate Optimization | | Waplington0