Why did the A/B test fail – a common hard-to-answer question in the world of A/B testing marketing and Conversion Rate Optimization. There are numerous pitfalls to conducting an A/B test, which will result in inaccurate data, corrupted test results and overall AB test incompleteness.
To help you recognize the reasons for a failed A/B test and therefore easily avoid them, we created this great article as a cheat sheet to keeping your A/B test failproof.
What is A/B testing?
Ler’s start from the very beginning – what is an A/B test? A/B testing, also known as split testing, is a Conversion Rate Optimization (CRO) technique for comparing two webpage versions – A and B – in order to find out which one performs better.
AB testing is commonly used along with Multivariate testing, which tests multiple elements and their combinations to discover the best match, and is described in more detail in our fresh What is Multivariate Testing? article.
To find out more on what is A/B testing, statistical significance and some of the best split testing practices, check out our excellent What is A/B Testing in Digital Marketing: CRO Best Practices. To find out why your A/B tests fail along with Ryan Reynolds, move on to the next section.
Main Reasons Why Your A/B Test Fails
Conversion Rate Optimization along with its most beloved creative testing solution – A/B testing – is one of the most effective and cost-productive options you can choose to increase sales on your website.
So here are the common reasons why A/B test fails and how to avoid them:
1. Limited Research before running an A/B test
The secret to a successful AB test lays in the research you do prior to running it. The more information you gather beforehand the easier it is to conduct an A/B test that will make a difference.
To find out what on your website really needs improvement or revisiting, and therefore should be A/B tested, make sure you’ve spent enough time exploring Google Analytics data analysis, live chat transcript analysis, user sessions video recordings, heatmaps analysis, user testing and other important quantitative and qualitative researches.
Here’s a great article on How to Run an AB Test in Google Optimize, in case you were looking for one.
2. Testing changes that are too small
Let’s be honest – to achieve quality A/B test results you need a large user group that is being tested. Testing changes that are too small – like font size or additional landing page line element – need a lot of time to reach statistical significance.
Generally, a statistical significance of a controlled experiment like A/B testing depends on two values – current conversion percentage and expected effect size. So don’t test just for the sake of it – choose AB testing variations that will make bigger conversion pages changes.
3. Stopping the A/B test too early
If you want clear, failproof A/B test results – run your A/B test at least for a 1 full week. According to VWO, the giant of Conversion Rate Optimization, an average A/B test runs over a week but no more than two months.
Noticed the dates? Never less than a week. To determine how long your A/B split testing should be running, calculate the minimal duration of the experiment based on the current conversion rate and expected effect size after an A/B test has been implemented.
Overwhelmed with details?
Contact the InsightWhale team and we’ll conduct all the tests and experiments for you 😉
4. Not using segmentation
Here’s another pitfall to watch out for when testing your website – not using segmentation when analyzing results of an A/B test. Failing to do so will result in a failed AB test that is, for instance, based on the traffic with low-quality leads or provides results applicable for certain user age group only.
To achieve a successful AB test, segment users based on a device category, traffic sources, returning users etc. so you can find a winner within particular segments of users and get quality insights for future tests.
To segment your users the right way, here’s an article on the Top A/B testing Tools for Mobile Apps that will do user segmentation for you.
5. Testing wrong elements or not important steps of the funnel.
A/B Testing is fun. Really-really fun. So the urge to test every single detail on your website is pretty overwhelming and sometimes overpowering, we know that. But hear us out.
Testing wrong elements or not important steps of the funnel will probably very much likely result in you wondering why did an A/B test fail. And the answer to that question is testing wrong elements or those that do not impact sales funnel will not improve your conversion rates.
To conduct an A/B test that is successful, define variations based on how important to users (and sales) those elements are. Here are some stats to help you prioritize elements to test on your conversion pages:
More than 1/3 of CRO specialists start by testing a call-to-action button, then in the decreasing significance order move on to testing headlines, landing page’s layout, website copy, checkout page and sign up form. By the way, here’s a great article on How to Increase Conversion Rate at the Checkout Page, you’re welcome 😉
6. Not Running a follow-up A/B test
Sometimes you gotta get more than once. That’s just the nature of the beast when it comes to the art of Conversion Rate Optimization and its creative testing solutions – multivariate and A/B testing.
The last item on the list of reasons to a failed A/B test is not running a follow-up A/B test if it is failed and you have assumptions as to why that happened. If you suspect a reason why it didn’t work out the first time and your test element is really important, there is nothing stopping you from repeating your experiment.
To fix this, pay attention to collected data or user’s feedback for a possible reason behind the AB test failure and if you find one run a follow-up test implementing the new changes.
Conclusion
Now when you hear someone in your CRO department sight out loud “Why did my A/B test fail ?!” or see any Conversion Optimization specialists lying under their desks silently asking the same question, you know what to do.
Show them our article as a reminder of what are the common reasons A/B tests fail and instructions on how to avoid them. And when you need to conduct a quality A/B testing experiment remember which amazing CRO agency you should contact 😉
