It doesn’t matter how smart your ecommerce idea looks on paper, it won’t prove its worth until it’s let loose on the buying public. Performance analysis using A/B testing has become the standard way of separating a great idea from a dud for at least a decade.
But the A/B test is struggling to cope in an increasingly complex ecommerce environment where deep personalisation experiences and adoption-based SaaS services have become the norm. In this new environment the way people shop online is undermining the basic foundations of A/B testing.
Ten years ago, ecommerce was a simpler beast. Retailers had yet to experiment with the kinds of AI-driven personalisation that we are using now. The idea of a bespoke homepage for each individual customer, for example, was out of reach as were in-page offers and incentivisation. A/B testing was more likely to be used to determine the relatively superficial optimisation of button styles and location on page, for example.
Crucially shoppers were also likely to use just one computer to make their purchases, now consumers might use a mobile, a tablet or multiple desktop machines in just one day.
User journeys were also far more straightforward and were most likely to start and finish on a desktop. These days the buying process goes through multiple stages ranging from discovery and validation to checkout and post purchase and each stage is just as likely to be carried out on a different device, which makes tracking individuals with cookies impossible.
The new, increasingly complex ecommerce landscape calls for an equally sophisticated solution which can be found in a process called matched cohort analysis (MCA). At True Fit, we use MCA because it gives a deeper and much more accurate picture of ecommerce performance and customer behaviour.
The sorry fact is that retailers and vendors who rely solely on A/B testing may be basing critical commercial decisions on significantly flawed data. To discuss the merits of MCA, we first need to take a closer look at A/B testing and then consider how MCA is different.
A/B testing, also known as split testing, has stood the test of time because it is so well suited to comparing the differences between two ecommerce experiences. It works best, as the name suggests, when there are two test branches and the new idea, or ‘treatment’, is applied to one of these branches. For example, you may want to test two different content layouts for the same product or moving navigation items around to see which results in more sales. To achieve a successful A/B test, users must be allocated randomly to each of the branches and stay there for the full course of the test. There also needs to be enough users to detect a change.
This criteria works just great for the majority of ecommerce performance tests, but as complexity of ecommerce increases some serious limitations to A/B testing are emerging:
A/B tools identify browsers, not users. It’s now common for shoppers to use multiple devices, so shoppers often see multiple test branches, undermining the whole test. Raising the retail bar.
The bottom line is that A/B testing is still great, but sometimes retailers need to raise the bar, for example when it comes to adoption-based technologies, and this is where matched cohort analysis (MCA) comes in.
We use MCA at True Fit because it overcomes many of the limitations of A/B testing. For example, there are no issues with multiple devices and it doesn’t require any modifications at all to the retailer’s website. For statistical reasons, MCA can often detect an effect when an A/B test can’t, and of course, it’s much better suited to exploratory data analysis. Successful MCA testing involves three steps:
Despite their limitations, A/B tests are still the go-to solution for measuring ecommerce success in most scenarios. But when ecommerce firms are faced with complex challenges, for example testing adoption-based technologies that alter consumer behaviour over time, MCA proves to be the trustworthy alternative.
For more information about matched cohort analysis, and how you can use it to improve the validation of adoption-based technologies like True Fit’s Personalization Platform, read the detailed report here.