Close this search box.

GUEST COMMENT A new testing method for a new era: the matched cohort analysis

Time to take ecommerce testing beyond A/B

It doesn’t matter how smart your ecommerce idea looks on paper, it won’t prove its worth until it’s let loose on the buying public. Performance analysis using A/B testing has become the standard way of separating a great idea from a dud for at least a decade.

But the A/B test is struggling to cope in an increasingly complex ecommerce environment where deep personalisation experiences and adoption-based SaaS services have become the norm. In this new environment the way people shop online is undermining the basic foundations of A/B testing.

Ten years ago, ecommerce was a simpler beast. Retailers had yet to experiment with the kinds of AI-driven personalisation that we are using now. The idea of a bespoke homepage for each individual customer, for example, was out of reach as were in-page offers and incentivisation. A/B testing was more likely to be used to determine the relatively superficial optimisation of button styles and location on page, for example. 

Crucially shoppers were also likely to use just one computer to make their purchases, now consumers might use a mobile, a tablet or multiple desktop machines in just one day.

User journeys were also far more straightforward and were most likely to start and finish on a desktop. These days the buying process goes through multiple stages ranging from discovery and validation to checkout and post purchase and each stage is just as likely to be carried out on a different device, which makes tracking individuals with cookies impossible.

Dealing with a complex retail landscape

The new, increasingly complex ecommerce landscape calls for an equally sophisticated solution which can be found in a process called matched cohort analysis (MCA). At True Fit, we use MCA because it gives a deeper and much more accurate picture of ecommerce performance and customer behaviour.

The sorry fact is that retailers and vendors who rely solely on A/B testing may be basing critical commercial decisions on significantly flawed data. To discuss the merits of MCA, we first need to take a closer look at A/B testing and then consider how MCA is different.

The limitations of A/B testing

A/B testing, also known as split testing, has stood the test of time because it is so well suited to comparing the differences between two ecommerce experiences. It works best, as the name suggests, when there are two test branches and the new idea, or ‘treatment’, is applied to one of these branches. For example, you may want to test two different content layouts for the same product or moving navigation items around to see which results in more sales. To achieve a successful A/B test, users must be allocated randomly to each of the branches and stay there for the full course of the test. There also needs to be enough users to detect a change.

This criteria works just great for the majority of ecommerce performance tests, but as complexity of ecommerce increases some serious limitations to A/B testing are emerging: 

  • It can’t be used to figure out the impact of events beyond the control of the retailer, such as tax rises and extreme weather. That’s because, during these so-called natural experiments, the tester can’t decide, for example, which user will experience a heatwave
  • A/B testing can’t be used to investigate the effect of past events such as a stock shortage or a partial website outage.

A/B tools identify browsers, not users. It’s now common for shoppers to use multiple devices, so shoppers often see multiple test branches, undermining the whole test. Raising the retail bar.

The bottom line is that A/B testing is still great, but sometimes retailers need to raise the bar, for example when it comes to adoption-based technologies, and this is where matched cohort analysis (MCA) comes in.

We use MCA at True Fit because it overcomes many of the limitations of A/B testing. For example, there are no issues with multiple devices and it doesn’t require any modifications at all to the retailer’s website. For statistical reasons, MCA can often detect an effect when an A/B test can’t, and of course, it’s much better suited to exploratory data analysis. Successful MCA testing involves three steps: 

  • The first involves grouping two sets of customers together in cohorts. This can be shoppers who decide to use a specific solution embedded in an ecommerce site and those who don’t. At True Fit, for example, we group together two cohorts; True Fit users and non-users
  • ‘Twins’ are then sought between the two cohorts. For example, they could be customers who share the same order count and order value over a historic period
  • The behaviour of the two cohorts is then tracked and measured over time. At True Fit we track orders per user, because this is easy to understand, easy to measure and highly relevant to the retailer.

Despite their limitations, A/B tests are still the go-to solution for measuring ecommerce success in most scenarios. But when ecommerce firms are faced with complex challenges, for example testing adoption-based technologies that alter consumer behaviour over time, MCA proves to be the trustworthy alternative.

For more information about matched cohort analysis, and how you can use it to improve the validation of adoption-based technologies like True Fit’s Personalization Platform, read the detailed report here.

Read More

Register for Newsletter

Group 4 Copy 3Created with Sketch.

Receive 3 newsletters per week

Group 3Created with Sketch.

Gain access to all Top500 research

Group 4Created with Sketch.

Personalise your experience on