Bain uses cookies to improve functionality and performance of this site. More information can be found in our Privacy Policy. By continuing to browse this site, you consent to the use of cookies.

Expert Commentary

A/B and Multivariate In-Market Testing: What's the Difference?

Marketers can learn a great deal about which messages, promotions or ads are effective through in-market testing.

  • July 13, 2016
  • min read

Article

A/B and Multivariate In-Market Testing: What's the Difference?

In marketing, the best way to learn what works and what doesn’t work is with in-market testing. Consumers or business customers receive and react to stimuli through the normal course of their activities. In most cases, recipients aren’t even aware that a test is going on. They just see an email, direct mail offer, a website, a price, a display or an ad.

But as marketers, we can learn a great deal about which messages, prices, promotions or ads are effective by observing the results of these tests. Conducting the tests with companies in such industries as telecommunications, retail, financial services and consumer products, we have covered a range of situations:

  • Pricing
  • Promotion (coupons, rebates, buy-one-get-one)
  • Marketing mix
  • Ad copy
  • Direct mail
  • Email
  • Banner ads
  • Search marketing
  • Website design

The most popular method for these in-market tests is the A/B test, which uses one test cell and one control cell. The winner is the one that performs better in market. You can easily administer A/B tests and run many at the same time. For example, Progressive Insurance’s research conducts thousands of A/B tests each year. With that platform in place, Progressive can quickly gather data-driven insight into many business questions that arise.

An alternative method is a multivariate test, designed to systematically test several attributes or variables at once. Several test cells contain combinations of the attributes. By running a statistical model at the end, you can predict what would have happened for any combination of attributes, whether they were in a test cell or not. Here is a comparison of the two methods:


ab-multivariate-test-table-01_embed

Note that test design can be complex. You have to make trade-offs among sample size, sample frame, the overall number of cells, number of attributes or levels, and restrictions. Another key issue, not always apparent, is the interactions among attributes, which you must plan for in advance and account for in the model.

We have conducted dozens of these tests. Occasionally, when we need a quick win, we will use the A/B test. Most often, though, we recommend multivariate tests. We typically want to test multiple attributes at one time, assign success to each of them and optimize for the best combination. The multivariate test gives us the flexibility we need to test several attributes simultaneously at the lowest cost, with the smallest sample. With a well-designed in-market test, we have seen companies achieve remarkable results (see the Bain Brief “Go Big and Get Smart: Boosting Your Digital Marketing Effectiveness” for more detail).

Paul Markowitz is a principal in Bain & Company’s Advanced Analytics practice. He is based in Boston.

Tags

Want to continue the conversation?

We help global leaders with their organization's most critical issues and opportunities. Together, we create enduring change and results.