Marketers often face the dilemma of whether to use conjoint tests, which are primary research experiments, or real in-market testing. This choice arises when introducing new products to the market, testing different pricing structures or running various promotions. To undertake these activities as efficiently as possible, how should marketers choose their advanced analytics tools?

Traditionally, the tool selected depends on the role of the person charged with doing the analysis. Because market researchers tend to work with traditional primary research survey tools, they usually select conjoint analysis as their main approach. Business intelligence analysts, by contrast, work mainly with operational analytics, so they typically choose an in-market test. Each group might not know of the other’s approach, or that both approaches can be used to solve the same business problem.

Read More

Advanced Analytics Expert Commentary

Success with advanced analytics requires both technical know-how and a thoughtful approach. In this series, Bain's experts offer practical advice on some of the most common data issues.

On the surface, the two methodologies don’t appear to have much in common. However, each can be useful in understanding consumers’ preferences for product features, sensitivity to price or willingness to pay—but in very different ways. Let’s explore how each works.

A conjoint experiment starts with primary research. The typical exercise contains 4 to 10 features (including price and promotions, if relevant), and each feature has 2 to 6 levels. Using experimental design—a powerful form of multivariate testing—you can generate hypothetical scenarios that each has 2 to 6 offers, roughly mimicking consumers’ real purchasing environment. Survey respondents choose among 12 scenarios. With that data collected, it takes about a week to construct statistical models to understand each consumer’s relative preferences on every product feature or price level.

You can also build simulators to estimate preference shares of any product combination you like, even if not tested in the survey or available in the market. Simulated preference shares often must be calibrated by product awareness or a diffusion curve, in order to reflect true market shares. Legal and financial approval for this type of survey approach is relatively easy, and the project will be straightforward to execute. In this relatively low-risk environment, you don’t have to create actual products, and tested products don’t have to be profitable. Figuring out profitability comes later, once it’s clear which products you will introduce.

An in-market test varies what a company actually sells in the market, in order to observe directly how consumers choose products. Experimental design still applies, but to a shorter list of attributes and levels. In general, you should limit the test to a simple design with 6 to 12 cells, with each cell being a real offer in the marketplace. The design consists of 2 to 6 attributes, each with 2 to 4 levels.

The company then sends paper or digital marketing materials, or in some campaigns, real products to real customers or prospects. Each consumer in a cell receives only one offer associated with that cell, whether it’s about message, product or price. After tracking their response or adoption rate, you use binary logistic-regression models to estimate the effect of each attribute and level. A simple spreadsheet simulator can forecast the potential take-up rate of any possible combination of the tested attributes and levels. With its real-market nature, you don’t need any calibration. One key consideration here is the substantial cost, often including legal and financial approval, of creating and introducing new products that will need to be profitable. Another is that most organizations find multivariate testing challenging to execute, well beyond their A/B testing capabilities. It requires targeting capabilities, the ability to build and perform quality assurance on many different marketing campaign features, and sophisticated test design software.

The table below compares these two methodologies.

So which one do we use? Conjoint tests may be the only choice when the offering features will be hard to change physically or the product is still under development or not available. On the other hand, in-market tests hold appeal when a company wants to measure the real adoption rate of certain intangible features such as messages of cash back.

Fortunately, in such circumstances, marketers don’t need to choose one method over the other. Because the two methods are so different, they actually work best when used together. You can start with a conjoint survey to understand the broad market and narrow down the range of possible products or promotions. Once that analysis produces high-potential offers, put those offers into an in-market experiment. This will confirm the true demand for and sensitivity to offer components. With a combined, sequential approach, you can test a wide variety of possibilities, minimize risks, reduce overall costs and boost adoption rates. No single approach can deliver all those benefits.

June Wu is an expert and Paul Markowitz is a principal in Bain & Company’s Advanced Analytics practice. They are based in Boston.