Article

Experimental design: Maximizing marketing effectiveness

Partner John Senior explains how companies are using experimental design to create more effective marketing campaigns.

By John Senior

First published on septiembre 11, 2013
min read


In this brief audio presentation, Bain partner John Senior explains how companies are using experimental design to create more effective marketing campaigns, increase conversion rates and retain high-value customers. Companies spend huge sums on direct marketing to acquire or retain customers through email, direct mail, catalogs and other tactics. Yet the return on investment (ROI) is often poor because the response rates tend to be very low—usually less than 5% and often less than 0.5%—and rates have been declining. Thanks to recent advances in statistical methods, modeling and analytics, leading marketers now have a far more powerful and sophisticated technique called experimental design. The method allows marketers to increase exponentially the variables tested in a single campaign (product offers, messages, incentives, mail formats and so on) and to test multiple offers in the market simultaneously. Marketers learn exactly which variables entice consumers to act. As a result, response rates rise dramatically, the effectiveness of future campaigns improves and overall return on spending increases. Transcript Slide 1: I’m John Senior, a partner in Bain’s Customer Strategy and Marketing practice, and I’d like to spend a few minutes talking about Bain’s point of view on improving marketing performance using experimental design. Slide 2: Experimental design is an efficient method of in-market testing that can improve the performance of direct marketing. The power of this approach is that it increases response rates and organizational learning by massively increasing the variables tested in marketing campaigns and identifying which variables actually drive its customers to act. Experimental design has been used in a variety of situations, including direct mail, email and online marketing and has now been extended to call centers and customer experience programs. However, experimental design tests can be more complex to develop and require greater organizational preparation. Slide 3: Many organizations still use traditional “champion-challenger” approaches, or A/B testing, that test one sort of marketing offers versus another. There are limitations to this approach, in that it can’t isolate which variables drive customers to act, and the rate of organizational learning is relatively slow. Slide 4: Experimental design allows you to test many variables in different offer combinations and isolate which attributes cause customers to respond. You can then combine the best-performing attributes together into the most powerful offers to launch in large-scale campaigns. Slide 5: We've applied this approach several times with a large cable operator to acquire, retain and upgrade customers with great success. We’ve seen response-rate lifts of many times the existing in-market offers, with a significant improvement in retention and a 20% increase in average revenue per user, which is a key metric in the industry. This is translated into hundreds of millions of dollars to the top and bottom line. Slide 6: In the experimentally designed acquisition campaign with the cable operator, we targeted a particular high-value customer segment that had been difficult to acquire historically. The best-performing experimentally designed offers delivered a 3x-4x response-rate lift over the existing in-market offer, which translated to several hundred thousand new subscribers. Slide 7: We were also able to identify which of the offer elements drove response rates, with some surprising findings. We found that the presentation of the incentive and the message itself were more important than the headline offers and discounts. We also found that including a free device had little impact on response rates, which was totally unexpected and saved the company a lot of money because these devices are very expensive. Slice 8: Having run many experimentally designed tests now across several industries, we've learned some important lessons: The richest offers don’t always drive the highest response rates; message, format and design can have a huge performance impact; It’s important to use a stable of offers in the market to prevent offer fatigue, which depresses response rates over time; And it’s also very important to work with a sales and service organizations to make sure they are adequately prepared and can build their capabilities to manage the more complex experimentally designed marketing tests. Slide 9: If you want to learn more about improving marketing performance using experimental design, please don’t hesitate to contact me or any of my colleagues in Bain’s Customer Strategy & Marketing practice. Read the full report: Act now! Triple your direct marketing effectiveness

In this brief audio presentation, Bain partner John Senior explains how companies are using experimental design to create more effective marketing campaigns, increase conversion rates and retain high-value customers.

Companies spend huge sums on direct marketing to acquire or retain customers through email, direct mail, catalogs and other tactics. Yet the return on investment (ROI) is often poor because the response rates tend to be very low—usually less than 5% and often less than 0.5%—and rates have been declining.

Thanks to recent advances in statistical methods, modeling and analytics, leading marketers now have a far more powerful and sophisticated technique called experimental design. The method allows marketers to increase exponentially the variables tested in a single campaign (product offers, messages, incentives, mail formats and so on) and to test multiple offers in the market simultaneously. Marketers learn exactly which variables entice consumers to act. As a result, response rates rise dramatically, the effectiveness of future campaigns improves and overall return on spending increases.

Transcript

Slide 1: I’m John Senior, a partner in Bain’s Customer Strategy and Marketing practice, and I’d like to spend a few minutes talking about Bain’s point of view on improving marketing performance using experimental design.

Slide 2: Experimental design is an efficient method of in-market testing that can improve the performance of direct marketing.

The power of this approach is that it increases response rates and organizational learning by massively increasing the variables tested in marketing campaigns and identifying which variables actually drive its customers to act.

Experimental design has been used in a variety of situations, including direct mail, email and online marketing and has now been extended to call centers and customer experience programs.

However, experimental design tests can be more complex to develop and require greater organizational preparation.

Slide 3: Many organizations still use traditional “champion-challenger” approaches, or A/B testing, that test one sort of marketing offers versus another. There are limitations to this approach, in that it can’t isolate which variables drive customers to act, and the rate of organizational learning is relatively slow.

Slide 4: Experimental design allows you to test many variables in different offer combinations and isolate which attributes cause customers to respond. You can then combine the best-performing attributes together into the most powerful offers to launch in large-scale campaigns.

Slide 5: We've applied this approach several times with a large cable operator to acquire, retain and upgrade customers with great success. We’ve seen response-rate lifts of many times the existing in-market offers, with a significant improvement in retention and a 20% increase in average revenue per user, which is a key metric in the industry. This is translated into hundreds of millions of dollars to the top and bottom line.

Slide 6: In the experimentally designed acquisition campaign with the cable operator, we targeted a particular high-value customer segment that had been difficult to acquire historically. The best-performing experimentally designed offers delivered a 3x-4x response-rate lift over the existing in-market offer, which translated to several hundred thousand new subscribers.

Slide 7: We were also able to identify which of the offer elements drove response rates, with some surprising findings. We found that the presentation of the incentive and the message itself were more important than the headline offers and discounts. We also found that including a free device had little impact on response rates, which was totally unexpected and saved the company a lot of money because these devices are very expensive.

Slice 8: Having run many experimentally designed tests now across several industries, we've learned some important lessons:

The richest offers don’t always drive the highest response rates; message, format and design can have a huge performance impact;
It’s important to use a stable of offers in the market to prevent offer fatigue, which depresses response rates over time;
And it’s also very important to work with a sales and service organizations to make sure they are adequately prepared and can build their capabilities to manage the more complex experimentally designed marketing tests.

Slide 9: If you want to learn more about improving marketing performance using experimental design, please don’t hesitate to contact me or any of my colleagues in Bain’s Customer Strategy & Marketing practice.

Read the full report: Act now! Triple your direct marketing effectiveness