Brief

To Market to an Upended World, Test, Test, Test

Grappling with Covid-19, companies are using frequent, simple tests to answer important marketing questions.

By Richard Lichtenstein, Eric Almquist, and Zak Prauer

First published on abril 13, 2020
min read


Executive Summary With so much changing in marketing, speed is critical. So keep testing simple, quick and focused on big things, and act fast on what you learn. Testing will help companies better market during the coronavirus pandemic, and build valuable skills for the time when recovery begins. In the first week of March, just after Washington State reported the first US death from Covid-19 and declared a state of emergency, grocery shopping spiked 30% week over week in Seattle. It was an unprecedented jump—until two weeks later, that is, when it climbed another 40% in Seattle and more than 60% across the country. In category after category, the models that marketers have painstakingly built and relied on have crumbled, overrun by the anomalous events of the coronavirus. Entire advertising categories, such as sporting and live events, are shuttered. Campaigns and messages that seemed fine a few months ago are being redesigned to better match our new world of social distancing. For Our Clients Macro Surveillance Platform For more detail on the business implications of coronavirus from Bain’s Macro Trends Group, log on to the Macro Surveillance Platform. Learn more about the platform > Clearly customers’ needs have changed, and there is no reason to believe they won’t continue to. Nuanced customer segmentation has given way to the tough reality that right now, for many businesses, there are just two key segments: unemployed and working from home. We can’t know for certain how the coronavirus pandemic will unfold, but it is possible to be responsive to customers along the way. Frequent tests focused on big, important questions leading to rapid action will help companies serve customers in a time of ongoing change. Leading companies are already beginning to make this shift. One global consumer products company, for example, recently decided to redeploy what it would have spent on live sporting events now canceled, to creating new messages, pushing them into digital channels, and continuing extensive testing. Gone are ads that encouraged the sharing of food, shaking hands and being in large groups, replaced by spots focused only on the product itself. In response to shifting media consumption, consumer behavior and changes to the media landscape, this company is moving more of its budget out of TV and into digital channels, such as YouTube and streaming TV. And because many of the traditional ways they measured their marketing are now being swamped by changing customer behavior, they are testing everything from their messaging to their media mix. Many of these tests focus on how to optimize marketing in real time, but they are beginning to run tests aimed at planning for the future. Essentially, everyone now has to start over in this unexpected new world. So for companies that have not yet embraced testing, this is an opportunity to begin, to start with simple tests, then build an operating model and culture of testing over time, and increase its sophistication as the economy begins to stabilize. It can be a way to come out of the recession stronger marketers than you went in. Three principles nurture effective testing in the current environment. Throw out old data—customer behavior and preferences have changed fundamentally. Historical mix models will give the wrong answers because they are trained on variance that no longer applies. Testing can help figure out which products and offers will meet customers’ new needs, and the best ways to communicate with empathy the value you can offer and the ways you can help your customers. Test quickly and often. The current situation is unprecedented, and it’s critical to market to customers in ways that address their new needs on today’s most popular channels. Agile principles enable fast-pace testing in sprints, with a focus on getting to market quickly with minimum complexity. Simple approaches like A/B testing offer speed and flexibility. Simple A/B tests make it possible to experiment with different messages, promotions or packaged offers. Covid-19 has led to some significant changes in consumer behavior, and that’s made testing audience important, too. Quarantined adults aged 65 and older who perhaps never would have considered buying groceries online are now turning to these services. To better understand the ramifications of that change, a retailer or food company might set up an A/B test to measure which messages are most effective with that new demographic. One test group would see a message emphasizing a value proposition unique to adults 65 and older, while the other would show the same message to existing online grocery shoppers. A/B testing can also assess how effective a certain marketing channel is or how effectively certain tactics target a specific audience. Such testing can compare, for example, the cost-effectiveness of more expensive ads targeting a consumer who has previously bought something in a category (online sporting goods, for example) to less-expensive ads aimed at a broader demographic. The testing approach is also helpful when assessing a company’s overall marketing mix in light of the recent rapid changes in pricing, inventory and consumption of different media. You could, for example, test a control of 50-50 streaming video vs. Facebook ads, compared first with a 25% streaming-75% Facebook mix, and next with the reverse, a 75% streaming–25% Facebook mix. Because companies must test new concepts and messages that were unthinkable a month ago, it is important to take time up front to articulate a clear testing hypothesis—what you think will work in our new reality. When considering your customers’ needs and how you might better meet them, it’s OK to rely on common sense. A telecom provider might have once had 8 or 10 typical customer profiles, for example. Given the current situation it would be reasonable for the provider to simplify those into three: people who are unemployed, people who are working from home (split into those with children at home and those without) and people who are displaced away from home. For the time being those characteristics are likely to be the most important for that company, and it’s reasonable to trust and move forward based on that understanding, using testing to clarify what works best. Assume you will be in testing mode for some time to come. It’s impossible to predict when consumer behavior and needs will return to normal, or what that new normal will look like, so increased testing will make sense for the foreseeable future. While things remain volatile, stick to simple and rapid tests, but when a more steady state develops, more sophisticated testing techniques such as multivariate or multi-armed bandit testing will reenter the toolkit. Even as things stabilize, it will be helpful to continue to do quick customer research as well to make sure you are staying abreast of customers’ needs. Even the best testing program is worthless if no action is taken based on the results. With everything changing so quickly, rapid deployment of anything you have learned is critical. Results will turn stale fast. Also, apply the insights from testing everywhere you can. If an email is a hit, for example, maybe its message or approach can be applied to your website, too. With the world changing, fast testing and fast action go hand in hand. Related Insights Coronavirus The global Covid-19 pandemic has extracted a terrible human toll and spurred sweeping changes in the world economy. Across industries, executives have begun reassessing their strategies and repositioning their companies to thrive now and in the world beyond coronavirus. Richard Lichtenstein is an expert vice president with Bain & Company based in New York. Eric Almquist is an Advanced Analytics partner in Bain’s Boston office. Zak Prauer is a senior expert in digital marketing based in Bain’s Chicago office.

Executive Summary

With so much changing in marketing, speed is critical. So keep testing simple, quick and focused on big things, and act fast on what you learn.
Testing will help companies better market during the coronavirus pandemic, and build valuable skills for the time when recovery begins.

In the first week of March, just after Washington State reported the first US death from Covid-19 and declared a state of emergency, grocery shopping spiked 30% week over week in Seattle. It was an unprecedented jump—until two weeks later, that is, when it climbed another 40% in Seattle and more than 60% across the country.

In category after category, the models that marketers have painstakingly built and relied on have crumbled, overrun by the anomalous events of the coronavirus. Entire advertising categories, such as sporting and live events, are shuttered. Campaigns and messages that seemed fine a few months ago are being redesigned to better match our new world of social distancing.

For Our Clients

Macro Surveillance Platform

For more detail on the business implications of coronavirus from Bain’s Macro Trends Group, log on to the Macro Surveillance Platform. Learn more about the platform >

Clearly customers’ needs have changed, and there is no reason to believe they won’t continue to. Nuanced customer segmentation has given way to the tough reality that right now, for many businesses, there are just two key segments: unemployed and working from home. We can’t know for certain how the coronavirus pandemic will unfold, but it is possible to be responsive to customers along the way. Frequent tests focused on big, important questions leading to rapid action will help companies serve customers in a time of ongoing change.

Leading companies are already beginning to make this shift. One global consumer products company, for example, recently decided to redeploy what it would have spent on live sporting events now canceled, to creating new messages, pushing them into digital channels, and continuing extensive testing. Gone are ads that encouraged the sharing of food, shaking hands and being in large groups, replaced by spots focused only on the product itself. In response to shifting media consumption, consumer behavior and changes to the media landscape, this company is moving more of its budget out of TV and into digital channels, such as YouTube and streaming TV. And because many of the traditional ways they measured their marketing are now being swamped by changing customer behavior, they are testing everything from their messaging to their media mix. Many of these tests focus on how to optimize marketing in real time, but they are beginning to run tests aimed at planning for the future.

Essentially, everyone now has to start over in this unexpected new world. So for companies that have not yet embraced testing, this is an opportunity to begin, to start with simple tests, then build an operating model and culture of testing over time, and increase its sophistication as the economy begins to stabilize. It can be a way to come out of the recession stronger marketers than you went in.

Three principles nurture effective testing in the current environment.

Throw out old data—customer behavior and preferences have changed fundamentally. Historical mix models will give the wrong answers because they are trained on variance that no longer applies. Testing can help figure out which products and offers will meet customers’ new needs, and the best ways to communicate with empathy the value you can offer and the ways you can help your customers.

Test quickly and often. The current situation is unprecedented, and it’s critical to market to customers in ways that address their new needs on today’s most popular channels. Agile principles enable fast-pace testing in sprints, with a focus on getting to market quickly with minimum complexity. Simple approaches like A/B testing offer speed and flexibility.

Simple A/B tests make it possible to experiment with different messages, promotions or packaged offers. Covid-19 has led to some significant changes in consumer behavior, and that’s made testing audience important, too. Quarantined adults aged 65 and older who perhaps never would have considered buying groceries online are now turning to these services. To better understand the ramifications of that change, a retailer or food company might set up an A/B test to measure which messages are most effective with that new demographic. One test group would see a message emphasizing a value proposition unique to adults 65 and older, while the other would show the same message to existing online grocery shoppers.

A/B testing can also assess how effective a certain marketing channel is or how effectively certain tactics target a specific audience. Such testing can compare, for example, the cost-effectiveness of more expensive ads targeting a consumer who has previously bought something in a category (online sporting goods, for example) to less-expensive ads aimed at a broader demographic. The testing approach is also helpful when assessing a company’s overall marketing mix in light of the recent rapid changes in pricing, inventory and consumption of different media. You could, for example, test a control of 50-50 streaming video vs. Facebook ads, compared first with a 25% streaming-75% Facebook mix, and next with the reverse, a 75% streaming–25% Facebook mix.

Because companies must test new concepts and messages that were unthinkable a month ago, it is important to take time up front to articulate a clear testing hypothesis—what you think will work in our new reality. When considering your customers’ needs and how you might better meet them, it’s OK to rely on common sense. A telecom provider might have once had 8 or 10 typical customer profiles, for example. Given the current situation it would be reasonable for the provider to simplify those into three: people who are unemployed, people who are working from home (split into those with children at home and those without) and people who are displaced away from home. For the time being those characteristics are likely to be the most important for that company, and it’s reasonable to trust and move forward based on that understanding, using testing to clarify what works best.

Assume you will be in testing mode for some time to come. It’s impossible to predict when consumer behavior and needs will return to normal, or what that new normal will look like, so increased testing will make sense for the foreseeable future. While things remain volatile, stick to simple and rapid tests, but when a more steady state develops, more sophisticated testing techniques such as multivariate or multi-armed bandit testing will reenter the toolkit. Even as things stabilize, it will be helpful to continue to do quick customer research as well to make sure you are staying abreast of customers’ needs.

Even the best testing program is worthless if no action is taken based on the results. With everything changing so quickly, rapid deployment of anything you have learned is critical. Results will turn stale fast. Also, apply the insights from testing everywhere you can. If an email is a hit, for example, maybe its message or approach can be applied to your website, too. With the world changing, fast testing and fast action go hand in hand.

Related Insights

Coronavirus

The global Covid-19 pandemic has extracted a terrible human toll and spurred sweeping changes in the world economy. Across industries, executives have begun reassessing their strategies and repositioning their companies to thrive now and in the world beyond coronavirus.

Richard Lichtenstein is an expert vice president with Bain & Company based in New York. Eric Almquist is an Advanced Analytics partner in Bain’s Boston office. Zak Prauer is a senior expert in digital marketing based in Bain’s Chicago office.