Brief

Crazy Idea or Game-changing Breakthrough? Put It to the Test

Companies with a nimble test-and-learn culture will seize new opportunities quickly.

By Chris Brahm, Richard Lichtenstein and Elizabeth Spaulding

First published on Νοεμβρίου 01, 2017
min read


Among the many clichés born of Silicon Valley’s start-up culture, few are as misunderstood as the notion of “fail fast”—the idea that a company should simply offer a new product or feature to the market and see what happens. While the spirit of experimentation is right, poor test-and-learn execution has been known to hobble a company’s fortunes for years to come. But just like social networks and search portals, the fail fast concept has grown up. Rather than introducing new products willy-nilly, companies are using technology to collect vast amounts of customer feedback in less time, extracting the lessons and errors from their trials sooner. These powerful test-and-learn techniques are helping companies get to the best answers faster. Consider the story of an American restaurant chain. Two years ago, the company set out to improve the quality of its cuisine—a mission that demanded a radical kitchen overhaul. Rather than waste months evaluating appliance and menu options in hopes of finding the perfect layout for its almost 1,000 restaurants, the company put its best options to the test. It set up learning labs in a handful of restaurants, allowing it to try different equipment configurations and collect feedback from chefs and diners about the resulting meals. After three months, one design earned the strongest customer satisfaction scores. The company brought the approach to four dozen sites to collect more feedback, allowing it to hone the concept further before deploying the final result to all of its restaurants. If you’re in direct marketing, you’re probably familiar with basic test-and-learn techniques, such as A/B tests that let marketers offer two different messages to the market and elevate the more effective one. Now, new digital tools are making it easier to run complex pilots, such as multivariate campaigns that can test five or more attributes with several variations on each to get to the best answer. With virtually endless possibilities, test-and-learn companies can: Study less, test more. When it comes to evaluating new product features and process changes, test-and-learn companies skip lengthy project studies and test their best options with customers and stakeholders. An online news outlet that wants to find the most effective headline for an article might try several options during the first hour of publication to see which one earns the most clicks from readers. Think more boldly. Companies use test-and-learn approaches to not only improve their marketing but to pilot changes to products, processes and even game-changing ideas that have the potential to shift a business’s direction. The best ideas often start with “I know this sounds crazy, but.…” Many software-as-a-service companies rigorously test disruptive “freemium” offers—free products or services that require customers to pay for additional features—to understand customer preferences and gauge the potential for cannibalization before expanding. Move forward when they have a minimum viable product. Companies benefit when they release incremental product improvements rather than wait for a complete solution that addresses every angle of a problem. Most of the time, the perfect answer doesn’t exist. And many customers welcome the opportunity to provide feedback on new features and products in return for early access. In fact, technology companies such as Apple and Google have been using beta testing to build customer loyalty for years. Embrace rapid feedback and iteration. The most effective tests last a week or two—enough time to gather feedback to highlight the options that resonate most. From there, a company can dump the weak options, improve the best ones and test them again, with an eye toward increasing sales or productivity. An organization that wants to change its workspace might start with a radically different concept (for instance, replacing all assigned desks with couches and communal tables) and gradually alter the layout (and desk-to-couch ratio) until it reaches the ideal conditions for collaboration. Executives that cling to waterfall planning models might dismiss the notion of putting unproven ideas out on the marketplace for feedback as too risky. After all, companies gain a certain level of comfort from spending months on plans, budgets and product updates, even if those plans are outdated by the time they’re finished. In our experience, test-and-learn companies have the opposite experience. By breaking down major initiatives into smaller, testable parts before rolling out plans widely, companies get ahead of problems and make critical improvements early. The results help them cut through office politics and resolve internal disputes by letting customers judge. Reducing the risk of large-scale failures often frees employees to come up with bigger, more innovative ideas. Companies find they can implement their best ideas faster, allowing them to reap the rewards sooner. That’s because a true test-and-learn approach is disciplined and rigorous. Before any tests begin, a team develops hypotheses for the options in question that consider the company’s strategy, target customers and financial goals. From there, the team decides how it will test the options and sets the right measures of cost and benefit. Over time, the company builds a body of knowledge around testing that improves each trial. Intuit has been a skillful user of test-and-learn approaches for more than a decade—ever since founder Scott Cook encouraged employees to step away from their slide presentations and traditional meetings and talk to actual customers. Under the Intuit approach, employees collect feedback from customers while they’re using its software in their homes or at work. From there, the company identifies pain points and quickly develops code to address them, circling back to customers for their feedback. When faced with international competition from the upstart Xero a few years ago, Intuit mobilized its test-and-learn approach to rapidly overhaul QuickBooks Online, relaunching it as a cloud-based open platform. The number of QuickBooks Online subscribers increased by 58% during its most recent fiscal year. More recently, the hip fashion brand Shinola has been using test-and-learn technology to stay ahead of changing tastes in the hypercompetitive accessories market. Thinking ahead to its 2018 women’s watch collection, Shinola’s merchandising team reached out to the analytics firm MakerSights to test style hypotheses about its flagship line. MakerSights used its technology to collect feedback from 1,700 customers in less than 48 hours. In the end, two watch silhouettes stood out, including a version with rose gold accents that was especially popular among women under 35—a strategically important customer segment. The data prompted Shinola to invest heavily in rose gold watch stylings, which it will feature prominently in its spring marketing and store displays. Holding tight to long-term waterfall-based planning is the fastest way for companies to stagnate and atrophy in a business climate of constant change. In contrast, a company with a strong outer game—a guiding strategy—and a nimble test-and-learn culture will seize new opportunities quickly. After all, today’s crazy idea might be tomorrow’s disruptive technology. So why not find out now? Chris Brahm leads Bain & Company’s Global Advanced Analytics practice and is based in San Francisco. Richard Lichtenstein is a New York–based principal with the firm’s Advanced Analytics practice. Elizabeth Spaulding coleads Bain’s Global Digital practice and is based in San Francisco.

Among the many clichés born of Silicon Valley’s start-up culture, few are as misunderstood as the notion of “fail fast”—the idea that a company should simply offer a new product or feature to the market and see what happens. While the spirit of experimentation is right, poor test-and-learn execution has been known to hobble a company’s fortunes for years to come.

But just like social networks and search portals, the fail fast concept has grown up. Rather than introducing new products willy-nilly, companies are using technology to collect vast amounts of customer feedback in less time, extracting the lessons and errors from their trials sooner. These powerful test-and-learn techniques are helping companies get to the best answers faster.

Consider the story of an American restaurant chain. Two years ago, the company set out to improve the quality of its cuisine—a mission that demanded a radical kitchen overhaul. Rather than waste months evaluating appliance and menu options in hopes of finding the perfect layout for its almost 1,000 restaurants, the company put its best options to the test. It set up learning labs in a handful of restaurants, allowing it to try different equipment configurations and collect feedback from chefs and diners about the resulting meals. After three months, one design earned the strongest customer satisfaction scores. The company brought the approach to four dozen sites to collect more feedback, allowing it to hone the concept further before deploying the final result to all of its restaurants.

If you’re in direct marketing, you’re probably familiar with basic test-and-learn techniques, such as A/B tests that let marketers offer two different messages to the market and elevate the more effective one. Now, new digital tools are making it easier to run complex pilots, such as multivariate campaigns that can test five or more attributes with several variations on each to get to the best answer. With virtually endless possibilities, test-and-learn companies can:

Study less, test more. When it comes to evaluating new product features and process changes, test-and-learn companies skip lengthy project studies and test their best options with customers and stakeholders. An online news outlet that wants to find the most effective headline for an article might try several options during the first hour of publication to see which one earns the most clicks from readers.
Think more boldly. Companies use test-and-learn approaches to not only improve their marketing but to pilot changes to products, processes and even game-changing ideas that have the potential to shift a business’s direction. The best ideas often start with “I know this sounds crazy, but.…” Many software-as-a-service companies rigorously test disruptive “freemium” offers—free products or services that require customers to pay for additional features—to understand customer preferences and gauge the potential for cannibalization before expanding.
Move forward when they have a minimum viable product. Companies benefit when they release incremental product improvements rather than wait for a complete solution that addresses every angle of a problem. Most of the time, the perfect answer doesn’t exist. And many customers welcome the opportunity to provide feedback on new features and products in return for early access. In fact, technology companies such as Apple and Google have been using beta testing to build customer loyalty for years.
Embrace rapid feedback and iteration. The most effective tests last a week or two—enough time to gather feedback to highlight the options that resonate most. From there, a company can dump the weak options, improve the best ones and test them again, with an eye toward increasing sales or productivity. An organization that wants to change its workspace might start with a radically different concept (for instance, replacing all assigned desks with couches and communal tables) and gradually alter the layout (and desk-to-couch ratio) until it reaches the ideal conditions for collaboration.

Executives that cling to waterfall planning models might dismiss the notion of putting unproven ideas out on the marketplace for feedback as too risky. After all, companies gain a certain level of comfort from spending months on plans, budgets and product updates, even if those plans are outdated by the time they’re finished.

In our experience, test-and-learn companies have the opposite experience. By breaking down major initiatives into smaller, testable parts before rolling out plans widely, companies get ahead of problems and make critical improvements early. The results help them cut through office politics and resolve internal disputes by letting customers judge. Reducing the risk of large-scale failures often frees employees to come up with bigger, more innovative ideas. Companies find they can implement their best ideas faster, allowing them to reap the rewards sooner.

That’s because a true test-and-learn approach is disciplined and rigorous. Before any tests begin, a team develops hypotheses for the options in question that consider the company’s strategy, target customers and financial goals. From there, the team decides how it will test the options and sets the right measures of cost and benefit. Over time, the company builds a body of knowledge around testing that improves each trial.

Intuit has been a skillful user of test-and-learn approaches for more than a decade—ever since founder Scott Cook encouraged employees to step away from their slide presentations and traditional meetings and talk to actual customers. Under the Intuit approach, employees collect feedback from customers while they’re using its software in their homes or at work. From there, the company identifies pain points and quickly develops code to address them, circling back to customers for their feedback. When faced with international competition from the upstart Xero a few years ago, Intuit mobilized its test-and-learn approach to rapidly overhaul QuickBooks Online, relaunching it as a cloud-based open platform. The number of QuickBooks Online subscribers increased by 58% during its most recent fiscal year.

More recently, the hip fashion brand Shinola has been using test-and-learn technology to stay ahead of changing tastes in the hypercompetitive accessories market. Thinking ahead to its 2018 women’s watch collection, Shinola’s merchandising team reached out to the analytics firm MakerSights to test style hypotheses about its flagship line. MakerSights used its technology to collect feedback from 1,700 customers in less than 48 hours. In the end, two watch silhouettes stood out, including a version with rose gold accents that was especially popular among women under 35—a strategically important customer segment. The data prompted Shinola to invest heavily in rose gold watch stylings, which it will feature prominently in its spring marketing and store displays.

Holding tight to long-term waterfall-based planning is the fastest way for companies to stagnate and atrophy in a business climate of constant change. In contrast, a company with a strong outer game—a guiding strategy—and a nimble test-and-learn culture will seize new opportunities quickly. After all, today’s crazy idea might be tomorrow’s disruptive technology. So why not find out now?

Chris Brahm leads Bain & Company’s Global Advanced Analytics practice and is based in San Francisco. Richard Lichtenstein is a New York–based principal with the firm’s Advanced Analytics practice. Elizabeth Spaulding coleads Bain’s Global Digital practice and is based in San Francisco.