Experimentation at Scale

The Experimentation at Scale Questions Leaders Are Facing Today

• What does experimentation at scale require beyond running more tests?

  Experimentation at Scale is Bain's approach to continuously improving digital marketing and other key business activities, using AI and machine learning to test messaging, pricing, and product variables across the customer journey to find the combinations that resonate. It helps companies test hundreds of variables at once, using techniques such as fractional factorial multivariate testing, geographic matched-market A/B/N testing, and multiarmed-bandit testing.

  Experimentation pressure tests assumptions against data on real customer response, instead of leaning on past beliefs or gut instinct. Firms that excel in marketing experimentation don’t run one-off tests in silos. They run a consistent, companywide program tied to a business objective; apply a standardized methodology and measurement approach so results can be trusted and compared across markets; and roll out the winning media mix and targeting decisions everywhere. For these companies, experimentation isn’t a side project. It’s a durable generator of higher marketing ROI and sustained sales growth.

• Why do in-market tests produce unreliable results?

  In-market tests can produce unreliable results when companies pour effort into designing the test variations but pay less attention to how the sample audience is selected. Sample selection is critical for in-market tests with a predefined target audience, because success depends in part on dividing the audience into multiple look-alike testing groups.

  The failure is subtle. If the test and control groups differ in their underlying propensity to act, any difference in results reflects that bias, not the test itself, and can confound the treatment and lead to a false conclusion. And a flawed test design can't be fixed after the fact, so this is a problem to solve in the design phase.

  The fixes are straightforward: select samples so groups are genuinely alike before launching, decide how to measure results as part of designing the test rather than after, and start simple and small to build confidence. Rigorous design is what gives marketers confidence in the insights and avoids false conclusions.

• How can we test hundreds of variables without running hundreds of separate tests?

  You can test hundreds of variables without running hundreds of separate tests by using experimental design.
  
  A/B testing has limitations. Marketers can test only a few variables at a time, and working through combinations takes many tests and a long time.

  Experimental design solves that. It lets marketers test many combinations at once, drawing out the effects of hundreds of variables from just a handful of structured tests, so they understand which variables compel customers to act. In Bain’s experience, experimental design has increased response and conversion rates three to five times over historical champion campaigns.

• When should we use adaptive testing methods like multiarmed bandits instead of traditional A/B tests?

  You should use adaptive methods like a multiarmed bandit (MAB) instead of a traditional A/B test when customer touches are frequent and you want to minimize the cost of testing while learning.

  The reason A/B testing falls short is that it’s manual to set up, implement, and interpret. The insights are often short-lived due to shifting preferences and seasonality in many markets. And with frequent customer touches, even the highest-performing messages lose effectiveness by the third time someone sees them.

  MABs, on the other hand, dynamically steer traffic toward winning messages and decrease the cost of testing from lost conversions. The decision rule? Use multivariate testing for touches that an individual sees infrequently, such as a subscription, and use MAB for continuous, high-frequency optimization, such as pricing or always-on campaigns. Matching the method to how often you touch the customer keeps testing fast and economical. Adaptive approaches let marketers earn while they learn instead of learning, then earning.

• How does experimentation become a durable competitive advantage rather than a one-off project?

  Experimentation becomes a durable competitive advantage when it runs as an ongoing capability rather than a one-time project. An embedded experimentation engine continuously improves the business and keeps delivering results over time.

  The best companies adopt an agile test-and-learn mindset, sharpening their digital, pricing, and media engines across the customer journey. They deliberately build the muscle to design high-impact tests, and they back it with a long-term testing roadmap that delivers steady ROI.

  To get there, companies can develop a pipeline of tests across every step of the customer journey, tie a measurement framework to business objectives so the returns are clear, and roll winning results out fast. In our experience, that's what lifts conversion rates and drives incremental revenue.