You’ve heard it before—China is the gold standard for many other markets in the world when it comes to adoption of new digital trends and technologies. The country is No.1 in e-commerce, with e-commerce sales accounting for 46% of retail sales in 2022, according to eMarketer. A survey conducted by Rakuten Insights shows that 76% of Chinese consumers follow at least one influencer on social media. Influencers’ high penetration contributes to a market with an estimated value of 98 billion RMB in 2021. And Bain’s research shows that about 40% of food is ordered online. There is little doubt that these digital platforms are on the rise, especially following the pandemic, when they already occupy the minds and hearts of many Chinese consumers.

Marketing spending allocated for these kinds of digital activities have risen in the past few years. Across a sample of our top Chinese fast-moving consumer goods (FMCG) clients, influencers and O2O platforms now accounts for 15% to 30% of their media budget. And this trend is not slowing down anytime soon, as a survey of top Chinese executives indicates that spending on new digital channels is expected to grow at an annual CAGR of more than 30% to 50%, outpacing the growth of overall digital media.

With greater power comes great responsibility, as the saying goes. While at the inception, executives were content with using metrics like “watch time,” “likes,” and “shares” as proxies for business outcomes, increasingly more of them are asking their marketing teams hard-hitting questions, like “How do I compare the key results you are showing me today with the TV report from last week?” “What is the role of these channels relative to the rest of the marketing mix and path to purchase?” “Why are these digital activities not reflected in the media mix modeling (MMM) report you presented to the management team last week?”

The reality is that the rise of new channels is a double-edged sword. The evolution of measurement standards and practices has not kept pace with the velocity of adoption. MMM measurement typically requires a three-year baseline of weekly sales data matched to the most closely related media metrics (gross rating points for TV, clicks for search marketing)—none of which are robustly established for most of these new channels, which have not been around long enough and do not have a clear winning metric. A case in point—online influencers, where impression fraud and follower farms make it hard for industry practitioners to agree on a common currency, unlike search marketing.

Compounding the issue is the fact that e-commerce penetration, although high, is not equal among all industries. For some FMCG companies, a large proportion of their sales still occurs in grocery or modern trade. This makes it harder to directly measure and attribute the effect of digital platforms on sales that happen offline. While companies are exploring new methods, such as tying influencer measurement to the blockchain and comparing that to digital wallet transactions to overcome this issue, such methods are still too nascent to be readily adopted at scale.

A consumer products company faced this same dilemma. As marketing spending on new platforms increased year over year, the imperative to hold these spends accountable became stronger. At the same time, there was a desire to move away from a smattering mix of ad hoc metrics (potential footfall, engagements) toward a more consistent and business-driven set of metrics (sales, e-commerce traffic). This provided extra motivation to reinvent the testing and measurement methodology to suit the reality of these new digital channels.

The company implemented a matched-market experimentation program that tested the spending mix hypotheses across various digital platforms to determine how it affected omnichannel sales. In all examples, the key hypotheses asserted that maximizing a digital platform’s media spending will result in higher sales returns (see Figure 1).

Figure 1

Three steps brought the experiment to life:

• Step 1: Select existing campaign to run experiment on and develop test design for control and experiment regions with relevant KPIs
• Step 2: Identify matched markets based on a host of internal and external factors and sense check qualitative elements
• Step 3: Measure the sales impact of the experiment and compare across regions

In addition, to eliminate any potential noise from the ongoing Covid-19 pandemic during the course of the experiments, daily Covid-19 case counts were a critical consideration in our matched-market models. This ensured fair representation of the impact of the pandemic across all key matched markets and helped to avoid any bias in the results of one matched market versus another. The data came from public sources provided by local governments and was fed into the model on an ongoing basis, alongside the other macro indicators mentioned above.

The experiment results helped determine the right mix of digital platforms spending by comparing sales percentage increase and identifying points of diminishing return. Furthermore, the ways the experiment affected full-funnel marketing metrics such as e-commerce site traffic and add-to-cart rates considered both online and offline sales data outcomes. As a result, the consumer products company adjusted its creative messaging and call-to-actions for future ads to suit the potential likelihood of conversion on various channels.

The benefits of embedding these capabilities were major catalysts for the company’s decision to adopt testing and learning at scale. By appointing key members of the marketing, sales, research, and e-commerce teams to an Agile testing squad, they were able to circumvent the typical approval processes required to share resources and data across different teams in the organization. Combined with the codification of tools (matched markets selector, media sufficiency calculator), this helped to create a clear bias to action, which then made it subsequently easier to scale existing processes for more testing across the broader organization.

That experience makes it clear that the future of marketing experimentation in China, considering the rise of new digital formats, is founded on an Agile methodology that goes beyond typical measurement methods. Until new technologies are stable enough for scaled adoption, and until e-commerce penetration reaches parity across all industries, a matched-market testing methodology capable of eliminating noise during times of uncertainty looks to be our best bet. Companies can consider testing methodologies like pre-post methodologies once the impact of the pandemic and sales seasonality on experimental noise is reduced or minimized.

New times call for new measures. Instead of using the same age-old measurement methods, executives and marketers need to reimagine a new world of measurement for today’s new digital platforms.