One great idea used to be enough to set a chemical company on the road to success. Leadership teams could make big bets on breakthrough ideas, and when blockbusters such as nylon, Kevlar and Gore-Tex reached the market, they often made companies—as well as the executives who championed these new ideas.

Now, the nature of innovation has changed, and there are fewer breakthrough chemicals and compounds. Innovation is still important, but increasingly, it's applied to enhancing functionality, customizing existing products or adapting products to new market opportunities.

This shift has major implications in the ways that chemical companies should manage their R&D functions. Most have recognized this shift, but Bain research finds that while two-thirds of executives say innovation is a top priority, less than 25% believe their companies are successful innovators. Many have not integrated recent innovation techniques occurring in other industries, such as Agile methodologies and digital technologies. Development cycles take too long, teams have difficulty prioritizing projects, and many senior executives see R&D as something of a black box and don't understand why returns from innovation are not higher.

Our work on innovation performance with hundreds of companies in chemicals and other industries offers insights into achieving top-quartile performance in chemicals based on the best practices of the most innovative companies from technology, healthcare, financial services and other industries. From this broad experience, three imperatives stand out for chemical companies.

• Clarify strategic direction. Chemical companies must link R&D priorities closely with corporate strategy, identifying core products, regions, end markets and customers. While it's tempting to chase the siren song of profitable or high-growth markets, concentrating R&D on a few areas of strength maximizes returns.
• Focus on customer needs. Chemical companies need to improve how they bring the customer's perspective into the R&D process. Leading innovators develop a deep understanding of the customer's strategy and product roadmap, and then tailor their materials to meet the customer's needs.
• Adapt the innovation operating model. Innovation requires effective interaction among R&D, strategy, marketing, sales, supply chain and support functions. The operating model serves as the blueprint for organizing and managing resources, including organizational structure, decision-making accountabilities, ways of working and governance.

With growth slowing in developed markets and demand slackening in some key developing ones, chemical executives are challenged to meet growth targets. What's more, studies of new product launches find that half fail to meet expectations. Innovation done right can help close the growth gap, but, too often, executives decide to either throw money at the problem or hope that the stage-gate process will deliver "good enough" products.

But innovation is too important to leave to chance. To demystify and improve it, executives need a better understanding of the elements of success and where they stand in relation to them.

Understand root causes with an innovation diagnostic

An innovation diagnostic should reveal how R&D resources are allocated by business unit, end market and horizon (incremental, radical, breakthrough). It should also give some idea of whether the innovation pipeline is sufficient to meet growth goals. Finally, a good diagnostic needs to shed light on decision effectiveness. How are innovation decisions made, by whom, and what's the result?

As R&D is highly cross functional, it's important to assess the effectiveness of interfaces with strategy, marketing, sales, supply chain, finance and human resources. Doing so allows executives to examine and rate R&D's integration with the rest of the company. Also, benchmarking an assessment against the performance of top innovators can give a clear picture of R&D effectiveness (see Figure 1).

Avoid common R&D failure modes

When R&D organizations in chemical companies are struggling to deliver on their promise, they are often suffering from one of three common failure modes.

Strong team running in too many directions. A lack of clarity about strategic direction or the source of profits can diffuse focus. Some companies spread their R&D investment too thinly across too many products, end markets and customers, hoping for a home run. These organizations frequently attract top researchers, but the lack of direction undermines R&D returns.

One specialty chemical company that was spreading its efforts across more than 40 end markets found that 70% of revenue and more than 95% of earnings came from just three markets. Its customer loyalty scores were higher than competitors in these areas, but it was spending less than half of its R&D resources there. By refocusing 80% of its R&D against those three areas, the company increased market share in its most profitable areas and was described by one analyst as having an R&D engine that is "the envy of the industry."

Narrowing the focus typically requires a few discrete actions.

• Build a robust strategic planning process. Ensure the strategic planning process involves leaders from strategy, the business lines, sales and R&D in order to reach the right level of specificity on priority market segments that can guide R&D resource allocation.
• Create a profit cube. Engage finance to do the hard work of building a profit cube that goes beyond revenue and gross margin to illustrate the full operating profit of each product, region, end market and sometimes even customer.
• Be disciplined about where to compete and where not to compete. Enforce difficult choices about when to kill projects and where to invest rather than spreading investments across too many end markets.

Fascinated by cool engineering problems rather than customer needs. Challenging problems may be more interesting to work on than delivering features that customers will buy. Sometimes R&D hasn't invested enough time understanding customer needs and their product roadmap, or is unable to translate that into requirements. Bringing the customer's voice into the process takes several forms.

• Understand customer needs. Ask how customers would allocate your R&D spending. When one company learned that its customers' highest priority was improving resin properties to allow a faster injection molding process, it shifted its focus from new products to solving this problem. R&D worked closely with the commercial group to evaluate existing designs and those of competitors to develop a more valuable resin, which then allowed the company to raise prices.
• Partner with customers. 3M likes to talk about the importance of deep customer intimacy or "spending time in the smokestacks," meaning go where the customers are. Leading chemical companies set up innovation exchanges where customers' product development teams share their roadmap. This helps identify how the properties of chemical products will need to evolve to meet new demands. These exchanges often involve product teardowns—for example, sorting through the metal parts of a vehicle to identify where high-performance plastics can reduce weight. Fast prototyping presents another opportunity for getting closer to customers—that is, if chemical companies can realize fast alignment among key technical people as well as tighter alignment with the outbound supply chain.
• Extend R&D investments across customers and markets. New products or applications are too often sold to a single customer, but they can frequently be reapplied to work for other customers and adjacent markets. Doing so requires tight integration among R&D, strategic marketing and sales organizations.

The operating model hinders execution. R&D's operating model should serve as a blueprint for the organization and management of resources. However, poor decision rights, weak stage-gate discipline and failure to define the right ways of working can make the operating model an obstacle to progress. Leading companies assess their operating models to ensure these imperatives.

• Clarify decision rights and accountability for critical decisions. One specialty chemical maker streamlined decision rights and reduced the typical number of decision makers on new R&D projects from more than 20 to two. The company introduced new roles, including a project manager to accelerate time to market for application development projects and a new technical sales role to source new projects. Finally, a new weekly review meeting among the business leadership, sales, R&D and supply chain teams gave new projects a go or no-go. These changes contributed to 15% growth in earnings in one year.
• Tailor the development processes to the cycle time of innovation with stage-gate discipline. Some R&D focuses on developing applications in response to customer needs. Stage gates are few, as the parameters for quality and success are well articulated. Many chemical companies rely on Agile techniques for these projects. In contrast, true research projects pursue an unshaped need based on market trends or vaguely defined opportunities. This type of innovation requires numerous, sequential stage gates and ruthless honesty from executives about the business case. It's critical to tailor the development process for these two very different types of innovation.
• Structure the organization to unlock sources of value. To determine the right organizational structure, executives need a clear picture of the role of a central R&D team, where resources should be located and which capabilities are most necessary. Begin by translating innovation strategy into a small set of specific principles, such as extending existing technologies to new markets, making the most of global capabilities or improving cross-functional coordination. These design principles then form the basis for evaluating organizational structure alternatives.

Benefit from an R&D transformation

Chemicals R&D doesn’t have to be a black box. Chemical companies that invest time assessing and transforming R&D reap significant benefits, including:

• greater ability to generate organic growth;
• reversal of share losses without increasing investments;
• clarity on which R&D projects to shelve and which to boost;
• more effective commercialization across the organization; and
• stronger decision rights and accountability, ensuring a focus on the customer.

Improving R&D requires time, commitment from senior leaders and persistence to see it through to full potential. But it's a journey that every chemical company should be considering.

Applying Agile in chemicals

The fundamentals of Agile are straightforward: To capture an opportunity, form a small, cross-functional, empowered and self-managing team to focus on it. The methodology originated in the software industry, where it has improved the productivity of tens of thousands of software development projects by as much as 39%. Other industries have adopted the methodology, too, but it's been slow to catch on in chemicals.

An Agile team's initiative owner, typically drawn from a business function, uses techniques such as design thinking to build a catalog of promising ideas or features and to continuously (and ruthlessly) prioritize the list based on potential value to customers and the company. The team breaks top-priority tasks into small modules, decides how to work on them and then starts building working versions in short cycles known as sprints.

Among the most important success factors for applying Agile methods to R&D in the chemical industry:

• Stay focused on customer needs. Agile teams must continuously refine priorities to ensure they are working on the right opportunities. Sometimes that means saying no to senior executives who ask to jump the queue with a pet project. Success depends on empowering the team to say no.
• Include the right functions on the team. It's essential to include general managers and product-line leaders, but also include sales, which brings the voice of the customer to the team. Marketing helps size the market beyond current customers, manufacturing and supply chain can tell whether a product can be made within the right cost and timing, and R&D defines what is technically possible.
• Build an Agile operating model. Agile teams can move quickly, but to bring products to market faster, companies usually have to change the larger organization. For example, support functions such as planning and procurement will need to become more Agile to keep up with R&D's productivity gains.

Kayvan Ardalan is a partner with the Performance Improvement practice and leads Research and Product Development globally. He is based in Los Angeles. Whit Keuer and Bram Vanassche are partners in Bain’s Chemicals practice. Whit is based in the Dallas office, and Bram is based in the Brussels office.