Report

Beyond Code Generation: More Efficient Software Development

Generative AI saves time, but meaningful improvements require a broader agenda.

David Crawford, Bill Radzevych, Jue Wang, Purna Doddapaneni, and Martin Goette

First published on сентября 25, 2024
min read


Резюме The arrival of generative AI puts pressure on software development organizations to demonstrate greater efficiency. In practice, generative AI appears to save about 10% to 15% of total software engineering time, and a lot of companies aren’t making profitable use of the savings. Improvements of 30% or more are possible, but they require using the full potential of generative AI and a broader agenda. This article is part of Bain's 2024 Technology Report. Explore the report The introduction of generative AI coding assistants has raised expectations of improving the efficiency of software development. In practice, engineering organizations that are using such tools are seeing efficiency improvements of about 10% to 15% on average (see Figure 1). In many cases, companies fail to monetize even these gains because they’re unable to reposition the saved time and resources to productive uses. Figure 1 Companies already see some efficiency gains with generative AI, but expect to see much more in the future But more is possible. Organizations that take a more comprehensive approach can see efficiency gains of 30% or more. The extra gains result from going beyond generative AI code generation, using generative AI for other tasks, and taking a more comprehensive approach to improving efficiency, including determining the right baselines and metrics. Real efficiency gains Developers spend about half their time writing and testing code, so although they report a 30% improvement from generative AI against those activities, this represents a net efficiency improvement of 15% across developers’ total time (see Figure 2). A more comprehensive approach to efficiency includes not only generative AI-assisted code generation and testing, but a comprehensive look at three dimensions: focusing on the right work, ensuring speedy, high-quality execution (including full potential use of generative AI), and optimizing resourcing costs. Figure 2 Developers spend most of their time creating new products and improving existing ones; generative AI use cases focus on test development and code generation Focus on the right work By far, the fastest way to improve efficiency is to refocus efforts on the work that creates the most value, concentrating on several sets of actions: Align investments with strategy across products and markets. Does the allocation of engineering time match company strategy? Are strategy and roadmaps informed by customer and market insights? Weigh expense-to-revenue ratio across products. Are older products taking resources from new developments? Should support be outsourced to reduce costs? Balance resource allocation across new developments, product improvement, maintenance, technical debt, and quality. Spending too little to address technical debt may eventually slow development. Link product strategy to day-to-day developer priorities. Unclear prioritization can lead to developers addressing the issues they believe are most critical, which are not always the same as those that support strategic goals. Better visibility into how time is actually spent often reveals a mismatch between leadership’s ambitions and the reality of how resources are allocated. Ensure speedy, high-quality execution There are many aspects to executing rapidly with high quality. Generative AI is top of mind today, but foundational elements such as continuous delivery and modern architecture can be more effective ways to drive efficiency (see Figure 3). Figure 3 Clear roadmaps, managing tech debt, and ensuring optimal resource allocation are the most effective ways to improve productivity Deploy full potential generative AI. Leaders in generative AI adoption can achieve up to 30% efficiency from optimal deployment. Intuit, a financial technology platform for consumers and small businesses, offers a good example with its initiative to move from “scrappy testing” to scale development. Intuit set out to improve efficiency and productivity around two themes. First, it wanted to increase development velocity to deliver innovative products and solutions to its 100 million customers, with speed and at scale. Second, it wanted to take full advantage of the inherent benefits of generative AI on its modern development platform to streamline end-to-end development by “shifting left”—that is, bringing critical tasks forward in the software development life cycle. Among the key takeaways from testing and scaling more than 30 different use cases are: Beyond code generation. Intuit used its proprietary generative AI operating system (GenOS) to analyze developer support documentation, logs, and other records to understand how developers have solved common problems in the past, extracting this knowledge to accelerate development velocity. The company created tools that serve up solutions to common developer tasks, meeting development teams where they are in their day-to-day work (integrated development environments, development portal, Slack, etc.) to drive efficiencies. Accelerating with context. While Intuit’s initial generative AI-driven code-generation tool sped up the process by 10% to 15%, by leveraging its generative AI tooling with Intuit-specific code context patterns (repositories, component libraries, etc.), the company reduced integration task completion times by two or three times. Improving the end-to-end development process. Intuit used its generative AI tools to improve standardization of code and documentation for product development teams across personas (software developers, designers, data engineers and analysts, technical program managers, etc.). Plan for continuous integration and delivery. Before developer teams deploy new code, they need to ensure that it won’t break anything in the live product or create security risks. Manual testing is time-consuming, and deploying to a live environment would be risky. Automating the testing in a virtual product environment is a more efficient and safer way to confirm the viability of new code. Continuous integration and delivery of new code also improves efficiency. It’s a more efficient way to manage risk because developers can assess the effects of each new deployment, and it allows companies to address security threats as they are discovered, limiting potential harm that could occur if the patches had to wait. Customers also appreciate a quick response to identified issues and the consistency of ongoing product improvements. Maintain a modern architecture. Modular architecture allows teams to adapt and improve products without reinventing the whole. A continuous investment in modular design avoids falling into technical debt—the cost incurred when companies fail to keep up with evolving technology and must invest heavily at some point to regain their competitive edge. Optimize resource costs Two software development organizations operating at similar speeds and quality can show very different cost profiles, depending on each organization’s model and talent structure. Geographical footprint, outsourcing levels, ratio of senior engineers to other team members, and the roles that various functions play all help determine costs. For example, a staff overloaded with senior engineers can be costly and may be slower to adopt new practices, whereas a staff with too many junior engineers may lack technical depth and result in higher costs despite savings. How to measure impact Many companies struggle to understand their baseline efficiency and measure the improvements they try to get from new initiatives like generative AI. About two-thirds of leaders surveyed aren’t satisfied with the insights they’re getting—or not getting. Many senior executives see software engineering as a black box: They don’t know where the money’s going. Building an effective measurement system requires a bespoke approach and focused attention. To avoid overload, a good target is 3 to 5 KPIs for the senior executive level and up to 10 KPIs for engineering leadership. Tiered systems address different needs of different groups: Executives need to focus on product performance, cost, and resource allocation across priorities. Technical leadership needs a view of whether their efforts are achieving the right business outcomes, and needs to identify barriers and upcoming challenges. Teams need to know if their deliveries are in line with requirements. A dedicated engineering productivity tool to measure efficiency is an essential enabler. Meaningful improvement is possible in software development, but the effort required is more far reaching than introducing a generative AI coding assistant. Investments that increase efficiency, improve execution, and optimize costs consistently pay off, making the effort worthwhile for any R&D or other software development organization. Read the Next Chapter Why Software Companies’ Customer Success Is Failing Read our 2024 Technology Report Download the PDF Explore the report More from the report Commercializing Innovation Investing in a Shifting Market Scope M&A Sovereign AI Top AI Functions The AI Opportunity Big and Small Computing The AI Chip Shortage The Software Cycle AI in Tech Services Deploying Generative AI Software Development Efficiency Customer Success AI Everywhere

Резюме

The arrival of generative AI puts pressure on software development organizations to demonstrate greater efficiency.
In practice, generative AI appears to save about 10% to 15% of total software engineering time, and a lot of companies aren’t making profitable use of the savings.
Improvements of 30% or more are possible, but they require using the full potential of generative AI and a broader agenda.

The introduction of generative AI coding assistants has raised expectations of improving the efficiency of software development. In practice, engineering organizations that are using such tools are seeing efficiency improvements of about 10% to 15% on average (see Figure 1). In many cases, companies fail to monetize even these gains because they’re unable to reposition the saved time and resources to productive uses.

Companies already see some efficiency gains with generative AI, but expect to see much more in the future

But more is possible. Organizations that take a more comprehensive approach can see efficiency gains of 30% or more. The extra gains result from going beyond generative AI code generation, using generative AI for other tasks, and taking a more comprehensive approach to improving efficiency, including determining the right baselines and metrics.

Real efficiency gains

Developers spend about half their time writing and testing code, so although they report a 30% improvement from generative AI against those activities, this represents a net efficiency improvement of 15% across developers’ total time (see Figure 2). A more comprehensive approach to efficiency includes not only generative AI-assisted code generation and testing, but a comprehensive look at three dimensions: focusing on the right work, ensuring speedy, high-quality execution (including full potential use of generative AI), and optimizing resourcing costs.

Developers spend most of their time creating new products and improving existing ones; generative AI use cases focus on test development and code generation

Focus on the right work

By far, the fastest way to improve efficiency is to refocus efforts on the work that creates the most value, concentrating on several sets of actions:

Align investments with strategy across products and markets. Does the allocation of engineering time match company strategy? Are strategy and roadmaps informed by customer and market insights?
Weigh expense-to-revenue ratio across products. Are older products taking resources from new developments? Should support be outsourced to reduce costs?
Balance resource allocation across new developments, product improvement, maintenance, technical debt, and quality. Spending too little to address technical debt may eventually slow development.
Link product strategy to day-to-day developer priorities. Unclear prioritization can lead to developers addressing the issues they believe are most critical, which are not always the same as those that support strategic goals.

Better visibility into how time is actually spent often reveals a mismatch between leadership’s ambitions and the reality of how resources are allocated.

Ensure speedy, high-quality execution

There are many aspects to executing rapidly with high quality. Generative AI is top of mind today, but foundational elements such as continuous delivery and modern architecture can be more effective ways to drive efficiency (see Figure 3).

Clear roadmaps, managing tech debt, and ensuring optimal resource allocation are the most effective ways to improve productivity

Deploy full potential generative AI. Leaders in generative AI adoption can achieve up to 30% efficiency from optimal deployment. Intuit, a financial technology platform for consumers and small businesses, offers a good example with its initiative to move from “scrappy testing” to scale development.

Intuit set out to improve efficiency and productivity around two themes. First, it wanted to increase development velocity to deliver innovative products and solutions to its 100 million customers, with speed and at scale. Second, it wanted to take full advantage of the inherent benefits of generative AI on its modern development platform to streamline end-to-end development by “shifting left”—that is, bringing critical tasks forward in the software development life cycle. Among the key takeaways from testing and scaling more than 30 different use cases are:

Beyond code generation. Intuit used its proprietary generative AI operating system (GenOS) to analyze developer support documentation, logs, and other records to understand how developers have solved common problems in the past, extracting this knowledge to accelerate development velocity. The company created tools that serve up solutions to common developer tasks, meeting development teams where they are in their day-to-day work (integrated development environments, development portal, Slack, etc.) to drive efficiencies.
Accelerating with context. While Intuit’s initial generative AI-driven code-generation tool sped up the process by 10% to 15%, by leveraging its generative AI tooling with Intuit-specific code context patterns (repositories, component libraries, etc.), the company reduced integration task completion times by two or three times.
Improving the end-to-end development process. Intuit used its generative AI tools to improve standardization of code and documentation for product development teams across personas (software developers, designers, data engineers and analysts, technical program managers, etc.).

Plan for continuous integration and delivery. Before developer teams deploy new code, they need to ensure that it won’t break anything in the live product or create security risks. Manual testing is time-consuming, and deploying to a live environment would be risky. Automating the testing in a virtual product environment is a more efficient and safer way to confirm the viability of new code.

Continuous integration and delivery of new code also improves efficiency. It’s a more efficient way to manage risk because developers can assess the effects of each new deployment, and it allows companies to address security threats as they are discovered, limiting potential harm that could occur if the patches had to wait. Customers also appreciate a quick response to identified issues and the consistency of ongoing product improvements.

Maintain a modern architecture. Modular architecture allows teams to adapt and improve products without reinventing the whole. A continuous investment in modular design avoids falling into technical debt—the cost incurred when companies fail to keep up with evolving technology and must invest heavily at some point to regain their competitive edge.

Optimize resource costs

Two software development organizations operating at similar speeds and quality can show very different cost profiles, depending on each organization’s model and talent structure. Geographical footprint, outsourcing levels, ratio of senior engineers to other team members, and the roles that various functions play all help determine costs. For example, a staff overloaded with senior engineers can be costly and may be slower to adopt new practices, whereas a staff with too many junior engineers may lack technical depth and result in higher costs despite savings.

How to measure impact

Many companies struggle to understand their baseline efficiency and measure the improvements they try to get from new initiatives like generative AI. About two-thirds of leaders surveyed aren’t satisfied with the insights they’re getting—or not getting. Many senior executives see software engineering as a black box: They don’t know where the money’s going.

Building an effective measurement system requires a bespoke approach and focused attention. To avoid overload, a good target is 3 to 5 KPIs for the senior executive level and up to 10 KPIs for engineering leadership. Tiered systems address different needs of different groups:

Executives need to focus on product performance, cost, and resource allocation across priorities.
Technical leadership needs a view of whether their efforts are achieving the right business outcomes, and needs to identify barriers and upcoming challenges.
Teams need to know if their deliveries are in line with requirements.

A dedicated engineering productivity tool to measure efficiency is an essential enabler.

Meaningful improvement is possible in software development, but the effort required is more far reaching than introducing a generative AI coding assistant. Investments that increase efficiency, improve execution, and optimize costs consistently pay off, making the effort worthwhile for any R&D or other software development organization.

Read the Next Chapter

Why Software Companies’ Customer Success Is Failing