Oil&Gas Financial Journal

Big Data analytics in oil and gas

With new tools and advanced analytic capabilities, oil and gas producers can capture more detailed data in real time at lower costs.

Von Vishy Padmanabhan

First published on September 22, 2014
Min. Lesezeit


This article originally appeared in the September 2014 issue of the Oil & Gas Financial Journal. Big Data analytics may be new to some industries, but oil and gas companies have dealt with huge quantities of data for decades in their quest to learn what lies below the surface. Seismic software, data visualization and now a new generation of pervasive computing devices—sensors that collect and transmit data—continue to open new possibilities. With these new tools and advanced analytic capabilities, oil and gas producers can capture more detailed data in real time at lower costs, which can help them improve oilfield and plant performance by 6% to 8%, according to research by Bain & Company. Those numbers are typical across industries. Our recent survey of more than 400 executives in many sectors revealed that companies with better analytics capabilities were twice as likely to be in the top quartile of financial performance in their industry, five times more likely to make decisions faster than their peers and three times more likely to execute decisions as planned. Our research also found that few companies are really ready to make the most of all this data: Only about 4% of companies across industries have the talent and skills they need to draw tangible business value from analytics. Although some oil and gas companies have invested in building up their capabilities, many struggle to get their arms around this powerful new opportunity. Our conversations with senior executives suggest that they are keenly aware of the promise of advanced analytics, but their teams have difficulty realizing the potential. Too often, companies delegate analytics to the IT department. But in practice, it belongs in the business, under the watchful eye of the CEO or another top executive, to make sure the effort delivers value to the business. As executives assess the potential in analytics, the first question they should ask is, where can analytics deliver the most value for the company? We see opportunities in unconventional and conventional production, as well as in midstream operations. For example, good analytics can help producers collect and analyze data on subsurface and geographic characteristics to get a more detailed view of shale basins. In conventional production, producers can move beyond measurement into predictive tools with a range of pattern-recognition techniques that help them spot trends and drill with more predictable outcomes. In midstream, data analytics can help monitor pipelines and equipment, enabling a more predictable and precise approach to maintenance. Knowing the potential value of better analytics is just the start. Oil and gas executives then need to define an organizational model that encourages collaboration across functions and puts the right data in the hands of decision makers. For example, in an asset-based model, with the functions deployed in the field and reporting into one geographically based organizational structure, good analytics can help managers become more efficient in managing the variety of data (including seismic, drilling logs, operational parameters such as drill bit RPMs and weight on bit, frack performance data and production rates) that help optimize well design and production. Each function may produce volumes of data, but unless the operating model can weave it together and put the right data in the right hands at the right time, it is difficult to improve performance. As in other industries, many energy companies have complex, legacy IT systems that have evolved over decades and that now contain many different islands of disparate data sets. Adding large volumes of real-time, unstructured data exacerbates the problem—but that’s where valuable insights arise. Some companies take a two-track approach that includes a long-term plan to modernize the entire IT stack while allowing a well-funded (but well-bounded) separate program that moves quickly on the most important opportunities. Finally, analytic talent is a scarce resource and most energy companies will need to hire data professionals to make the most of this opportunity. They should also avoid the temptation of trying to solve this issue just by throwing high-priced analytic labor at the problem. The right talent model should balance industry knowledge with analytic skills in a model focused on solving specific problems and identifying new opportunities. This capabilities upgrade should include people who understand open-source models, cloud technologies, pervasive computing and iterative development methodologies. Oil and gas companies will need to improve their analytics capabilities in order to compete in an industry where decisions are moving faster and the stakes are growing ever higher. Creating a world-class analytics capability takes time and investment, and it can happen only with a sustained focus by top management. Now is the time to develop a plan that maps an organization’s ambitions against its capabilities—and describes the path toward a world-class, advanced analytics capability. Vishy Padmanabhan is a partner in Bain & Company’s Dallas office. He works with the firm’s Global Oil & Gas practice, and is also a member of the Global IT practice.

This article originally appeared in the September 2014 issue of the Oil & Gas Financial Journal.

Big Data analytics may be new to some industries, but oil and gas companies have dealt with huge quantities of data for decades in their quest to learn what lies below the surface. Seismic software, data visualization and now a new generation of pervasive computing devices—sensors that collect and transmit data—continue to open new possibilities.

With these new tools and advanced analytic capabilities, oil and gas producers can capture more detailed data in real time at lower costs, which can help them improve oilfield and plant performance by 6% to 8%, according to research by Bain & Company. Those numbers are typical across industries. Our recent survey of more than 400 executives in many sectors revealed that companies with better analytics capabilities were twice as likely to be in the top quartile of financial performance in their industry, five times more likely to make decisions faster than their peers and three times more likely to execute decisions as planned.

Our research also found that few companies are really ready to make the most of all this data: Only about 4% of companies across industries have the talent and skills they need to draw tangible business value from analytics. Although some oil and gas companies have invested in building up their capabilities, many struggle to get their arms around this powerful new opportunity.

Our conversations with senior executives suggest that they are keenly aware of the promise of advanced analytics, but their teams have difficulty realizing the potential. Too often, companies delegate analytics to the IT department. But in practice, it belongs in the business, under the watchful eye of the CEO or another top executive, to make sure the effort delivers value to the business.

As executives assess the potential in analytics, the first question they should ask is, where can analytics deliver the most value for the company? We see opportunities in unconventional and conventional production, as well as in midstream operations. For example, good analytics can help producers collect and analyze data on subsurface and geographic characteristics to get a more detailed view of shale basins. In conventional production, producers can move beyond measurement into predictive tools with a range of pattern-recognition techniques that help them spot trends and drill with more predictable outcomes. In midstream, data analytics can help monitor pipelines and equipment, enabling a more predictable and precise approach to maintenance.

Knowing the potential value of better analytics is just the start. Oil and gas executives then need to define an organizational model that encourages collaboration across functions and puts the right data in the hands of decision makers. For example, in an asset-based model, with the functions deployed in the field and reporting into one geographically based organizational structure, good analytics can help managers become more efficient in managing the variety of data (including seismic, drilling logs, operational parameters such as drill bit RPMs and weight on bit, frack performance data and production rates) that help optimize well design and production. Each function may produce volumes of data, but unless the operating model can weave it together and put the right data in the right hands at the right time, it is difficult to improve performance.

As in other industries, many energy companies have complex, legacy IT systems that have evolved over decades and that now contain many different islands of disparate data sets. Adding large volumes of real-time, unstructured data exacerbates the problem—but that’s where valuable insights arise. Some companies take a two-track approach that includes a long-term plan to modernize the entire IT stack while allowing a well-funded (but well-bounded) separate program that moves quickly on the most important opportunities.

Finally, analytic talent is a scarce resource and most energy companies will need to hire data professionals to make the most of this opportunity. They should also avoid the temptation of trying to solve this issue just by throwing high-priced analytic labor at the problem. The right talent model should balance industry knowledge with analytic skills in a model focused on solving specific problems and identifying new opportunities. This capabilities upgrade should include people who understand open-source models, cloud technologies, pervasive computing and iterative development methodologies.

Oil and gas companies will need to improve their analytics capabilities in order to compete in an industry where decisions are moving faster and the stakes are growing ever higher. Creating a world-class analytics capability takes time and investment, and it can happen only with a sustained focus by top management. Now is the time to develop a plan that maps an organization’s ambitions against its capabilities—and describes the path toward a world-class, advanced analytics capability.

Vishy Padmanabhan is a partner in Bain & Company’s Dallas office. He works with the firm’s Global Oil & Gas practice, and is also a member of the Global IT practice.