The Era of Experimentation Is Over with Arvind Krishna, CEO of IBM

Episode Transcript

Arvind Krishna: Have conviction. The technology is useful. I don't think there's any doubts left on that. Have conviction and figure out, that's the art of the leader, to figure out the two or three areas where AI could be a huge unlock, and put enough resource and enough investment behind those.

Sarah Elk: We are excited to welcome Arvind Krishna, Chairman and CEO of IBM. Arvind is a charismatic and humble leader. He is reinventing IBM for the era of AI, and he's looking ahead at the next frontier of quantum, and how that's going to interact in the era of AI. I'm Sarah Elk with Andrew Ng, and this is Winning with AI.

Arvind, thank you so much for being with us today. We're very excited to have you on Winning with AI.

Arvind Krishna: Sarah, it's a pleasure to be here speaking with both you and Andrew.

Andrew Ng: Yeah, it's good to see you, Arvind.

Sarah Elk: One of the things that I think would be interesting to talk about is the changes over the past year. I've heard that you've described day zero and that the time is now, which is different from maybe the proof-of-concept era of the last 12 months. How do you think about what has changed, in terms of the moment that CEOs are facing now?

Arvind Krishna: In most new technologies, there is an era of experimentation. People are trying [it] out because they don't fully trust it, they don't fully believe in it, they're not quite sure whether it's really transformative or not. I think that that has passed for AI. There is no question that it is going to be incredibly productive in a number of areas.

Now, perhaps the mistake some people make is that it's going to be incredibly productive in every single area. That's probably not quite true. I advise all our clients, "Please don't do 100 experiments, pick three, four, five things which you can scale like crazy. And go put the correct investment, the correct teams, and the correct technology behind that to go get that benefit."

Customer experience, customer service, coding, I think we can say that these have been proven out. I don't think there's much of a debate on that. The one where I have a lot of faith and belief is on enterprise operations, or what people might call the back office or how an enterprise functions. And I actually believe it's ready for that also.

Sarah Elk: Yeah, it's interesting. [According to] the Bain CEO survey that we did in 2026, 80% of CEOs are doing something related to AI but are not seeing the return they expect. And I've often wondered, how much activity is there really that's going on? What are you seeing from your vantage point?

Arvind Krishna: Let's say a reasonable enterprise is willing to spend a couple of hundred million dollars a year, which is not a trivial amount of money. But if you now spread it across 100 experiments, you probably have three to five people in each experiment, you got a couple of million dollars of total spend, and then you don't have the ability on how to scale it because you've not though through all of it. If you instead put 100 people on one of those things, and you put 15 or $20 million of spend, and you thought about scaling it, which one is going to be more successful? And it is shocking to me that most people are in the first bucket, not the second.

Andrew Ng: I'd actually love to understand your insider view of IBM over the last 10, 15 years. I think a lot of people, probably everyone in AI, closely follow IBM's technical innovations with the deepest respect. And 15 years ago with IBM Watson, you pretty much owned the AI brand, or people thought IBM AI. And then over these last 15 years, I think under your predecessor, IBM stock did not outperform the market. It was kind of flat, including dividends, total return was kind of flat. And then since you took over, certainly from a share price, shareholder return point of view, IBM's done much better. But I'd love to hear the insider view of what IBM's been up to these last 15 years, as waves of AI came.

Arvind Krishna: So before I get to the complete answer of IBM, let's talk about that AI journey. So, we clearly saw that AI could make a difference. I think IBM Watson winning Jeopardy was a huge proof point to the world, that the mixture of compute and algorithmic techniques, in this case deep learning, can be applied and can be applied to the real world, because you had to understand language, you had to look up answers, you had to understand written text. And understanding not just language in a very clinical way, but idiomatic spoken language, analogies, innuendo, all of that as part of Jeopardy proved that.

So now, what was the mistake? The huge mistake we made was we believed, now how do you unlock value or how do you monetize that? And we took the approach that if we construct monolithic vertical applications in certain use cases, then you can derive a lot of value, and that can come back to us, the corporation. That has never been the first four to five years of any new technology. We should have decomposed it and come out with piece parts that let people gain trust, gain knowledge, get their own expertise. But instead, we took the monolithic approach and that was a terrible idea. That is with 20/20 hindsight, and we are going to avoid making that mistake again.

The second mistake we made is that not only did we do monoliths, we then took a monolith in an area where we didn't know very much, which was the area of health. So, is AI applied to health going to be incredibly productive? I really do believe that. However, who is going to derive that value? So we certainly deal with payers, we deal with insurance companies, but we don't deal much with doctors. So in some sense, we didn't know the actual client. Second, we didn't know the regulator. And three, we don't have a lot of medical staff that is inside IBM. It's easy again in hindsight to say that, but you should always try to at least get only one of those areas wrong. So if we had known the doctor or we had known the regulator, it might have played out differently. So, those are my two observations.

The second one I think is that this moment of LLMs is really fundamentally different. It's really about much more, I'll say, brute force compute than it is about throwing hundreds of really smart people at a problem. So, that is a really big inflection point that is going on right now.

Andrew Ng: Yeah. You're a CEO of one of the largest, most consequential companies in the world, and you still talk like an engineer, which I intend as a compliment. I think people that don't know I think are often, he's a PhD from UIUC, engineer researcher, ran IBM research. And I actually love that you've carried that thinking about building blocks, rather than monoliths and offering building blocks to people to assemble themselves. It's a very engineering way of thinking about it, but I think this is the right one for this era. So I'm curious, in light of that, what are the most important building blocks or components to your mind these days?

Arvind Krishna: Look, I think it's not a surprise to our audience here for me to say, platforms tend to survive and win, not so much just individual components or individual products. And I think that IBM should be thought about as having four platforms. One, the mainframe is still going strong, even though it's been 60 years and it runs a lot of the world's transaction workloads. Ones where there is, I'll call it large vertical. There used to be a phrase used, but it's fallen out of woke, scale-up versus scale-out systems. And I think the mainframe is the survivor of the scale-up systems.

The second platform is our hybrid cloud platform. People can think of maybe Red Hat as the anchor products with OpenShift in there. The third is what we are doing around AI. And the fourth we are investing in, but it's probably still a couple of years away from mainstream, is quantum.

Sarah Elk: So Arvind, when you think about the lessons of the last 15 years that you described, how do those apply to the AI platform offerings that you're building now?

Arvind Krishna: So as opposed to monoliths, we want to open it up. So people get into this debate of which LLM, and I say it's going to be an and world. I actually believe people will use multiple LLMs for various reasons. Could be technical, could be political, could be diversity. They would also use open weight models. I've held the belief for five years, but in the last nine to 10 months it's come true in some sense.

As you go across these, you're then going to also need all of the guardrails. People are going to worry about, how do I optimize cost? But also, how do I maintain a full audit trail? How do you do evals? How do you do harnesses? How do you really capture the value? And people talk about hallucination. Look, any statistical technique is always going to be somewhat imprecise, but if we can get it into the 95, 96% accuracy, we are probably beating humans. I think we just have this innate desire that machines have to be perfect and humans can be imperfect, but if you're trying to do cognitive tasks, then maybe up in the 90s is effective enough for many things, maybe not all.

Andrew Ng: Since you touched on mainframes, I'd love to dig more into that. So, IBM has long had a thriving mainframe business. I know that they had questions when Anthropic started using AI to write COBOL and other legacy code. I think I remember you saying, was it to CNBC, that you think only 2% of all the code of IBM could be automated by AI. Do you still feel that way? And how do you explain the reason behind this?

Arvind Krishna: Two separate statements in there, Andrew, that maybe the media coupled together, which should not be. My 2% statement was of the software products that IBM has on the market, how many could be easily replaced by AI, including agents or LLMs? And I said 2% of the products we have in terms of revenue mass could be replaced by AI, I think pretty easily. So those are products that are more of an interaction style than those that have deep data or transactional or moving data around, which I don't think can be easily replaced by AI or just agents. That was the first statement.

The second, absolutely. I actually would agree with Anthropic. I think LLMs are great at writing code. LLMs could be great at taking hundreds of thousands of lines of code and replacing one language with another. After all, those are in some sense deterministic routines. So, I actually agree with that. What I disagreed with is that it's going to be a concern for the mainframe. People don't use the mainframe because of language lock-ins or it runs a bunch of languages. The mainframe gets used when people need to get a certain massive volume of tasks done. If I think about credit card authorization, approving 15 or 20 billion transactions an hour, think about the tokens you would take to do that. I would say in that case, the mainframe is probably 100 times more cost-effective. That's why I think the mainframe survives, not because of a language lock-in.

Sarah Elk: Yeah, Arvind, we've talked about the state of the software industry. There's certainly a tremendous amount of data work that needs to be done, in terms of unstructured data, the connection of that data from an ontology or semantic layer standpoint to make it ready for agentic. I'm curious, as you look across both those topics, how do you think about the need for companies to build their software development lifecycle capability again? Many companies let this atrophy over the last 20 years, and when we think about agentic almost as a new class of software and the need to have those kind of capabilities in-house to take advantage of that analytical layer you're describing on top of the data, I'm curious to get your perspective on how you think about the need for that.

Arvind Krishna: So I'll always begin from, it's got to be in chase of a business goal. So let's for a moment imagine that you're trying to replace, and I've heard you talk about this, Sarah, before, like you're going to go across an end-to-end process.

For example, let's take quote to cash. Somebody's prospecting a customer, somebody's getting a price, somebody is entering that order. You got to make sure that there is inventory, you got to make sure there's distribution, you got to make sure you can invoice to go collect it. Think of that whole end-to-end process. This is probably touching a half dozen or a dozen different systems. Today, how do we do it? We organize the company into silos. Here is sales, here is distribution, here is finance, here is legal, etc. And then we do it within each step and there's a lot of human touch offs and handoffs, and that's what you could call friction or cost inside an enterprise.

Well, agents can go deal with all these systems and actually deal with multiple of them. So that then opens up the question, how do you organize and how do you make sure your data is clean enough? Opening that up and solving that problem for an enterprise, I think has incredible value because that opens up how you can scale the business, how you can begin to grow revenue, how you can make your customer happier because you're doing things faster on their behalf. And that is where I think the massive unlock is, without a whole lot of risk coming into the picture.

Sarah Elk: Yeah, I agree with that. The other thing you said, Arvind, that really resonated with me is the organizational silos and the complexity around that. We're starting to see that translate into complexities and the technical scaffolding and AI componentry that companies are starting to build. And so if there's an organizational silo over here that's starting to build an agentic system, some of those pieces could be composable to another business unit or another function over here, but they're building separately and they're not really thinking through how to take advantage of an AI platform in-house or what that would mean for scaling and compounding internally. I'm curious if you have felt that inside IBM as you've built out your capability.

Arvind Krishna: Absolutely. Look, we started down the journey about four years ago, in early '23. I'll take a simple example from HR. We began by saying, "Can people interact with an agent as opposed to all of the underlying systems, whether it's directories or whether it's your HCM, human capital management system, etc?"

And I'll use a simple example, employee verification for the sake of getting a mortgage. What would happen in the past? You would go to your manager and say, "Get me a letter." The manager had no desire to write that letter. They would ask their HR partner who had no desire to write the letter. They would call up some HR back office who knew nothing about the employee who would go look up three or four different systems, write the letter, get it out. I think we counted up, it was 17 different human touch points, probably maybe an hour of work in total, and then the letter would go out.

So, we asked a simple question. If you are on our network, we know exactly who you are, otherwise you're not on our network. If that's the case, why can't you ask an agent, and you don't even tell the agent who you are, you just say, "I need an EVL, employee verification letter." All the agent should ask you is, "Which email or which physical address?" It can look up the system on your behalf. It can then produce that letter. And what used to be 30 minutes of human work is now you spending 15 seconds, and 30 seconds later the thing is done.

Sarah Elk: Yeah. So as CEO, how did you decide which two or three big domains to go after?

Arvind Krishna: So, I'll give you my internal. We did a piece of work to say, if I think about running the enterprise, there's about 200 processes. There's a few more than that, let's call it 200. I take an approach that if it's too small, like you're only spending a couple of million, it's too small to get a benefit. If it's too large, like you're spending a billion, that's too big.

So, let's call them about $100 million per area that we wanted to go after. So, there's 200 such areas. Okay. Now you take the approach, you can't do them all. So you see who raises their hand as the owner of those to say, "Hey, I'm really excited." In the first two years, actually about 60 people raised their hand. We made progress on five. HR was actually one of the first ones. Then you can say, okay, now I'm getting more skill. It became 10. Over the first two and a half years, we got about 60 of them done. That gives a lot of confidence. The next 70 are now raring to go because they saw what their friends had done. The last 30 or 40 are probably a little recalcitrant. Is that that the human doesn't trust the AI? Is it that the AI is not ready? Is it that they're really much more complicated? It's some mixture of all of those for the last 40.

Sarah Elk: You obviously have a technical background that Andrew alluded to, which helps you maybe see what the potential is for the technology and help frame and break down how to think about that. How critical is it, do you think, that the modern CEO or up and coming CEOs have a technology background of some kind?

Arvind Krishna: I think I would actually say curiosity and being willing to ask really naive and dumb questions is probably far more important than having technical acumen.

Look, as the other partners at Bain, as well as Andrew's compadres will tell you, you can be technical, but very quickly you learn that in adjacent technical areas, you know almost nothing. You know a few areas really well, and in the vast majority you actually don't know very much. I think what it helps is, I'll call it maybe a mindset of, let's deconstruct a problem. What is the smallest size you can tackle that makes progress, but it's not so big that it's like trying to eat a whale? So, I think that's the mindset that's more important, and then be willing to acknowledge maybe the first and second time we try it, we'll make mistakes, it won't be good, but does that teach us that it can go forward or not? So it's more of the risk appetite and just a way of thinking about problems that's useful. But I think a lot of business people, a lot of people who actually tackle tough things have those skills. So I would actually say that that's what is more important.

And have conviction. I think when people are unsure, and this has been used in military, it's called there, I think they call it put both feet in the boat, don't keep one on the shore and one on the boat. I just saw the movie Odyssey, so there it was about burn the boats, right? Don't have a way out that is there. I think those are much more important skills than just depth of technical knowledge.

Andrew Ng: Yeah. Leadership is hard. Right? You got to pick a direction, lead the team there, and then if it fails, it was the leader's fault, [inaudible 00:20:02] the time. So it's hard, but I think in moments like these, leadership really matters.

Arvind Krishna: I completely agree. And be willing to take accountability. It's accountability, not just ownership.

Andrew Ng: Yeah. Hey, one thing I know, I love to talk about AI, my favorite subject for sure. And one thing I know you've spoken extensively about is quantum. Even before you were CEO, you were driving quantum innovations in IBM and I've been following with great interest innovations with Nighthawk, Loon, error correction, and so on. A lot of people are debating, when will anyone's quantum investments come to fruition? Would it be 10 years from now like people have been saying for the last 20 years, or would it be a much shorter time horizon? I'd love to understand what you're seeing and what signals you are seeing that allow you to forecast when it will start to have real business impact.

Arvind Krishna: Yeah. So I'm going to make a provocative statement on the business impact and analogy, and then come back to the signals, Andrew, if that's okay with you. I'm going to tell you that in 2029, so that's only now two and a half years, maybe three years away, quantum will do something that'll surprise us. The analogy I would take is in the AI analogy, it'll be similar to the ChatGPT moment of 2022, November, I think is when they first put the free version out to download onto a smartphone. So, that's not very far away. So that's from a perspective of a business being able to use it.

Then I think going from there into, what are some of the milestones and signals that we see? We have been very motivated to figure out, how can our partners who want to use it, how can they use it to do problems that are meaningful? Here is an example that the Cleveland Clinic did, and they had a lot of conviction in quantum, so they've been working on it for about three years. But I would say for the first three years, there were a lot of skeptics also, like, is it really going to do anything useful? Earlier this year, they finally were able to model a 12,000 atom protein, a fragment of something called trypsin, on a quantum computer. I think that was a remarkable step, because nobody thought that it would get there that quickly. And the breakthroughs were really around computational chemistry and how do you combine classical supercomputing and quantum together?

Another milestone came when people began to be able to model how plasma flows happen, which can then predict both magnetic properties. So maybe we can find a different material with the same properties as rare earth materials, if we can predict magnetism by computing it, not by discovering a material. So, that was a really important step. And these are both problems that I think classical, I'll say, likely cannot do. I'll say likely because I have a lot of respect for the computational folks as well as math people who can come up with better ways to do things. But at almost six months out, and these two people have not yet found better techniques.

Andrew Ng: I was going to ask about something, this may be very speculative, but people often talk about AI now as transforming businesses and society, and will quantum be the next thing they'll know, maybe. And and I see you're bullish on it, I hope you turn out to be right. I'm marking my calendar to look back in 2029 and see what breakthroughs surprises at that time. And do you think that will be where quantum treats most of the value, physics simulation and materials? Which is important, but then also maybe not as big as AI, which is not to say it's not also big.

Arvind Krishna: So I think to the question you're asking, let's give the audience a way to think about it. Quantum is not going to solve every problem in the world. We should just state that. I think there are four categories so far that we have found where we are pretty comfortable that quantum will have a significant advantage over every other technique. I'm going to use a little bit of a geeky name, but then also try to translate it into language that everybody can understand.

The first category is Hamiltonians. You're right, that's the world of quantum physics and quantum chemistry. So let's call it, it can do computational chemistry. That world you can say, how big is it? The physical world around materials is about $10 trillion. So if we could improve how a chemical plant operates by three or 4%, or if we can take 5% more oil out of an oil well, or if we can compute the properties of a small molecule so we know what its attachment to a protein could be for drug discovery, I'll say that aggregate world is about 10% of the world's GDP.

The second category is partial differential equations. And like you, most people are going to react with, "What the hell are those useful for?" Well, those can predict things like black shoals, they can predict risk, but more importantly, they can predict things like solving the Navier-Stokes equation, which is important for flow in pipes. And so there is a whole world of problems you can solve. So those two, I would say people will say, "Okay, we can see that's the physical world which you just spoke of."

But there is a third category around optimization. So optimization, if it's just linear, can be done easily. The moment you put constraints, it's very hard to do optimization in any effective way. We also have problems like are called by the traveling salesman problem, think of fuel or moving trucks around on roads. Quantum we know can solve constrained linear programs up to a certain scale much better than classical supercomputers. The fourth, it has been shown that quantum can find hidden patterns and data, AKA, applications to AI. I feel that those are going to need bigger quantum computers in the first five years, so that's probably more a category in the middle of the 2030s than early on. But the first three are going to be trackable in the next three years.

Andrew Ng: I like that you said quantum isn't going to solve all the problems in the world, words to that effect. We happen to know AI also won't solve all the problems in the world that I wish more people would just come out and admit that, compared to some of the AGI hype statements that get made.

Arvind Krishna: Yeah. You and I probably agree on that, but we are probably the minority. Why do we expect techniques and technologies that are looking at past data are going to suddenly go way beyond? And that is why I think that there's a lot to be learned, and I think there's a lot of these things. And they're very useful, but they're not silver bullets.

Sarah Elk: That's great. Well, I know we're reaching the end of time. Is there anything that we haven't spoken about that you would love to make sure we cover as we think about advice for CEOs related to AI?

Arvind Krishna: I would sort of hit three quick points, Sarah, if I could. One, I think that the big LLMs are incredibly important and incredibly useful. I would strongly urge everyone to also pay attention to open-weight models, especially run on premise, not just on the cloud, because that allows you to make sure that really critical proprietary IP is not taken away. But also because of geopolitics, people outside the US may worry a lot about where their queries and data is going. So that is one that I would offer up, other than the pure cost advantage that is going to be there.

The second, you touched on already. How do you really unlock your internal data to be well-used by AI, albeit with some governance around it? I think that's really important. And the third is one I mentioned again, have conviction. The technology is useful. I don't think there's any doubts left on that. Have conviction and figure out that's the art of the leader to figure out the two or three areas where AI could be a huge unlock, and put enough resource and enough investment behind those.

Andrew Ng: Hear, hear. And I think one of the things that's hard about choosing those two to three areas is it takes judgment. And when you choose two or three things, you could be wrong. Whereas if you choose 100 things, you're less likely to be wrong. But I think that's why real leadership is so hard and why it really matters.

Sarah Elk: Yeah. I love the discussion of being extremely choice-ful in the focus areas, but then also de-risking it by testing and learning and being adaptable. Having spent a lot of time looking at agile in the late '20 teens and having written a book on that, I don't think a lot of companies live that in practice.

Arvind Krishna: I think leaders find it hard to acknowledge that they're wrong and they have to pivot. And I think that that is why 100% of the market value is driven by 20% of the companies, because those are the ones who are actually willing to pivot, adapt, and move quick. I think acknowledging a mistake in six months is far better than going three years and getting a mediocre result.

Sarah Elk: Well, that's great advice.

Andrew Ng: Hear, hear.

Sarah Elk: Thank you so much for being on the show.

Arvind Krishna: It was a pleasure, Sarah, speaking with both you and Andrew.

Andrew Ng: Thank you, Arvind.