The discussions in the video center around the costs and considerations of integrating AI into businesses, focusing on insights shared by Rebecca Gott, CTO for IBM Power platform, and Penny Madsen of IDC. The dialogue begins with Penny's journey from a technology journalist to analyzing cloud buyer behavior at IDC, highlighting the factors driving AI adoption, such as cloud models and macroeconomic influences. Rebecca shares her experience as a CTO navigating businesses through IT landscape modernizations, where AI frequently plays a key role.

The video delves into the intricacies and hidden costs associated with implementing AI in a business setting. Penny outlines the high expenses associated with infrastructure, skills, and data management, projecting an increase in costs as AI projects scale. Rebecca emphasizes the importance of data governance and the complexity of integrating AI results into backend systems. The discussion covers the essential need for cloud transformations and the widespread requirement for company-wide AI initiatives.

Main takeaways from the video:

Please remember to turn on the CC button to view the subtitles.

Key Vocabularies and Common Phrases:

1. infrastructure [ˈɪnfrəˌstrʌktʃər] - (noun) - The basic physical and organizational structures needed for the operation of a society or enterprise. - Synonyms: (framework, system, organization)

There's a multitude of reasons why this is so expensive, and I don't think it's even as expensive as what it's going to be for a lot of companies going forward as well. We sort of see that at the moment about 1.9 million per company for big companies going forward in 2024, but this increases to 4.1 when we look at what they expect to spend in 2025. And I think you've got to look at it from both the infrastructure viewpoint, what's happening with the applications, also the amount of cultural change and business change that has to take place in the industry, things like skills, for example, which are highly critical and increasingly expensive, the network, the infrastructure, when you're looking at having higher performance compute and then all the business change, and then you've got to think all of this is all based on data and just the cost and being able to work out the data governance, the right storage requirements and get this all into play.

2. macroeconomic [ˌmækroʊˌɛkəˈnɑːmɪk] - (adjective) - Relating to the branch of economics concerned with large-scale factors, such as interest rates and national productivity. - Synonyms: (large-scale, broad, total)

And what external influences, from macroeconomic influences to things like Genai are having on the industry.

3. integration [ˌɪntɪˈɡreɪʃən] - (noun) - The process of combining parts into a whole, ensuring compatibility and coherence. - Synonyms: (combination, unification, merger)

A second thing that we often see is so then projecting of much further, and I'm going to skip a few steps here, another, I'll say, kind of hidden cost that isn't necessarily thought about upfront, is the integration of those results into the backend systems

4. data governance [ˈdætə ˈɡʌvərnəns] - (noun) - The management of data availability, usability, integrity, and security in enterprise systems, based on internal data standards and policies that also control data usage. - Synonyms: (data management, data policy, data administration)

You also have to think about the data governance, you know, basically what are the controls, you know, the privacy permissions around that data.

5. inferencing [ˈɪnfrənsɪŋ] - (noun) - The process of deriving logical conclusions from premises known or assumed to be true using models, especially in AI. - Synonyms: (reasoning, deduction, conclusion-making)

Often building the model, you have the model in place, you're doing inferencing, but once you start getting those inferencing results, how do you integrate that into whatever the backend data system is?

6. pathology [pəˈθɒlədʒi] - (noun) - The study of the causes and effects of disease or injury. - Synonyms: (disease study, diagnosis, pathology research)

We were working with a hospital in Asia Pacific doing pathology. Their inferencing was on essentially digital probes, and it was for essentially object recognition, image classification type of thing.

7. data sovereignty [ˈdætə ˈsɒvərənti] - (noun) - The concept that data is subject to the laws and governance structures within the nation where it is collected. - Synonyms: (data jurisdiction, data law compliance, data territoriality)

And interestingly, we see, for example, companies in western Europe might be a little bit more prepared for some of this because they've dealt with data sovereignty needs, with GDPR and other things.

8. hybrid cloud [ˈhaɪbrɪd klaʊd] - (noun) - A computing environment that combines a public cloud and a private cloud by allowing data and applications to be shared between them. - Synonyms: (mixed cloud, combined cloud, composite cloud)

They haven't set themselves up for hybrid cloud companies want to access a lot of data from where it might be resting, or they might need to get data to somewhere else or use different models.

9. Proof Of Concept (Poc) [pruːf ʌv ˈkɒnsɛpt] - (noun) - A realization of a certain method or idea to demonstrate its feasibility, or a demonstration in principle to verify a concept has practical potential. - Synonyms: (prototype, preliminary model, feasibility test)

It's one thing to step into a POC, and pocs actually can be really great learning vehicles for companies, particularly as they're entering the space to get a handle on a true understanding of the skills that are needed, the type of infrastructure that may be needed.

10. exorbitant [ɪɡˈzɔːbɪtənt] - (adjective) - (of a price or amount charged) unreasonably high. - Synonyms: (excessive, outrageous, exorbitant)

And what we're finding is a lot of companies are then finding out that they don't have the right cloud foundations, they haven't set themselves up for hybrid cloud companies want to access a lot of data from where it might be resting, or they might need to get data to somewhere else or use different models, and they might find that they're then hit with the exorbitant network work costs.

Can you afford gen AI?

A lot of Genai says it's free, but is anything in life really free? So with all of the hype around using AI everywhere, it's worth it to ask, should we be using AI everywhere, and can we afford it? So I wanted to ask two people who are on the front lines of this question, Rebecca Gott and Penny Madsen. Welcome, Rebecca and Penny. Glad to be here, thanks to having me. Great to see you.

Now, Penny, I want to start with you, though. First, can you tell me a little bit about how you got into talking about the cost of AI? It was quite a long road. I started off as a journalist covering technology back when really all the conversations were around ERP and CRM and very little about data centers, and then ended up focusing on data centers and saw the rise of that industry. And I'm now at IDC where my focus is looking at what's happening with the cloud buyer, what challenges they might have, what some, and what's making them adopt different models of cloud, and what external influences, from macroeconomic influences to things like Genai are having on the industry. Oh, wonderful. You've got such a diverse background. I can't wait for us to, like, pick into that today.

Now, Rebecca, what about you? So I'm CTO for IBM Power platform, and so I spend my days working with customers, understanding where they need to take their businesses and where they need to take their it landscapes. And much of that discussion has been around modernization, essentially taking some of their existing applications, their workloads, and taking them to essentially the next level, modernizing them. And that's where often AI enters the picture. So then I know that they always have questions like, Rebecca, do I really need this? Exactly, yes. Yeah.

And as you know, there's so much talk within the industry and just the general public around AI and the use cases around AI. So businesses are no different. Right. But we're gonna get into all of that, Penny, like, I know that you also are somebody who has a ton of conversations with people who've started big AI initiatives point blank. Why is it so expensive? There's a multitude of reasons why this is so expensive, and I don't think it's even as expensive as what it's going to be for a lot of companies going forward as well. We sort of see that at the moment about 1.9 million per company for big companies going forward in 2024, but this increases to 4.1 when we look at what they expect to spend in 2025.

And I think you've got to look at it from both the infrastructure viewpoint, what's happening with the applications, also the amount of cultural change and business change that has to take place in the industry, things like skills, for example, which are highly critical and increasingly expensive, the network, the infrastructure, when you're looking at having higher performance compute and then all the business change, and then you've got to think all of this is all based on data and just the cost and being able to work out the data governance, the right storage requirements and get this all into play. It's not cheap. A lot of companies are only doing this in very small scale at the moment. But I think as we start to scale out, we're going to see costs look very different.

So as you continue to reveal these costs to me, Rebecca, I know that you probably got some good insight when it comes to some hidden costs, some things that maybe are laying within some of those layers that penny just broke down for us. Can you please pull the curtain back even more for us? Absolutely, yes. Yeah. So one thing penny hit on was just the data. And often, you know, when we're working with customers and they have ideas around use cases, one of the first realizations can be, do we even have the data? Do we collect the right data? And that can actually take some amount of time, sometimes quite a long time, to put the right mechanisms in place to actually collect that data.

You also have to think about the data governance, you know, basically what are the controls, you know, the privacy permissions around that data. A second thing that we often see is so then projecting of much further, and I'm going to skip a few steps here, another, I'll say, kind of hidden cost that isn't necessarily thought about upfront, is the integration of those results into the backend systems. Often building the model, you have the model in place, you're doing inferencing, but once you start getting those inferencing results, how do you integrate that into whatever the backend data system is? And that can take some amount of time. And so one example I'll point to is we were working with a hospital in Asia Pacific doing pathology. Their inferencing was on essentially digital probes and it was for essentially object recognition, image classification type of thing.

And so they had the inferencing in place, but then to take those results and actually integrate them into their back end system, a pathology information system, that actually was another journey for them to actually make use of that data. So that integration piece can be a big piece as well. The other thing that we see, which I, I've been doing a lot of work on recently, particularly because I'm looking at what's happening with cloud buyers. There's a lot of organizations might be starting on this and they go, we want a genai project. They then quickly find out that they've got a data governance project, so they've got to go back to square one with data. And interestingly, we see, for example, companies in western Europe might be a little bit more prepared for some of this because they've dealt with data sovereignty needs, with GDPR and other things.

They've looked at their data. A lot of companies then have to get their data into the right shape to do this, and then they need to also factor in the account that actually can our infrastructure enable us to do this? And what we're finding is a lot of companies are then finding out that they don't have the right cloud foundations, they haven't set themselves up for hybrid cloud companies want to access a lot of data from where it might be resting, or they might need to get data to somewhere else or use different models, and they might find that they're then hit with the exorbitant network work costs. So they need to have the right automation and processes in place. So companies are then having to go back and carry out a cloud transformation project to build the foundations for this. And we find three quarters of companies are actually on that journey today and they want to have this done in two years, three years, four years. They're not thinking five years. And it's pretty transformational.

They're looking at changing up to three quarters of any cloud strategy they might have in place already. Okay, first off, I want to take a moment just to thank both of you for keeping it 100% real. Neither of you are painting a fantasy of like, here's how easy it is, here's how you can get AI on the cheap. That's not what this podcast is right now. So thank you for keeping it real there. We've talked a bit about some of these hidden costs and back end costs. And you know, Penny, as you mentioned, you enter the project like with your eyes focused on one specific area. Then you realize, oh, hold on, this is actually a data management thing.

So I want to talk a little bit of, I want to go in reverse and talk about some of the upfront costs, right? So before we even get to some of the hidden ones, what if a company is not prepared for the full upfront costs? Like is there any space that they can move into next or some way that they can help to place them in that position. Like what can you do if you don't think that you can afford it? We have an IBM Institute of Business values that does surveys occasionally, and we did, we did one recent survey, 47% responded that they, they don't have a handle on what to expect for the costs of AI. How much will it cost? They're seeking essentially advice through external services, third parties that have been through these processes with other customers to get a scope on the upfront costs, and then they can step back and work on digesting what they can potentially bite off.

We actually find that there's a pretty high number of projects that we've been tracking that don't end up realizing the true value or return on investment that companies want to get from them. Its early days and thats why were finding most of these projects are being then linked to business KPI's. Theyre having to look at driving consolidation and cutting tech debt wherever they are. And they were already on this journey because of inflationary pressures to make things work better and faster. But now theyve got this added pressure of doing it for Genai, which requires so much more. Theres a lot of organizations that just out there playing around on the licensed models, for example, that might seem cheap to start with, but then very quickly when you look to scale those, they can ramp up in costs and you might find other problems as well.

So I think it's really, there's a difference in companies doing the proof of concepts to getting those into production, and that's when the cost comes in. It's one thing to step into a POC, and pocs actually can be really great learning vehicles for companies, particularly as they're entering the space to get a handle on a true understanding of the skills that are needed, the type of infrastructure that may be needed. And so getting through that POC, actually there can be tremendous, really solid learning, but then taking that next step in terms of, okay, we have this great and sometimes successful POC, but taking it into a production setting is a whole other barrier of entry often. And so that's when you really start to need to understand any regulatory processes or compliance issues, data governance that you need to be mindful of, and security also.

So security often can be an afterthought, so that barrier to production can be a big one. I was talking to a company recently, I was doing a number of interviews with organizations that have been doing genai projects, and it was really interesting because so many of them said, we just had this directive from the c level, really quickly through the it team, go get some genai projects on the table. We want to see things working. And they, in a lot of cases, it was put out to the whole company. What can you do with Genai? Go do what you can.

We've opened up some licenses and nothing happened. Nothing happened until organizations actually formed their own center of excellence around AI or had a team that could actually manage this. Because could you imagine if you had everybody in the company doing their own genai projects? One, lack of efficiency across the whole business, but then lack of the ability to then be able to look at and orchestrate these projects and then opening up the element of risk as well. That might come from doing some of these.

Penny, you said three words that made my ears perk up. You said a lot of words, but three in particular. Now that I want to drill in a little bit more on center of excellence. That sounds like a really cool academy, but I'm sure that that's not exactly what that is in the context of what you shared. Can you tell me more about exactly how a center of excellence works? Yeah. So I actually see this as the exact same thing. Right. In terms of exactly what Penny's saying is that we work with companies and it's so spot on. There's a directive to do ideation.

They say, think of all these great ideas. And so we talked to some teams and different lines of businesses are coming in with some really great ideas. But there has to be some sort of, I'll say, kind of center to guide and lead the efforts. You see chief data and analytics officers being named or even chief AI officers being named to put some organization around it and establish best practices, practices with regard to data governance and tools and processes to get a handle on things. Because what you want to prevent is exactly what Penny was touching on was that this risk of, I'll say, shadow it, where you have the different departments, department tends to have their own funding lines, so you don't want them buying their own infrastructure, kicking off, buying their own licenses, and having this very, I'll say, very different ways of doing things just depending on what team you're working in. And so I think that's actually a really good practice that companies, companies are understanding they need to establish essentially that center of excellency.

Yeah. And I think the lines of communication as well, being able to have that coordination across line of business and cloud, because these Genai projects, they're not just about Genai. There is that underlying infrastructure element here. And considering the impact some of these will have on that, but also what the infrastructure needs to do to be able to accommodate this. So we're actually finding a more and more that we're getting lines of business become involved in cloud decisions, for example, at the preliminary stage, and test and development, for example, where decisions can be made and should be made about where things are housed and located.

Can either of you, or both of you together, can you help me get some best practices in terms of how can a company really drill down and figure out, do we need generative aih, because you both have done a beautiful job of painting a lot of the barriers and a lot of the concerns. But how can one know that this is something that's still worth it for us to enter into? So I think one thought is with generative AI introducing it, either embedding it in your products or introducing it into your back end processes. For developer efficiency, let's say, does that give you a competitive advantage, or do you believe your competitors are pursuing these technologies and will that give them a competitive advantage that then basically you are left behind? Yeah, I think organizations that are doing this really well are focusing first on those projects, for example, that might drive some efficiency and bring them pretty direct small gains and enable them to actually get their head around what's happening here. But I think also a lot of companies are looking at Genai and saying, do we actually need Genai? Maybe we just need AI. You got to remember there are other tools around before Genai.

If I'm an organization that's going to take your advice, penny, and think about perhaps considering do I need Genai right now or is AI enough? Are there resources that have not even been tapped into yet that I could be tapping on into? Seems like that's one way that as an organization, I could help to keep my costs under control. Can both of you kind of give me some more ideas in terms of other ways that companies might be able to keep their costs under control if they know, like you said, rebecca, hey, my competitors over here already leaning on in, I can't afford to not invest, but how can I still do this in a considerate manner? Taking a look at essentially, I think starting small is, is a good approach. What we're seeing is more customers are understanding that AI really is a fit for purpose with regard to infrastructure. And so what can you do with your existing it assets?

Can you start with essentially maybe the compute farm, the servers you have in place, or do you need some very expensive GPU's? Often customers are finding they can start smaller and then learn and then see where they need to go to scale. I think also anybody going down this route needs to think very carefully about what their partners are doing down this route. You need to consider who you're working with, what flexibility, because there's going to be so many innovations take place in this space, what you're using today, there might be something different you require in a year's time. So having that flexibility built in. But flexibility isn't just about, can I access different clouds? Can I access different environments? Can I do things at the edge? It's not just a technical question. It's also about the contracts and the arrangements you might have. It's about where the vendors that you're working with on this are going, because no one vendor is going to be able to meet every need for every organization in this space.

Well, then how often have either of you seen companies right in the midst of making that initial investment? And then, as you mentioned, Penny, there's some sort of industry shift or a technological leap that then kind of makes that initial investment or that initial project that was being worked on kind of obsolete. If you embark on a project and it ends up being valuable, but you don't want to take it further, there's generally still some very good learning behind that that informs essentially where you want to go next. And so I tend not to think that things are just a throwaway.

And it was, I'll say, a wasted effort because there is learning and understanding of, okay, we're going in this direction, but maybe we need to shift a little bit and go in this direction because, hey, there's this new tool or technology. And as Penny mentioned, the whole ISV and the ecosystem, there's so much new things and services coming out enabled through our system, integrators, the partners you work with, and just having that flexibility, companies need to have that in mind. I guess the one constant for a lot of organizations is that nothing is constant anymore, and we don't know what's going to be around the corner. And Genai kind of brings us to the, it's the technical version of that, I guess, whereas just adding to those pressures. But we can't forget that there's other macroeconomic things happening behind the scenes. There's a certain political element to some of this. There's definitely also regulatory element that we don't know what's going to change there. Once again, it's a moving space.

And to protect themselves, businesses really should set themselves up so they can think on their feet. So if I'm a business that's listening right now, then I've heard both Rebecca and Penny. I've heard both of you basically say, though, that AI is absolutely something that can help businesses to compete. You just gotta figure out what is the right AI or Gen AI approach for yourselves. But if I'm a business that's now discovered, okay, so I need to do this. What would you tell me in order to get me to actually foot the bill, because I've still now got to write that check in order to make the investment. What would you say to help, to put me at ease and or to inspire me, at a very minimum, understanding what the technology can do for you.

Start that education that can be some initial investment in terms of basically just the learning and understanding possible use cases that have a return on investment that you find attractive. And so I think looking at some of that as very, I'll say, just opening steps can then lead to the next step. To put it in context, if I had to say to somebody, you need to be investing in this today. A lot of the proofs of concepts that I've seen become production successfully. The returns I've seen or heard about have all been around that 50% figure. So companies saying we're seeing 50% better efficiencies, we're seeing 50% savings, we're getting 50% more back from efforts we're putting in in terms of revenue.

So there's definite advantages there if you can get it right. But I think, think about the foundations, because even if you do one project, you want to be able to replicate that, or you want to be able to scale that project, you want to be able to and turn it around and do things quickly. Take that knowledge that you've built up, and the only way you're going to get there is by investing. Okay, I like that. And I like that 50% number too. That's quite compelling.

Now let's look at the other side of the coin. What am I risking if I don't take the leap? What are companies potentially going to miss out on if they don't take the leap into investing in AI? I think competitive advantage. And we do see many, many companies looking at some of their back end processes and efficiencies. So what can I do to enable Genai or just AI tools that add to my developer efficiencies and that can really pay dividends if you're able to get your products to market faster than competitors, because you have put in efficiencies into your backend development processes that actually can be a winning game and lead to true competitive advantage. So I also think there's going to be, you mentioned skills, and the reason why skills are so expensive is it's really competitive for those skills at the moment. And if you're not doing something around this, the next generation of people coming up looking at where they're going to be working are going to be wondering why, and you're going to find yourself becoming more and more of a laggard with some of these industry adoption areas.

Well, look, I'm very inspired right now, and on behalf of our listeners, I'm just going to assume that they're also really inspired because they've listened this long. So thank you all for listening. But Rebecca, Penny, thank you both for being here today, for sharing insights, for being so generous with this discussion. I really appreciate it. And everyone who is listening, everyone who's watching, thank you as well. If you happen to have any additional thoughts about this episode, please leave them in the comments and we will see you next time. Promise.

Artificial Intelligence, Technology, Business, Ai Adoption, Data Management, Infrastructure Challenges, Ibm Technology