How Can Non-Engineering Teams Adopt AI at Scale?

This transcript is based on the episode’s English auto-captions and has been formatted for readability. Please allow for occasional transcription errors in names, acronyms and specialised terms.

[00:00:00]

This is also a technology front that you put uh against your prospects and your customers because the ones the the the marketing and the sales teams going to uh be facing first with the with the customers and you need to show them that you are an AI first company and I think it's very important to every business unit to have this uh leader or what how we call it it's a a business outcome that push the whole business unit to be much more identify and and measure the the ROI and adoption and create the different um solutions and automations within the business unit and and most of the the processes today have too much decision makers, too much processes, too much people involved. you have the ability when you're using AI to really condense that and to have like less and less and less decision makers uh on the process. I think that the the the main proxy of understanding AI economics or AI ROI it's maybe uh revenue per employee but then uh it's much more harder to measure that uh within business unit that doesn't have like direct impact on the revenue. Uh so it's it's very important to let people try and create stuff and come up with ideas but when you create something that you want the company to use in scale you need to take an ownership somebody need to take an ownership on that and and make sure that uh you build a practice on top of this tool. AI builders need to be a product with some technical sense and they need to put within the business unit not outside the the engineering that creating the solutions or the change of or the identify of the workflows and and and creating the agent need to be sit uh in the same area with the same people and in the business unit. In the end there is a lot of similarities between humans and agents but uh humans have this psychology layer that I don't think have agent have right and a lot of the time most of the the the transformation success is people Hello and welcome to the enterprise tech talk podcast. I am your host Sumitra Kalikar. Now the topic of today's conversation focuses on a specific challenge that most of the organizations are facing today and that is while AI is now becoming widely available um it is still not deeply embedded in the organization workflows in the day-to-day uh work that is carried out within the various business operations and while organizations are running various AI experiments um for the business teams such as marketing sales, HR, finance and legal.

[00:03:16]

The AI experimentation itself is not a problem. The hard problem is making AI more trustworthy, more governable, more reputable and measurable. That's the real challenge. And to help me unpack this important topic, I'm joined today by Ohad Rogan. Ohad is um AI lead transformation leader at AMDOCS where he is helping various business teams embed AI into their day-to-day business operations. Oh also brings um almost decade plus experience in startup building and currently works at the intersection of uh enterprise AI automation uh product leadership and digital data labor. Oh welcome to the podcast. Good to have you here. Yeah, thank you to having me. Pleasure. >> Well, before we get into the topic, um would you mind provide uh a brief overview about your professional journey? Um your experience with startups and how you landed into the world of enterprise.

[00:04:22]

>> Yeah. So, um I'm on entrepreneurship uh since I remember myself uh even before school. Uh I studied computer science and entrepreneurship and then I had my own startup for five and a half years. Uh we were a tech store company. Uh we raised a million and a half uh dollars and we tried to do different stuff on the AI. Um we had a product for uh that understand user behavior and then we moved to create the QA AI engineer that fully automated and it was much before open AAI had uh CHPT or um then after they launched the CHP we had something uh that can control your browser much before open AI operator. So we have a lot of experience on how to create different AI solutions. Um then I moved to be an AI lead in a a multi-level marketing company that created a sales enablement tool very consumer focused for one and a year. So I I really learned the the bits and the bites when you're creating something that every user should uh use which uh provide me a lot of experience when you using or you serving uh such type of non-engineering population uh that you need to be very specific and create something broad that everybody going to use and then I moved to to AMDox uh to lead basically the end transformation for the non-engineering with a amazing coord Right. Okay. That's good. Um let's start with our uh with the enterprise context.

[00:06:01]

Um look um when in uh most air conversations these days they revolve around um uh improving the u the the developer productivity, the developer experience uh bringing the agent AI practices into software engineering etc. And which is important right absolutely important but the but the fact remains that in the large corporate environments the day-to-day operations happen outside engineering right in finance legal HR and so on >> and you have been working closely with those teams. So I wanted to start with a with a kind of overview question to you. Um in your experience working with these teams, how do you see what are your observations? Uh what kind of challenges surface working with those teams which potentially may not be the challenges or there are different challenges compared to say engineering teams.

[00:07:01]

>> So um it's a very good question. Uh I think that um we put very very large population that are not the same in the same bucket. This is how we are looking at like the non-engineering and when we talk about engineering most of the flows are very you know well known you have the CI/CD pipeline and and you you know how you do a um software development life cycle it's pretty much the same in every place you go but finance and HR are two different uh things uh >> and you try to to provide some methodology to those uh non-engineering type of population which are not necessarily match with one another. So you need to create something else to create practice to create uh understanding to be able to automate and identify uh the flows and and it's it's all about like to change the mindset uh how you create new things how you automate things so I think it's a it's a different approach when you come to those organizations because also that doesn't have like the you know engineering conversation although today as I see that the the line is is very the gap between being engineering and non or non-engineering is very um you know it's it's it's very vague today you can be very you can create beautiful things and I think the the most important thing that you need to provide those population is the confidence to create something uh because when they understand that they can build and get and they try and they have the you know the enablement uh they they're doing really really amazing stuff because they understand the business much more than the engineering uh that they need to understand what they develop and then to develop. They really understand the business really understand the outcome that they want to create within the company. So you just need to put with them the confidence provide them with the right enablement uh um and the abstraction on top of the the tools that they already have um and amazing stuff will be happen. And I want to add on top of what you're saying regarding to uh the the denial engineering why it's so important this is also a technology front that you put uh against your prospects and your customers because the ones the the the marketing and the sales teams going to uh be facing first with the with the customers and you need to show them that you are an AI first company. this is the first team or people that you're going to uh meet in the journey to have kind of a you know relationship with uh with Amdox or any company that you sell service or product to.

[00:09:55]

>> Yeah. Yeah. And I think you mentioned this but I want to emphasize that not necessarily all these business operations are same when it comes to AI uh understanding. First thing is um across HR, finance, legal, customer support etc. Their risk appetites could be different. Their workloads definitely are difficult right and their uh I would also say potentially their literacy around AI could also be different. Um so uh in your experience do you see some some of these particular operations are more progressive in terms of looking forward for adoption while some other operations are more cautious on a >> I think it's very depend on the the leader and then the persona within this uh business unit let's uh they're ones that willing to take more risk and try things even I don't want to say against the company policy but they try to understand what they can do they push the system to change right uh they they move into changing the risk appetite they talking with the right people they pushing it to be much more identified um and other business units there is no leader to take an ownership on on this transformation information and I think it's very important to every business unit to have this uh leader or what how we call it it's a a business outcome lead that push the whole business unit to be much more identify and and measure the the ROI and adoption and create the different um solutions and automations within the business unit and we build a framework how to do that but uh it's very depend on on on the persona and the risk appetite and and and the finance like the the token consumption is right is the most um like painful uh chain in this process uh which is which is taking a lot of time to a big corporate or enterprise to deal with.

[00:12:12]

>> Yeah. Yeah. And um the other thing I wanted to unpack with you was um the potentially difference between someone say uh um having tool access within a within a within a business operation AI tools access versus the the real adoption of AI within that operation. And what I'm trying to say is that yes, let's say a finance operation or a nature operation might claim that >> 50% of their uh employ their workforce now is regularly using AI tools. Um but but does that really mean that yeah has been well adopted within that operation or how do you draw about differentiate that and how do you say um the the business that operation has really adopted well uh yeah into the into their so tools adoption is a good proxy right we measure that as well but I think that we need to defer the the the tools Right?

[00:13:22]

Because every tools has a different evaluation scope of of what capable with and and the market move very very rapidly. So my matrix is that uh I have like four different uh uh pillars of tools and they differ into personal and centralized. So we have the productivity tools. Personal can be uh chpt co-work cloud co-work copilot co-work all those kind of tools right and then you have the centralized which can be uh agents that you created in copilot studio or other like solution that you can take vertical AI that help you uh to do your job but it's centralized when you something change in those tools everybody get the benefit it's not something that you configure it to yourself then we move to autonomous agent right autonomous agent agents um is basically much more advanced capabilities uh agents that do do the work for you. So if you look on personal side of things, you have Grockbot, you have um scout by Microsoft, you have those kind of tools, maybe even open claw, but it's yours. You configure it, it's it's helping you to do your job better. And you have autonomous agent that on uh centralized which is uh autopilot or open claw infrastructure that governed by the enterprise to create job for you and they have like uh you know email and teams and they work with you right we're not there by the way and then there is another pillar which called uh I called agentive systems that you put in place uh in your personal life it's more than one agent you have few agents that talks with each other to do a real job and in a centralized place is much more like a a cohesive agentic system that work with you. It's not just one agent is few agents that work to do a a very specific domain work and we have more tool that for building stuff for our AI builders.

[00:15:20]

This is how we call them AI domain builders which is cursor and others. Uh so there we have on the personal the cursor the this the tool and they have like more harnesses and stuff like that that enable a more centralized governance on top of the the development for the the builders. It's not for the uh uh software development life cycle. So you can measure that adoption different in the different stages to understand where you're at. But also you can measure the the way of work the change of the of the names and the transformation. So for example in our uh uh methodology that we put within Amdox for the non-engineering which we call the business process life cycle or the business uh uh process value. We uh guided them to add AI domain builders. So we can measure how AI domain builders you have in your business unit. And we have the outcome lead, the AI outcome lead like I said that need to to create kind of a a uh road map for what he need to automate a success criteria. How we going to how much he going to save to the company? If it's a you know stop working with third party vendors or uh um like save hiring or stuff like that then he execute on this plan and can showcase the ROI he created uh in inside the business unit. we add that into our KPIs that we are measuring and of course there is much more like uh efficiency and and uh more broader KPIs but that are much much harder to to measure when you talk about this kind of population because it doesn't have like a task that you can measure that relevant to the all population so you need to do that differently >> yeah I want to come back to that in more details a bit around um how to measure AI value because sometimes you start with um simple KPIs like AI consumption, usage, license consumption, the very simple transactional KPIs but they do not reflect the real value that this AI is giving to you right but but I want to come back to that >> before we go there um one question I I have is around um the the struggle the struggle many organizations and operations in terms of moving from pilot phase, experimentation phase into operationalizing su something successfully and um and many organizations operations struggle there. uh and I was looking at one of the McKenzie's um research paper of called 2025 state of AI which also says the same thing that in most most of the times >> pilots do not necessarily progress production successful productions at least. >> Mhm. I wanted to ask that question to you as to have you seen this challenge um in your work as well and if yes what are your key common patterns what are the key observations here as to why these pilots do not translate into good production use cases.

[00:18:40]

>> Mhm. So yeah, it's an issue. Uh we see that we see that uh on our side as well. But I think it's it's come to a three different uh three big main issues that uh lead to that. One is that um it's it's the ownership, right? Somebody build something that is very very cool and very beautiful but then after he created that he understood understand that he need to maintain that and nobody want to maintain a big product because it's not part of his uh work or scope. Uh so it's it's very important to let people try and create stuff and come up with ideas but when you create something that you want the company to use in scale you need to take an ownership. Somebody need to take an ownership on that and and make sure that uh you build a practice on top of this tool for every product. Uh it can be a cool demo but when you want to you know even a third party application if you want to uh have it in your side on your organization and you want to buy it and and you need to do the adoption you need to make sure people adapt and create a practice around the tool. It's the same when you create something internally. So the the main issue is is the the ownership the long-term ownership uh that maintain this product. The the second thing is uh on the building phase you're build to see the the immediate outcome or the immediate solution and you don't think about the scale. So you doesn't have this practice to to build right to build in long-term in mind uh because you think very narrow and and the tools uh you're using are very very good in provide you what you want very quickly cursor uh cloud code those harnesses and somebody that doesn't have the required uh uh you know u understanding on how to create product at last will not provide this specification when they building the product uh which lead in the end to a product that doesn't scale even and if you if uh you want it and the the third uh issue is um like the the the practice you create around like I I mentioned it the the around the adoption it's not just the ownership it's like how you adapt how you create things that people can can take uh to their day-to-day workflow those and this change of of behavior on on people it's something that is very very hard to do like think about like people that work on Excel for 30 years right 30 years they work on Excel now you provide them a portion of what they can do on Excel in in another tool maybe much more capable on on providing much more faster what you need but a lot of other uh behaviors and and solutions still using this uh Excel sheet and now you ask him don't use one tool use 30. It's not going to happen.

[00:21:51]

>> It's not going to happen. Uh so you need to be very uh cautious when you look on those three things when you're creating the solutions and and take that in account. I think this uh the supporting point I wanted to discuss then with you and probably that is where where you are heading on when you said that that third point in your argument is uh I mean AI has uh I mean you might want sometimes organizations or business operations might just introduce AI and try to automate the existing processes themselves as they are right um and yeah and and using the opportunity to missing the opportunity to really look step step back and see is this the correct way of doing the things or now with these tools should we now re-engineer the processes as well right and and that that's potentially is the missed opportunity many times is is that what you have observed as well >> yeah amazing amazing amazing uh uh question or observation and also Gartner talking about that uh right now AI provided the opportunity to rethink things. Um, and and most of the the processes today have too much decision makers, too much processes, too much people involved. you have the ability when you're using AI to really condense that and to have like less and less and less decision makers uh on the process and most of the time people really uh or companies or enterprises goes to let's look on the current process and let's find ways to automate certain places on those uh on this uh process instead of saying how we can radically change this process. how we can radically move from I don't know 10 steps to one step and a human in the loop. Um and this is need to be the conversation. This is need to be the mindset. Uh think how you can radically change the way you operate.

[00:23:54]

And it's much more harder for an enterprise with a lot of governance and a lot of uh uh bureaucracy than a new startup that uh coming up today from still that uh establishing in day one as a AI native company creating the age and creating the infrastructure and then growth to change an enterprise is much more harder. Um and and his mindset is very very very important because uh I always say that today's startup can run very very quickly and very fast and the gap between like to be an AI uh first company to a startup and and an enterprise I don't know in three years let's say and the technology progress so much fast that to the AI native company it will be very easy to adapt and the enterprise just will uh increase the gap So in one year from now if we not do any radically changes uh uh moves uh so one years from now the gap between AI first startup and enterprise company will be six year gap because the the technology progress so so much fast. So we need to change our mindset and we need to take much more risk and to understand how we radically change things uh in a cautious way in a governant way but we need to talk and discuss about those things and it's I believe it's a it's a huge risk for for enterprises and a huge opportunity for startup in this uh market uh time.

[00:25:25]

>> Yeah. I um just to build on the the the this the business adoption paths for different operations with this legal, HR, finance uh etc. um what has been your experience in how um because the reason I'm asking this is um many times business teams do not necessarily understand AI well enough as as the technical people would do uh and they would rely on let's say internal engineering teams or they would rely on the vendors who are selling something to them. >> Mhm. Um in your observation uh is there something engineering teams or vendors miss or do not understand about the operations where they uh to basically effectively provide them a proper pathway or solutions. The reason I'm asking is if you take a a um let's say legal department they would be more cautious of um uh the privacy of the data and so on right when if you talk about pilots they will be more focused on the auditing uh aspect of the of the of the pilots data etc. So different business operations they are different focus areas uh and they and I'm not sure so that with the engineering teams or vendors really provide very tailored solutions to their business needs.

[00:26:52]

>> Yeah. So first there is like also some engineering's bias. I think today I talk with a lot of engineers that uh doesn't have also mindset the AI mindset they should have because they do things in the old way right so also the engineering need to to change because sometimes engineering believing that they can do a product work and they can do QA work but product cannot do an engineering work and I really challenge this assumption. Um second this is why we um we move to uh uh the understanding or the decision that AI builders need to be a product with some technical sense and they need to put within the business unit not outside the the engineering that creating the solutions or the change of or the identify of the workflows and and and creating the agent need to be sit uh in the same area with the same people as in the business unit.

[00:27:55]

Uh it's not a service that you consume. This is how it was like back then. We have the IT department and the IT department had like uh you know uh developers and they was oversee on each VU and I believe that in order to understand the business you need to sit next to the people that do the things understand it deeply. So I believe or I want to believe that the the builders that we put within uh legal or finance will have the same focus on the same interest and the same understanding in I don't know four 6 months as the legal and the finance person they they really understand the bits and the bites because they are in the same area of the of of those people. I really believe that your location is is mattered and and provide a lot of of of who you are and the value that you can perceive and also provide based on where you're at.

[00:28:57]

If you're a long distance, it doesn't have the same focus and the same interest like your peers. uh if they are long distance from you, it's very important to have the same you know relationship and and then they these developers which are builders which is kind of product developersish they will have this narrow focus or vertical understanding based on where they sit and live. Um one thing uh because you said uh something that triggered my thoughts to be honest. Uh when you said you product product um owners of product builders cannot you cannot be the good engineers. Um and I wanted to unpack that a bit more. Um because where um some many organizations are heading and you you would very much acknowledge that is >> when there is a operating model which is uh supporting or encouraging more autonomous functions right um uh autonomous teams uh whether it's a business teams etc. uh the they that that culture encourages or or enables those teams to make more decisions and make make implement their own tools as well to to support their specific needs.

[00:30:15]

What AI is doing on top of it is making those things probably a bit more simpler for them is easier for them to build no new fix solutions workflows etc. uh you can have cursor rolled out in your organization and with little bit of training um uh of anyone any I mean I will not say anyone but with some training a business person in finance a business person in in legal or they can build their own silo simple solutions. So my question is uh the probably twofold questions. So one is uh do you see a risk in ter testing your argument that product builders cannot be good engineers is that um they do you see they still need to build uh rely on internal engineering teams uh to to build a products and but second uh followup argument is is there a risk that this is now heading towards more of a what we typically used to call a shadow IT uh aspect were different or operations now have many technology solutions they are building uh and not necessarily having a good control across the technology landscape as to what lu are emerging how they relate to each other what whether operation teams have understood the broader other aspects like security privacy aspects uh about those solutions what's your take on that >> yeah let's let's unpack it So what I'm I said is that developers uh doesn't have the the the belief that product can be developers. I believe that the gap is very uh you know it's it's become uh less and less uh notable and we will have much more solutions that will wrap everything and will enable people to um you know to to do the job in a a more govern and and secure uh way. But for sure right now we're not there. Like the the technology is still not there. Uh there is a lot of things that need to be done on on the enterprise level to to make it happen. A lot of enablement to those people. Uh it can be cursor rule. It could be much more like dedicated IDE for them to develop CI/CD that wrap and and devel deployment ability to much more you know to a centralized place and not to create this shadow because exactly how you say when you move from uh centralized place that you're creating solutions and move people to the business unit you basically create uh a tradeoff you pick people that will understand very good the business but you're creating a lot of risk when you do that and you need to manage this risk and you need to understand this trade-off basically it's right for any decision that uh you're doing but um when we we understood that we created something that will support this business unit that we call it the how uh it's if somebody want to read about that this is the habok model but in the hub we're basically creating a road map for creating ating tool for enable the the the spokes to work as expected and for example when they develop something they develop something on the GitHub and the Confluence and we can monitor that you understand they reach out to the COE and can talk about what they going to build and create uh and understand uh the the IT perspective of it they need to register the app when they do the deployment they they have the only approved deployment uh uh server that they can use in order to deploy their app in order other in the organization to use it. They need to going through different agents that will look on what they're build in order to improve and make sure it's governed and and uh it's hinged the the company policy. So we're creating all this enablement around those uh AI domain builders to make sure that we doesn't have bridge and we doesn't have like shadow at right now I must say because we doesn't have that we we or we have part of it we have a lot of shadow at we have a lot of people that they have the ability or get carouser or other tools develop things and creating a lot of bridges to the company and when you do not provide a sufficient solution uh to those people you're creating just g this is exactly the the Spotify use case right before Spotify a lot of people had a lot of uh pirate music you download that from emu kaza a lot of but when you pay subscription that is as is reasonable for a streaming solution that work well you don't need to download music you use Spotify so the IT need to create their own Spotify in order people to consume that and not create a shadow IT like we have today.

[00:35:25]

>> Yeah, interesting take on that. Um and I wanted to unpack this uh from the angle of the governance and operating model uh around AI and I think you you mentioned few things around it already. Um and you talked about uh hover spoke model. Um uh what I wanted to understand there is when you have a sport model where the central team has certain kind of responsibilities potentially making sure there's a consistency in some policies role et is there a uh I figured out what accountability should be with folks with with the restrictive teams and what should be what is the final balance between what accountability should sit within the hub central team versus the the operation teams. >> Yeah, it's it's a good question that I'm not sure that I have like a good answer to. Um because the balance is very different between what these folks want to build, right? Sometimes they come with a very complex things that uh required a lot of enablement from the hub side and uh or very deep knowledge understanding on different uh AI solution or architects. So you kind of say maybe it's need to be developed on the hub side and to maintain on the hub side which is it basically uh because we're creating the hab within it um and sometimes 100% of the ownership and accountability can be inside of the spoke and the process and the communication between the hub and and the spoke is is very important >> um and to learn as as as as as we go.

[00:37:20]

But as I see it, there is one thing that is very very important for both of the sides to be very agile and to be able to create and and delete things and to have like a mindset of uh of moving fast like we we can try things very fast and we can ditch things very fast and delete them and archive them. Uh but we need to have this ability this muscle to move fast and change. uh most of the time what I'm seeing a lot of a big enterprise is to uh double down on a on a project for a long long time and the technology is shifting and changing and it's really hard to let it go and sometimes you you put like the pain and the value you um potentially going to see it's the same but you're too much into the technicality and you need the fast to optim optimize to the value very very quickly and if you understand that you have a new technology that you can uh build very quickly in order to get them to get to the value higher value on on or faster to the value you need to ditch what you're you're doing and moving in another direction and it's okay and it this is the communication I want to see between these folks and the hub uh have this conversation and and understanding how they can move and get to the value very quickly And I believe that the way we're going to get there is to have those AI domain builders within the spokes within the business units which are have the capabilities and their 100% uh role right it's 100% uh uh time of their of of their job and they have the capability to talk with the same terminologically and and understanding with the hab in order to have this good conversation. If we have somebody that is very doesn't have like any AI uh background or experience and and didn't build anything and we put that role on top of him and say to him now you're AI domain builder this conversation will be harm right um and and I think in order that these roles and responsibility to work well we need fresh to have this conversation to work well >> yeah and uh in if if we uh um take that whole operating model discussion a bit little bit at higher level at executive level where where do you see the overall ownership of AI strategy or road map for the organization should sit and there have there are examples where it sits under it and CIO there are examples where it sits under the product team product business there are examples where there is a standalone separate AI function and they have a chief AI officer u so all these scenarios are there out in the market what what is what are your thoughts of >> um I can say what we have in anocks but I think it's like a transformation what we had uh in the end uh a lot of things happened that put us in this position so and and I'm not sure this is the right way right but I think that when you talk about AI transformation or AI strategy.

[00:40:46]

Uh you have your technology front, right? This is uh what we're are doing the AI transformation. How you create an AI first company. Uh it's true for the SDLC. It's it's true for uh business processes and and and the non-engineering side of things. But you also have how you create uh uh product for customers. If you're selling something, if you're selling product, how this what is the strategy for the product to be much more identified and how it could be competing the market which change very rapidly and uh there is big vendors and low margin there is the strategy layer is is very involve a lot a lot of different things um and I think it's it's need to be separate organization under the the chief AI officer uh which maybe in the future gonna be not AI maybe gonna be quantum and and stuff but the the the the this strategy function that need to be talking with the CEO and basically make sure that we have strategy for all those different areas and to be able to measure that the the the old ship going to this direction uh in in any one of the verticals and uh this is how I I think it's it's it's need to be in in Amdoc uh I'm part of the A3 team which is a separate team that under uh uh same executives that lead the IT and security and we kind of a strategy enablement arm that working with IT and security in order to move things to the direction that we want. We are separating into SDLC and nonSDLC. The SDLC are overseas all the engineering and the transformation for the engineering and uh the non-engineering like what we're talking here it's it's regarding to the finance, HR, marketing, sales, all those kind of uh department. This is how Amdox is building and um it's always changing.

[00:43:03]

I think uh you know maybe we'll talk in few months from now it's it's not going to be like that. >> Uh yeah we are chief data officers now we are chiefs officers and if you are correct your prediction is correct five years down the line and we might achieve point of officers. So >> yeah, >> right. Um I wanted to combine our >> maybe also uh physical AI officers because maybe we're going to switch the AI uh the digital AI in some physical AI. >> Uh so yeah, it's always you have like the the frontier technology that shifting or disrupting the market and you need to deal with that. >> I agree with that. Look um I wanted to come back now on the the the how to measure the value of AI which we had started a conversation and you said you are uh MDOX is going through or has gone through transformation journey around AI and for any business I'm not just just taking that MDOX example but for any business which starts on those kind of transformations you would I would imagine start with very simple metrics as I said AI the coverage across the business user users um then uh the licensing consumptions or token consumptions what there could be some transactional kind of metrics you start with uh but real that that as I said that's not necessarily the real metrics you want to continue to report on when you try to measure the value um from your experience what kind of value focus u measures or metrics have been most effective in your trans in the So we started a little background like we had like a form that we sent to the employees to understand how much they using AI. Uh and we measured we start mainly from the SDLC. We measured you know how much people use cursor, how much line of code was uh u uh committed by AI. H we have some partnership with Stanford regarding to AI velocity and quality uh adoption for sure cursor adoption uh we we look on those kind of uh kind of numbers and when we put the way of work of transformation we start to measure like the change of the the roles and and stuff like that. how much like of the the units was transformed to the way uh to the way of work that we uh we establish and I think that it it's not sufficient right I think that we need to start discussing discussing about uh AI economics uh which involve basically understanding our why of AI because ROI of AI is is much more complex than uh measuring those proxies.

[00:46:15]

I think that the the the main proxy of understanding AI economics or AI ROI it's maybe uh revenue per employee but then uh it's much more harder to measure that uh within business unit that doesn't have like direct impact on the revenue. So uh it's can be much more border um like measurement but then you need to do some complex uh calculation when you try to measure every business unit. Uh but is a good very good proxy uh revenue per employee because it's it see it it seems it says that you're much more efficient and and you do more with the same people you have and maybe you can also have less people in the long run uh and and it says something about your um ability to to run but it it doesn't say nothing about the AI consumption right which is a big huge topic for a big company I'm 23,000 employees and now if every employee consump I don't know $1,000 per month uh with AI tokens it says a lot and everybody uses very differently and the ROI comes from or the collective or the personal ability to use those tools there's a lot of things that affect of of this and I really like an open AI article that uh of the their CFO they're trying to break that trying to break uh uh this ability to measure not everything right now but to take a task and then to optimize on this task for example if I create a P&L in finance how much time it's it's taking me to create a P&L how much AI cost is provide me like then to create a kind of a unit economics around this uh uh task and to optimize it right and then I can say okay if before the AI took took me uh 10 hours. Now on uh if I put $10 to uh AI which is uh uh it provide me to create this P&L in two hours right when when I move to do thinking I can start putting ROI on tasks and then to accumulate that but I didn't see any good solution in scale to really to to measure those. So this is why we try to provide the the accountability and responsibility to these uh uh this unit that we put within the business unit that this person that we call it outcome lead business outcome lead. He need to put those those task.

[00:49:03]

He need to put those uh uh ROI he want to create and he need to measure everything. It mean the return on investment. how much AI put what it saved him in hours in in money and then we can accumulated all different tables or data that we get from the business unit from the outcome lead to one unified uh um kind of AI economic ROI and saying we put that kind of money this is what expected this is what we get um uh and and trying to improve this uh um like the the baseline right to improve the baseline. Yeah, this is a bit of complex calculation and when I ask this question to anyone, I mean I I mean first thing comes to our mind is uh what you just said uh how we can measure the individual task productivity and then we accumulate that and that's an important dimension. Um but it tells us the one aspect of our benefit which is around productivity uh the cost aspect right.

[00:50:17]

Um the the broader business benefits could be definitely productivity but what about the acquisitions new revenue model revenue generation and other things right >> uh retention of customers and so on and so forth >> and um may I'm not sure and maybe that's a a bit more complex calculation to understand how we directly supports um new customer acquisitions or customer retentions etc. Right. >> Um yeah, I think that when you put uh uh this ROI table within for example uh sales or marketing, you will see user acquisition. Although when you talk about user acquisition stuff like that, there is much more than one >> uh business involved, right? It's it's much broader uh uh much broader process and uh because I'm very focused on internal transformation uh I want to see this change and not like external but it's very very important to measure those as well and I always saying that for me I believe that we need to you know to have more employees >> because it means that we have much more business because in the end you will need also you know people that are communicating with other people that buying your product. it's not going to be AI uh because you need to create this trust and a lot of so you want to be able to have like more employees but it means that you have much more revenue right it says that uh uh and and the conversation not always need to be about how how you become much productivity productive and and you have less people uh it's not the right mindset and and I 100% agree about that >> and and because you focus more on on the internal AI uh enablement transformation information. Um the one thing that is uh typically talked about for when it comes to the internal adoption is uh within every business operation you will have multiple um a the digital level the the the uh AI agent as new AI agent as new employees new type of employees and potentially every person as an employee would have potentially two 10 different AI it does working with alongside him and do you see that's kind of operating model uh or business operation structure that is how we are getting um and um >> yeah I want to get your thoughts on that >> uh and leave it uh I have like uh 19 agents that uh by the way this the all the conversation you had with me on the email was with my agents uh they set up this call, they set up the the podcast, they send everything.

[00:53:13]

>> Uh I Yeah, I just, you know, I just like uh um read read everything before and uh um the community manager within Aldox is AI agent. She lead the community. She talked with the people all the cursor approval for the engineering leading by AI that uh talk with the people, enforce the policy. I 100% believe in that. I don't see a lot of people have the same system as I have. Uh so I think that there is still very very long way to go for a production system to to run as as what I uh running on my computer uh with those agents working together from marketing to uh uh orchestration and operation and uh context management in in the level that uh my my system uh able to do. But we have like a huge project with the Microsoft to creating what we called Adele project. It's Amdoc's digital employee lightning. The first AI uh employee that will run and work with people with Amdoc. already happen in in uh in Microsoft on top of their autopilot uh infrastructure which provide you with the 365 uh agent 365s that basically govern and have the observability on top of the agent and give them uh uh exactly in like a an employee somebody to report to. They have their own email. They have their own teams. They part of the workforce. And I really believe that we are going there. And then the the the next phase will be how you design not just uh you know a specific role that do the job for you in a good way. Uh how you create a system around it. Uh not just like one digital employee that report to a human. How you uh create a team of agents that report to an agent that report to a uh to a human that really do much more complex work that um need the the the intelligence of different scopes to work together into one shared goal. It's not one one goal for an employee. It's a shared goal for the for the system.

[00:55:34]

>> Yeah. Look, as we wrap up now, uh I wanted to uh ask you our final question and that is given you have been part of an AI transformation journey at at dogs. Are there any lessons that you have learned in that process that you want to share with any organization who might be just thinking on that kind of transformation now they want to be at early stage? a lot. Um from where where I where I can start um in the end there is a lot of similarities between humans and agents but uh humans have this psychology layer that I don't think have agent have >> right and a lot of the time most of the the the transformation success is people >> and and I think we start with that and maybe we close with that as the one that leads, the one that take the ownership, uh the ability of the enterprise to take risk, the the the number of uh decision maker in the process, the mindset, the confidence, all those kind of things are uh from the same place that it's it's human fear and human behavior and psychology and a lot of polit politics within the enterprise.

[00:57:01]

You need to manage that much more than the technology. The technology is already there. The technology is already there. You need to find the people, the right people. You need to find the right processes. You need to um uh to deal with this complex human uh uh in the loop uh uh situation in order to transformation to work. And sometime it's much more harder than than the technology than to put the the technology in in the people hands. We talk about you know the practice the adaption it's all come to the this uh human behavior and psychology. And I think this is the number one to understand when you are running transformation in that uh scale. >> So essentially the traditional wisdom prevails that at the end of the day it's a people problem and people need to own it. >> Yeah. 100%. >> Right. Okay. Yeah 100%. >> So thanks on that note. Thanks. Thanks for your time. It was a fantastic conversation. A lot many insights I'm pretty sure for our audience. Um so thanks for time. It was really pleasure hosting you. If you found this discussion valuable, please follow and subscribe to Enterprise Tech Talk and thanks for listening. I look forward to seeing you in the next