S4 | E23 | Moving Fast Without Breaking Trust: The Reality of AI Governance with Rowan Stewart @ Transcend

Welcome to the Think Data podcast brought to you in partnership with Mydataworks. If you want to stay up to date with the latest breakthroughs and trends in the world of data and artificial intelligence, and if you're curious about some of the strategies that companies and founders use to launch data and AI products, then you're in the right place. Our aim is to bring together a diverse lineup of fantastic guests from the founders, through to accomplished leaders and product owners at some of the most fascinating data and AI companies worldwide. They will each offer you their own unique insight into what it takes to launch and scale a great data business. Thanks for tuning in and I hope you enjoy the episode. Welcome back to the Think Data podcast and on today's show I welcome Rowan Stewart. She is the AI and data safety product leader at Transcend. Before joining Transcend, Rowan worked at a pocket of BCG where she saw firsthand what happened when companies rushed AI products to the market without the right governance and data foundations to support them. Today, she helps companies including Robinhood, Brex and Groupon build AI products that are not only innovative, but also trustworthy. compliant and built to scale. As AI continues to move at this incredible pace, governance often feels like something businesses promise to tackle later. But the question here is what are the real risks of taking that approach and how can startups continue to move quickly without creating problems they'll have to deal with and solve further down the line. Rowan, it's really good to have you on today. And your background is kind of not necessarily the most typical. background from where you are now because I noticed you've spent what around six six and a half years with BCGX which I know we're like the the technology incubator arm of Boston Consulting so you left them you joined Transcend what a couple of years back so I was really keen to kind of I suppose start from the beginning to kind of what led you to be in this space because the role you hold now at Transcend in probably the hottest space in AI right now all around kind of AI governance risk appliance. I could talk for hours about this. So I'm really excited to dig on this. But starting back at the beginning, you know, why did you choose Transcend and what led you to this point?

Yeah, yeah. So I mean, even right out of college, I ended up joining a startup where I worked on natural language processing algorithms. This was before transform models were even a thing, if you can believe it, before those. First papers from Google even came out for LLMs. And so I'd been in the kind of AI space, although we called it machine learning back then, for a long time. And when I joined BCG, that became my specialty. I would work with kind of Fortune 500 companies and a lot of different sectors, but especially in finance to stand up kind of AI. initiatives and do builds for them. And one of the things I noticed very quickly is that it's really hard to build fast and build right. You know, it's really, really difficult to balance between proving out the value of the technology and doing it in a way that is safe and governed. And I found over and over again, having this experience with so many companies where they struggled to hit that balance that I was very interested in making, I guess you might say, like the safe and ethical choice, the easy business choice as well. And so when I left BCG, I really wanted to explore that space. I worked on a workshop with Aspen Policy Institute, which is a foremost kind of AI think tank, amongst other things, about what AI governance means. I really wanted to explore that space because I'd been on the other side so many times.

Yeah, it's interesting because it's coming from that technical background or that you understand that space. It kind of puts you in such a different position now. And I think when you look at Transcend specifically, what attracted you to them? Because I know I touched on the beginning of the AI governance piece and how there's so many companies doing it wrong. And actually where you're sitting is in that real kind of almost that firefighter mode. But what appealed you to Transcend and what is your role for those guys?

Yeah, so I joined Transcend with an eye towards actually building out our AI governance capability. You know, historically Transcend's focus has really been on data privacy and ensuring that data was used in accordance with laws like GDPR, etc. And that gave a really, really strong technical infrastructural foundation for understanding how to govern complex, context-heavy decisions. For instance, in order to decide whether you can use some customer's data, you have to know Alex Hutchings in San Francisco right now, but he's visiting Paris. And so that makes him subject to GDPR. But his IP address from his device says this. and the purpose that we want to use his data for. is marketing, but he's opted into analytics only, et cetera. So all of these different things make those kind of decisions very hard to resolve. And I felt like there was a strong basis there of that is going to be one of the hardest problems to crack in AI. Everyone who has ever worked in machine learning or AI knows that data is 80% of the battle. And so I really wanted to take that. and push it into like the AI world. And that is really my role at Transcend now as a senior director of product is to take this like deep infrastructural knowledge we have about how to permission data dynamically and bring it into the world of agents and AI where it is most needed because they're the ones who are going to be gobbling up so, so much of the data.

Yeah, and on the flip side, creating a lot of it as well, right? Yeah. like the sheer... scale you know being in data analytics for 15 or so years and exponentially now what we've seen with all of our clients and just generally on the amount of data that's being created is creating a lot of issues and it's really interesting on that point about a cross-border piece because it's something you don't really necessarily think about you know you're an international company you've got workers or even customers based in say Germany which have very strict GDPR rules however in the US loosely speaking it's quite it's like the wild west a bit so How do you approach this? Because so many companies are getting it wrong, you know, so from you in your role, what are you approaching first and foremost? Or what should companies be approaching first?

Yeah, I think there are a couple of common, I would say, you know, sand traps that you can get yourself into with AI governance. I think one is the incentives are misaligned. You, as a large company, a lot of folks are saying, hey, you need to like adopt AI. And I want to see, you know, X percent increase in your KPIs. You know, you need to sell more or serve customers faster, et cetera, by adopting this technology. And at the same time, they're turning around to their, you know, IT teams often. It's often not even like compliance and legal and saying and make sure that they don't do anything that messes anything up. So then you've actually pitted your own company against itself. Right. So you have people whose jobs depend on actually adopting this. technology and people whose jobs depend on making them not adopt this technology. And that just like really, really doesn't make for an effective, you know, governance setup because you're always having your people fight against each other. So I think one thing that you need to get right is having a really clear eyed view of how do you want to balance business risk and business value when you're adopting technology. like this. And then I would say another sand trap that's more specific and more technical is a lot of folks are saying, okay, you know what? We're going to just give the agents all of the rights of the person who's associated with them. So you, Alex, maybe have a, you know, assistant in like a cloud desktop or a chat GPT or cursor or whatever, and it just gets all of your rights and privileges. But here's the thing is that I don't actually trust Alex's agent as much as I trust Alex. Right. And I shouldn't. And, you know, for instance, if I gave an agent to the CEO of our company, he has so many permissions that would be disastrous. No agent should be able to run around and act on his behalf with as much of that leeway. And then there's another problem that comes up on the back end of it, which is if you have turnover of your staff and all of your agents are actually really associated with one person and bound to those credentials, then you have a big problem. So I think we need to move towards breaking this association between a single person is the owner of a single agent who then takes on all of their credentials. And as I said earlier, we need to think a little bit harder about how we're incentivizing the folks at our company. and make that trade-off between business value and business risk. really, really clear in a collaboration rather than just pitting people against each other.

Yeah, it's a fascinating point you touched on because we've seen the rise of this role called the AI transformation lead, which ultimately is for these whole scale transformation initiatives where it's probably one of the most exciting roles. It's like a poison chalice because you're going into an organization and saying, look, let me learn what you're doing. But ultimately, the goal is to try and avoid people picking themselves up against each other but the end goal let's be honest is either cross-saving or an efficiency so when you're a start so say you're a startup would you say it's easier for a company to implement this put the guardrails in place get the government's process right from ground zero i.e a startup versus a much larger organization that's being told to change because of ai and how are they two different challenges or can you still with the same brush stroke. solve the problem?

Yeah, I think they have a lot in common, but the challenges of, I mean, at BCGI, I would be working with extremely large organizations and they do inherently have different challenges than very small ones. I think very small organizations are under tremendous, very direct market pressure. And so they have to, you know, respond to the needs of the business extremely quickly because they're often trying to find that product market fit and they can't, you know, take on the weight of a heavy compliance process. So the way that they often handle this moment of trade-off is saying like, screw it. Like, I don't know if you've like been in a claw and you're just like, oh my God, yes, allow, allow, allow, allow, just like, oh, you know, just do it. And I think that's where a lot of those folks end up and Then you get, you know, splashed across the headlines, some, you know, crazy incident that's happened or what have you. And then on the really, really large enterprise, especially if they're not, you know, the large tech folks, the response is often just to choke it out entirely. Like you're rolling it out so slowly that your employees are like, I'm just going to like put it in my, you know, personal chat GPT and, you know, it'll help me do my job and hit my KPIs and like no one needs no. about it. But on the surface, their actual like large scale implementations are extremely slow and just like stuck in that pilot purgatory. So while I, you know, very conservative, I think they often break towards the we're going to just go very high risk and the enterprises break towards the we're going to go very safety first. And I think a system, what they need to do is the same thing of... finding that right balance and really making that choice going closer to the optimal point between those two things instead of just saying shoot because i'm a startup i'm going to go way over here or shoot because i'm a large enterprise i'm going to go way over here yeah

yeah it's really interesting isn't it because also it's the risk appetite isn't it it's that risk how quickly you move and you might have one of those kind of uh founders who's like i sold it we're going to go for it but then equally You've got those companies that want to do the AI transformation, but they're moving so slowly by the time they deploy, the technology's already moved so far ahead. Yeah, exactly. That was a waste of time. In your opinion, then, from AI governance, if you look at, you touched on it right at the beginning about GDPR, what's your honest views around, you know, whether it was global or regionally about AI governance from a... state level from a federal level where they actually are saying these are the guardrails these are the rules because it seems for me at the moment there isn't necessarily a in some way you refer to and go like gdpr for example these are the rules if you break them yeah yeah yeah what

where do how close do you think we are to getting that i mean i think i mean we have the uai act we have tons of legislation coming out from a number of different states i think the thing that's going to happen in the boardrooms and does happen in the boardrooms of large enterprises, is that they'll think, okay, yes, this is coming. And they'll either say, okay, well, we're just not going to do anything until the rules have, you know, really solidified and our guardrails are clear. But it's also equally clear that then they are under huge threat from, you know, others who are willing to take that risk, etc. To be totally frank with you, I think the question is less going to be Thank you. compliance with a certain regulation and more going to be literally the business risk you take on. It's not necessarily that, you know, you're afraid an auditor is going to come find you. It's that, you know, your agent breaks out of its sandbox and breaks into Hugging Face and creates like a massive PR. issue for you. Or you're the Facebook director of privacy, or I'm not sure what it was, who's sprinting back to her desk because her email server is being deleted. Those are business concerns, whether or not there are regulations backing them up and whether or not there are auditors with actual budget to do the auditing thing. So I would guess that it will run ahead because these are real business risks. even outside of them being regulatory slash compliance risks.

Yeah, I think it's a really salient point. It all ties into this kind of cliche about responsible AI, doesn't it? It's like, what does that ultimately mean? And it's such a buzzword, you're hearing out responsible AI this. But in your experience of Transcend, obviously, as a pocket of Boston who's consulting with huge organizations, in your eyes, what is responsible AI? Because actually, you made something really important there. It's less maybe about the rules and regulations above, but actually what guardrails we're putting for these AI agents themselves to prevent them from having the next PR disaster for you.

Yeah, I think responsible AI, I mean, there have been so many like words for this. Is it like ethical AI, responsible AI, AI governance, etc. And I think a lot of it boils down to the same mechanics of what you want to be able to do. And it's almost a rhetorical point of where you want to motivate people from that really differentiates the terms. Like, I would say responsible AI if I wanted to have a conversation, you know, about ethics. And I might say AI governance if I wanted to have a conversation about business risk. But at the same, at the end of the day, a lot of those mechanics are the same mechanics of if... an agent is attempting to do X action, am I able to prevent that? So I think responsible AI, if I had to give it a definition, would be how you build and implement AI systems in a way that is pro-humanity, broadly speaking, whether you... take that as, you know, does not include bias in the data sets to does not make decisions about individuals, which is like a common piece of something like the EU AI framework, or, you know, whether you even think about it as like, is it displacing jobs or making, you know, the roles of human workers better or worse? And then on the AI governance side, a lot of those same concerns are there through a different lens, but a lot of the mechanics of I need to be able to inspect the, you know, payloads of any request or I need for PII or I need to be able to suspend an agent immediately in its runtime if I don't like what it's doing and I consider it dangerous. A lot of those mechanics are the same and it's just really the motivation behind them that is the difference between like what we like to call like AI governance and what we like to call responsible AI.

Yeah, that's pretty clear. And I think if you, going back to what you said earlier about you've got your, you've got your two sides of the coin here, you've got your slow moving, risk averse, you know, chief AI officer or CEO of a large Fortune 500 who's, who's bringing in AI because they've got XYZ projects to deliver versus the cavalier founder who's shipping AI, you know like tomorrow You've got two very different profiles there. And obviously two different consequences if they get it wrong. But from your broad view, what is the actual consequence of getting this wrong? Because at the moment, I'm not sure I'm aware of any legislation and laws, but is there a such, can people be penalized? Is there kind of potential, you know, fines, prison centers? What actually happens if they get this really wrong?

So the EU AI Act is... technically enforce, but that does not mean that it is enforced, right? So there are plenty of regulations across various states and federally and internationally that have opinions about AI, but it's really a question of whether those regulations are backed up with the budgets to do the audits that would be required in order to actually enact those penalties. And I would say It's less about a fine. For most of the large companies anyway, the fine is almost inconsequential in its size. It's more about the restrictions that are put on how they operate afterwards. that are more significant and cumbersome and or the PR that comes from it. So I would say, yes, absolutely, there are laws in force. And hypothetically, those laws can come with penalties of X amount, but the monetary penalties are not going to be as important as the bad press or as important as the requirements that come with being called out.

Yeah, interesting. And especially if they spend a lot of time, money, resources, in turning to them to be told, actually, you're not allowed to do this, then they're like, oh, what do we do next? We've got all these stakeholders. Yeah,

it's disastrous, right? Like you've poured so much, so much into these, you know, initiatives. And this is true, whether you're a large company or a small company, that's a, that's potentially a career ending, right? If you have spent that many resources on something that then you have to scrap.

Yeah.

So it is a very worthwhile investment to think it through. But I would just say, you know, one of the things that we're building right now is because we've encountered this ourselves, a system that helps you encode those policies and enforce them across your kind of AI agent landscape, but makes it very clear the tradeoffs between business value and business risk that you are taking by surfacing things like this agent. has been stopped for, you know, 40% of its actions, that means that you are losing value. You should inspect those things and say, yes, that is worth it for me in terms of risk, or no, actually, we need to consider tweaking this and the converse of, you know, if maybe a policy is constantly blocking tons and tons of activity across your organization, maybe that is a hint that either people need to be reeducated. about what their agents should be used to do or that your policies are too strict, you know?

Yeah.

And you need to be able to adapt. You really do. Like the years, days, millennia of agent like policy being just like a series of papers that gets dusty on a shelf is just not going to cut it.

No. No, and on that point then, when companies are looking at this kind of, I know you use AI transformations, they're going to broad a term here. Is it important that they put these guardrails in, the governance, the policies, or they look to put those in before they go down the rabbit hole? Or is it almost like they have to deploy and then they put the guardrails around things based on what they're seeing in terms of feedback and output from these agents? Is it kind of, is that the way around it? Or should they always look and say... How are we going to control it? This is how we deploy it.

So I think the reality for most companies is that there's so much red tape, if you're very concerned about that, that if you haven't even proved the value of what you're trying to build and you're already circumscribing and putting a heavy compliance burden on that project, you're dead in the water. You need to at least show that there's something worth building, right?

Interesting. Okay.

And so I think the answer is being able to either A, prove what value there is extremely quickly, which is just very, very hard, or B, massively reduce the burden of compliance for these initiatives. And so that's part of why I think AI governance will need to be kind of this like encoded living thing, because if it's a super heavy process. that is very burdensome for the teams trying to stand something up, then you've choked innovation before it even has a chance, right? But if it's a kind of a living thing that's just like the water in which you exist and, you know, you have policies that can be differentiated between like, this is early stage, it's this late stage, this is like full deployment, then you stop having the experience that I think a lot of people, a lot of enterprises and companies have of saying, okay, Thank you. okay we finally like got this thing and it's showing a lot of promise and we're thinking of going live to a bigger thing you know and then compliance comes in and it's just like we don't want to be the bad guys but we're going to have to be the bad guys because you guys have not built this so instead of having to like reconstruct at the choke point of going to production release all of this compliance that then kills a lot of projects honestly or you know traps them in pilot Purgatory. Like having this continuously encoded setup where those projects can evolve and go through those checkpoints much more easily, I think is the answer to how enterprises are going to be able to. actually adopt and actually ship instead of have yet another cool sandbox demo that goes nowhere.

Yeah, it's so articulately put. And I know from first-hand experience talking to some of the companies we recruited for how many projects have been shelved at this stage, because great idea, they built the team, they've got the reviews, everyone's excited and never makes it to production, never makes it to production. And actually these companies are so excited. But I'd imagine a lot of that is because internally... They've not either brought the right stakeholders in at the early stage. They've not scoped it out. And then they've got to the end and gone, actually, we can't do that because of this policy. Or actually, have you considered the impact over here? And actually, it's the excitable CEO, right?

Exactly. And then it just like takes the wind right out of your sails. And I've been on the other side of that so many times that, you know, we'll be working with an innovation group and we'll be building something amazing. and then we just can't get through that final piece where it actually goes to, you know, like wide deployment, which is where the true like business value would actually be seen. And it just, it drove me so absolutely crazy that this is like, that's why Transcend was so exciting to me because I was like, oh, wow, this is an intersection of like where you want to do the right thing, but it's really hard, you know, either from a business like risk perspective or like a pure compliance perspective or. you know, even an ethical perspective. Like, why is it so hard to do the right thing?

Yeah, exactly. And there's also, if you've got the appetite, that kind of experimentation culture, and you start to stifle that, then it's very hard to bounce back. And people are inside the engineers, the product owners, they're kind of, this is just, it's not going anywhere. So when you generally, and from your experience with Transcend, obviously before, what do you think is next for this kind of broader term AI governance specifically, where do you see this space evolving and some of the potential challenges we might be coming up against?

Yeah, I mean, I think you're going to see a lot like on in terms of the market, I think you're going to see a shift from here's my personal agent that I like chat with all day to kind of service level agents that work much more autonomously. I think that is just like the way the world is going. And we're going to see a lot of challenges, as I mentioned, around trying to reconstruct our idea of permissioning and governing much more autonomous agents. So I think that's one thing is we're going to get more autonomous and it's going to go badly for a while until we all figure that out. I think we are also going to see a rise of kind of this role of chief AI officer, I think like... Jensen from NVIDIA called this out a while back of like every company will have a chief agentic AI officer like soon enough. I think we will have kind of specialists who are focused on finding that balance between business risk and business reward for agents. And for governance per se, I think we're going to. see the rise of a lot of kind of like MCP gateway type solutions, you know, like the run layers or mint MCPs of the world, which will be really, really useful, but won't be enough. I think we will try to say, okay, if I can just restrict what tools my agents have access to, that is going to be kind of the end all be all of my agent governance. And we're going to find that there's a lot more. that goes into governing agents, like access, you know, controlling their access to memory, you know, controlling their ability to even perform inference on it, inspecting payloads, both inbound and outbound for PII, you know, which is kind of transcends core competence, etc. Like there's so much more than just tools. But I think we're going into a phase right now where everyone's like, okay, MCP Gateway, I get that. All agents go through the MCP and then that will be kind of a final piece. So those are my kind of three predictions for you.

Yeah, it's exciting. I feel like we could go on forever because I think we've seen this kind of, I've never seen a space evolve as quickly as we can, but it's almost like the train has left the station, right? I think for a lot of companies now, they're like, okay, we need to probably rein this back in a bit. We need to, I feel like we've hit this. level where yes the innovation is still exponentially growing every day but I almost think companies now are trying to slow things down slightly just to make sure that they have things in govern the right way they have the right frameworks on your point the fluid policies they've got everything ready to avoid that pr disaster i like that word fluid yeah exactly yeah it's it's yeah no adaptive responsive you can't you can't realistically hope to be

right about every policy you're no defining right now you just have to be adaptive and that can feel a little bit like keeping up with the joneses which is exhausting i think we've all I felt the tool burn out. but there has to be a way.

Yeah, I think the tool burnout's really not kind of a final point to finish on. It's like there's so, you've got to be aware and people listen to this, be mindful. The next shiny tool doesn't necessarily solve the next problem. It's actually, I know myself, I was like, download this, use this. And I was like, I'm just getting nothing done. It's kind of, you know, I've stripped it back so much now. I'm like, okay, I feel comfortable with everything I've built. I'm not happy with it. But you probably see yourself with some customers you speak to.

Yeah, yeah. I think a lot of customers we've talked to are like, man, I feel like I'm building on sand here.

Yeah.

Stuff changes so fast. And, you know, these frontier model laboratories are releasing something like literally every day, multiple things every day. And it's exhausting. It's really, really hard to work against. And I think the answer can't be like, okay, well, we're just going to like... put a pin in it and just like no just say we're we're never gonna like we're not gonna look at anything new for the next three months because inevitably there will be something new but building and architecting your systems in such a way that they can tolerate having a much more composable composable setup so that when the next thing comes along it's not so painful to cut over

100 well look i'm i'm really you appreciative of your time this morning. So what you and Transcender doing, your background's fascinating anyway, but I think what Transcender doing, the fact you found that kind of niche, the home where you get to be at the forefront of what I genuinely think is probably the next hottest, or probably the current most hottest pocket of AI right now. So people listening here, check Rowan out, check Transcender out. But yeah, thanks for coming on this morning.

Of course. Thank you for having me.

Pleasure. Thanks so much.