Chief Content Officer,
Techstrong Group


In this Digital CxO Leadership Insights Series video, Mike Vizard talks to Diveplane CEO Mike Capps about what makes weeding out bias in AI models so challenging.



Mike Vizard: Hello and welcome to the latest edition of the Digital CxO Leadership Insights video series. I am your host Mike Vizard. Today we’re talking with Dr. Michael Capps, who is CEO for Diveplane. And we’re talking about AI and ethics and everything that goes with that. Mike, welcome to the show.

Mike Capps: Hey, it’s a pleasure, can’t wait.

Mike Vizard: We’ve been talking about this issue, now, for several years, and it’s not clear to me that everybody fundamentally understands the issue. I mean, it’s not like people have ethics; they may decide that they even don’t want to adhere to certain principles because of whatever viewpoint they have. And not everybody has the same viewpoint about the same thing. So, how do we navigate this issue? Is it really about ethics? Or is it about visibility and how AI is applied, and people having some insights as to what some of the biases are in the whole process?

Mike Capps: Well, you cheated and answered the question for me in the question. ‘Cause, I mean, ethics is based in morality, and morality is cultural, right? So the notion of you shouldn’t be allowed to do, say, a social credit score like they do in China, I abhor the idea, the EU is absolutely against it, but who am I to say they can’t do it? The issue is that we want to know whether it’s happening, and be able to apply, with transparency, know what’s going on inside it. If we are a culture, here in the United States, that doesn’t think you should be able to bias decisions based on, say, racial origin, okay, fine, we should make sure that we can look and make sure that that isn’t happening, right? Simple as that.

Mike Vizard: So what is the challenge in getting that level of insight and visibility? What’s required and what do we need to make that a standard approach for everybody to easily implement?

Mike Capps: I wish it were easy, right, it would be great if it were easy. First of all, you know, imagine you train an AI with real-world data and it’s great data. Unfortunately, some of the real-world history and great history that we have involves decisions that we wouldn’t make today. So if you train your AI based on, say, 60 years of loan decisions here in North Carolina, you’d probably see some racial bias; you’d definitely see gender bias. So that’s, like, problem one is just, even if you build it perfectly, you might be training it with something that you don’t want to continue. But the real issue here is that neural networks are so complicated that we can’t understand what’s going on inside them.

It’s as simple as that: humans can’t look at it and inspect. It doesn’t matter if the code is opensource. _____ is opensource; doesn’t matter. My code is opensource. If you wanna look at my genetic code and guess what I’m gonna have for dinner, good luck, right? There’s this big difference between being able to see the code and understand why it’s doing what it’s doing, in a transparent manner. And that is the most popular technique, right now, is neural networks, and they are just not built for human understanding.

Mike Vizard: It seems like there’s a lot of interest, in various governments around the world, in this issue, but it’s not clear to me that they understand the core technology problem themselves. So, are we in danger of maybe creating some overly-prescriptive regulations that would be counterproductive? Or how do we have an intelligent conversation about where government’s role in this is?

Mike Capps: I mean, it’s a hard problem and there’s a lot of well-meaning folks in government that just don’t have the technical background. I think one of the tricks, you know, I just mentioned opensource, well, there are some very well-meaning regulators who are thinking that we need to make sure AI is opensource, in order to make sure it’s trustworthy. There’s no real correlation between that. You can have an opensource system with 160 billion decision nodes, which is more than I have in my head; I can’t possibly understand what’s going on. But they’re legislating opensource, which is great for ad-supported companies who give away their technology.

I can’t afford to give away and opensource everything that I have. And so, it’s almost anticompetitive to – and it’s regulators trying to do the best thing they can. I hope to see, you know, the new, say, AI Act in the EU has a lot of notions of, “We’ll change this as time goes,” but as soon as you put something down on paper and law, it’s really hard to adjust to a changing technology field. And that’s what I worry about, ’cause we’re building new techniques that are more explainable, more transparent, and the last thing we wanna do is codify something by mistake.

Mike Vizard: Seems like there’s a fine line between trying to optimize for a certain result and what you might consider a hidden bias. So do we need some of these AI models to be validated by third-parties, to say, “Hey, at the very least, there wasn’t something that somebody inadvertently wound up putting their thumb on the proverbial AI scale and tilting it one way for a particular use case, only to discover that there are ethical complications”?

Mike Capps: Well, I would say it’s a good start to have a third-party validate that the approach was right. You know, we just worked on this great process in the Data and Trust Alliance, which I am delighted to talk about; it’s a number of companies that are focused on data and trust. And the issue isn’t simply, “Can I validate this model with ex post facto testing?” it’s, “Is this a mission-based organization that’s constantly throughout the process, from soup to nuts, thinking about bias issues?” Because in the end, how would you verify that a credit score decision wasn’t racially biased? We would have to have a massive test set, and then even if, for example, someone who’s in a disadvantaged class doesn’t get the job, it might’ve been the right decision, and that’s okay. But how do you know it was the right decision?

It’s really, really hard, again, to look at a black-box system and decide, “Well, it doesn’t seem to be making too many, say, gender-biased decisions, so therefore, it probably isn’t gender-biased.” Well, you just haven’t caught the fact that it’s gender-biased in some small region of the model, because you haven’t tested thoroughly, right? That is the problem with the black box, you can’t know if it’s right, so therefore, the only way to test it is perfectly, which is not possible.

Mike Vizard: Is there some way to get at this by looking at the data before we create the AI model, to make sure that it isn’t reflective of some, to your previous example, moment in history where there was a bias in the process that resulted in the way the data was collected, and the data is just a reflection of that bias?

Mike Capps: I mean, absolutely, right? You know, I like to joke that AI is like coffee, and it matters, like, where were the beans sourced, and was that done in a way that was sustainable, all the way through are you underpaying your barista and not allowing them to organize a union to get better workplace protections, right? The whole thing has to be clean, so, yes, we’ve gotta start with data that’s good, but it doesn’t mean that you can’t learn poorly. I think, you know, maybe American politics is a great example of everybody looking at the same data and coming up with very different decision models based on that data. So, for sure, starting with real scientific information and data is a great way to start. It doesn’t mean that you’ll learn properly.

And, you know, we’ve seen black boxes that were, for example, a malignant tumor detector that had perfect data in front of it, and it learned and had a perfect result whenever it would look at a new patient, as long as the odd patients were malignant and the even ones were benign. Because that’s what the training set was. It was a clean direct training set, but it just learned shallowly off of this dataset, and so, on a brand-new person, it would be 50-50 right if it wasn’t already pre-categorized that way. So it’s entirely possible to have useful good data and then have a black box learn the entirely wrong thing off of it, and it’s happened and, sadly, those systems have been fielded. So it’s step one, right? Good data, not biased data, absolutely, but it’s just one step of the process.

Mike Vizard: So, people trust the AI models? Because today there’s a longstanding issue in IT where we don’t always trust the data because the business executives that created the data know how flawed it is. And then when I [crosstalk] with an analytics app that says, “Here is this outcome that’s gonna come,” they gently nod their head and smile, but they know that the data used to create the analytics was spotty, at best. So the question becomes, will they come to the same conclusion about AI models and we’ll have some sort of backlash?

Mike Capps: I hope so, right, and I think you see that. You know, when I’m in sort of the top financial enterprises, they have fantastic world-class data functions, and they just don’t use it in critical situations. Now, if you’re trying to optimize advertising spend on the one to three cents range in Facebook targeting, yeah, sure, AI is fine. But if you’re trying to make a hundred-million-dollar decision using an AI model and saying, “Well, the AI told me so. If I’m risking the whole business or my career, who cares?” nah, they’re not doing it at all. And I think why we are building explainable transparent AI is all about that notion of conviction, like, it doesn’t do me any good to give you a prediction without some notion of how convicted can you be about that prediction. If you don’t have that, then what’s the point of it? Then it’s just tea leaves.

Mike Vizard: Do we maybe need to think more about training the data scientists in the ethical use of AI? And maybe this is a moral issue, but it comes down to the people, and we could maybe certify them as being ethical AI advocates and proponents, is that something we should be looking at?

Mike Capps: You know, “certify” is a big word, but that notion of having everyone throughout the process who’s touching data have a real understanding of what can go wrong is critical. And that goes all the way up to the executive level, right? The last thing you want is you’ve got this Oracle class of data scientists with hoods on and they’re slaughtering the pig and bringing the entrails and saying, “This means we should, you know, purchase this new company,” right? And the CEO nods and says, “All right, well, they told me so.” You don’t want that. You want, all the way through the process, data scientists saying, “Here are the biases built into this data. We didn’t have this information. We have this. This is why we think this is useful. Here is the range in which it’s useful.”

And being very intelligent about the way they communicate that information up the chain into decision-making, whether that’s healthcare or, you know, weapons target, whatever it is, needs to be some notion of, “How did we get this? What’s the provenance? How effective is this?” And that, like you say, requires training and a little bit of, I hope, mission and ethics.

Mike Vizard: You mentioned this alliance. Maybe you can describe the mission for that a little bit more? But more importantly, are you looking for folks to join you? And what kind of folks?

Mike Capps: Sure. Data Trust Alliance is, we were lucky to be a cofounder. It was started by Sam Palmisano, who was a longtime CEO and chairman of IBM. And it’s a CEO-led alliance with the CEOs of Walmart, Starbucks, I get to talk football with Roger Goodell ’cause he is a member, American Express, Mastercard, about 20 global enterprises, and weird little Diveplane because we do a lot in the space and we’re glad to be a part of that. Our first effort was tackling AI bias in workplace, and for workplace systems, we built sort of a series of evaluations that procurement officials could do when looking at AI technology to decide, “Is this something that we’re willing to buy?” And then, the whole goal, here, is let’s share it with every enterprise out there, let’s share it with government, and we had a number of government stakeholders and others, and then share it with providers.

So if you’re going to sell, right now, who to promote or who to hire software into the organizations I just mentioned, you have to go through this auditing process. You have to be able to prove what you’re doing about bias, right now, and they won’t allow black-box models in many of those decision processes. My opinion, at least, is the quickest way to change a business is to have all the buyers say what they need, and then suddenly, all those workplace vendors have started adopting, I think, really ethical approaches to what they’re doing. I’m not sure that we’re looking for new members, we’ve got a few, you know, Johnson & Johnson, CVS, others that have kind of come in over time, but it’s more about rolling out the findings.

We’re working on data quality. We just finished one on how to evaluate a purchase of a company, and, you know, whether it’s a startup or whatever else, what are the ways you can do due diligence on a company to make sure you’re not accidentally swallowing a poison pill. Where, you know, a lot of startups play fast and loose with data, you know, buy it where they shouldn’t and use it in ways they weren’t allowed to, in order to get quick results. And the last thing a vaunted organization like Johnson & Johnson want to do is have that be within their risk profile. But I hope that all leads to transformation in the way startups and small tech companies are using data, when they realize there’s no exit unless you do it right.

Mike Vizard: Do you think people realize that this is more or less a continuous process? ‘Cause when you first think about it, everybody’s, like, “Well, we’ll just take care of that as we build the AI model,” but the AI model is subject to drift and the data can change. And as someone once said, “It’s one thing to be wrong, it’s another thing to be wrong at scale,” so how do we kind of figure this whole thing out?

Mike Capps: Yes. Yes, no, that’s the problem with trading on, say, gender-biased data and then scaling it, right, it’s the last thing you wanna do. No, it absolutely is, and, you know, there’s two parts to that. You brought up drift, and that’s a big issue. We’ve got a model monitoring approach, and that’s something I’d like to see more in general is, once you deploy a model, you wanna monitor that it still works. And healthcare is the perfect example. You know, you buy a fantastic model that’s built at Harvard on fantastic wonderful data pre-Covid, and then take it into your hospital today and apply it. Well, it may not work, anymore, because the people who are walking through that door are a completely different health profile.

Is that model still accurate? Tough to say. You need to monitor to make sure that’s the right approach. But in general, yes, this is going to be a never-stopping problem, and it’s just like security on the Internet, right? We’re not gonna solve this one. There will always be a new worse problem that can come up, and we’ve got to have a mix of hygiene and education and sort of aggressive defense against the problems that can arise, and just keep getting better and better. ‘Cause AI is gonna keep getting stronger and stronger, right? We’ve just gotta be ready for it being used anywhere, and be aware of what could go wrong. [Crosstalk] teach kids.

Mike Vizard: So what’s your best advice to folks? We have a lot of digital CxOs out there, and a lot of them are enthusiastic about AI and may not understand exactly what they’re getting involved in. But, you know, should they all go take a course at MIT, or is there some other way to get and kind of educate?

Mike Capps: Well, I mean, you know, the course at Stanford is a famous one for getting up to speed on machine learning techniques. You know, it’s absolutely evolving. I think just coming in with skepticism about, “Where’s the data coming from? How are we maintaining that data? Who can touch that data in the future?” And then, the next side of it is, “How is this model built?” and then just use it accordingly, right? There’s nothing, like I said, wrong with using advertising optimization or, let’s say, cohort churn in your videogame, like, you know, “Oh, Mike’s not enjoying this videogame so much, he might quit tomorrow.” Using a black-box AI to figure that out and offer you a sale? Why not. Like, I have no problem with that whatsoever.

The issue is entirely in sort of these high-concern decisions about access to public resources, jobs, healthcare, and the like, that we really need to be smart about it. There’s no great answer, yet. I look forward to us having more explainable AI and machine learning techniques that are available for problems at scale. You know, like I said, we’re building one in the structured data realm, but there’s nothing. If you wanna do image analysis, facial recognition, and do it in a way that’s safe and correct and unbiased, that’s transparent, doesn’t exist, there’s nothing like that right now. That’s all based on neural nets, and the creator of neural nets says it’s a dead evolutionary pathway. The trick is, how do we have some of these other techniques catch up to this massive effort of hardware and compute and giant organizations building, as fast as they can, these black-box solutions.

Mike Vizard: Should people hire somebody into the role of what might be called the AI skeptic, and their whole job is just to kind of look at all this stuff with a jaundiced eye and say, “Hey, what’s real here and what’s not and what might get us in trouble”?

Mike Capps: That’s a neat idea. There are certainly some consulting firms that are doing that, but I would say that anyone who’s in the AI and data space that isn’t an ethics czar and a skeptics czar, like, all of them should be, right? It’s just that general notion. It’s kind of like, anyone who touches my network better understand what phishing is, you know, right? Like, you don’t get to just have someone in your CISO’s office who understands phishing; it’s gotta be everyone who touches e-mail in your entire organization. I think it’s the same way, when you’re talking about data, AI, and making decisions from that, everyone needs to have sort of that level of literacy. Unfortunately, but that’s the new normal, right?

Mike Vizard: All right, so no matter how smart that machine is, if it sounds too good to be true, it still probably isn’t. [Crosstalk]

Mike Capps: No such thing as a free lunch, sir.

Mike Vizard: Thank you all for watching this latest episode of our show. You can find this one on the website, along with our others. We invite you to check them all out. And once again, thanks for spending time with the two Mikes.

Mike Capps: Hey, I’ve enjoyed it, thank you.

Modernizing Digital Banking Services

Step 1 of 7

What are the primary business drivers for your organization to modernize its digital banking services? (Select up to three)(Required)