CONTRIBUTOR
Enterprises are moving to deploy GenAI in a significant way. According to recent research from Infosys, businesses in North America will be increasing their investments in GenAI by 67% in 2024 to reach an anticipated $6 billion. The question is: Will organizations reap the benefits they expect?
“Artificial intelligence and machine learning are non-deterministic by nature, so simply investing in artificial intelligence will not guarantee meaningful results — and failed implementations are commonplace,” says Chandini Jain, founder and CEO at AI and data services provider Auquan.
Chandini says to succeed with GenAI, organizations need well-defined metrics, established goals and ways to measure progress. Organizations must also understand what level of errors they can tolerate.
Ram Ramamoorthy, director of AI research at ManageEngine, the IT division of Zoho Corp, agrees. “Simply implementing GenAI is not enough to guarantee success if you’re not looking at the key performance indicators [KPIs] in your organization,” Ramamoorthy says.
That is sound advice; however, many organizations are rushing forward without having such established goals or ways to measure performance.
Ramamoorthy advises companies to consider how GenAI has “increased employee productivity, saved time, aligned with organizational goals, and paved the way for innovations. And, on the business front, look out for growth and retention of customers, and seek their feedback.”
When it comes to specific measurements, Ramamoorthy and others advise organizations to look for metrics that will help measure the quality of their models, the system quality (data and AI asset reusability, throughput, latency, integration, and compatibility with other systems) as well as GenAI’s business impact. These can include the speed of adoption, how often the tools are used, accelerated time to value, and business savings such as employee hours saved.
Orla Daly, chief information officer at Skillsoft, suggests enterprise business-technology leaders take a step back and look at the big picture before choosing where to deploy GenAI.
According to a survey commissioned by AI work assistant provider Glean, enterprises haven’t yet established ways to measure GenAI returns. The survey of 224 senior IT leaders within organizations with 1,000 employees and at least $100 million in revenue found that 28% of respondents said they generate positive ROI from generative AI initiatives. In contrast, 31% said they believe they’re generating positive ROI but don’t have data to support that assertion.
Finally, 17% of respondents said they’re not generating an ROI but expect to do so within a year, another 17% said it’s too early to tell about ROI, and 6% don’t expect an ROI in the next year. Forty-six percent of respondents said the results were a little better or better than they’d initially expected. That’s all undoubtedly promising.
How are respondents measuring ROI? For 57% of respondents, the top metric used was employee productivity.
For those who have yet to establish how they’ll measure AI success, Skillsoft’s Daly suggests they start by identifying current business priorities, which will lead to identifying internal and external opportunities to deploy GenAI. Daly explains Skillsoft’s approach: “As an education technology company, for example, [our] GenAI strategy focuses on three key areas: What we learn, how we learn, and how we work. We think about this regarding our customers and employees, informing both the solutions we bring to the market and our internal workflows and processes.”
Maulik Bhagat, executive vice president of services and innovation at management and technology consulting firm AArete, advises organizations to measure ROI and business value from GenAI deployments as they have with other technologies. “As far as measuring the success of GenAI implementations, I don’t see it as much different from any other investment or initiative that you would measure the success for,” he says. Bhagat. Bhagat explains, however, that enterprises need to have clearly defined metrics and expected business outcomes, a baseline for pre-implementation, and an understanding of the costs and timeline for implementation.
Risk tolerance also needs to be reflected. “An additional aspect to consider will be risk, especially in terms of what the output from GenAI applications is being used for,” Bhagat adds. Bhagat cites healthcare clinical decision-making as an example where the risks may be too high to consider despite a very appealing ROI on paper.
While some organizations may struggle to quantify a financial ROI in all cases, don’t expect GenAI deployments to slow down soon. “There is only a short window until the market catches up, where early adopters will likely get disruptive value/ROI from well-implemented GenAI use cases,” Bhagat says.
