Many deployments go bad as enterprise leaders race to deploy their artificial intelligence/machine learning models (AI/ML) to improve their businesses and operations. This issue poses problems for companies hoping to digitally transform processes using these tools.
Consider this 2021 survey from Gartner research that found only 54% of AI models successfully move from pilot to production. Perhaps that percentage improved over the past few years? Unlikely. According to this 2023 Data Science survey from Rexer Analytics, only 32% of such deployments move successfully from pilot to production.
Experts say that one of the reasons for the AI/ML model’s abysmal outcomes is the lack of an AI/ML deployment and machine learning operations (MLOps) strategy. Overall, those we asked cited the following as the top reasons for failure:
Failure to Identify and Satisfy AI/ML Business Objectives
Olga Kupriyanova, principal consultant with global technology research and advisory firm ISG, says this is where organizations should begin. “Companies should comprehensively assess their current data landscape, machine-learning capabilities, and business objectives. This evaluation will help identify the key drivers for MLOps adoption and align them with overall organizational goals,” Kupriyanova says.
It sounds like common sense, but it’s an area many organizations miss. Business objectives can be anything from improving operational efficiency, increasing revenue, or improving customer satisfaction. Whatever the AI/ML deployment objectives are, they must be defined before the success of MLOps initiatives can be defined.
Organizations May Go Too Big With Their Models
Instead, experts advise enterprises to start small and deliver quick MLOps wins. The MLOps teams can learn how to best work together before tackling more significant challenges by hitting initial, straightforward objectives. They can also gain trust among other organizational stakeholders and build momentum. “Developing a roadmap that prioritizes quick wins and addresses critical pain points can foster early success and buy-in from stakeholders,” Kupriyanova says.
Developing quick value will help demonstrate to the rest of the organization the value of MLOps. Consider AI/ML MVPs. By building minimum viable AI/ML models, at least at first, MLOps teams can iterate and increase complexity over time. By avoiding “big bang” deployments, MLOps teams will avoid becoming overwhelmed by unwieldy ML models.
“Beginning with small, manageable projects can help demonstrate value and gather learnings to scale MLOps practices effectively across the organization,” adds Maksym Lushpenko, a senior DevOps engineer and founder of the DevOps Assessment Platform, provider Brokee.
There are No Clearly Established Goals and Metrics
Progress can be successfully gauged by creating specific goals and regularly measuring against key performance indicators (KPIs). It’s expected that the appropriate KPIs should be chosen carefully and will vary from organization to organization. Operational KPIs may include model training time, model inference time (prediction latency), model resource utilization and model deployment time.
CI/CD Principles are Not Applied to MLOps
Miroslav Klivansky, global practice leader of Pure Storage, in analytics and AI, says modern development patterns should be brought to MLOps. “When establishing MLOps initiatives, applying CI/CD principles to machine learning is important. This approach can enable organizations to automate and continuously integrate code changes, model training, and deployments,” Klivansky says.
“MLOps should also follow infrastructure-as-code principles to eliminate potential issues and ensure consistency from development to production. It’s also important to use tools and best practices for monitoring model performance and establish feedback loops to train models to enhance their accuracy over time. The main difference to remember is that in MLOps, everything needs to be versioned and tracked – the code, the models, and even the data sets, since they may change based on data cleaning, different feature definitions, and more. Tools like DVC and LakeFS make that easier and integrate with MLOps workflows,” Klivansky adds.
They Don’t Plan for Scalability
While starting small, have a long-term plan and vision for automating and scaling your MLOps processes as your needs grow. “Organizations should also focus on establishing a robust MLOps infrastructure that includes the necessary tools, platforms, and processes to support the entire machine-learning lifecycle,” says Kupriyanova.
Joseph Thacker, principal AI engineer and security researcher at AppOmni, states that When thinking about their MLOps initiatives, they should focus on creating a flexible infrastructure that can accommodate different models, architectures, and tools. This is because new model versions become available relatively quickly, and new architectures or tools may be required to meet customer requests,” Thacker says.
They Don’t Encourage Cross-Functional Collaboration
Organizations must promote collaboration and communication among data scientists, data engineers, DevOps teams and other stakeholders for efficient operations. “Fostering a culture of collaboration between data scientists, engineers and operational teams is crucial to integrating and streamlining the MLOps workflow,” Kupriyanova adds.
Experts say that experimentation should also be encouraged among this team. However, Lou Flynn, senior manager for AI at SAS, warns that caution is advised when it comes to MLOps and AI/ML model experimentation. Encouraging a culture of experimentation, exploration and curiosity among data scientists and modelers is crucial,” Flynn says. “However, this creative freedom should be balanced with a consistent organizational rhythm. This rhythm or “heartbeat” is essential for evaluating the ROI from machine learning initiatives, ensuring that efforts contribute to strategic goals rather than wandering into the territory of mere science experiments. By striking this balance, organizations can foster an environment where MLOps flourishes and drives meaningful, measurable outcomes,” Flynn advises.
However, by not merely avoiding – but reversing – these pitfalls, organizations can lay a solid foundation for their MLOps initiatives, build trust and buy-in throughout their organization, and incrementally scale their efforts while minimizing risks and unnecessary complexity.
