Who owns data in your organization? It’s a question Sharad Kumar, field CTO of Qlik, would often ask in his meetings with CTOs and IT leaders – and the answer he’d almost never get is the one that is right.
This is revealing of a larger problem that’s prevailing in IT today. As data architectures are quietly changing under the hood, there is a glaring lack of ownership that’s caused a rift to open up between producers and consumers of data.
“Data users are not getting what they really want,” said Kumar, while explaining the problem at the Qlik Tech Field Day Showcase in December.
More mature enterprises are adopting a federated, and service-oriented approach called data products. Data products are units of data that are productized for easy consumption. More specifically, these are assets that contain data and code rolled into one package. Also loaded in the packages are sundry items like business logic and semantics, data quality rules, access policies, specific use cases they are designed for.
The additional information makes the data products re-useable for analytical consumption across a variety of domain-specific use cases.
“Data products simplify your data environment. Think of it as data products are to centralized data platform what microservices are to applications,” explained Kumar. “They are more modular, more consumable, and more usable.”
But most importantly, they have designated owners. “Data products have measurable value because they have ownership,” he said.
This owner – whether it’s an individual or a team – is responsible for creating and maintaining the assets, starting with curating and cleansing the data, ensuring high quality, keeping them updated and trustworthy, lifecycles, versioning and finally retirement.
“Different teams within each domain build and manage their own data products. This federation brings agility; they don’t need to build complex central pipelines or large enterprise-wide data models because each team understands the need of the users, what the use cases are, and how they want to consume it. They build the pipelines and shape the data to their own needs and move at their own velocity,” Kumar explained.
An IDC survey shows that 28% companies that are in mature stages of the AI journey have data products deployed in their environment, while 4% are in the early stages of adoption.
At the Tech Field Day Showcase, Qlik announced that it now offers support for data product management on Qlik Talend Cloud. Users can design domain-specific data products and deploy them using the platform’s data catalog capabilities.
Data Products Catalog caters to both data producers and consumers within the organization. Under the hood, it features three capability planes, each designed for specific personas, Kumar told.
At the bottom is the data product engineering plane that serves as the foundational layer for all things data. Here the data platform team performs the steps of extraction, ingestion, integration and transformation of datasets before passing it on to the data product managers.
A level above is the data product management plane where it all comes together. This layer comes with a host of out-of-the-box self-serve capabilities for building and managing data products. Various teams come together on this plane to create these products by choosing the required datasets, adding rules and semantics, enforcing access control, and finally activating them for consumption. This is also the layer where the full data product lifecycle from activation to retirement is managed.
All activated products are published in an internal data product marketplace where consumers can search and find them for building analytics solutions.
“I could expose a SQL interface on the data product so somebody could build a business intelligence report, or I could provide the data as a file for example, so that somebody could train a machine learning model, or I could make it available as a vector interface for somebody to build a RAG,” said Kumar while explaining the options.
Users assigned to the role of data product manager can add to the data products drawing from relevant datasets. After a data product is created, Qlik shows an aggregated view of the quality indicators – validity and completeness – based on the samples, make it easy to make improvements.
For more information, head over to Qlik.com, or check out presentations from the Qlik Tech Field Day Showcase.
