A survey of 157 data management decision-makers in North America conducted by Forrester Consulting on behalf of Capital One suggests that one of the primary reasons so many organizations are finding it challenging to achieve their digital business transformation goals is data management.
Nearly 80% of survey respondents identified data cataloging as a major challenge, without which they struggle to understand what data they have, how the data is used, and who owns the data.
Other key concerns surfaced include poor data quality (80%), difficulty understanding data (76%) and a lack of data observability (74%). A total of 82% also cited confusing data governance policies as a top challenge, while 80% note a lack of entitlements and enforcing role-based access to specific data. A full 80% said they have difficulty governing data at scale overall, while 82% said they find it difficult to forecast and control costs.
Those challenges will only continue to grow as data sets not only increase in size but also become more diverse as new types of data are added to the data management mix, noted Salim Syed, head of engineering for Capital One Software.
The only way to address that challenge is to adopt a more federated approach to data management than enables an IT organization to centrally manage data while access to data is governed and secured via a series of distributed self-service portals, he added.
The survey suggests many organizations are moving down that path over the next two years, with cloud data investments being made in protection and security (49%), tools that make data easier to find and use (47%), and tools that reduce and manage costs (39%). Decision-makers anticipate that investment in entitlements is expected to nearly double in one to two years, the survey finds.
A full 84% said they also plan to use different or multiple providers to house data. Only 10% of respondents use different vendors for different use cases, but over the next one to two years, multivendor use will increase nearly four-fold to 38%, the survey finds. A full 87% also report relying on commercial tools or vendors compared to just over a quarter (28%) employing a mix of homegrown or open-source tools.
Ultimately, the goal of any IT team should be to enable organizations to derive the maximum value from investments in data, said Syed. “The goal is to bring data out of the darkness and into the light,” he said.
The challenge is IT teams are often not able to distinguish one data set from another, much less track its data lineage. Historically, IT teams have tended to focus more on data storage issues rather than maximizing business value. That issue, in fact, is at the core of the historic divide between IT and the rest of the business. Only the business units that created the data in the first place tend to understand its value.
Unfortunately, those same business units don’t trust the data they create because it is often conflicting and inconsistent, which makes investing in the data analytics tools needed to drive digital transformation initiatives often problematic.
It may be years before organizations resolve data management issues, However, as digital business transformation initiatives continue to advance, a much larger percentage of organizations are finally coming to terms with data management issues that have been festering in enterprise IT organizations for decades.