Use cases for business intelligence (BI), analytics, and artificial intelligence (AI) are growing more complex, involving more data sources, platforms, locations, and a greater variety and number of data producers and consumers. This means data flows are becoming exponentially more complex, transformation logic is much more sophisticated, and more opportunities exist for errors and leaks. At the same time, organizations’ emphasis on data governance—of quality, security, privacy, lifecycle management, and costs—is also increasing, as data leaders, business leaders, and management teams recognize the importance of leveraging and protecting data assets.
These collective forces are driving the need for greater control of data flowing through the infrastructure supporting BI, analytics, and AI environments—and as a result, demand is growing for data observability. Key use cases where these needs are acute include operational data monitoring, dashboards and KPI reporting, compliance and audit readiness, forecasting, and fraud detection and risk management. As BI, analytics, and AI use cases become more mission-critical, the supporting data pipelines likewise become equally important, requiring higher degrees of transparency and monitoring.
More technology providers, both larger analytical data infrastructure (ADI) platform players and specialists, are bringing data observability capabilities to the market. Conditions are right for rapid growth in demand for data observability features and functionality, as growing complexity and mission-criticality increase pressure on end user organizations to stay “in control” of their data. However, capitalizing on this burgeoning demand requires vendors to understand customers’ data observability needs and adapt their positioning accordingly.
Adding data observability capabilities may also represent an opportunity for a broader data governance stance in the future. Data observability increasingly will become a foundational capability for governance, since governance starts with an understanding of data assets, how they are accessed, and where they are flowing and consumed within the enterprise. Surfacing insight about data flows and metrics about data usage will provide governance-focused teams and roles with a strong fact base they can use to operationalize governance policies around security, privacy, quality, lifecycle management, and the cost of critical data.
Data observability functionality can provide tremendous value for organizations that have the right set of conditions —primarily “lossiness,” lack of reliability, and quality issues in important data pipelines. These challenges increase as the underlying data flows supporting BI, analytics, and AI use cases are more frequently considered just as mission critical as the applications they fuel.
You do not have permission to access this document. Make sure you are logged in and/or please contact Danielle with further questions.