Self-service data infrastructure as a platform is the technical pillar of Data Mesh. This pillar aims to resource the data domains and host the developed data products. It is essential in the Data Mesh approach to provide a federated and interoperable platform or infrastructure with self-service addressed resources. This promotes domain autonomy & data distribution.
Self-service data infrastructure as a platform is therefore intended to position IT as a facilitator of the data mesh approach. It also aims to avoid an overload of used technologies. However, the efforts made in recent years in terms of streamlining should not be written off as Data Mesh is all about consolidating infrastructure & data services.
Data Mesh: a technology-agnostic approach
One of the key points of Data Mesh is that it is technology agnostic.
It can therefore be implemented with any type of tool or database: ETL tool, SQL or NoSQL databases, with traditional reporting or data visualization tools or even with Data Science studios or programming languages! The most important thing is that the access and exchange of data is standardized across the organization. This facilitates a better interaction between the domains.
Above all, the infrastructure must be federated and accessible in the form of an on-demand service platform. This platform is structured with services within an application catalog that will be used by the domains.
What are the benefits of the Data Mesh with this self-service data platform?
What changes is mainly the streamlining, but also the supervision which is now managed with a single point. This platform is thus a resource that can be shared, but whose development is managed within a roadmap defined according to well-defined user needs.
The data platform must also be managed as a shared resource, whose development is controlled and operated within a roadmap organized according to the needs of its users (the domains).
IT team at the heart of the data platform with the Data Mesh
By providing the application catalog, the organization's IT department maintains its function and evolves it in more deployment, maintenance and technical support aspects. It is less involved in the "project" aspect (the development of data products is the responsibility of the domains) and focuses on the deployment, maintenance and technical support of infrastructure and application services.
IT enables and facilitates all data initiatives undertaken by domains and sets the rules for the technology pillar.
IT is involved in three areas
- Infrastructure (resource provisioning capabilities, compute, storage, orchestration, etc.)
- The oversight of the infrastructure and spending (via FinOps), and its governance.
- The development of the platform as a product for the benefit of the domains.
Cloud & Data Mesh
Cloud Data Platforms, that are emerging more & more today, are a key asset when implementing Data Mesh as they are supporting the self-service data infrastructure platform.
Nevertheless, let’s remember that the Cloud as part of Data Mesh must follow few good practices to generate a real value. To deployment of the platform need to be aligned with the implementation of a FinOps approach. It is crucial to have the resources to maintain the federated vision & ensure a good control of the expenditure. In this way, all domains can manage their own costs.
