The data engineering domain has benefited from many advances over the recent years, including streaming technologies, on-demand infrastructure, and the proliferation of cloud-native tooling to digest, process, store, and serve data at scale. Despite these advances, one thing remains underdeveloped: the approach to the overall data engineering platform architecture and team structure. The current state of the art, unfortunately, remains centralized, functionally organized and managed by hyperspecialized teams, difficult to scale, and hard to leverage.
Zhamak Dehghani shares observations and lessons learned working with and consulting many organizations that are busy developing their next generation data platforms. Zhamak introduces an alternative data platform architecture that she calls Data Mesh – a decentralized and ubiquitous data platform – with an organizational model that can enable your business to truly become data driven.