Data mesh

Dec 30, 2019Download as PPTX, PDF1 like2,252 views

A Work of Zhamak Dehghani Principal consultant ThoughtWorks https://martinfowler.com/articles/data-monolith-to-mesh.html https://fast.wistia.net/embed/iframe/vys2juvzc3?videoFoam How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh Many enterprises are investing in their next generation data lake, with the hope of democratizing data at scale to provide business insights and ultimately make automated intelligent decisions. Data platforms based on the data lake architecture have common failure modes that lead to unfulfilled promises at scale. To address these failure modes we need to shift from the centralized paradigm of a lake, or its predecessor data warehouse. We need to shift to a paradigm that draws from modern distributed architecture: considering domains as the first class concern, applying platform thinking to create self-serve data infrastructure, and treating data as a product.

Data Mesh
A Work of Zhamak Dehghani
Principal consultant
ThoughtWorks
https://martinfowler.com/articles/data-monolith-to-mesh.html
https://fast.wistia.net/embed/iframe/vys2juvzc3?videoFoam

Data Mesh is an innovative concept addressing many data challenges from an architectural, cultural, and organizational perspective. But is the world ready to implement Data Mesh? In this session, we will review the importance of core Data Mesh principles, what they can offer, and when it is a good idea to try a Data Mesh architecture. We will discuss common challenges with implementation of Data Mesh systems and focus on the role of open-source projects for it. Projects like Apache Spark can play a key part in standardized infrastructure platform implementation of Data Mesh. We will examine the landscape of useful data engineering open-source projects to utilize in several areas of a Data Mesh system in practice, along with an architectural example. We will touch on what work (culture, tools, mindset) needs to be done to ensure Data Mesh is more accessible for engineers in the industry. The audience will leave with a good understanding of the benefits of Data Mesh architecture, common challenges, and the role of Apache Spark and other open-source projects for its implementation in real systems. This session is targeted for architects, decision-makers, data-engineers, and system designers.

Data Lakehouse, Data Mesh, and Data Fabric (r2)James Serra

So many buzzwords of late: Data Lakehouse, Data Mesh, and Data Fabric. What do all these terms mean and how do they compare to a modern data warehouse? In this session I’ll cover all of them in detail and compare the pros and cons of each. They all may sound great in theory, but I'll dig into the concerns you need to be aware of before taking the plunge. I’ll also include use cases so you can see what approach will work best for your big data needs. And I'll discuss Microsoft version of the data mesh.

Data Mesh 101ChrisFord803185

Data Mesh is a new socio-technical approach to data architecture, first described by Zhamak Dehghani and popularised through a guest blog post on Martin Fowler's site. Since then, community interest has grown, due to Data Mesh's ability to explain and address the frustrations that many organisations are experiencing as they try to get value from their data. The 2022 publication of Zhamak's book on Data Mesh further provoked conversation, as have the growing number of experience reports from companies that have put Data Mesh into practice. So what's all the fuss about? On one hand, Data Mesh is a new approach in the field of big data. On the other hand, Data Mesh is application of the lessons we have learned from domain-driven design and microservices to a data context. In this talk, Chris and Pablo will explain how Data Mesh relates to current thinking in software architecture and the historical development of data architecture philosophies. They will outline what benefits Data Mesh brings, what trade-offs it comes with and when organisations should and should not consider adopting it.

Data Mesh for DinnerKent Graziano

Data mesh is a decentralized approach to managing and accessing analytical data at scale. It distributes responsibility for data pipelines and quality to domain experts. The key principles are domain-centric ownership, treating data as a product, and using a common self-service infrastructure platform. Snowflake is well-suited for implementing a data mesh with its capabilities for sharing data and functions securely across accounts and clouds, with built-in governance and a data marketplace for discovery. A data mesh implemented on Snowflake's data cloud can support truly global and multi-cloud data sharing and management according to data mesh principles.

Webinar Data Mesh - Part 3Jeffrey T. Pollock

The document discusses data mesh vs data fabric architectures. It defines data mesh as a decentralized data processing architecture with microservices and event-driven integration of enterprise data assets across multi-cloud environments. The key aspects of data mesh are that it is decentralized, processes data at the edge, uses immutable event logs and streams for integration, and can move all types of data reliably. The document then provides an overview of how data mesh architectures have evolved from hub-and-spoke models to more distributed designs using techniques like kappa architecture and describes some use cases for event streaming and complex event processing.

How to Build the Data Mesh Foundation: A Principled Approach | Zhamak Dehghan...HostedbyConfluent

Organizations have been chasing the dream of data democratization, unlocking and accessing data at scale to serve their customers and business, for over a half a century from early days of data warehousing. They have been trying to reach this dream through multiple generations of architectures, such as data warehouse and data lake, through a cambrian explosion of tools and a large amount of investments to build their next data platform. Despite the intention and the investments the results have been middling. In this keynote, Zhamak shares her observations on the failure modes of a centralized paradigm of a data lake, and its predecessor data warehouse. She introduces Data Mesh, a paradigm shift in big data management that draws from modern distributed architecture: considering domains as the first class concern, applying self-sovereignty to distribute the ownership of data, applying platform thinking to create self-serve data infrastructure, and treating data as a product. This talk introduces the principles underpinning data mesh and Zhamak's recent learnings in creating a path to bring data mesh to life in your organization.

Intuit's Data Mesh - Data Mesh Leaning Community meetup 5.13.2021Tristan Baker

Data Mesh Part 4 Monolith to MeshJeffrey T. Pollock

This is Part 4 of the GoldenGate series on Data Mesh - a series of webinars helping customers understand how to move off of old-fashioned monolithic data integration architecture and get ready for more agile, cost-effective, event-driven solutions. The Data Mesh is a kind of Data Fabric that emphasizes business-led data products running on event-driven streaming architectures, serverless, and microservices based platforms. These emerging solutions are essential for enterprises that run data-driven services on multi-cloud, multi-vendor ecosystems. Join this session to get a fresh look at Data Mesh; we'll start with core architecture principles (vendor agnostic) and transition into detailed examples of how Oracle's GoldenGate platform is providing capabilities today. We will discuss essential technical characteristics of a Data Mesh solution, and the benefits that business owners can expect by moving IT in this direction. For more background on Data Mesh, Part 1, 2, and 3 are on the GoldenGate YouTube channel: https://www.youtube.com/playlist?list=PLbqmhpwYrlZJ-583p3KQGDAd6038i1ywe Webinar Speaker: Jeff Pollock, VP Product (https://www.linkedin.com/in/jtpollock/) Mr. Pollock is an expert technology leader for data platforms, big data, data integration and governance. Jeff has been CTO at California startups and a senior exec at Fortune 100 tech vendors. He is currently Oracle VP of Products and Cloud Services for Data Replication, Streaming Data and Database Migrations. While at IBM, he was head of all Information Integration, Replication and Governance products, and previously Jeff was an independent architect for US Defense Department, VP of Technology at Cerebra and CTO of Modulant – he has been engineering artificial intelligence based data platforms since 2001. As a business consultant, Mr. Pollock was a Head Architect at Ernst & Young’s Center for Technology Enablement. Jeff is also the author of “Semantic Web for Dummies” and "Adaptive Information,” a frequent keynote at industry conferences, author for books and industry journals, formerly a contributing member of W3C and OASIS, and an engineering instructor with UC Berkeley’s Extension for object-oriented systems, software development process and enterprise architecture.

Enabling a Data Mesh Architecture with Data VirtualizationDenodo

Watch full webinar here: https://bit.ly/3rwWhyv The Data Mesh architectural design was first proposed in 2019 by Zhamak Dehghani, principal technology consultant at Thoughtworks, a technology company that is closely associated with the development of distributed agile methodology. A data mesh is a distributed, de-centralized data infrastructure in which multiple autonomous domains manage and expose their own data, called “data products,” to the rest of the organization. Organizations leverage data mesh architecture when they experience shortcomings in highly centralized architectures, such as the lack domain-specific expertise in data teams, the inflexibility of centralized data repositories in meeting the specific needs of different departments within large organizations, and the slow nature of centralized data infrastructures in provisioning data and responding to changes. In this session, Pablo Alvarez, Global Director of Product Management at Denodo, explains how data virtualization is your best bet for implementing an effective data mesh architecture. You will learn: - How data mesh architecture not only enables better performance and agility, but also self-service data access - The requirements for “data products” in the data mesh world, and how data virtualization supports them - How data virtualization enables domains in a data mesh to be truly autonomous - Why a data lake is not automatically a data mesh - How to implement a simple, functional data mesh architecture using data virtualization

Time to Talk about Data MeshLibbySchulze

This document discusses data mesh, a distributed data management approach for microservices. It outlines the challenges of implementing microservice architecture including data decoupling, sharing data across domains, and data consistency. It then introduces data mesh as a solution, describing how to build the necessary infrastructure using technologies like Kubernetes and YAML to quickly deploy data pipelines and provision data across services and applications in a distributed manner. The document provides examples of how data mesh can be used to improve legacy system integration, batch processing efficiency, multi-source data aggregation, and cross-cloud/environment integration.

Data Mesh in Practice - How Europe's Leading Online Platform for Fashion Goes...Dr. Arif Wider

A talk presented by Max Schultze from Zalando and Arif Wider from ThoughtWorks at NDC Oslo 2020. Abstract: The Data Lake paradigm is often considered the scalable successor of the more curated Data Warehouse approach when it comes to democratization of data. However, many who went out to build a centralized Data Lake came out with a data swamp of unclear responsibilities, a lack of data ownership, and sub-par data availability. At Zalando - europe’s biggest online fashion retailer - we realised that accessibility and availability at scale can only be guaranteed when moving more responsibilities to those who pick up the data and have the respective domain knowledge - the data owners - while keeping only data governance and metadata information central. Such a decentralized and domain focused approach has recently been coined a Data Mesh. The Data Mesh paradigm promotes the concept of Data Products which go beyond sharing of files and towards guarantees of quality and acknowledgement of data ownership. This talk will take you on a journey of how we went from a centralized Data Lake to embrace a distributed Data Mesh architecture and will outline the ongoing efforts to make creation of data products as simple as applying a template.

Data Mess to Data Mesh | Jay Kreps, CEO, Confluent | Kafka Summit Americas 20...HostedbyConfluent

Companies are increasingly becoming software-driven, requiring new approaches to software architecture and data integration. The "data mesh" architectural pattern decentralizes data management by organizing it around domain experts and treating data as products that can be accessed on-demand. This helps address issues with centralized data warehouses by evolving data modeling with business needs, avoiding bottlenecks, and giving autonomy to domain teams. Key principles of the data mesh include domain ownership of data, treating data as self-service products, and establishing federated governance to coordinate the decentralized system.

Apache Kafka® and the Data MeshConfluentInc1

This document discusses how Apache Kafka and event streaming fit within a data mesh architecture. It provides an overview of the key principles of a data mesh, including domain-driven decentralization, treating data as a first-class product, a self-serve data platform, and federated governance. It then explains how Kafka's publish-subscribe event streaming model aligns well with these principles by allowing different domains to independently publish and consume streams of data. The document also describes how Kafka can be used to ingest existing data sources, process data in real-time, and replicate data across the mesh in a scalable and interoperable way.

Data Lakehouse Symposium | Day 4Databricks

The document discusses the challenges of modern data, analytics, and AI workloads. Most enterprises struggle with siloed data systems that make integration and productivity difficult. The future of data lies with a data lakehouse platform that can unify data engineering, analytics, data warehousing, and machine learning workloads on a single open platform. The Databricks Lakehouse platform aims to address these challenges with its open data lake approach and capabilities for data engineering, SQL analytics, governance, and machine learning.

Data Mesh at CMC Markets: Past, Present and FutureLorenzo Nicora

This document discusses CMC Markets' implementation of a data mesh to improve data management and sharing. It provides an overview of CMC Markets, the challenges of their existing decentralized data landscape, and their goals in adopting a data mesh. The key sections describe what data is included in the data mesh, how they are using cloud infrastructure and tools to enable self-service, their implementation of a data discovery tool to make data findable, and how they are making on-premise data natively accessible in the cloud. Adopting the data mesh framework requires organizational changes, but enables autonomy, innovation and using data to power new products.

How a Semantic Layer Makes Data Mesh Work at ScaleDATAVERSITY

Data Mesh is a trending approach to building a decentralized data architecture by leveraging a domain-oriented, self-service design. However, the pure definition of Data Mesh lacks a center of excellence or central data team and doesn’t address the need for a common approach for sharing data products across teams. The semantic layer is emerging as a key component to supporting a Hub and Spoke style of organizing data teams by introducing data model sharing, collaboration, and distributed ownership controls. This session will explain how data teams can define common models and definitions with a semantic layer to decentralize analytics product creation using a Hub and Spoke architecture. Attend this session to learn about: - The role of a Data Mesh in the modern cloud architecture. - How a semantic layer can serve as the binding agent to support decentralization. - How to drive self service with consistency and control.

Gartner: Master Data Management FunctionalityGartner

MDM solutions require tightly integrated capabilities including data modeling, integration, synchronization, propagation, flexible architecture, granular and packaged services, performance, availability, analysis, information quality management, and security. These capabilities allow organizations to extend data models, integrate and synchronize data in real-time and batch processes across systems, measure ROI and data quality, and securely manage the MDM solution.

Data Mesh in Practice: How Europe’s Leading Online Platform for Fashion Goes ...Databricks

Databricks FundamentalsDalibor Wijas

This document is a training presentation on Databricks fundamentals and the data lakehouse concept by Dalibor Wijas from November 2022. It introduces Wijas and his experience. It then discusses what Databricks is, why it is needed, what a data lakehouse is, how Databricks enables the data lakehouse concept using Apache Spark and Delta Lake. It also covers how Databricks supports data engineering, data warehousing, and offers tools for data ingestion, transformation, pipelines and more.

Data Catalogs Are the Answer – What is the Question?DATAVERSITY

Organizations with governed metadata made available through their data catalog can answer questions their people have about the organization’s data. These organizations get more value from their data, protect their data better, gain improved ROI from data-centric projects and programs, and have more confidence in their most strategic data. Join Bob Seiner for this lively webinar where he will talk about the value of a data catalog and how to build the use of the catalog into your stewards’ daily routines. Bob will share how the tool must be positioned for success and viewed as a must-have resource that is a steppingstone and catalyst to governed data across the organization.

Data Lakehouse, Data Mesh, and Data Fabric (r1)James Serra

How to govern and secure a Data Mesh?confluent

This document discusses challenges with centralized data architectures and proposes a data mesh approach. It outlines 4 challenges: 1) centralized teams fail to scale sources and consumers, 2) point-to-point data sharing is difficult to decouple, 3) bridging operational and analytical systems is complex, and 4) legacy data stacks rely on outdated paradigms. The document then proposes a data mesh architecture with domain data as products and an operational data platform to address these challenges by decentralizing control and improving data sharing, discovery, and governance.

DW Migration Webinar-March 2022.pptxDatabricks

The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.

Introduction SQL Analytics on Lakehouse ArchitectureDatabricks

This document provides an introduction and overview of SQL Analytics on Lakehouse Architecture. It discusses the instructor Doug Bateman's background and experience. The course goals are outlined as describing key features of a data Lakehouse, explaining how Delta Lake enables a Lakehouse architecture, and defining features of the Databricks SQL Analytics user interface. The course agenda is then presented, covering topics on Lakehouse Architecture, Delta Lake, and a Databricks SQL Analytics demo. Background is also provided on Lakehouse architecture, how it combines the benefits of data warehouses and data lakes, and its key features.

Data Architecture Brief OverviewHal Kalechofsky

Data architecture defines the target state for an information system by describing how data is processed, stored, and utilized. It shows data structures, flows, and usage across business applications and systems. Data architecture sets data standards and addresses both stored and moving data. Its benefits include higher quality, reduced costs, quicker time to market, clearer scope, faster performance, better documentation, fewer errors, and managed risks. Defining the target state involves conceptual, logical, and physical architectural processes to represent enterprise entities, their relationships, and specific data mechanisms. Influencers include requirements, technology, economics, policies, and processing needs. Principles include building decoupled systems, using the right tools, leveraging managed services, and using log-

Data platform architectureSudheer Kondla

The document discusses data architecture solutions for solving real-time, high-volume data problems with low latency response times. It recommends a data platform capable of capturing, ingesting, streaming, and optionally storing data for batch analytics. The solution should provide fast data ingestion, real-time analytics, fast action, and quick time to value. Multiple data sources like logs, social media, and internal systems would be ingested using Apache Flume and Kafka and analyzed with Spark/Storm streaming. The processed data would be stored in HDFS, Cassandra, S3, or Hive. Kafka, Spark, and Cassandra are identified as key technologies for real-time data pipelines, stream analytics, and high availability persistent storage.

Databricks Platform.pptxAlex Ivy

The document provides an overview of the Databricks platform, which offers a unified environment for data engineering, analytics, and AI. It describes how Databricks addresses the complexity of managing data across siloed systems by providing a single "data lakehouse" platform where all data and analytics workloads can be run. Key features highlighted include Delta Lake for ACID transactions on data lakes, auto loader for streaming data ingestion, notebooks for interactive coding, and governance tools to securely share and catalog data and models.

Developing Security Orchestration, Automation, and Response ApplicationsVICTOR MAESTRE RAMIREZ

Secure_File_Storage_Hybrid_Cryptography.pptx..yuvarajreddy2002

Data mesh

1. Data Mesh A Work of Zhamak Dehghani Principal consultant ThoughtWorks https://martinfowler.com/articles/data-monolith-to-mesh.html https://fast.wistia.net/embed/iframe/vys2juvzc3?videoFoam

Data mesh

Recommended

More Related Content

What's hot (20)

Recently uploaded (20)

Data mesh