Apache Kafka and the Data Mesh | Ben Stopford and Michael Noll, Confluent

Editor's Notes

• #6: Small scale data pipelines constantly broken. Large scale: finance and risk have completely different numbers. Story of one path for books in an investment bank. Boose Allen Hamilton. 3 months analysis. 2 hours to explain.
• #7: This allows the applications to connect around data in motion Acts as a kind of central nervous system Let’s something happening in one part of the company, trigger the right updates and response everywhere else as it occurs
• #9: Thanks Ben for setting the stage and covering the influences of the data mesh concept. Let us now dive into building a data mesh with event streaming and Kafka.
• #10: Ben and I will discuss the four data mesh principles: (1) giving local autonomy to teams via domain-driven decentralization, (2) establishing product thinking to treat data as a first-class product, (3) providing a self-serve data platform so the local domains can easily participate in the data mesh, and (4) federated governance to tie these independent domains and data assets together so they actually “mesh” and interoperate.
• #11: The data mesh concept has both organizational and technical dimensions. But, essentially, we can think of a data mesh as a network to exchange data about the business, with nodes and connections. The nodes in the mesh are the “data products”—we’ll learn more about them later, but for now let’s just imagine a node could be a microservice, a database, an application, etc. The nodes function is to produce and consume high-quality, locally-curated data within the mesh for the benefits of other nodes. “Related” nodes, so to speak, are grouped into domains, like “Inventory”. And the data flows from node-to-node and thus from domain-to-domain as needed.
• #13: A centralized approach, essentially, does not work at non-trivial scale (might work for a small startup, but not an enterprise).
• #14: OUTRO: This ensures that data does not diverge in different ways across the organization; e.g., the risk department has different closing position to the finance department.
• #16: Let’s talk about the second principle: treating data as a first-class product. This is arguably the most important principle of the four.
• #17: This principle is about product thinking: Your domain’s shared data is managed as a true product. And your objective is to provide this data product in the right way so that other teams can make good use of it. In other words, you treat the other teams as internal customers of your data. This product thinking is really important to prevent data chauvinism, which is a risk of principle 1 that Ben just covered. That’s because decentralization and granting local autonomy to domains may cause the creation of even more data silos along with data quality problems if the origin domains only care about themselves. So, what is “a data product” then? Sounds a bit abstract, doesn’t it?
• #18: Think of a data product as a microservice for the data world. It is a node on the data mesh, situated within a particular domain. The role of the data product is to produce—and possibly consume—high-quality data within the mesh. And, like a microservice, it encapsulates everything it needs for its functioning: (1) the actual data that we want to share with the outside world, (2) the code that creates, manipulates, and shares that data, (3) the infrastructure that we need to store the data and to run the code. So, as you can see, it’s really similar to microservices. [OUTRO] Now, before we take a closer look at a data product, let’s first zoom out again to the bigger picture of the data mesh.
• #19: Because a key question is: How is data shared across data products in the mesh? That is, how do we implement the links and connections between the nodes in the network? And, even visually, we can see that this interconnectivity lends itself very naturally to … [NEXT SLIDE]
• #20: ...Event Streaming with Kafka. Here, data is provided to other data products through streams in Kafka. And any data product can consume via Kafka from the high-quality data streams of other data products. As we can see, this idea of a data mesh is very similar to the idea of a Central Nervous System, where data is continuously flowing, being processed, analyzed, acted upon. Now, we must remember that the data mesh shown here is a LOGICAL view, not a physical one. [OUTRO] If you know Kafka, you know that the reality looks a bit different and...a bit better.
• #21: Event streaming is not a point-to-point architecture that leads to this spaghetti mess we have showed in the beginning of this talk. Instead, it is an architecture that decouples data producers from data consumers in a way that is highly scalable, fault-tolerant, and persistent (because streams are durably stored). Here, a data product talks to other data products only INDIRECTLY, and this layer of indirection is event streams in Kafka. So this setup is Write-Once, Read-Many. And all a data product has to do to participate in a data mesh is to publish its “public” data as streams, and then other data products can independently consume this data in the way they see fit. [OUTRO] Now, as a quick refresher, let’s recap the key properties of event streams and why they are such a great fit to build a data mesh?
• #22: Here are some of the reasons. For example, streams are real-time, so you can propagate data throughout the mesh immediately, as soon as new information is available. Streams are also persisted and replayable, so they let you capture both real-time AND historical data with one setup. And because they are immutable, they make for a great source of record, which is useful for governance purposes.
• #23: So, let’s get back to data products and the question of how do we get data into and out of a data product? There are three main options. A) Use streams. We already discussed why this is a good fit in a data mesh. A strong advantage is that they continuously import and export data as soon as new information is available in the business. B) Use Request-Response APIs to request a data snapshot on demand, like a file export. C) Use traditional ETL for batch import/export of nightly snapshots. This gives you a fixed cadence of importing and exporting these snapshots, but, compared to option B, less control over exactly what snapshot data you’ll be working with.
• #24: With event streaming, you can quickly onboard data from existing systems through Kafka connectors that stream data from these systems into the mesh in real-time. There are two hundred of such connectors readily available, so in most cases you don’t have to write anything yourself. So this is a great way to set data in motion when you are bootstrapping a data mesh.
• #25: Now that there is some data flowing into our data product, what’s happening then? Well, really, what’s happening inside a data product is of no concern to the data mesh. Apart from the team responsible for a data product, everybody else in the data mesh only cares about the public data that is shared by this domain. HOW this public data is created internally doesn’t matter, as long as it meets criteria such as being addressable, trustworthy, have good quality, and so on.
• #26: That being said, the local team does care about how the data is being made. And, among other interesting technologies, event streaming is also very useful to implement the insides of a data product. Naturally, you’d use streaming tools like ksqlDB and Kafka Streams to process any streaming data that enters or is created within a data product, and streaming data that leaves and thus is shared with the outside world in the data mesh.
• #27: Likewise, if you use non-streaming systems like databases and cloud storage services within a data product, you can “streamify” them via Kafka connectors. For example, as shown here, you can feed real-time updates into relational database like MySQL from upstream data products to keep the database current with the latest business information, and any existing applications that interact with the database can continue to work as usual. And similarly, any changes to the database itself—like updates to a customer profile—can be captured and shared as event streams with the outside world in the data mesh.
• #28: This brings us to the last point: In which way should we share our data product’s streams within the mesh? The two main considerations here are the use of schemas and the use of versioning. When you publish a stream, you should use schemas with appropriate backwards and/or forwards compatibility. This ensures, for example, that when you add a new data field to a stream, that any existing consumers of that stream can continue to operate without having to make any changes. However, if you do need to introduce breaking changes to the schema of a stream, then use versioned streams. For example, if you are being required to obfuscate certain data fields moving forward, which likely DOES break existing consumers, then publish a new version of your stream, and deprecate the original version.
• #35: This is the architecture of most event streaming applications that have any UI component (to show sth to a human). Maybe you streaming-import data from other databases into Kafka, then perform stream processing on top of that, and then use Kafka Connect to streaming-export the results into a traditional database, against which a UI application runs lookup queries or other tasks. This architecture is good, and many great applications have been built with it. But: it does suffer from unnecessary complexity.
• #36: ksqlDB turns the data mesh into something you can query, while still having all the benefits of being decentralized
• #39: Let’s talk about the last principle: Federated governance.
• #40: Like principle 1 (Decentralization), this principle is largely an organizational concern. The objective here is to ensure that all these independent, autonomous teams and data products within the mesh can work actually together—that is, we want to ensure that the various constituents in the mesh actually mesh so that we generate network effects for all this data we share across the organization. Predominantly, this task is about establishing global standards like data governance, and applying these across the full mesh. So, even though we started with decentralization in principles 1 and 2, we now want to strike the right balance between decentralization and centralization to tie everything together. This isn’t easy, and in practice the execution will likely differ from company to company. Let’s take a look at concrete examples.
• #41: One important task is to ensure that “the same” data or data attribute is represented identically across different domains. Otherwise we can’t join or correlate, for example, data about our customers across the various domains like Inventory, Orders, and Shipments. We won’t have the time in this talk to cover the how to define and agree upon such data models. But what we can say is that—once you have them—then data contracts, schemas, and schema registries will help you to implement and enforce such standards throughout the data mesh.
• #42: Another example is how data products should detect and recover from errors. Here, we can use common strategies like application logs, data profiling, and data lineage to unearth errors in the mesh. Streams are very useful here because they capture live AND historical data in a sequence of events, which let you identify cause-and-effect relationships much more easily—particularly if you can join and correlate streams from different domains as discussed in the slide before. Also, streams let you then recover and fix any such errors by replaying and reprocessing historical data.
• #43: In this final example, we can see again that there are lots of data streams within a data mesh. These data streams may span across systems, data centers, clouds, and so on. For the purpose of tracking data lineage, we ideally want to cover the full mesh, so we must follow the data. Event streaming is again a key technology to implement this in practice, because it lets you track data-in-motion all the way from its origins to intermediate and to the final destinations.
• #44: Before we wrap up, let’s talk about some further considerations for federated governance. In my opinion, the most important recommendation is having the right mindset: be pragmatic! Don’t expect a governance system to be perfect. It only needs to be sufficiently good. There will always be parts of a data architecture that is not covered or that was just recently changed. So governance is really more an process than a technology. Lastly, we talked quite a bit about decentralization vs. centralization. Governance is on the centralized side. So while you probably want to standardize a number of things, please be aware of the downsides of centralized DATA MODELS, as they will eventually become too slow to change. If you really need something like this, use processes and tools like GitHub to collaborate efficiently, and have good luck! :-)
• #45: Michael: As Ben just said, the data mesh is a concept, and you should really think of [building a data mesh in your company] as a journey. You didn’t have one last week, and you will not have a finished data mesh next week. So get the necessary management commitment, and then ideally start small with the first concrete use cases, and iterate from there. And finally, while the data mesh is certainly a valuable concept, it’s not the holy grail. The data mesh sits next to other useful concepts, like microservices and DDD. And with its focus on analytical data, the data mesh is but one of the ways you can apply event streaming & Kafka to drive your business, just like you can use event streaming with microservices to implement the more operational systems. Apply these concepts in the way you see fit so that you can make technology useful in your company. [OUTRO] Ben and I hope you enjoyed our session on data mesh and event streaming, and we wish you a lot of fun with the rest of Kafka Summit.
• #48: [OUTRO] So, what’s a Data Product then?