Domain Driven Data: Apache Kafka® and the Data Mesh

Apr 1, 20212 likes404 views

James Gollan, Confluent, Senior Solutions Engineer From digital banking to industry 4.0 the nature of business is changing. Increasingly businesses are becoming software. And the lifeblood of software is data. Dealing with data at the enterprise level is tough, and their have been some missteps along the way. This session will consider the increasingly popular idea of a 'data mesh' - the problems it solves and, perhaps most importantly, how an event streaming platform forms the bedrock of this new paradigm. Recording to be available cnfl.io/meetup-hub https://www.meetup.com/KafkaMelbourne/events/277076626/

Domain Driven Data
Apache Kafka and the Data
Mesh
James Gollan, Senior Solutions Engineer at Conﬂuent

Why are we talking about Data
Mesh at a Kafka meetup?

Copyright 2021, Conﬂuent, Inc. All rights reserved. This document may not be reproduced in any manner without the express written permission of Conﬂuent, Inc.
Producer Guarantees
P
Broker 1 Broker 2 Broker 3
Topic1
partition1
Leader Follower
Topic1
partition1
Topic1
partition1
Producer Properties
acks=all
min.insync.replica=2
ack

Data mesh is an attempt to solve
these problems

This seems kinda familiar - where
have I heard this before....
Could it be microservices?

Domain Driven Data: Apache Kafka® and the Data Mesh

Apache Kafka is the de facto standard for data streaming to process data in motion. With its significant adoption growth across all industries, I get a very valid question every week: When NOT to use Apache Kafka? What limitations does the event streaming platform have? When does Kafka simply not provide the needed capabilities? How to qualify Kafka out as it is not the right tool for the job? This session explores the DOs and DONTs. Separate sections explain when to use Kafka, when NOT to use Kafka, and when to MAYBE use Kafka. No matter if you think about open source Apache Kafka, a cloud service like Confluent Cloud, or another technology using the Kafka protocol like Redpanda or Pulsar, check out this slide deck. A detailed article about this topic: https://www.kai-waehner.de/blog/2022/01/04/when-not-to-use-apache-kafka/

Deep Dive into Apache Kafkaconfluent

In the last few years, Apache Kafka has been used extensively in enterprises for real-time data collecting, delivering, and processing. In this presentation, Jun Rao, Co-founder, Confluent, gives a deep dive on some of the key internals that help make Kafka popular. - Companies like LinkedIn are now sending more than 1 trillion messages per day to Kafka. Learn about the underlying design in Kafka that leads to such high throughput. - Many companies (e.g., financial institutions) are now storing mission critical data in Kafka. Learn how Kafka supports high availability and durability through its built-in replication mechanism. - One common use case of Kafka is for propagating updatable database records. Learn how a unique feature called compaction in Apache Kafka is designed to solve this kind of problem more naturally.

From Message to Cluster: A Realworld Introduction to Kafka Capacity Planningconfluent

6 Nines: How Stripe keeps Kafka highly-available across the globe with Donny ...HostedbyConfluent

Availability is a key metric for any Kafka deployment, but when every event is critical the system must be centered around keeping publishers and consumers highly available, even when a Kafka cluster goes down. At Stripe our core business relies on Kafka, and as we outgrew a single Kafka cluster we had to build a multi-cluster system which would fit our needs while supporting a target of 99.9999% availability for our most critical use cases. In this talk we’ll discuss our solution to this problem: an in-house proxy layer and multi-cluster toplogy which we’ve built and operated over the past 3 years. Our proxy layer enables multiple Kafka clusters to work in coordination across the globe, while hitting our ambitious availability targets and providing clean client abstractions. In this talk we’ll discuss how our Kafka deployment provides: availability for both publishers and consumers in the face of cluster outages, increased security and observability, simplified cluster maintenance, and global routing for constraints such as data locality. We’ll highlight the benefits & tradeoffs of our approach, the design of our proxy layer, Kafka configuration decisions, and where we’re planning to go from here.

Stream processing using KafkaKnoldus Inc.

Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...HostedbyConfluent

At Stripe, we operate a general ledger modeled as double-entry bookkeeping for all financial transactions. Warehousing such data is challenging due to its high volume and high cardinality of unique accounts. aFurthermore, it is financially critical to get up-to-date, accurate analytics over all records. Due to the changing nature of real time transactions, it is impossible to pre-compute the analytics as a fixed time series. We have overcome the challenge by creating a real time key-value store inside Pinot that can sustain half million QPS with all the financial transactions. We will talk about the details of our solution and the interesting technical challenges faced.

Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward

Flink Forward San Francisco 2022. Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover. by Mason Chen

Amazon S3 Best Practice and Tuning for Hadoop/Spark in the CloudNoritaka Sekiyama

This document provides an overview and summary of Amazon S3 best practices and tuning for Hadoop/Spark in the cloud. It discusses the relationship between Hadoop/Spark and S3, the differences between HDFS and S3 and their use cases, details on how S3 behaves from the perspective of Hadoop/Spark, well-known pitfalls and tunings related to S3 consistency and multipart uploads, and recent community activities related to S3. The presentation aims to help users optimize their use of S3 storage with Hadoop/Spark frameworks.

Scaling your Data Pipelines with Apache Spark on KubernetesDatabricks

There is no doubt Kubernetes has emerged as the next generation of cloud native infrastructure to support a wide variety of distributed workloads. Apache Spark has evolved to run both Machine Learning and large scale analytics workloads. There is growing interest in running Apache Spark natively on Kubernetes. By combining the flexibility of Kubernetes and scalable data processing with Apache Spark, you can run any data and machine pipelines on this infrastructure while effectively utilizing resources at disposal. In this talk, Rajesh Thallam and Sougata Biswas will share how to effectively run your Apache Spark applications on Google Kubernetes Engine (GKE) and Google Cloud Dataproc, orchestrate the data and machine learning pipelines with managed Apache Airflow on GKE (Google Cloud Composer). Following topics will be covered: – Understanding key traits of Apache Spark on Kubernetes- Things to know when running Apache Spark on Kubernetes such as autoscaling- Demonstrate running analytics pipelines on Apache Spark orchestrated with Apache Airflow on Kubernetes cluster.

Introducing the Apache Flink Kubernetes OperatorFlink Forward

Flink Forward San Francisco 2022. The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples." by Thomas Weise

Apache Spark on K8S Best Practice and Performance in the CloudDatabricks

Kubernetes As of Spark 2.3, Spark can run on clusters managed by Kubernetes. we will describes the best practices about running Spark SQL on Kubernetes upon Tencent cloud includes how to deploy Kubernetes against public cloud platform to maximum resource utilization and how to tune configurations of Spark to take advantage of Kubernetes resource manager to achieve best performance. To evaluate performance, the TPC-DS benchmarking tool will be used to analysis performance impact of queries between configurations set. Speakers: Junjie Chen, Junping Du

From Zero to Hero with Kafka Connectconfluent

Watch this talk here: https://www.confluent.io/online-talks/from-zero-to-hero-with-kafka-connect-on-demand Integrating Apache Kafka® with other systems in a reliable and scalable way is often a key part of a streaming platform. Fortunately, Apache Kafka includes the Connect API that enables streaming integration both in and out of Kafka. Like any technology, understanding its architecture and deployment patterns is key to successful use, as is knowing where to go looking when things aren't working. This talk will discuss the key design concepts within Apache Kafka Connect and the pros and cons of standalone vs distributed deployment modes. We'll do a live demo of building pipelines with Apache Kafka Connect for streaming data in from databases, and out to targets including Elasticsearch. With some gremlins along the way, we'll go hands-on in methodically diagnosing and resolving common issues encountered with Apache Kafka Connect. The talk will finish off by discussing more advanced topics including Single Message Transforms, and deployment of Apache Kafka Connect in containers.

Running Apache Spark on Kubernetes: Best Practices and PitfallsDatabricks

Kafka replication apachecon_2013Jun Rao

The document discusses intra-cluster replication in Apache Kafka, including its architecture where partitions are replicated across brokers for high availability. Kafka uses a leader and in-sync replicas approach to strongly consistent replication while tolerating failures. Performance considerations in Kafka replication include latency and durability tradeoffs for producers and optimizing throughput for consumers.

Building a Streaming Microservice Architecture: with Apache Spark Structured ...Databricks

Kafka Connect - debeziumKasun Don

Apache Kafka® and API Managementconfluent

Kafka and Confluent are nice, but what about the integration with public clouds like Azure. Or even better, to integrate Kafka and Confluent with a managed API management like Azure API Gateway. In this talk I will show you how it is possible to integrate an event streaming platform like Confluent into an enterprise API Management and different other services to build up a lambda based data platform architecture.

Change Data Streaming Patterns for Microservices With Debezium confluent

(Gunnar Morling, RedHat) Kafka Summit SF 2018 Debezium (noun | de·be·zi·um | /dɪ:ˈbɪ:ziːəm/): secret sauce for change data capture (CDC) streaming changes from your datastore that enables you to solve multiple challenges: synchronizing data between microservices, gradually extracting microservices from existing monoliths, maintaining different read models in CQRS-style architectures, updating caches and full-text indexes and feeding operational data to your analytics tools Join this session to learn what CDC is about, how it can be implemented using Debezium, an open source CDC solution based on Apache Kafka and how it can be utilized for your microservices. Find out how Debezium captures all the changes from datastores such as MySQL, PostgreSQL and MongoDB, how to react to the change events in near real time and how Debezium is designed to not compromise on data correctness and completeness also if things go wrong. In a live demo we’ll show how to set up a change data stream out of your application’s database without any code changes needed. You’ll see how to sink the change events into other databases and how to push data changes to your clients using WebSockets.

Getting Started with Confluent Schema Registryconfluent

Aggregated queries with Druid on terrabytes and petabytes of dataRostislav Pashuto

The document discusses Druid, an open-source distributed column-oriented data store designed for low latency queries on large datasets. It outlines Druid's capabilities for real-time ingestion, aggregation queries in sub-seconds, and storing petabytes of historical data. Examples are given of companies like Netflix and PayPal using Druid at large scales to analyze streaming data. The key components, data formats, and query types of Druid are described.

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...Databricks

Building data product requires having Lambda Architecture to bridge the batch and streaming processing. AirStream is a framework built on top of Apache Spark to allow users to easily build data products at Airbnb. It proved Spark is impactful and useful in the production for mission-critical data products. On the streaming side, hear how AirStream integrates multiple ecosystems with Spark Streaming, such as HBase, Elasticsearch, MySQL, DynamoDB, Memcache and Redis. On the batch side, learn how to apply the same computation logic in Spark over large data sets from Hive and S3. The speakers will also go through a few production use cases, and share several best practices on how to manage Spark jobs in production.

ksqlDB: A Stream-Relational Database Systemconfluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward

Flink Forward San Francisco 2022. At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing. by Xiang Zhang & Pratyush Sharma & Xiaoman Dong

Fundamentals of Apache KafkaChhavi Parasher

Building large scale transactional data lake using apache hudiBill Liu

Data is a critical infrastructure for building machine learning systems. From ensuring accurate ETAs to predicting optimal traffic routes, providing safe, seamless transportation and delivery experiences on the Uber platform requires reliable, performant large-scale data storage and analysis. In 2016, Uber developed Apache Hudi, an incremental processing framework, to power business critical data pipelines at low latency and high efficiency, and helps distributed organizations build and manage petabyte-scale data lakes. In this talk, I will describe what is APache Hudi and its architectural design, and then deep dive to improving data operations by providing features such as data versioning, time travel. We will also go over how Hudi brings kappa architecture to big data systems and enables efficient incremental processing for near real time use cases. Speaker: Satish Kotha (Uber) Apache Hudi committer and Engineer at Uber. Previously, he worked on building real time distributed storage systems like Twitter MetricsDB and BlobStore. website: https://www.aicamp.ai/event/eventdetails/W2021043010

Autoscaling Flink with Reactive ModeFlink Forward

Flink Forward San Francisco 2022. Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo. by Robert Metzger

Kafka 자료 v0.1Hyosang Hong

Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit

This document provides an overview of stream processing with Apache Flink. It discusses the rise of stream processing and how it enables low-latency applications and real-time analysis. It then describes Flink's stream processing capabilities, including pipelining of data, fault tolerance through checkpointing and recovery, and integration with batch processing. The document also summarizes Flink's programming model, state management, and roadmap for further development.

Apache Kafka® and the Data MeshConfluentInc1

This document discusses how Apache Kafka and event streaming fit within a data mesh architecture. It provides an overview of the key principles of a data mesh, including domain-driven decentralization, treating data as a first-class product, a self-serve data platform, and federated governance. It then explains how Kafka's publish-subscribe event streaming model aligns well with these principles by allowing different domains to independently publish and consume streams of data. The document also describes how Kafka can be used to ingest existing data sources, process data in real-time, and replicate data across the mesh in a scalable and interoperable way.

Evolution from EDA to Data Mesh: Data in Motionconfluent

Thoughtworks Zhamak Dehghani observations on these traditional approaches’s failure modes, inspired her to develop an alternative big data management architecture that she aptly named the Data Mesh. This represents a paradigm shift that draws from modern distributed architecture and is founded on the principles of domain-driven design, self-serve platform, and product thinking with Data. In the last decade Apache Kafka has established a new category of data management infrastructure for data in motion that has been leveraged in modern distributed data architectures.

More Related Content

What's hot (20)

Scaling your Data Pipelines with Apache Spark on KubernetesDatabricks

Introducing the Apache Flink Kubernetes OperatorFlink Forward

Apache Spark on K8S Best Practice and Performance in the CloudDatabricks

From Zero to Hero with Kafka Connectconfluent

Running Apache Spark on Kubernetes: Best Practices and PitfallsDatabricks

Kafka replication apachecon_2013Jun Rao

Building a Streaming Microservice Architecture: with Apache Spark Structured ...Databricks

Kafka Connect - debeziumKasun Don

Apache Kafka® and API Managementconfluent

Change Data Streaming Patterns for Microservices With Debezium confluent

Getting Started with Confluent Schema Registryconfluent

Aggregated queries with Druid on terrabytes and petabytes of dataRostislav Pashuto

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...Databricks

ksqlDB: A Stream-Relational Database Systemconfluent

Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward

Fundamentals of Apache KafkaChhavi Parasher

Building large scale transactional data lake using apache hudiBill Liu

Autoscaling Flink with Reactive ModeFlink Forward

Kafka 자료 v0.1Hyosang Hong

Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit

Scaling your Data Pipelines with Apache Spark on KubernetesDatabricks

Introducing the Apache Flink Kubernetes OperatorFlink Forward

Apache Spark on K8S Best Practice and Performance in the CloudDatabricks

From Zero to Hero with Kafka Connectconfluent

Running Apache Spark on Kubernetes: Best Practices and PitfallsDatabricks

Kafka replication apachecon_2013Jun Rao

Building a Streaming Microservice Architecture: with Apache Spark Structured ...Databricks

Kafka Connect - debeziumKasun Don

Apache Kafka® and API Managementconfluent

Change Data Streaming Patterns for Microservices With Debezium confluent

Getting Started with Confluent Schema Registryconfluent

Aggregated queries with Druid on terrabytes and petabytes of dataRostislav Pashuto

Building Data Product Based on Apache Spark at Airbnb with Jingwei Lu and Liy...Databricks

ksqlDB: A Stream-Relational Database Systemconfluent

Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward

Fundamentals of Apache KafkaChhavi Parasher

Building large scale transactional data lake using apache hudiBill Liu

Autoscaling Flink with Reactive ModeFlink Forward

Kafka 자료 v0.1Hyosang Hong

Flexible and Real-Time Stream Processing with Apache FlinkDataWorks Summit

Similar to Domain Driven Data: Apache Kafka® and the Data Mesh (20)

Apache Kafka® and the Data MeshConfluentInc1

Evolution from EDA to Data Mesh: Data in Motionconfluent

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...HostedbyConfluent

Apache Kafka and the Data Mesh | Ben Stopford and Michael Noll, ConfluentHostedbyConfluent

The document discusses the principles of a data mesh architecture using Apache Kafka for event streaming. It describes a data mesh as having four key principles: 1) domain-driven decentralization where each domain owns the data it creates, 2) treating data as a first-class product, 3) providing a self-serve data platform for easy access to real-time and historical data, and 4) establishing federated governance with global standards. Event streaming is presented as a good fit for data meshing due to its scalability, ability to handle real-time and historical data, and immutability. The document provides examples and recommendations for implementing each principle in a data mesh.

Continus sql with sql stream builderTimothy Spann

This document discusses Continuous SQL with SQL Stream Builder. It provides an agenda for a meetup covering an introduction, overview of Apache Flink and SQL Stream Builder, demos and Q&A. SQL Stream Builder allows anyone familiar with SQL to create powerful stream processors for real-time analytics by leveraging Apache Flink's scalable and high performance processing. It provides an interactive interface and deep integration features beyond just UI capabilities.

Comprehensive Guide for Microsoft Fabric to Master Data AnalyticsSparity1

Citi Tech Talk: Messaging Modernizationconfluent

This document discusses moving to an event-driven architecture using Confluent. It begins by outlining some of the limitations of traditional messaging middleware approaches. Confluent provides benefits like stream processing, persistence, scalability and reliability while avoiding issues like lack of structure, slow consumers, and technical debt. The document then discusses how Confluent can help modernize architectures, enable new real-time use cases, and reduce costs through migration. It provides examples of how companies like Advance Auto Parts and Nord/LB have benefitted from implementing Confluent platforms.

Apache Kafka and the Data Mesh | Michael Noll, ConfluentHostedbyConfluent

Data mesh is a relatively recent term that describes a set of principles that good modern data systems uphold. A kind of “microservices” for the data-centric world. While the data mesh is not technology-specific as a pattern, the building of systems that adopt and implement data mesh principles have a relatively long history under different guises. In this talk, we share our recommendations and picks of what every developer should know about building a streaming data mesh with Kafka. We introduce the four principles of the data mesh: domain-driven decentralization, data as a product, self-service data platform, and federated governance. We then cover topics such as the differences between working with event streams versus centralized approaches and highlight the key characteristics that make streams a great fit for implementing a mesh, such as their ability to capture both real-time and historical data. We’ll examine how to onboard data from existing systems into a mesh, modelling the communication within the mesh, how to deal with changes to your domain’s “public” data, give examples of global standards for governance, and discuss the importance of taking a product-centric view on data sources and the data sets they share.

Flink vs. SparkSlim Baltagi

Slim Baltagi – Flink vs. SparkFlink Forward

This document provides an overview of a presentation comparing Apache Flink and Apache Spark. The presentation aims to address marketing claims, confusing statements, and outdated information regarding Flink vs Spark. It outlines key criteria to evaluate the two platforms, such as streaming capabilities, state management, and scalability. The document then directly compares some criteria, such as their support for iterative processing and streaming engines. The presenter hopes this evaluation framework will help others assess Flink and Spark for stream processing use cases.

Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent

As businesses strive to remain at the cutting edge of innovation, the demand for scalable and up-to-date conversational AI solutions has become paramount. Generative AI (GenAI) chatbots that seamlessly integrate into our daily lives and adapt to the ever-evolving nuances of human interaction are crucial. Real-time data plays a pivotal role in ensuring the responsiveness and relevance of these chatbots, empowering them to stay abreast of the latest trends, user preferences, and contextual information.

data-mesh-101.pptxTarekHamdi8

The document provides an overview of data mesh principles and hands-on examples for implementing a data mesh. It discusses key concepts of a data mesh including data ownership by domain, treating data as a product, making data available everywhere through self-service, and federated governance of data wherever it resides. Hands-on examples are provided for creating a data mesh topology with Apache Kafka as the underlying infrastructure, developing data products within domains, and exploring consumption of real-time and historical data from the mesh.

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...HostedbyConfluent

In this session we share our experience of building a real-time data pipelines at Tencent PCG - one that handles 20 trillion daily messages with 700 clusters and 100Gb/s bursting traffic from a single app. We discuss our roadmap of enhancing Kafka to break its limits in terms of scalability, robustness and cost of operation. We first built a proxy layer that aggregates physical clusters in a way agnostic to the clients. While this architecture solves many operational problems, it requires significant development to stay future-proof. With retrospection with our customer and careful study of the ongoing work from the community, we then designed a region federation solution in the broker layer, which allows us to deploy clusters at a much larger scale than previously possible, while at the same time providing better failure recovery and operability. We discuss how we make this development compatible with KIP-500 and KIP-405, and the two KIP (693, 694) that we submitted for discussion.

Confluent Messaging Modernization Forumconfluent

Confluent provides a platform for modernizing enterprise messaging infrastructure by leveraging Kafka. Kafka uses an immutable log to share data across producers and consumers in a scalable, fault-tolerant, and efficient manner. This allows enterprises to build real-time applications and enable data-in-motion across the organization. Confluent offers tools like Schema Registry, ksqlDB, and connectors to help standardize data, build stream processing applications, and integrate Kafka with other systems.

WhatIsData-Blitzpharvener

Citi Tech Talk: Hybrid Cloudconfluent

Developing Actors in Azure with .netMarco Parenzan

Real Time Streaming with Flink & CouchbaseManuel Hurtado

Big Data Companies and Apache SoftwareBob Marcus

The document provides an overview of leading big data companies in 2021 and the Apache Hadoop stack, including related Apache software and the NIST big data reference architecture. It lists over 50 big data companies, including Accenture, Actian, Aerospike, Alluxio, Amazon Web Services, Cambridge Semantics, Cloudera, Cloudian, Cockroach Labs, Collibra, Couchbase, Databricks, DataKitchen, DataStax, Denodo, Dremio, Franz, Gigaspaces, Google Cloud, GridGain, HPE, HVR, IBM, Immuta, InfluxData, Informatica, IRI, MariaDB, Matillion, Melissa Data

Lesson 2Sandra Ahn

Apache Kafka® and the Data MeshConfluentInc1

Evolution from EDA to Data Mesh: Data in Motionconfluent

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...HostedbyConfluent

Apache Kafka and the Data Mesh | Ben Stopford and Michael Noll, ConfluentHostedbyConfluent

Continus sql with sql stream builderTimothy Spann

Comprehensive Guide for Microsoft Fabric to Master Data AnalyticsSparity1

Citi Tech Talk: Messaging Modernizationconfluent

Apache Kafka and the Data Mesh | Michael Noll, ConfluentHostedbyConfluent

Flink vs. SparkSlim Baltagi

Slim Baltagi – Flink vs. SparkFlink Forward

Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...confluent

data-mesh-101.pptxTarekHamdi8

Enhancing Apache Kafka for Large Scale Real-Time Data Pipeline at Tencent | K...HostedbyConfluent

Confluent Messaging Modernization Forumconfluent

WhatIsData-Blitzpharvener

Citi Tech Talk: Hybrid Cloudconfluent

Developing Actors in Azure with .netMarco Parenzan

Real Time Streaming with Flink & CouchbaseManuel Hurtado

Big Data Companies and Apache SoftwareBob Marcus

Lesson 2Sandra Ahn

More from confluent (20)

Webinar Think Right - Shift Left - 19-03-2025.pptxconfluent

Migration, backup and restore made easy using Kannikaconfluent

In this presentation, you’ll discover how easily you can migrate data from any Kafka-compatible event hub to Confluent using Kannika’s intuitive self-service interface. We’ll guide you through the process, showing how the same approach can be applied to define specific event data sets and effortlessly spin up secure environments for demos, testing, or other purposes. You’ll also learn how to back up event data in just a few steps by transferring compressed data to the cloud storage location of your choice. In addition, we’ll demonstrate how to restore filtered datasets of topics, ensuring quick recovery and maintaining business continuity when needed.

Five Things You Need to Know About Data Streaming in 2025confluent

Topics that Peter covers: Tapping into the Potential of Data Products: Data drives some of today's most important business use cases. Data products enable instant access to reliable and trustworthy data by eliminating the data mess created by point-to-point connections. The Need to Tap into 'Quick Thinking': The C-level has to reorient itself so it doesn't become the bottleneck to adaptability in a data-driven world. Nine in 10 (90%) business leaders say they must now react in real-time. Learn what you can do to provide executive access to real-time data to enable 'Quick Thinking.' Rise Above Data Hurdles: Discover how to enforce governance at data production. Reestablishing trustworthiness later is almost always harder, so investing in data tools that solve business problems rather than add to them is essential. Paradigm to Shift Left: Shift Left is a new paradigm for processing and governing data at any scale, complexity, and latency. Shift Left moves the processing and governance of data closer to the source, enabling organisations to build their data once, build it right and reuse it anywhere within moments of its creation. The Need for a Strategic View: The positive correlation between data streaming maturity and significant business returns underscores the importance of a long-term, strategic view of data streaming investments. It also highlights the value of advancing beyond initial, siloed use cases to a more integrated approach that leverages data streaming across the enterprise.

Data in Motion Tour Seoul 2024 - Keynoteconfluent

Data in Motion Tour Seoul 2024 - Roadmap Democonfluent

From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...confluent

In this presentation, we’ll demonstrate how Confluent and Lightstreamer come together to tackle the last-mile challenge of extending your Kafka architecture to web and mobile platforms. Learn how to effortlessly build real-time web applications within minutes, subscribing to Kafka topics directly from your web pages, with unmatched low latency and high scalability. Explore how Confluent's leading Kafka platform and Lightstreamer's intelligent proxy work seamlessly to bridge Kafka with the internet frontier, delivering data in real-time.

Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...confluent

Data in Motion Tour 2024 Riyadh, Saudi Arabiaconfluent

Data streaming platforms are becoming increasingly important in today’s fast-paced world. From retail giants who need to monitor inventory levels to ensure stores never run out of items, to new-age, innovative banks who are building out-of-the-box banking solutions for traditional retail banks, data streaming platforms are at the centre, powering these workflows. Data streaming platforms connect all your applications, systems, and teams with a shared view of the most up-to-date, real-time data. From Gen AI, stream governance to stream processing - it’s these cutting edge developments that will be featured during the day.

Build a Real-Time Decision Support Application for Financial Market Traders w...confluent

Strumenti e Strategie di Stream Governance con Confluent Platformconfluent

Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeksconfluent

As businesses strive to stay at the forefront of innovation, the ability to quickly develop scalable Generative AI (GenAI) applications is essential. Join us for an exclusive webinar featuring MIA Platform, MongoDB, and Confluent, where you'll learn how to compose GenAI apps with real-time data integration in a fraction of the time. Discover how these three powerful platforms work together to ensure applications remain responsive, relevant, and adaptive to user preferences and contextual changes. Our experts will guide you through leveraging MIA Platform's microservices architecture and low-code development, MongoDB's flexibility, and Confluent's stream processing capabilities. Experience live demonstrations and practical insights that will transform your approach to AI-driven app development, enabling you to accelerate your development process from weeks to mere minutes. Don't miss this opportunity to keep your business at the cutting edge.

Building Real-Time Gen AI Applications with SingleStore and Confluentconfluent

Unlocking value with event-driven architecture by Confluentconfluent

Sfrutta il potere dello streaming di dati in tempo reale e dei microservizi basati su eventi per il futuro di Sky con Confluent e Kafka®. In questo tech talk esploreremo le potenzialità di Confluent e Apache Kafka® per rivoluzionare l'architettura aziendale e sbloccare nuove opportunità di business. Ne approfondiremo i concetti chiave, guidandoti nella creazione di applicazioni scalabili, resilienti e fruibili in tempo reale per lo streaming di dati. Scoprirai come costruire microservizi basati su eventi con Confluent, sfruttando i vantaggi di un'architettura moderna e reattiva. Il talk presenterà inoltre casi d'uso reali di Confluent e Kafka®, dimostrando come queste tecnologie possano ottimizzare i processi aziendali e generare valore concreto.

Il Data Streaming per un’AI real-time di nuova generazioneconfluent

Per costruire applicazioni di AI affidabili, sicure e governate occorre una base dati in tempo reale altrettanto solida. Ancor più quando ci troviamo a gestire ingenti flussi di dati in continuo movimento. Come arrivarci? Affidati a una vera piattaforma di data streaming che ti permetta di scalare e creare rapidamente applicazioni di AI in tempo reale partendo da dati affidabili. Scopri di più! Non perdere il nostro prossimo webinar durante il quale avremo l’occasione di: • Esplorare il paradigma della GenAI e di come questa nuova tecnnologia sta rimodellando il panorama aziendale, rispondendo alla necessità di offrire un contesto e soluzioni in tempo reale che soddisfino le esigenze della tua azienda. • Approfondire le incertezze del panorama dell'AI in evoluzione e l'importanza cruciale del data streaming e dell'elaborazione dati. • Vedere in dettaglio l'architettura in continua evoluzione e il ruolo chiave di Kafka e Confluent nelle applicazioni di AI. • Analizzare i vantaggi di una piattaforma di streaming dei dati come Confluent nel collegare l'eredità legacy e la GenAI, facilitando lo sviluppo e l’utilizzo di AI predittive e generative.

Break data silos with real-time connectivity using Confluent Cloud Connectorsconfluent

Building API data products on top of your real-time data infrastructureconfluent

This talk and live demonstration will examine how Confluent and Gravitee.io integrate to unlock value from streaming data through API products. You will learn how data owners and API providers can document, secure data products on top of Confluent brokers, including schema validation, topic routing and message filtering. You will also see how data and API consumers can discover and subscribe to products in a developer portal, as well as how they can integrate with Confluent topics through protocols like REST, Websockets, Server-sent Events and Webhooks. Whether you want to monetize your real-time data, enable new integrations with partners, or provide self-service access to topics through various protocols, this webinar is for you!

Speed Wins: From Kafka to APIs in Minutesconfluent

Evolving Data Governance for the Real-time Streaming and AI Eraconfluent

Catch the Wave: SAP Event-Driven and Data Streaming for the Intelligence Ente...confluent

Santander Stream Processing with Apache Flinkconfluent