Event & Data Mesh as a Service: Industrializing Microservices in the Enterprise | Pavan Keshavamurthy, Platformatory

Aug 3, 20211 like905 views

Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better. The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka. We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices. Some potential takeaways for the discerning audience: 1. Opportunities in a platform approach to Event Driven Architecture in the enterprise 2. Adopting a product mindset around Data & Event Streams 3. Seeking harmony with allied enterprise applications

Data & Eventing Mesh
The next frontier of modernization
Pavan Keshavamurthy
Kafka Summit APAC 2021

Whoami?
● Founder
- OpenDevX → www.opendevx.io
- Platformatory → www.platformatory.io
● Technologist, Programmer
- Distributed Systems
- Modernization
● OSS Contributor
● Hobby Genomics
● Classical music
● Amateur Astronomy
● ..and a lot of other things

Layers & Silos
Big Data (Warehouse, Lake, Mart)
Transactional DB Transactional DB Transactional DB
{API Management}
no|new-sql
<Service/> <Service/> <Service/> <Service/>
<Service/> <Service/> <Service/> <Service/>
ETL / Data
Engg
Analytics & BI Advanced
Analytics (ML)
Data
Governance
…... …... …...
no|new-sql no|new-sql no|new-sql
ESB | MoM | BPM

The Reality.
That Middleware... That Warehouse... That Lake...
Some things about that didn’t scale very well in practice.

Layers & Silos
Big Data (Warehouse, Lake, Mart)
Transactional DB Transactional DB Transactional DB
no|new-sql
ETL / Data
Engg
Analytics & BI Advanced
Analytics (ML)
Data
Governance
…... …... …...
no|new-sql no|new-sql no|new-sql
The biggest impediment = Hyperspecialization Silos

“A database is a giant global
variable that pervades your
code. How, exactly, is that a
good thing?”
“We pride ourselves on
creating the biggest monolith
of them all, the big data
platform”
https://twitter.com/allenholub/status/140379691815
3531393
https://martinfowler.com/articles/data-monolith-to-mesh.html

- Decompose data (just like
services) by product ←→
domain & context bounds
- Evolve a Reactive Core
(Event Sourcing & CQRS)
Break the Monolith: Let the data (& events) flow
- Mesh Topology over
layered architecture

Real-Time Event
Store
Data Flow &
Observability
ProductC
atalog
Data
Security &
Governance
Service Mesh
Observability
(OpenTracing)
Product
Catalog
Security &
Governance
Data & Event Mesh Control Plane
API Mgmt & Service Mesh Control Plane
Traffic &
Routing
Bounded
Context
Bounded
Context
Bounded
Context

The Data Product
{{Data}}
Bulk /Raw Data Sets
(REST / File Transfer)
Querying, Aggregation
(OData, GraphQL..)
Real Time Events
(AsyncAPI, CloudEvents, *RPC
Polyglot Pipeline Composition
(Beam, DAG…)
X X X
X X X
Developer
BI / Analyst
X X X
X X
… {as a service}
Stakeholder Use-cases

Closing Thoughts
- Culture eats strategy for breakfast
- Small, distributed & self contained → resilient in
the long haul
- Not about tooling alone. People & process
transformation crucial
- Data Ops is the new DevOps - Embed Domain
Data Product ownership into cross functional
teams

Thank you
pavan@platformatory.com
www.platformatory.io

Apache Kafka users who want to leverage Google Cloud Platform's (GCPs) data analytics platform and open source hosting capabilities can bridge their existing Kafka infrastructure on-premise or in other clouds to GCP using Confluent's replicator tool and managed Kafka service on GCP. Using actual customer examples and a reference architecture, we'll showcase how existing Kafka users can stream data to GCP and use it in popular tools like Apache Beam on Dataflow, BigQuery, Google Cloud Storage (GCS), Spark on Dataproc, and Tensorflow for data warehousing, data processing, data storage, and advanced analytics using AI and ML.

Keynote: Jay Kreps, Confluent | Kafka ♥ Cloud | Kafka Summit 2020confluent

Cassandra Lunch #23: Lucene Based Indexes on CassandraAnant Corporation

During this week's Cassandra Lunch we covered packaged and DIY methods for Lucene-based indexes on Cassandra; as well as some pros and cons for using Lucene Based Indexes on Cassandra. Accompanying Blog: https://blog.anant.us/apache-cassandra-lunch-23-lucene-based-indexes-on-cassandra/ Cassandra Lunch Recording: https://youtu.be/Z0NXWmZAB8s Additional Resources: https://docs.datastax.com/en/dse/6.0/dse-dev/datastax_enterprise/search/searchTOC.html https://github.com/instaclustr/cassandra-lucene-index https://github.com/strapdata/elassandra https://github.com/tjake/Solandra Join Cassandra Lunch Weekly at 12 PM EST Every Wednesday: https://www.meetup.com/Cassandra-DataStax-DC/events/ Awesome Cassandra: https://github.com/Anant/awesome-cassandra Follow Us and Reach Us At: Anant: https://www.anant.us/ Cassandra.Link: https://cassandra.link/ Email: [email protected] LinkedIn: https://www.linkedin.com/company/anant/ Twitter: https://twitter.com/anantcorp Eventbrite: https://www.eventbrite.com/o/anant-1072927283 Facebook: https://www.facebook.com/AnantCorp/

DataOps Automation for a Kafka Streaming Platform (Andrew Stevenson + Spiros ...HostedbyConfluent

DataOps challenges us to build data experiences in a repeatable way. For those with Kafka, this means finding a means of deploying flows in an automated and consistent fashion. The challenge is to make the deployment of Kafka flows consistent across different technologies and systems: the topics, the schemas, the monitoring rules, the credentials, the connectors, the stream processing apps. And ideally not coupled to a particular infrastructure stack. In this talk we will discuss the different approaches and benefits/disadvantages to automating the deployment of Kafka flows including Git operators and Kubernetes operators. We will walk through and demo deploying a flow on AWS EKS with MSK and Kafka Connect using GitOps practices: including a stream processing application, S3 connector with credentials held in AWS Secrets Manager.

Streaming Data Lakes using Kafka Connect + Apache Hudi | Vinoth Chandar, Apac...HostedbyConfluent

Apache Hudi is a data lake platform, that provides streaming primitives (upserts/deletes/change streams) on top of data lake storage. Hudi powers very large data lakes at Uber, Robinhood and other companies, while being pre-installed on four major cloud platforms. Hudi supports exactly-once, near real-time data ingestion from Apache Kafka to cloud storage, which is typically used in-place of a S3/HDFS sink connector to gain transactions and mutability. While this approach is scalable and battle-tested, it can only ingest data in mini batches, leading to lower data freshness. In this talk, we introduce a Kafka Connect Sink Connector for Apache Hudi, which writes data straight into Hudi's log format, making the data immediately queryable, while Hudi's table services like indexing, compaction, clustering work behind the scenes, to further re-organize for better query performance.

Should You Read Kafka as a Stream or in Batch? Should You Even Care? | Ido Na...HostedbyConfluent

Should you consume Kafka in a stream OR batch? When should you choose each one? What is more efficient, and cost effective? In this talk we’ll give you the tools and metrics to decide which solution you should apply when, and show you a real life example with cost & time comparisons. To highlight the differences, we’ll dive into a project we’ve done, transitioning from reading Kafka in a stream to reading it in batch. By turning conventional thinking on its head and reading our multi-petabyte Kafka stream in batch using Spark and Airflow, we’ve achieved a huge cost reduction of 65% while at the same time getting a more scalable and resilient solution. We’ll explore the tradeoffs and give you the metrics and intuition you’ll need to make such decisions yourself. We’ll cover: Costs of processing in stream compared to batch Scaling up for bursts and reprocessing Making the tradeoff between wait times and costs Recovering from outages And much more…

Lessons Learned - Monitoring the Data Pipeline at HuluDataWorks Summit

This document summarizes lessons learned about monitoring a data pipeline at Hulu. It discusses how the initial monitoring approach had some issues from the perspectives of users and detecting problems. A new approach is proposed using a graph data structure to provide contextual troubleshooting that connects any issues to their impacts on business units and user needs. This approach aims to make troubleshooting easier by querying the relationships between different components and resources. Small independent services would also be easier to create and maintain within this approach.

Cloud-Based Event Stream Processing Architectures and Patterns with Apache Ka...HostedbyConfluent

The Apache Kafka ecosystem is very rich with components and pieces that make for designing and implementing secure, efficient, fault-tolerant and scalable event stream processing (ESP) systems. Using real-world examples, this talk covers why Apache Kafka is an excellent choice for cloud-native and hybrid architectures, how to go about designing, implementing and maintaining ESP systems, best practices and patterns for migrating to the cloud or hybrid configurations, when to go with PaaS or IaaS, what options are available for running Kafka in cloud or hybrid environments and what you need to build and maintain successful ESP systems that are secure, performant, reliable, highly-available and scalable.

Streaming Data in the Cloud with Confluent and MongoDB Atlas | Robert Walters...HostedbyConfluent

This document discusses streaming data between Confluent Cloud and MongoDB Atlas. It provides an overview of MongoDB Atlas and its fully managed database capabilities in the cloud. It then demonstrates how to stream data from a Python generator application to MongoDB Atlas using Confluent Cloud and its connectors. The presentation concludes by providing a reference architecture for connecting Confluent Platform to MongoDB.

The evolution of the big data platform @ Netflix (OSCON 2015)Eva Tse

The document summarizes the evolution of Netflix's big data platform to meet the challenges of their growing scale. Key points include: - Netflix now has over 65 million members in over 50 countries and supports over 1000 devices. They stream over 10 billion hours of content per quarter. - Their traditional business intelligence stack could no longer meet the demands of scale. They transitioned to using AWS services like S3 for storage and open source tools like Kafka, Cassandra, and Parquet to enable real-time analytics and machine learning on their massive data volumes. - Netflix has adopted an open source-first strategy and contributes back to the community as their own tools evolve to meet processing needs and achieve the necessary scale to

Druid + Kafka: transform your data-in-motion to analytics-in-motion | Gian Me...HostedbyConfluent

Apache Druid is a high-performance distributed analytics store for modern analytics applications. It supports ingesting millions of events per second and sub-second query processing. Druid supports various types of data sources for ingestion, including Apache Kafka. You can immediately query on stream events once they get ingested into Druid. Since Kafka provides scalable and robust data delivery while Druid supports advanced complex analysis on streams, Kafka and Druid are widely used together for BI and operational analytics use cases, which require interactivity, scalability, real-time, and performance. This talk is based on our real-world experiences building out streaming analytics stacks powering production use cases across many industries.

Distributed Data Quality - Technical Solutions for Organizational ScalingJustin Cunningham

The document discusses Yelp's distributed data architecture and quality solutions for organizational scaling. It describes how Yelp connects over 500 engineers across many services through shared data stored in databases like MySQL, Cassandra and Elasticsearch. The data is ingested through Kafka and processed using tools like Flink. Schematizer provides documentation, discovery and ownership of data. It also enables data lineage tracking and auditing to ensure quality as the data is transformed and loaded into data lakes and warehouses. The goal is to provide reliable, up-to-date shared data to align teams and enable autonomy through self-service data access.

Cloud native data platformLi Gao

Cloud Native Data Platform at Fitbit - Fitbit collects 100 TB of user data daily from 30 million users across fitness trackers, smartwatches, and apps for internal teams like data science, research, and customer support as well as enterprise wellness programs. - The data platform includes MySQL, Kafka, Cassandra, S3, EMR, Presto/Spark and supports both batch and real-time workflows across multiple AWS accounts for compliance. - Key challenges included diverse user needs, multiple compliance requirements, and a lean team. The multi-tenant architecture in AWS with fine-grained S3 buckets and IAM roles helps address these challenges.

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...HostedbyConfluent

This document discusses streaming data between Confluent Cloud and MongoDB Atlas. It provides an overview of MongoDB Atlas and its fully managed database capabilities in the cloud. It then demonstrates how to stream data from a Python generator application to MongoDB Atlas using Confluent Cloud and its connectors. The document promotes using MongoDB Atlas as a turnkey database as a service solution and shows how it can be integrated with Confluent Cloud for streaming data workflows.

A unified analytics platform with Kafka and Flink | Stephan Ewen, VervericaHostedbyConfluent

Apache Kafka and Apache Flink together are a winning stack for data analytics that is used by many companies across industries. The two projects complement each other perfectly: Kafka offers a world-class log for event stream storage and transport, while Flink is a powerful system for analytics and applications on top of those event streams. This talk will demonstrate how to use Kafka and Flink together for "unified analytics": Analytics that seamlessly combine processing of real-time data and historic data. Using SQL as the language for our sample applications, we will walk though various scenarios for unified analytics, such as - Running the same query for processing real-time data from Kafka and for batch-accelerated processing of the historic data stored in Kafka. - Writing queries that combine data in Kafka with tables in external systems (like S3) - Switching between streams of historic data (from S3) and real-time streams in Kafka. The audience will learn how combining real-time and historic data is becoming convenient with the combination of Kafka and Flink.

ASPgems - kappa architectureJuantomás García Molina

Kappa Architecture is an alternative to Lambda Architecture that simplifies real-time data processing. It uses a distributed log like Kafka to store all input data immutably to allow reprocessing from the beginning if the processing code changes. This avoids having to maintain separate batch and real-time processing systems. The ASPgems team has implemented Kappa Architecture for several clients using Kafka, Spark Streaming, and Cassandra to provide real-time analytics and metrics in sectors like telecommunications, IoT, insurance, and energy.

AWS Re-Invent 2017 Netflix Keystone SPaaS - Monal Daxini - Abd320 2017Monal Daxini

Over 100 million subscribers from over 190 countries enjoy the Netflix service. This leads to over a trillion events, amounting to 3 PB, flowing through the Keystone infrastructure to help improve customer experience and glean business insights. The self-serve Keystone stream processing service processes these messages in near real-time with at-least once semantics in the cloud. This enables the users to focus on extracting insights, and not worry about building out scalable infrastructure. I’ll share the details about this platform, and our experience building it.

Time Series Analysis Using an Event Streaming PlatformDr. Mirko Kämpf

Advanced time series analysis (TSA) requires very special data preparation procedures to convert raw data into useful and compatible formats. In this presentation you will see some typical processing patterns for time series based research, from simple statistics to reconstruction of correlation networks. The first case is relevant for anomaly detection and to protect safety. Reconstruction of graphs from time series data is a very useful technique to better understand complex systems like supply chains, material flows in factories, information flows within organizations, and especially in medical research. With this motivation we will look at typical data aggregation patterns. We investigate how to apply analysis algorithms in the cloud. Finally we discuss a simple reference architecture for TSA on top of the Confluent Platform or Confluent cloud.

Cloud Connect 2012, Big Data @ NetflixJerome Boulon

Enterprise Metadata IntegrationDr. Mirko Kämpf

Low-latency data applications with Kafka and Agg indexes | Tino Tereshko, Fir...HostedbyConfluent

If a real-time dashboard takes 5 minutes to refresh, it’s not real-time. With data lakes increasingly enabling massive amounts of unprocessed data sets, delivering low-latency analytics is not for the faint-hearted. Learn how to stream massive amounts of data which used to be impossible to handle from Kafka, to serve real-time applications using lake-scale optimized approaches to storage and indexing.

Netflix Big Data Paris 2017Jason Flittner

How Disney+ uses fast data ubiquity to improve the customer experience Martin Zapletal

Disney+ uses Amazon Kinesis to drive real-time actions like providing title recommendations for customers, sending events across microservices, and delivering logs for operational analytics to improve the customer experience. In this session, you learn how Disney+ built real-time data-driven capabilities on a unified streaming platform. This platform ingests billions of events per hour in Amazon Kinesis Data Streams, processes and analyzes that data in Amazon Kinesis Data Analytics for Apache Flink, and uses Amazon Kinesis Data Firehose to deliver data to destinations without servers or code. Hear how these services helped Disney+ scale its viewing experience to tens of millions of customers with the required quality and reliability. Learn more about re:Invent 2020 at http://bit.ly/3c4NSdY

Kafka Summit SF 2017 - Riot's Journey to Global Kafka Aggregationconfluent

This document summarizes Riot Games' journey to establishing a global Kafka aggregation platform. It describes how Riot previously had complex, siloed architectures with operational data challenges. It then outlines how Riot transitioned to using Kafka for scalable, easy aggregation across regions. The document details Riot's current regional collection and global aggregation approach using Kafka Connect. It also discusses challenges encountered and solutions implemented around areas like message replication, partition reassignment, and low latency needs. Finally, it previews Riot's plans for real-time analytics, bi-directional messaging, streaming metrics, and handling of personal information with their Kafka platform.

Money Heist - A Stream Processing Original! | Meha Pandey and Shengze Yu, Net...HostedbyConfluent

Netflix will spend roughly 19B+ on content in 2021 over 100 countries to entertain over 200M subscribers. With this scale of investment in internationalized productions, it is essential to intelligently invest and track the movement of cash, in near-real-time, from pre-production to launch. This involves complex near real time financial calculations, data movement, consistent data views of transactional data of the present and the cash predictions of the future. It needs exactly once processing semantics and idempotency while guaranteeing a low latency serving layer. We leverage kafka to build a scalable and fault tolerance event-based processing engine to provide the right SLA guarantees. We also embrace event-driven data materialization to provide low latency lookups at scale. The aim of this talk is to provide an insight into how we built such a scalable system while embracing a full blown Kappa architecture, with Kafka at the heart of it all.

How Credit Karma Makes Real-Time Decisions For 60 Million Users With Akka Str...Lightbend

In this webinar, Engineering Manager at Credit Karma, Dustin Lyons, discusses how not long ago his team was facing a common challenge shared by many financial services architects and engineering leaders: not only how to move from the offline, batch-mode processing of Big Data to streaming, Fast Data, and how to enable real-time decision making based on the information flowing in from over 60 million members. Dustin reviews how his team migrated away from PHP and successfully implemented Akka Streams with Apache Kafka to ingest, process and route real-time events throughout their data ecosystem. At the end of this presentation, you’ll better understand: * The design considerations for new Fast Data architectures, from streaming to microservices to real-time analysis. * Some lessons learned when it comes to progressing from batch to streaming using Akka, Spark and Kafka * Why Akka’s self-healing actor model and the resilience that it provides is actually what matters most when delivering real-time customer experiences

Kafka Summit SF 2017 - DNS for Data: The Need for a Stream Registryconfluent

This document discusses the need for a stream registry to manage streaming data. It summarizes the evolution of a company's use of streaming from an initial problem in 2015 to developing a microservices architecture in 2016 and introducing a stream registry in 2017 to allow for self-service streams across multiple regions, clouds, and companies. The stream registry provides functions like stream registration, discovery, health monitoring, and throttling to help democratize access to streams.

Salesforce & SAP IntegrationRaymond Gao

Engineering practices in big data storage and processingSchubert Zhang

Data Preparation Offline Data (CRM) Data Warehouse Data Mining Data Visualization 离线批量分析为主 19 大数据服务平台用户应用 Web应用移动应用数据服务平台数据源数据仓库数据处理数据分析数据可视化原则: 1. 提供统一的数据访问接口 2. 支持多种数据源与多种应用 3. 以离线批量分析为主,兼顾实时

Architecting a next generation data platformhadooparchbook

More Related Content

What's hot (19)