Eron Wright - Introducing Flink on Mesos
2 likes659 views
This document introduces Flink on Mesos, including the Flink Master Process and Dispatcher framework. The Flink Master Process acts as both an Application Master and Mesos framework, hosting components like the JobManager and ResourceManager. The Dispatcher is a highly available service for launching Flink clusters through REST. It uploads configuration and files to the Flink Master Process, which then launches TaskManagers. Future directions include dynamic scaling, improved dispatcher capabilities, and Docker/Mesos disk support.
1 of 17
Ad
Recommended
Stephan Ewen - Running Flink Everywhere
Stephan Ewen - Running Flink EverywhereFlink Forward http://flink-forward.org/kb_sessions/running-apache-flink-everywhere-standalone-yarn-mesos-docker-kubernetes-etc/
The world of cluster managers and deployment frameworks is getting complicated. There is zoo of tools to deploy and manage data processing jobs, all of which have different resource management and fault tolerance slightly different. Some tools have a only per-job processes (Yarn, Docker/Kubernetes), while others require some long running processes (Mesos, Standalone). In some frameworks, streaming jobs control their own resource allocation (Yarn, Mesos), while for other frameworks, resource management is handled by external tools (Kubernetes). To be broadly usable in a variety of setups, Flink needs to play well with all these frameworks and their paradigms. This talk describes Flink’s new proposed process and deployment model that will make it work together well with the above mentioned frameworks. The new abstraction is designed to cover a variety of use cases, like isolated single job deployments, sessions of multiple short jobs, and multi-tenant setups.
Eron Wright - Flink Security Enhancements
Eron Wright - Flink Security EnhancementsFlink Forward http://flink-forward.org/kb_sessions/flink-security-enhancements/
Recent security enhancements to Flink make it easy to access secure data and to protect the associated credentials. In this talk we’ll describe and demonstrate the new features, including Kerberos-based access to HDFS and Kafka, transport security (TLS), and service-level authorization which protects your Flink cluster from unauthorized access.
Flink Forward SF 2017: Till Rohrmann - Redesigning Apache Flink’s Distributed...
Flink Forward SF 2017: Till Rohrmann - Redesigning Apache Flink’s Distributed...Flink Forward As stream processing engines become more and more popular and are used in different environments, the demand to support different deployment scenarios increases. Depending on the user's infrastructure, a stream processor might be run on a bare metal cluster in standalone mode, deployed via Apache Yarn and Mesos, or run in a containerized environment. In order to fulfill the requirements of different deployment options and to provide enough flexibility for the future, the Flink community has recently started to redesign Flink's distributed architecture. This talk will explain the limitations of the old architecture and how they are solved with the new design. We will present the new building blocks of a Flink cluster and demonstrate, using the example of Flink's Mesos and Docker support, how they can be combined to run Flink nearly everywhere.
Power of the Log: LSM & Append Only Data Structures
Power of the Log: LSM & Append Only Data Structuresconfluent LSM trees provide an efficient way to structure databases by organizing data sequentially in logs. They optimize for write performance by batching writes together sequentially on disk. To optimize reads, data is organized into levels and bloom filters and caching are used to avoid searching every file. This log-structured approach works well for many systems by aligning with how hardware is optimized for sequential access. The immutability of appended data also simplifies concurrency. This log-centric approach can be applied beyond databases to distributed systems as well.
Flink Forward SF 2017: Cliff Resnick & Seth Wiesman - From Zero to Streami...
Flink Forward SF 2017: Cliff Resnick & Seth Wiesman - From Zero to Streami...Flink Forward Apache Flink provides powerful stream processing capabilities which can allow organizations to move directly from batch to real time analytics, skipping the lambda architecture entirely. However, getting to production is not always as simple as rewriting your job in a new API, but requires rethinking your application design with a stream first mindset. This talk will cover MediaMath’s journey in rebuilding its reporting infrastructure using Apache Flink. We will discuss high level architectural designs when building an extensible reporting platform as well as deep dive into specific technical hurdles. Topics will include managing a Flink cluster on EC2 spot instances, reconciling Flink’s consistency model with S3’s, handling massive data skew as well as tools and techniques for building performant, fault tolerant streaming applications.
Better Kafka Performance Without Changing Any Code | Simon Ritter, Azul
Better Kafka Performance Without Changing Any Code | Simon Ritter, AzulHostedbyConfluent Apache Kafka is the most popular open-source stream-processing software for collecting, processing, storing, and analyzing data at scale. Most known for its excellent performance, low latency, fault tolerance, and high throughput, it's capable of handling thousands of messages per second. For mission-critical applications, how do you ensure that the performance delivered is the performance required? This is especially important as Kafka is written in Java and Scala and runs on the JVM. The JVM is a fantastic platform that delivers on an internet scale. In this session, we'll explore how making changes to the JVM design can eliminate the problems of garbage collection pauses and raise the throughput of applications. For cloud-based Kafka applications, this can deliver both lower latency and reduced infrastructure costs. All without changing a line of code!
What is the State of my Kafka Streams Application? Unleashing Metrics. | Neil...
What is the State of my Kafka Streams Application? Unleashing Metrics. | Neil...HostedbyConfluent "Just as the Apache Kafka Brokers provide JMX metrics to monitor your cluster's health, Kafka Streams provides a rich set of metrics for monitoring your application's health and performance. The metrics to observe for a given use-case of Kafka Streams will vary significantly from application to application. Learning how to build and customize monitoring of those applications will help you maintain a healthy Kafka Streams ecosystem.
Takeaways
* An analysis and overview of the provided metrics, including the new end-to-end metrics of Kafka Streams 2.7.
* See how to extract metrics from your application using existing JMX tooling.
* Walkthrough how to build a dashboard for observing those metrics.
* Explore options of how to add additional JMX resources and Kafka Stream metrics to your application.
* How to verify you built your dashboard correctly by creating a data control set to validate your dashboard.
* Go beyond what you can collect from the Kafka Stream metrics."
Exactly-once Stream Processing with Kafka Streams
Exactly-once Stream Processing with Kafka StreamsGuozhang Wang I will present the recent additions to Kafka to achieve exactly-once semantics (0.11.0) within its Streams API for stream processing use cases. This is achieved by leveraging the underlying idempotent and transactional client features. The main focus will be the specific semantics that Kafka distributed transactions enable in Streams and the underlying mechanics to let Streams scale efficiently.
From Newbie to Highly Available, a Successful Kafka Adoption Tale (Jonathan S...
From Newbie to Highly Available, a Successful Kafka Adoption Tale (Jonathan S...confluent The document discusses running Apache Kafka across multiple datacenters and provides guidance on how to configure Kafka for disaster recovery in a multi-datacenter setup. It recommends having separate Kafka clusters in each datacenter rather than a single cluster spanning datacenters, due to high replication latency over long-distance links. It also outlines three ways to prepare for disaster recovery: running in separate clusters per datacenter, using Kafka MirrorMaker to replicate topics between clusters, and using a shared storage system.
The Log of All Logs: Raft-based Consensus Inside Kafka | Guozhang Wang, Confl...
The Log of All Logs: Raft-based Consensus Inside Kafka | Guozhang Wang, Confl...HostedbyConfluent KRaft is Kafka's implementation of the Raft consensus protocol that replaces Zookeeper for managing Kafka cluster metadata and electing controllers. It stores metadata in a replicated log with a single writer and allows fast controller failovers. The Quorum Controller runs on top of KRaft, caching metadata from the log. It provides ordered, consistent metadata updates and faster broker shutdowns and startups compared to the previous Zookeeper-based controller. KRaft will be available in early access in the next Kafka release while the Zookeeper dependency is deprecated and eventually removed.
What's new in Confluent 3.2 and Apache Kafka 0.10.2 
What's new in Confluent 3.2 and Apache Kafka 0.10.2 confluent With the introduction of connect and streams API in 2016, Apache Kafka is becoming the defacto solution for anyone looking to build a streaming platform. The community continues to add additional capabilities to make it the complete solution for streaming data.
Join us as we review the latest additions in Apache Kafka 0.10.2. In addition, we’ll cover what’s new in Confluent Enterprise 3.2 that makes it possible for running Kafka at scale.
A Journey through the JDKs (Java 9 to Java 11)
A Journey through the JDKs (Java 9 to Java 11)Markus Günther This presentation gives an overview of the most relevant updates and features going from Java 9 to Java 11 from a developer's perspective.
Streaming and Messaging
Streaming and MessagingXin Wang Part-1: Distributed Computing and Storage (Resource Management & Job/Data Management)
Part-2: Streaming and Messaging (Apache Storm & Apache Kafka)
Part-3: Streaming in User Behavior Analysis (Issues & Practices, New Streaming/Messaging Technologies)
Containerizing Distributed Pipes
Containerizing Distributed Pipesinside-BigData.com Hagen Toennies from Gaikai Inc. presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"In this talk we will present how we enable distributed, Unix style programming using Docker and Apache Kafka. We will show how we can take the famous Unix Pipe Pattern and apply it to a Distributed Computing System. We will demonstrate the development of two simple applications with the focus on "Do One Thing and Do It Well." Afterwards we demonstrate how we make these two programs work to together using Apache Kafka. By encapsulating our applications in containers we will also show how that enables us to go from the limited resources of a development machine to cluster of computers in a data center without changing our applications or containers."
Watch the video: http://wp.me/p3RLHQ-goG
Learn more: http://www.hpcadvisorycouncil.com/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Flume and HBase 
Flume and HBase Alexander Alten This document discusses Apache Flume and using its HBase sink. It provides an overview of Flume's architecture, components, and sources. It describes how the HBase sink works, its configuration options, and provides an example configuration for collecting data from a sequential source and storing it in an HBase table.
Robust Operations of Kafka Streams
Robust Operations of Kafka Streamsconfluent (Bill Bejeck, Confluent) Kafka Summit SF 2018
Apache Kafka added a powerful stream processing library in mid-2016, Kafka Streams, which runs on top of Apache Kafka. The community has embraced Kafka Streams with many early adopters, and the adoption rate continues to grow. Large to mid-size organizations have come to rely on Kafka Streams in their production environments. Kafka Streams has many advanced features to make applications more robust.
The point of this presentation is to show users of Kafka Streams some of the latest and greatest features, as well as some that may be advanced, that can make streams applications more resilient. The target audience for this talk are those users already comfortable writing Kafka Streams applications and want to go from writing their first proof-of-concept applications to writing robust applications that can withstand the rigor that running in a production environment demands.
The talk will be a technical deep dive covering topics like:
-Best practices on configuring a Kafka Streams application
-How to meet production SLAs by minimizing failover and recovery times: configuring standby tasks and the pros and cons of having standby replicas for local state
-How to improve resiliency and 24×7 operability: the use of different configurable error handlers, callbacks and how they can be used to see what’s going on inside the application
-How to achieve efficient scalability: a thorough review of the relationship between the number of instances, threads and state stores and how they relate to each other
While this is a technical deep dive, the talk will also present sample code so that attendees can view the concepts discussed in practice. Attendees of this talk will walk away with a deeper understanding of how Kafka Streams works, and how to make their Kafka Streams applications more robust and efficient. There will be a mix of discussion.
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...HostedbyConfluent Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security.
A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka.
In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.
PaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at YelpNathan Handler Capacity planning is a difficult challenge faced by most companies. If you have too few machines, you will not have enough compute resources available to deal with heavy loads. On the other hand, if you have too many machines, you are wasting money. This is why companies have started investing in automatically scaling services and infrastructure to minimize the amount of wasted money and resources.
In this talk, Nathan will describe how Yelp is using PaaSTA, a PaaS built on top of open source tools including Docker, Mesos, Marathon, and Chronos, to automatically and gracefully scale services and the underlying cluster. He will go into detail about how this functionality was implemented and the design designs that were made while architecting the system. He will also provide a brief comparison of how this approach differs from existing solutions.
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto 
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto Docker, Inc. Terraform is a tool for building and safely iterating on infrastructure, while Consul provides service discovery, monitoring and orchestration. In this talk we discuss using Terraform and Consul together to build a Docker-based Service Oriented Architecture at scale. We use Consul to provide the runtime control plane for the datacenter, and Terraform is used to modify the underlying infrastructure to allow for elastic scalability.
Espresso Database Replication with Kafka, Tom Quiggle
Espresso Database Replication with Kafka, Tom Quiggleconfluent This document discusses using Apache Kafka for database replication in LinkedIn's ESPRESSO database system. It provides an overview of ESPRESSO's architecture and transition from per-instance to per-partition replication using Kafka. Key aspects covered include Kafka configuration, the message protocol for ensuring in-order delivery, and checkpointing by the Kafka producer to allow resuming replication from the last committed transaction after failures.
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...confluent The last 5 years, Kafka and Flink have become mature technologies that have allowed us to embrace the streaming paradigm. You can bet on them to build reliable and efficient applications. They are active projects backed by companies using them in production. They have a good community contributing, and sharing experience and knowledge. Kafka and Flink are solid choices if you want to build a data platform that your data scientists or developers can use to collect, process, and distribute data. You can put together Kafka Connect, Kafka, Schema Registry, and Flink. First, you will take care of their deployment. Then, for each case, you will setup each part, and of course develop the Flink job so it can integrate easily with the rest. Looks like a challenging but exciting project, isn't it? In this session, you will learn how you can build such data platform, what are the nitty-gritty of each part, how you can plug them together, in particular how to plug Flink in the Kafka ecosystem, what are the common pitfalls to avoid, and what it requires to be deployed on kubernetes. Even if you are not familiar with all the technologies, there will be enough introduction so you can follow. Come and learn how we can actually cross the streams!
Introduction to Apache Mesos
Introduction to Apache MesosJoe Stein Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers.
Managing multiple event types in a single topic with Schema Registry | Bill B...
Managing multiple event types in a single topic with Schema Registry | Bill B...HostedbyConfluent With Apache Kafka, it's typical to place different events in their own topic. But different event types can be related. Consider customer interactions with an online retailer. The customer searches through the site and clicks on various items before deciding on a final purchase. But businesses can gain insight by processing these events in sequence. Using the event type per topic leaves a lot of work for developers. Is there a better way?
Fortunately, there is. Schema Registry now supports having multiple event types in the same topic. By placing various event types in a single topic, you can now handle different related events in-order. In this presentation, I'll introduce Schema Registry then we'll dive into how it handles multiple event types in a single topic, including examples.
You will learn how and when to apply the multiple event types per topic pattern. Additionally, you'll learn how schema references work in Schema Registry.
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent Speaker: Matthias J. Sax, Software Engineer, Confluent
ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries.
Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems.
https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/
Introduction to Akka-Streams
Introduction to Akka-Streamsdmantula This document provides an introduction to Akka Streams, which implements the Reactive Streams specification. It discusses the limitations of traditional concurrency models and Actor models in dealing with modern challenges like high availability and large data volumes. Reactive Streams aims to provide a minimalistic asynchronous model with back pressure to prevent resource exhaustion. Akka Streams builds on the Akka framework and Actor model to provide a streaming data flow library that uses Reactive Streams interfaces. It allows defining processing pipelines with sources, flows, and sinks and includes features like graph DSL, back pressure, and integration with other Reactive Streams implementations.
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQLconfluent This document introduces KSQL, a streaming SQL engine for Apache Kafka. It provides concise summaries of KSQL's capabilities and how to use it in 3 sentences or less:
KSQL allows users to easily query and transform data in Kafka streams using SQL-like queries. It provides simplicity, flexibility, and scalability compared to directly using Kafka Streams APIs. KSQL can be run in standalone, client-server, or application modes and is well-suited for tasks like streaming ETL, anomaly detection, monitoring, and IoT data processing.
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignMichael Noll Apache Kafka 0.8 basic training (120 slides) covering:
1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka
2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers
3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning
4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps
5. Playing with Kafka using Wirbelsturm
Audience: developers, operations, architects
Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/
Verisign is a global leader in domain names and internet security.
Tools mentioned:
- Wirbelsturm (https://github.com/miguno/wirbelsturm)
- kafka-storm-starter (https://github.com/miguno/kafka-storm-starter)
Blog post at:
http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/
Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!
Federated mesos clusters for global data center designs
Federated mesos clusters for global data center designsKrishna-Kumar The document describes Huawei's approach to federating Mesos clusters across multiple data centers. It proposes a multi-master federation approach where each data center runs its own Mesos master that coordinates with other masters. Gossiper modules in each data center gossip with each other to exchange framework and resource information. When one data center reaches a resource threshold, its gossiper will direct work to other data centers based on a simple policy engine. A demo visualization is shown to illustrate work load balancing during normal and failure scenarios.
Márton Balassi Streaming ML with Flink- 
Márton Balassi Streaming ML with Flink- Flink Forward http://flink-forward.org/kb_sessions/streaming-ml-with-flink/
As continuous big data processing is gaining popularity it naturally implies that there is a need to transition many of the distributed machine learning functionality to a streaming backend. The most common use case is to give streaming predictions based on the model learnt in batch, however in some cases it is beneficial to also update the model on the fly. It is not uncommon that streaming learners need different algorithms than their batch counterparts. The talk discusses the common use cases and the pitfalls of the streaming ML transition through the example of recommender systems. It also offer a dive into the implementation of a Scala library augmenting FlinkML with streaming predictors.
Stephan Ewen - Scaling to large State
Stephan Ewen - Scaling to large StateFlink Forward The document discusses scaling state management in Apache Flink streaming applications to very large state. It describes how Flink uses state sharding and increasing operator parallelism to scale stateful computation. For fault tolerance, it discusses scaling checkpointing by making checkpoints asynchronous and less frequent, and scaling recovery by replicating state so fewer operators need recovery. It presents work in progress on incremental checkpointing and recovery to further optimize state management for large, stateful streaming applications.
Ad
More Related Content
What's hot (20)
From Newbie to Highly Available, a Successful Kafka Adoption Tale (Jonathan S...
From Newbie to Highly Available, a Successful Kafka Adoption Tale (Jonathan S...confluent The document discusses running Apache Kafka across multiple datacenters and provides guidance on how to configure Kafka for disaster recovery in a multi-datacenter setup. It recommends having separate Kafka clusters in each datacenter rather than a single cluster spanning datacenters, due to high replication latency over long-distance links. It also outlines three ways to prepare for disaster recovery: running in separate clusters per datacenter, using Kafka MirrorMaker to replicate topics between clusters, and using a shared storage system.
The Log of All Logs: Raft-based Consensus Inside Kafka | Guozhang Wang, Confl...
The Log of All Logs: Raft-based Consensus Inside Kafka | Guozhang Wang, Confl...HostedbyConfluent KRaft is Kafka's implementation of the Raft consensus protocol that replaces Zookeeper for managing Kafka cluster metadata and electing controllers. It stores metadata in a replicated log with a single writer and allows fast controller failovers. The Quorum Controller runs on top of KRaft, caching metadata from the log. It provides ordered, consistent metadata updates and faster broker shutdowns and startups compared to the previous Zookeeper-based controller. KRaft will be available in early access in the next Kafka release while the Zookeeper dependency is deprecated and eventually removed.
What's new in Confluent 3.2 and Apache Kafka 0.10.2
What's new in Confluent 3.2 and Apache Kafka 0.10.2 confluent With the introduction of connect and streams API in 2016, Apache Kafka is becoming the defacto solution for anyone looking to build a streaming platform. The community continues to add additional capabilities to make it the complete solution for streaming data.
Join us as we review the latest additions in Apache Kafka 0.10.2. In addition, we’ll cover what’s new in Confluent Enterprise 3.2 that makes it possible for running Kafka at scale.
A Journey through the JDKs (Java 9 to Java 11)
A Journey through the JDKs (Java 9 to Java 11)Markus Günther This presentation gives an overview of the most relevant updates and features going from Java 9 to Java 11 from a developer's perspective.
Streaming and Messaging
Streaming and MessagingXin Wang Part-1: Distributed Computing and Storage (Resource Management & Job/Data Management)
Part-2: Streaming and Messaging (Apache Storm & Apache Kafka)
Part-3: Streaming in User Behavior Analysis (Issues & Practices, New Streaming/Messaging Technologies)
Containerizing Distributed Pipes
Containerizing Distributed Pipesinside-BigData.com Hagen Toennies from Gaikai Inc. presented this deck at the 2017 HPC Advisory Council Stanford Conference.
"In this talk we will present how we enable distributed, Unix style programming using Docker and Apache Kafka. We will show how we can take the famous Unix Pipe Pattern and apply it to a Distributed Computing System. We will demonstrate the development of two simple applications with the focus on "Do One Thing and Do It Well." Afterwards we demonstrate how we make these two programs work to together using Apache Kafka. By encapsulating our applications in containers we will also show how that enables us to go from the limited resources of a development machine to cluster of computers in a data center without changing our applications or containers."
Watch the video: http://wp.me/p3RLHQ-goG
Learn more: http://www.hpcadvisorycouncil.com/
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
Flume and HBase
Flume and HBase Alexander Alten This document discusses Apache Flume and using its HBase sink. It provides an overview of Flume's architecture, components, and sources. It describes how the HBase sink works, its configuration options, and provides an example configuration for collecting data from a sequential source and storing it in an HBase table.
Robust Operations of Kafka Streams
Robust Operations of Kafka Streamsconfluent (Bill Bejeck, Confluent) Kafka Summit SF 2018
Apache Kafka added a powerful stream processing library in mid-2016, Kafka Streams, which runs on top of Apache Kafka. The community has embraced Kafka Streams with many early adopters, and the adoption rate continues to grow. Large to mid-size organizations have come to rely on Kafka Streams in their production environments. Kafka Streams has many advanced features to make applications more robust.
The point of this presentation is to show users of Kafka Streams some of the latest and greatest features, as well as some that may be advanced, that can make streams applications more resilient. The target audience for this talk are those users already comfortable writing Kafka Streams applications and want to go from writing their first proof-of-concept applications to writing robust applications that can withstand the rigor that running in a production environment demands.
The talk will be a technical deep dive covering topics like:
-Best practices on configuring a Kafka Streams application
-How to meet production SLAs by minimizing failover and recovery times: configuring standby tasks and the pros and cons of having standby replicas for local state
-How to improve resiliency and 24×7 operability: the use of different configurable error handlers, callbacks and how they can be used to see what’s going on inside the application
-How to achieve efficient scalability: a thorough review of the relationship between the number of instances, threads and state stores and how they relate to each other
While this is a technical deep dive, the talk will also present sample code so that attendees can view the concepts discussed in practice. Attendees of this talk will walk away with a deeper understanding of how Kafka Streams works, and how to make their Kafka Streams applications more robust and efficient. There will be a mix of discussion.
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...
Securing the Message Bus with Kafka Streams | Paul Otto and Ryan Salcido, Raf...HostedbyConfluent Organizations have a need to protect Personally Identifiable Information (PII). As Event Streaming Architecture (ESA) becomes ubiquitous in the enterprise, the prevalence of PII within data streams will only increase. Data architects must be cognizant of how their data pipelines can allow for potential leaks. In highly distributed systems, zero-trust networking has become an industry best practice. We can do the same with Kafka by introducing message-level security.
A DevSecOps Engineer with some Kafka experience can leverage Kafka Streams to protect PII by enforcing role-based access control using Open Policy Agent. Rather than implementing a REST API to handle message-level security, Kafka Streams can filter, or even transform outgoing messages in order to redact PII data while leveraging the native capabilities of Kafka.
In our proposed presentation, we will provide a live demonstration that consists of two consumers subscribing to the same Kafka topic, but receiving different messages based on the rules specified in Open Policy Agent. At the conclusion of the presentation, we will provide attendees with a GitHub repository, so that they can enjoy a sandbox environment for hands-on experimentation with message-level security.
PaaSTA: Autoscaling at Yelp
PaaSTA: Autoscaling at YelpNathan Handler Capacity planning is a difficult challenge faced by most companies. If you have too few machines, you will not have enough compute resources available to deal with heavy loads. On the other hand, if you have too many machines, you are wasting money. This is why companies have started investing in automatically scaling services and infrastructure to minimize the amount of wasted money and resources.
In this talk, Nathan will describe how Yelp is using PaaSTA, a PaaS built on top of open source tools including Docker, Mesos, Marathon, and Chronos, to automatically and gracefully scale services and the underlying cluster. He will go into detail about how this functionality was implemented and the design designs that were made while architecting the system. He will also provide a brief comparison of how this approach differs from existing solutions.
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto
Orchestrating Docker with Terraform and Consul by Mitchell Hashimoto Docker, Inc. Terraform is a tool for building and safely iterating on infrastructure, while Consul provides service discovery, monitoring and orchestration. In this talk we discuss using Terraform and Consul together to build a Docker-based Service Oriented Architecture at scale. We use Consul to provide the runtime control plane for the datacenter, and Terraform is used to modify the underlying infrastructure to allow for elastic scalability.
Espresso Database Replication with Kafka, Tom Quiggle
Espresso Database Replication with Kafka, Tom Quiggleconfluent This document discusses using Apache Kafka for database replication in LinkedIn's ESPRESSO database system. It provides an overview of ESPRESSO's architecture and transition from per-instance to per-partition replication using Kafka. Key aspects covered include Kafka configuration, the message protocol for ensuring in-order delivery, and checkpointing by the Kafka producer to allow resuming replication from the last committed transaction after failures.
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...
Cross the streams thanks to Kafka and Flink (Christophe Philemotte, Digazu) K...confluent The last 5 years, Kafka and Flink have become mature technologies that have allowed us to embrace the streaming paradigm. You can bet on them to build reliable and efficient applications. They are active projects backed by companies using them in production. They have a good community contributing, and sharing experience and knowledge. Kafka and Flink are solid choices if you want to build a data platform that your data scientists or developers can use to collect, process, and distribute data. You can put together Kafka Connect, Kafka, Schema Registry, and Flink. First, you will take care of their deployment. Then, for each case, you will setup each part, and of course develop the Flink job so it can integrate easily with the rest. Looks like a challenging but exciting project, isn't it? In this session, you will learn how you can build such data platform, what are the nitty-gritty of each part, how you can plug them together, in particular how to plug Flink in the Kafka ecosystem, what are the common pitfalls to avoid, and what it requires to be deployed on kubernetes. Even if you are not familiar with all the technologies, there will be enough introduction so you can follow. Come and learn how we can actually cross the streams!
Introduction to Apache Mesos
Introduction to Apache MesosJoe Stein Apache Mesos is a cluster manager that simplifies the complexity of running applications on a shared pool of servers.
Managing multiple event types in a single topic with Schema Registry | Bill B...
Managing multiple event types in a single topic with Schema Registry | Bill B...HostedbyConfluent With Apache Kafka, it's typical to place different events in their own topic. But different event types can be related. Consider customer interactions with an online retailer. The customer searches through the site and clicks on various items before deciding on a final purchase. But businesses can gain insight by processing these events in sequence. Using the event type per topic leaves a lot of work for developers. Is there a better way?
Fortunately, there is. Schema Registry now supports having multiple event types in the same topic. By placing various event types in a single topic, you can now handle different related events in-order. In this presentation, I'll introduce Schema Registry then we'll dive into how it handles multiple event types in a single topic, including examples.
You will learn how and when to apply the multiple event types per topic pattern. Additionally, you'll learn how schema references work in Schema Registry.
ksqlDB: A Stream-Relational Database System
ksqlDB: A Stream-Relational Database Systemconfluent Speaker: Matthias J. Sax, Software Engineer, Confluent
ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries.
Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems.
https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/
Introduction to Akka-Streams
Introduction to Akka-Streamsdmantula This document provides an introduction to Akka Streams, which implements the Reactive Streams specification. It discusses the limitations of traditional concurrency models and Actor models in dealing with modern challenges like high availability and large data volumes. Reactive Streams aims to provide a minimalistic asynchronous model with back pressure to prevent resource exhaustion. Akka Streams builds on the Akka framework and Actor model to provide a streaming data flow library that uses Reactive Streams interfaces. It allows defining processing pipelines with sources, flows, and sinks and includes features like graph DSL, back pressure, and integration with other Reactive Streams implementations.
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQL
Kafka Summit SF 2017 - Kafka Stream Processing for Everyone with KSQLconfluent This document introduces KSQL, a streaming SQL engine for Apache Kafka. It provides concise summaries of KSQL's capabilities and how to use it in 3 sentences or less:
KSQL allows users to easily query and transform data in Kafka streams using SQL-like queries. It provides simplicity, flexibility, and scalability compared to directly using Kafka Streams APIs. KSQL can be run in standalone, client-server, or application modes and is well-suited for tasks like streaming ETL, anomaly detection, monitoring, and IoT data processing.
Apache Kafka 0.8 basic training - Verisign
Apache Kafka 0.8 basic training - VerisignMichael Noll Apache Kafka 0.8 basic training (120 slides) covering:
1. Introducing Kafka: history, Kafka at LinkedIn, Kafka adoption in the industry, why Kafka
2. Kafka core concepts: topics, partitions, replicas, producers, consumers, brokers
3. Operating Kafka: architecture, hardware specs, deploying, monitoring, P&S tuning
4. Developing Kafka apps: writing to Kafka, reading from Kafka, testing, serialization, compression, example apps
5. Playing with Kafka using Wirbelsturm
Audience: developers, operations, architects
Created by Michael G. Noll, Data Architect, Verisign, https://www.verisigninc.com/
Verisign is a global leader in domain names and internet security.
Tools mentioned:
- Wirbelsturm (https://github.com/miguno/wirbelsturm)
- kafka-storm-starter (https://github.com/miguno/kafka-storm-starter)
Blog post at:
http://www.michael-noll.com/blog/2014/08/18/apache-kafka-training-deck-and-tutorial/
Many thanks to the LinkedIn Engineering team (the creators of Kafka) and the Apache Kafka open source community!
Federated mesos clusters for global data center designs
Federated mesos clusters for global data center designsKrishna-Kumar The document describes Huawei's approach to federating Mesos clusters across multiple data centers. It proposes a multi-master federation approach where each data center runs its own Mesos master that coordinates with other masters. Gossiper modules in each data center gossip with each other to exchange framework and resource information. When one data center reaches a resource threshold, its gossiper will direct work to other data centers based on a simple policy engine. A demo visualization is shown to illustrate work load balancing during normal and failure scenarios.
Viewers also liked (20)
Márton Balassi Streaming ML with Flink-
Márton Balassi Streaming ML with Flink- Flink Forward http://flink-forward.org/kb_sessions/streaming-ml-with-flink/
As continuous big data processing is gaining popularity it naturally implies that there is a need to transition many of the distributed machine learning functionality to a streaming backend. The most common use case is to give streaming predictions based on the model learnt in batch, however in some cases it is beneficial to also update the model on the fly. It is not uncommon that streaming learners need different algorithms than their batch counterparts. The talk discusses the common use cases and the pitfalls of the streaming ML transition through the example of recommender systems. It also offer a dive into the implementation of a Scala library augmenting FlinkML with streaming predictors.
Stephan Ewen - Scaling to large State
Stephan Ewen - Scaling to large StateFlink Forward The document discusses scaling state management in Apache Flink streaming applications to very large state. It describes how Flink uses state sharding and increasing operator parallelism to scale stateful computation. For fault tolerance, it discusses scaling checkpointing by making checkpoints asynchronous and less frequent, and scaling recovery by replicating state so fewer operators need recovery. It presents work in progress on incremental checkpointing and recovery to further optimize state management for large, stateful streaming applications.
Automatic Detection of Web Trackers by Vasia Kalavri
Automatic Detection of Web Trackers by Vasia KalavriFlink Forward This document describes research into automatically detecting web trackers. It explores using network structure and behavior to classify nodes in a bipartite graph of website referrers and hosts as trackers or not. Various graph-based and machine learning algorithms are tested, including relevance search, clustering, and community detection on a projected graph. The best approach involved preprocessing logs into graphs, running community detection, and achieving very high accuracy in classifying nodes as trackers or not. Lessons learned include starting simply, choosing features incrementally, visualizing data, and using a flexible system like Flink.
Ted Dunning-Faster and Furiouser- Flink Drift
Ted Dunning-Faster and Furiouser- Flink DriftFlink Forward http://flink-forward.org/kb_sessions/faster-and-furiouser-flink-drift/
Not long ago, we had the opportunity to test Apache Flink to see just how fast it would go on a moderately realistic task with fast hardware and with a good streaming transport layer underneath. Our goal was not so much careful comparison with other software, but flat-out speed, Flink against Flink. In the process, we learned a lot about what it takes to go fast. Some of the lessons were ones that we had “learned” a number of times before: – the bottleneck isn’t where you thought it was – copying data is expensive – context switches are expensive – measure twice, cut once But there were some real surprises along the way. The really important knobs weren’t quite what people say you should turn. One of the biggest surprises was the degree to which high performance libraries have threading built into them which makes the actual concurrrency much higher than the apparent concurrency. The result was that at least one cluster parameter needed to be adjusted by 30x to get real
Julian Hyde - Streaming SQL
Julian Hyde - Streaming SQLFlink Forward http://flink-forward.org/kb_sessions/streaming-sql/
Streaming is necessary to handle data rates and latency, but SQL is unquestionably the lingua franca of data. Is it possible to combine SQL with streaming, and if so, what does the resulting language look like? Apache Calcite is extending SQL to include streaming, and Apache Flink is using Calcite to support both regular and streaming SQL. In this talk, Julian Hyde describes streaming SQL in detail and shows how you can use streaming SQL in your application. He also describes how Calcite’s planner optimizes queries for throughput and latency.
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with Flink
Sanjar Akhmedov - Joining Infinity – Windowless Stream Processing with FlinkFlink Forward http://flink-forward.org/kb_sessions/joining-infinity-windowless-stream-processing-with-flink/
The extensive set of high-level Flink primitives makes it easy to join windowed streams. However, use cases that don’t have windows can prove to be more complicated, making it necessary to leverage operator state and low-level primitives to manually implement a continuous join. This talk will focus on the anomalies that present themselves when performing streaming joins with infinite windows, and the problems encountered operating topologies that back user-facing data. We will describe the approach taken at ResearchGate to implement and maintain a consistent join result of change data capture streams.
Aljoscha Krettek - The Future of Apache Flink
Aljoscha Krettek - The Future of Apache FlinkFlink Forward http://flink-forward.org/kb_sessions/the-future-of-apache-flinktm/
In this session we will first have a look at the current state of Apache Flink before diving into some of the upcoming features that are either already in development or still in the design phase. Some of the features currently in development that we are going to cover are: – Dynamic Scaling: Adapting a running program to changing workloads. – Queryable State: External querying of internal Flink state. This has the power to replace key/value stores by turning Flink into a key value store that allows for up to date querying of results. – Side Inputs: Having additional data that evolves over time as input to a stream operation. For the glimpse at the far-off future of Apache Flink™ we dare not make any predictions yet. In the session we will look at the latest whisperings and see what the community is currently thinking up as solutions to existing problems and predicted future challenges in the stream processing space.
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...
Thomas Lamirault_Mohamed Amine Abdessemed -A brief history of time with Apac...Flink Forward Many use cases in the telecommunication industry require producing counters, quality metrics, and alarms in a streaming fashion with very low latency. Most of this metrics are only valuable when they’re made available as soon as the associated events happened. In our company we are looking for a system able to produce this kind of real-time indicator, which must handle massive amounts of data (400,000 eps) with often peak loads (like New Year’s Eve) or out-of-order events like massive network disorder. Low latency and flexible window management with specific watermark emission are also a must-haves. Heterogeneous format, multiple flow correlation, and the possibility of late data arrival are other challenges. Flink being already widely used at Bouygues Telecom for real-time data integration, its features made it the evident candidate for the future System. In this talk, we'll present a real use case of streaming analytics using Flink, Kafka & HBase along with other legacy systems.
Jamie Grier - Robust Stream Processing with Apache Flink
Jamie Grier - Robust Stream Processing with Apache FlinkFlink Forward The document discusses Apache Flink, an open source platform for distributed stream and batch data processing. It describes how Flink allows for stateful stream processing, windowed computations over streams, and robust handling of time, failures, planned downtimes, and reprocessing. The presentation provides an overview of these concepts and includes a code example and details on Flink's distributed and parallel deployment architecture.
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...
Kostas Tzoumas_Stephan Ewen - Keynote -The maturing data streaming ecosystem ...Flink Forward http://flink-forward.org/kb_sessions/keynote-tba-2/
The past 12 months saw the data streaming ecosystem mature and grow tremendously with new open source projects and products being offered in the market, and more large-scale production applications of streaming data. It is now understood that streaming data is not a fad, but a growing industry that is here to stay.
Apache Flink was one of the pioneering communities advocating that stream processing is a great fit for the continuous nature of data production, and that batch processing can be seen and efficiently performed as a special case of stream processing. Flink saw tremendous growth since the last Flink Forward conference, with the project boasting now more than 200 contributors from several companies, several production installations and broad adoption.
In this talk, we discuss several large-scale stream processing use cases that we see at data Artisans. Additionally, we discuss what this accelerated growth means for Flink, how we can sustain this growth moving forward, as well as a vision for the next big directions in Flink.
S. Bartoli & F. Pompermaier – A Semantic Big Data Companion
S. Bartoli & F. Pompermaier – A Semantic Big Data CompanionFlink Forward This document introduces Okkam, an Italian company that uses Apache Flink for large-scale data integration and semantic technologies. It discusses Okkam's use of Flink for domain reasoning, RDF data processing, duplicate detection, entity linkage, and telemetry analysis. The document also provides lessons learned from Okkam's Flink experiences and suggestions for improving Flink.
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...
Fabian Hueske_Till Rohrmann - Declarative stream processing with StreamSQL an...Flink Forward http://flink-forward.org/kb_sessions/declarative-stream-processing-with-streamsql-and-cep/
Complex event processing (CEP) and stream analytics are commonly treated as distinct classes of stream processing applications. While CEP workloads identify patterns from event streams in near real-time, stream analytics queries ingest and aggregate high-volume streams. Both types of use cases have very different requirements which resulted in diverging system designs. CEP systems excel at low-latency processing whereas engines for stream analytics achieve high throughput. Recent advances in open source stream processing yielded systems that can process several millions of events per second at sub-second latency. Systems like Apache Flink enable applications that include typical CEP features as well as heavy aggregations. In this talk we will show how Apache Flink unifies CEP and stream analytics workloads. Guided by examples, we introduce Flink’s CEP-enriched StreamSQL interface and discuss how queries are compiled, optimized, and executed on Flink.
Flink Case Study: OKKAM
Flink Case Study: OKKAMFlink Forward Okkam is an Italian SME specializing in large-scale data integration using semantic technologies. It provides services for public administration and restaurants by building and managing very large entity-centric knowledge bases. Okkam uses Apache Flink as its data processing framework for tasks like domain reasoning, managing the RDF data lifecycle, detecting duplicate records, entity record linkage, and telemetry analysis by combining Flink with technologies like Parquet, Jena, Sesame, ELKiBi, HBase, Solr, MongoDB, and Weka. The presenters work at Okkam and will discuss their use of Flink in more detail in their session.
Dongwon Kim – A Comparative Performance Evaluation of Flink
Dongwon Kim – A Comparative Performance Evaluation of FlinkFlink Forward This document provides a summary and analysis of a performance evaluation comparing the big data processing engine Flink to other engines like Spark, Tez, and MapReduce. The key points are:
- Flink completes a 3.2TB TeraSort benchmark faster than Spark, Tez, and MapReduce due to its pipelined execution model which allows more overlap between stages compared to the other engines.
- While Tez and Spark attempt to overlap stages, in practice they do not due to the way tasks are scheduled and launched. MapReduce shows some overlap but is still slower.
- Flink causes fewer disk accesses during shuffling by transferring data directly from memory to memory instead of writing to disk like
RocksDB compaction
RocksDB compactionMIJIN AN The document discusses compaction in RocksDB, an embedded key-value storage engine. It describes the two compaction styles in RocksDB: level style compaction and universal style compaction. Level style compaction stores data in multiple levels and performs compactions by merging files from lower to higher levels. Universal style compaction keeps all files in level 0 and performs compactions by merging adjacent files in time order. The document provides details on the compaction process and configuration options for both styles.
Gábor Horváth - Code Generation in Serializers and Comparators of Apache Flink
Gábor Horváth - Code Generation in Serializers and Comparators of Apache FlinkFlink Forward This document discusses code generation for serializers and comparators in Apache Flink. Currently, Flink serializers use reflection which reduces performance. The document proposes generating serialization code at runtime to eliminate reflection and improve efficiency. It shows benchmark results demonstrating a 6x speedup from generated serializers and comparators compared to Flink's current approach. Future work is discussed to further optimize serialization performance and address challenges of dynamically generated code.
RocksDB detail
RocksDB detailMIJIN AN RocksDB is an embedded key-value store written in C++ and optimized for fast storage environments like flash or RAM. It uses a log-structured merge tree to store data by writing new data sequentially to an in-memory log and memtable, periodically flushing the memtable to disk in sorted SSTables. It reads from the memtable and SSTables, and performs background compaction to merge SSTables and remove overwritten data. RocksDB supports two compaction styles - level style, which stores SSTables in multiple levels sorted by age, and universal style, which stores all SSTables in level 0 sorted by time.
Gyula Fóra - RBEA- Scalable Real-Time Analytics at King
Gyula Fóra - RBEA- Scalable Real-Time Analytics at KingFlink Forward http://flink-forward.org/kb_sessions/rbea-scalable-real-time-analytics-at-king/
This talk introduces RBEA (Rule-Based Event Aggregator), the scalable real-time analytics platform developed by King’s Streaming Platform team. We have built RBEA to make real-time analytics easily accessible to game teams across King without having to worry about operational details. RBEA is built on top of Apache Flink and uses the framework’s capabilities to it’s full potential in order to provide highly scalable stateful and windowed processing logic for the analytics applications. We will talk about how we have built a high-level DSL on the abstractions provided by Flink and how we tackled different technical challenges that have come up while developing the system.
Matthias Kricke_Martin Grimmer_Michael Schmeißer - Building a real time Tweet...
Matthias Kricke_Martin Grimmer_Michael Schmeißer - Building a real time Tweet...Flink Forward http://flink-forward.org/kb_sessions/building-a-real-time-tweet-map-with-flink-in-six-weeks/
It is often necessary to build a proof of concept for a bigdata project to show the feasibility to customers. With OSTMap (Open Source Tweet Map) we proved that it is possible to accomplish this with the right choice of technologies in a short time frame. OSTMap enables the user to view live, geotagged tweets of the last hours on a map, search all collected tweets by a term or user name and view the amount of incoming tweets per minute distinguished by language as a graph. With OSTMap we cover several important areas of a typical big data PoC project: scalability, stream processing and ingest of incoming data, batch processing of stored data, performant queries on the data and visualization of the data. To achieve this, we use Apache Flink and Apache Accumulo as backend technologies, AngularJS and Leaflet for the frontend. OSTMap was developed iteratively using the walking skeleton approach which allowed us to increase the feature set constantly even with a strict deadline.
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...
Javier Lopez_Mihail Vieru - Flink in Zalando's World of Microservices - Flink...Flink Forward http://flink-forward.org/kb_sessions/flink-in-zalandos-world-of-microservices/
In this talk we present Zalando’s microservices architecture, introduce Saiki – our next generation data integration and distribution platform on AWS and show how we employ stream processing with Apache Flink for near-real time business intelligence.
Zalando is one of the largest online fashion retailers in Europe. In order to secure our future growth and remain competitive in this dynamic market, we are transitioning from a monolithic to a microservices architecture and from a hierarchical to an agile organization.
We first have a look at how business intelligence processes have been working inside Zalando for the last years and present our current approach – Saiki. It is a scalable, cloud-based data integration and distribution infrastructure that makes data from our many microservices readily available for analytical teams.
We no longer live in a world of static data sets, but are instead confronted with endless streams of events that constantly inform us about relevant happenings from all over the enterprise. The processing of these event streams enables us to do near-real time business intelligence. In this context we have evaluated Apache Flink vs. Apache Spark in order to choose the right stream processing framework. Given our requirements, we decided to use Flink as part of our technology stack, alongside with Kafka and Elasticsearch.
With these technologies we are currently working on two use cases: a near real-time business process monitoring solution and streaming ETL.
Monitoring our business processes enables us to check if technically the Zalando platform works. It also helps us analyze data streams on the fly, e.g. order velocities, delivery velocities and to control service level agreements.
On the other hand, streaming ETL is used to relinquish resources from our relational data warehouse, as it struggles with increasingly high loads. In addition to that, it also reduces the latency and facilitates the platform scalability.
Finally, we have an outlook on our future use cases, e.g. near-real time sales and price monitoring. Another aspect to be addressed is to lower the entry barrier of stream processing for our colleagues coming from a relational database background.
Ad
Similar to Eron Wright - Introducing Flink on Mesos (20)
Containerization - The DevOps Revolution
Containerization - The DevOps RevolutionYulian Slobodyan Containerization allows for standardized and isolated application deployment across infrastructure through container runtimes like Docker. Key features that enable containers include namespaces which isolate processes and their views of the system, cgroups which control resource usage, and copy-on-write storage for efficient application packaging. A container orchestration system like Mesos provides scalability, fault tolerance, and unified resource management across clusters. This allows maximizing infrastructure utilization through flexible scheduling of containerized applications and services on shared clusters.
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...
Making Distributed Data Persistent Services Elastic (Without Losing All Your ...Joe Stein Making Distributed Data Persistent Services Elastic (Without Losing All Your Data)
Mesos Overview
Roles, Attributes, Constraints, Modules, Hooks
Marathon, Kafka, Cassandra, HDFS, MySQL, YARN and more!
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating System
OSDC 2015: Bernd Mathiske | Why the Datacenter Needs an Operating SystemNETWAYS Developers are moving away from their host-based patterns and adopting a new mindset around the idea that the datacenter is the computer. It?s quickly becoming a mainstream model that you can view a warehouse full of servers as a single computer (with terabytes of memory and tens of thousands of cores). There is a key missing piece, which is an operating system for the datacenter (DCOS), which would provide the same OS functionality and core OS abstractions across thousands of machines that an OS provides on a single machine today. In this session, we will discuss:
How the abstraction of an OS has evolved over time and can cleanly scale to spand thousands of machines in a datacenter.
How key open source technologies like the Apache Mesos distributed systems kernel provide the key underpinnings for a DCOS.
How developers can layer core system services on top of a distributed systems kernel, including an init system (Marathon), cron (Chronos), service discovery (DNS), and storage (HDFS)
What would the interface to the DCOS look like? How would you use it?
How you would install and operate datacenter services, including Apache Spark, Apache Cassandra, Apache Kafka, Apache Hadoop, Apache YARN, Apache HDFS, and Google's Kubernetes.
How will developers build datacenter-scale apps, programmed against the datacenter OS like it?s a single machine?
Modern Elastic Datacenter Architecture
Modern Elastic Datacenter ArchitectureWeston Bassler The document discusses modern elastic datacenter architecture using Apache Mesos and DC/OS. It provides an introduction to Mesos and DC/OS, explaining how they allow building scalable, fault-tolerant distributed systems. It outlines the benefits of using Mesos and DC/OS, and describes how the speakers have implemented a solution using tools like Packer, Terraform, Ansible, and DC/OS to achieve scalability, automation, and high availability. Demos are presented on deploying and managing applications with DC/OS tools like Marathon and running Spark frameworks.
Musings on Mesos: Docker, Kubernetes, and Beyond.
Musings on Mesos: Docker, Kubernetes, and Beyond.Timothy St. Clair 2014 ApacheCon EU talk on Mesos and it's integration with Docker as well the ecosystem, and outline the future of Kubernetes integration
Running Spark on Mesos
Running Spark on MesosPeker Mert Öksüz Running a batch Spark jobs on Mesos cluster. Installing Mesos cluster and running spark dispatcher on it.
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OS
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OSLightbend Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available.
In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like:
* The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers.
* The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements.
* How to address the particular needs of stateful services in a model that natively favors stateless, transient services.
TechBeats #2
TechBeats #2applausepoland Applause's journey from a monolith to a microservice environment running on Mesosphere DC/OS by Christopher Gianelloni
Introduction to mesos
Introduction to mesosOmid Vahdaty Mesos is a cluster manager that provides efficient resource sharing across distributed applications. It sits between applications and the operating system to make deploying and managing applications in large clusters more efficient. Mesos introduces distributed two-level scheduling where it decides how to allocate resources to frameworks, which then decide how to use those resources. It provides features like fault tolerance, scalability, resource isolation, and APIs for building distributed apps. Common uses include running Hadoop, Spark, Storm, Jenkins, and Docker on Mesos clusters.
Orbit GSM UMTS LTE parser platform - ETL tool
Orbit GSM UMTS LTE parser platform - ETL toolAhmet Ozturk Orbit is an end-to-end data management solution for Extract-Transform-Load (ETL) processes where they can be defined, managed and monitored from a single point with a centralized approach.
DockerCon14 Cluster Management and Containerization
DockerCon14 Cluster Management and ContainerizationDocker, Inc. This document discusses cluster management and containerization. It describes the challenges of managing clusters at scale, including failures, maintenance, and utilization. It introduces Mesos as a modern cluster manager that can help address these challenges by coordinating resources across clusters and launching and monitoring tasks. Mesos provides a general purpose platform for scheduling different types of workloads like services, batch jobs, and streaming applications.
Real Time Operating System
Real Time Operating SystemSharad Pandey This presentation includes a detail of various real time operating systems and it focuses on Vx Works. It will also help you understand what not is a RTOS.
OSDC 2016 - Mesos and the Architecture of the New Datacenter by Jörg Schad
OSDC 2016 - Mesos and the Architecture of the New Datacenter by Jörg SchadNETWAYS Apache Mesos has the ability to run on every private and cloud instance, anywhere. In this talk, Jörg Schad (Software Developer at Mesosphere) will explain the momentum behind the “single computer” abstraction that has put Mesos at the center of one of the most exciting architecture shifts in recent information technology history. He will explain how Mesos is enabling application developers and devops to redefine their responsibilities and shorten the amount of time it takes to write and ship production code. Jörg will outline how Mesos is empowering the new class of “datacenter developers” to program directly against datacenter resources, and draw correlations to how the Linux revolutionized the server industry.
DC/OS: The definitive platform for modern apps
DC/OS: The definitive platform for modern appsDatio Big Data DC/OS is an open source platform that provides container orchestration and management using Mesos. It allows running applications and services across data center infrastructure including bare metal, VMs, and cloud. DC/OS provides services like Marathon for container orchestration, security, monitoring, load balancing and service discovery. It has features like high resource utilization, mixed workload support, elastic scalability, high availability and zero downtime upgrades.
Containerized Data Persistence on Mesos
Containerized Data Persistence on MesosJoe Stein Elodina is a startup focused on supporting open source software like Mesos frameworks. They have experience implementing and assisting with Kafka, Mesos, Hadoop, Cassandra, and other big data systems. The document discusses containerizing data persistence on Mesos with various data systems like Kafka, MySQL, Cassandra and HDFS. It provides overviews of file systems, databases, and Mesos concepts like roles and resources. It also summarizes efforts to run these data systems as Mesos frameworks.
DEPLOYING A DOCKERIZED DISTRIBUTED APPLICATION IN MESOS
DEPLOYING A DOCKERIZED DISTRIBUTED APPLICATION IN MESOSJulia Mateo - Mesos is a cluster management platform that dynamically allocates resources to distributed applications running on Docker containers. It provides deployment, execution, resource sharing, and data locality capabilities.
- Marathon is a Mesos framework that provides easy deployment and management of Docker containers running long-running applications on Mesos. It exposes a REST API.
- MesosDNS provides service discovery by mapping Mesos tasks to DNS names, but has limitations around identifying ports, caching, health checking, and failover that alternative solutions like lb-marathon aim to address.
Apache Mesos Overview and Integration
Apache Mesos Overview and IntegrationAlex Baretto Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications running on a shared pool of nodes. It provides APIs for resource management and scheduling and can run applications like Hadoop, Spark, and Kafka across large compute clusters. Mesos was created at Berkeley and adopted by companies like Twitter, Airbnb, and Hubspot to improve resource utilization, allow dynamic scaling of services, and reduce the operational complexity of managing large fleets of servers. Mesos also integrates with Docker and Kubernetes to enable containerized workloads and services to be deployed and orchestrated on Mesos clusters.
Modern apps with dcos
Modern apps with dcosSam Chen This document provides an overview of how to implement modern applications on DC/OS architecture quickly. It begins with introductions of the speaker and defines concepts like microservices, modern applications, and what DC/OS is. Then it explains the principles of microservices and characteristics of modern applications. The document compares DC/OS, Docker, Kubernetes and OpenStack. It also illustrates how to implement modern applications on DC/OS and why DC/OS is different. The document will share best practices and case studies.
DCOS Presentation
DCOS PresentationJan Repnak The document discusses Mesosphere's Datacenter Operating System (DCOS). It provides an overview of containers, Apache Mesos, Marathon, and how DCOS integrates these components to function as an operating system for datacenters. Key points include that DCOS treats resources across a cluster as a single shared pool, provides scheduling and orchestration of workloads, and enables easy deployment and management of applications.
minimesos. Apache Mesos made easy
minimesos. Apache Mesos made easyViktor Sadovnikov Are you developing new Apache Mesos framework and wonder how you can test it as part of your CI builds? Are you looking for a quick way to try out new Mesos framework without starting a massive cluster and crewing your development box? Do you want to test your application in clusters of different size and switch among them within a minute? In other case minimesos (http://minimesos.org/) is the tool to use. During this talk we will show how you can manage Mesos cluster from your test code and how you can make multiple clusters wait to be started on your single box
Ad
More from Flink Forward (20)
Building a fully managed stream processing platform on Flink at scale for Lin...
Building a fully managed stream processing platform on Flink at scale for Lin...Flink Forward Apache Flink is a distributed stream processing framework that allows users to process and analyze data in real-time. At LinkedIn, we developed a fully managed stream processing platform on Flink running on K8s to power hundreds of stream processing pipelines in production. This platform is the backbone for other infra systems like Search, Espresso (internal document store) and feature management etc. We provide a rich authoring and testing environment which allows users to create, test, and deploy their streaming jobs in a self-serve fashion within minutes. Users can focus on their business logic, leaving the Flink platform to take care of management aspects such as split deployment, resource provisioning, auto-scaling, job monitoring, alerting, failure recovery and much more. In this talk, we will introduce the overall platform architecture, highlight the unique value propositions that it brings to stream processing at LinkedIn and share the experiences and lessons we have learned.
Evening out the uneven: dealing with skew in Flink
Evening out the uneven: dealing with skew in FlinkFlink Forward Flink Forward San Francisco 2022.
When running Flink jobs, skew is a common problem that results in wasted resources and limited scalability. In the past years, we have helped our customers and users solve various skew-related issues in their Flink jobs or clusters. In this talk, we will present the different types of skew that users often run into: data skew, key skew, event time skew, state skew, and scheduling skew, and discuss solutions for each of them. We hope this will serve as a guideline to help you reduce skew in your Flink environment.
by
Jun Qin & Karl Friedrich
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...Flink Forward Flink Forward San Francisco 2022.
To improve Amazon Alexa experiences and support machine learning inference at scale, we built an automated end-to-end solution for incremental model building or fine-tuning machine learning models through continuous learning, continual learning, and/or semi-supervised active learning. Customer privacy is our top concern at Alexa, and as we build solutions, we face unique challenges when operating at scale such as supporting multiple applications with tens of thousands of transactions per second with several dependencies including near-real time inference endpoints at low latencies. Apache Flink helps us transform and discover metrics in near-real time in our solution. In this talk, we will cover the challenges that we faced, how we scale the infrastructure to meet the needs of ML teams across Alexa, and go into how we enable specific use cases that use Apache Flink on Amazon Kinesis Data Analytics to improve Alexa experiences to delight our customers while preserving their privacy.
by
Aansh Shah
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Flink Forward Flink Forward San Francisco 2022.
Probably everyone who has written stateful Apache Flink applications has used one of the fault-tolerant keyed state primitives ValueState, ListState, and MapState. With RocksDB, however, retrieving and updating items comes at an increased cost that you should be aware of. Sometimes, these may not be avoidable with the current API, e.g., for efficient event-time stream-sorting or streaming joins where you need to iterate one or two buffered streams in the right order. With FLIP-220, we are introducing a new state primitive: BinarySortedMultiMapState. This new form of state offers you to (a) efficiently store lists of values for a user-provided key, and (b) iterate keyed state in a well-defined sort order. Both features can be backed efficiently by RocksDB with a 2x performance improvement over the current workarounds. This talk will go into the details of the new API and its implementation, present how to use it in your application, and talk about the process of getting it into Flink.
by
Nico Kruber
Introducing the Apache Flink Kubernetes Operator
Introducing the Apache Flink Kubernetes OperatorFlink Forward Flink Forward San Francisco 2022.
The Apache Flink Kubernetes Operator provides a consistent approach to manage Flink applications automatically, without any human interaction, by extending the Kubernetes API. Given the increasing adoption of Kubernetes based Flink deployments the community has been working on a Kubernetes native solution as part of Flink that can benefit from the rich experience of community members and ultimately make Flink easier to adopt. In this talk we give a technical introduction to the Flink Kubernetes Operator and demonstrate the core features and use-cases through in-depth examples."
by
Thomas Weise
Autoscaling Flink with Reactive Mode
Autoscaling Flink with Reactive ModeFlink Forward Flink Forward San Francisco 2022.
Resource Elasticity is a frequently requested feature in Apache Flink: Users want to be able to easily adjust their clusters to changing workloads for resource efficiency and cost saving reasons. In Flink 1.13, the initial implementation of Reactive Mode was introduced, later releases added more improvements to make the feature production ready. In this talk, we’ll explain scenarios to deploy Reactive Mode to various environments to achieve autoscaling and resource elasticity. We’ll discuss the constraints to consider when planning to use this feature, and also potential improvements from the Flink roadmap. For those interested in the internals of Flink, we’ll also briefly explain how the feature is implemented, and if time permits, conclude with a short demo.
by
Robert Metzger
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Flink Forward Flink Forward San Francisco 2022.
Flink consumers read from Kafka as a scalable, high throughput, and low latency data source. However, there are challenges in scaling out data streams where migration and multiple Kafka clusters are required. Thus, we introduced a new Kafka source to read sharded data across multiple Kafka clusters in a way that conforms well with elastic, dynamic, and reliable infrastructure. In this presentation, we will present the source design and how the solution increases application availability while reducing maintenance toil. Furthermore, we will describe how we extended the existing KafkaSource to provide mechanisms to read logical streams located on multiple clusters, to dynamically adapt to infrastructure changes, and to perform transparent cluster migrations and failover.
by
Mason Chen
One sink to rule them all: Introducing the new Async Sink
One sink to rule them all: Introducing the new Async SinkFlink Forward Flink Forward San Francisco 2022.
Next time you want to integrate with a new destination for a demo, concept or production application, the Async Sink framework will bootstrap development, allowing you to move quickly without compromise. In Flink 1.15 we introduced the Async Sink base (FLIP-171), with the goal to encapsulate common logic and allow developers to focus on the key integration code. The new framework handles things like request batching, buffering records, applying backpressure, retry strategies, and at least once semantics. It allows you to focus on your business logic, rather than spending time integrating with your downstream consumers. During the session we will dive deep into the internals to uncover how it works, why it was designed this way, and how to use it. We will code up a new sink from scratch and demonstrate how to quickly push data to a destination. At the end of this talk you will be ready to start implementing your own Flink sink using the new Async Sink framework.
by
Steffen Hausmann & Danny Cranmer
Tuning Apache Kafka Connectors for Flink.pptx
Tuning Apache Kafka Connectors for Flink.pptxFlink Forward Flink Forward San Francisco 2022.
In normal situations, the default Kafka consumer and producer configuration options work well. But we all know life is not all roses and rainbows and in this session we’ll explore a few knobs that can save the day in atypical scenarios. First, we'll take a detailed look at the parameters available when reading from Kafka. We’ll inspect the params helping us to spot quickly an application lock or crash, the ones that can significantly improve the performance and the ones to touch with gloves since they could cause more harm than benefit. Moreover we’ll explore the partitioning options and discuss when diverging from the default strategy is needed. Next, we’ll discuss the Kafka Sink. After browsing the available options we'll then dive deep into understanding how to approach use cases like sinking enormous records, managing spikes, and handling small but frequent updates.. If you want to understand how to make your application survive when the sky is dark, this session is for you!
by
Olena Babenko
Flink powered stream processing platform at Pinterest
Flink powered stream processing platform at PinterestFlink Forward Flink Forward San Francisco 2022.
Pinterest is a visual discovery engine that serves over 433MM users. Stream processing allows us to unlock value from realtime data for pinners. At Pinterest, we adopt Flink as the unified streaming processing engine. In this talk, we will share our journey in building a stream processing platform with Flink and how we onboarding critical use cases to the platform. Pinterest has supported 90+near realtime streaming applications. We will cover the problem statement, how we evaluate potential solutions and our decision to build the framework.
by
Rainie Li & Kanchi Masalia
Apache Flink in the Cloud-Native Era
Apache Flink in the Cloud-Native EraFlink Forward Flink Forward San Francisco 2022.
This talk will take you on the long journey of Apache Flink into the cloud-native era. It started all the way from where Hadoop and YARN were the standard way of deploying and operating data applications.
We're going to deep dive into the cloud-native set of principles and how they map to the Apache Flink internals and recent improvements. We'll cover fast checkpointing, fault tolerance, resource elasticity, minimal infrastructure dependencies, industry-standard tooling, ease of deployment and declarative APIs.
After this talk you'll get a broader understanding of the operational requirements for a modern streaming application and where the current limits are.
by
David Moravek
Where is my bottleneck? Performance troubleshooting in Flink
Where is my bottleneck? Performance troubleshooting in FlinkFlink Forward Flinkn Forward San Francisco 2022.
In this talk, we will cover various topics around performance issues that can arise when running a Flink job and how to troubleshoot them. We’ll start with the basics, like understanding what the job is doing and what backpressure is. Next, we will see how to identify bottlenecks and which tools or metrics can be helpful in the process. Finally, we will also discuss potential performance issues during the checkpointing or recovery process, as well as and some tips and Flink features that can speed up checkpointing and recovery times.
by
Piotr Nowojski
Using the New Apache Flink Kubernetes Operator in a Production Deployment
Using the New Apache Flink Kubernetes Operator in a Production DeploymentFlink Forward Flink Forward San Francisco 2022.
Running natively on Kubernetes, using the new Apache Flink Kubernetes Operator is a great way to deploy and manage Flink application and session deployments. In this presentation, we provide: - A brief overview of Kubernetes operators and their benefits. - Introduce the five levels of the operator maturity model. - Introduce the newly released Apache Flink Kubernetes Operator and FlinkDeployment CRs - Dockerfile modifications you can make to swap out UBI images and Java of the underlying Flink Operator container - Enhancements we're making in: - Versioning/Upgradeability/Stability - Security - Demo of the Apache Flink Operator in-action, with a technical preview of an upcoming product using the Flink Kubernetes Operator. - Lessons learned - Q&A
by
James Busche & Ted Chang
The Current State of Table API in 2022
The Current State of Table API in 2022Flink Forward Flink Forward San Francisco 2022.
The Table API is one of the most actively developed components of Flink in recent time. Inspired by databases and SQL, it encapsulates concepts many developers are familiar with. It can be used with both bounded and unbounded streams in a unified way. But from afar it can be difficult to keep track of what this API is capable of and how it relates to Flink's other APIs. In this talk, we will explore the current state of Table API. We will show how it can be used as a batch processor, a changelog processor, or a streaming ETL tool with many built-in functions and operators for deduplicating, joining, and aggregating data. By comparing it to the DataStream API we will highlight differences and elaborate on when to use which API. We will demonstrate hybrid pipelines in which both APIs interact with one another and contribute their unique strengths. Finally, we will take a look at some of the most recent additions as a first step to stateful upgrades.
by
David Andreson
Flink SQL on Pulsar made easy
Flink SQL on Pulsar made easyFlink Forward Flink Forward San Francisco 2022.
Based on the new Flink-Pulsar connector, we implemented Flink's TableAPI and Catalog to help users to interact with the Pulsar cluster via Flink SQL easily. We would like to go through the design and implementation of the SQL connector in the following aspects:
1. Two different modes of use Pulsar as a metadata store
2. Data format transformation and management
3. SQL semantics support within Pulsar context
by
Sijie Guo & Neng Lu
Dynamic Rule-based Real-time Market Data Alerts
Dynamic Rule-based Real-time Market Data AlertsFlink Forward Flink Forward San Francisco 2022.
At Bloomberg, we deal with high volumes of real-time market data. Our clients expect to be notified of any anomalies in this market data, which may indicate volatile movements in the markets, notable trades, forthcoming events, or system failures. The parameters for these alerts are always evolving and our clients can update them dynamically. In this talk, we'll cover how we utilized the open source Apache Flink and Siddhi SQL projects to build a distributed, scalable, low-latency and dynamic rule-based, real-time alerting system to solve our clients' needs. We'll also cover the lessons we learned along our journey.
by
Ajay Vyasapeetam & Madhuri Jain
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Exactly-Once Financial Data Processing at Scale with Flink and PinotFlink Forward Flink Forward San Francisco 2022.
At Stripe we have created a complete end to end exactly-once processing pipeline to process financial data at scale, by combining the exactly-once power from Flink, Kafka, and Pinot together. The pipeline provides exactly-once guarantee, end-to-end latency within a minute, deduplication against hundreds of billions of keys, and sub-second query latency against the whole dataset with trillion level rows. In this session we will discuss the technical challenges of designing, optimizing, and operating the whole pipeline, including Flink, Kafka, and Pinot. We will also share our lessons learned and the benefits gained from exactly-once processing.
by
Xiang Zhang & Pratyush Sharma & Xiaoman Dong
Processing Semantically-Ordered Streams in Financial Services
Processing Semantically-Ordered Streams in Financial ServicesFlink Forward Flink Forward San Francisco 2022.
What if my data is already in order? Stream Processing has given us an elegant and powerful solution for running analytic queries and logic over high volumes of continuously arriving data. However, in both Apache Flink and Apache Beam, the notion of time-ordering is baked in at a very low level, making it difficult to express computations that are interested in a semantic-, rather than time-ordering of the data. In financial services, what often matters the most about the data moving between systems is not when the data was created, but in what order, to the extent that many institutions engineer a global sequencing over all data entering and produced by their systems to achieve complete determinism. How, then, can financial institutions and others best employ Stream Processing on streams of data that are already ordered? I will cover various techniques that can make this work, as well as seek input from the community on how Flink might be improved to better support these use-cases.
by
Patrick Lucas
Tame the small files problem and optimize data layout for streaming ingestion...
Tame the small files problem and optimize data layout for streaming ingestion...Flink Forward Flink Forward San Francisco 2022.
In modern data platform architectures, stream processing engines such as Apache Flink are used to ingest continuous streams of data into data lakes such as Apache Iceberg. Streaming ingestion to iceberg tables can suffer by two problems (1) small files problem that can hurt read performance (2) poor data clustering that can make file pruning less effective. To address those two problems, we propose adding a shuffling stage to the Flink Iceberg streaming writer. The shuffling stage can intelligently group data via bin packing or range partition. This can reduce the number of concurrent files that every task writes. It can also improve data clustering. In this talk, we will explain the motivations in details and dive into the design of the shuffling stage. We will also share the evaluation results that demonstrate the effectiveness of smart shuffling.
by
Gang Ye & Steven Wu
Batch Processing at Scale with Flink & Iceberg
Batch Processing at Scale with Flink & IcebergFlink Forward Flink Forward San Francisco 2022.
Goldman Sachs's Data Lake platform serves as the firm's centralized data platform, ingesting 140K (and growing!) batches per day of Datasets of varying shape and size. Powered by Flink and using metadata configured by platform users, ingestion applications are generated dynamically at runtime to extract, transform, and load data into centralized storage where it is then exported to warehousing solutions such as Sybase IQ, Snowflake, and Amazon Redshift. Data Latency is one of many key considerations as producers and consumers have their own commitments to satisfy. Consumers range from people/systems issuing queries, to applications using engines like Spark, Hive, and Presto to transform data into refined Datasets. Apache Iceberg allows our applications to not only benefit from consistency guarantees important when running on eventually consistent storage like S3, but also allows us the opportunity to improve our batch processing patterns with its scalability-focused features.
by
Andreas Hailu
Recently uploaded (20)
How to join illuminati Agent in uganda call+256776963507/0741506136
How to join illuminati Agent in uganda call+256776963507/0741506136illuminati Agent uganda call+256776963507/0741506136 How to join illuminati Agent in uganda call+256776963507/0741506136
Template_A3nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn
Template_A3nnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnncegiver630 Telangana State, India’s newest state that was carved from the erstwhile state of Andhra
Pradesh in 2014 has launched the Water Grid Scheme named as ‘Mission Bhagiratha (MB)’
to seek a permanent and sustainable solution to the drinking water problem in the state. MB is
designed to provide potable drinking water to every household in their premises through
piped water supply (PWS) by 2018. The vision of the project is to ensure safe and sustainable
piped drinking water supply from surface water sources
Just-In-Timeasdfffffffghhhhhhhhhhj Systems.ppt
Just-In-Timeasdfffffffghhhhhhhhhhj Systems.pptssuser5f8f49 Just-in-time: Repetitive production system in which processing and movement of materials and goods occur just as they are needed, usually in small batches
JIT is characteristic of lean production systems
JIT operates with very little “fat”
Calories_Prediction_using_Linear_Regression.pptx
Calories_Prediction_using_Linear_Regression.pptxTijiLMAHESHWARI Calorie prediction using machine learning
CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...
CTS EXCEPTIONSPrediction of Aluminium wire rod physical properties through AI...ThanushsaranS Prediction of Aluminium wire rod physical properties through AI, ML
or any modern technique for better productivity and quality control.
md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptx
md-presentHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHation.pptxfatimalazaar2004 BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB
Data Science Courses in India iim skills
Data Science Courses in India iim skillsdharnathakur29 This comprehensive Data Science course is designed to equip learners with the essential skills and knowledge required to analyze, interpret, and visualize complex data. Covering both theoretical concepts and practical applications, the course introduces tools and techniques used in the data science field, such as Python programming, data wrangling, statistical analysis, machine learning, and data visualization.
Thingyan is now a global treasure! See how people around the world are search...
Thingyan is now a global treasure! See how people around the world are search...Pixellion We explored how the world searches for 'Thingyan' and 'သင်္ကြန်' and this year, it’s extra special. Thingyan is now officially recognized as a World Intangible Cultural Heritage by UNESCO! Dive into the trends and celebrate with us!
FPET_Implementation_2_MA to 360 Engage Direct.pptx
FPET_Implementation_2_MA to 360 Engage Direct.pptxssuser4ef83d Engage Direct 360 marketing optimization sas entreprise miner
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management By James Francis, CEO of Paradigm Asset Management
In the landscape of urban safety innovation, Mt. Vernon is emerging as a compelling case study for neighboring Westchester County cities. The municipality’s recently launched Public Safety Camera Program not only represents a significant advancement in community protection but also offers valuable insights for New Rochelle and White Plains as they consider their own safety infrastructure enhancements.
How to join illuminati Agent in uganda call+256776963507/0741506136
How to join illuminati Agent in uganda call+256776963507/0741506136illuminati Agent uganda call+256776963507/0741506136
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...
Safety Innovation in Mt. Vernon A Westchester County Model for New Rochelle a...James Francis Paradigm Asset Management
Eron Wright - Introducing Flink on Mesos
- 1. Introducing Flink on Mesos Eron Wright – [email protected] DELL EMC @eronwright
- 2. 2 of 15 What is Apache Mesos? • A popular cluster manager (similar to YARN) • MakesavailableCPU, memory, & diskresources • Uniquecapabilitiesforstorageservices • Emerging asa foundationfordata-centric,convergedinfrastructure • Provides a programming model for using cluster resources • A Mesosprogram is calleda “framework” • Packaged into an open-source distribution called DCOS • Prescribesbestpracticesrelatedto Mesosframeworks, relatedservices,etc.
- 3. 3 of 15 Why Flink on Mesos? • Flink works best on a cluster manager – Easy to scale each job independently – Externalize scheduling logic (fairness, quota, …) – Good job isolation • Flink can benefit from unique Mesos capabilities – Disk resources – Dynamic resource management – Unique management features (e.g. inverse offers for controlled downscaling & maintenance)
- 4. Demo
- 6. 6 of 15 Introduction Flink Master Process • The Flink Master Process is: – The “Application Master” for a single Flink cluster – A Mesos framework! • Hosts numerous components: – Job Manager – Resource Manager (acts as Mesos scheduler) – Artifact Server (HTTP server for Mesos fetcher) • Responsible for TM scaling and recovery – Handles JobManager scale change requests – Stores task state in ZooKeeper host1host2 Master JM RM HTTPD TM TM Mesos
- 7. 7 of 15 How it Works Flink Master Process • Offer handling: – Uses Netflix Fenzo as an optimizer – Gathers offers until all tasks launched • Recovery: – Stores intentional state in ZooKeeper – Master uses leader election – Mesos allows some time for recovery before killing tasks • Monitoring: – Detects task failure; launches replacement automatically. host1host2 Master TM TM 4. Launch Mesos 2. Resource Offers 1. Register 5. Fetch (HTTP) 6. Status update 3. Optimize
- 8. 8 of 15 Configuration Flink Master Process (Con’t) • Framework Info – mesos.resourcemanager.framework.secret – mesos.resourcemanager.framework.principal – mesos.resourcemanager.framework.role • Mesos Master Info – mesos.master: (IP address or ZK lookup info) – mesos.failover-timeout Note: no port configuration is necessary; Mesos automatically assigns ports.
- 9. Dispatcher
- 10. 10 of 15 Introduction Dispatcher • A highly-available service for launching Flink clusters. • A Mesos framework! • Accessed via REST by the CLI • DCOS compatibility: – HTTP-based – Accessible via the Admin Router – (future) JWT authentication • Aligned with FLIP-6 host1 1D 1C 1B 1A host2 2D 2C 2B 2A host3 3D 3C 3B 3A host4 4D 4C 4B 4A Dispatcher Master TM TM TMTM Master CLI TM Mesos
- 11. 11 of 15 Framework Hierarchy Dispatcher (Con’t) • Nesting of frameworks is a common Mesos pattern. Here, Marathon launches the dispatcher, which launches the Flink Master Process, etc. • Architecturally, it avoids a dependency on the Marathon API. For example, Aurora could be used here in place of Marathon. Dispatcher Master Maratho n TM (Task) (Task) (Task)
- 12. 12 of 15 Launching a Session Dispatcher (Con’t) • Use: mesos-session.sh • CLI uploads files to dispatcher via HTTP – Flink Configuration – Supplemental files (--ship) – Keytabs – Certificates • Dispatcher adds additional elements: – Configuration › ZooKeeper Namespace – Flink JAR – … host1 1D 1C 1B 1A host2 2D 2C 2B 2A host3 3D 3C 3B 3A host4 4D 4C 4B 4A Dispatcher Master TM TM CLI HTTP(S) TM HTTP(S) Mesos
- 13. 13 of 15 Dispatcher Deployment Modes Dispatcher (Con’t) • Dispatcher is usable in two ways • Remote Mode: – Recommended for detached execution • Local Mode: – Recommended for simple, interactive sessions (e.g. flink shell) 3C 3B 3A 4C 4B 4A Dispatcher Master Master CLI HTTP(S) 3C 3B 3A 4C 4B 4A Master CLI + Dispatcher Local Mode Remote Mode
- 14. Summary
- 15. 15 of 15 Future Directions • Dynamic Scaling – Add/remove Task Managers in response to scale changes over a job’s lifetime – Support Mesos maintenance procedures (e.g. inverse offers) • Dispatcher Evolution (FLIP-6) – Generalize to support all deployment scenarios, unified CLI – Provide a centralized Web UI (incl. job history) – Authentication Support (e.g. OAuth 2.0) • Docker Image Support – Tracking the “Mesos unified containerizer” • Mesos Disk Support – Allocate multiple disks for Task Manager temp space – Scale up the I/O
- 16. 16 of 15 Project Status • Targeted for: Flink 1.2 • Contributors: – Eron Wright (Dell EMC) – Maximilian Michels (data Artisans) • Design Doc: – Mesos Integration on Google Docs • JIRAs: – FLINK-1984 – Integrate Flink with Apache Mesos • Code: – https://github.com/EronWright/flink/tree/feature-FLINK-1984-T2