Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink Forward 2019 SF
11 likes1,726 views
Container technology experiences an ever increasing adoption throughout many industries. Not only does this technology make your applications portable across different machines and operating systems, it also allows to scale applications in a matter of seconds. Moreover, it significantly simplifies and speeds up deployments which decreases development and operation costs. Consequently, more and more Flink deployments run in containerized environments which poses new challenges for Flink. In this talk, we will take a look at Flink's current and future container support which will make it a first class citizen of the container world. First of all, we will explain how the new reactive execution mode will solve the problem of seamless application scaling and how it blends in with any environment. Complementary to the reactive mode, the active execution mode demonstrates its strengths when it comes to changing workloads such as batch jobs. Last but not least, we will take a look beyond Flink's own nose and investigate how Flink can be used together with Kubernetes operators or data Artisans' Application Manager. We will conclude the talk with a short demo of Flink's native Kubernetes support and giving an outlook on future developments in the container realm.
Ad
More Related Content
What's hot (20)
Similar to Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink Forward 2019 SF (20)
Ad
More from Till Rohrmann (16)
Ad
Recently uploaded (20)
Future of Apache Flink Deployments: Containers, Kubernetes and More - Flink Forward 2019 SF
- 1. © 2019 Ververica Till Rohrmann, Engineering Lead @stsffap Future of Apache Flink Deployments: Containers, Kubernetes and More
- 2. © 2019 Ververica2 Advancements of APIs & Supported Use Cases Batch Processing process static and historic data Data Stream Processing realtime results from data streams Event-driven Applications data-driven actions and services • Deployment & operations features did not keep up with APIs
- 3. © 2019 Ververica3 Flink as a Library • Deploying Flink applications should be as easy as starting a process • Application-oriented instead of cluster-oriented • Start multiple processes which form distributed execution • Removing the cluster out of the equation P1 P2 P3 P4 New process
- 4. © 2019 Ververica4 Resource Management & Autoscaling • Multiple short lived stages • Different resource requirements per stage • Efficient execution requires control over resources • Flink allocates actively resources • Long running, continuously processing operators • Constrained by external systems and SLAs • External system with holistic view over resources better suited for scaling decisions • Flink reacts to available resources Real-time & data-driven Applications Batch Analytics Continuous Processing
- 5. © 2019 Ververica5 Reactive vs. Active Execution Mode • Active mode – Flink is aware of underlying cluster framework – Flink allocate resources – E.g. existing YARN and Mesos integration • Reactive mode – Flink is oblivious to its runtime environment – External system allocates and releases resources – Flink scales with respect to available resources – Relevant for environments: Kubernetes, Docker, as a library
- 6. © 2019 Ververica6 Auto Scaling • Latency • Throughput • Connector signals • Available resources Scale up
- 7. © 2019 Ververica7 Containerized Deployments • Popular to deploy Flink in containerized environments • Why Flink on containers? – Self-contained runtime environment that includes application code, libraries, dependencies, and configuration files – Ease of operations by clean separation of concerns – Offers dynamic resource allocation • Reactive mode suits containerized deployments well • Demand for active mode to run batch workloads
- 8. © 2019 Ververica8 Containerized Deployments (cont.) • De-facto industry standard for container orchestration • Resource-oriented with declarative configuration – You tell K8s the desired state, and a background process asynchronously makes it happen – “3 replicas of this container should be kept running” – “a load balancer should exist, listening on port 443, backed by containers with this label” • Core resource types: – Pod: a group of one or more containers running on a node – Deployment: set of n identical pods Brief Introduction to Kubernetes
- 9. © 2019 Ververica Active & Reactive Mode
- 10. © 2019 Ververica10 Revamped Distributed Architecture (FLIP-6) – Better separation of concerns – RESTful API for client-cluster communication – Full resource elasticity Dispatcher Client ResourceManager ClusterManager JobManager TaskManager 1. Submit job 2. Start job 3. Request slots 4. Allocate resources 5. Start TaskManager 6. Execute job
- 11. © 2019 Ververica11 Active K8s Integration K8s deployment controller Client TaskManager JobManager K8sResMngr ApplicationMaster TaskManager (3) Submit job (1) Submit AM deployment (4) Start JM (5) Request slots (7) Start TM pod Dispatcher
- 12. © 2019 Ververica12 Reactive Mode • Relies on external system to start/release TaskManagers – Kubernetes Horizontal Pod Autoscaler – GCP Autoscaling – AWS Auto Scaling Group • Re-scale job if resources are added/removed • By definition works with all cluster managers Flink cluster JM TM TM ASG Start new TM if CPU% > threshold Register & offer slots Event rate over time
- 13. © 2019 Ververica Does Any of This Work? • Demo: Active K8s Integration & Reactive Container Mode
- 14. © 2019 Ververica14 Current State • Recommendations on how to deploy job and session cluster on K8s – Docker image build script – K8s resource configs • StandaloneJobClusterEntryPoint to start a (containerized) job cluster • Work in progress: – Active K8s integration • https://issues.apache.org/jira/browse/FLINK-9953 • Kudos to Chunhui-Shi • git clone -b native-k8s [email protected]:chunhui-shi/flink.git – Reactive Container Mode with automatic rescaling • https://issues.apache.org/jira/browse/FLINK-10407
- 15. © 2019 Ververica15 TL; DL • Reactive container mode – Support applications as first class citizen – Ease of operations • Active K8s integration coming soon • Join the community! – Subscribe to mailing lists – Participate in Flink development – Design the future
- 16. © 2019 Ververica Thank you! Questions?