Apache flink 1.7 and Beyond

Editor's Notes

• #7: State evolution was missing piece to properly evolve Flink applications (topology changes were already possible) State evolution is particularly important for users who have amassed a lot of state which they cannot discard/recompute because it would be too expensive Evolving the state is a cheap solution by adding/removing fields to capture new features
• #8: State schema evolution is currently supported when using Avro types Limited by what Avro supports wrt schema changes In the future, state transformations are conceivable  Mapping state to a completely different type A => B Dedicated talk available
• #9: Example to motivate temporal tables and temporal joins: Converting currencies of buys and sales which are executed in different currencies Stream A are the incoming buys/sales, when they happened, in which currency they were executed and the amount of transferred money Stream B are the currency rates Goal join buys/sales with the latest currency rate to calculate exact costs/income in some given currency
• #10: Temporal tables are tables which have an additional time/version column The table can contain multiple entries for the same key originating from different times Temporal table allows to join with latest row for a given key wrt some timestamp Perfect for joining buys/sales with their respective currency rate for conversion
• #11: How do you use SQL if you want to extract temporal patterns (e.g. you want to find out how many of your warehouse orders have not been processed correctly) Easy to do with complex event processing which has been developed for this purpose CEP allows you to define patterns via a regular expression like language which can then be extracted from your data For example, let’s assume we have a stream of taxi ride events (passenger X starts ride, passenger Y stops ride) We want to find rides where the taxi picks up two other passengers after the first passenger starts its ride How do we express this with SQL?
• #12: MATCH_RECOGNIZE comes to our rescue MATCH_RECOGNIZE allows to define temporal patterns in your query Support for this key word was added with Flink 1.7 which is also compliant to SQL:2016 Purple defines the pattern: First a start event where a passenger X enters the taxi, then two events where another ride starts (new passengers enter the taxi) and last the event where passenger X leaves the taxi The events of the purple pattern are defined in the DEFINE clause Passengers are identified by the rideId
• #13: Many more SQL improvements where added with Flink 1.7.0 SQL views define virtual tables from a SQL query Refer to dedicated talk by Piotr
• #14: Scala 2.12 is now supported  Users can now use newer Scala version and more powerful Scala 2.12 ecosystem The StreamingFileSink which has been added in 1.6.0 now also supports writing to S3 with exactly once processing guarantees Flink has now a Kafka 2.0 connector being able to read from Kafka 2.0 Flink’s REST API is versioned  No more breaking changes for third party integrations which use the REST API The legacy mode has been removed  Flip-6 only
• #16: If we look at the capability spectrum, Flink shines most in the domain of streaming analytics, continuous processing, event driven applications and with some limitations in batch However, in order to process batch workloads as good as streaming only workloads Flink needs still to improve a bit (e.g. scheduling, dynamic memory management, persistent intermediate results, …) The same applies for the near real-time applications which have low latency SLAs (Flink needs to recover faster in order to fulfill the SLAs, auto-scaling, …) The goal is for Flink to become the primary choice when trying to solve a problem from any of these domains Flink 1.8 will take some incremental steps to extend Flink’s capability spectrum
• #17: More and more people use Flink to develop event-driven applications These applications should be as easy to deploy as starting a local application Atm users need to manage a Flink cluster to deploy an application to The job mode has eased the situation a bit but you still need to operate a cluster Idea: Completely remove the cluster out of the equation by by bundling Flink and application code into one image A process started from this image will automatically connect to the other Flink processes and form a cluster on its own User only needs to start and stop processes that’s all
• #18: In order to make deployment as easy as just described, the community is working on a new execution mode The existing execution mode is the active mode In the active mode, Flink is aware of its underlying resource manager and can talk to it in order to allocate/free resources (e.g. Yarn, Mesos) The community is currently working on adding support for running Flink in active mode on Kubernetes (needs to implement a KubernetesResourceManager) The reactive mode differs in the sense that Flink does not know about its runtime environment Thus, Flink cannot allocate/free resources. Allocation/Freeing needs to be done by an external process New resources (TaskManagers) once started will join the cluster and Flink will automatically make use of these new resources Depending on the job requirements Flink will automatically scale up/down the job wrt to the available resources
• #19: Enable auto-scaling Based on some metrics and wrt to the available resources Flink will decide when and how to rescale a Flink job The community will add a RescalingPolicy interface which allows the operators to define when to signal a rescaling request Flink will make sure that all resources are used efficiently The reactive mode will enable dynamic scaling also in container environments where an external system controls the resources
• #20: A long sought-after goal is the unification of batch and stream processing There are no fundamental differences between batch and stream processing which would prevent the unification Atm Flink handles batch and streaming on the API level separately (exception is the Table API) Big parts of the runtime are shared by batch and streaming workloads (network stack, distributed components) However from the Task level upwards (StreamTask, StreamOperator, BatchTask) there are still separate code paths for batch and streaming Flink 1.8 will help paving the way for a truly unified runtime for batch and streaming workloads
• #21: Scheduling of batch jobs is currently sub-optimal Flink uses lazy scheduling starting from the sources Depending on the intermediate result type it schedules the consumers either when the first data has been produced or when the intermediate result has been completed Often different branches have dependencies and should be scheduled accordingly hash-join has a build and probe side, the build side needs to be processed before the probe side is consumed  No need to schedule the probe side operators Atm, these dependencies are not respected which leads to resource under-utilization
• #22: Better scheduling would be by taking dependencies into account First schedule build side Second schedule probe side and build side of second join operator Last schedule probe side of second join operator
• #23: In order to support different schedulers, the community is currently working on making the scheduler extendable and pluggable Schedulers will receive topology (including operators dependencies) and signals from the ExecutionGraph (e.g. task finished, data available, etc) Based on these signals the scheduler can make scheduling decisions Specialized schedulers for different use cases conceivable (e.g. streaming scheduler which schedules all operators vs. batch scheduler which schedules the topology stage by stage)
• #24: Another way to improve Flink’s batch capabilities is to make Flink’s shuffle service pluggable Atm Flink’s shuffle service is coupled to the lifetime of the TaskManager The shuffle service either keeps results in memory or spills to disk (depending on the exchange mode) The problem is that results are bound to the lifetime of the TaskManager and, thus, it is not possible to release containers after tasks have finished Moreover, there is only one shuffle service implementation which has to work for streaming and batch workloads
• #25: Idea is to make the shuffle service pluggable so that an external and persistent shuffle service can be supplied Such a shuffle service allows to decouple the result partitions from the underlying containers and, thus, containers can be released earlier Possible shuffle service implementations could be based on persisting results to DFS or implemeting a Yarn based shuffle service Making the shuffle service pluggable will make Flink more flexible wrt supported use cases
• #26: Two important features to allow users to define end-to-end SQL only pipelines (without writing any (Java/Scala) code) are in the making Support for external catalogs to easily read from and write to SQL tables Examples: Hive meta store for batch SQL jobs or the confluent schema registry for streaming SQL Prerequisite is that the SQL sources/sinks have been registered at the catalog If this has not happened, then support for a data definition language (DDL) could be helpful With the DDL it will be possible to define SQL tables using a SQL query (schema + meta information) These two features will simplify the usage of SQL with Flink tremendously