Getting started with streaming analytics: Deep Dive
0 likes185 views
Now that we know how to create simple streaming analytics pipelines, it is time to learn something more interesting. In this session I will show you how to add Complex Event Processing to your Apache Flink (or Kinesis Data Analytics) application using JAVA. For those of you that prefer SQL, I will show you how to run streaming analytics using only SQL.
Ad
More Related Content
Similar to Getting started with streaming analytics: Deep Dive (7)
More from javier ramirez (20)
Ad
Recently uploaded (20)
Ad
Getting started with streaming analytics: Deep Dive
- 1. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Part 3 of 3: Deep Dive Getting started with streaming analytics Javier Ramirez AWS Developer Advocate @supercoco9
- 2. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Agenda Working with Avro and schemas Time windows, session windows, and accumulators Complex Event Processing Streaming analytics with SQL
- 3. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Ingestion/in-stream storage: Apache Kafka A distributed streaming platform Concepts: Producers Topics Brokers Consumers
- 4. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Apache Avro Apache Avro™ is a data serialization system. Avro provides: • Rich data structures. • A compact, fast, binary data format. • A container file, to store persistent data. • Remote procedure call (RPC). • Simple integration with dynamic languages. Code generation is not required
- 5. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Confluent Schema Registry https://docs.confluent.io/current/schema-registry/index.html
- 6. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Stream Processing: Apache Flink Stateful computation over Data Streams Concepts: Job Manager/Workers Source DataStream Transforms/Operators TableAPI/SQL Sinks
- 7. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Different types of streaming windows 13:00 14:008:00 9:00 10:00 11:00 12:00 Processing Time Event Time Processing Time 11:0010:00 15:0014:0013:0012:00 11:0010:00 15:0014:0013:0012:00 Input Output Event Time Processing Time 11:0 0 10:0 0 15:0 0 14:0 0 13:0 0 12:0 0 11:0 0 10:0 0 15:0 0 14:0 0 13:0 0 12:0 0 Input Output
- 8. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Apache Flink Windows https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html
- 9. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential FlinkCEP Complex Event Processing for Flink FlinkCEP is the Complex Event Processing (CEP) library implemented on top of Flink. It allows you to detect event patterns in an endless stream of events, giving you the opportunity to get hold of what’s important in your data.
- 10. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Kinesis Data Analytics for SQL • Sub-second end to end processing latencies • SQL steps can be chained together in serial or parallel steps • Build applications with one or hundreds of queries • Pre-built functions include everything from sum and count distinct to machine learning algorithms • Aggregations run continuously using window operators
- 11. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Automatic schema discovery which works for CSV and JSON data Supports multiple event types, arbitrary object nesting, single level of array nesting Connect to Streaming Data Sources Easily connect to Kinesis Data streams and Kinesis Data Firehose delivery streams Amazon Kinesis Data Streams Amazon Kinesis Data Firehose
- 12. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Built-in AWS Lambda integration provides flexible pre- processing ahead of SQL code for: • Normalizing different event types • Converting other data formats (AVRO, Protobuf, ZIP) to JSON and CSV • Custom enrichment from database tables or API calls Pre-process Data Streams Using AWS Lambda AWS Lambda function raw data Amazon Kinesis Data Analytics application transformed data SQL code source destination
- 13. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Fast, iterative development with SQL templates in console to get started Interactive SQL Editor
- 14. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Window Types • Sliding, tumbling, and custom windows • Tumbling windows are fixed size and grouped keys do not overlap
- 15. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential • Add a SQL table to your streaming application from Amazon S3 • Periodically update the table by calling the update application API Enrich your Data Stream using Amazon S3 Data In-application stream Amazon Kinesis Data Analytics application SQL code joining table and stream streaming source destination Amazon S3 In-application table
- 16. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential Getting Started https://ci.apache.org/projects/flink/flink-docs-stable/ Apache Flink official docs https://docs.aws.amazon.com/msk/latest/developerguide/what-is-msk.html Getting started with Apache Kafka/Amazon MSK https://aws.amazon.com/kinesis/ Amazon Kinesis Services for streaming data https://aws.amazon.com/elasticsearch-service/ Amazon ElasticSearch Service https://kafka.apache.org/documentation/ Apache Kafka official docs
- 17. © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential and Trademark © 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential ThanksJavier Ramirez AWS Developer Advocate @supercoco9