Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWS
0 likes337 views
AWS has offered streaming data services for over 5 years, starting with Kinesis Data Streams. Now customers use streaming across most AWS services, including two that support Apache Flink - EMR and Kinesis Data Analytics. The keynote described how customer use of streaming data on AWS has evolved from early customers processing tens of millions of events daily to now processing trillions of events, and how Flink usage has grown both internally at AWS and for customers.
Keynote: Customer Journey with Streaming Data on AWS - Rahul Pathak, AWS
- 1. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Customer Journey with Streaming Data on AWS Amazon Web Services (AWS) offers over 165 fully featured cloud services from data centers globally. AWS launched its first data streaming service, Amazon Kinesis Data Streams, over five years ago. Now, customers are using streaming data across most AWS services including two that support running Apache Flink, Amazon EMR and Amazon Kinesis Data Analytics. In this keynote, we will describe how customers and their use of streaming data has evolved on AWS. We will look at how streaming data and Apache Flink are used externally and internally on AWS, and where we see usage of Apache Flink growing.
- 2. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Rahul Pathak General Manager of Databases, Analytics, and Blockchain Amazon Web Services Customer Journey with Streaming Data on AWS
- 3. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Building with our customers 50+ fully managed streaming capabilities deployed globally in 22 AWS Regions 2019 2013 First fully managed streaming service, Amazon Kinesis Data Streams 2018 Support for Apache Flink based apps in Amazon Kinesis Data Analytics Support for Apache Flink in Amazon EMR 2016
- 4. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Streaming data on AWS in 2013 Internal and external customer struggling with high volume data ā¢ Low latency, continuous data capture ā¢ Durable storage to quickly get data from unreliable sources ā¢ Scale to cost effectively handle lots of data AWS Metering and Billing
- 5. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What did customers look like in the past? The āAverageā customerā¦ was attracted by ease of use ā¢ Java developer ā¢ Had one application processing 10s of millions of events per day ā¢ Application performed extract, buffer, and load to Amazon S3 The āLargeā customerā¦ was attracted by performance and scale ā¢ Had distributed systems experience and was familiar with Hadoop ā¢ Had two applications processing billions of events per day ā¢ Application performed advanced ETL like joins
- 6. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Early streaming data customers on AWS Streaming extract and load to Amazon S3 50 billion daily ad impressions, sub-50 ms responses 100 GB/day clickstreams from 250+ sites
- 7. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Supercell Delivers World-Class Mobile Games Supercell is a Finnish game company known for the hit games Clash of Clans, Hay Day, and Boom Beach. The world of gaming never sleeps ... We owe every player a great experience, and AWS is our platform to make that happen. Sami Yliharju Services Lead, Supercell ā ā ā¢ Started with a non- streaming architecture ā¢ Use streaming data for faster ETL and analytics ā¢ Started with archival apps and kept adding use cases
- 8. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Supercellās data pipeline circa 2012
- 9. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Supercellās streaming pipeline circa 2018
- 10. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simplify common use cases Every customer built a streaming delivery app ā¢ Load streaming data into streams, data lakes and warehouses ā¢ Zero administration and seamless elasticity ā¢ Direct-to-data store integration ā¢ Serverless continuous data transformations
- 11. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Make advanced use cases accessible Few customers were able to move to real-time analytics ā¢ Analyze data streams in real time ā¢ Interact with streaming data in real-time using Apache Flink-based apps ā¢ Build fully managed and serverless stream processing applications
- 12. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apache Flink at AWS Customers run Apache Flink on AWS on different services with varying degree of flexibility and management Amazon Kinesis Data Analytics Amazon Elastic Kubernetes Service Amazon EMR
- 13. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Streaming data on AWS today
- 14. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Streaming data on AWS in 2019 <40% delivery apps 3 apps 3 data stores
- 15. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What do customers do today? The āAverageā customerā¦ is attracted by the ease of use and rich features ā¢ New to streaming data (~50% of customers are still net new to streaming) ā¢ Has several applications processing 100s of millions of events per day The āLargeā customerā¦ is attracted to the above but needs flexibility and elasticity ā¢ Uses many languages including SQL, Java, Python ā¢ Has 10s of applications processing 10s of billions of events per day The āPlatformā customer isā¦attracted to all of the above plus performance ā¢ Builds abstracted platforms on top of streaming services ā¢ Has 100s of applications processing trillions of events per day
- 16. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Streaming services are foundational at Amazon Amazon Go video analytics Amazon.com online catalog Amazon CloudWatch logs Amazon S3 events AWS metering
- 17. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Customers built a wide variety of use cases Near-real-time home valuation (Zestimates) Live clickstream dashboards refreshed under 10s 1 billion events per week from connected devices Real-time game events analytics Built event driven, micro services arch StreamHub Online stylist processing 10 million events/day Facilitate communications between 100+ microservices IoT predictive analytics Log analytics for real-time āsingle pane of glassā Serverless event bus and ingestion pipeline
- 18. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Autodesk makes software for people who make things Autodesk, a leading provider of 3D design and engineering software, wants to do more than create and deliver software. āUltimately, we are improving our software products and offering better service to our customers because of the real-time visibility weāre getting into log data.ā Tommy Li Senior Software Architect, Autodesk ā ā ā¢ Provides cloud services for its design software ā¢ Uses streaming analytics to monitor and improve their customer experience
- 19. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Autodeskās Streaming Architecture Amazon Kinesis Data Streams Amazon EC2 Amazon Elastic Container Service AWS Lambda Amazon Kinesis Data Firehose Amazon Kinesis Data Analytics Amazon Elasticsearch Service Amazon Athena Amazon CloudWatch
- 20. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Data Center Services (DCS) uses Apache Flink DCS maintains and monitors electrical and physical topologies for all AWS Data Centers āWe chose Apache Flink because it provided simple, extensible interfaces and scalability. We chose Kinesis Data Analytics because of its guaranteed uptime, simple deployment mode, and lower ops cost. AWS Data Center Team ā ā ā¢ Write software to interface with equipment in data centers ā¢ Includes electrical power draw, water usage, weather, fan speeds, and host temperatures ā¢ Use insights to drive down cost of data center operations
- 21. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Data Center Services ā¢ Data is captured from Kinesis Data Streams and CDC from a noSQL store (Amazon DynamoDB streams) ā¢ Analytics are calculated using Apache Flink on Kinesis Data Analytics ā¢ Analytics include drift in circuit breaker settings to power utilization and much more
- 22. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What will customers look like in the future? The āAverageā customerā¦ doesnāt even know they are using Flink or a streaming data service The āLargeā customerā¦ has teams across the company with varying levels of technical sophistication The āPlatformā customer isā¦ any company, no longer requiring teams of engineers and years of investment You may already be here but streaming is still new for most
- 23. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Apache Flink at AWS in the future ā¢ Flink is the fastest growing framework reading Kinesis Data Streams ā¢ Usage is still relatively small compared to simplest of solutions (e.g. KafkaConsumer, Kinesis Clients) running on EC2 ā¢ Excited to work with community of further simplifying running Flink both on AWS and anywhere else
- 24. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. It is still Day 1
- 25. Ā© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!