Introduction to AWS Kinesis
Download as PPTX, PDF2 likes648 views
A quick overview of AWS Kinesis: What is Kinesis, what problems does Kinesis solve, and how might you integrate Kinesis with an existing data warehouse.
Ad
More Related Content
Similar to Introduction to AWS Kinesis (20)
Recently uploaded (20)
Ad
Introduction to AWS Kinesis
- 1. Wellington AWS Meetup Introduction to Kinesis
- 2. Who Am I? • Team Leader/Architect in Business Intelligence/databases • 17 years experience. • MCSE BI, OCP DBA, MCDBA • AWS-ASA-2505
- 3. Who are OptimalBI? • Wellington based BI Consultancy • “Making Information Visible”
- 4. Talk Outline 1. Why do we need Kinesis? 2. What is Kinesis? 3. Demo 4. How does it fit into an existing data warehouse 5. When to use Kinesis
- 5. Big Data 1. Volume 2. Velocity 3. Variety
- 6. Kinesis is an answer to Velocity Machine learning looks simple: Data is collected, magic happens, and we output it to our users
- 7. Traditional Business Intelligence Data Store Data Warehouse Query Tool • Periodic, Batch Extract- Transform-Load. • Persistent data source • High latency
- 8. Internet of Things • Large number of sensors. • Self registering • Pushing data • May or may not retain any historic data. = Only one chance to get data
- 9. Batch ETL • Data needs to wait somewhere between loads. • If data is only loaded six hours per day, then four-times as much hardware is needed. • Latency of hours
- 10. DIY Streaming ETL “Realtime” “ETL” cluster
- 11. DIY Streaming ETL 2.0 Add a queue
- 12. DIY Streaming ETL 3+ Cluster more Getting messy, still problems
- 13. Problems with DIY Streaming ETL 1. Message queues deliver once. If you want to fan out to many readers the application in front needs to know about each of them and queue the same message repeatedly. 2. Order of message delivery is not guaranteed. 3. If the program reading data crashes partway through aggregating, messages are lost.
- 14. What is Kinesis • Kinesis is like a message queue, but more scalable and with multiple readers of each message. • Kinesis is like a NOSQL database, but with message delivery and daily purging. • Kinesis is like an Enterprise Service Bus focused on Analytics. • For a limited, if common, use case Kinesis is the best of all.
- 15. Kinesis Qualities • Scalable • Elastic • Durable • Fault Tolerant • Replayable
- 16. Kinesis Components • Each Queue/DB is called a Stream • Each stream scales by adding Shards • Each Shard provides 1 MB/s in and 2MB/s out • Shards are only $0.44/day, so autoscale them to give some safety margin • Also pay about 2 cents per million puts
- 17. Kinesis Client Library • Kinesis expects you to write bespoke producer and consumer programs • KCL provides automatic multi-threading with one worker thread per shard. • Similar to Hadoop, framework handles the lifting the bespoke program does the “reduce” • You have to autoscale the EC2 groups.
- 18. Kinesis Application instances Auto Scaling group instances Auto Scaling group instances Auto Scaling group Amazon Kinesis
- 19. Existing Kinesis Connectors HTTP POST AWS SDK Log4j Flume Fluentd Get* APIs Amazon Kinesis Client Library + Connector Library Apache Storm Amazon Elastic MapReduce Sending Reading
- 20. http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-kinesis.html Standard AWS Demo Script 1. HIVE already running in EMR 2. Create Kinesis Stream 3. Start Producer 4. Configure HIVE as consumer
- 21. Integrating Kinesis into an existing Data Warehouse 1. Access data in near real-time 2. Facilitate more-traditional ETL 3. Archive
- 23. Near Real-time Data 1. Analyze individual transactions 2. Send alerts for both individual transactions and trends 3. Aggregate to feed a live dashboard
- 24. Facilitate Traditional ETL 1. Write lightly transformed data to S3 to batch COPY into Redshift 2. Pre-compute aggregates, then write them to S3 3. Provide a durable, replayable buffer in front of traditional ETL tools.
- 25. Archive 1. In addition to using your data, Kinesis makes it easy to log the full incoming data set to S3. 2. An object store makes more sense for write-once/read-never data than a database.
- 26. When to use Kinesis 1. Internet of Things (IOT) 2. Use for near-real-time access to data. 3. Have more than one consumer for each piece of data.
- 27. Thanks 1. Our sponsors: • API Talent • AWS • OptimalPeople 2. Bronwyn and Wyn 3. AWS for images on slides