Scribd
Upload
69 views

Becoming An Expert On Streaming Data Architectures With AWS Lambda SVS314

The document discusses using AWS Lambda with streaming data architectures. It provides examples of using Lambda with DynamoDB Streams, Kinesis Data Streams, Enhanced Fan-Out consumers, and Amazon Managed Streaming for Kafka. It also addresses challenges around iterator age spikes, optimizing consumer latency, and increasing processing throughput.

Uploaded by

Datei
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
69 views

Becoming An Expert On Streaming Data Architectures With AWS Lambda SVS314

The document discusses using AWS Lambda with streaming data architectures. It provides examples of using Lambda with DynamoDB Streams, Kinesis Data Streams, Enhanced Fan-Out consumers, and Amazon Managed Streaming for Kafka. It also addresses challenges around iterator age spikes, optimizing consumer latency, and increasing processing throughput.

Uploaded by

Datei
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 12

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.

SVS314

Becoming an expert on
streaming data architectures
with AWS Lambda
Heeki Park
Principal Solutions Architect
Amazon Web Services

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon DynamoDB Streams

Client DynamoDB DynamoDB AWS Lambda


application Stream Lambda function

Client application Lambda service


writes to the reads from the
DynamoDB table DynamoDB stream

DynamoDB stream Lambda service synchronously


captures a time-ordered invokes the function with a
sequence of item-level batch of records
modifications
© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams

Client Amazon AWS Lambda


application Kinesis Lambda function

Client application Lambda service


produces records reads from the
into the stream stream

Lambda service synchronously


invokes the function with a
batch of records

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams

Client Amazon AWS


application Kinesis Lambda

Client application Lambda service


reads from the Lambda
produces records
into the stream stream function

Lambda service synchronously


invokes the function with a
batch of records

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams

Client Amazon EFO AWS Lambda


application Kinesis consumer Lambda function

Client application Lambda service


produces records reads from the EFO
into the stream consumer

Enhanced Fan-Out Lambda service synchronously


consumer functions as a invokes the function with a
dedicated consumer batch of records

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Kinesis Data Streams

Client Amazon EFO AWS


application Kinesis consumer Lambda

Client application Lambda service


reads from the EFO Lambda
produces records
into the stream consumer function

Enhanced Fan-Out Lambda service synchronously


consumer functions as a invokes the function with a
dedicated consumer batch of records

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon Managed Streaming for Kafka

Client Amazon AWS Lambda


application MSK Lambda function

Client application Lambda service


produces records reads from the
into the stream stream

Lambda service synchronously


invokes the function with a
batch of records

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenge 1
The IteratorAge metric on my Kinesis or DynamoDB stream periodically
grows rapidly. How do I address those spikes?

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenge 2
What are ways that I can optimize consumer latency for my streaming
applications in order to get the closest to real time as possible?

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenge 3
What are ways that I can increase processing throughput for my
streaming applications in order to process large amounts of streaming
data quickly?

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thank you!
Heeki Park
[email protected]

© 2021, Amazon Web Services, Inc. or its affiliates. All rights reserved.

You might also like