Serverless computing with AWS Lambda
3 likes3,066 views
The document discusses the evolution of cloud computing towards serverless architecture with a focus on AWS Lambda, a serverless and event-driven compute service. It outlines its capabilities, benefits, and various use cases for real-time data processing, highlighting workflows using services such as Amazon S3, Kinesis, and DynamoDB. Additionally, it provides tips for effectively utilizing AWS Lambda, including automation and deployment strategies.
![Example – Video Clip Transcode
// Transcode video files uploaded to an S3 bucket
var async = require('async');
var fs = require('fs');
var exec = require('child_process').exec;
var aws = require('aws-sdk');
var s3 = new aws.S3();
// Lambda entry point
exports.handler = function(event, context) {
var inputVideo = '/tmp/myVideo.avi';
var outputVideo = '/tmp/myVideo.mp4';
var inputBucket = event.Records[0].s3.bucket.name;
var inputKey = event.Records[0].s3.object.key;
var outputBucket = inputBucket + '-out';
var outputKey = inputKey.substr(0,inputKey.length-4) + '.mp4';
// Runs the array of functions in series, each passing their results to the next
async.waterfall([
Set up variables
Serialize steps](https://image.slidesharecdn.com/serverlesscomputingwithawslambda-160311105646/85/Serverless-computing-with-AWS-Lambda-18-320.jpg)
![Example – Video Clip Transcode
// Download the object from S3 into a buffer
function download(next) {
s3.getObject({Bucket: inputBucket, Key: inputKey}, next); },
// Write the buffer to a file
function write(response, next) {
fs.writeFile(inputVideo, response.Body, next); },
// Spawn a process to transcode
function transcode(next) {
exec("./ffmpeg -i " + inputVideo + " -vcodec mpeg4 -b:v 1200k “
+ outputVideo, next); },
// Read the file to a buffer
function read(next) {
fs.readFile(outputVideo, next); },
// Upload the buffer to an object in S3
function upload(objectBuffer, next) {
s3.putObject({Bucket: outputBucket, Key: outputKey, Body: objectBuffer},
context.done); },
], context.fail);
};
Get file from S3
Write to disk
ffmpeg transcode
Read from disk
Upload to S3](https://image.slidesharecdn.com/serverlesscomputingwithawslambda-160311105646/85/Serverless-computing-with-AWS-Lambda-19-320.jpg)
Serverless computing with AWS Lambda
- 1. Serverless computing with AWS Lambda Matt McClean AWS Solutions Architect 1
- 2. What to expect from this session • Evolution of Cloud Compute toward Serverless • What is AWS Lambda? • Building a serverless data processing architecture • Real-time data processing architecture patterns • Tips and tricks for using AWS Lambda • Live demo of Apigee + AWS Lambda app
- 3. Evolution of Cloud Compute towards Serverless 3 ©2016 Apigee. All Rights Reserved.
- 4. Evolution of Compute – Public Cloud Infrastructure Instances Application code
- 5. Evolution of Compute – Containers Infrastructure Instances Application code Containers
- 6. Evolution of Compute – Serverless Application code
- 7. What is AWS Lambda? 7 ©2016 Apigee. All Rights Reserved.
- 8. AWS Lambda Serverless, event-driven compute service Lambda = microservice without servers
- 9. AWS Lambda – Benefits EVENT-DRIVEN SCALE SERVERLESS SUBSECOND BILLING
- 10. AWS Lambda – Capabilities BRING YOUR OWN CODE SIMPLE RESOURCE MODEL FLEXIBLE INVOCATION PATHS GRANULAR PERMISSIONS CONTROL
- 11. AWS Lambda – How it Works AUTHORING STATELESS DEPLOYMENT MONITORING & LOGGING
- 12. AWS Lambda – Partner Ecosystem
- 13. Building a Serverless Data Processing Architecture 13 ©2016 Apigee. All Rights Reserved.
- 14. SOURCE of data Fleet of servers poll for changes Listening to source Pushes to queue Pull off queue Fleet of servers act as workers to process the data Auto-scale worker nodes to adjust with load S3 objects Amazon Kinesis records DynamoDB tables SNS messages IoT devices … Cross-AZ replication for high availability Load balancer to distribute traffic Data Processing Architecture with Servers Application code
- 15. SOURCE of data Attach a Lambda function And that’s it! Lambda does Listening/polling Queuing Auto scaling Redundancy Load balancing Data Processing Architecture Without Servers Application code
- 16. Real-Time Data Processing Architecture Patterns 16 ©2016 Apigee. All Rights Reserved.
- 17. Sample Real-Time File Processing Workflow Using Amazon S3 & AWS Lambda Amazon S3 AWS Lambda Amazon S3 New file uploaded Amazon DynamoDB
- 18. Example – Video Clip Transcode // Transcode video files uploaded to an S3 bucket var async = require('async'); var fs = require('fs'); var exec = require('child_process').exec; var aws = require('aws-sdk'); var s3 = new aws.S3(); // Lambda entry point exports.handler = function(event, context) { var inputVideo = '/tmp/myVideo.avi'; var outputVideo = '/tmp/myVideo.mp4'; var inputBucket = event.Records[0].s3.bucket.name; var inputKey = event.Records[0].s3.object.key; var outputBucket = inputBucket + '-out'; var outputKey = inputKey.substr(0,inputKey.length-4) + '.mp4'; // Runs the array of functions in series, each passing their results to the next async.waterfall([ Set up variables Serialize steps
- 19. Example – Video Clip Transcode // Download the object from S3 into a buffer function download(next) { s3.getObject({Bucket: inputBucket, Key: inputKey}, next); }, // Write the buffer to a file function write(response, next) { fs.writeFile(inputVideo, response.Body, next); }, // Spawn a process to transcode function transcode(next) { exec("./ffmpeg -i " + inputVideo + " -vcodec mpeg4 -b:v 1200k “ + outputVideo, next); }, // Read the file to a buffer function read(next) { fs.readFile(outputVideo, next); }, // Upload the buffer to an object in S3 function upload(objectBuffer, next) { s3.putObject({Bucket: outputBucket, Key: outputKey, Body: objectBuffer}, context.done); }, ], context.fail); }; Get file from S3 Write to disk ffmpeg transcode Read from disk Upload to S3
- 20. Sample Real-Time Stream Processing Workflow Using Amazon Kinesis & AWS Lambda Amazon Kinesis AWS Lambda Amazon Redshift Amazon SNS New records available
- 21. Sample Real-Time DB Triggers Workflow Using Amazon DynamoDB & AWS Lambda New data available Amazon DynamoDB AWS Lambda Amazon DynamoDB Amazon Redshift
- 22. Sample Real-Time Message Handling Workflow Using Amazon SNS & AWS Lambda New message published Amazon SNS AWS Lambda Amazon SNS Amazon Kinesis
- 23. Sample CRUD Backend Workflow Using Amazon API Gateway & AWS Lambda New API called Amazon API Gateway AWS Lambda Amazon S3 Amazon DynamoDB
- 24. More Triggers for AWS Lambda AWS CloudFormation Custom Resources Amazon SES Actions Amazon Cognito Sync Triggers … and the list will continue to grow! AWS IoT ActionsAWS Lambda Scheduled Events Amazon Echo Skills Amazon SWF Tasks
- 25. Tips and Tricks for using AWS Lambda 25 ©2016 Apigee. All Rights Reserved.
- 26. Using Lambda to Audit CloudTrail Activity AWS Lambda Amazon S3Amazon CloudTrail Amazon SNS AWS IAM
- 27. Using Lambda for Automated Infrastructure Management AWS Lambda Amazon SNS Amazon CloudWatch Alarm ec2 runInstance ecs startTask beanstalk updateApp kinesis splitShard Any API call https://aws.amazon.com/blogs/compute/scaling-amazon-ecs-services-automatically-using-amazon-cloudwatch-and-aws-lambda/ Amazon CloudWatch Event
- 28. Using Lambda to Forward AWS Events to Slack AWS Lambda Amazon SNS Amazon CloudWatch Events Auto Scaling Slack https://aws.amazon.com/blogs/aws/new-slack-integration-blueprints-for-aws-lambda/
- 29. Using Lambda to Deploy Lambda Functions https://aws.amazon.com/blogs/compute/dynamic-github-actions-with-aws-lambda/ AWS Lambda Amazon SNS GitHub Repo lambda createFn ()
- 30. Using Lambda in your software delivery pipeline AWS Lambda AWS CodeCommit AWS CodePipeline https://aws.amazon.com/about-aws/whats-new/2016/01/aws-codepipeline- adds-support-for-triggering-aws-lambda-functions/
- 31. A Few Other Tips and Tricks for Extending AWS Lambda Functionality Ø Use /tmp space as cache Ø Run arbitrary binaries Ø Use Grunt and Jenkins plugins for deploying Ø Build event-driven Amazon ECS Ø Try out Serverless Framework
- 33. API Management covers the entire API Lifecycle 33 Design Develop Secure Publish Monitor Analyze Monetize Scale AWS API Gateway
- 34. Demo 34
- 35. Visit http://aws.amazon.com/lambda, the AWS Compute blog, or the Lambda forum to learn more and to get started using Lambda. More info
- 36. Thank you