Serverless in production, an experience report (CoDe-Conf)
2 likes1,068 views
AWS Lambda has changed the way we deploy and run software, but this new serverless paradigm has created new challenges to old problems - how do you test a cloud-hosted function locally? How do you monitor them? What about logging and config management? And how do we start migrating from existing architectures? In this talk Yan and Scott will discuss solutions to these challenges by drawing from real-world experience running Lambda in production and migrating from an existing monolithic architecture.
![if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi](https://image.slidesharecdn.com/serverlessinprod-181011120942/85/Serverless-in-production-an-experience-report-CoDe-Conf-96-320.jpg)
![if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi
install Serverless framework
as dev dependency](https://image.slidesharecdn.com/serverlessinprod-181011120942/85/Serverless-in-production-an-experience-report-CoDe-Conf-97-320.jpg)
![if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION
elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run int-$STAGE
elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then
STAGE=$2
REGION=$3
PROFILE=$4
npm install
AWS_PROFILE=$PROFILE npm run acceptance-$STAGE
else
usage
exit 1
fi
install Serverless framework
as dev dependency
mitigate version conflicts](https://image.slidesharecdn.com/serverlessinprod-181011120942/85/Serverless-in-production-an-experience-report-CoDe-Conf-98-320.jpg)
More Related Content
What's hot (19)
Similar to Serverless in production, an experience report (CoDe-Conf) (19)
More from Yan Cui (20)
Recently uploaded (20)
Serverless in production, an experience report (CoDe-Conf)
- 1. in production an experience reportan experience report what you should know before you go to production ServerlessServerless
- 2. Yan Cui http://theburningmonk.com @theburningmonk Principal Engineer @
- 3. “Netflix for sports” offices in London, Leeds, Katowice and Amsterdam
- 7. available in Austria, Switzerland, Germany, Japan and Canada, Italy and USA
- 8. available on 30+ platforms
- 10. “Netflix for sports” offices in London, Leeds, Katowice and Amsterdam We’re hiring! Visit engineering.dazn.com to learn more. follow @dazneng for updates about the engineering team. WE’RE HIRING!
- 13. apr, 2016
- 15. hey guys, vote on this post and I’ll announce a winner at 10PM tonight
- 16. 10PM traffic
- 18. low utilisation to leave room for spikes EC2 scaling is slow, so scale earlier
- 19. lots of $$$ for unused resources
- 20. up to 30 mins for deployment deployment required downtime
- 21. - Dan North “lead time to someone saying thank you is the only reputation metric that matters.”
- 23. “what would good look like for us?”
- 24. be small be fast have zero downtime have no lock-step DEPLOYMENTS SHOULD...
- 25. FEATURES SHOULD... be deployable independently be loosely-coupled
- 26. WE WANT TO... minimise cost for unused resources minimise ops effort reduce tech mess deliver visible improvements faster
- 27. nov, 2016
- 28. 170 Lambda functions in prod 1.2 GB deployment packages in prod 95% cost saving vs EC2 15x no. of prod releases per month
- 29. time is a good fit
- 30. 1st function in prod! time is a good fit
- 31. ? time is a good fit 1st function in prod!
- 33. Practices ToolsPrinciples what is good? how to make it good? with what?
- 35. 170 functions ? ? time is a good fit 1st function in prod!
- 38. rebuilt search
- 39. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearch
- 40. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda
- 42. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery
- 43. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery 1 developer, 2 days design production (his 1st serverless project)
- 44. Legacy Monolith Amazon Kinesis Amazon Lambda Google BigQuery “nothing ever got done this fast at Skype!” - Chris Twamley
- 45. - Dan North “lead time to someone saying thank you is the only reputation metric that matters.”
- 55. choose a tried-and-tested deployment framework, don’t invent your own
- 63. TESTING
- 64. amzn.to/29Lxuzu
- 65. Level of Testing 1.Unit do our objects do the right thing? are they easy to work with?
- 67. Level of Testing 1.Unit 2.Integration does our code work against code we can’t change?
- 68. handler
- 69. handler test by invoking the handler
- 70. Level of Testing 1.Unit 2.Integration 3.Acceptance does the whole system work?
- 72. “…We find that tests that mock external libraries often need to be complex to get the code into the right state for the functionality we need to exercise. The mess in such tests is telling us that the design isn’t right but, instead of fixing the problem by improving the code, we have to carry the extra complexity in both code and test…” Don’t Mock Types You Can’t Change
- 73. “…The second risk is that we have to be sure that the behaviour we stub or mock matches what the external library will actually do… Even if we get it right once, we have to make sure that the tests remain valid when we upgrade the libraries…” Don’t Mock Types You Can’t Change
- 74. Don’t Mock Types You Can’t Change Services
- 75. Paul Johnston The serverless approach to testing is different and may actually be easier. http://bit.ly/2t5viwK
- 77. LambdaAPI Gateway DynamoDB Unit Tests
- 78. LambdaAPI Gateway DynamoDB Unit Tests Mock/Stub
- 79. is our request correct? is the request mapping set up correctly?is the API resources configured correctly? are we assuming the correct schema? LambdaAPI Gateway DynamoDB is Lambda proxy configured correctly? is IAM policy set up correctly? is the table created? what unit tests will not tell you…
- 81. most Lambda functions are simple have single purpose, the risk of shipping broken software has largely shifted to how they integrate with external services observation
- 83. optimize towards shipping working software, even if it means slowing down your feedback loop…
- 84. …if a service can’t provide you with a relatively easy way to test the interface in reality, then you should consider using another one. Paul Johnston
- 85. “…Wherever possible, an acceptance test should exercise the system end-to- end without directly calling its internal code. An end-to-end test interacts with the system only from the outside: through its interface…” Testing End-to-End
- 86. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda
- 87. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda Test Input
- 88. Legacy Monolith Amazon Kinesis Amazon Lambda Amazon CloudSearchAmazon API Gateway Amazon Lambda Test Input Validate
- 89. integration tests exercise system’s Integration with its external dependencies my code
- 90. acceptance tests exercise system End-to-End from the outside my code
- 91. integration tests differ from acceptance tests only in HOW the Lambda functions are invoked observation
- 94. CI + CD PIPELINE
- 95. Jenkins build config deploys and tests unit + integration tests deploy acceptance tests
- 96. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run int-$STAGE elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run acceptance-$STAGE else usage exit 1 fi
- 97. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run int-$STAGE elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run acceptance-$STAGE else usage exit 1 fi install Serverless framework as dev dependency
- 98. if [ "$1" = "deploy" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE 'node_modules/.bin/sls' deploy -s $STAGE -r $REGION elif [ "$1" = "int-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run int-$STAGE elif [ "$1" = "acceptance-test" ] && [ $# -eq 4 ]; then STAGE=$2 REGION=$3 PROFILE=$4 npm install AWS_PROFILE=$PROFILE npm run acceptance-$STAGE else usage exit 1 fi install Serverless framework as dev dependency mitigate version conflicts
- 101. build.sh allows repeatable builds on both local & CI
- 103. Auto Auto Manual
- 105. LOGGING
- 107. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now?
- 108. 2016-07-12T12:24:37.571Z 994f18f9-482b-11e6-8668-53e4eab441ae GOT is off air, what do I do now? UTC Timestamp API Gateway Request Id your log message
- 109. Me Logs are not easily searchable in CloudWatch Logs.
- 110. CloudWatch Logs
- 111. AWS Lambda invokes AWS Lambda stdout asynchronously any log aggregation service CloudWatch Logs
- 112. CloudWatch Logs AWS Lambda AWS Lambda stdout any log aggregation service asynchronously invokes
- 113. …
- 114. CloudWatch Events
- 117. DISTRIBUTED TRACING
- 119. a user my followers didn’t receive my new post!
- 120. where could the problem be?
- 121. correlation IDs* * eg. request-id, user-id, yubl-id, etc.
- 122. wrap HTTP client & AWS SDK clients to forward captured correlation IDs
- 123. kinesis client http client sns client
- 126. use X-Ray for performance tracing
- 127. Amazon X-Ray
- 128. Amazon X-Ray
- 129. X-Ray traces do not span over async event sources
- 132. no place to install agents/daemons
- 133. • invocation Count • error Count • latency • throttling • granular to the minute • support custom metrics
- 134. • same metrics as CW • better dashboard • support custom metrics https://www.datadoghq.com/blog/monitoring-lambda-functions-datadog/
- 138. my code
- 139. my code
- 140. my code internet internet press button something happens
- 141. those extra 10-20ms for sending custom metrics would compound when you have microservices and multiple APIs are called within one slice of user event
- 142. Amazon found every 100ms of latency cost them 1% in sales. http://bit.ly/2EXPfbA
- 143. no more background processing, other than what the platform provides
- 144. console.log(“hydrating yubls from db…”); console.log(“fetching user info from user-api”); console.log(“MONITORING|1489795335|27.4|latency|user-api-latency”); console.log(“MONITORING|1489795335|8|count|yubls-served”); timestamp metric value metric type metric namemetrics logs
- 145. CloudWatch Logs AWS Lambda ELK stack logs metrics CloudWatch
- 147. don’t forget to setup dashboards & alarms
- 148. CONFIG MANAGEMENT
- 149. design for easy & quick propagation of config changes
- 151. me Environment variables make it hard to share configurations across functions.
- 152. me Environment variables make it hard to implement fine-grained access to sensitive info.
- 157. SSM Parameter Store
- 159. Secrets Manager
- 160. sensitive data should be encrypted in-flight, and at-rest
- 161. enforce role-based access to sensitive configuration values
- 162. SSM Parameter Store HTTPS role-based access encrypted in-flight
- 163. SSM Parameter Store encrypt role-based access
- 164. SSM Parameter Store encrypted at-rest
- 165. HTTPS role-based access SSM Parameter Store encrypted in-flight
- 166. invest into a robust client library
- 167. fetch & cache at cold-start
- 171. https://www.cloudzero.com/blog/2018-cloud-and-serverless-ecosystem-survey take the survey and enter to win a $100 Amazon Gift Card understand the activity, cost, performance and relationships that occur in your cloud and serverless environments