Amazon ECS at Coursera: A unified execution framework while defending against untrusted code
- 1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Frank Chen, Coursera Brennan Saeta, Coursera October 2015 CMP406 Amazon ECS at Coursera Powering a general-purpose near-line execution microservice, while defending against untrusted code
- 2. What to Expect from the Session • Techniques for a unified near-line, batch, and scheduled micro-service powered by Amazon ECS • Security vulnerabilities and countermeasures when running untrusted code in Docker with Amazon ECS • Reasons to modify the Amazon ECS agent
- 3. Session Outline • Introduction to Coursera • Near-line, batch and scheduled job execution framework • Motivations and background • Amazon ECS benefits and limitations • Iguazú and its architecture • Evaluating programming assignments • System requirements • Security threat model • Attacks and defenses
- 7. Education at Scale 15 million learners worldwide 2.5 million course completions 1,300+ courses 125+ partners
- 8. A unified execution framework
- 9. Batch Processing Enables… Reporting Instructor Reports • Grade exports • Learner demographics • Course progress statistics Internal Reports • Business metrics • Payments reconciliation
- 10. Scheduled Processing Enables… Marketing • Recommendation emails • Targeted marketing / reactivation emails
- 11. Nearline Processing Enables… Pedagogical Innovations • Peer-review matching & analysis • Auto-graded programming assignments
- 12. The early days… January 2012
- 13. Bad Old Days of Batch Processing @ Coursera Cascade • PHP-based job runner • Originally ran in screen sessions • Polled APIs for new jobs • Forced restarts on regular basis due to unidentified memory leaks • Fragile and unreliable The early days…
- 14. Bad Old Days of Batch Processing @ Coursera Saturn • Scala scheduled batch job runner • Powered by Quartz Scheduler library • Better than Cascade, but… • All jobs ran on same JVM, causing interference The not- so early days?
- 15. Looking for something better…
- 16. What We Wanted Reliable Easy Development Easy Deployment High Efficiency Low Ops Load Cost Effective
- 17. What We Wanted Reliable Easy Development Easy Deployment High Efficiency Low Ops Load Cost Effective
- 18. What We Wanted Reliable Easy Development Easy Deployment High Efficiency Low Ops Load Cost Effective
- 19. What We Wanted Reliable Easy Development Easy Deployment High Efficiency Low Ops Load Cost Effective
- 20. What We Wanted Reliable Easy Development Easy Deployment High Efficiency Low Ops Load Cost Effective
- 21. What We Wanted Reliable Easy Development Easy Deployment High Efficiency Low Ops Load Cost Effective
- 22. What Else Did We Look At? Home-grown Tech • Tried, but proved to be unreliable • Difficult to handle coordination and synchronization • Powerful, but hard to productionize • Needs developers with experience • Designed for GCE first • Not a managed service, higher Ops load
- 23. Amazon ECS to the Rescue Amazon re:Invent 2014 – Dr. Werner Vogels introducing Amazon ECS Screenshot from https://www.youtube.com/watch?v=LE5uBqNp2Ds by Amazon Web Services
- 24. Amazon ECS to the Rescue Little maintenance Integrated with rest of AWS Easy to develop for
- 25. Amazon ECS to the Rescue Little maintenance Integrated with rest of AWS Easy to develop for
- 26. Amazon ECS to the Rescue Little maintenance Integrated with rest of AWS Easy to develop for
- 27. However… Amazon ECS is a great building block, but we still need to build tools around it for our purposes.
- 28. What We Built: Iguazú Marissa Strniste (https://www.flickr.com/photos/mstrniste/5999464924) CC-BY-2.0 • Batch Job Scheduler for Amazon ECS • Immediately • Deferred (run once at X time) • Scheduled recurring (cron-like) • Programmatically accessible internally via our standard APIs and clients • Named for Iguazú falls • World’s largest waterfall by volume • We hope Iguazú handles a similar volume of jobs
- 29. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 30. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 31. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 32. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 33. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 34. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 35. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 36. Iguazú Frontend Iguazú Scheduler Iguazú Backend Iguazú: Architecture CassandraServices Services Iguazú Admin ECS Workers SQS ECS API Devs Users
- 37. Developing Iguazú Jobs class Job extends AbstractJob with StrictLogging { override val reservedCpu = 1024 // 1 CPU core override val reservedMemory = 1024 // 1 GB RAM def run(parameters: JsValue) = { logger.info("I am running my job! ") expensiveComputationHere() } }
- 38. Running Jobs from Other Services // invoking a job with one function call // from another service via Naptime RPC/REST framework val invocationId = IguazuJobInvocationClient .create(IguazuJobInvocationRequest( jobName = "exportQuizGrades", parameters = quizParams))
- 39. Iguazú: Developer / Ops User Interface
- 40. Deploying Jobs Easy Deployment 1. Developers Merge into master. Done! Jenkins Build Steps: 1. Builds zip package from master 2. Prepares Docker image with zip file 3. Pushes image into Docker registry 4. Registers updated jobs with Amazon ECS API
- 41. Logs • Logs are in /var/lib/docker/containers/* • Upload into log analysis service (Sumologic) • Wrapper prints out job name and job ID at the start for easy searching • Good enough for now
- 42. Metrics • Using third-party metrics collector (Datadog) • Metrics for both jobs and container instances • So long as the worker machines can talk to Internet, things will work out pretty well
- 43. Since April 2015… 65 jobs in production >1000 runs per day 44 different scheduled jobs
- 47. Programming Assignments at Coursera
- 48. The Security Challenge Compiling and running untrusted, arbitrary code in Amazon EC2 Would you like to compile and run C code from random people on the Internet on your servers?
- 49. 1st Generation System Class graders in separate AWS acct Custom grader systems on cloud providers Course grader under the instructor’s desk Learners Coursera Servers Queue Service
- 50. 1st Generation System: Weaknesses No Auto Scaling No standard security Graders crashed
- 51. 1st Generation System: Weaknesses No Auto Scaling No standard security Graders crashed
- 52. 1st Generation System: Weaknesses No Auto Scaling No standard security Graders crashed
- 53. Design Goals Cost Savings No Maintenance Near Real-time Secure Infrastructure
- 54. Design Goals Cost Savings No Maintenance Near Real-time Secure Infrastructure
- 55. Design Goals Cost Savings No Maintenance Near Real-time Secure Infrastructure
- 56. Design Goals Cost Savings No Maintenance Near Real-time Secure Infrastructure
- 57. Threat Model Prevent submitted code from: • impacting the evaluation of other submissions. • disrupting the grading environment (e.g., DoS) • affecting the rest of the Coursera learning platform Additional goals: • Minimize exfiltration of information • Test cases, solutions, etc… • Minimize risk of submissions changing own scores • Avoid turning into bitcoin miners or part of botnet
- 58. Threat Model - Assumptions • Run arbitrary binaries • Instructor grading scripts may have vulnerabilities • ∴ Grading code is untrusted • Unknown vulnerabilities in Docker and Linux name- spacing and/or container implementation
- 59. Attack / Vulnerability Classes Divided into 2 main categories: • Assuming basic containers are secure, prevent any negative impacts to running arbitrary code. • Assuming basic container technology is vulnerable, mitigate negative impacts as much as possible.
- 60. What We Built: GrID Patrick Hoesly (https://www.flickr.com/photos/zooboing/5665221326/) CC-BY-2.0 • Service + architecture for grading programming assignments • Builds on Amazon ECS and Iguazú • Named for Tron’s “digital frontier” • Backronym: Grading Inside Docker
- 61. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS APIs Grading MachinesVPC Firewalls Coursera Production Account Coursera GrID Grading Account
- 62. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS APIs Grading MachinesVPC Firewalls Coursera Production Account Coursera GrID Grading Account
- 63. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS API Grading MachinesVPC Firewalls Production Acct GrID Grading Account
- 64. High-level GrID Architecture Learners GrID Iguazú S3 Bucket ECS API Grading Machines VPC Firewalls Production Acct GrID Grading Account
- 65. Attacks: Resource Exhaustion Defenses: • Docker / CGroups: • CPU quotas • Memory limits • Swap limits • Hard timeouts for container execution • btrfs limits • file system storage quotas • IOPS throttling
- 66. Attacks: Kernel Resource Exhaustion Defenses: • Open file limits per container (nofile) • nproc Process limits • Limit kernel memory per cgroup • Limit execution time
- 67. Attacks: Network attacks Attacks: • Bitcoin mining • DoS attacks on third-party systems • Access Amazon S3 and other AWS APIs Defense: • Deny network access
- 68. Modifying the ECS Agent: Network Modes • NetworkDisabled too restrictive • Some graders require local loopback • Feature also deprecated • --net=none + deny net_admin + audit network • Isolation via Docker creating an independent network stack for each container • github.com/coursera/amazon-ecs-agent
- 69. Attacks: Namespace / Container Vulnerabilities • App Armor & Mandatory Access Control • Required modifying the Amazon ECS Agent • Allows auditing or denying access to a variety of subsystems • Drop capabilities • No need for NET_BIND_SERVICE, CAP_FOWNER • No root within container
- 70. Attacks: Root escalations within the container • We modify instructor grader images before allowing them to be run • Clears setuid • Inserts C wrapper to drop privileges from root and redirect stdin/stdout/stderr • Required Amazon ECS Agent modification • Grant root privileges • Map Docker socket into Docker containers to run Docker in Docker!
- 71. Attacks: If all else fails… • Utilizes VPC security measures to further restrict network access • No public internet access • Security group to restrict inbound/outbound access • Network flow logs for auditing • Separate AWS account • Run in an Auto Scaling group • Regularly terminate all grading EC2 instances
- 72. Other Security Measures • Utilize AWS CloudTrail for audit logs • Third-party security monitoring (Threat Stack) • No one should log in, so any TTY is an alert • Penetration testing by third-party red team (Synack)
- 73. Technique: Co-process • Environment has no network, but has to get submissions in and results out • Python co-process watches Amazon ECS / Docker • Python co-process then: • Mounts a shared folder containing submission • Reads back the grade from the shared folder after container exits • Monitors and cleans up
- 74. Future Improvements • Priority queues for different grading priorities • Re-grades vs on-demand grades • Better instructor tooling • Automated “unit-testing” for new graders • Better simulation of production environment on instructor machines • Support scheduling GPUs
- 75. Lessons Learned • Run the latest kernels • Latest security patches • btrfs wedging on older kernels • Default Ubuntu 14.04 kernel not new enough! • Carefully monitor disk usage • Docker-in-docker can’t clean up after itself (yet). • Reliable deploy tooling pays for itself
- 76. Related Sessions Also from Coursera: • BDT404 - Building and Managing Large-Scale ETL Data Flows with AWS Data Pipeline and Dataduct - Friday Containers and Amazon ECS: • CMP302 - Amazon EC2 Container Service: Distributed Applications at Scale – Next timeslot in Venetian H
- 77. Thank you! Questions? Also, we are hiring! www.coursera.org/jobs tech.coursera.org Brennan Saeta github/saeta @bsaeta [email protected] Frank Chen github/frankchn @frankchn [email protected]
- 78. Remember to complete your evaluations!
Editor's Notes
- #3: This is a 400-level session…
- #4: This talk is organized as follow…
- #5: Introducing Coursera. - Mission - Founding. Transition: we partner…
- #6: “World class institutions from around the globe. We have partners on 6 continents (most of them)! You might recognize a name or two up here. These institutions take their best programs and their best instructors, and their best courses, and put them on the Coursera learning platform.”
- #7: Courses span a range from highly technical… to the humanities, including… Listing these courses: - Machine Learning - Data Science - Learning How To Learn - Social Psychology - Irrational Behavior Raise of hands: who has taken a Coursera course?
- #8: … Our scale brings a number of challenges, and opportunities. For example, the export gradebook function is relatively trivial for a class of 20 or even 200 students. Our largest classes have over 200,000 individuals. In order to power our global learning platform, we’ve had to develop techniques and build systems that meet our needs. And now, I’d like to invite Frank Chen, a founding engineer at Coursera, to the stage to discuss our near line job scheduler and execution framework.”
- #9: Three things: Batch Processing, Scheduled Processing, Nearline Processing
- #10: Instructor reports - As you know, our instructors put classes on our platform for free – and in return, they want data about their learners! We also want to give them this data because they can use this for further studies and to improve their courses. 20 learners easy – 200k learners hard. Of course, like any other company, we have a lot of Internal reports as well – finance, bizdev, marketing wants to know updated data, make sure data is accurate. Payment reconciliation makes sure we are gettingthe right product to the right people etc…
- #11: Our main use case of scheduled processing would be for marketing and recommendation emails. How many of you have received a recommendation email from Coursera? That is from a batch job that is run every week. We also do smaller scale things – reactivation and targeted marketing emails.
- #12: What is peer review? In a lot of our classes (e.g. Fiction Writing, Modern Poetry), instructors cannot accurately assess student performance just by using MCQ questions – you need students to write short answer questions and essays or submit drawings or recordings. You need human beings to evaluate these. In an ideal world, you can hire 1,000 TAs to grade these submissions, but we can’t afford that. You can do the next best thing – getting other students to evaluate you. As part of this system, we need to assign reviewers to your assignments in a fair and efficient manner according to complex criteria and near-line processing enables us to do this.
- #13: Before I jump into what we did with ECS, I want to talk about the early days of batch processing at Coursera.
- #16: Looking for something BETTER in our next system.
- #17: Saturn / Cascade were flaky Developers became frustrated with jobs not running properly
- #18: Developing & testing locally was difficult in old system – develop bad habit of pushing code to prod without testing it Required no boilerplate code to be written and little environment setup should be required
- #19: Deployment was difficult and often interfered with running jobs. “Other services have one-click tools, why can’t your service have that too?”
- #20: Low startup and shutdown time – responsive enough to start within 30 seconds of the job being requested if there are enough resources.
- #21: Only one dev-ops engineer -- can’t manage everything Developers own their services Developers shouldn’t have to actively monitor services
- #22: We are a startup and we are cost-conscious Most jobs complete < 20 minutes but EC2 rounds costs up to full hour
- #25: Amazon does the hard work of coordination and synchronization in a distributed system, and even provides an agent to run jobs itself. Our DevOps engineer is very happy – one AMI with one agent + Docker on it, simple and easy to use.
- #26: Tightly integrated with rest of AWS APIs. For instance, we can use Amazon IAM roles and users to restrict access to the ECS API, and this makes Brennan, who is also our security engineer, very happy because we get security without him doing a lot of extra work.
- #27: Amazon traditionally has a very good set of APIs, documentation and SDKs and ECS is no different. We found it very easy to grasp the key concepts and get started with using Amazon ECS.
- #28: No scheduled tasks No fine-grained monitoring of tasks No retries or delays when cluster runs out of resources / prioritization Does not integrate well with our existing infrastructure (e.g. Scala APIs and tooling)
- #30: User submits a request to a frontend service – e.g. to our quiz service for an export of all the grades of the students in the class for a specific quiz.
- #31: The online service sends a batch job request to the Iguazu frontend.
- #32: The iguazu frontend persists pertinent job information to Cassandra, our database.
- #33: The frontend then submits the job request to a SQS queue.
- #34: The Iguazu backend reads pending jobs off the SQS queue and processes them.
- #35: In this case, it will talk to the ECS APIs to get a list of all container instances and select a container instance to run the job. A special note here: In our original design, we handed off the scheduling portion of this to the ECS system itself by calling RunTask – which would randomly choose an instance with enough CPU and RAM resources to run my job on. However, we found that this is not flexible enough for our purposes. Specifically, as we were integrating the Iguazu backend with the EC2 Autoscaling Lifecycle APIs in order to autoscale our ECS container instances, we found that RunTask would sometimes schedule jobs on instances that were in the process of termination. By switching to the StartTask API and writing our own scheduling system for the backend, we eliminated that problem by simply not scheduling jobs on instances undergoing termination. In addition, Iguazu can also receive notifications from the EC2 AS Lifecycle APIs that will automatically notify it when an instance is getting terminated and Iguazu will even block the termination of instances until all jobs that were already running on that instance had completed. This greatly improved reliability especially for our long running jobs. If jobs cannot be run, then they will go back into the SQS queue for retry after 10 minutes. If the job still fails to be scheduled after an hour, then the job will be deleted and a exception logged.
- #36: If we have successfully identified an instance with enough resources, then we will call the StartTask API and ask ECS to run the task on that specific container instance. The Iguazu backend will periodically monitor the job and update the status of the job in Cassandra. Each online service can then query the Iguazu frontend for job updates.
- #37: Similarly, developers can use an admin interface to schedule recurring jobs. This goes into an alternate Iguazu frontend (aka Scheduler) that wakes up every second and sends all the tasks that it has to run to the backend via the same SQS queue.
- #38: Our backend is written entirely in Scala, so naturally Iguazu jobs are written in Scala too. Almost no boilerplate and easy to get started.
- #39: Running jobs from other services (e.g. the quiz service) is even easier. We have an internal REST RPC framework called Naptime that abstracts the details of inter-service API calls away from each individual developer. Developers can just take the Iguazu Job Invocation Client that Naptime provides and call the create method with the parameters of the job he or she wants to run. In this case, the developer is running the exportQuizGrades job.
- #40: Our interface is very simple. Engineers can just click the red + button on the lower right to add a new schedule job and they can also click to edit existing ones. Of course, all changes are logged for auditing purposes.
- #44: Developers are definitely happier with this system than the previous ones but we are always continually improving the system.
- #45: Now, I’d like to talk about a special application of Iguazu, Docker, and Amazon ECS: evaluating programming assignments. First Brennan Slide (Round 2)
- #49: … Which of you would like to run untrusted C code from random people on the internet on your servers? Raise your hands! …. Ah you must be from Amazon Web Services. But see for us, we don’t even require a credit card!
- #50: … And some professors set up their graders on machines under their desks. Naturally, the power cord always became unplugged hours before the submission deadline…
- #51: Procrastinators! And then we’d forget about the instance for a couple months after the course finished.
- #52: Securing un-trusted code is hard!
- #54: Now, Coursera recently underwent a complete revamp of the entire course platform, as we shifted from running sessions 2-3 times a year, to every 2-6 weeks. This gave us an opportunity to take another run at programming assignment infrastructure. Coursera is a startup, and we provide generous financial aid, so we are very cost conscious. When we analyzed our cloud spend, we found that we spent a disproportionate amount of money on the graders for programming assignments. Since it is a given that we need to support more courses, with more sophisticated assignments, we needed a system that would provide an order of magnitude or more in cost savings. Cost savings implied autoscaling and a shared pool of resources.
- #55: Have only one devops engineer. No maintenance implied immutable infrastructure, and highly automated environment.
- #56: For pedagogical reasons, we would like to provide feedback as quickly as possible. Ideally, we are able to execute fast graders and turn around their scores within 30 seconds at the 90th percentile. Near realtime means cannot boot a new EC2 instance. In combination with others, implied containers and Docker.
- #57: We wanted to bake security into the infrastructure to automatically relieve instructors from worrying about the vast majority of vulnerabilities. This also has the added benefit of making the system more robust to more innocent occurrences. ... But, what does “Secure Infrastructure” even mean?
- #65: … And with that, I’d like to delve into some attacks, and defenses.
- #74: Now, some of you may have noticed, we have a little problem…
- #75: First talk about success of the system: - Thousands of assignments evaluated daily. - Over a hundred assignments on the platform - Used by dozens of courses Despite the success, there are a number of future improvements to be made… - Provide a number of sample submissions, and their expected scores… - Now, a number of our security measures cannot be replicated out of AWS, but some like App Armor could be. As we have time, we may look into open sourcing our app armor profile, and our list of kernel capabilities. - Finally, we would like to support mapping GPUs into our containers for that CUDA class… While this would require another modification to the Amazon ECS agent, we think that it won’t be too bad.
- #76: … Finally, building a platform for code execution is much harder than building an API in front of a database. Thank you to all of the engineers at AWS who are building the secure, and reliable systems we all here have come to rely on.
- #78: If any of what we’ve talked about today sounds interesting to you, please know that Coursera is always looking for talented engineers, managers, and designers to join the team. If you are interested, please don’t hesitate to reach out. Thank you all very much for attending. <Pause> Questions?