Scaling Engineering with Docker
4 likes1,590 views
GameChanger adopted Docker to enhance scalability and eliminate deploy-time risks associated with their growing application and infrastructure needs. The transition to Docker allowed for immutable images, reduced dependency concerns, and streamlined the deployment pipeline, fostering collaboration between development and operations teams. This resulted in improved efficiency and responsiveness to user demands in managing amateur sports data and services.
Scaling Engineering with Docker
- 1. SCALING ENGINEERING WITH DOCKER A Case Study https://flic.kr/p/ba2mjn
- 3. WHAT WE’LL COVER 1. What motivated GameChanger to adopt Docker? 2. Walkthrough of GameChanger Deploy Pipeline
- 4. WHAT MOTIVATED GAMECHANGERTO ADOPT DOCKER?
- 5. • Scorekeeping • 150+ Stats • Live Gamestream • Team management • 12TB (10 histories of pro sports) • 10 MongoDB shards • 100-400 app servers • 50K games/day (10K concurrent) • 3000 w/s, 30,000 r/s GameChanger’s market is amateur sports. Whereas ESPN caters to handful of top professional teams in the country, GameChanger provides free tools to the millions of amateur sports teams around the world.
- 6. ELIMINATING DEPLOY-TIME RISK This graph shows the number of requests/second received by one of our services over the last week. The area under the graph is broken down by host. You can see that we are scaling our hosts up and down in response to demand. At GC we have an extremely spike traffic profile so using autoscaling is important to control costs. Therefore it’s very important not only to deploy new application code to existing servers but also to be able to very reliably build new servers with minimal risk.
- 7. Chef Server App Server App Server App Server App Server App Server App Server 30 30 30 30 30 30 To illustrate the risks associated with a traditional Configuration Management approach to building servers, let’s look at the typical Chef architecture. This is a CM server which hosts the current valid configuration data for the cluster (and by configuration data we also mean setup scripts etc) The Developer is responsible for pushing new configuration to the CM server and then all app servers periodically pull and execute the latest scripts. Risks: - CM server is a SPOF. Chef is painful to scale out. - CM server needs to be scaled to support max conceivable cluster size (or we have problems when we need it most) - Thundering herd
- 8. github.com/miketheman/knife-role-spaghetti This is a visualization of GC’s role/recipe dependencies in Chef before we moved to Docker. Risks: - Spaghetti-like dependencies are impossible to reason about (what happens if I upgrade node.js?) - Dependencies are indirect and not explicit - Testing is expensive and time consuming. Devs are disincentivized from testing. - Coupling issues not discovered until deploy time -> can take down your cluster. - Rollback can be painful
- 9. App Server PyPI npm Ubuntu apt-get rubygems .tar.gz files apache binaries S3 GitHub Deploy-time dependencies on multiple external repositories is a big risk. - Build AMIs (complex, heavy, time consuming, does not allow us to iterate fast enough) - Host you own mirror for services like PyPI. But then who owns the maintenance of that mirror?
- 10. HOW DOES DOCKER ELIMINATETHESE RISKS? • Assets are baked into an immutable image at build time • No deploy-time dependencies on 3rd party repos • Docker registry is simple and easy to scale • Dependencies simple, explicit and direct • Rollback is trivial
- 11. SCALING ENGINEERING A less obvious problem with traditional CM approaches is how they inhibit the scaling of engineering. Let’s illustrate with an example…
- 12. Application FeatureTeam Like many companies, GC’s product started out as a small Python app developed by a couple of people. At this point we only have a few users so we can run on a couple of servers and deployment is simple manual step.
- 13. Application FeatureTeam FeatureTeam As we build more features our application gets bigger and we hire more people to help build and maintain those features. We’re still doing some form of manual deployment at this point, and though it’s starting to become a bottleneck we’re still prioritizing feature development.
- 14. Monolithic Application FeatureTeam FeatureTeam FeatureTeam We grow further and our application grows accumulating more and more responsibilities. The need to coordinate test + build + deploy necessitates an Ops team to own this problem.
- 15. Monolithic Application FeatureTeam FeatureTeam FeatureTeam Deployment (Test + Build + Deploy) OpsTeam Following a more “Dev Ops” mantra, these responsibilities form more of a continuum. Devs care about getting their code to prod, Ops care about what the code does, both cooperate. Deploying a monolithic app in this way actually works pretty well. The tech stack is fairly static, and forms a shared context which minimizes the friction between dev and ops teams. The problem for GC was that this monolithic architecture scaled poorly for us: - Poor ownership boundaries - Quality of shared components suffered - Introducing new languages is difficult to sell - Different features have different CAP requirements - Operational problems derived from indirect coupling
- 16. μ FeatureTeam FeatureTeam FeatureTeam OpsTeam μ μ μ μ μ μ μ μ μ μ μ Deployment (Test + Build + Deploy) Solution: Teams own collections of independently-scalable microservices with clear ownership boundaries. Teams ar But this poses a problem for our previous deployment approach: - Suddenly Ops need to know how to deploy an ever growing list of technologies - Information friction between Dev and Ops is high as the context is dynamic - Deployment using something like Chef becomes more and more complex - As feature teams are added, Ops becomes a bottleneck, the relationship risks becoming adversarial
- 17. –Melvin Conway, 1968 “Any organization that designs a system … will inevitably produce a design whose structure is a copy of the organization's communication structure.” Conway’s Law A collection of teams that design a system will inevitably produce a design which evolves from the minimum amount of out-of-band communication required between those teams.
- 18. μ Feature + OpsTeam Feature + OpsTeam Feature + OpsTeam μ μ μ μ μ μ μ μ μ μ μ Deployment Deployment Deployment In the face of the need high-traffic high-complexity communication to get software deployed, teams will be motivated towards compartmentalizing the way they approach deployment. This is much better as the contextual footprint for each mini-Ops team is manageable. But there is still a problem here. We risk duplicating effort across teams on “core” deployment activities.
- 19. CORE DEPLOYMENTTASKS • Log rotation • User account creation, sudoers, SSH keys • Continuous Integration • Metrics • DNS • Monitoring & alerting • ulimits • Tool installation • … All of these are important. Doing them well requires that they be owned and continuously improved and maintained as a first class system asset. On feature teams they will not be treated in this way, we’ll duplicate effort building several half-formed implementations of these things.
- 20. μ FeatureTeam FeatureTeam FeatureTeam μ μ μ μ μ μ μ μ μ μ μ Build Build Build OpsTeam “Core” Deployment Pipeline We still needed an Ops Team to own the core parts of deployment, but needed a way to ensure the interface between the feature and Ops teams to have low information friction and not require the Ops team to understand n different tech stacks. We could have tried to use Chef to do this by making each team own its own roles, but you end up running into problems around shared dependencies, global state and indirect coupling. Docker provides a neat abstraction which allows these responsibilities to be separated clearly and scalably.
- 21. HOW DOES DOCKER ALLOW USTO SCALE ENGINEERING? 1. Development team has complete control over what they deploy 2. Core deployment can still be owned by a dedicated team as a first class concern 3. Small shared context needed for cross-team communication 1. allows us to scale teams out linearly without creating a centralized bottleneck 2. eliminates wasted duplicate effort and the effort of maintaining a substandard system 3. eliminates waste effort communicating complex requirements in an out of band way
- 23. Test Build Deploy We’re going to run through the test-build-deploy pipeline at GameChanger. We’re using a separate service for each of those, so let’s introduce the cast of characters.
- 24. Drone TheTest Runner Docker-based testing Fully isolated and concurrent tests Easily tests against service containers, e.g. Postgres
- 25. Jenkins The Build Server In charge of building Docker containers, app packages, etc. Can do almost anything, but isolation not a strong suit
- 26. Bagel The Deploy Service Manages versioning of deployable apps across environments One-click deploy, rollback mmm, bagel
- 28. Test Build Deploy We’re going to go through what it takes to wire up an application to work with our pipeline…
- 29. Test Build Deploy …and while we’re doing it, we’re going to highlight the ways in which Docker helps us achieve the goals that Tom was talking about earlier.
- 30. Python + Postgres:A Simple Application Let’s consider a simple Python application that works with a Postgres database. It has a bunch of unit tests, including unit tests that require connecting to an actual Postgres instance to run.
- 31. Test Removes some dependency setup concerns from application dev Tests are fully isolated, coupling is minimized Fast and parallelizable, multiple teams can work on a single app without slowing each other down Drone TheTest Runner Drone, as we mentioned before, is what we use to run tests. What are the benefits we get from using Drone?
- 32. Drone uses a simpleYAML specification Test
- 33. Test Drone uses a simpleYAML specification
- 34. Application devs do not need to know how to install these Use official images or write a generalized image once, share with all your teams Test
- 35. Server Host OS Each test run is fully isolated using containers Test
- 36. Server Host OS Each test run is fully isolated using containers Parallel testing becomes trivial Fast testing of PRs reduces likelihood of breaking the build Test
- 37. Build Receives Git hash and static dependencies from Drone Builds Docker images, pushes to our private Docker registry Jenkins The Build Server
- 39. Tested application code (specific Git hash) Same library versions used to test that code, e.g. pip freeze, npm shrinkwrap All system libraries, drivers, etc. Build What exactly are we putting into our images? Note that our image will *not* contain service dependencies like the Postgres we want to run against. We have a few options for how to connect to a database at runtime.
- 40. Drone PyPi Jenkins Registry Image Build So this is how we make the build work with Docker. Why is this actually better than the traditional model? Well…
- 41. Drone PyPi Jenkins Registry Image Build X …what if we put a bullet in PyPi and are no longer able to get our library dependencies? Before we answer that, what happened in the old world? We’d deploy new code, our servers would all try to pull from PyPi, fail, and freak out. Is our site down? Are we up but with incorrect or partial dependencies? It’s not great. What happens with Docker?
- 42. Drone PyPi Jenkins Registry Image Build X With Docker, that risk is moved from deploy time to build time. Jenkins will try to pull from PyPi and fail. It won’t be able to push a new image with your updated code. This is usually a good thing! You can have confidence that all the images you *do* have will have all their dependencies and be fully working images.
- 43. Mostly a thin API on top of our Docker registry Also owns triggering deploys across our infrastructure Deploy Bagel The Deploy Service Bagel gives us a way to coordinate the images in our Docker registry with their corresponding Git tags, the dependencies that were baked into the images, etc. I think just go into a quick demo here, show some cool dependency diffs and PR messages or something.
- 45. Deploy deploy.travisthieman.com Deploys triggered via gossip protocol Coordinated with distributed locks What happens when we hit the Deploy button in Bagel?
- 46. All our machines run identical OS-level images Images and runtime config specified viaYAML Deploy mechanism on each machine reconciles spec and containers currently running Similar to Docker Compose All our boxes run off the same machine image (AMI). A YAML file specifying which of our apps should be deployed to that box is all that distinguishes it. Our pretty-dumb deploy scripts (triggered by Bagel) handle matching the running state of that box’s containers to what’s in this YAML file and the current deployed versions according to Bagel.
- 47. Test Build Deploy That’s our deploy pipeline. Using Docker, we’ve seen significant gains in simplicity and developer productivity across our test, build, and deploy stages. Our feature teams can release new services with ease, and our Ops team has been phased out of existence. Our engineers are now free to focus on problems that benefit our customers and our business. Before I go, just a few closing thoughts on Docker as someone who’s spent a bit of time with it…
- 48. Young Ecosystem RollYour Own? Build Process Intricacies Docker on OS X Development Environments
Editor's Notes
- #6: GameChanger’s market is amateur sports. Whereas ESPN caters to handful of top professional teams in the country, GameChanger provides free tools to the millions of amateur sports teams around the world.
- #7: This graph shows the number of requests/second received by one of our services over the last week. The area under the graph is broken down by host. You can see that we are scaling our hosts up and down in response to demand. At GC we have an extremely spike traffic profile so using autoscaling is important to control costs. Therefore it’s very important not only to deploy new application code to existing servers but also to be able to very reliably build new servers with minimal risk.
- #8: To illustrate the risks associated with a traditional Configuration Management approach to building servers, let’s look at the typical Chef architecture. This is a CM server which hosts the current valid configuration data for the cluster (and by configuration data we also mean setup scripts etc) The Developer is responsible for pushing new configuration to the CM server and then all app servers periodically pull and execute the latest scripts. Risks: - CM server is a SPOF. Chef is painful to scale out. - CM server needs to be scaled to support max conceivable cluster size (or we have problems when we need it most) - Thundering herd
- #9: This is a visualization of GC’s role/recipe dependencies in Chef before we moved to Docker. Risks: - Spaghetti-like dependencies are impossible to reason about (what happens if I upgrade node.js?) - Dependencies are indirect and not explicit - Testing is expensive and time consuming. Devs are disincentivized from testing. - Coupling issues not discovered until deploy time -> can take down your cluster. - Rollback can be painful
- #10: Deploy-time dependencies on multiple external repositories is a big risk. - Build AMIs (complex, heavy, time consuming, does not allow us to iterate fast enough) - Host you own mirror for services like PyPI. But then who owns the maintenance of that mirror?
- #12: A less obvious problem with traditional CM approaches is how they inhibit the scaling of engineering. Let’s illustrate with an example…
- #13: Like many companies, GC’s product started out as a small Python app developed by a couple of people. At this point we only have a few users so we can run on a couple of servers and deployment is simple manual step.
- #14: As we build more features our application gets bigger and we hire more people to help build and maintain those features. We’re still doing some form of manual deployment at this point, and though it’s starting to become a bottleneck we’re still prioritizing feature development.
- #15: We grow further and our application grows accumulating more and more responsibilities. The need to coordinate test + build + deploy necessitates an Ops team to own this problem.
- #16: Following a more “Dev Ops” mantra, these responsibilities form more of a continuum. Devs care about getting their code to prod, Ops care about what the code does, both cooperate. Deploying a monolithic app in this way actually works pretty well. The tech stack is fairly static, and forms a shared context which minimizes the friction between dev and ops teams. The problem for GC was that this monolithic architecture scaled poorly for us: - Poor ownership boundaries - Quality of shared components suffered - Introducing new languages is difficult to sell - Different features have different CAP requirements - Operational problems derived from indirect coupling
- #17: Solution: Teams own collections of independently-scalable microservices with clear ownership boundaries. Teams ar But this poses a problem for our previous deployment approach: - Suddenly Ops need to know how to deploy an ever growing list of technologies - Information friction between Dev and Ops is high as the context is dynamic - Deployment using something like Chef becomes more and more complex - As feature teams are added, Ops becomes a bottleneck, the relationship risks becoming adversarial
- #18: Conway’s Law A collection of teams that design a system will inevitably produce a design which evolves from the minimum amount of out-of-band communication required between those teams.
- #19: In the face of the need high-traffic high-complexity communication to get software deployed, teams will be motivated towards compartmentalizing the way they approach deployment. This is much better as the contextual footprint for each mini-Ops team is manageable. But there is still a problem here. We risk duplicating effort across teams on “core” deployment activities.
- #20: All of these are important. Doing them well requires that they be owned and continuously improved and maintained as a first class system asset. On feature teams they will not be treated in this way, we’ll duplicate effort building several half-formed implementations of these things.
- #21: We still needed an Ops Team to own the core parts of deployment, but needed a way to ensure the interface between the feature and Ops teams to have low information friction and not require the Ops team to understand n different tech stacks. We could have tried to use Chef to do this by making each team own its own roles, but you end up running into problems around shared dependencies, global state and indirect coupling. Docker provides a neat abstraction which allows these responsibilities to be separated clearly and scalably.
- #22: allows us to scale teams out linearly without creating a centralized bottleneck eliminates wasted duplicate effort and the effort of maintaining a substandard system eliminates waste effort communicating complex requirements in an out of band way
- #24: We’re going to run through the test-build-deploy pipeline at GameChanger. We’re using a separate service for each of those, so let’s introduce the cast of characters.
- #27: mmm, bagel
- #29: We’re going to go through what it takes to wire up an application to work with our pipeline…
- #30: …and while we’re doing it, we’re going to highlight the ways in which Docker helps us achieve the goals that Tom was talking about earlier.
- #31: Let’s consider a simple Python application that works with a Postgres database. It has a bunch of unit tests, including unit tests that require connecting to an actual Postgres instance to run.
- #32: Drone, as we mentioned before, is what we use to run tests. What are the benefits we get from using Drone?
- #40: What exactly are we putting into our images? Note that our image will *not* contain service dependencies like the Postgres we want to run against. We have a few options for how to connect to a database at runtime.
- #41: So this is how we make the build work with Docker. Why is this actually better than the traditional model? Well…
- #42: …what if we put a bullet in PyPi and are no longer able to get our library dependencies? Before we answer that, what happened in the old world? We’d deploy new code, our servers would all try to pull from PyPi, fail, and freak out. Is our site down? Are we up but with incorrect or partial dependencies? It’s not great. What happens with Docker?
- #43: With Docker, that risk is moved from deploy time to build time. Jenkins will try to pull from PyPi and fail. It won’t be able to push a new image with your updated code. This is usually a good thing! You can have confidence that all the images you *do* have will have all their dependencies and be fully working images.
- #44: Bagel gives us a way to coordinate the images in our Docker registry with their corresponding Git tags, the dependencies that were baked into the images, etc. I think just go into a quick demo here, show some cool dependency diffs and PR messages or something.
- #46: What happens when we hit the Deploy button in Bagel?
- #47: All our boxes run off the same machine image (AMI). A YAML file specifying which of our apps should be deployed to that box is all that distinguishes it. Our pretty-dumb deploy scripts (triggered by Bagel) handle matching the running state of that box’s containers to what’s in this YAML file and the current deployed versions according to Bagel.
- #48: That’s our deploy pipeline. Using Docker, we’ve seen significant gains in simplicity and developer productivity across our test, build, and deploy stages. Our feature teams can release new services with ease, and our Ops team has been phased out of existence. Our engineers are now free to focus on problems that benefit our customers and our business. Before I go, just a few closing thoughts on Docker as someone who’s spent a bit of time with it…