Scaling Development Environments with Docker

Editor's Notes

• #2: * The title of this talk is “Scaling Development Environments with Docker”, but that does not really cover everything * * Producing scalable, state heavy, workspaces for users * Maintaining multiple projects with many dependencies * Deploying large scale containers rapidly * Anecdote about Maxes project * * Shared code between containers * * Many containers in a single project * * Easy and fast deployment of changes
• #3: * Weeby co is a games company in Mountain View, California * How do you bring new developers onto your project, how can you make a new developer a useful developer * How do you manage dependencies for containers, two main types of containers
• #4: * Our goal is to create the best tools that exist for developing social mobile games * Take a 2 man development team and put them in a place where they can compete with Candy Crush or Dragon Flight * Our javascript games run just as well as a pure native game, and we can – in a few clicks – integrate rich social features into that game * * Send lives, daily bonuses, leaderboards, in game currency * Rapid feedback of in browser development – hit refresh and see your changes immediately * Develop once, deploy anywhere * Make it social in a mater of minutes – all it takes is a few clicks
• #5: * Unfortunately that is not easy to do, if you are a developer or a product manager * Our tools require a lot of dependencies, and require the setup to be done exactly correct NEXT SLIDE * Working with node is not always simple * * Sometimes broken state, difficult on your computer, difficult when its not your computer, even more difficult when you don’t have direct access to the computer * Most of our tools assume you are working on a Mac OS, and if not at least a UNIX OS
• #6: * Unfortunately that is not easy to do, if you are a developer or a product manager * Our tools require a lot of dependencies, and require the setup to be done exactly correct THIS SLIDE * For example: working with node is not always simple * * Sometimes broken state, difficult on your computer, difficult when its not your computer, even more difficult when you don’t have direct access to the computer * Most of our tools assume you are working on a Mac OS, and if not at least a UNIX OS
• #7: * Obviously the most universally accessible solution is virtualization * Move all of our tools off of your computer and on to a server that your computer has access to
• #8: * A lot of different ways to virtualize software, unfortunately most are easy for the user but complicated for the developers * Cannot have a server per user * * Not cost effective * * Doesn’t scale well * * Ton of overhead, as well as long startup and shutdown times
• #9: * With Docker we can condense this entire server in to a container * * Easily distributed and scaled * * Cost effective * What is actually running inside this container * * Agent * * Weeby * * Devkit * Now apparent why a games company needs docker. Providing all of our tools in the most simple to consume way possible, over the web
• #10: * There are a lot of servers running inside containers on our machine, and we have a limited number of ports that we can expose * * Issue of linking to a non standard port in a web browser * * Issue of accessing non standard ports from behind firewalls
• #11: * Easily solved by using NGINX as a reverse proxy * Solves the problem of ports * The system is still not easy for us to use – must spin up containers individually, edit NGINX configurations, and then reload the routing * Manually managed systems are not fun
• #12: * Make a server to manage docker and nginx * * Using python Flask server with beaker for sessions and uWSGI for threading * * Responsible for not only creating new containers, reloading routing configuration, but also monitoring container health, and spinning down inactive containers to save space on the system
• #13: * Easily add an authentication method through NGINX to make all access to our user containers secure * * Sweet docker goodness * * No container is exposed to the outside world, so the only way to access them is through NGINX
• #14: * …Getting toward the end of things now * Need a way to easily persist user projects, simple with docker volumes * Still have this problem of manually managing this “server”, there are a lot of things going on and what is worse than having to keep track of all that
• #15: * Docker is awesome, again * Allows us to completely wrap the manager in a way that it can now be deployed to any system running docker * Problem: What happens when we need to update the manager? * * Shut down manager, no NGINX, no route from user computer to user container
• #16: * Again we were shown how using docker to compartmentalize makes development much simpler * Move NGINX into a separate container, so that we can start a new manager whenever we want * * The procedure is fairly straight forward * * New manager * * Starts up all its internal servers * * Adopts existing user containers * * Tell NGINX container that it is the new master, and routes should point to itself * * Tell the old manager it can now safely shut down
• #17: * It turns out developing this whole system is complicated * There are a ton of things going on inside our manager and agent, and we haven’t even looked at the rest of the stack * What do each of these production servers do? * * Registry is backed by Amazon’s S3, each manager has access to a registry * * Build server because uploading large containers is not possible on all networks * * Google App Engine servers to run our authentication and game logic (leaderboards, push notifications, etc.) * * NGINX container to route containers * * Manager * * Agent
• #18: * Developing servers is always a difficult task because you have to do your best to make sure your dev environment matches your deployment environment * Should something change, an interaction, a dependency, it could cause the entire system to fail * Difficult to keep track of interactions in your head, and verify based on your personal knowledge that a change will not effect some other part of the system * People are bad at programming, failure is always a possibility, perfect test coverage is difficult
• #19: * We rely on Google App Engine for our game server, as well as our producer authentication server * These are separate servers because we want devkit to remain open and usable to anyone with no affiliation to the producer * One generic App Engine container running the local python development tools * Mount the specific server files in to the container at run time
• #20: * Right now our system is relatively simple, we are just looking at local running versions of our app engine servers * Being able to completely isolate them is great because we are able to see a much more accurate depiction of how the servers will run in deployment
• #21: * Next we add in the NGINX routing container and the manager * We have support in these servers to configure important URLs, so we can easily just choose a URL and have it route based on our /etc/hosts file * * Instead of “producer.weeby.co”, it is “producer-dev.weeby.co”
• #22: * How to deal with a bunch of web servers all running on port 80 inside of boot2docker * * Fully configurable URLs, so we can just tell docker that newServer should be running on 8080, nimbus on 8081, etc * Explain the interactions between containers
• #23: * Next we start up the manager container * The agent container is started as part of the log in procedure, which means it is not started here, but it will be in the diagrams for clarity * Really the meat and beans, these are the two servers that see near constant development and improvement
• #24: * This is the complete development environment, without any of the interactions between the containers * Nothing too complicated, each container has a very specific task. Each server has a very concise repository of code
• #25: * Now we are able to see the interactions of all our containers and we can look at our entire system as a whole, from the comfort of our local development machine * What does the flow look like for a user coming into the system for the first time? * How the manager is able to talk back to the reload server * How do we actually edit the servers running inside of these containers?
• #26: * This shows all of the file mounts into the containers that originate in our host development machine * * These mounts allow us to interact with all of the files as though they are local files, using vim or sublime, git, etc * * All the files are in one place, in one repository so that we can easily track the entire deployment * Changes made locally are available inside the containers, so we can develop each server as though it were actually running on our local machine, except each server is actually running in its own isolated environment
• #27: * There is a common way of setting up a docker build * * Use a Dockerfile and a launch script * * We do this, but we also add build-container and run-container scripts so that any special arguments or mounts are easily reproduced * The templating is part of a tool that we started developing for this project * * Dockerfile_base template and Dockerfile_patch template * * We are building 4-5 GB images, and cannot afford to build the entire image on a regular basis * * * Leverage docker’s union filesystem * * Patch builds only run a checkout instead of the entire build proces
• #28: * Our build tools know when to run the base template, based on git diffs on the base template * If there are no diffs, then it runs the patch template to only insert new changes made to the other dependencies
• #29: * Configuring the templates are easy – just define where (relatively) the dependencies are, build tools automatically determine the commit, and automatically inject that in to the dockerfile template * Build tools also automatically determine if it is a minor build or a patch build, by checking for changes made to the dockerfile_base template. If it is a patch, the most recent image is used, and then a checkout is run on the repositories, making patch builds almost instant * Why have a template? Why have two templates?
• #30: * This is just the agent container, it has been reduced in size by about 1 GB since the beginning of the project * Running all of our software takes a lot of space and takes a long time to clone dependencies * Cannot rely on a strong internet connection to allow us to develop (strong meaning ability to upload 4+ GB)
• #31: * Our build server is ridiculously simple, since all of our build tools are packaged in to the main repository * * Build server listens to github webhook requests, looks for tags matching a certain pattern, ‘agent-0.6.0’, and then runs the build tools * * Checkout the new tag, run the build scripts, tag and upload the new image, remove any old images to keep build server storage free * This build server is super resilient ... not only are you checking out the content to build, but you are also checking out the scripts (and the procedure) for building the content * * You know that the build server will work because you can test all of the scripts it will use locally, with the same mentality as running all of the containers locally * * Break it without any repercussions
• #32: * This is spread out across many servers, and is pretty complicated to conceptualize
• #33: * This is Martin and myself, we are the two people developing these tools and building this architecture
• #34: * No way that 2 developers could work on this project without being able to work on the entire thing locally, without being able to examine interactions first hand, and have the ability to break things without causing disruptions in service * Started off taking 30 minutes to build and 30 minutes to upload * * Now takes 30 minutes total for a rebuild and 30 seconds for a patch * * Build server uses our main code repository, and the same build scripts that we use locally * * git hook looking for a new agent or a new manager, automatically gets the next version from our registry, increments, and builds * Full circle of development to build to upload to accessible by a user
• #35: * Where are we going next with this project and these tools * Work with shippable to integrate their build server support into our tools
• #36: * Thank you