Using Ansible at Scale to Manage a Public Cloud

Editor's Notes

• #2: I'm Jesse Keating I work at Rackspace I'm going to talk about what Rackspace does with Ansible
• #3: At Rackspace we care about scale. Scale of number of server systems Scale of product environments Scale of engineers doing awesome things at Rackspace. Going to cover three scale challenges with three case studies that will highlight key Ansible features that have made it my go to tool in the box.
• #5: First is the scale of servers. I work in the Rackspace Public Cloud product group. We have... It is a lot to handle. Have existing inventory files for use with pssh/etc. Admins worry about what's there, engineers work on growing capacity and automation, developers work on new code and new tools to deploy code. We all work together, DevOps.
• #6: A real world example from a couple days ago Needed to copy one file out to nova-compute Vms and restart nova-compute service Want to avoid flooding the admins with alerts Want easy to read output to know what happened. Before would have been manual actions on nagios hosts, bash script around pssh, lots of output noise, repeat delays on inactive hosts
• #7: Key things Ansible brings to the party
• #8: Example of existing inventory contents. Regions with cells with groups
• #9: More
• #10: Json output that ansible can use. Groups of groups, group_vars, addresses.
• #11: Fairly simple python script to hand to ansible (but it can be anything, so long as it hands back json)
• #12: Silly example of a one-off task
• #13: Actual playbook used to hot-patch production
• #14: This is how we're using Ansible RIGHT NOW with our production environment Building up a toolbox as we go
• #16: Next I want to talk about the scale of our environments. Again I'll be focusing on our public cloud, which is powered by OpenStack. Stop me when you spot the problem. Servers, block storage, object storage, networks, auth, usage, etc... CI is really just for automated tests to gauge health Way too many moving parts for one pre-production environment, puts risk on deploying code in timely manner. Not easy to deploy from personal branch/fork
• #17: What we want to do is build out preproduction environments for each group or individual developer. Big task Before could be days or weeks before an environment could be created, then could sit unused for long periods of time. Devs couldn't do it, Engineers had to find time to fit it in.
• #18: Why we went with Ansible to back this service
• #19: Apologize for puppet/mco stuff here, but that is what is pre-existing Localhost actions to prepare files for new hosts
• #20: Use the host loop to parallelize host boot up in one of our internal Nova environments Eventually this will use the rax module, which could do the DNS step for us
• #21: Now do some actions on the remote hosts. Not showing everything Still in development
• #22: Inventory files look a little different here, more details per host. Making use of some yaml syntax to have defaults that can be overloaded.
• #23: Plugin to read the files, and use --host
• #24: What could take days/weeks to get done can now take minutes. Automating the part that isn't already automated, filling the gap. Will hook it into a web service where developers can make a reservation and provide input as to what they want deployed. Significant overlap with process to roll out new production environments, obvious next step
• #26: Finally lets talk about the scale of our Engineering organization(s) No hard rules about what tech must be used. Best practices bubble up A real challenge to bring on new employees, worse to bring on intern and make most use of their time
• #27: Once more talking about our cloud group, ozone. Not the full story, but some idea of what has to happen. Took me weeks to get fully set up, and I think I'm still missing some stuff, exacerbated by being remote and off-hours from main group some times.
• #28: How can Ansible help here?
• #29: Ansibox is a project I'm working on personally to help with onboarding. Taking inspiration from Github's Boxen project. Roles are where the magic happens.
• #30: Engineers should have to give limited input to Ansibox in order for Ansibox to be able to perform the setup. These could be prompted for in the future. Engineer names a role and provides a location to find that role.
• #31: The top level playbook fetches all the roles, can update them optionally. Generates another playbook to actually go through and apply the roles to the host. Generated playbook comes from a template and is very simple.
• #32: Here is a look at after it gets generated. Doing sudo no at this level, each task in each role can decide to do sudo if author wants it.
• #33: A very simple start to a ansibox executable. Two playbooks are necessary due to Ansible design Prompt is there for second play in case any role wants sudo
• #34: This is the start of a task list for the ozone role. Repos get cloned, tools get installed, configuration files get put into place. Here we could also check for permissions to services and prompt the engineer on what to do to gain access
• #35: With this system it becomes easy for an engineer to boot strap a system, and easy for a group to own that process for the group. Engineers can also add their own roles for personal setups, and be unafraid to refresh devices. Engineers can also contribute to the system as gaps are found