Deploying OpenStack with Ansible

Editor's Notes

• #5: OpenStack is hard. plain and simple. Now I’m here to talk about how Ansible makes Operating and Deploying OpenStack clouds easier it by no means makes it simple. I have no magic pixie dust that makes OpenStack simple. Deployers that claim to have a scalable production ready OpenStack cloud in > 10 minutes are on crack. People writing configuration management software for OpenStack know that OpenStack is hard but we’re all out there trying to make life easier for everyone in the community.
• #7: Talk about why I’m here presenting about Ansible and OpenStack Where did we come from? - Rackspace Private Cloud has been here a while. I’d go as far as saying we were the first. Don’t call it a comeback we’ve been here for years. Packaging OpenStack sucks, say why.
• #8: Talk about why I’m here presenting about Ansible and OpenStack Where did we come from? - Rackspace Private Cloud has been here a while. I’d go as far as saying we were the first. Don’t call it a comeback we’ve been here for years. Packaging OpenStack sucks, say why.
• #9: RCBOPS chef was a good example of the “run thrice” philosophy. - Stackforge chef cookbooks is not much better Upgrading required a lot of retool for ever release, even if it's a point release. If you're using Puppet or Chef you’re learning a “DSL” which is more like a language than a task driven system. Additionally when coming from the greater OpenStack community telling people that they need to learn Ruby or some variant there of is a hard sell.
• #10: The controller model makes it hard or impossible to scale past 2 controllers and in production under heavy workloads we’ve found that operators need the ability to scale beyond the two node limit. If you use the controller model and you have two of them, then you likely have a VIP that fails over between the two nodes this VIP failover is error prone and makes services like plain jane MySQL and RabbitMQ very unhappy. The controller model generally does not account for the issues that can be caused when using mirrored queues.
• #11: Talk about why I’m here presenting about Ansible and OpenStack Where did we come from? - Rackspace Private Cloud has been here a while. I’d go as far as saying we were the first. Don’t call it a comeback we’ve been here for years. Packaging OpenStack sucks, say why.
• #12: * Community, community, community… * The power of true orchestration and task driven deployments, not a system of run thrice until nice. * YAML is not code, YAML is easy to read, YAML is not code, YAML is easy to read. * Everything is SSH, no agent, no CVEs due to agents. * If the environment is large enough simply set Ansible forks accordingly and go… * We made the LXC module. ** Pull request from rackspace for use of lxc in ansible natively: https://github.com/ansible/ansible-modules-extras/pull/123
• #13: * LXC is almost more bare metal. With LXC we can simulate additional host machines and treat the containers like just the same as we would another physical node. * LXC is compatible with a lot of networks: veth, vlan, macvlan, and even physical device management. * LXC can be built in an LVM using a real filesystem that can handle a production workload. * LXC is rock solid. Container don’t crash under our workload, we’ve had containers up with impressive uptime, though we still treat them like disposable resources.
• #14: * OSAD is in stackforge and is gated using the OpenStack development process and model. * Everything is tested with tempest. * Containers for process and service separation. * OpenStack services are installed from upstream sources. * No proprietary software that you have to buy into. ^ and we have scale using OpenStack as it was intended from the upstream developers. Our OpenStack deployment includes: galera, rabbitmq, repository servers, rsyslog, memcached, keystone, glance, nova, neutron, heat, cinder, tempest, swift, horizon
• #15: * Ansible tagging allows me to run one logical set of tasks in a given role. - Within the roles everything is a namespaced, even the tags. - there are presently 319 tags in master. * Process and service separation in containers means everything is a “node”.
• #16: * In the spirit of all things open source, we use MariaDB + Galera. * Your own personal PyPi index, local to your deployment is always available to you, but it’s also mirrored at: - http://rpc-repo.rackspace.com/ - https://mirror.rackspace.com/rackspaceprivatecloud/
• #17: * All in One simulates a larger environment than most production clouds. * We used OVS, it worked, until it didn’t. - For production we use LinuxBridge and in the future we’ll visit other plugins.
• #18: * We have an internally elected PTL at this point, though we’ll have a formal election soon. * Everything is gated through gerrit. The community commitment within the project forced us to refactor to make the system more supportable from the perspective of the greater community. That refactor forced us to “keystone-lite” the repo such that it removed all of the Rackspace-isms making the deployment system more generic. Contributor list # git log --format='%aN' | sort -u | wc -l
• #19: The community commitment within the project forced us to refactor to make the system more supportable from the perspective of the greater community. That refactor forced us to “keystone-lite” the repo such that it removed all of the Rackspace-isms making the deployment system more generic. * We have an internally elected PTL (me) at this point, though we’ll have a formal election soon. * Everything is gated through gerrit. * When we committed to stackforge we excised cruft and deployment decisions that only benefited the Rackspace Private Cloud * We made the decision to follow Ansible best practices to the letter where we could. Total lines of content in juno which includes all the things within the repo. # find . -type f -exec grep -v -e '^#' -e '^$' {} \; | wc -l 77391 Total lines of content in master which includes all the things within the repo. # find . -type f -exec grep -v -e '^#' -e '^$' {} \; | wc -l 37045 Lines of YAML no comments no new lines in master # find . -type f -name '*.yml' -exec grep -v -e '^#' -e '^$' {} \; | wc -l 9881
• #20: Vanilla OpenStack, in terms of the bits that power all of OpenStack is simpler to use, operate, and understand. Simple is amazing! * https://github.com/docker/docker/issues/7229 * https://www.mail-archive.com/[email protected]/msg03847.html * http://www.linuxquestions.org/questions/linux-general-1/which-linux-distros-use-aufs-unionfs-630594/ Cloud components are cattle, spend 30 minutes troubleshooting a broken component and if its not simple to fix kill them when they misbehave.
• #21: This is what a basic openstack_user_config.yml file looks like. * It’s easy to get started * the config is simple to understand * can become as complex as you want it. The basic openstack_user_config.yml file is essentially your entry point into Ansible inventory.
• #23: ascii diagram of stack.
• #24: This is what a basic openstack_user_config.yml file looks like. * Its easy to get started * the config is simple to understand * can become as complex as you want it. The basic openstack_user_config.yml file is essentially your entry point into Ansible inventory.
• #25: This is what a basic openstack_user_config.yml file looks like. * Its easy to get started * the config is simple to understand * can become as complex as you want it. The basic openstack_user_config.yml file is essentially your entry point into Ansible inventory.
• #26: This is what a basic openstack_user_config.yml file looks like. * Its easy to get started * the config is simple to understand * can become as complex as you want it. The basic openstack_user_config.yml file is essentially your entry point into Ansible inventory.
• #27: Thanks to Kevin and the OpenStack Ansible Deployment team, all the people who have contributed. As usual we are standing on the shoulders of giants, from OSAD to Ansible to OpenStack to Linux and more I’m not used to speaking in front of this many people, so forgive my mistakes AURO - one of the few OpenStack public clouds in Canada, we have a lot of work ahead of us with with such a great community we can get the job done Canadian data residency and ownership is important to many of our customers Fairly stock OpenStack other than using Midonet
• #28: We started our second generation deployment while the OSAD team was working on moving from Juno to Kilo and removing “raxisms”, we had to get started and that has caused us not to use as much of OSAD as we would like We definitely have some thinking and learning to do in terms of creating a process and workflow to consume OSAD and to integrate our particular infrastructure choices When we upgrade from Juno to Kilo we will bring in much more of OSAD, if not all We have a lot of work to do in terms of getting our organization up to speed and into a more “devopsy” style of working Having full, working config files is a tremendous help to anyone deploying OpenStack
• #29: We like the emphasis on testing, that is the only way we will be able to continuously improve our deployment, the only way we will be able to operate a cloud over a long period of time We need to get off of our dependency on the os packages, we will deploy from source using OSAD’s methodology We really want to be part of the Ansible, Openstack, and OSAD community, we are committed to giving back where we can, low on resources at the moment though Segregation of services is important to us
• #30: Public cloud is in many ways quite different than private cloud We have multiple tiers of support and need to ensure they have the tools to do their job but also keep segregation of duties We have to bill people, will be implementing stacktach, currently our own internal system
• #31: I think one of the most powerful things about ansible is the ability to use it to operate openstack over time, not just initial deployment
• #32: These are things that I personally struggle with and are not necessarily issues with Ansible or OSAD; have to watch I don’t shoot myself in the foot so to speak Ansible’s power and flexibility are...very powerful, almost too powerful in some cases
• #33: I wrote a quick callback plugin to send a notification to slack when a playbook causes changes or fails
• #34: I think it’s good that we are a public cloud and want to use OSAD Mostly we just need to figure out how to use as much as OSAD as possible while still having a unique environment Though almost all OpenStack deployments are unique
• #35: I do struggle with secrets and variables in Ansible If we need to do ITIL like things, how do we do that with config mgmt? Ansible as the “execution engine” for change mgmt, “continuous improvement” CONCLUSION: Basically we consume as much of OSAD as we can, add our custom requirements and account for differences, then wrap that all in monitoring, continuous integration and change management Again thanks to the community, we have a lot of work to do for AURO and a lot of learning to do and changes to implement Thanks to all the people writing modules too
• #36: Talk about where Ansible and OpenStack go from here. modules commits upstream improving ansible issues we’ve faced