Dockerizing the Hard Services: Neutron and Nova

Editor's Notes

• #2: My name is Clayton O’Neill. I work Charter Communications and I’m a Principal Engineer there on the OpenStack team We’ve found that Nova and Neutron are some of the harder services to get working happily inside of Docker containers I’m here to share with you how we’re doing that
• #3: So quick overview of what I’ll be covering today: First, What crazy command-line flags are needed to run Neutron and Nova inside of Docker at all? Second thing, when we run Neutron L3 agent inside of Docker, how do we prevent service restarts from causing HA router failovers Third, how do we make Neutron’s use of network namespaces work nicely with Docker without breaking everything Lastly, if we have time, I’ll cover how we can use the same fixes for network namespaces to also make Cinder storage backends work properly.
• #4: At the last summit a co-worker and I gave a talk titled “Deploying OpenStack Using Docker in Production” As a quick review, in that talk we covered: Why do we use Docker? How do we deploy OpenStack with Docker? And finally what pain points have we encountered. If you’re interested in a bigger picture view, that talk can be found on Youtube at the URL here. One thing to note is that a lot of the specifics we talked about then, and I’ll cover today are codified in the puppet module we use for deploying these services
• #5: This is just a quick review of where we’re at in terms of deploying docker and our environment overall We put our first service using Docker into production in July of 2015 Since we’ve switched to deploying most of our OpenStack services using Docker, you can see the list here We’re using Ceph and Solidfire for volume storage, which is relevant for Nova and Cinder docker work And we’re using VXLAN tenant networks with HA routers for networking, which affects Neutron docker work Lastly, we’re using Docker 1.12 in production, and we’ll talk about why that specific version later.
• #6: One thing I want to get out of the way up front, but you’re going to see me talk about the paths /run and /var/run interchangeably On almost all Linux systems now days, /var/run is a symlink to /run. Sometimes the symlink isn’t good enough, and we need to specify the /run path Sometimes documentation still refers to the old /var/run path. I’m going to use the one that makes sense in the context, but know that they’re the same thing
• #7: A lot of the difficulty with getting any complex service to work inside of docker is just figuring out the command-line flags you need to start the container with And with Openstack, command-lines get kind of crazy, especially for Neutron and Nova So I’m going to show you some examples from our environments But I’m going to warn you, these are…..a bit overwhelming
• #8: This is the Docker run command-line we used to start the Neutron OVS agent in our environments This is actually very slightly simplified, there were a few options I removed that weren’t relevant So, this looks pretty intimidating….
• #9: ...and then when you look at the command-line we use for nova-compute, it gets even worse. So, this is not as bad as it seems. We’re not going to go through every option here, but there are some common themes in all of these that you should be aware of. So let’s look at what some of these flags do, and why they’re important
• #10: The “--net host” flag turns off Docker networking for the container we’re going to create Normally docker would start the container walled off into it’s own network namespace, to isolate it from the rest of the system What --net host does it tells it to skip this step This leaves the container in the host network namespace, meaning that the container can see all of the network interfaces, routes, iptables rules, etc This makes networking for the container slightly faster, we do this for all of our OpenStack services This is required, because both Nova and Neutron interact directly with the host networking.
• #11: The --privileged flag is a little more straightforward. It’s not unreasonable to think of this as giving the container the ability to run processes as root This is needed so that Nova and Neutron can do things like run iptables, mount iscsi devices, and other things that require higher privileges than Docker normally allows. Nova and Neutron services still run as an unprivileged user inside the container They use the root-wrap library to execute privileged commands via sudo This works the same as if you were running the services outside of a container This is a sledgehammer and since we started putting these services in Docker, more fine grain controls have been added to Docker The --cap-add and --cap-drop flags have been added that allow controlling specific capabilities like “can control network interfaces” or “can mount devices” This is more secure, but requires more work to determine the minimal privilege set for a service.
• #12: The bulk of the command-line given in the OVS agent and nova-compute examples before were taken up by these “-v” commands These are what Docker calls “data volumes” and the way they’re being used here, they expose a part of the host’s filesystem into the container. This are similar to bind mounts that you may already be familiar with. These data volumes, or volume mounts tend to fall into two usage categories for us: Storing state on the host, instead of inside of the container, so that these containers remain disposable. Exposing host resources into the container for it’s use This slide has some of the more interesting volume mounts we use from the OVS agent example before The first example is an example of sharing resources from the host to the container: This tells Docker that we want to mount /etc/neutron from the host into the container at the same place, /etc/neutron The ‘ro’ at the end here makes that a read-only mount, so that the container can’t make changes to it. The second data volume here is mounting the /var/log/neutron directory from the host into the container, so that neutron can write logs files to the host The next one is allowing the container to have access to the /run/openvswitch directory on the host. This allows processes in the container to talk to the OVS and OVSDB apis via the unix domain sockets found in those directories. Lastly we’re mounting a few additional directories read-write into the container. These are the additional places that neutron wants to store state, and we want to ensure that is is persisted across container restarts. This will be things like pid files, DHCP lease files, etc
• #13: Simplifying the format here because we’re generally always mapping from and to the same place on the host and in the container This is a list of the more interesting directories we mount from the host into the nova-compute container /etc/ceph is mounted, read-only so that nova has access to ceph keys We need /etc/iscsi so that the iscsi commands for mounting Solidfire volumes can read their configuration and /dev so that nova can validate those block devices present before attaching them to an instance In order for live migration to work, nova needs to be able to ssh to the target node, so we want to share the host’s ssh_known_hosts file with the container also The next one, /run/openvswitch, we saw on the previous slide, this is needed so that nova-compute can plug OVS ports before instances are created Lastly we need to expose /run/libvirt into the nova-compute container, so that it can create instances, destroy instances, etc This is very similar to the OVS case. This directory contains a unix domain socket that allows nova to talk to the libvirt service that is running outside the container Unfortunately this isn’t the last time I’m going to bring up docker command line arguments in our remaining time, but that’s most of it.
• #14: Now for Nova, getting these magic incantations correct is most of the hard work of getting it to play nicely with Docker But Neutron is a little more involved.. So one feature we were eager to roll out after upgrading to the Liberty release was Neutron’s HA routers HA routers allow a virtual router to fail over automatically between two network nodes if one of them has an issue In the past not having that was a big pain point for us When we started talking about moving Neutron into docker, I was really important that we didn’t give up any reliability when we did. However, pretty early on we realized that Docker and HA routers weren’t going to work out very well without some extra work
• #15: So let’s start with a quick review of how Neutron makes routers highly available Keepalived does the bulk of the work here. It’s used for failure detection and managing ownership of the IP addresses associated with the virtual router Neutron L3 agent starts a pair of keepalived processes on two different network nodes The keepalived on each network node then sends out heartbeat messages every 2 seconds If keepalived misses a few heartbeats from its counterpart on the other network node, it assumes that something bad has happened, and it takes over their shared IP addresses Also, if keepalived is shutdown cleanly, it’s going to notify it’s counterpart that it’s going away and that it should pick up the shared IPs This IP failover isn’t instantaneous, so there is some loss of data path traffic while the other node picks up those IP addresses But traffic can also be lost because NAT mappings aren’t replicated to the other node. That means existing sessions may be dropped. So, this isn’t a seamless process, but it’s a lot better than doing it manually.
• #16: So this gets us to the crux of the problem with HA routers and Docker Because we want to be running the L3-agent inside a container, that means that keepalived will *also* be running inside the container That means that when we shutdown l3 agent, the container is going to shut down, and so will keepalived. When it shuts down, we’re going to have a failover event This leads to spurious failovers For example, if you want to change a config file option, you have to restart all your L3 agents in order for it take take effect This means you have to restart all your keepalived processes And that means that every virtual router will see at least one and sometimes two failover events while you deploy that change This is definitely customer impacting, and it would be a regression in functionality compared to not running things in Docker We also see this problem on the DHCP agent side of things, with DNSMasq Because it spawns DNSmasq inside the container by default, you can see a DHCP outage or worse a DNS outage depending on how you have it configured. That’s not as big of a deal as a router failover, but we can solve both of these problems in pretty much the same way. So let’s look at how we worked around these issues.
• #17: In order to get around this problem we want to separate keepalived and dnsmasq out from the L3 agent and DHCP agent containers To do that, we run each Keepalived and DNSMasq process in a separate container. In order to do that, we have to start the keepalived and dnsmasq containers, from inside of the L3-agent and DHCP agent containers! By doing this, we can avoid keepalived and dnsmasq failures during agent restarts To do this, we had to make a few changes:
• #18: First step is to be able to start Docker containers from inside the L3 agent and DHCP agent containers We need two things in order to be able to do that: We need to be able to talk to Docker engine from inside our agent containers This is similar to what we’ve seen already with OVS and libvirt before. Docker engine by default exposes its API over a unix domain socket That socket is found at /var/run/docker.sock This is a little different than OVS and libvirt, in that it’s a socket we share with the L3 agent and DHCP agent containers, not a directory, but docker can do that. Next, we need the Docker client inside of the container. We do that by just adding the docker package to our neutron image One thing to note here, the Docker engine API is versioned, and the engine has to support the version of the API that the client requests Docker API version tends to change every “minor” rev, so 1.10 to 1.11, 1.11 to 1.12. The docker engine does typically support a few versions of the API, so you don’t have to upgrade the docker client in the image every time you upgrade docker, but you will have to upgrade it occasionally. Lastly, I want to note that the Neutron user inside the container *cannot* access Docker Engine. Only root or a member of the Docker group can access the Docker Engine socket. That means that only commands run via root-wrap have access Docker
• #19: So the next question is: How do we get Neutron to launch a docker container? Our solution for this is very simple, and not too gross We wrote scripts that can pretend to be keepalived or dnsmasq These were placed in /usr/local/bin inside our neutron image Since /usr/local/bin is first in the PATH, this allowed them to intercept neutron’s attempts to start these binaries These scripts look at the command line, determine the router or network that is being affected, and start a dnsmasq or keepalived container using the docker run command-line These containers are named just “dnsmasq-<networkid>” or “keepalived-<routerid>” In order to make that work we did have to update the rootwrap filters to allow the binaries in /usr/local/bin to be executed We needed to set the --pid host flag on the L3 agent and DHCP agent containers This flag allows the container to see all processes on whole host, instead of just the ones in the container. This is needed because Neutron keeps track of whether or not the Keepalived and DNSmasq processes are still alive by reading the pid files that they write, and then checking to see if those processes are still alive.
• #20: We needed one more thing before we could really call this stable, which was Docker 1.12 Until 1.12, restarting the docker engine would restart *all* containers Clearly it wasn’t going to make sense to put all this work into preventing keepalived restarts, then still have router failovers because of docker restarts. Docker 1.12 upgrade went pretty smoothly. However, as I said before, we Initially we mounted the docker socket at /var/run/docker.sock into the L3 and DHCP containers and this all worked fine Until we restarted Docker engine for an unrelated configuration change The good news was that all of our existing containers continued running as intended The bad news is that the existing containers were still holding on to the old docker socket and couldn’t create containers anymore! Our work around was to configure Docker engine to create a second control socket in a directory in /var/run. (/var/run/docker-sock/docker.sock) Updated everything to mount the /var/run/docker-sock directory, instead of the file inside of it We then updated our tooling for starting the agent containers and the wrapper scripts to use that new socket in the new directory This works because each time those wrapper scripts need to call the docker client, it will see the latest socket in that directory, instead of a stale mounted socket from a previous docker engine
• #21: The final outcome here is that it works! This is an excerpt from docker ps on one of our nodes hosting Neutron virtual routers If you can read this, you’ll see we have Neutron L3-agent, metadata agent and OVS agent all running inside of containers And we also have a number of keepalived processes running inside of containers also. We’ve been running with this approach for a while and haven’t had any problems We hope this approach might be useful to others So let’s move on to our last major topic for today
• #22: So the last topic I’m going to talk about today is to how to get Docker to play nicely with Neutron’s use of network namespaces Network namespaces are kind of an esoteric topic, but there can be some confusing interactions between Docker and network namespaces
• #23: So we’ll break this topic up into three parts: What is a network namespace? How do they work? it’s simpler and weirder than you might expect How is Docker tied up into this?
• #24: So what is a network namespace? The ‘ip netns’ tool is the primary CLI tool you use for interacting with network namespaces If you look at the man page for that, then this is the explanation it gives for network namespaces Let’s look at this first sentence here “A network namespace is logically another copy of the network stack, with its own routes, firewall rules, and network devices. This isn’t too hard to follow: What this means is that network namespaces allow you to partition the networking on a server into multiple areas (or namespaces) that don’t interact except where explicitly specified. This allows you to do things like have per tenant private networks with overlapping IP space without the kernel getting confused about what traffic belongs to which tenant If you have a networking background, you probably know this as VRF (Virtual routing and forwarding)
• #25: So the second paragraph is also not too bad What it’s saying is that by default, everything runs in a single network namespace. This is the behavior you’re used to If you want to do anything special with network namespaces you have to *explicitly* ask for them to be used In our case, the thing asking for network namespaces to be used will be Neutron It uses network namespaces for both DHCP and virtual router services to enforce isolation between networks.
• #26: This last paragraph is the part that is most relevant to the issue I want to cover. One thing we can pretty easily get out of this is that the interface to network namespaces is via the /var/run/netns directory So we’re going to have to make sure our containers using namespaces have access to that directory. This is pretty easy, we can use the -v flag we’ve already talked about But that’s not going to be enough However, the primary thing this paragraph is trying to tell us something about how network namespaces are implemented, but it seems to assume you already know a lot about namespaces What it’s trying to talk about is how namespaces are created and persisted, and that’s the part that we’re interested in today so let’s dig into it a bit.
• #27: So what happens when you (or neutron) run “ip netns add test” to create a new namespace? So the short version is that it is going to call a syscall named “unshare” with a CLONE_NEWNET flag This puts the ip netns process in a new empty namespace It also changes what /proc/self/ns/net points to. This is symlink to a magic file that represents the network namespace for the current process You can see an example of what that looks like at the bottom here The last paragraph from that man page is trying to explain that the network namespace has a lifetime that is determined by something having that magic file open I.e. something using the namespace Using the namespace is determined in one of two ways, either a process is actually *in* the namespace, or a process has this magic file open, directly or indirectly. If nothing is using the namespace, then will disappear. So how does ip netns keep this namespace from disappearing after it creates it and it exits?
• #28: So this part is where the author of ip netns was clever Any reference to the magic file that represents the namespace will keep the namespace alive If you run “strace ip netns add test” you’ll see the bulk of the work is this are the 4 syscalls shown here Creates a new file, /var/run/netns/test Closes the file without writing anything to it, so the file is just going to be empty It creates the namespace, using the unshare call we talked about before. it calls the mount syscall to mount the new namespace file into the /var/run/netns directory, *over* the empty file it just created
• #29: This mount line is the key to how namespaces are persisted This creates an alias from the magic namespace file in /proc to a file that it creates in /var/run/netns When the ip netns add test process exits, the file in proc disappears, but the reference in /var/run/netns persists Because of that, other processes on the system can now use this network namespace So that’s all great, you have learned something about how network namespaces work that you didn’t already know But you didn’t sign up to come to a talk on network namespaces So let’s get back to what we’re here for
• #30: What does this have to do with Docker? Docker data volumes interact with other filesystem mounts in ways that are unintuitive And we just talked about how network namespaces are persisted and represented as filesystem mounts! And as we mentioned before, we need to share /var/run/netns as a data volume for L3 agent and DHCP agent to be able to use namespaces So how do these things interact? So this is the scenario we’re interested in: If you expose a host directory into a container, what happens if the host, or the container mounts or unmounts something underneath that directory? Because that’s effectively what ip netns is going to be doing when it adds or creates new network namespaces
• #31: The behavior is not what you would expect: When the container starts up, any mounts inside a shared directory will be visible inside the container Any new mounts made inside the container aren’t visible outside of the container Any new mounts made outside the container aren’t visible inside the container Any filesystems unmounted inside the container or on the host won’t be unmounted This is because the mounts are effectively copied into the container on startup, but nothing keeps the container and host in sync (by default) This is critically important if you’re using network namespaces, inside of a container So what happens if you’re using network namespaces and start up a container that mounts /var/run/netns? This is exactly what we need to do with L3-Agent and DHCP-Agent
• #32: So let’s look a quick test that I did to show the problem This example shows creating a network namespace called test1 on the host Then I created a container, mounted /run/netns into the container and set it up to sleep for 1 hour in the background I then just added a second network namespace on the host, named test2 So the question at this point is, what does the host and container netns directory look like? We want them to look the same!
• #33: So we see here that in /run/netns on the host, we’ve got two normal looking files If we look in /proc/mounts, then we see that those two files are mounts of the nsfs filesystem, representing the magic namespace files So that’s good, but what does it look like inside the container?
• #34: Inside the container, it looks not quite as good. We see two files, but one of them is not readable, and when we look at the mounts, the second namespace isn’t actually there The all zeros mode that you see on test2 here is a big warning sign. If you ever see this in an environment, it means your namespaces are broken In fact, we learned this the hard way, in a variety of different ways :) Let me give you an example We run neutron-netns-cleanup out of cron every 30 minutes, to clean up unused namespaces When it starts up, it catalogs all namespaces on the system before it does anything If you have normal files in this /var/run/netns, this cataloging at the beginning will fail In fact, it will never delete any namespaces, because it fails before it gets to that part I’m too embarrassed to say how long we had this broken in production before we noticed, I can tell you that on our systems, it takes about 5000 namespaces before it takes more than 30 minutes to fail with a broken file in this directory! At that point, we noticed pretty quickly, because we ended up with a *lot* of neutron-netns-cleanup commands running! You can extrapolate this to a lot of other types of scenarios with adding or deleting namespaces inside the container, or outside the container, etc. Many of these will end up with stale namespaces like above, or with undeletable namespaces ….So how do we fix this?
• #35: So the solution to this is pretty straightforward, and uses flags to Docker volume mounts Underneath the covers this uses a feature of the Linux kernel that is called “shared subtree flags”. These flags have a lot of flexibility about how mount propagation is handled between the host and the container In Docker 1.10 they added support for using these shared subtree flags with volume bind mounts If you’re interested there are links here to the kernel documentation Oddly enough probably the most readable documentation on this is found in the yet to be published documentation on docker service creation The short version of all of this though is that any time you’re using network namespaces inside of a container, you need to add /run/netns as a volume mount, and you need to specify the shared flag This will ensure that any namespace creation or deletion (and mount/unmounts as a result) properly propagate between the host and the container So this solves the network namespace issue, but there are other areas where these flags can be useful This flag can be used any time you need to make sure filesystem mounts are in sync between the host and container, not just network namespaces So that leads us into our bonus topic...
• #36: How are network namespaces related to Cinder?! Now that you know that the issue with namespaces and docker have to do with mount points You may be able to guess how this relates to Cinder also
• #37: So specifically the way this helps with Cinder is with the NFS backend for Cinder In our smaller development environments, we run Cinder with the NFS backend, so that we don’t need to have a Ceph cluster to do testing The NFS backend wants to be able to run the mount commands for shares itself. Since we’re running Cinder and Nova-Compute inside a container, that poses a problem Nova compute can mount the share inside the container, With the default mount behavior, libvirt won’t be able to see it when you ask it to attach the volume to an instance So we need to figure out how to make mounts inside the container for /var/lib/cinder/mnt show up outside the container This is basically the same solution as we had for network namespaces, but with a few extra steps We have to run a few commands before we start the container When we were dealing with namespaces, we didn’t need to run these first two commands there, why? For the docker shared flag to work, two things must be true. The volume you’re mounting into the container must be it’s own mount point It must already have the shared flag set on that mount on the host So why don’t we need this for network namespaces? /var/run/netns is already a filesystem of its own ip netns makes it shared every time it creates a namespace With Cinder we have to do that ourselves The first command here makes /var/lib/cinder/mnt a mount point by mounting it on top of itself. This is kind of gross The next command marks that mount point as shared, so that docker can share it into the container And then finally when we start the cinder-volume and nova-compute containers, we need to explicitly share the mount directory into those containers with the shared flag With these 3 changes, this will get Cinder and Nova working inside of docker containers, even with the NFS backend.
• #38: So that’s everything I have today. If you have any questions or comments later, feel free to contact me, but I think we’ll have time for a few questions right now.