4

vol.

NETWORKING,
SECURITY
& STORAGE
WITH
DOCKER
& CONTAINERS
EDITED & CURATED BY ALEX WILLIAMS
The New Stack:
The Docker and Container Ecosystem Ebook Series

Alex Williams, Founder & Editor-in-Chief

Benjamin Ball, Technical Editor & Producer
Hoang Dinh, Creative Director
Lawrence Hecht, Data Research Director

Contributors:
Judy Williams, Copy Editor

Norris Deajon, Audio Engineer

TABLE OF CONTENT
Sponsors ........................................................................................................................................ 4
Introduction .................................................................................................................................. 5
NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS

IBM: Bridging Open Source and Container Communities ................................................... 8

The Container Networking Landscape Explained ................................................................. 9
Cisco: Uniting Teams with a DevOps Perspective ................................................................30
Three Perspectives on Network Extensibility .......................................................................31
Twistlock: An Automated Model for Container Security.....................................................37
Assessing the Current State of Container Security ..............................................................38
Joyent: A History of Security in Container Adoption ..........................................................52
Methods for Dealing with Container Storage........................................................................53
Nuage Networks: ............................................71
Identifying and Solving Issues in Containerized Production Environments ..................72
Docker: Building the Foundation of Secure Containers .....................................................85
NETWORKING, SECURITY & STORAGE DIRECTORY

Networking ..................................................................................................................................87
Security.........................................................................................................................................90
Storage .........................................................................................................................................95

Disclosures...................................................................................................................................98

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 3

 SPONSORS
We are grateful for the support of the following ebook series sponsors:

And the following sponsors for this ebook:

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 4

INTRODUCTION
Keeping pace with the technology, practitioners and vendors in the

publishing our container ecosystem ebook series. Every time we narrow

our area of focus, we’ve been opened up to yet another microcosm of
experienced users, competing products and collaborative projects. Our
solutions directory for the container ecosystem series has expanded with
each book, and currently we have catalogued over 450 active products
and projects. Calling this container technology space an ecosystem has

community makes greater strides.

Container technology has the ability to add so much speed to the

development and deployment process, but deciding what option to

Comparatively, there are relative veterans who have long been composing

pipeline, and automated much of the orchestration around containers.

These practitioners are thinking more about how to securely network
containers, maintain persistent storage, and scale to full production
environments. With this ebook series, we look to educate both
newcomers and familiars by going beyond operational knowledge and
into analysis of the tools and practices driving the market.

Networking is a necessary part of distributed applications, and

networking in the data center has only become more complex. In
introducing container networking, we take a closer look at the demands
that are driving this change in complexity, the evolution of types of

service discovery, and networking with OpenStack.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 5

INTRODUCTION

It was also important for us to include a solid perspective on the best

practices and strategies around container security. Container security has

been cited as a barrier to entry for containers. This ebook explains how
containers can facilitate a more secure environment by addressing

include topics such as image provenance, security scanning, isolation and

least privilege, auditing and more.

portable container lifecycle. We cover how to account for the temporary

architectures, host-based persistence, multi-host storage, volume plugins

storage strategies with the intent to show some of the patterns that have
worked for others implementing container storage.

Networking, security and storage are all topics with broad and deep
subject matter. Each of these topics deserves a full book of its own, but
setting the stage in this initial ebook on these topics is an important
exercise. The container ecosystem is becoming as relevant for operations
teams as it is for developers who are packaging their apps in new ways.
This combined interest has created a renaissance for technologists, who
have become the central players in the emergence of new strategic
thinking about how developers consume infrastructure.

There are more ways than one to skin a cat, and while we try to educate
on the problems, strategies and products, much of this will be quickly
outgrown. In two years’ time, many of the approaches to networking,
security and storage that we discuss in the ebook will not be as relevant.
But the concepts behind these topics will remain part of the conversation.
Containers will still need to communicate with each other securely,

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 6

INTRODUCTION

container storage and security will need policy management, third-party

storage and databases will need to be integrated so that stateful apps can

worthy of their own book. So be on the lookout for more publications

from us. In the meantime, please reach out to our team any time with
feedback, thoughts, and ideas for the future. Thanks so much for your
interest in our ebook series.

Thanks,

Benjamin Ball
Technical Editor and Producer
The New Stack

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 7

BRIDGING OPEN SOURCE
AND CONTAINER
COMMUNITIES
McGee talks about bringing together various tools
in the open source and container ecosystems,
including the many networking tools looking to
address the needs of containers. IBM is focused
on bringing these communities together by contributing to core
technologies and building a world-class cloud platform. Listen on
SoundCloud or Listen on YouTube

Jason McGee

Estes talks about the challenges of networking

containers, the evolution of container namespaces,
and the current state of container security, to

discussion extends into the plugin ecosystem for Docker, and how

Listen on SoundCloud or Listen on YouTube

Phil Estes

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 8

THE CONTAINER
NETWORKING
LANDSCAPE EXPLAINED
LEE CALCOTE

etworking is an inherent component to any distributed applica-

N tion, and one of the most complicated and expansive technolo-

gies. As application developers are busily adopting container
technologies, the time has come for network engineers to prepare for the
unique challenges brought on by cloud-native applications.

With the popularization of containers and microservices, data center

networking challenges have increased in complexity. The density by which
containers are deployed on hosts (servers) presents challenges in terms of

from a few network interfaces on bare metal hosts, to a few network

interfaces per virtual machine (VM) with twenty or so VMs per host, to a
few interfaces per container with hundreds of containers per host.

Despite this increased density, the demands and measurements of

reliability made of conventional networking hardware are the same
demands and expectations made of container networking. Inevitably,
operators will compare the performance of virtual machine networking to

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 9

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

improved performance when containers are run directly on bare metal.

mistakenly set.

This article is split into two primary areas of focus around types of

Networking starts with connectivity. Part one starts with the various ways
in which container-to-container and container-to-host connectivity is
provided.

This focuses on a breakdown of current container networking types,

including:

• None

• Bridge

• Overlay

• Underlay

For the second half of this article, there are two container networking

• Container Network Model (CNM)

• Container Network Interface (CNI)

greatly.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 10

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Types of Container Networking

While many gravitate toward network overlays as a popular approach to
addressing container networking across hosts, the functions and types of
container networking vary greatly and are worth better understanding as
you consider the right type for your environment.

Some types are container engine-agnostic, and others are locked into a

breadth of functionality or on being IPv6-friendly and multicast-capable.

Which one is right for you depends on your application needs,
performance requirements, workload placement (private or public cloud),
etc. Let’s review the more commonly available types of container
networking.

Antiquated Types of Container Networking

The approach to networking has evolved as container technology
advances. Two modes of networking have come and all but disappeared
already.

Links and Ambassadors

Prior to having multi-host networking support and orchestration with
Swarm, Docker began with single-host networking, facilitating network
connectivity via links as a mechanism for allowing containers to
discover each other via environment variables or /etc/hosts
and transfer information between containers. The links capability was
commonly combined with the ambassador pattern to facilitate linking
containers across hosts and reduce the brittleness of hard-coded links.
The biggest issue with this approach was that it was too static. Once a

related containers or services moved to new IP addresses, then it was

impossible to change the values of those variables.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 11

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Container-Mapped Networking
In this mode of networking, one container reuses (maps to) the
networking namespace of another container. This mode of networking
may only be invoked when running a docker container like this:
--net:container:some_container_name_or_id.

inside of the network stack that has already been created inside of
another container. While sharing the same IP and MAC address and port

on the two containers will be able to connect to each other over the
loopback interface.

This style of networking is useful for performing diagnostics on a running

container and the container is missing the necessary diagnostic tools (e.g.,
curl or dig). A temporary container with the necessary diagnostics tools

Container-mapped networking may be used to emulate pod-style

networking, in which multiple containers share the same network

sharing the same IP address, are inherent to the notion that containers
run in the same pod, which is the behavior of rkt containers.

Current Types of Container Networking

Lines of delineation of networking revolve around IP-per-container versus
IP-per-pod models and the requirement of network address translation
(NAT) versus no translation needed.

None
None is straightforward in that the container receives a network stack, but
lacks an external network interface. It does, however, receive a loopback

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 12

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

interface. Both the rkt and docker container projects provide similar
behavior when none or null networking is used. This mode of container
networking has a number of uses including testing containers, staging a
container for a later network connection, and being assigned to
containers with no need for external communication.

Bridge
A Linux bridge provides a host-internal network in which containers on the
same host may communicate, but the IP addresses assigned to each
container are not accessible from outside the host. Bridge networking
leverages iptables for NAT and port-mapping, which provide single-
host networking. Bridge networking is the default Docker network type
(i.e., docker0), where one end of a virtual network interface pair is
connected between the bridge and the container.

1. A bridge is provisioned on the host.

2. A namespace for each container is provisioned inside that bridge.

3. Containers’ ethX are mapped to private bridge interfaces.

4. iptables with NAT are used to map between each private container
and the host’s public interface.

NAT is used to provide communication beyond the host. While bridged

containers running on one host, there’s a performance cost related to

using NAT.

Host
In this approach, a newly created container shares its network namespace

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 13

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

the container has access to all of the host’s network interfaces, unless

network stack.

Host networking is the default type used within Mesos. In other words, if
the framework does not specify a network type, a new network
namespace will not be associated with the container, but with the host
network. Sometimes referred to as native networking, host networking is
conceptually simple, making it easier to understand, troubleshoot and
use.

Overlay
Overlays use networking tunnels to deliver communication across hosts.
This allows containers to behave as if they are on the same machine by
tunneling network subnets from one host to the next; in essence,
spanning one network across multiple hosts. Many tunneling technologies
exist, such as virtual extensible local area network (VXLAN).

VXLAN has been the tunneling technology of choice for Docker libnetwork,
whose multi-host networking entered as a native capability in the 1.9
release. With the introduction of this capability, Docker chose to leverage

neighbor table exchange and convergence times.

For those needing support for other tunneling technologies, Flannel may
be the way to go. It supports udp, vxlan, host-gw, aws-vpc or gce.
Each of the cloud provider tunnel types creates routes in the provider’s
routing tables, just for your account or virtual private cloud (VPC). The
support for public clouds is particularly key for overlay drivers given that
among others, overlays best address hybrid cloud use cases and provide
scaling and redundancy without having to open public ports.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 14

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Multi-host networking requires additional parameters when launching the

Docker daemon, as well as a key-value store. Some overlays rely on a
distributed key-value store. If you’re doing container orchestration, you’ll
already have a distributed key-value store lying around.

Overlays focus on the cross-host communication challenge. Containers on

segmented from one another.

Underlays
Underlay network drivers expose host interfaces (i.e., the physical network
interface at eth0) directly to containers or VMs running on the host. Two
such underlay drivers are media access control virtual local area network
(MACvlan) and internet protocol vlan (IPvlan). The operation of and the
behavior of MACvlan and IPvlan drivers are very familiar to network
engineers. Both network drivers are conceptually simpler than bridge

Moreover, IPvlan has an L3 mode that resonates well with many network

clouds, underlays are particularly useful when you have on-premises

VLAN, underlay networking allows for one VLAN per subinterface.

MACvlan
MACvlan allows creation of multiple virtual network interfaces behind the
host’s single physical interface. Each virtual interface has unique MAC and
IP addresses assigned, with a restriction: the IP addresses need to be in
the same broadcast domain as the physical interface. While many
network engineers may be more familiar with the term subinterface (not
to be confused with a secondary interface), the parlance used to describe

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 15

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

MACvlan virtual interfaces is typically upper or lower interface. MACvlan

networking is a way of eliminating the need for the Linux bridge, NAT and
port-mapping, allowing you to connect directly to the physical interface.

MACvlan uses a unique MAC address per container, and this may cause
issue with network switches that have security policies in place to prevent

interface.

which completely isolates the host from the containers it runs. The host
cannot reach the containers. The container is isolated from the host. This
is useful for service providers or multi-tenant scenarios, and has more
isolation than the bridge model.

Promiscuous mode is required for MACvlan; MACvlan has four modes of

operation, with only the bridge mode supported in Docker 1.12. MACvlan
bridge mode and IPvlan L2 mode are just about functionally equivalent.

protocols were designed with on-premises use cases in mind. Your public
cloud mileage will vary as most do not support promiscuous mode on
their VM interfaces.

A word of caution: MACvlan bridge mode assigning a unique MAC address

end-to-end visibility; however, with a typical network interface card (NIC),

e.g., Broadcom, having a ceiling of 512 unique MAC addresses, this upper-
limit should be considered.

IPvlan
IPvlan is similar to MACvlan in that it creates new virtual network

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 16

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

same MAC address of the physical interface. The need for this behavior is

more than one MAC address.

Best run on kernels 4.2 or newer, IPvlan may operate in either L2 or L3

modes. Like MACvlan, IPvlan L2 mode requires that IP addresses assigned
to subinterfaces be in the same subnet as the physical interface. IPvlan L3
mode, however, requires that container networks and IP addresses be on

ip link, is
ephemeral, so most operators use network startup scripts to persist

stands to improve. For example, when new VLANs are created on a top of
rack switch, these VLANs may be pushed into Linux hosts via the exposed
container engine API.

MACvlan and IPvlan

When choosing between these two underlay types, consider whether or
not you need the network to be able to see the MAC address of the
individual container. With respect to the address resolution protocol (ARP)

802.1D packets. In IPvlan L3 mode, however, the networking stack is

in. In this sense, IPvlan L3 mode operates as you would expect an L3

router to behave.

Note that upstream L3 routers need to be made aware of networks

created using IPvlan. Network advertisement and redistribution into the

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 17

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

network still needs to be done. Today, Docker is experimenting with

Border Gateway Protocol (BGP). While static routes can be created on top
of the rack switch, projects like goBGP have sprouted up as a container
ecosystem-friendly way to provide neighbor peering and route exchange
functionality.

Although multiple modes of networking are supported on a given host,

MACvlan and IPvlan can’t be used on the same physical interface
concurrently. In short, if you’re used to running trunks down to hosts, L2
mode is for you. If scale is a primary concern, L3 has the potential for
massive scale.

Direct Routing
For the same reasons that IPvlan L3 mode resonates with network
engineers, they may chose to push past L2 challenges and focus on
addressing network complexity in Layer 3 instead. This approach

manage the container networking. The container networking solutions

focused at L3 use routing protocols to provide connectivity, which are
arguably easier to interoperate with existing data center infrastructure,
connecting containers, VMs and bare metal servers. Moreover, L3

Project Calico is one such project and uses BGP to distribute routes for

it to seamlessly integrate with existing data center infrastructure without

the need for overlays. Without the overhead of overlays or encapsulation,
the result is networking with exceptional performance and scale. Routable
IP addresses for containers expose the IP address to the rest of the world;
hence, ports are inherently exposed to the outside world.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 18

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Network engineers trained and accustomed to deploying, diagnosing and

to digest. However, it’s worth noting that Calico doesn’t support

overlapping IP addresses.

Fan Networking
Fan networking is a way of gaining access to many more IP addresses,
expanding from one assigned IP address to 250 IP addresses. This is a
performant way of getting more IPs without the need for overlay
networks. This style of networking is particularly useful when running
containers in a public cloud, where a single IP address is assigned to a
host and spinning up additional networks is prohibitive, or running
another load-balancer instance is costly.

Point-to-Point
Point-to-point is perhaps the simplest type of networking, and the default
networking used by CoreOS rkt. Using NAT, or IP Masquerade (IPMASQ), by
default, it creates a virtual ethernet pair, placing one on the host and the
other in the container pod. Point-to-point networking leverages iptables

for internal communication between other containers in the pod over the
loopback interface.

Capabilities
Outside of pure connectivity, support for other networking capabilities
and network services needs to be considered. Many modes of container
networking either leverage NAT and port-forwarding or intentionally avoid
their use. IP address management (IPAM), multicast, broadcast, IPv6, load-

and performance are all additional considerations when selecting

networking.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 19

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

The question is whether these capabilities are supported and how

developers and operators are empowered by them. Even if a container
networking capability is supported by your runtime, orchestrator or plugin
of choice, it may not be supported by your infrastructure. While some tier

in top public clouds reinforces the need for other networking types, such
as overlays and fan networking.

In terms of IPAM, to promote ease of use, most container runtime engines

default to host-local for assigning addresses to containers, as they are

is universally supported across container networking projects. Container

Network Model (CNM) and Container Network Interface (CNI) both have

key capability to adoption in many existing environments.

Linux containers: The container network model (CNM) and the container
network interface (CNI). As stated above, networking is complex and there
are many ways to deliver functionality. Arguments can be made as to
which one is easier to adopt than the next, or which one is less tethered to
their benefactor’s technology.

When evaluating any technology, some important considerations are

community adoption and support. Some perspectives have been formed
on which model has a lower barrier to entry. Finding the right metrics to
determine the velocity of a project is tricky. Plugin vendors also need to
consider the relative ease by which plugins may be written for either of
these two models.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 20

 THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Container Network Model

The Container Network Model
Docker, adopted by projects such as libnetwork, with plugins built by
projects and companies such as Cisco Contiv, Kuryr, Open Virtual
Networking (OVN), Project Calico, VMware and Weave.

Libnetwork provides an interface between the Docker daemon and

network drivers. The network controller is responsible for pairing a driver
to a network. Each driver is responsible for managing the network it owns,
including services provided to that network like IPAM. With one driver per
network, multiple drivers can be used concurrently with containers

(built-in to libnetwork or Docker supported) or remote (third-party

FIG 1:

Container Network Model (CNM) Drivers

Docker
Runtime

Container Network Model (libnetwork)

None Bridge Overlay 3rd-Party

Plugins

Native Drivers (built-in to libnetwork or Docker supported) Remote Drivers

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 21

Source: Lee Calcote

 Container Network Model Interfacing
THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Network 1 Network 2

Endpoint Endpoint Endpoint

Network Sandbox Network Sandbox

Docker Container Docker Container

Source: Docker, Inc.

FIG 2:

plugins). The native drivers are none, bridge, overlay and MACvlan. Remote

having local scope (single host) or global scope (multi-host).

• Network Sandbox: Essentially the networking stack within a

container, it is an isolated environment to contain a container’s

• Endpoint: A network interface that typically comes in pairs. One end

of the pair sits in the network sandbox, while the other sits in a
designated network. Endpoints join exactly one network, and multiple
endpoints can exist within a single network sandbox.

• Network:
group of endpoints that are able to communicate with each other.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 22

 THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

--label
libnetwork and drivers. Labels are powerful in that the runtime may
inform driver behavior.

Container Network Interface

The Container Network Interface (CNI) is a container networking
Apache
Mesos, Cloud Foundry, Kubernetes, Kurma and rkt. There are also plugins
created by projects such as Contiv Networking, Project Calico and Weave.

network vendor engineers to be a simple contract between the container

and output from CNI network plugins.

FIG 3:

Container Network Interface (CNI) Drivers

Container
Runtime

Container Network Interface (CNI)

Loopback Bridge PTP MACvlan IPvlan 3rd-Party

Plugin Plugin Plugin Plugin Plugin Plugin

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 23

Source: Lee Calcote

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Multiple plugins may be run at one time with a container joining networks

in JSON format, and instantiated as new namespaces when CNI plugins

are invoked. CNI plugins support two commands to add and remove
container network interfaces to and from networks. Add gets invoked by
the container runtime when it creates a container. Delete gets invoked
by the container runtime when it tears down a container instance.

CNI Flow

container and assign it a container ID, then pass along a number of

attaches the container to a network and reports the assigned IP address

back to the container runtime via JSON.

Mesos is the the latest project to add CNI support, and there is a Cloud
Foundry implementation in progress. The current state of Mesos
networking uses host networking, wherein the container shares the same
IP address as the host. Mesos is looking to provide each container with its
own network namespace, and consequently, its own IP address. The
project is moving to an IP-per-container model and, in doing so, seeks to
democratize networking such that operators have freedom to choose the
style of networking that best suits their purpose.

Currently, CNI primitives handle concerns with IPAM, L2 and L3, and
expect the container runtime to handle port-mapping (L4). From a Mesos
perspective, this minimalist approach comes with a couple caveats, one of

rules to be used for a container; this capability may be handled by the

container runtime. A second caveat is the fact that while operators should

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 24

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

associated with the particular instance of the container.

CNM and CNI

democratize the selection of which type of container networking may be

used, in that both are driver-based models, or plugin-based, for creating
and managing network stacks for containers. Each allows multiple
network drivers to be active and used concurrently, in that each provide a
one-to-one mapping of network to that network’s driver. Both models
allow containers to join one or more networks. And each allows the
container runtime to launch the network in its own namespace,

the network to the network driver.

This modular driver approach is arguably more attractive to network

responsibility for ensuring service-level agreements (SLAs) are met and

security policies are enforced.

Both models provide separate extension points, aka plugin interfaces, for

per function encourages composability.

CNM does not provide network drivers access to the container’s network

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 25

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

access to the container network namespace. CNI is considering how it

might approach arbitration
future.

CNI supports integration with third-party IPAM and can be used with any
container runtime. CNM is designed to support the Docker runtime engine
only. With CNI’s simplistic approach, it’s been argued that it’s
comparatively easier to create a CNI plugin than a CNM plugin.

These models promote modularity, composability and choice by fostering

an ecosystem of innovation by third-party vendors who deliver advanced
networking capabilities. The orchestration of network micro-
segmentation can become simple API calls to attach, detach and swap
networks. Interface containers can belong to multiple networks, and each

to detach a network service from an old container and attach it to a new

container.

Container Networking in OpenStack

Initially focused on infrastructure automation for virtual machines,
OpenStack has come to focus on the needs of containers. Kuryr and
Magnum are not their only container-related projects, but certainly the
two concerned with container networking.

Kuryr
Kuryr, a project providing container networking, currently works as a
remote driver for libnetwork to provide networking for Docker using
Neutron as a backend network engine. Support for CNM has been
delivered and the roadmap for this project includes support for CNI.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 26

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

Magnum
Magnum, a project providing Containers as a Service (CaaS) and
leveraging Heat to instantiate clusters running other container
orchestration engines, currently uses non-Neutron networking options for
containers.

Work is ongoing to integrate Kuryr and Magnum. The notion that

challenges for the project teams to overcome.

Networking with Intention

behavior in infrastructure-agnostic terms by using policy. With the

increased density of network interfaces, volume of IP addresses, and
complexity of communication across containers running interdependent

networking draws from the successes of established concepts, referring to

them by the terms invariant, portable, composable, scalable.

Invariant, for example, is similar to the idempotency concept in

this as invariant, meaning the that intent doesn’t change as a result of

equipment.

Another intent-based concept is portability, a concept characterized as

policy supporting environments with heterogenous vendor equipment or

is composable when policy can concurrently leverage disparate network

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 27

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

The scalable quality of intent-based networking is self-explanatory; it

enables each node or host to make networking decisions in a distributed
fashion, with each node being aware of the intent-based policy. We’ve

advance the capabilities of container networking technologies; intent-

based networking stands to carry the ball further.

Summary
We discussed a number of considerations for choosing which type of

combination. Performance is certainly one of those considerations and

will be the subject of further research. Outside of the various types of

highlighted is understanding to what extent you need to integrate with

incumbent systems in your environment. For example, in the case of
on-premises workloads, you’ll have an existing IPAM solution. IPAM is
provided by most container network vendors’ drivers, but only some have
integration with leading IPAM providers, such as Cisco, Infoblox,
SolarWinds, etc.

As vendors and projects continue to evolve, the networking landscape

Docker’s acquisition of SocketPlane, and the transition of Flannel to Tigera

formed around Canal. Canal is a portmanteau of
Calico and Flannel and a combination of those two projects. CoreOS will
provide ongoing support for Flannel as an individual project, and will be
integrating Canal with Tectonic, their enterprise solution for Kubernetes.
Other changes come in the form of new project releases. Docker 1.12’s
release of networking features, including underlay and load-balancing
support, is no small step forward for the project.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 28

THE CONTAINER NETWORKING LANDSCAPE EXPLAINED

While there’s a large number of container networking technologies and

distinctly unique ways of approaching them, we’re fortunate in that much
of the container ecosystem seems to have converged and built support
around only two networking models, at least for now. Developers would
like to eliminate manual network provisioning in containerized
environments, and barring those who have misconceptions of their job
insecurity, network engineers are ready for the same.

Like other resources, an intermediary step to automated provisioning is

pre-provisioning, meaning network engineers would preallocate networks
with assigned characteristics and services, such as IP address space, IPAM,
routing, QoS, etc., and developers or deployment engineers would identify
and select from a pool of available networks in which to deploy their
applications. Pre-provisioning needs to become a thing of the past, as
we’re all ready to move on to automated provisioning.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 29

UNITING TEAMS
WITH A
DEVOPS PERSPECTIVE
In this discussion with Ken Owens of Cisco, we talk
about how to apply existing models of
networking, security and storage to containerized
environments. Owens talks about how bringing a
DevOps mindset forward to containers reveals the strong
foundations for security, networking and storage and how they still

involved in operating these environments, from Linux and network

administrators to security and storage teams. It’s important to link
up these perspectives through the components they control, while
creating policy around resource management. The discussion
moves on to the roles of Contiv and Mantl, and how they address
these issues. Listen on SoundCloud or Listen on YouTube

Ken Owens

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 30

THREE PERSPECTIVES
ON NETWORK
EXTENSIBILITY
SCOTT FULTON III

critical aspect of any cloud-based deployment is managing the

A networking between the various components of the workload. In

this chapter, we’ll present perspectives on three styles of inte-
grated container networking by way of plugins. The previous section intro-
duced the Container Network Model (CNM) and the Container Network
Interface (CNI). We’ll discuss the origin of these models, as well as a third
area that includes the Apache Mesos ecosystem.

Container Network Model and

Libnetwork
Docker’s extensibility model adds capability to the daemon in the way a
library adds capability to an operating system. It involves a code library,
similar to Docker’s runtime, but used as a supplement. That library is
called libnetwork, produced as a third-party project by the development
team SocketPlane, which Docker Inc. acquired in March 2015.

Essentially, libnetwork provides a kind of plank on which developers may

write network drivers. The binding principle of libnetwork is called the

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 31

THREE PERSPECTIVES ON NETWORK EXTENSIBILITY

Container Network Model (CNM), which was conceived as a container’s bill

of rights. One of those rights is equal access to all other containers in a

dividing network addresses. A service discovery model provides a means

for containers to contact one another.

The intention is for libnetwork to implement and use any kind of

networking technology to connect and discover containers. It does not
specify one preferred methodology for any network overlay scheme.
Project Calico is an example of an independent, open source project to
develop a vendor-neutral network scheme for Layer 3; developers have
recently made Project Calico’s calicoctl library an addressable component
of a Docker plugin.

container system for databases, called Flocker. It uses libnetwork, and is

addressable using Weaveworks’ Weave Net overlay. As ClusterHQ Vice
President of Product Mohit Bhatnagar told us:

“I think we are at a point where customers who initially thought of

containers for stateless services need to realize both the need and the
potential for stateful services. And we are actually very pleasantly
surprised about the number of customer engagements ... regarding
stateful services.”

The critical architectural distinction for the Docker scheme concerns just
what part is being extended. In Docker architecture, the daemon of
Docker Engine runs on the host server where the applications are being

replacing it with an amalgamated view of servers running in a cluster.

Engine at a lower layer.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 32

THREE PERSPECTIVES ON NETWORK EXTENSIBILITY

Container Network Interface

Kubernetes published guidelines for implementing networked
extensibility. It should be capable of addressing other containers’ IP
addresses without resorting to network address translation (NAT), and
should permit itself to be addressed the same way. Essentially, as long as

context, theoretically anything could extend what you do with

Kubernetes, but nothing had to be bound to it.

“We looked at how we were going to do networking in Kubernetes,”

explained Google Engineering Manager Tim Hockin, “and it was pretty
clear that there’s no way that the core system could handle every network

system. We had to externalize it, and plugins are the way we’re doing that.”

Then CoreOS produced its Container Network Interface (CNI). It’s more
rudimentary than CNM, in that it only has two commands: create a

instantiate the container’s contents and set it up with an IP address. But

with CNI. As a result, Flannel and Weave Net have been implemented as
Kubernetes plugins using CNI.

forms of networking into a Kubernetes environment, they also incur some

costs. “The general Kubernetes position on overlays is, you should only
use them if you really, really have to. They bring their own levels of

more users of Kubernetes are going directly to L3 routing, and they’re

having a better time with that than they are with overlays.”

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 33

THREE PERSPECTIVES ON NETWORK EXTENSIBILITY

concluded that which model you choose will depend, perhaps entirely, on
how much integration you require between containers and pre-existing
workloads.

“If your job previously ran on a VM, and your VM had an IP and could talk
to the other VMs in your project,” explained ClusterHQ Senior Vice
President of Engineering and Operations Sandeepan Banerjee, “you are

Banerjee then cited Kubernetes’ no-NAT stipulation as the key reason.

“If that is not a world that you are coming from,” he continued, “and you
want to embrace the Docker framework as something you see as

then the Docker proposal is powerful, with merits, with probably a lot
more tunability overall.”

Mesosphere and Plugins from the

Opposite End
Mesosphere has produced perhaps the most sophisticated commercial
implementation of Mesos with
competitor against Kubernetes in the form of its Marathon orchestrator.

As a scheduling platform, the job of extending the reach of Mesos has

historically been done from the opposite side of the proverbial bridge.
Enabling scheduling for big data processes in Hadoop, job management
processes in Jenkins, and container deployment in Docker, has all been
done from within those respective platforms.

Mesosphere began to pave the way for properly interfaced containers to

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 34

THREE PERSPECTIVES ON NETWORK EXTENSIBILITY

extend this library by way of CNI. At the time of this writing, Mesosphere
had published a document stating its intent to implement CNI support in

with varying interfaces and implementations for networking and storage,”

said Ben Hindman, founder and chief architect at Mesosphere, “that the
means of doing plugins, I think, is a pretty important part. What I think is

Docker will become the universal plugins. And I think what you’re seeing in
the industry already today is, that’s not the case.”

discovery system called Minuteman to connect containers to one another.

It works by intercepting packets as they’re being exchanged from a

the proper destination IPs. This accomplishes the cross-cloud scope that

establishing routing rules between containers in that virtual network.

Mesosphere does not reinvent the wheel here at all; actually, it gives users
their own choice of overlay schemes, based on performance or other
factors.

Hindman told us he sees value in how Flannel, Weave, and other network
overlay systems solve the problem of container networking, at a much
higher level than piecing together a VXLAN. The fact that such an
alternative would emerge, he said, “is just capturing the fact that we, as an
industry, are sort of going through and experimenting with a couple of

settle on a handful of things, and overlays are still going to be there. But
there are going to be some other ways in which people link together and

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 35

THREE PERSPECTIVES ON NETWORK EXTENSIBILITY

connect up containers that are not using pre-existing, SDN-based

technologies.”

Integration Towards the Future

don’t quite understand the concepts behind the processes they are

require to invest both their faith and their capital expense. Some might
think this is a watering down of the topic. In truth, integration is an
elevation of the basic idea to a shared plane of discussion. Everyone
understands the basic need to make old systems coexist, interface and
communicate with new ones. So even though the methodologies may
seem convoluted or impractical in a few years, the inspiration behind
working toward a laudable goal will have made it all worth pursuing.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 36

AN AUTOMATED
MODEL FOR CONTAINER
SECURITY
In this discussion with John Morello of Twistlock,
we talk about how containers can actually be a
better medium for automating and securing
applications. Containers being immutable and
lightweight makes it easier to follow images from early in the
development life cycle all the way to the registry and compute
environments. Twistlock collects data from this life cycle and creates
a predictive model for a container’s behavior. This model looks for
inconsistent behaviors, and depending on what you want, it can set

we talk about Twistlock’s focus on four distinct use cases, recent

changes to its core features, the value of partner integration and
more. Listen on SoundCloud or Listen on YouTube

John Morello

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 37

ASSESSING THE
CURRENT STATE OF
CONTAINER SECURITY
ADRIAN MOUAT

ny rational organization that wishes to run mission-critical

A services on containers will at some point ask the question: “But

is it secure? Can we really trust containers with our data and
applications?”

machines (VMs) debate and a discussion of the protection provided by the

hypervisor layer in VMs. While this can be an interesting and informative
discussion, containers versus VMs is a false dichotomy; concerned parties
should simply run their containers inside VMs, as currently happens on
most cloud providers. A notable exception is Triton from Joyent, which
uses SmartOS Zones to ensure isolation of tenants.

There is also a growing community who believe that container security

and isolation on Linux has improved to the point that one can use bare
metal container services not using VMs for isolation; for example, IBM has
built a managed container service on the public Bluemix cloud service
that is running without VM isolation between tenants.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 38

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

To retain the agility advantage of containers, multiple containers are run

within each VM. Security conscious organizations may use VMs to separate

processing billing information may be scheduled on separate nodes to

Hyper_, Intel and VMware

VM-based frameworks that implement the Docker API in an attempt to

Once we accept that moving to containers does not imply surrendering

step is to investigate the security gains that can be achieved through the

will push to the continuous integration (CI) system, which will build and
test the images. The image will then be pushed to the registry. It is now
ready for deployment to production, which will typically involve an
orchestration system such as Docker’s built-in orchestration, Kubernetes,
Mesos, etc. Some organizations may instead push to a staging
environment before production.

In a system following security best practices, the following features and

properties will be present:

• Image Provenance: A secure labelling system is in place that

production environment came from.

• Security Scanning: An image scanner automatically checks all

images for known vulnerabilities.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 39

 ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

Image Scanner Ongoing

(security check) Auditing

DEVELOPER
COMMITS PULLS FEEDBACK
LATEST FEEDBACK
CODE LOOP
SIGNED LOOP
IMAGE

PULLS
CI/CD SENDS
LATEST Production
System SIGNED Registry STABLE Environment
IMAGES
IMAGE

TRIGGERS
UPDATE

Source: Adrian Mouat

FIG 1:
-

• Auditing: The production environment is regularly audited to ensure

all containers are based on up-to-date containers and both hosts and

• Isolation and Least Privilege: Containers run with the minimum

able to unduly interfere with the host or other containers.

• Runtime Threat Detection and Response: A capability that detects

active threats against a containerized application in runtime and
automatically responds to it.

• Access Controls: Linux security modules, such as AppArmor or

SELinux, are used to enforce access controls.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 40

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

Image Provenance

especially in production environments. It is essential to avoid running

or tampered with in some way. For this reason, it is important to be able

to identify and verify the source of any container, including who built it
and exactly which version of the code it is running.

The gold standard for image provenance is Docker Content Trust. With
Docker Content Trust enabled, a digital signature is added to images
before they are pushed to the registry. When the image is pulled, Docker
Content Trust will verify the signature, thereby ensuring the image comes
from the correct organization and the contents of the image exactly
match the image that was pushed. This ensures attackers did not tamper
with the image, either in transit or when it was stored at the registry.

implementation of The Update Framework (TUF).

At the time of writing, Docker Content Trust is supported by Docker Hub,

Artifactory and the Docker Trusted Registry (currently experimental). It is
possible to setup the open source private registry with Docker Content
Trust, but this requires also standing up a Notary server (see these
instructions for more details).

In the absence of Docker Content Trust, it is still possible to verify image

provenance using digests, which are cryptographic hashes of the contents
of an image. When an image is pushed, the Docker client will return a
string (such as sha256:45b23dee09af...6677c5cb2) that represents the
digest of the image. This digest can then be used to pull the image.
Whenever an image is pulled in this manner, Docker will verify the digest

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 41

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

matches the image. Unlike tags, a digest will always point to exactly the
same image; any update to the image will result in the generation of a new
digest. The problem with using digests is organizations need to set up a
proprietary system for automatically extracting and distributing them.

trust information attached and should be considered the safest Hub

images. Discretion should be applied when using other images, but note
that “automated builds” are linked to the source code they are built from,
and should be considered more trustworthy than regular user images.
Organizations should consider building images from source themselves
rather than pulling from untrusted repositories. This situation is currently
changing somewhat with the emergence of Docker Store, which will provide
a trusted store for publishers along the same lines as Apple’s App Store.

Security Scanning

several companies. The basic idea is simple: take a Docker image and

vulnerabilities to produce a “bill of health” for the image. Based on this

information, organizations can then take action to mitigate vulnerabilities.

Atomic Scan from Red Hat, Bluemix

Vulnerability Advisor from IBM, Clair from CoreOS, Docker Security
Scanning from Docker Inc., Peekr from Aqua Security, and Twistlock Trust.
They vary widely in how they work, how they are accessed and how much

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 42

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

Some scanners, including Clair, will just interrogate the package manager

managers the scanner doesn’t recognize. In contrast, Docker Security

Scanning performs a binary-level analysis of images that works regardless
of the package manager and can also identify versions of statically-linked
libraries. Twistlock is also interesting in that it performs scanning on

vulnerability scanning, and works in air-gapped environments.

It is essential to consider how security scanning can be integrated into

your systems. Docker Security Scanning is available as an integrated part
of Docker Cloud and Docker Datacenter, but not as a stand-alone service.

scanners can be installed on-premises, which will be important to

boundaries.

be to have a blanket ban on running any images with vulnerabilities in

have some vulnerabilities and this isn’t a realistic option. For example,

vulnerability due to the version of Perl used (most other images have

to investigate discovered vulnerabilities individually to ascertain whether

or not they represent a real risk to your system.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 43

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

to do this is to use a very small base image, such as Alpine, which comes
in at only 5MB. Another, somewhat extreme, possibility is to build a
statically linked binary and copy it on top of the empty “scratch” image.
That way there are no OS level vulnerabilities at all. The major
disadvantage of this approach is that building and debugging become

Automated scanning is a huge move forward for security in our industry. It

quickly surfaces potential risks and places pressure on vendors to patch
vulnerable base images in a timely manner. By paying attention to the
results of scans and reacting quickly, organizations can stay one step
ahead of many would-be attackers.

Auditing
Auditing directly follows security scanning and image provenance. At any
point in time, we would like to be able to see which images are running in
production and which version of the code they are running. In particular, it
is important to identify containers running out-of-date, potentially
vulnerable images.

When working with containers, it is strongly recommended to follow what

is sometimes called a “golden image” approach: do not patch running
containers, but instead replace them with a new container running the
blue-green deployments and rolling upgrades can be
used to avoid downtime. With this approach, it is possible to audit large
numbers of running containers by looking at the image they were built

Note that it isn’t enough to scan images before they are deployed. As new
vulnerabilities are reported, images with a previous clean bill of health will

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 44

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

become known-vulnerable. Therefore, it is important to keep scanning all

images that are running in production. Depending on the scanning
solution used, this doesn’t necessarily involve an in-depth rescan;

reference this against new vulnerabilities.

It is still important to audit the hosts in a container-based system, but this

can be made easier by running a minimal distribution, such as CoreOS,
Red Hat Atomic or Ubuntu Snappy, which are designed to run containers

Docker Bench for Security

Isolation and Least Privilege

processes. In addition, cgroups are used to control the level of access to

resources such as CPU and RAM. Further, the Linux kernel calls that a
container can be controlled through Linux capabilities and seccomp. One
of the fundamental concepts in information security is the principle of
least privilege

“ Every program and privileged

user of the system should operate
using the least amount of privilege
necessary to complete the job.”
Jerome Saltzer

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 45

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

With reference to containers, this means that each container should run

Applying this principle makes an attacker’s life much harder; even if a

vulnerable feature, it cannot be exploited.

A large and easy win for security is to run containers with read-only

can be accommodated by using tmpfs

directories.

amount of memory available to a container will prevent attackers from

consuming all the memory on the host and starving out other running
services. Limiting CPU and network bandwidth can prevent attackers
from running resource-heavy processes such as Bitcoin mining or torrent
peers.

Perhaps the most common mistake when running containers in

production is having containers which run as the root user. While building

when the container starts should not run as root. If it does, any attacker
who compromises the process will have root-level privileges inside the
container. Much worse, as users are not namespaced by default, should
the attacker manage to break out of the container and onto the host, they
might be able to get full root-level privileges on the host.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 46

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

and switch to it before executing the main process. Since Docker 1.10,
there has been optional support for enabling user namespacing, which
automatically maps the user in a container to a high-numbered user on
the host. This works, but currently has several drawbacks, including

problems are being resolved upstream in the Linux community at

publication time, so expect that user namespace support will become
more viable for a larger percentage of use cases in the near future.

the attack surface, both by constraining what an attacker can do and

reducing exposure to vulnerabilities in the kernel code. The primary

around 40 capabilities, which map onto sets of kernel calls. Container

runtimes, including rkt and Docker, allow the user to select which
privileges a container should run with. These capabilities map onto
around 330 system calls, which means several capabilities, notably SYS_

which kernel calls are allowed, Docker now has seccomp support for
specifying exactly which calls can be used, and ships with a default
seccomp policy that has already shown to be at mitigating
problems in the Linux kernel. The main problem with both approaches is

untested parts of code.

Potentially, existing tools can be helpful to determine your application’s

use of syscalls without resorting to trial and error. If you are able to fully
exercise your application’s code paths with tracing capabilities, like

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 47

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

strace2elastic, this will provide a report of used syscalls within your

application during the container’s runtime.

While OS-level isolation and the enforcement of least privilege is critical,

isolation also needs to be tied to application logic. Without understanding
of the application that is running on the host, OS-level isolation may not

Runtime Threat Detection and Response

No matter how good a job you do with vulnerability scanning and
container hardening, there are always unknown bugs and vulnerabilities
that may manifest in the runtime and cause intrusions or compromises.

detection and incident response capabilities.

Containerized applications, compared to their monolithic counterparts,

are distinctly more minimal and immutable. This makes it possible to

traditional, monolithic applications. Using this baseline, you should be

able to detect real-time threats, anomalies, and active compromises, with
a lower false-positive rate than seen with traditional anomaly detection.

Behavior baselining, where a security mechanism focuses on

understanding an application or system’s typical behavior in order to
identify anomalies, was one of the hottest trends at Blackhat 2016. The

derivation of the baseline, the continuous monitoring, and the detection

and response. Today, most organizations accomplish behavior baselining
with a combination of manual labor and data science. However, due to the
transient nature of containers, it is especially important that the whole
process be automated.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 48

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

Active response goes hand-in-hand with baselining. Active response is

how to respond to an attack, a compromise or an anomaly as soon as it is

alerting responsible personnel, communicating with enterprise ticketing

systems, or applying some pre-determined corrective actions to the
system and the application.

In the container environment, an active response could mean performing

additional logging, applying additional isolation rules, disabling a user
dynamically, or even actively deleting the container. Again, automation is

in a negative way, such as getting the system in an inconsistent state or

interfering with non-idempotent operations.

detection and response include Aqua Security, Joyent Triton SmartOS and
Twistlock. As more mission-critical applications move to containers,
automating runtime threat detection and response will be increasingly
important to container security. The ability to correlate information,
analyze indicators of compromise, and manage forensics and response
actions, in an automated fashion, will be the only way to scale up runtime
security for a containerized world.

Access Controls
The Linux kernel has support for security modules that can apply policies
prior to the execution of kernel calls. The two most common security
modules are AppArmor and SELinux, both of which implement what is
known as mandatory access control (MAC). MAC will check that a user or
process has the rights to perform various actions, such as reading and

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 49

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

discretionary access control or DAC.

SELinux was originally developed by the National Security Agency (NSA),

but is now largely developed by Red Hat and found in their distributions.

to control their labels. AppArmor is similar, but less comprehensive than

SELinux, and doesn’t have the same control over volumes. It is enabled by
default on Debian and Ubuntu distributions.

In both cases, it is possible to create special policies for running particular

containers; e.g., a web server policy for running Apache or NGINX that
allows certain network operations but disallows various other calls.

creating such policies tends to be a frustrating exercise, eased slightly by

third party utilities such as bane. In the future, we can expect to see an

Further to the topic of access control, it’s important to note that anyone

upon execution) binaries that can be copied back to the host. In most
situations, this is just something to be aware of, but some organizations

may want to look at using higher-level platforms such as Docker

Twistlock, to add such controls.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 50

ASSESSING THE CURRENT STATE OF CONTAINER SECURITY

Conclusion
It is essential for organizations to consider security when implementing a

Image provenance starts with the developers building, pulling and

pushing images on their laptops, continues through the CI and testing
phases, and ends with the containers running in production.

Containers and the golden image approach enable new ways of working
and tooling, especially around image scanning and auditing.

production and can much more easily and quickly react to vulnerabilities.
Updated base images can be tested, integrated and deployed in minutes.
Image signing validates the authenticity of containers and ensures that
attackers have not tampered with their contents.

images become common and features, such as integrated security

be a strong reason for organizations to make the move to containers.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 51

A HISTORY OF SECURITY
IN CONTAINER
ADOPTION
In this discussion with Bryan Cantrill of Joyent, we
delve into a discussion surrounding the security of
containers at scale, how hardware visualization
impacts container security, and the ways in which
Joyent has contributed to the ongoing use of containers in
production. Cantrill mentions that the security issue for them is
especially focused on multi-tenant security with containers. You
can remove the problems around hardware virtualization entirely
by deploying multi-tenant environments on bare metal in Triton.
It’s not just a matter of what’s old is new again in the security
world, but a real and qualitative increase in the complexity and
velocity of systems. Listen on SoundCloud or Listen on YouTube

Bryan Cantrill

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 52

METHODS FOR DEALING
WITH CONTAINER
STORAGE
JANAKIRAM MSV

few years ago, when virtualization was introduced to IT adminis-

A trators, there was an attempt to standardize the virtual machine

(VM) as the unit of deployment. Each new build and version
resulted in a new VM template. Eventually, developers and administrators
started creating new images and provisioning VMs, which resulted in a
phenomenon called “VM sprawl.” Each template and the provisioned VM

management. Given the large size of VM images, organizations realized

that it wasn’t practical to create a new VM template for each version.

One of the goals for Docker was to avoid the pitfall illustrated by “VM
sprawl.” The only way Docker could avoid the trap of fragmented images

to create isolation between the host system and containers. This isolation
was core to the security of containerized applications. To meet these
requirements, Docker adopted a for the
images and containers.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 53

METHODS FOR DEALING WITH CONTAINER STORAGE

a branch, which becomes a separate layer. Docker images are based on a

images to be constructed and deconstructed as needed instead of

creating a large, monolithic image.

applications that can rapidly scale. Since they are self-contained, Docker

workloads, it becomes a challenge to run stateful applications that deal

with persistent data. Workloads, such as relational databases, NoSQL
databases, content management systems, and big data stacks, demand
persistence and durability of data. Some workloads, like content
management systems, also require shared data access across multiple
application instances.

When a Docker image is pulled from the registry, the engine downloads all
the dependent layers to the host. When a container is launched from a
downloaded image comprised of many layers, Docker uses the copy-on-
write

container process. When a Docker image is created from an existing

approach enables reuse of images without duplication or fragmentation.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 54

METHODS FOR DEALING WITH CONTAINER STORAGE

topmost working layer. All other processes using the original image’s
layers will continue to access the read-only, original version of the layer.
This technique optimizes both image disk space usage and the
performance of container start times.

Strategies to Manage Persistent Data

Docker’s layered storage implementation is designed for portability,

deleted, all of the data written to the container is deleted along with it.

As a best practice, it is recommended to isolate the data from a container

should be distinctly separate from the container lifecycle. There are

multiple strategies to add persistence to containers. We will evaluate the
options that are available out-of-the box with Docker, followed by the
scenarios that are enabled by the ecosystem.

Host-Based Persistence
Host-based persistence is one of the early implementations of data
durability in containers, which has matured to support multiple use cases.
In this architecture, containers depend on the underlying host for

stored within the directory is visible inside the container mount

namespace. The data is persisted outside of the container, which means it
will be available when a container is removed.

In host-based persistence, multiple containers can share one or more

volumes. In a scenario where multiple containers are writing to a single
shared volume, it can cause data corruption. Developers need to ensure

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 55

 METHODS FOR DEALING WITH CONTAINER STORAGE

that the applications are designed to write to shared data stores.

Data volumes are directly accessible from the Docker host. This means
you can read and write to them with normal Linux tools. In most cases
you should not do this, as it can cause data corruption if your containers
and applications are unaware of your direct access.

There are three ways of using host-based persistence, with subtle

Implicit Per-Container Storage

container that requested host-based persistence. The directory is created

the container. When the running container is removed, the directory is

FIG 1: -

Host-Based Persistence Per Container

Host

/
container.
var
lib
docker
volumes
233ran5d5o2m
Container
r9a3n8745dom
3ran506d87om /
content content
dir dir

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 56

Source: Janakiram MSV

 METHODS FOR DEALING WITH CONTAINER STORAGE

automatically deleted on the host by the Docker Engine. The directory

may also become unavailable if the Docker Engine crashes on the host.
The key thing to understand is that the data stored in the sandbox is not
available to other containers, except the one that requested it.

Explicit Shared Storage (Data Volumes)

We can choose the second technique if there is a need to share data
across multiple containers running on the same host. In this scenario, an

or more containers.

This becomes especially useful when multiple containers need read-write

access to the same directory. For example, containers running an Apache
web server can centrally store logs to the same directory, making it easier
to process the logs.

FIG 2: -

Host-Based Persistence Shared Among Containers

Host Container 1

/ /
opt web
web

Container 2

The directory on the host is Changes bypass the storage /

created outside of Docker backend and are made
Engine’s context. directly to data volume.
web

Data volumes are shared These applied changes will

across containers & are not not be included when the
deleted when containers are image updates.
NETWORKING,
removed. SECURITY & STORAGE WITH DOCKER & CONTAINERS 57

Source: Janakiram MSV

METHODS FOR DEALING WITH CONTAINER STORAGE

Since the directory on the host is created outside of Docker Engine’s

stopping Docker Engine. Since this shared mount point is fully outside the
control of Docker Engine’s storage backend, it is not part of the layered,

This technique is the most popular one used by DevOps teams. Referred
to as data volumes

• Data volumes can be shared and reused across multiple containers.

• Changes made to a data volume are made directly, bypassing the

engine’s storage backend image layers implementation.

• Changes applied to a data volume will not be included when the image
gets updated.

• Data volumes are available even if the container itself is deleted.

Shared Multi-Host Storage

containers nonportable. The data residing on the host will not move along
with the container, which creates a tight bind between the host and
container.

them in a clustered environment, where multiple hosts participate to

deliver required compute, network and storage capabilities. This scenario
demands distributed storage that is made available to all hosts, and is
then exposed to the containers through a consistent namespace.

Ceph, GlusterFS, Network File System (NFS)

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 58

 Multi-Host Persistence Shared Among Containers
METHODS FOR DEALING WITH CONTAINER STORAGE
Container 1 Container 2 Container 3

/ / /
web web web

Host 1 Host 2 Host 3

/ / /
opt opt opt
web web web

Distributed Filesystem
Source: Janakiram MSV

FIG 3: -

running Docker containers. By creating a consistent naming convention

underlying durable storage backend, irrespective of the host from which

they are deployed.

combined with the explicit storage technique. Since the mount point is
available on all nodes, it can be leveraged to create a shared mount point
among containers.

Containerized workloads running on orchestration engines deployed in

subset of cluster nodes. These nodes will be designated for scheduling

containers that need long-term durability and persistence.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 59

METHODS FOR DEALING WITH CONTAINER STORAGE

Orchestration engines provide a mechanism to specify hosts during the

container

running containers. In Kubernetes, labels can be used to target a set of

nodes when deploying pods. Kubernetes also utilizes Pet Sets, a group of
stateful pods that have a stronger requirement for identity.

Typical Operations Supported By Host-Based

Persistence
Thanks to tight integration with the core container engine, host-based

considering explicit shared storage and shared multi-host storage

• Create volumes:
containers. It results in the Docker Engine creating a designated

• Launching stateful containers: The persistent volumes are then

associated with one or more containers during launch time.

• Backing up data: Data stored in volumes can be easily backed up to

Docker documentation for guidance on this
process.

• Migrating and restoring data: The backed up data can then be

• Deleting volumes:
removing associated containers; they will need to be manually deleted
by the operations team.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 60

METHODS FOR DEALING WITH CONTAINER STORAGE

• In multi-host

multiple physical or virtual servers.

Top Use Cases for Using Host-based

Persistence
Host-based persistence may be considered for the following scenarios:

• Databases: It can be faster to write to a volume than the copy-on-

write layer. This is applicable when running relational and NoSQL
databases.

• Host-mounting source code: In a development environment where

source code needs to be shared between the host and containers,
host-based persistence comes in handy. Since the container accesses
the same version as the host, it’s easy to debug and test in a container

inside the container.

• Master-Worker: In a scenario where data needs to be shared with

two containers acting as master and worker, host-based persistence
should be used. For example, the data aggregated by the master
container is processed by a worker container.

Volume Plugins
Although host-based persistence is a valuable addition to Docker for

of specialized storage backends optimized for data-intensive workloads.

To solve these limitations, volume plugins have been added to Docker to
extend the capabilities of containers to a variety of storage backends,

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 61

METHODS FOR DEALING WITH CONTAINER STORAGE

without forcing changes to the application design or deployment

architecture.

Starting with version 1.8, Docker introduced support for third-party

volume plugins. Existing tools, including Docker command-line interface
(CLI), Compose and Swarm, work seamlessly with plugins. Developers can
even create custom plugins based on Docker’s
guidance.

According to Docker, volume plugins enable engine deployments to be

integrated with external storage systems and data volumes to persist
beyond the lifetime of a single engine host. Customers can start with the
default local driver that ships along with Docker, and move to a third-party

As of June 2016, Docker supports over a dozen third-party volume plugins

for use with Azure File Storage, Google Compute Engine persistent disks,
NetApp Storage and vSphere. In addition, projects like Rancher Convoy
can provide access to multiple backends at the same time.

Basics of Volume Plugin Architecture

Docker ships with a default driver that supports local, host-based

extended to support new backends. This architecture is based on

Docker’s philosophy of “batteries included, but replaceable.” The third-
party volume plugins are installed separately, which typically ship with
their own command line tools to manage the lifecycle of storage volumes.

Docker’s volume plugins can support multiple backend drivers that

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 62

 Docker Volume Plugin Architecture
METHODS FOR DEALING WITH CONTAINER STORAGE

Basic volume create Vendor-supported

and delete operations. snapshot and copy operations backend drivers

Docker Client Plugin Client Storage Backend 1

Storage Backend 2

Docker Daemon Plugin Daemon

Storage Backend 3

Storage Backend...

Source: Janakiram MSV

FIG 4: -
ers.

Typical Operations Supported By Volume

Plugins
Volume plugins typically install a daemon responsible for managing the
interaction with storage backends. A client in the form of a command-line

the volume. The operations supported by the CLI go beyond the standard
tasks that the Docker CLI can perform.

The volume plugin clients enable the following tasks as part of the
lifecycle management:

• Creating provisioned volumes: This step involves creating devices

that can be accessed while creating a volume from a standard Docker
CLI.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 63

 METHODS FOR DEALING WITH CONTAINER STORAGE

• Taking snapshot of volumes: Many plugins support creating point-

in-time snapshots of volumes. These incremental snapshots only
contain the delta of changes made since the last snapshot, thus
maintaining a small size.

• Backing up snapshots to external sources: Optionally, volume

plugin tools support backing up snapshots to sources such as Amazon
S3 and Azure Storage.

• Restoring volumes on any supported host: Plugins enable easy

migration of data from one host to another by restoring the backups
and snapshots.

Flocker from ClusterHQ

Docker. The Flocker data volume, called a dataset, is portable and can be

FIG 5:

Flocker: Apps and Data Flocking Together

Why no registry? Flocker Storage

Flocker currently uses a Unlike other orchestration
simple, custom service Master systems in this comparison,
local to the master to store Flocker is tightly integrated
cluster state. Integration Flocker with a shared storage backend.
with a distributed system Control Service The product uses this tight
(e.g. Zookeeper or etcd) integration to enable migration
is anticipated for a future of containers and their data.
release.

Host 1 Host 2 Host “n” Shared Storage

Backend
Flocker Agent Flocker Agent Flocker Agent

Containers Containers Containers

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 64

Source: The New Stack

METHODS FOR DEALING WITH CONTAINER STORAGE

used with any container within the cluster. It manages Docker containers
and data volumes together, enabling the volumes to follow the containers

Flocker works with mainstream orchestration engines such as Docker

Swarm, Kubernetes and Mesos. It supports storage environments ranging
from Amazon Elastic Block Store (EBS), GCE persistent disk, OpenStack
Cinder, vSAN, vSphere and more.

There are many open source volume plugins to support a variety of

storage backends. Please refer to Docker’s plugin page for the latest list of
available plugins.

Top Use Cases for Volume Plugins

Volume plugins target scenarios typically used in production

• Data-intensive applications: Since volume plugins have drivers for

specialized storage backends, they can deliver the required
performance demanded by data-intensive workloads such as big data
processing and video transcoding.

• Database migration: Volume plugins make it easy to move data

across hosts in the form of snapshots, which enable migration of
production databases from one host to another with minimum
downtime. Through this, containers in production environments can
be migrated to powerful hosts or virtual machines.

• Stateful application failover: Using volume plugins with a

supported shared storage backend like Amazon EBS, customers can
manually failover containers to a new machine and reattach an
existing data volume. This enables transparent failover of stateful
applications.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 65

METHODS FOR DEALING WITH CONTAINER STORAGE

• Reduced Mean Time Between Failures (MTBF): With volume

plugins connected via a shared storage backend, operations teams
can speed up cluster time-to-recovery by attaching a new database
container to an existing data volume. This results in faster recovery of
failed systems.

choices made available by this vibrant ecosystem.

Container Storage Ecosystem

Since storage is a key building block of the container infrastructure, many

storage providers. While there are a few dozen entities delivering storage
solutions from the container ecosystem, we will explore some of the
prominent players from each category.

The rise of containers in enterprise has led to the creation of a new class
of storage optimized for containerized workloads. Existing storage
technologies, such as network-attached storage (NAS) and storage area
network (SAN), are not designed to run containerized applications.

expose the virtual disks to the more modern applications.

commodity hardware, featuring a scale-out block storage, which in itself is

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 66

METHODS FOR DEALING WITH CONTAINER STORAGE

money by selling these services on top of commodities or bundled with a

containers is the ability to virtualize storage, which may be based on faster

solid-state drives (SSD) or magnetic disks. Aggregating disparate storage

on faster SSDs while moving the archival data to magnetic disks. This
delivers the right level of performance for workloads, such as online

operations per second (IOPS).

containers, with many of them selling appliances or Storage as a Service.

Portworx, Hedvig, CoreOS Torus, EMC libStorage, Joyent Manta and
Blockbridge
requiring them to buy something else. StorageOS, Robin Systems and
Quobyte are examples of companies that do not provide unbundled

Storage Appliance Providers

The virtualization of compute, storage and networking led to the evolution

started to ship appliances that delivered datacenters in a box. These

appliances came with bundled hypervisor, storage and networking

infrastructure.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 67

METHODS FOR DEALING WITH CONTAINER STORAGE

With containers becoming a popular choice, some startups are building

appliances that deliver end-to-end infrastructure for containerized
workloads. They have purpose-built converged infrastructure for
containers that comes with network interoperability and persistent
storage.

Diamanti is an early mover in the

space of container-based converged infrastructure. Its appliance comes

networking at the hardware level. Other companies, such as Datera

storage appliances as solutions for container use cases.

As containers become mainstream in the enterprise, we can expect to see

the rise of converged infrastructure for containerized workloads.
Containers solve the problem of assembling the right technology stack for
cloud-native applications and microservices.

Object and Block Storage Providers

object storage services, such as Amazon S3, IBM Bluemix Object Storage
and Joyent Manta, as well as block storage devices such as Elastic Block
Storage (EBS) or Google Compute Engine (GCE) persistent disks. To enable
easy integration with the infrastructure, these cloud providers are
investing in storage drivers and plugins that bring persistence to
containers. DevOps teams can host image registries in the public cloud
backed by object storage. Block storage devices deliver performance and
durability to workloads.

These investments in storage drivers and plugins by cloud providers are

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 68

METHODS FOR DEALING WITH CONTAINER STORAGE

primarily meant to run hosted container management services or

Docker announced native support for AWS and Azure. This will accelerate
the development and optimization of storage drivers for object and block
storage.

Summary
Storage is one of the key building blocks of a viable enterprise container
infrastructure. Though Docker made it easy to add persistence based on
data volumes, the ecosystem is taking it to the next level. Volume plugins
are a major step towards integrating containers with some of the latest
innovations in the storage industry. Providers are making it possible to tap
into the power of enterprise storage platforms.

poised to witness a huge growth with containers. The concept of mixing

as IBM FlashSystem arrays being used to power cognitive computing

services. This combination can use existing commodity hardware to build
storage pools consumed by cloud-native applications.

It’s only a matter of time before converged infrastructure embraces

containers to deliver turn-key infrastructure platforms optimized to run
complex workloads. The compute building block that relies on hypervisors

gearing up to ride the new wave of converged infrastructure powered by

containers.

Public cloud providers with robust storage infrastructure are getting ready

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 69

METHODS FOR DEALING WITH CONTAINER STORAGE

services will get dedicated drivers and plugins to maintain images in

cloud. Containers as a Service will also drive the demand for native drivers
for public cloud storage.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 70

SOFTWARE-BASED
NETWORKING AND
SECURITY
In this discussion with Hari Krishnan and Harmeet
Sahni of Nuage Networks, we talk about how to

the networking and security needs of containers.

SDN can help prevent infrastructure-related performance

that allows users to handle dynamic container environments.

Nuage Network’s Virtual Services Platform (VSP) provides these
capabilities in all types of cloud environments, featuring user intent-

more. Listen on SoundCloud or Listen on YouTube

Hari Krishnan

Harmeet Sahni,

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 71

IDENTIFYING
AND SOLVING ISSUES
IN CONTAINERIZED
PRODUCTION
ENVIRONMENTS
VIVEK JUNEJA

o you have an application that is composed around containers.

S You have lightweight base images, a centralized container

registry, and integration with the deployment and continuous

full scale on your hardware. For running a multitier application, you spent
time on using a service discovery mechanism for your application
containers. You have a logging mechanism that pulls out the information
from each container and ships them to a server to be indexed. Using a
monitoring tool that is well suited for this era when machines are dispos-
able, you see an aggregate of your monitoring data, giving you a view of
the data grouped around container roles. Everything falls nicely into place.

You’re ready to take this to the next level by connecting your pipeline to
production. The production environment is where the containers will see
the most entropy. Rolling containers into production requires that you
spend your time building a canary release system to implement a rolling
upgrade process. Every change travels neatly from the development
environment to your production environment, shutting down one

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 72

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

container at a time and replacing them with a brand new version of your
code. This is what usually comes to mind when we talk about adopting
containers at a high level.

However, to the true practitioner, this is the tip of the iceberg. Doing
everything mentioned earlier still does not guarantee a perfect
environment for your containers. There’s still potential to have your plans

containers. We’ll explore these issues around container networking,

storage and security.

Container Networking
Containers do not live in isolation; they need to connect with other
services. Containers need to be discoverable and available for connection

the goal is to reliably and quickly reach out to a destination container.

discovery. While networks change across development, testing and

production environments, service discovery remains consistent across
environments. This means that the service discovery mechanism must
remain common across the varied networks where containers are
deployed.

If you have just started using containers in production, there are some key
questions that need answering to help stabilize your approach:

•
scenario? Should you use a bridge, overlay, underlay or another
networking approach?

• How does service discovery integrate with the various container

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 73

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

• How do you monitor a container network and identify bottlenecks

with its performance?

• How do you visualize a network topology running across multiple

hosts?

• How do you secure container networks?

• How do you isolate networks when running containers belonging to

varied tenants on the same physical or virtual hosts?

We will address each of these concerns before moving on to other

misunderstood aspects of containers in production.

It is recommended to have the containers use the host network, instead of

a bridged network, if the services running in the container need to be
exposed to outside users. This is primarily because the bridged network
causes latency due to the virtual Ethernet (vEth) network connection.

cause of concern. To resolve that, the application service in the container

default port. For example, when running a Tomcat container for a Java
application, the server and Apache JServ Protocol (AJP) port numbers
could be supplied at runtime using the operating system (OS) environment
variables.

docker run -d -e SERVER_HTTP_PORT=8080 -e SERVER_

AJP_PORT=8005 tomcat:8

The environment (ENV) variables SERVER_HTTP_PORT and SERVER_

AJP_PORT

containers from binding to a consistent port on the host, and will allow

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 74

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

multiple instances of containers running the same image, at the same

time, on the same host.

The host network also prevents the constant change of iptables, which is
common with the bridge network. You would not want to do that change
in a production environment where iptables could be used for
. The bridged network is commonly used in development
and testing environments to allow multiple concurrent containers of the
same kind to run on a set of shared hosts. Port mappings are the way to
allow the bridged containers to be accessed from end users.

model for container networks that share the same IP address. This is
useful for grouping application services in containers that usually work
together.

Docker features overlay networks that enable easy creation of multi-host

networks with per-container internet protocol (IP) support. Other
solutions, like Calico, Flannel and Weave, can integrate with Docker as a
network plugin. This space is rapidly developing, so the advice is to test

them into production.

Service Discovery and Container Networking

Service discovery is usually an infrastructure concern; it allows
applications deployed as containers to transparently access each other
via means like domain name system (DNS). If the containers are deployed
on the host network in production, then a proxy must exist that is able to
route incoming requests to the containers. Previously, a service discovery
solution based on Consul and Registrator was an easy setup mechanism
for discoverable containers.

This has evolved, thanks to the introduction of overlay networks and an

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 75

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

array of third-party plugins like Calico and Weave. Implementation

requires use of a key-value store to coordinate network updates across
hosts. In some solutions, DNS is used as the basis of the service discovery.

discovery process in cases where containers are frequently changing or

moving across hosts. Other solutions include services, like HAProxy or

backends.

scale clusters with minimal applications. Managing the lifecycle of ports

allows IP per-container, use a service discovery mechanism that is

integrated into the network provider. This reduces the number of moving

service discovery.

Monitoring and Visualizing Container Networks

When dealing with containers in production, it is important to understand
how they interact with each other. This is vital to help diagnose issues and

ecosystem actively supports this requirement. For instance, Weave Scope

provides an overview of a containerized application’s interconnections
across a given set of hosts.

This is crucial to understanding how containers communicate between

each other and with other uncontained services. Container-native
monitoring tools, like Sysdig
across container instances. When the container count increases and

together.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 76

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

Monitoring and performance management in container networks is the

Ecosystem series, Monitoring & Management with Docker and Containers.

We’ll go more in-depth on this topic then, as the monitoring space itself is
complicated and extremely important for showing the value of containers
in production.

Isolating Networks

infrastructure, isolation needs to be created to protect the network

connection between related containers. Docker addresses this need with

are connected by a network that is separate from other tenants. Docker

also provides overlay networks that span across multiple hosts. There
could be any number of overlay networks, with each ideally being used for
a related container.

Container Storage
Containers have become an essential component of the continuous
delivery (CD) story for most organizations. Moreover, containers are the
best packaging means for microservices. Running containers in a

commonly used in Docker’s popular container runtime.

Some adoptees have their applications bundled as containers, but still

rely on local container storage or some form of host-mounted volume.
While this works on a small scale, this practice quickly runs out of steam
as it scales across tens, hundreds or thousands of hosts. Most container
advocates recommend steering away from ephemeral container storage
and towards moving state outside of the container.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 77

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

Lightweight kernels have emerged over the last few years, purpose-built
for the demands of applications. This rise is due to the fact that a
lightweight container image contributes to a faster deployment, which
then leads to rapid feedback loops for developers. Teams running

general housekeeping, which entails getting rid of old container volumes

from continuous deployments of new image versions. Regular clean-up
activity ensures the hosts never run out of storage space and optimizes

Similar to networking, we want to go over some key questions that need

answering before adopting storage for containers:

• How do you select the right persistent storage backend for containers?

• How do you reduce the size of the container images?

Addressing the Filesystem Drivers

Many drivers are available for use with Docker. Advanced multi-layered
AUFS) is very common; it is stable and has had

better to use a data volume from Docker.

OverlayFS
compared to AUFS; however, it is important to have this tested for a
certain period of time before moving to production, primarily due to its

the container runtime rkt also uses it in the new Linux kernels.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 78

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

If you choose to use Device Mapper, and are running multiple containers
on a host, it is preferred to use real block devices for data and metadata.

Persistent Storage for Containers

The ideal case for a containerized application is to have its state managed
completely outside the realms of its execution; this means the container is
leveraging an external data source for all of its state requirements.
However, you can run stateful services as containers. In that case, you
could either choose between having a container-managed volume or
mounting a directory from the host to the containers. You could also use a
Docker volume plugin, like Flocker, thus allowing the containers to access
shared storage. Utilizing plugins makes it much easier for containers to use
multiple hosts while still accessing a persistent block storage.

It is also possible to build data-only containers. However, that practice is

now discouraged for production environments because named volumes
provide better volume management as well as the ability to use other
drivers for volume storage. Data-only containers can create problems
when identifying the right data container to use, which becomes
problematic if you have multiple data containers in your production
environment.

issues when containerized applications access them. Then you need to

user inside the container. This could be painful to manage if the shared

outside the context of the containerized application. If the persistent store

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 79

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

changed by the containerized application, then it’s prudent to use named

volumes. If the change to the persistent data store is also made outside
the container, then it’s better to use a data volume that uses host-

the application inside the container accesses them.

Size of Container Images

The general rule of thumb is to keep the container image size as minimal
as possible. This helps to reduce the amount of data the container host
has to fetch when pulling the image from the repository. One way to
alleviate the problem of overgrown and bloated container images is to
avoid installing unnecessary packages, especially through the package
manager built inside the container. For container runtimes that use copy-

one line, and making them available as one layer when building container
images.

When using lightweight base images like Alpine for Docker, you need to
make sure they comply with apk package manager, which may have

package managers like Apt.

Retiring Old Container Images, Volumes and

Instances
One common issue when deploying containers is running out of disk

Most host housekeeping mechanisms need to watch out for old container
images that have been around for a while and do not need to be on the
host. If your continuous delivery pipeline has been active, putting out tens

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 80

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

or hundreds of changes a day in production, it is likely the old images are

piling up. Deleting the unused container images is one tactic available
through container runtime tooling. A container runtime like Docker
prevents deletion of container volumes when deleting the container
instances.

Data containers behave like ordinary containers and can be mistakenly

volume containers prevents mix-up with the ordinary application

containers. As for the actual named volumes, the removal of the container

reused across the lifecycle of multiple containers.

Container Security

was whether containers were secure. This concern became more

prominent when considering whether to use containers on a shared host
running varied tenants. With the granular permission models now
available in container runtimes, and with the isolated user, network and
process namespaces, the state of container security has greatly stabilized.
There are still some areas of debate and concern, however. Allowing
containers to use secrets like credentials or access keys is still in
contention.

Security concerns also impact the way container storage and networks
are implemented. Vulnerability scans and signed container images are
becoming well-known practices amongst developers. Container
marketplaces like Docker’s Hub and Store, have already implemented

understanding of these problems.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 81

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

Here are some key questions that need to be answered when considering
the security of container deployments in production environments:

• How do you propagate critical security patches to container images in

the production environment?

• How do you validate container images obtained from external

sources?

•
deploying them in production?

• How do you allow containers to have access to secure keys and

credentials without exposing them to the host or other colocated
containers?

• How do you prevent a compromised container from disturbing the

host or other colocated containers?

• How do you test for secure container deployments?

Integrating Security Patch to Container Base

Images
Containerized applications are composed of a base image and the

deployment. Usually, the base image is the one that remains less prone to
changes, while the application changes are constantly baked in through
continuous integration. When security changes are proposed in the form
of patches to the operating environment, the change is passed on to the
base image.

The base image is rebuilt with the planned change, and is then used as a
new base image to rebuild the application containers. It is important to
have a consistent image across all environments. A change in the base

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 82

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

image is like any other change and is passed to the production

environment using the continuous delivery pipeline.

Trusted Container Images and Deployment

Docker introduced Content Trust from version 1.8 onwards. It uses the
developer or publisher’s private key to sign the image before pushing the
image to the registry. When fetching the same image, the public key of the

enterprise-grade image storage solution called Docker Trusted Registry.

obtain the keys either from the local store or a remote metadata store.

Access Secrets from Containers

Secrets include any information that you would not want to leak out to
unauthorized agents, but are required by the container to access external
resources. Common ways to inject these secrets has been to use
environment variables or to bundle them in a build manifest.

unauthorized agents. Deploying containers on AWS that need secret keys

can take advantage of the identity and access management (IAM) roles
instead of passing keys insecurely. The EJSON library from Shopify allows
you to encrypt secret information, which can be committed to the source
code repository. Keywhiz from Square and Vault from HashiCorp also act
as key-value stores.

Separating Compromised Containers

A compromised container can potentially disarm the container host. One
way to address this is to enable SELinux and SVirt. SVirt works with Docker
to disallow container processes from getting access to the host system. A
detailed account has been shared by Project Atomic.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 83

IDENTIFYING AND SOLVING ISSUES IN CONTAINERIZED PRODUCTION ENV...

CoreOS has also integrated SVirt with the rkt runtime; it is available by
default whenever a new container is run. This is especially critical if the
developer uses an untrusted container image from the public web without

permissions to be revoked from a given container.

Secure Container Deployments

The Docker Bench program provides an automated tool that checks
against all the best practices as advocated in the Center for Internet
Security (CIS) Benchmark v1.11 report. This will help identify hotspots in
your infrastructure that need attention before putting services into
production. Another important step is to have a secure container host.
The Docker security deployment guidelines is a good resource to
strengthen your Docker container runtime environment.

Summary

use of containers in production environments is not possible without

giving an enormous amount of consideration to container networking,
storage and security. As more organizations practice this art, new patterns
and practices will emerge, making container deployments the de facto for
any applications built in the future. The goal is to rethink how security,
storage and networking will evolve when container count increases in
production, running not just tens or hundreds of nodes, but thousands at
a time.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 84

BUILDING THE
FOUNDATION OF
SECURE CONTAINERS
In this discussion with Nathan McCauley of Docker,

come to think about Docker security over the

years. McCauley explained that many early
security concerns stemmed from users not being familiar with the
technology at play. Docker has addressed these concerns in the
base platform over the years, aided by a movement in the DevOps
community towards embracing container technology and how it
can help them achieve their security goals. The discussion also
covers Docker 1.12’s native Swarm orchestration mode, and some
additional security features like cryptographic node identity.
Listen on SoundCloud or Listen on YouTube

Nathan McCauley

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 85

CHAPTER #: CHAPTER TITLE GOES HERE, IF TOO LONG THEN...

NETWORKING, SECURITY
& STORAGE DIRECTORY

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 86

NETWORKING
Product/Project (Company or Supporting Org.) Sub-category

Avi Vantage Platform (Avi Networks) Overlays and Virtual Networking Tools

A software-based application delivery solution that integrates with container-based environments to provide

Aviatrix (Aviatrix)

Big Cloud Fabric (Big Switch Networks)

Bluemix Virtual Private Network (VPN) (IBM)

on-premises data center and An

Open Source Canal (Tigera) Overlays and Virtual Networking Tools

Open Source Cisco Application Centric Infrastructure (Cisco) Overlays and Virtual Networking Tools

Open Source Container Network Interface (CNI) (N/A)

Open Source Contiv Network (Cisco)

container-based

Diamanti (Diamanti)

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 87

CONT’D: NETWORKING

Product/Project (Company or Supporting Org.) Sub-category

A purpose-built container infrastructure that addresses the challenges of deploying containers to production

Open Source Flannel (CoreOS) Overlays and Virtual Networking Tools

Joyent Triton (Joyent)

DataCenter and SmartOS functionality

Open Source Joyent Triton DataCenter (Joyent)

Triton DataCenter converges container orchestration

Open Source Joyent Triton SmartOS (Joyent)

A lightweight container hypervisor that delivers

Open Source Kuryr (OpenStack Foundation) Overlays and Virtual Networking Tools

Open Source libnetwork (Docker) Overlays and Virtual Networking Tools

libnetwork provides a

Nuage Networks VSP (Virtualized Services Platform) Overlays and Virtual Networking Tools
(Nokia)

overlay for

Open Source OpenContrail (Juniper Networks)

PLUMgrid Open Networking Suite (PLUMgrid) Overlays and Virtual Networking Tools

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 88

CONT’D: NETWORKING

Product/Project (Company or Supporting Org.) Sub-category

Open Source Project Calico (Tigera) Overlays and Virtual Networking Tools

Open Source Project Skyhook (Aviatrix)

Robin Systems (Robin Systems)

Open Source Romana (N/A) Overlays and Virtual Networking Tools

Open Source VPNKit (Docker) Overlays and Virtual Networking Tools

A set of tools and services

Weave Net (Weaveworks) Overlays and Virtual Networking Tools

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 89

SECURITY
Product/Project (Company or Supporting Org.) Sub-category:

Amazon EC2 Container Registry (ECR) (Amazon Web

Services)

Open Source Anchore (Anchore)

Apcera Platform (Apcera) Policy/Compliance Management

Aqua Container Security Platform (Aqua Security

Software)

Aqua Peekr (Aqua Security Software)

Open Source Atomic Registry (Red Hat)

Open Source Atomic Scan (Red Hat)

Aviatrix (Aviatrix) Policy/Compliance Management

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 90

CONT’D: SECURITY

Product/Project (Company or Supporting Org.) Sub-category:

BanyanOps (BanyanOps)

Bluemix (IBM)

Select your prefered managed platform for your applications from

Open Source Canal (Tigera) Policy/Compliance Management

Open Source Cisco Application Centric Infrastructure (Cisco) Policy/Compliance Management

Open Source Clair (CoreOS)

A container vulnerability analysis service providing static analysis of vulnerabilities in appc and Docker

CloudPassage Halo (CloudPassage)

Conjur (Conjur)

Open Source Docker Bench for Security (Docker)

The Docker Bench for Security is a script around deploying

Docker containers

Docker Cloud (Docker)

Docker Cloud includes Docker Security

which reviews images in private repositories to verify that they are free from known security

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 91

CONT’D: SECURITY

Product/Project (Company or Supporting Org.) Sub-category:

Docker Datacenter (Docker)

Provides on-premises container management and deployment services to enterprises with a production-ready
platform supported by Docker and

Open Source Docker Distribution (Docker)

Open Source Docker Engine (Docker)

Docker Content Trust is a feature that makes it possible to verify the publisher

Docker Store (Docker)

and Docker Store-curated content

Docker Trusted Registry (Docker)

Allows users to store and manage Docker images on premises or

Open Source Dockyard (N/A)

Enterprise Registry (CoreOS)

FlawCheck Private Registry (FlawCheck)

FlawCheck Private Registry Enterprise (FlawCheck)

Google Container Registry (Google)

Open Source Harbor (VMware)

Illumio Adaptive Security Platform (ASP) (Illumio)

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 92

CONT’D: SECURITY

Product/Project (Company or Supporting Org.) Sub-category:

ASP is a distributed software platform designed to continuously protect communications within and

Open Source Joyent Triton SmartOS (Joyent)

A lightweight container hypervisor that delivers

Open Source Joyent Triton DataCenter (Joyent)

Triton DataCenter converges container orchestration

Open Source Notary (Docker)

Open Source OpenSCAP (Red Hat)

The OpenSCAP Base is both a library and a command line tool which can be used to parse and evaluate each

Polyverse (Polyverse)

Open Source Portus (SUSE)

Private Image Registry Service (IBM)

The
private registry supports group access policies to

Open Source Project Calico (Tigera) Policy/Compliance Management

Open Source Project Skyhook (Aviatrix) Policy/Compliance Management

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 93

CONT’D: SECURITY

Product/Project (Company or Supporting Org.) Sub-category:

Quay Enterprise (CoreOS)

Quay.io (CoreOS)

Open Source Registrator (Glider Labs)

Twistlock Runtime (Twistlock) Policy/Compliance Management

detects compromises and

Twistlock Trust (Twistlock)

Scans images and registries to

Vulnerability Advisor (IBM)

the latest
providing insight to the security posture before instantiating a live

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 94

STORAGE
Product/Project (Company or Supporting Org.) Sub-category:

Acropolis (Nutanix)

Bluemix (IBM)

Select your prefered managed platform for your applications from

Bluemix Object Storage (IBM)

to

Open Source Ceph (Red Hat)

ClusterHQ Volume Hub (ClusterHQ) Management or Data Volume

Open Source Contiv Storage (Cisco)

Open Source Convoy (Rancher Labs) Management or Data Volume

Open Source Crate (Crate.io)

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 95

CONT’D: STORAGE

Product/Project (Company or Supporting Org.) Sub-category:

Datera Elastic Data Fabric (Datera)

Diamanti (Diamanti)

A purpose-built container infrastructure that addresses the challenges of deploying containers to production

Open Source dvol (ClusterHQ) Management or Data Volume

Open Source Flocker (ClusterHQ) Management or Data Volume

Hedvig Distributed Storage Platform (Hedvig)

IBM Cloud Object Storage (IBM)

Joyent Triton (Joyent)

DataCenter and SmartOS functionality

Open Source Joyent Triton SmartOS (Joyent)

A lightweight container hypervisor that delivers

Open Source Kubernetes (Cloud Native Computing Foundation)

Open Source libstorage (EMC)

Open Source Manta (Joyent)

Manta is an
Manta

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 96

CONT’D: STORAGE

Product/Project (Company or Supporting Org.) Sub-category:

Open Source Pachyderm (Pachyderm)

Open Source Polly (EMC) Management or Data Volume

Open Source Portworx PX-Developer (Portworx)

Portworx PX-Enterprise (Portworx)

Quobyte (Quobyte)

Open Source REX-Ray (EMC)

Robin Systems (Robin Systems)

StorageOS (StorageOS)

provides enterprise storage array functionality delivered via software on a pay-as-you-go basis

Open Source Torus (CoreOS) Management or Data Volume

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 97

DISCLOSURES
The following companies mentioned in this ebook are sponsors of The
New Stack: Apcera, Arcadia Data, Bitnami, Capital One, CloudFabrix, Cloud

CoreOS, DigitalOcean, Hewlett Packard Enterprise, Intel, Iron.io,

Mesosphere, New Relic, Red Hat, Sysdig, Weaveworks.

NETWORKING, SECURITY & STORAGE WITH DOCKER & CONTAINERS 98

thenewstack.io