Part III Automation and Orchestration

Chapter 10 Orchestration: Automated Replication And Parallelism

Introduction

• The previous chapter points out that the reason the vast number of automated tools and technologies have been
developed is due to the scale and complexity of a cloud data center.

• This chapter considers the topic of orchestration in which an automated system configures, controls, and manages
all aspects of a service.

• A later chapter explains the microservices programming paradigm that orchestration enables.

The Legacy Of Automating Manual Procedures

• The easiest way to build an automation tool consists of starting with the manual processes humans use and
looking for ways to have software handle some steps automatically.

• Automating a manual process can lead to a clumsy, inefficient solution that mimics human actions instead
of taking an entirely new approach.

Orchestration: Automation With A Larger Scope

• In the case of cloud data centers, early automation systems derived from manual processes led to separation of
functionality, with some tools helping deploy virtualized servers, others handling network configuration, etc.

• The advent of containers required designers find a way to build more comprehensive systems that cross
functional boundaries and handle all aspects needed for a service. Particularly:

o Rapid creation - low-overhead of containers means that it takes significantly less time to create a
container than to create a VM.

o Short lifetime - Unlike a VM that remains in place semi-permanently once created, a container is
ephemeral. A typical container is created when needed, performs one application task, and then exits.

o Replication – Replication is key for containers. When demand increases for a particular service,
multiple containers for the service can be created and run simultaneously. When demand declines,
unneeded container replicas can be terminated.

• Rapid creation and termination of containers requires an automated system. In addition, the network (and
possibly a storage system) must be configured quickly when a container is created or terminated.

• An automated container management system needs a larger scope that includes communication and storage.
Industry uses the term orchestration to refer to an automated system that coordinates the many subsystems
needed to configure, deploy, operate, and monitor software systems and services.

• Unlike an automation tool that focuses on one aspect of data center operations, an orchestration system
coordinates all the subsystems needed to operate a service, including deploying containers and configuring both
network communication and storage.
 • In addition to automated configuration and deployment, a container orchestrator usually handles three key
aspects of system management:
o Dynamic scaling of services - When demand increases, the orchestrator automatically increases the
number of simultaneous copies. When demand decreases, the orchestrator reduces the number of
copies, either by allowing copies to exit without replacing them or by terminating idle copies.
o Coordination across multiple servers - An orchestrator deploys copies of a container on multiple
physical servers and then monitors performance and balances the load by starting new copies on lightly
loaded servers.
o Resilience and automatic recovery - If a container fails or the containers providing a service become
unreachable, the orchestrator can either restart the failed containers or switch over to a backup set,
thereby guaranteeing that the service remains available at all times.

Kubernetes: An Example Container Orchestration System

• Kubernetes (abbreviated K8s), which was developed at Google and later moved to open source, is a very popular
technology that manages many aspects of running a service. Below are seven key features of Kubernetes:

o Service naming and discovery - Kubernetes allows a service to be accessed through a domain name or
an IP address. Typically, names and addresses are configured to be global, allowing applications running
outside the data center to access the service.

o Load balancing - Kubernetes does not limit a service to a single container. Instead, if traffic is high,
Kubernetes can automatically create multiple copies of the container for a service and use a load balancer
to divide incoming requests among the copies.

o Storage orchestration - Kubernetes allows an operator to mount remote storage automatically when a
container runs. The system can accommodate many types of storage, including local storage and
storage from a public cloud provider.

o Optimized container placement - When creating a service, an operator specifies a cluster of servers
(called nodes) that Kubernetes can use to run containers for the service. The operator specifies the
processor and memory (RAM) that each container will need. Kubernetes places containers on nodes in
the cluster in a way that optimizes the use of servers.

o Automated Recovery -After creating a container, Kubernetes does not make the container available to
clients until the container is running and ready to provide service. Kubernetes automatically replaces a
container that fails and terminates a container that stops responding to a user-defined health check.

o Management of configurations and secrets - Kubernetes separates management information from

container images, allowing users to change the information needed for configuration and management
without rebuilding container images. In addition to storing conventional configuration information, such
as network and storage configurations, Kubernetes allows one to store sensitive information, such as
passwords, authentication tokens, and encryption keys.

o Automated rollouts and rollbacks - Kubernetes allows an operator to roll out a new version of a service
at a specified rate by creating a new version of a container image and telling Kubernetes to start replacing
running containers with the new version. More important, Kubernetes allows each new container to
inherit all the resources the old container owned.
Limits On Kubernetes Scope
• Kubernetes does not handle all aspects of deploying and managing containerized software. Specifically:

o Kubernetes does not focus on a specific application type

▪ Kubernetes is application agnostic in the sense that it can be used for an arbitrary type of
application and an arbitrary programming paradigm. For example, it can be used to deploy
stateful or stateless applications. It can be used with serverless or traditional client-server
paradigms and can support arbitrary workloads. Hence, Kubernetes does not include special
facilities for any specific type.

o Does not manage source code or build containers

▪ Kubernetes relies on conventional container technology to handle source code and build
containers such as Docker containers. Thus, when it deploys a container on a node, we will
assume the node runs Linux along with the Docker software needed to run a container (i.e.,
dockerd).

o Does not supply event-passing middleware

▪ One programming paradigm used with containers passes asynchronous events among
containers. Kubernetes does not supply the middleware needed to pass events, but instead
assumes a programmer will use an extant middleware facility.

o Does not handle monitoring or event logging

▪ Kubernetes monitors and controls containers, but does not provide built-in mechanisms to
collect, log, or report measurements.

The Kubernetes Cluster Model

• Kubernetes provides container orchestration, which means it automates the deployment and operation of a set
of one or more containers to provide a computation service.

• Kubernetes uses the term cluster to describe the set of containers plus the associated support software used to
create, operate, and access the containers. The number of containers in a cluster depends on demand, and
Kubernetes can increase or decrease the number as needed.

• Software in a cluster can be divided into two conceptual categories: one category contains software invoked by
the owner of the cluster to create and operate containers. The other category contains software invoked by users
of the cluster to obtain access to a container.

Figure 10.2 The conceptual model for Kubernetes in which an owner runs software to create and
operate a set of containers, and the containers provide a computation service that users
access.
 • Although the terms owner and user seem to refer to humans, the roles do not have to be filled by entering
commands manually. Each of the two categories of software provides multiple APIs. Although a human can
indeed interact with the software through a command-line interface, apps can also perform the functions (e.g.,
through a RESTful interface). In the case of a container, a user typically runs an app that uses the traditional
client-server paradigm to communicate with a container.

Kubernetes Pods
• Although many applications consist of a single container, some programming paradigms, such as the
microservices paradigm, encourage a software engineer to divide an application into small autonomous pieces
that each run as a container.

• The pieces are tightly-coupled which means they are designed to work together, usually by communicating over
the network.

• To run a copy of a multicontainer application, all containers for the application must be started, and the network
must be configured to provide communication.

• Kubernetes uses the term pod to refer to an application. Thus, a pod can consist of a single container or multiple
containers; single container pods are typical.

• A pod defines the smallest unit of work that Kubernetes can deploy. When it deploys an instance of a pod,
Kubernetes places all containers for the pod on the same node.

• In terms of networking, Kubernetes assigns an IP address to each running pod. If a pod has multiple containers,
all containers in the pod share the IP address.

• Communication among containers in the pod occurs over the localhost network interface, just as if the containers
in the pod were processes running on a single computer.

• The use of a single IP address for all containers means a programmer must be careful to avoid multiple containers
attempting to use the same transport layer port. For example, if one of the containers in a pod uses port 80
for a web server, none of the other containers will be allowed to allocate port 80.

Pod Creation, Templates, And Binding Times

• To create a pod, Kubernetes uses a late binding approach in which a programmer creates a template for the pod
(sometimes called a pod manifest) that specifies items to use when running the pod. A template assigns a
name to the pod, specifies which container or containers to run, lists the network ports the pod will use, and
specifies the version of the Kubernetes API to use with the pod. A template can use yaml or json formats.
For example:

apiVersion: v1 kind: Pod metadata:

name: example-pod labels:
purpose: web-server spec:
containers:
- name: example-pod image: f34cd9527ae6 ports:
- containerPort: 8080

• When it deploys a pod on a node, Kubernetes stores information from the template with the running pod. Any
changes to a template apply to any new pods that are created from the template but have no effect on already
running pods.
 • The metadata specifications in the example are kept with each running pod, including labels that contain
information useful for humans. Tools allow humans to search for running pods with a label.

• A template can specify many more items such as a set of environment variables and initial values to be passed
to containers in the pod, specify external storage mount points for each container and an initial command to
run when the container executes.

Init Containers
• Kubernetes allows a designer to include one or more init containers.

• An init container is intended to perform initialization that might be needed before the main containers run so all
init containers in a pod must complete successfully before the main containers are started.

• An init container is useful for testing the environment to ensure that the facilities the main containers need
are available.

• A pod designer uses a set of init containers to check the environment before the main containers of the
pod execute. Doing so guarantees that either all the main containers in a pod start successfully or none
of them start.

Kubernetes Terminology: Nodes And Control Plane

• Kubernetes uses a broad definition of a node to be a computational engine on which software runs. A node
can be a physical server but is likely to be a virtual machine. When an owner creates a cluster, the owner specifies
a set of nodes that can be used to run the cluster, and Kubernetes spreads containers from the cluster across the
nodes.

• Kubernetes uses the term Control Plane (also known by the term master) to refer to the software components
that an owner uses to create and operate containers. When the control plane software runs on a node, the node
is known as a master node.

• A node that Kubernetes uses to run containers is Kubernetes node (worker node). Some sources use the term
worker node, which helps clarify the meaning. We will use the terms master node and worker node to make
the purpose of each node explicit.

• A typical Kubernetes cluster runs a single copy of the control plane components, and all control-plane
components run on a single master node.

• It is possible to create a cluster by running multiple copies of the control plane software on multiple master
nodes, but this text assumes that in their all the control plane software components run on a single master node.

Control Plane Software Components

• The Kubernetes control plane software consists of five main software components:
o API Server (kube-api-server)- The API server provides an interface to all the control plane
components. Whenever a human or a management app needs to interrogate or control a cluster,
communication goes through the API server.

o Scheduler (kube-scheduler) - The Scheduler handles the assignment of pods to nodes in the cluster.
It watches for a pod that is ready to run but has not yet been assigned to a node. It then chooses a
node on which to run the pod and binds the pod to the node. The Scheduler remains executing after
the initial deployment of pods, which allows Kubernetes to increase the number of pods in the cluster
dynamically.
 o Cluster State Store - The Cluster State Store holds information about the cluster, including the set of
nodes available to the cluster, the pods that are running, and the nodes on which the pods are currently
running. When something changes in the cluster, the Cluster Store must be updated. Kubernetes uses
the etcd key-value technology as the underlying storage technology. Etcd is a reliable distributed
system that keeps multiple copies of each key-value pair and uses a consensus algorithm to ensure that
a value can be retrieved even if one copy is damaged or lost.

o Controller Manager (kube-controller-manager) - The Controller Manager component consists of a

daemon that remains running while the cluster exists. The daemon uses a synchronous paradigm to run
a set of control loops that are sometimes called controllers. Kubernetes includes useful controllers,
such as a Replication Controller, Endpoints Controller, and Namespace Controller. In addition to
running controllers, the Controller Manager performs housekeeping functions such as garbage collection
of events, terminated pods, and unused nodes.

o Cloud Controller Manager - The Cloud Controller Manager provides an interface to the underlying
cloud provider system. Such an interface allows Kubernetes to request changes and to probe the
underlying cloud system when errors occur. The interface also allows Kubernetes to check the status of
hardware.

• In addition to the five main software components listed above, Kubernetes offers a command-line app that
allows a human to connect to the cluster and enter management commands that operate the cluster. The
command-line app, which is named kubectl, connects to the API server.

Communication Among Control Plane Components

• Control plane components communicate with one another, and some components communicate with outside
endpoints. All internal and external communication goes through the API server with the exception of the Cloud
Controller Manager, which uses a provider-specific protocol to communicate with the cloud system.

Figure 10.5 Control plane components on a master node and the communication among them.

Worker Node Software Components

• A pod is merely an environment, so each worker node runs software components to manage the containers that
comprise each pod. These components include:

o Service Proxy (kube-proxy) - Responsible for configuring network forwarding on the node to provide
network connectivity for the pods running on the node. Specifically, the Service Proxy configures the
Linux iptables facility.
 o Kubelet - Provides the interface between the control plane and the worker node. It contacts the API
server, watches the set of pods bound to the node, and handles the details of running the pods on the
node. Kubelet sets up the environment to ensure that each pod is isolated from other pods, and interfaces
with the Container Runtime system to run and monitor containers and it also monitors the pods running
on the node and reports their status back to the API server. Kubelet includes a copy of the cAdvisor
software that collects and summarizes statistics about pods and then exports the summary through a
Summary API, making them available to monitoring software (e.g., Metrics Server).

o Container Runtime - Kubernetes does not include a Container Runtime system. Instead, it uses a
conventional container technology and assumes that each node runs conventional Container Runtime
software. When Kubernetes needs to deploy containers, Kubelet interacts with the Container Runtime
system to perform the required task.

Figure 10.7 Software components on a worker node and the communication among them.

Kubernetes Features
• Kubernetes contains many additional features and facilities, some of which are quite sophisticated and complex.
The following highlights some of the most significant.
o Replicas - When deploying a cluster, an owner can specify and control the number of replicas for the
pod. In essence, an owner can explicitly control how an application scales out to handle higher load.

o Deployments - Kubernetes uses the term Deployment to refer to a specific technology that offers an
automated technology for scale out. Like other Kubernetes facilities, Deployments follow the intent-
based approach. An owner specifies the desired number of replicas and the Deployment system
maintains the required number of replicas automatically.

o StatefulSet - The StatefulSets facility allows an owner to create and manage stateful applications. A
user can deploy a set of pods with guaranteed ordering. Each pod in the set is assigned a unique identity,
and the system guarantees that the identity will persist.

o DaemonSet - The DaemonSet facility allows an owner to run a copy of a pod on all nodes in a cluster
(or some of them). As the name implies, the facility is typically used to deploy a permanently-running
daemon process that other containers on the node can use.

o Garbage Collection - Kubernetes objects have dependencies. For example, a Replicaset owns a set of
pods. When the owner of an object terminates, the object should also be terminated (e.g., collected).
The Garbage Collection facility allows one to set dependencies and specify how and when terminated
objects should be collected.

o TTL Controller for Finished Resources - The TTL Controller allows an owner to specify a maximum
time after an object has finished (either terminates normally or fails) before the object must be
collected. The value is known as a Time-To-Live (TTL), which leads to the name of the facility.
 o Job - The job facility creates a specified number of pods, and monitors their progress. When a specified
number of the pods complete, the Job facility terminates the others. If all pods exit before the required
number has been reached, the Job facility restarts the set. As an example, the facility can be used to
guarantee that one pod runs to completion.

o CronJob - The CronJob facility allows an owner to schedule a job to be run periodically (e.g., every
hour, every night, every weekend, or once a month). The idea and the name are derived from the Unix
cron program, and the CronJob facility uses the same specification format as cron.

o Services - The Services facility allows an owner to create a set of pods and specify an access policy
for the set. In essence, the Services facility hides the details of individual pods and passes each request
to one of the pods automatically. Decoupling a Service from the underlying pods allows pods to
exit and restart without affecting any apps that use the service. The facility works well for a
microservices architecture.

• Kubernetes offers many facilities, and often allows one to choose between a mechanism that offers
explicit control and a mechanism that handles the task automatically.

Summary
• An orchestration system handles all the data center subsystems needed to deploy a service, including the network,
storage facilities, and container execution.

• Orchestration typically handles dynamic scaling, coordination across multiple physical servers, and recovery.

• Kubernetes provides the primary example of an orchestration technology. Kubernetes handles service naming
and discovery, load balancing, storage, container placement, automated container restart, management of
configurations, and automated rollouts.

• Kubernetes uses a cluster model in which each cluster consists of a master node that runs control plane
software and worker nodes that run pods. Each pod represents an application, and may consist of one or more
containers that work together on one application.

• The owner of a service uses the Kubernetes control plane to deploy and control the service, External management
apps also communicate with the API server; the kubectl app provides a command-line interface.

• Each worker node runs a Service Proxy that configures network communication among pods on the node and
Kubelet that runs and monitors pods.

• Kubernetes contains many facilities that handle the tasks of deploying replicas, managing stateful applications,
running a background daemon on each node, controlling the collection of terminated objects, guaranteeing a job
runs to completion, running jobs periodically, and offering a service that can scale out automatically.