Kubernetes for

Operators
K U B E R N E T E S F O R O P E R AT O R S

Table of Contents
Containers and Kubernetes: Why Operators Should Care 3 Which Flavor of Kubernetes Should You Choose? 12

Who Should Read This Book? 3 Managed Kubernetes 12

Why Containers? 4 Kubernetes Distributions 12

A Kubernetes Primer 4 Do It Yourself 12

Kubernetes Controllers 5 Avoiding Missteps 13

How Kubernetes Help Operators 5 What Should Do Next? 15

Two Elements of Kubernetes that All Operators Should Know About 6

Operators Tailor Kubernetes to Application Needs 6

Cluster API Streamlines Kubernetes Provisioning and Management 6

First Principles for Operators 7

Top Six Questions to Answer Upfront 8

Question 1: Where should I put my Kubernetes cluster? 8

Question 2: How should I build my Kubernetes cluster? 8

Question 3: How many clusters should I build? 9

Question 4: What about underlying infrastructure? 9

Question 5: What about security? 9

Question 6: How can I extend Kubernetes to address specific 10

operational needs?

2
K U B E R N E T E S F O R O P E R AT O R S

Containers and Kubernetes: Run an application on almost any infrastructure—on-premises or in

the cloud

Why Operators Should Care Move applications easily between environments in response to need
If you’re part of an operations team, you know that your company’s success or cost
depends on your team’s ability to operate digital services reliably at scale.
Enterprises are increasingly turning to cloud native technologies, including Cloud native technologies are new and evolving fast. A whole ecosystem of
containers and Kubernetes, to achieve this goal. solutions and services is emerging to address a wide variety of use cases
and needs. There’s a lot to learn. This eBook will help you map your
If you are contemplating this transition, you are likely under pressure from
company’s journey to containers and Kubernetes, including important
above and below. Executives are asking you to deliver more reliable services
questions your team should ask itself and an exploration of the most
at a lower price point, while development teams want containerized
common missteps.
infrastructure on which to create and deliver new applications.

In the traditional approach, your team has to manage a large number of

dependencies with custom configurations for each application. Kubernetes Who Should Read This Book?
is a way to commoditize the way you deliver infrastructure, simplifying
management and leveling the playing field for operations of all sizes. No two organizations are alike, and titles can vary widely from one
Containers enable much greater application portability. to the next. However, this book is targeted to people that focus on
platform and infrastructure operations.
For operators, containers and Kubernetes have many benefits:

Eliminate the need to manage application dependencies at the Infrastructure engineers, systems engineers, and site reliability
infrastructure level engineers (SREs)—anyone responsible for the infrastructure on which
Kubernetes will run—can benefit.

Deliver infrastructure in a more cloud-like way

3
K U B E R N E T E S F O R O P E R AT O R S

Why Containers? database with some interesting features layered on top of it. Kubernetes uses
a set of controllers that each implement specific capabilities and work
together to produce the end result. Kubernetes controllers can be ripped out
One of the big challenges that operations teams face is the complexity of
and replaced to extend the system and adapt it to new requirements and
managing highly custom applications and the difficulty of moving applications
environments.
from one environment to another. By encapsulating all of an application’s
dependencies, containers make applications much more portable—and The diagram below shows the parts of a typical Kubernetes system. The core
therefore make an operator’s job simpler. A container can move from of the system is the database, etcd. The state of the cluster is stored there
development to QA to production—or from one cloud environment to (and only there). In front of etcd is the API Server. Nothing else in Kubernetes
another—without requiring any changes to the container—and with no talks to etcd directly. The API Server exposes a RESTful interface and
hardware and software reconfigurations in the target environment. provides the services necessary in a distributed system.
While many organizations are moving existing legacy applications into ALL interactions with Kubernetes are mediated via APIs. This approach
containers more or less as is, new application development makes use of can be a big change for both operations teams and developers.
containers in conjunction with a microservices architecture that breaks down
an application into component services. From an operational standpoint, there
are two critical things to understand about the microservices approach: Containers encapsulate an application in a form that’s
applications scale out instead of scaling up, and there are many more portable and easy to deploy. Containers can run on any
application components to manage compatible system—in any cloud—without changes.
Containers consume resources efficiently, enabling high
density and utilization.
A Kubernetes Primer
Kubernetes makes it possible to deploy and run complex
Container environments change more rapidly than VM environments. Having applications requiring multiple containers by clustering
a way to manage containerized applications effectively is an essential physical or virtual resources for application hosting.
element of cloud native and microservices architecture. Kubernetes has Kubernetes is extensible, self-healing, scales applications
emerged as the leading solution for orchestrating and managing automatically, and is inherently multi-cloud.
containerized applications.
Microservices architecture breaks down an application into
The components of Kubernetes “play off” each other to coordinate activities multiple component services, enabling greater parallelism
and react to events like musicians playing jazz. At its core, Kubernetes is a during both development and execution.

4
K U B E R N E T E S F O R O P E R AT O R S

make the state of the world match its configuration. If a controller can’t fully
achieve the desired state, it retries. Controllers are both patient and diligent,
resulting in a very stable distributed system pattern that is self-healing. If
MASTER NODE something goes wrong, a controller will work to fix it. If the desired state
changes while a controller is working, it changes course and works toward the
etcd API Server Kublet Docker new desired state. Controllers react to each other very quickly, making
Kubernetes extremely responsive. The actions of the system adapt to the
state of the world in real time.
NODE

Scheduler
Controller
Kublet Docker
How Kubernetes Helps Operators
Manager

For platform operators, infrastructure resources in Kubernetes are clustered

and can be consumed and released elastically, enabling seamless scaling and
higher resource utilization. Kubernetes eliminates many of the manual
provisioning and other tasks of conventional enterprise IT.
The Scheduler and the Controller Manager implement most of the orchestration
Kubernetes clusters deployed in different private and public clouds provide a
logic of Kubernetes. Together, etcd, the API Server, Scheduler, and Controller
uniform (or very similar) management environment and identical principles of
Manager make up the Kubernetes control plane; they can run on a single node
operation, reducing the learning curve associated with managing a multi-cloud
or across multiple nodes for availability.
environment and minimizing the risk of operator errors.
Worker nodes make up the data plane of Kubernetes; each worker node runs
the container runtime (Docker in the diagram) and a local daemon called the
Kubelet that communicates with the API Server.

Kubernetes Controllers
Kubernetes Controllers ensure that the observed state of the cluster is as close Portability Integration Community

as possible to your desired state. Each controller monitors its configuration, Run Kubenetes everywhere. Build on your SDDC Move in lockstep with the
infrastructure. community.
stored in the API Server. It then looks at the state of the world and tries to Leverage a common upstream
framework to run workloads Apply Kubernetes as a practical Work with a partner that
on-premises, in public clouds path to public cloud adoption demonstrates leadership in the
or hybrid cloud open source community.

5
K U B E R N E T E S F O R O P E R AT O R S

Two Elements of manually, Operators allow you to more easily

deploy and manage applications on Kubernetes.
use the Cluster API to change the number to eight
and it will automatically spin up another virtual

Kubernetes that All Operators encapsulate domain-specific knowledge

machine, applying Kubernetes logic to
infrastructure. If one of those servers fails, Cluster

Operators Should for a specific application. You can think of this as

embodying the knowledge and logic that might
API will automatically spin up a new one to take its
place, using self-healing to return to the desired
Know About traditionally be captured in run books. The open
source Operator Framework provides the
state.

necessary tools to facilitate Operator creation. To make Cluster API work for a particular type of
If you are new to Kubernetes, there are two
environment, you need a provider for that
emerging elements of the environment that you
should know about. These two things are rapidly environment. Provider implementations are already
changing the way that operators interact with Cluster API Streamlines available for major public clouds as well as VMware
Kubernetes:
Kubernetes Provisioning and vSphere. GitHub has a list of many of the available
providers.
Operators. Custom Kubernetes Management
controllers that implement domain-
specific logic for an application Infrastructure-level management is an area of rapid Where to Learn More
evolution for Kubernetes. Cluster API is a
Cluster API. Declarative APIs that Kubernetes project to bring declarative, The What and the Why of the Cluster API
facilitate cluster creation, configuration, [blog]
Kubernetes-style APIs to cluster creation,
and management in the Kubernetes
ecosystem configuration, and management. It provides
Cluster API Gitbooks
additive functionality on top of core Kubernetes.

Operators Tailor Kubernetes VMware is actively contributing to the

development of the Kubernetes Cluster API.
to Application Needs
One of the goals is to provide a way to let
Kubernetes Custom Resource Definitions or CRDs operators declaratively define what a cluster
extend the resources the API Server can manage. should look like. For example, using the Cluster API
CRDs are often paired with a custom controller you might declare that you want seven servers
called an Operator. By automating application- running in your Kubernetes cluster. If you later
specific tasks that otherwise have to be done decide that you want eight servers running, you

6
K U B E R N E T E S F O R O P E R AT O R S

First Principles for Operators

There are a few basic principles that will help your organization move forward
with Kubernetes.

You don’t have to decide everything upfront. Because Kubernetes

is flexible and extensible, decisions you make today don’t
necessarily lock you in for the rest of time. For example, you can
choose NGINX ingress controller now, and change to something else
like Contour in six months or a year as your needs change.

Everything doesn’t have to be greenfield. You don’t have to start

from scratch and deploy all new tools. Look for integration points
with the tools you are already using like Jenkins.

Enable your stakeholders. Don’t ignore the needs of others that will
be using the platform. If developers are used to being able to access
system logs for debugging, make sure they still have access and
know how to take advantage of any new tools.

7
K U B E R N E T E S F O R O P E R AT O R S

Top Six Questions to Answer Upfront Question 2: How should I build my

While you don’t have to answer every design question upfront, there are a
Kubernetes cluster?
few questions you should consider carefully before you start deploying There are two dimensions to this question: What infrastructure environment
Kubernetes. In this section, we look at the top six questions. While the should you choose? And, what software should you install in that
decisions are ultimately up to your team, we’ve tried to provide guidance environment? If you choose a cloud service, those decisions are largely
based on experience with many teams adopting Kubernetes. decided for you.

If you’re deploying on-premises, will the infrastructure be virtualized or bare-

Question 1: Where should I put my metal? Both can work, so it may come down to what you have and what
Kubernetes cluster? you’re most comfortable with. The major cloud providers deploy Kubernetes
on top of virtual machines, as do many enterprises with running VM
The first question that teams grapple with when they approach Kubernetes is environments.
where to deploy it. Should they deploy on-premises? In the cloud? There are
a range of factors you need to consider: Will you purchase new hardware for your deployment or repurpose existing
hardware? This largely depends on whether you have up-to-date hardware
On-premises. If you deploy Kubernetes on-premises, you’ll have available for the purpose.
complete control and more flexibility to tailor the deployment to
your business needs. On the other hand, Kubernetes deployment When it comes to software, the biggest decision is whether to choose:
requires some expertise, and there are a lot more decisions you’ll
• A hosted cloud service
have to make—and a lot more things you’ll ultimately be
responsible for managing. As you’ll learn shortly, there are a few • A packaged software distribution
decisions you have to get right from the beginning.
• A full do-it-yourself (DIY) installation from open source
Cloud. Cloud-based Kubernetes services like Amazon Elastic
Kubernetes Service (Amazon EKS), Google Kubernetes Engine This topic is discussed in more detail in the following chapter. Our general
(GKE), and Azure Kubernetes Service (AKS) provide key primitives guidance is to stay as close to upstream Kubernetes as possible no matter
that can make it easier to get started. However, you may be which option you choose. Compatibility ensures that you can move
trading future flexibility for speed and ease of initial deployment. operations between environments with no or minimal changes. Kubernetes is
evolving quickly so you want to ensure compatibility with the latest versions
and avoid wandering into any walled gardens.

8
K U B E R N E T E S F O R O P E R AT O R S

Question 3: How many clusters Persistent storage. A surprising number of teams get fairly far along in the
Kubernetes planning process without thinking about storage. Similar to
should I build? networking, Kubernetes provides storage flexibility through drivers that
conform to the Container Storage Interface (CSI). Many storage vendors offer
A question that comes up all the time is whether or not to have everything in drivers for compatibility with Kubernetes. You can view a list of available CSI
a single large cluster. You should think of Kubernetes as a multi-cluster drivers on GitHub.
solution and cultivate a multi-cluster mindset. Having multiple clusters
reduces the size of each failure domain and provides you with greater Connectivity. Plan ahead to make sure you have the appropriate
flexibility going forward. For example, you can stand up a new cluster with a infrastructure surrounding Kubernetes to support your application needs.
new feature and migrate services to that cluster to take advantage of the This may include things like load balancers and ingress controllers.
feature.
Security. A final item that’s hard to bolt-on to Kubernetes after the fact is
As a purely practical matter, the maximum “comfortable” size of a security. Be sure and involve your security team in Kubernetes planning.
Kubernetes cluster is around 500 nodes. Beyond that, you will have to start
tuning Kubernetes itself to continue scaling; that’s an arduous process..
Question 5: What about security?
Question 4: What about underlying The security model is an area that people often overlook during Kubernetes
planning phases:
infrastructure?
Multi-team vs. Multi-tenant. This distinction can be useful to think about
There are several infrastructure “plumbing” considerations that you definitely during security planning:
need to think about and understand before you begin to deploy. These may
be difficult to change after the fact: Multi-team. You need infrastructure to support different teams within the
same organization so there is a certain level of trust.
Container networking. Kubernetes gives you flexibility regarding networking
through Container Network Interface (CNI) plugins. There are a variety of CNI Multi-tenant. You need infrastructure to support separate organizations
plugins that support various approaches to software-defined networking where there is no trust relationship.
(SDN) and various network options. If you have to have a particular network
This distinction affects your approach to security. Kubernetes does not have
capability (for example multicast) you’ll need a CNI that supports that
a hard multi-tenancy design. Multi-tenancy can be enabled, but it doesn’t
feature. You can view a list of available CNI plugins on GitHub.
come right out of the box.

9
K U B E R N E T E S F O R O P E R AT O R S

Authentication/authorization. Use something like OIDC and connect your Kubernetes projects in areas ranging from database to key management to
existing authentication system to Kubernetes from the beginning. It’s simple observability, making it a little easier to identify the ones that you want to
to do and worth the effort. either adopt immediately or track for possible future use. Be warned,
however, that the number of items in the landscape is already a little
Policies. People often deploy Kubernetes and forget to configure things overwhelming.
like resource quotas and pod security policies. These are essential both to
secure your cluster and to achieve high levels of utilization.

Backup and restore. Having a regular backup policy is essential to protect

your Kubernetes environment. It enables you to move applications to
another cluster in a disaster situation and is also useful for moving a
collection of resources from one cluster to another. Velero is a useful tool,
providing backup and migration of Kubernetes applications and their
persistent volumes.

Question 6: How can I extend

Kubernetes to address specific
operational needs?
Unlike other platforms like OpenStack, Kubernetes is highly extensible. If
you have a scenario that Kubernetes doesn’t address, you can extend
Kubernetes without having to worry about the upgrade path going forward.
You should approach Kubernetes with that mindset and be prepared to take
advantage of existing Kubernetes extensions where possible—and to
develop your own when necessary. The Cloud Native Computing
Foundation (CNCF) maintains a landscape that shows many of the active

10
K U B E R N E T E S F O R O P E R AT O R S

11
K U B E R N E T E S F O R O P E R AT O R S

Which Flavor of Kubernetes Should Kubernetes Distributions

You Choose? A variety of vendors—Red Hat, Canonical, and many others—offer packaged
Kubernetes distributions. For anyone who has used Linux, the packaged
distribution model will seem familiar. However, there’s an important distinction:
Earlier you learned that there are three basic flavors of Kubernetes: Kubernetes is young and changing extremely quickly. Because they may limit
future compatibility and flexibility, avoid deep forks that diverge significantly
from upstream Kubernetes.

Distributions provide greater control than managed Kubernetes, including full

access to the control plane. They can work well if your operations stay within
Managed Cloud Kubernetes Do it yourself
the confines of what the distribution offers. However, if you need to extend
Service distribution
beyond the boundaries created by the distribution, you could experience issues.

One of your first decisions, as noted earlier, is to decide which one(s) to

choose. The good news is that if you get six months into deployment and Do It Yourself
decide for whatever reason that you made the wrong initial decision, you
can still switch strategies and recover. Whatever work you’ve done in one The final approach is to take a complete, do-it-yourself approach and build
environment should be largely compatible with the other environments. Kubernetes from source, either on your own or with the help of a partner. This
After all, that’s one of the key strengths of Kubernetes and containers. approach requires the most technical expertise and more personnel. You’ll
need to pay more attention to where Kubernetes is headed and create a
cluster lifecycle strategy to keep up with changes, and you’ll be more
Managed Kubernetes dependent on the community for support.

The major cloud providers all offer Kubernetes platforms that provide an However, this is the option that gives you the most flexibility and the greatest
easy, turnkey solution that you can use to get up and running with control. Our opinion is that it’s not as hard as people think and you shouldn’t
Kubernetes in a managed environment. The hosted cloud solutions have all simply rule it out. We continue to believe that this is the best option for many
committed to maintaining compatibility with upstream Kubernetes. However, organizations.
if portability is important to your operations, you still need to be careful to
avoid incorporating other services that are only available in a particular cloud.

12
K U B E R N E T E S F O R O P E R AT O R S

Avoiding Missteps
The following table describes some of the most
common missteps that we see organizations
make as they move to adopt Kubernetes—along
with tips on how to avoid them.

Common Kubernetes Missteps

Misstep 1: Misstep 2: Misstep 3:

Going all-in on managed Kubernetes Solving problems you don’t yet have Letting perfect be the enemy of done

It’s awesome to no longer have to manage With platforms that aren’t extensible, if you This misstep is a corollary to the previous
your control pane. However, there are don’t start out with a capability, it’s hard to one. You can waste a lot of cycles guessing
downsides associated with not having full add later. Kubernetes turns this paradigm on about future requirements and trying to build
control. Some limitations may be its head. Designing solutions for problems an ideal solution. If you narrow your scope,
showstoppers. with tips on how to avoid you think may arise in the future just adds you can finish deployment more quickly and
them. complexity and delays deployment. start gaining operational experience.

13
K U B E R N E T E S F O R O P E R AT O R S

Common Kubernetes Missteps

Misstep 4: Misstep 5: Misstep 6:

Trading “battle-tested” for “cutting-edge” Open source is not free Mixing too many workloads on one cluster

If you look at the CNCF landscape, there are The burden to support open source software It is easy to add workloads to an existing
a lot of interesting technologies. Resist the falls on you. Be diligent in evaluating a cluster, but new apps may require changes
temptation to bet your business on project before adopting it: How many stars that have undesired effects elsewhere, even
technology that’s brand new. Track does it have? How healthy is its community? compromising security. It’s better to have
interesting projects and give them time to Are people responsive? Are pull requests multiple smaller clusters. Managed
mature. reviewed and merged? Are there guidelines Kubernetes makes it fast and easy to add
for contributors? new clusters for unique requirements.

Misstep 7: Misstep 8: Misstep 9:

Using available tools for federation Not implementing resource quotas Not using operators

Federation is a hard problem, and current To increase resource utilization, you must Operators are becoming the preferred way
tools likely won’t meet your needs. Instead, implement resource quotas that ensure that for managing domain-specific knowledge
ensure you have copies of application no one person or application consumes too and simplifying application management in
containers and data where they can be used many Kubernetes resources. This allows you Kubernetes. Take advantage of existing
for DR. A modest substitution of human to do tighter bin packing on each node. operators when they are available and
effort for fancy federation insulates you from modify or build your own operators when
outages that span zones. necessary.

14
K U B E R N E T E S F O R O P E R AT O R S

What Should I Do Next? Kubernetes Academy

Kubernetes Academy provides an accessible learning path to

If you’re a developer starting out with Kubernetes, the most important thing is to get
advance your skill set, regardless of where you are on your
started. Use the resources links in this eBook to learn more, watch videos, and engage
Kubernetes journey. Comprised of a series of 8-minutes long
with your peers. In addition to connecting online, you may find local Kubernetes
videos, courses range from beginner to advanced.
meetups in your area, and there are annual KubeCon + CloudNativeCon conferences in
North America, Europe, and China.

Consider downloading one of the development tools mentioned earlier and kick the
tires, or install minikube on your local machine to get more familiar with Kubernetes. In
addition, VMware has a variety of resources for everyone on the Kubernetes journey:

TGIK

Every Friday at 1PM Pacific Time, VMware holds an informal hangout session focusing
on a specific Kubernetes-related topic. You can see the archive of past sessions on
YouTube and subscribe to view the live sessions.

Cloud Native Apps Blog

Read our regular blog to find out the latest. Posts cover diverse topics and new blogs
are posted regularly.

Watch a webinar on Cluster API

Learn about Cluster API, how it works, its current state, and why it’s crucial for the
future of Kubernetes.

And be sure and follow @cloudnativeapps on Twitter to keep up with all the latest
cloud native developments.

15