233 BROUGHT TO YOU IN PARTNERSHIP WITH

CONTENTS

∠ WHAT IS KUBERNETES?

Getting Started with ∠ KEY KUBERNETES CONCEPTS

∠ KUBERNETES ARCHITECTURE

Kubernetes
∠ GETTING STARTED WITH KUBERNETES

∠ EXAMPLE APPLICATION

∠ NAMESPACE, RESOURCE QUOTAS,

AND LIMITS

∠ AND MORE!

UPDATED BY ALAN HOHN, LOCKHEED MARTIN FELLOW

ORIGINAL BY ARUN GUPTA, PRINCIPAL OPEN SOURCE TECHNOLOGIST AT AMAZON WEB SERVICES

WHAT IS KUBERNETES?
apiVersion: v1
Kubernetes is an open source container orchestration system. It kind: Pod
manages containerized applications across multiple hosts for spec:
containers:
deploying, monitoring, and scaling containers. Originally created
- name: nginx
by Google, in March of 2016 it was donated to the Cloud Native image: nginx
Computing Foundation (CNCF). volumeMounts:
- mountPath: /srv/www
DZON E .COM/ RE FCA RDZ

name: www-data
Kubernetes, or “k8s” or “kube” for short, allows the user to
readOnly: true
declare the desired state of an application using concepts such as - name: git-monitor
“deployments” and “services.” For example, the user may specify image: kubernetes/git-monitor
env:
that they want a deployment with three instances of a Tomcat - name: GIT_REPO
web application running. Kubernetes starts and then continuously value: http://github.com/some/repo.git
volumeMounts:
monitors containers, providing auto-restart, re-scheduling, and - mountPath: /data
replication to ensure the application stays in the desired state. name: www-data
volumes:
Kubernetes is available as a standalone installation or in a variety - name: www-data
emptyDir: {}
of distributions, such as Red Hat OpenShift, Pivotal Container
Service, CoreOS Tectonic, and Canonical Kubernetes. Not only can containers in a pod share the same network
interfaces, but they can also share volumes, as shown in the
KEY KUBERNETES CONCEPTS
Kubernetes resources can be created directly on the command
line but are usually specified using Yet Another Markup Language
(YAML). The available resources and the fields for each resource
may change with new Kubernetes versions, so it’s important to
double-check the API reference for your version to know what’s
available. It’s also important to use the correct “apiVersion” that
matches your version of Kubernetes. This Refcard uses the API
from Kubernetes 1.12, released 27 September 2018.

POD
A Pod is a group of one or more containers. Kubernetes will
schedule all containers for a pod into the same host, with the
same network namespace, so they all have the same IP address
and can access each other using localhost. Here is an example
pod definition:

1
 fission.io

Get Started Today at Fission.io

 GETTING STARTED WITH KUBERNETES

example. This example uses the “git-monitor” container to keep a By declaring a service, we can provide a single point of entry for
Git repository updated in /data so the “nginx” container can run all the pods in a deployment. This single point of entry (hostname
a web server to serve the repository files. and IP address) remains valid as pods come and go.

When a pod is created, Kubernetes will monitor it and kind: Service

apiVersion: v1
automatically restart it if a process terminates. In addition, metadata:
Kubernetes can be configured to attempt to connect to a name: nginx-service
spec:
container over the network to determine if the pod is ready selector:
(readinessProbe) and still alive (livenessProbe). app: nginx
ports:
- protocol: TCP
port: 80
DEPLOYMENT
A deployment provides pod scaling and rolling updates.
Kubernetes will make sure that the specified number of pods is Services and deployments can be created in any order. The
running, and on a rolling update will replace pod instances one at service actively monitors Kubernetes for pods matching the
selector field. For example, in this case the service will match
a time, allowing for application updates with zero downtime.
pods with metadata.labels content of app: nginx, like the one
Deployments graduated from beta in Kubernetes 1.11, and replace shown in the deployment example above.
the older concept of Replica Sets. A deployment creates a replica
Services rely on Kubernetes to provide a unique IP address and route
set, but it is not necessary to interact with the replica set directly.
traffic to them, so the way services are configured can be different
apiVersion: apps/v1 depending on how your Kubernetes installation is configured.
kind: Deployment
metadata:
name: nginx-deployment PERSISTENT VOLUME CLAIM
labels: Kubernetes has multiple types of storage resources. The Pod
app: nginx
DZON E .COM/ RE FCA RDZ

spec: example above shows the simplest, an empty directory mounted

replicas: 3 into multiple containers in the same pod. For truly persistent storage,
selector:
matchLabels:
the most flexible approach is to use a Persistent Volume Claim.
app: nginx
template: A Persistent Volume Claim requests Kubernetes to dynamically
metadata: allocate storage from a Storage Class. The Storage Class is typically
labels:
app: nginx created by the administrator of the Kubernetes cluster and must
spec: already exist. Once the Persistent Volume Claim is created, it can
containers:
be attached to a Pod. Kubernetes will keep the storage while the
- name: nginx
image: nginx:1.15.4 Persistent Volume Claim exists, even if the attached Pod is deleted.
ports:
- containerPort: 80 kind: PersistentVolumeClaim
apiVersion: v1
metadata:
Deployments include a “template” that specifies what the created name: web-static-files
pods should look like, so there is no need to separately define a spec:
resources:
pod. Kubernetes will automatically create the required number of requests:
pods from the deployment template. storage: 8Gi
storageClassName: file-store
---
Note that a deployment will identify matching pods using the
apiVersion: v1
matchLabels field. This field must always have the same data as kind: Pod
the metadata.labels field inside the template. The deployment metadata:
name: webserver
will take ownership of any other running pods that match the spec:
matchLabels selector, even if they were created separately, so containers:
- image: nginx
keep these names unique. name: nginx
volumeMounts:
- mountPath: /usr/share/nginx/html
SERVICE name: web-files
A service provides load balancing to a deployment. In a volumes:
- name: web-files
deployment, each pod is assigned a unique IP address, and when
persistentVolumeClaim:
a pod is replaced, the new pod typically receives a new IP address. claimName: web-static-files

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 GETTING STARTED WITH KUBERNETES

The above example declares both the Persistent Volume Claim run them; and kube-proxy, which manages networking rules so
and a Pod that uses it. This example takes advantage of the ability connections to service IP addresses are correctly routed to pods.
to place multiple YAML documents in the same file using --- as
As shown in the picture, each node can run multiple pods, and
a separator. Of course, in a real example, it would be better to
each pod can include one or more containers. The pod is purely a
separate the Persistent Volume Claim so it can easily be retained
Kubernetes concept; the kubelet configures the container engine
even if the Pod is deleted.
to place multiple containers in the same network namespace so
For more information on the available providers for Kubernetes those containers share an IP address.
Storage Classes, and for multiple examples on configuring persistent
storage, see the DZone Refcard Persistent Container Storage. GETTING STARTED WITH KUBERNETES
SETTING UP KUBERNETES
KUBERNETES ARCHITECTURE There are a variety of ways to set up, configure, and run
Kubernetes uses a client-server architecture, as seen here:
Kubernetes. It can be run in the cloud using providers such
as Amazon Elastic Container Service for Kubernetes, Google
Kubernetes Engine, Azure Kubernetes Service, Packet, Pivotal
Container Service, and others. It can be also run on-premise by
building a cluster from scratch on physical hardware or via virtual
machines. The various options are described in the Kubernetes
setup documentation, where you can find out which solution is
best for you and get step-by-step instructions. The most popular
option for building a multi-host Kubernetes cluster from scratch
is kubeadm, while the recommended way to get started and run a
DZON E .COM/ RE FCA RDZ

A Kubernetes cluster is a set of physical or virtual machines and single-node cluster for development and testing is to use Minikube.
other infrastructure resources that are used to run applications.
However you set up your cluster, you will interact with it using the
The machines that manage the cluster are called Masters, and the
standard Kubernetes command-line client program kubectl.
machines that run the containers are called Nodes.

MINIKUBE
MASTER
Minikube uses virtualization software like VirtualBox, VMware, or
The Master runs services that manage the cluster. The most
important is kube-apiserver, which is the primary service that KVM to run the cluster. Once Minikube is installed, you can use

clients and nodes use to query and modify the resources running the minikube command-line to start a cluster by running the

in the cluster. The API server is assisted by: etcd, a distributed key- following command:

value store used to record cluster state; kube-controller-manager, minikube start

a monitoring program that decides what changes to make when

To stop the cluster, you can run:
resources are added, changed, or removed; and kube-scheduler,
minikube stop
a program that decides where to run pods based on the available
nodes and their configuration. To determine the IP address of the cluster is using:
minikube ip
In a highly-available Kubernetes installation, there will be
multiple masters, with one acting as the primary and the others If you are having problems, you can view the logs or ssh into the
as replicas. host to help debug the issue by using:

minikube logs
NODE
minikube ssh
A Node is a physical or virtual machine with the necessary
services to run containers. A Kubernetes cluster should have as
You can also open a dashboard view in the browser to see and
many nodes as necessary for all the required pods. Each node
change what is going on in the cluster.
has two Kubernetes services: kubelet, which receives commands
to run containers and uses the container engine (e.g. Docker) to minikube dashboard

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 GETTING STARTED WITH KUBERNETES

KUBECTL kubectl create deployment nginx --image=nginx

kubectl is a command-line utility that controls the Kubernetes deployment.apps/nginx created

cluster. Commands use this format:

This command will start a Deployment, which contains a Repilca
kubectl [command] [type] [name] [flags]
Set, which contains a Pod, which contains a Docker container
running an NGINX web server. We can use kubectl to get the
• [command] specifies the operation that needs to be
status of the deployment:
performed on the resource. For example, create, get,
describe, delete, or scale. kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE
• [type] specifies the Kubernetes resource type. For AGE
example, pod (po), service (svc), deployment (deploy), or nginx 1 1 1 1
1m
persistentvolumeclaim (pvc). Resource types are case-
insensitive, and you can specify the singular, plural, or The status of the Replica Set can be seen by using:
abbreviated forms. kubectl get rs
NAME DESIRED CURRENT READY AGE
• [name] Specifies the name of the resource, if applicable. nginx-65899c769f 1 1 1 1m
Names are case-sensitive. If the name is omitted, details
The status of the Pod can be seen by using:
for all resources will be displayed (for example, kubectl get
pods). kubectl get po
NAME READY STATUS RESTARTS
• [flags] Options for the command. AGE
nginx-65899c769f-kp5c7 1/1 Running 0
1m
Some examples of kubectl commands and their purpose:
DZON E .COM/ RE FCA RDZ

Of course, most of the time we will use a YAML configuration file:

COMMAND PURPOSE
kubectl create -f nginx-deployment.yaml

kubectl create -f nginx. Create the resources specified The file nginx-deployment.yaml contains the Deployment
yaml in the YAML file. If any speci- definition shown above.
fied resources exist, an error is
returned.
SCALE APPLICATIONS
Deployments can be scaled up and down:
kubectl delete -f nginx. Delete the resources spec-
yml ified in the YAML file. If any kubectl scale --replicas=3 deploy/nginx
resources do not exist, they deployment.extensions/nginx scaled
are ignored.
The Kubernetes controller will then work with the scheduler to
kubectl get pods List all pods in the “default” create or delete pods as needed to achieve the requested number.
namespace. See below for This is reflected in the deployment:
more information on name-
spaces. kubectl get deploy
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE
kubectl describe pod Show metadata for the “nginx” AGE
nginx pod. The name must match nginx 3 3 3 3
exactly. 3m

kubectl --help You can verify there are three pods by running:
Show the complete list of
available commands. kubectl get po
NAME READY STATUS RESTARTS
AGE
RUN YOUR FIRST CONTAINER nginx-65899c769f-c46xx 1/1 Running 0
Most of the time when using kubectl, we create YAML resource 38s
nginx-65899c769f-j484j 1/1 Running 0
files, so we can configure how we want our application to run.
38s
However, we can create a simple Deployment using kubectl nginx-65899c769f-kp5c7 1/1 Running 0
without using a YAML file: 3m

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 GETTING STARTED WITH KUBERNETES

Note that the original Pod continued to run while two more were Note that we use a PersistentVolumeClaim so we get persistent
added. Of course, we would also want to create a Service to assist data; we’ll assume this has been created already since we want it
in load balancing across these instances; see below for a more to stay around for a long time even as we update the application.
complete example.
Even though there will only be one database instance, we will

DELETE APPLICATIONS create a Service so the IP address will stay the same even if the
Once you are done using the application, you can destroy it with PostgreSQL pod is replaced.
the delete command. kind: Service
apiVersion: v1
kubectl delete deployment nginx
metadata:
deployment.extensions "nginx" deleted
name: postgres-service
spec:
Because Kubernetes monitors pods to achieve the desired selector:
number of replicas, we must delete the Deployment to remove app: postgresql
ports:
the application. Simply stopping the container or deleting the
- protocol: TCP
pod will just cause Kubernetes to create another pod. port: 5432

Next, we create the deployment for the Node.js application:

EXAMPLE APPLICATION
Let’s put multiple Kubernetes apiVersion: apps/v1
features together to deploy a Node.js kind: Deployment
metadata:
application together with a PostgreSQL
name: nodejs
database server. Here is the planned labels:
architecture (see right). app: nodejs
spec:
DZON E .COM/ RE FCA RDZ

We’ll work from the bottom of the replicas: 3

diagram. First, we’ll define the selector:
matchLabels:
PostgreSQL deployment: app: nodejs
template:
apiVersion: apps/v1
metadata:
kind: Deployment
labels:
metadata:
name: postgresql app: nodejs
labels: spec:
app: postgresql containers:
spec: - name: nodejs
replicas: 1 image: nodejs:10-alpine
selector: command: ["npm"]
matchLabels: args: ["start"]
app: postgresql env:
template: - name: NODE_ENV
metadata: value: production
labels: workingDir: /app
app: postgresql volumeMounts:
spec: - mountPath: /app
containers: name: node-app
- name: postgresql readOnly: true
image: postgres:10.4 - name: git-monitor
env: image: kubernetes/git-monitor
- name: PGDATA env:
value: "/data/pgdata" - name: GIT_REPO
volumeMounts: value: http://github.com/some/repo.git
- mountPath: /data volumeMounts:
name: postgresql-data - mountPath: /data
volumes: name: node-app
- name: postgresql-data volumes:
persistentVolumeClaim: - name: www-data
claimName: postgresql-data emptyDir: {}

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 GETTING STARTED WITH KUBERNETES

This example uses a “sidecar” container, kubernetes/git-monitor, Once we’ve created the namespace, we can create resources in it
to keep our application up to date based on a Git repository. using the --namespace (-n) flag, or by specifying the namespace
The sidecar populates a volume which is shared with the in the resource’s metadata:
Node.js container.
apiVersion: v1
kind: Pod
Finally, we create the service that provides the user entrypoint for
metadata:
our application: name: webserver
namespace: development
kind: Service
…
apiVersion: v1
metadata:
name: nodejs-service By using separate namespaces, we can have many pods called
spec: webserver and not have to worry about name collisions.
selector:
app: nodejs
ports: ACCESS CONTROL
- protocol: TCP
Kubernetes supports Role Based Access Control (RBAC).
port: 3000
type: LoadBalancer
Here’s an example that limits developers to read-only access for

Providing services with external IPs so they can be visible from pods in production. First, we create a ClusterRole, a common

outside the cluster is a complex topic because it depends on your set of permissions we can apply to any namespace:

cluster’s environment ( e.g. cloud, virtual machine, or bare metal). kind: ClusterRole
This example uses a LoadBalancer service, which requires some apiVersion: rbac.authorization.k8s.io/v1
external load balancer such as an Amazon Elastic Load Balancer metadata:
name: pod-read-only
DZON E .COM/ RE FCA RDZ

to be available and configured in the cluster.

rules:
- apiGroups: [""] # "" indicates the core API group
NAMESPACE , RESOURCE QUOTAS, AND LIMITS resources: ["pods"]
Kubernetes uses namespaces to avoid name collisions, to verbs: ["get", "watch", "list"]

control access, and to set quotas. When we created resources

above, these went into the namespace default. Other resources Next, we use a RoleBinding to apply this ClusterRole to a
that are part of the cluster infrastructure are in the namespace specific group in a specific namespace:
kube-system.

kind: RoleBinding
To see pods in kube-system, we can run:
apiVersion: rbac.authorization.k8s.io/v1
metadata:
$ kubectl get po -n kube-system
name: read-only
NAME READY STATUS
namespace: production
RESTARTS AGE
subjects:
…
- kind: Group
kube-apiserver-minikube 1/1 Running 0
name: developers
17m
apiGroup: rbac.authorization.k8s.io
…
roleRef:
kind: ClusterRole
name: pod-read-only
RESOURCE ISOLATION apiGroup: rbac.authorization.k8s.io
A new namespace can be created from a YAML resource definition:
Alternatively, we can use a ClusterRoleBinding to apply a role
apiVersion: v1
kind: Namespace to a user or group in all namespaces.
metadata:
name: development RESOURCE QUOTAS
labels: By default, pods have unlimited resources. We can apply a quota
name: development
to a namespace:

7 BROUGHT TO YOU IN PARTNERSHIP WITH

 GETTING STARTED WITH KUBERNETES

apiVersion: v1 Note that we can request fractions of a CPU and use varying units
kind: ResourceQuota for memory.
metadata:
name: compute-resources
namespace: sandbox
spec:
hard:
cpu: "5"
memory: 10Gi

Kubernetes will now reject unlimited pods in this namespace.

Instead, we need to apply a limit:

apiVersion: v1
kind: Pod
metadata:
name: webserver
namespace: sandbox
spec:
containers:
- image: nginx
name: nginx
resources:
limits:
memory: "128Mi"
cpu: "500m"
DZON E .COM/ RE FCA RDZ

Written by Alan Hohn, Lockheed Martin Fellow

Alan Hohn is a Lockheed Martin Fellow who has worked as a software architect, lead, and manager. After
many years writing and teaching Java, he now mostly creates services in Go and Python. He is an advocate,
trainer, and coach for Agile and DevOps, and is the author of recent video courses on Ansible.

DZone, Inc.
DZone communities deliver over 6 million pages each 150 Preston Executive Dr. Cary, NC 27513
month to more than 3.3 million software developers, 888.678.0399 919.678.0300
architects and decision makers. DZone offers something
Copyright © 2018 DZone, Inc. All rights reserved. No part of this publication
for everyone, including news, tutorials, cheat sheets,
may be reproduced, stored in a retrieval system, or transmitted, in any form
research guides, feature articles, source code and more. or by means electronic, mechanical, photocopying, or otherwise, without
"DZone is a developer’s dream," says PC Magazine. prior written permission of the publisher.

8 BROUGHT TO YOU IN PARTNERSHIP WITH

 BROUGHT TO YOU IN PARTNERSHIP WITH

CONTENTS

Advanced
∙ About Kubernetes
∙ Basics
∙ Kubernetes Constructs
∙ Extending Kubernetes

Kubernetes
∙ Kubectl
∙ Managing Nodes
∙ Test Plan
∙ Security and Troubleshooting
∙ Feeding Results to Other Things
∙ Etcd

WRITTEN BY CHRIS GAUN ∙ Security Checklist

PRODUCT MANAGER OF KUBERNETES PLATFORM, D2IQ
∙ Additional Resources
CONTRIBUTIONS BY JASON HERNANDEZ
CERTIFIED KUBERNETES ADMINISTRATOR (CKA) & SOLUTIONS ARCHITECT, D2IQ

ABOUT KUBERNETES This research points out how quickly Kubernetes is being adopted
Kubernetes is a distributed cluster technology that manages a and what a strategic role it plays in the global digital landscape.
container-based system in a declarative manner using an API.
Kubernetes differs from the orchestration offered by configuration
Kubernetes is an open-source project governed by the Cloud Native
management solutions in that it provides a declarative API that
Computing Foundation (CNCF) and counts all the largest cloud
collects, stores, and processes events in an eventually consistent
providers and software vendors as its contributors along with a long
manner. A few traits of Kubernetes include:
list of community supporters.
• Abstraction – Kubernetes abstracts the application
There are currently many learning resources to get started with the orchestration from the infrastructure resource and as-a-
fundamentals of Kubernetes, but there is less information on how to service automation. This allows organizations to focus on
manage Kubernetes infrastructure on an ongoing basis. This Refcard the APIs of Kubernetes to manage an application at scale
aims to deliver quickly accessible information for operators using any in a highly available manner instead of the underlying
Kubernetes product. infrastructure resources.

For an organization to deliver and manage Kubernetes clusters

to every line of business and developer group, Operations needs
to architect and manage both the core Kubernetes container
orchestration and the necessary auxiliary solutions (add-ons) —
e.g., monitoring, logging, and CI/CD pipeline. The CNCF maps out
many of these solutions and groups them by category in the “CNCF
Landscape”.

Last year, Gartner predicted that by 2022, more than 75% of global
organizations would be running containerized applications in
production, up from less than 30% at that time. The CNCF 2019
survey, published roughly one year later in 2020, says that more than
that are already in production.

1
Technology

Training

Services
& Support

The Leading Independent

Platform for Enterprise
Kubernetes

Explore the
D2iQ Kubernetes Platform

D2iQ simplifies and automates the really

difficult tasks needed for enterprise
Kubernetes in production at scale.

Simplify your
Kubernetes Journey
 REFCARD | ADVANCED KUBERNETES

• Declarative – Kubernetes’ control plane decides how the CONTROLLER MANAGER

hosted application is deployed and scaled on the underlying Kubernetes uses a control plane to perform non-terminating control
fabric. A user simply defines the logical configuration of the loops to observe the state of Kubernetes objects and reconcile it with
Kubernetes object, and the control plane takes care of the the desired state.
implementation.
For more information, see the Kubernetes documentation: https://
• Immutable – Different versions of services running on
kubernetes.io/docs/reference/command-line-tools-reference/kube-
Kubernetes are completely new and not swapped out.
controller-manager/
Objects in Kubernetes, say different versions of a pod, are
changed by creating new objects. KUBELET (AGENT)
Kubernetes clusters have a worker node called a kubelet that has
several functions (e.g., kubelets create Pods from the Podspec).
BASICS
Figure 1: Simplified Kubernetes architecture and components For more information, see the Kubernetes documentation:
https://kubernetes.io/docs/reference/command-line-tools-
reference/kubelet/

SCHEDULER
Kubernetes relies on a sophisticated algorithm to schedule
Kubernetes objects. The scheduling takes into account filters such
as resources, topology, and volume, and then uses prioritization
settings such as affinity rules to provision pods on particular
Kubernetes worker nodes.

CONTROL PLANE, NODES, AND PERSISTENT STORAGE For more information, see the Kubernetes documentation: https://
Kubernetes’ basic architecture requires a number of components. kubernetes.io/docs/reference/command-line-tools-reference/
kube-scheduler/
API SERVER
The main way to communicate with Kubernetes clusters is through KUBE-PROXY
the Kubernetes API Server. A user, and other Kubernetes components, Each Kubernetes cluster worker node has a network proxy for
sends declarative events to the Kubernetes API Server through connectivity.
HTTPS (port 443). The API Server then processes events and updates
the Kubernetes persistent data store — e.g., etcd. The API Server also For more information, see the Kubernetes documentation: https://

performs authentication and authorization. kubernetes.io/docs/reference/command-line-tools-reference/

kube-proxy/
For more information, see the Kubernetes documentation:
https://kubernetes.io/docs/concepts/architecture/master-node- OTHER STANDARD COMPONENTS OF KUBERNETES
communication/ In order to run the most basic Kubernetes cluster, a number of
additional components and add-ons are required.
ETCD CLUSTER
As Kubernetes collects declarative events, it needs to store them. KUBE-DNS
Kubernetes uses etcd for this persistent storage, providing a single All services and pods (when enabled) on Kubernetes are assigned
source of truth of all Kubernetes objects. etcd requires an odd a domain name that is resolved by the DNS. This DNS is handled by
number of nodes. A loss of etcd storage results in a loss of the Kube-DNS, which itself is a pod and a service on Kubernetes. Kube-
cluster's state, so etcd should be backed up for disaster recovery. DNS resolves DNS of all services in the clusters.

For more information, see the Kubernetes documentation: KUBECTL

https://kubernetes.io/docs/tasks/administer-cluster/configure- The official command line for Kubernetes is called kubectl (the
upgrade-etcd/ pronunciation is up for debate). All industry-standard Kubernetes
commands start with kubectl.

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | ADVANCED KUBERNETES

METRICS SERVER AND METRICS API EXTENDING KUBERNETES

Kubernetes has two resources for giving usage metrics to users and Kubernetes has a number of points to extend its core functionality:
tools. First, Kubernetes can include the metrics server, which is the
POINT DESCRIPTION
centralized aggregation point for Kubernetes metrics. Second is
the Kubernetes metrics API, which provides an API to access these Custom Resource CRD allows users to extend Kubernetes with
Definition (CRD) custom APIs for different objects beyond the
aggregated metrics. standard ones supported by Kubernetes.

WEB UI (DASHBOARD) Container Runtime CRI is a plugin API that enables Kubernetes to
Kubernetes has an official GUI called Dashboard. That GUI is distinct Interface (CRI) support other container runtimes beyond Docker
and Containerd.
from vendor GUIs that have been designed for specific Kubernetes
derivative products. Note that the Kubernetes Dashboard release Container Network CNI gives users a choice of network overlay that
Interface (CNI) can be used with Kubernetes to add SDN features.
version does not necessarily match the Kubernetes release version.
Container Storage CSI empowers users to support different storage
KUBERNETES CONSTRUCTS Interface (CSI) systems through a driver model.

Kubernetes has a number of constructs for defining and managing

objects on the cluster:
KUBECTL
CONSTRUCT DESCRIPTION Below are some commands useful for IT professionals getting started

Namespaces Kubernetes includes a means to segment a single with Kubernetes. A full list of kubectl commands can be found at the
physical cluster into separate logical clusters reference documentation: https://kubernetes.io/docs/reference/
using namespacing.
generated/kubectl/kubectl-commands
Pods Whatever the runtime, Kubernetes fundamentally
manages a logical grouping of one or more MAKING LIFE EASIER
containers called a pod.
Finding Kubernetes command short name:
StatefulSets A Kubernetes controller for managing workloads
that require proper management of state. $ kubectl describe

ReplicaSets One of the control loops available in Kubernetes

Find out more:
that ensures that the desired number of pods
are running. • Using kubectl aliases –
https://github.com/ahmetb/kubectl-aliases
Roles Kubernetes has several access control schemes,
and users should always default to role-based
• Switching among Kubernetes clusters and
access controls to maximize security.
namespaces (context switching) –
Ingresses and Load In order to expose a service outside a cluster, a https://github.com/ahmetb/kubectx
Balancing user should set up an ingress for layer 7 or define
the configuration of a layer 4 load balancer using • Tailing logs from multiple pods –
type=loadbalancer in the service definition.
https://github.com/wercker/stern

Deployments The declarative controller in Kubernetes that

Setting your Bash prompt to mirror Kubernetes context:
manages replicasets of pods.

Services Defined by a label, a Kubernetes service prompt_set() {

is a logical layer that provides IP/DNS/etc. if [ "$KUBECONFIG" != "" ]; then
persistence to dynamic pods. PROMPT_KUBECONTEXT="k8s:$(kubectl config current-
context 2>/dev/null)\n"
DaemonSet A Kubernetes construct that enables users to run
a pod on every node in the cluster. fi
PS1="${PROMPT_KUBECONTEXT}[\u@\h \W]\$ "
Jobs and Cronjobs Kubernetes includes the logic to run jobs, }
processes that run to completion, and
k8s:admin@local
cronjobs — processes that run at specific
intervals and run to completion. [kensey@sleeper-service ~]$

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | ADVANCED KUBERNETES

COMMANDS $ kubectl create service Create a clusterIP for a service

clusterip foo named foo
KUBECTL COMMAND FORMAT
--tcp=5678:8080

$ kubectl version Print the client and server version

$ kubectl autoscale Autoscale pod foo with a minimum
information
deployment foo --min=2 of 2 and maximum of 10 replicas
$ kubectl cluster-info IP addresses of master and services --max=10 --cpu-percent=70 when CPU utilization is equal to or
greater than 70%
$ kubectl get namespaces List all the namespaces in the
Kubernetes cluster

$ kubectl cordon NODE Mark node as unschedulable — used MANAGING NODES

for maintenance of the cluster Sometimes it is necessary to perform maintenance on underlying
nodes. In those cases, it is important to use eviction and make sure
$ kubectl uncordon NODE Mark node as available for scheduling
— used after maintenance the application owners have set a pod disruption budget.

$ kubectl drain NODE Removes pods from node via graceful To properly evict a node, use:
termination for maintenance
1. Cordon node: $ kubectl cordon $NODENAME
$ kubectl drain NODE Find the names of the objects that
--dry-run=client will be removed 2. Drain node: $ kubectl drain $NODENAME
• Respects PodDisruptionBudgets
$ kubectl drain NODE Remove pods even if they are not
--force=true managed by controller 3. Uncord node: $ kubectl uncordon $NODENAME

$ kubectl taint nodes Taint a node so they can only run

node1 key=value:NoSchedule dedicated workloads or certain pods TEST PLAN
that need specialized hardware
According to the CNCF, most Kubernetes clusters are still provisioned
$ kubectl run nginx Start instance of nginx and managed software distributions (instead of public cloud provider
--image=nginx --port=8080 options). For these clusters, once a new cluster is up and running, it is
imperative to devise a test plan and design a run book to make sure
$ kubectl expose rc Create a service for a replicated nginx,
nginx --port=80 --target- which serves on port 80 and connects it is operational. For new public cloud providers or new versions of
port=8080 to the containers on port 8000 Kubernetes on a provider, it is also important to run a testing plan.

$ kubectl get RESOURCE Print information on Kubernetes

Below is testing plan used as part of D2iQ's testing of Kubernetes.
resources, including:
• all
After each step, validate that the cluster is working properly.
• certificatesigningrequests
(aka csr) TEST CLUSTERS
• clusterrolebindings
• clusterroles 1. Provision a highly available Kubernetes cluster with 100
Kubernetes nodes
$ kubectl explain RESOURCE Print documentation of resources
2. Scale that cluster down to 30 nodes after it has finished
$ kubectl scale Scale a ReplicaSet (rs) named foo provisioning
--replicas=COUNT rs/foo
Can also scale a Replication 3. Provision 3 additional highly available Kubernetes clusters
Controller or StatefulSet with 5 nodes
$ kubectl apply -f Perform rolling update (where v2 4. Scale all 3 clusters (in parallel) to 30 nodes simultaneously
frontend-v2.json is the next version of your service)
5. Provision 16 more highly available Kubernetes clusters with
$ kubectl label pods foo Update the labels of resources 30 nodes
GPU=true
6. Kill 5 Kubernetes nodes on a single cluster simultaneously
$ kubectl delete pod foo Delete foo pods 7. Kill 3 control-plane nodes on a single cluster (fewer if the

$ kubectl delete svc foo Delete foo services

nodes will not automatically reprovision)

8. Kill the etcd leader

TABLE CONTINUES IN NEXT COLUMN

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | ADVANCED KUBERNETES

TEST WORKLOADS You can also remove pods from the environment.

1. Provision storage (potentially using CSI and a specific driver)

Waiting —> Incorrect Image Information
• Test different provider-specific storage features
TROUBLESHOOTING COMMAND
2. Run the e2e cluster loader with the higher end of pods
per node (35-50) Check YAML URL spec:
containers:
3. Run Kubernetes conformance testing to stress the cluster — name: example
image: url:port/image:v
4. Provision Helm to the Kubernetes cluster

5. Test specific Kubernetes workloads (using Helm charts) Pull an image onto desktop to docker pull <image>

• Deploy Nginx determine whether registry and

• Deploy Redis image information are correct

• Deploy Postgres
• Deploy RabbitMQ Also check that the secret information is correct.

6. Deploy services with type=loadbalancer to test load

CRASH LOOPING DEPLOYMENT
balancer automation
• Use ctrl + C to exit the crash loops
• Test different provider-specific load
balancer features • Troubleshoot
• Roll back deployment
7. Expose a service using Ingress

TROUBLESHOOTING COMMAND
SECURITY AND TROUBLESHOOTING
General issue checking $ kubectl describe deployments
TROUBLESHOOTING WITH KUBECTL
The Kubernetes command line kubectl provides many of the Rolling back a $ kubectl rollout undo [Deployment
deployment Name]
resources needed in order to debug clusters.

Pausing a deployment $ kubectl rollout pause [Deployment

CHECK PERMISSIONS Name]

COMMAND EXAMPLE

Check your $ kubectl auth $ kubectl auth can-i create GENERAL DEBUGGING
permissions can-i deployments --namespace dev
Watch the stdout of a container:
Check $ kubectl auth $ kubectl auth can-i create
permissions can-i [Options] deployments --namespace dev $ kubectl attach bash-rand-77d55b86c7-bntxs
of other users --user=chris

Launch a temporary pod with an interactive shell:

PENDING AND WAITING PODS $ kubectl run busybox --rm -i --tty --restart=Never
--image busybox -- /bin/sh
Pending —> Check Resources

TROUBLESHOOTING COMMAND Copy files into and out of containers (note that this requires a tar

General issue checking $ kubectl describe pod <name of binary in the container):
pending pod>
$ kubectl cp default/bash-rand-77d55b86c7-bntxs:/
Check if pod is using $ kubectl top pod root/rand .
too many resources tar: Removing leading `/' from member names

Check node resources $ kubectl top node

$ cat rand
Get node resources $ kubectl get nodes -o yaml | grep 567bea045d8b80cd6d007ced02849ac4
-A 15 allocatable

Get all pod resources $ kubectl top pod --all-namespaces

--containers=true

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | ADVANCED KUBERNETES

Increase the verbosity of kubectl output (99 is the most TROUBLESHOOTING WITH JQ
verbose — “get everything”): Kubectl provides an option to output results in JSON format, which
allows for additional parsing and filtering. This can be very beneficial
$ kubectl -v 99 get nodes
when writing scripts or monitoring the state of resources in a
I1211 11:10:28.611959 24842 loader.go:359]
Kubernetes cluster. A popular open-source utility called “jq” makes it
Config loaded from file /home/kensey/bootkube/
easy to parse JSON from the command line.
cluster/auth/kubeconfig
I1211 11:10:28.612482 24842 loader.go:359]
Instead of writing code to parse the JSON objects, the output can
Config loaded from file /home/kensey/bootkube/
be piped into the jq utility to filter out resources that meet certain
cluster/auth/kubeconfig
I1211 11:10:28.614383 24842 loader.go:359] criteria. For example, kubectl can be used to print out all pods in
Config loaded from file /home/kensey/bootkube/ json format, but jq can add value by parsing out only pods with a
cluster/auth/kubeconfig specific start date or container image.
I1211 11:10:28.617867 24842 loader.go:359]
Config loaded from file /home/kensey/bootkube/ Basic usage:
cluster/auth/kubeconfig
[some JSON content] | jq [flags] [filter expression]
I1211 11:10:28.629567 24842 round_trippers.
go:405] GET https://192.168.122.138:6443/api/v1/
nodes?limit=500 200 OK in 11 milliseconds Use the kubectl json format and pipe to the jq utility using a
I1211 11:10:28.630279 24842 get.go:558] no kind simple dot filter to pretty-print the output:
is registered for the type v1beta1.Table in scheme
$ kubectl get nodes -o json | jq '.'
"k8s.io/kubernetes/pkg/api/legacyscheme/scheme.
go:29"
NAME STATUS ROLES AGE VERSION Use the hierarchy to filter the input so only node labels are displayed:

$ kubectl get nodes -o json | jq '.items[].

Get more information about Kubernetes resources:
metadata.labels'
$ kubectl explain crd
KIND: CustomResourceDefinition
MORE ADVANCED FUNCTIONS
VERSION: apiextensions.k8s.io/v1beta1
Find pods with containers that have not passed a readiness probe:
DESCRIPTION: • select – cherry-pick a piece of the input by criteria
CustomResourceDefinition represents a resource
and use a boolean expression to match desired value
that should be exposed on
the API server. Its name MUST be in the format • Pipe the output of one filter into another filter to clean
<.spec.name>.<.spec.group>. up the results

FIELDS: $ kubectl get pods -o json | jq '.items[] | select(

[...] .status.containerStatuses[].ready == false ) |
.metadata.name'
Control output:

$ kubectl get deploy bash-rand -o yaml

TROUBLESHOOTING WITH CURL
apiVersion: extensions/v1beta1
Adding headers to requests are often used for:
kind: Deployment
metadata: • Setting accepted content types for replies.
annotations: • Setting content type of posted content.
deployment.kubernetes.io/revision: "1" • Injecting bearer auth tokens.
creationTimestamp: 2018-12-11T05:20:32Z
generation: 1
$ curl --header "Authorization: Bearer [token]"
labels:
[API server URL]
run: bash-rand
[...]

7 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | ADVANCED KUBERNETES

Building and submitting requests: ETCD

• GET – Request is contained in the URL itself (default method) A production Kubernetes cluster will typically have three etcd nodes,
and used to read/list/watch resources which provides high availability and can withstand the loss of a single
• POST – Submit a data blob to create resources etcd node. The etcdctl utility provides a number of commands for
• PATCH – Submit a data blob to merge-update resources running operations on the etcd cluster.
• PUT – Submit a data blob to replace a resource
DESCRIPTION COMMAND
• DELETE – Submit options to control deletion of a resource
Set an environment ETCDCTL="ETCDCTL_API=3 etcdctl
$ curl --cert client.cert --key client.key --cacert
variable for the etcdctl --cert=server.crt --key=server.key
cluster-ca.cert \
binary, endpoints, and --cacert=ca.crt
https://192.168.100.10:6443/api/v1/namespaces/ the required certificates --endpoints=https://127.0.0.1:2379"
default/pods and key.
{
"kind": "PodList", List the members of the $ ETCDCTL member list
"apiVersion": "v1", etcd cluster
"metadata": {
Check the etcd endpoint $ ETCDCTL endpoint health --cluster
"selfLink": "/api/v1/namespaces/default/pods",
health -w table
"resourceVersion": "126540"
}, Check the etcd endpoint $ ETCDCTL endpoint status --cluster
"items": [ status -w table

List any active alarms $ ETCDCTL alarm list

FEEDING RESULTS TO OTHER THINGS Create an etcd backup $ ETCDCTL snapshot save snapshot.db

Basic method – bash while read loop:

$ while read -r serverevent; do echo "$serverevent" \ SECURITY CHECKLIST

| jq '.["operation"] = .type | .["node"] = .object. Security is fundamental for Kubernetes day-two operations. Users
metadata.name | { "operation": .operation, "node":
should ensure that some basic cluster security is being maintained.
.node}' ; \
Here is a security checklist for Kubernetes administrators:
done < <(curl -sN --cert client.cert --key client.
key --cacert cluster-ca.cert \
https://192.168.100.10:6443/api/v1/nodes?watch=true) Kubernetes patch releases – i.e., 1.x.y (ensure it has all CVE fixes)
{ A new version of Kubernetes is released every three months, and
"operation": "ADDED",
patches with bug fixes come out at regular intervals for the latest
"node": "192.168.100.10"
} three versions.
[...]
Kubernetes versions must be kept current due to a lack of
widespread support for older versions and the many bug fixes
Something completely different – bash copro:
in each patch release. Also, Kubernetes is new software, and
#!/bin/bash security vulnerabilities are announced on a regular basis.

coproc curl -sN --cacert cluster-ca.cert --cert ./

Transport Layer Security (TLS)
client.cert --key ./client.key \
Kubernetes components must use an encrypted connection.
https://192.168.100.10:6443/api/v1/nodes?watch=true
Make sure that Kubernetes clusters have end-to-end TLS enabled.
exec 5<&${COPROC[0]}

Kubernetes RBAC and authentication

while read -ru 5 serverevent; do
if [[ $(echo $serverevent | jq -r '.type') == Kubernetes has several forms of access management, including
"ADDED" ]]; then role-based access control (RBAC). Make sure that RBAC is enabled
echo "Added node $(echo $serverevent | jq -r and that users are assigned proper roles.
'.object.metadata.name') in namespace \
$(echo $serverevent | jq '.object.metadata. Network policies enabled
namespace')" A Kubernetes network is flat. Every pod is given an IP address
fi
done and can communicate with all other pods on its namespace. It is
possible to use network policies to limit the interactions among
trap 'kill -TERM $COPROC_PID' TERM INT
pods and lock down this cross-communication.

8 BROUGHT TO YOU IN PARTNERSHIP WITH

 REFCARD | ADVANCED KUBERNETES

Different clusters or namespaces based on the security profile Secure Dashboard

Namespaces can create logical clusters in a single physical The Kubernetes Dashboard is a functional utility for users, but it
cluster, but they only provide soft isolation. Do not include vastly can also be a vector for malicious attacks — limit its exposure as
different services in the same cluster. much as possible. Employ three specific tactics:

Using different Kubernetes clusters reduces the potential for • Do not expose the Dashboard to the Internet
vulnerabilities to affect all systems. Also, large clusters with • Ensure the Dashboard ServiceAccount is not open
unrelated services become harder to upgrade, which violates the and accessible to users
first item in this checklist. • Configure the login page and enable RBAC

Implement resource quotas

To prevent “noisy neighbors” and potential denial of service
ADDITIONAL RESOURCES
• Tracking Kubernetes Releases: https://github.com/kubernetes/
situations, do not let containers run without an upper bound on
sig-release/tree/master/releases
resources. By default, all resources are created with unbounded
- Enhancements, Bug Triage, and Release Calendar
CPU and memory limits. To avoid the default behavior, assign
• Tracking Kubernetes Enhancement Proposals: https://github.
resource quotas at the namespace level, which are critical in
com/kubernetes/enhancements/tree/master/keps
preventing overconsumption of resources.
• Check if the Kubernetes distribution or installer has been
Limit access to insecure API server ports through conformance testing: https://docs.google.com/
The API Server is the main means to communicate with spreadsheets/d/1LxSqBzjOxfGx3cmtZ4EbB_BGCxT_wlxW_
xgHVVa23es/edit#gid=0
Kubernetes components. Besides enabling authentication and
RBAC, lock down all unsecure API server ports. • Run end-to-end tests to validate a Kubernetes cluster: https://
github.com/kubernetes/test-infra
Limit access of pod creation for users • Kubernetes declarative test tool (KUTTL): https://github.com/
One of the most widely used vectors for attacking container kudobuilder/kuttl
management systems are the containers themselves. To ensure • CIS Security Benchmark tool: https://github.com/mesosphere/
the containers are secure and protect the system from attacks: kubernetes-security-benchmark

• Limit who can create pods. • YAML Templates: https://cheatsheet.dennyzhang.com/

• Limit the use of unknown or unmaintained libraries. kubernetes-yaml-templates

• Use a private container registry and tagged container • Use jq to process JSON output from the command line: https://
images, keeping tagged images immutable. stedolan.github.io/jq/

WRITTEN BY CHRIS GAUN, CONTRIBUTIONS BY JASON HERNANDEZ,

PRODUCT MANAGER OF KUBERNETES PLATFORM, CERTIFIED KUBERNETES ADMINISTRATOR (CKA) &
D2IQ SOLUTIONS ARCHITECT, D2IQ

Chris Gaun is a CNCF ambassador and Jason Hernandez is a Certified Kubernetes

product manager for the D2iQ Kubernetes Administrator (CKA) and solutions architect at
Platform. He has presented at Kubecon several D2iQ. Prior to D2iQ, he worked as a Site Reliability
times and has hosted over 40 Kubernetes workshops across Engineer helping to build robust monitoring systems and platform
the US and EU. He lives in Mississippi with his beautiful wife automation. He lives in Illinois with his wife and their recently
Jasmin, kids, and dog Panda. adopted puppy.

DZone, a Devada Media Property, is the resource software Devada, Inc.

developers, engineers, and architects turn to time and 600 Park Offices Drive
again to learn new skills, solve software development Suite 150
problems, and share their expertise. Every day, hundreds of Research Triangle Park, NC 27709
thousands of developers come to DZone to read about the
latest technologies, methodologies, and best practices. That 888.678.0399 919.678.0300
makes DZone the ideal place for developer marketers to Copyright © 2020 Devada, Inc. All rights reserved. No part
build product and brand awareness and drive sales. DZone of this publication may be reproduced, stored in a retrieval
clients include some of the most innovative technology and system, or transmitted, in any form or by means of electronic,
tech-enabled companies in the world including Red Hat, mechanical, photocopying, or otherwise, without prior
Cloud Elements, Sensu, and Sauce Labs. written permission of the publisher.

9 BROUGHT TO YOU IN PARTNERSHIP WITH

 BROUGHT TO YOU IN PARTNERSHIP WITH

CONTENTS

∠ Kubernetes: A Brief Overview

∠ Monitoring Kubernetes: The

Monitoring
Challenges

∠ Monitoring Kubernetes from the

Sources

∠ Prometheus

Kubernetes ∠ Sensu

∠ Sensu and Prometheus

∠ Conclusion

∠ References

WRITTEN BY STEFAN THORPE

HEAD OF DEVOPS AND SECURITY AT CHERRE AND CTO AT CAYLENT

years of experience running production workloads, and is designed to

Kubernetes: A Brief Overview
Kubernetes allows you to deploy and manage cloud-native applications handle tens, thousands, and even millions of containers. Google itself

with more flexibility than any container orchestration platform has ever initiates more than 2 billion container deployments weekly. At one point, it

provided before. Kubernetes, affectionately referred to as Kube or K8s for ran everything through its internally developed platform: Borg.

short, is an open-source platform for developers that eliminates many

Borg was a large-scale internal cluster management system and Kuberne-
of the manual processes involved in managing, deploying, and scaling
tes' predecessor. The internal program ran hundreds of thousands of jobs
containerized applications.
from a multitude of different applications across countless clusters, each

The fault-tolerant and scalable platform can be deployed within almost with a vast number of machines. Much of the education garnered over the

any infrastructure. Kubernetes will integrate with anything, from public years from developing Borg was distilled into the Kubernetes technology

clouds (Google Cloud, Microsoft Azure, Amazon Web Services, and so on) we see today.

to private or hybrid clouds to server clusters, data centers, or on-premises

Alongside the release of Kubernetes v1.0 in July 2015, Google partnered
— or, indeed, any combination of these solutions. The remarkable fact that
with the Linux Foundation to establish the Cloud Native Computing Foun-
Kubernetes supports the automatic placement and replication of contain-
ers over such a large number of hosts is no doubt one of the many reasons
why the platform is becoming the de-facto stage for running microservices
infrastructure in the cloud. Future-proof,
multi-cloud
THE KUBERNETES ORIGIN STORY
Kubernetes was initially conceived and developed by the engineers at monitoring
Google. Open-sourced since its launch in mid-2014, K8s is now main-
at scale.
tained, upgraded, and managed by a large community of contributors that
includes software titans like Microsoft, Red Hat, IBM, and Docker.
Download Sensu Now
As well as being publicly open about how the company's entire infrastruc-
ture runs in containers, Google was an early contributor to Linux container
technology (the tech that powers all of Google's extensive cloud services).
Kubernetes originally came into being through the cloud platform's 15

1
 MONITORING KUBERNETES

dation (CNCF). The CNCF's purpose is to build sustainable ecosystems As well as being truly container runtime agnostic, Kubernetes' features
while fostering a community around a network of high-quality projects include everything necessary to centrally automate, control, and scale
that orchestrate containers within either a monolithic or a microservices containerized applications. Here are the highlights:
architecture. Kubernetes is one of the highest velocity CNCF projects in the
• Enterprise-scale container deployments
history of open source.
• Rollout management
HOW KUBERNETES WORKS • Built-in service discovery
Kubernetes, at its fundamental level, is a system for coordinating contain- • Self-healing
erized applications across a cluster of machines. It's designed to allow • Autoscaling
developers to fully manage the lifecycle of containerized applications and • Persistent storage management
services using features that deliver predictability, scalability, and high • High-availability features
availability. Simply put, K8s helps developers pilot container clusters, • Interconnected clusters
hence the name, which means helmsman (or pilot) in Greek. It is the most • Resource bin packing
popular option available today because it provides an abstraction to make • Secrets
a cluster of machines act identically to one big machine — a vital feature
when running large-scale environments. These highlighted features are why Kubernetes has become widespread
for leveraging different application architectures from monolithic or
By default, Kubernetes optimizes the use of Docker to run images and batch-driven applications to highly distributed microservice applications.
manage containers. However, K8s can also use other container engines,
such as rkt from CoreOS, for example. Monitoring Kubernetes: The Challenges
Traditionally, servers were treated like pets, with "all hands on deck" being
called to fix one that went down. With containerization, servers are num-
bered as cattle in a herd; when one goes down, it’s simply replaced. Since
its release in 2014, Kubernetes has revolutionized container technology
and become a critical tool for quickly launching applications at scale.

However, challenges arrive hand-in-hand with innovation and speed, and

they are not easy to overcome from the control panel. Containers produce
a significant level of complexity when it comes to orchestration.

While Kubernetes has made it dramatically more accessible for everyone to

implement and manage containers — including everything from scheduling
and provisioning to automatically maintaining the desired state — there are
This architecture diagram shows how Kubernetes works. It outlines a set key obstacles to overcome in terms of monitoring both Kubernetes itself
of master components, including "pods," that model an application-spe- and the applications running on it. Among these challenges are:
cific "logical host." A Kubernetes pod envelops an application container (or,
sometimes, multiple containers), storage resources, an exclusive network • Interconnected applications distributed across multiple
IP address, and details about how the container(s) should run. They are vendors: At its core, Kubernetes serves as a universal platform that
the smallest atomic unit for containers. Theoretically, pods contain one companies can use to deploy applications wherever they wish them
or more tightly coupled applications — with best practice aiming for one to run. Typically, companies adopt a multi-cloud solution to remain
container per pod. More on pods later. cloud-agnostic and avoid vendor lock-in. Multiple cloud-hosted
environments also come with their own inherent value of mitigating
Essentially, the process includes an API server, a scheduler, and controllers.
downtime and data loss, which is why it's so common. However,
The API server exposes the Kubernetes API and processes REST operations
with assets, software, and applications distributed across multi-
as well as subsequent updates. The scheduler attaches the undeployed
cloud solutions comes monitoring issues for extracting real-time
pods with a suitable VM or physical machine. If none are available, the pod
data about the health of all widely dispersed resources.
stays unscheduled until it can identify an appropriate node. The master runs
the other cluster-level functions through multiple embedded controllers to • Constantly moving applications on dynamic infrastructure:
achieve endpoint creation, node discovery, and replication control, among With constant movement of multiple applications, it can be
other operations. Because this controller design is flexible and extendable, difficult to achieve complete visibility across every platform and
Kube admins can build their own controllers. Kube tracks the shared state of protocol at any given moment. This can result in problems with
the Kubernetes cluster via the API server and implements changes to make isolating any bottlenecks or constraints in the system. As many
the current state and the desired state of the cluster correspond. established companies have multiple applications within their

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 MONITORING KUBERNETES

infrastructure, this challenge is practically unavoidable. Without a

Monitoring Kubernetes from the Sources
robust monitoring system to capture the detailed picture, compa- Like most container orchestration platforms, Kubernetes comes with a set
nies are vulnerable to visibility and buried problems that can burn of elementary tools that monitor servers, though they aren't all exactly
through money and resources. "built-in." With some additional minor tinkering, it's possible to leverage
these base additional components to gain more visibility into Kubernetes
• Numerous complex components to observe: Akin to migrating
from the get-go. These rudimentary options include:
from a monolithic application to a microservices architecture,
deploying and monitoring even a small cluster on K8s can present
• The Kubernetes Dashboard: Not deployed by default, this add-
a challenge. Kubernetes is a complex machine with a lot of
on is worth running for providing the basic visualization of the
components operating together, so adopting the platform means
resources running on each deployed K8s cluster. It also delivers a
monitoring all of the following (and then some):
primary means of managing and interacting with those resources

• The capacity and resource utilization of clusters including: as well as the environment.

− Nodes: To ensure the state of all K8s nodes, track the • Container probes: These are diagnostics which actively monitor
CPU, memory, and disk utilization. the health of a container. Each probe results in either one of
three outcomes:
− Deployments/pods: Confirm all implemented pods in a
deployment are healthy.
• Success: The container passed the diagnostic probe.
− Containers: Watch the CPU and memory consumption
• Failure: If the probe determines that a container is no
in relation to configured limits.
longer healthy, the probe will restart it.

• Applications: Track the performance and availability of all • Unknown: The diagnostic failed, so no action is necessary.
applications running in the cluster through request rates,
throughput, error rate, and so on. • Kubelet: The kubelet is a primary node agent that runs on every
node in the cluster and ensures the containers are working. It's
• End-user experience: Supervise mobile application and
also the mechanism through which the master communicates
browser performance to respond to insights and errors.
with the nodes. The kubelet exposes many individual container
Ensure customer satisfaction by following and improving
usage metrics straight from cAdvisor.
load time and availability.

• Supporting infrastructure: As mentioned above, it's vital • cAdvisor: This Kubernetes component is an open-source container

to track Kubernetes clusters running across different plat- resource usage and performance analysis agent that is integrated

forms and multiple vendors. with the kubelet. It is purpose-built for containers, identifying all
containers running in a machine to collect statistics about memory,
• Monitoring the minutiae of operational complexity: All the
network usage, filesystem, and CPU. cAdvisor operates per node,
Kubernetes core elements (i.e., the kubelet, API server, Kube
and also analyzes overall machine usage by assessing the "root"
controller manager, and Kube scheduler) involve numerous
container on the machine. It is simple to use, but also limited:
flags that determine how the cluster operates and performs. The
default values may be fine for smaller clusters at the beginning, • cAdvisor only covers basic resource utilization; it doesn't
but as deployments scale up, it's important to make and monitor have the capacity to analyze how applications are actual-
adjustments. It also becomes crucial to keep track of Kubernetes ly performing.
details such as labels and annotations.
• cAdvisor also doesn't provide any trending, long-term
storage or analysis facilities.
Ironically, monitoring solutions can be responsible for bringing down
clusters by pulling extreme amounts of data from the Kubernetes APIs.
• Kube-state-metrics: Another add-on service that listens to the
Therefore, it is a good idea to determine a key baseline for monitoring, and
Kubernetes API and translates information about Kubernetes
dial it up when necessary to further diagnose issues that require addition-
constructs into metrics. In general, the model interrogates the
al attention. Kubernetes API server and recovers data about the states of
Kubernetes objects.
If a high level of monitoring is justified, it may be mindful to deploy more
masters and nodes (which you'll need to do for HA anyways). Another • Metrics server: The metrics server collects metrics from the
practical technique, especially when it comes to large-scale implementa- Summary API, exposed by the kubelet on each node. It is a clus-
tions, is to implement a lone cluster for storing Kubernetes events. This ter-wide aggregator of resource usage data. As of Kubernetes v1.8,
way the performance of the main instances won't be affected by any it's deployed by default in clusters created by kube-up.sh script as
volatility in event creation and event retrieval for monitoring purposes. a deployment object.

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 MONITORING KUBERNETES

The path of the core monitoring pipeline is as follows: environment. When first installing Prometheus, it's important to define a
few parameters that allow the system to scrape and collect data, starting
1. As cAdvisor is installed by default on all cluster nodes, it collects
with the scrape interval. Prometheus can be configured to provide either
data metrics about those containers and nodes.
real-time monitoring or interval-based analysis of Kubernetes nodes.
2. The kubelet exposes these metrics through the kubelet APIs. (The
default is a one-minute resolution.) By assigning external labels and attaching them to time-series alerts, it's
3. The metrics server identifies all available nodes and requests the also possible to use Prometheus as an alert system for node failures and oth-
kubelet API to report all container and node resource usage. er issues. The rest of the system is defined through scrape configurations.
4. The metrics server then exposes these metrics through the Kuber-
netes aggregation API. THE PROMETHEUS OPERATOR
The difference between controllers and operators often confuses many
Fundamentally, this pipeline outlines a sense of the initial steps necessary users. To deconstruct the two: a Kubernetes Operator is the name of a
to incorporate reasonable monitoring for Kubernetes environments. pattern that involves a Controller adding new objects to the Kubernetes
While we still don't have detailed application monitoring through these API to configure and manage applications such as Prometheus. Put simply,
rudimentary components, we can at least observe and monitor the under- an operator is a domain-specific controller.
lying hosts and Kubernetes nodes, and receive some metrics about any
Kubernetes abstractions. "The Prometheus Operator serves to make running Prometheus on top
of Kubernetes as easy as possible, while preserving Kubernetes-native
Customarily, the average K8s cluster administrator is interested in configuration options." ("Prometheus Operator", 2019)
monitoring from a holistic point of view, while application developers
who build services tend to lean towards monitoring from an app-centric Using the Prometheus Operator allows for easy monitoring of K8s services
point of view. Recently, however, these lines have blurred. Regardless of and deployments. It is viable to run Prometheus using a predetermined
the perspective required, all teams want to minimize the amount of input configuration .yml file, but this can later be automated according to how
required to monitor their systems and manage data collection. We'll now Kubernetes is set up. The Prometheus Operator creates, configures, and
take a look at two viable monitoring options that bring in full visualization manages all monitoring instances atop Kubernetes. When deploying a
and system-level metrics, allowing developers to track, troubleshoot, and new version of an app, K8s creates a new pod (container). Once the pod is
receive alerts about the most crucial parts of K8s clusters. ready, Kube destroys the old one. Prometheus is on constant watch, track-
ing the API; when it detects any differentials, it creates a new Prometheus

Prometheus configuration based on the services or pods changes.

OVERVIEW OF PROMETHEUS
The second major project to graduate CNCF's process from incubation to CORE COMPONENTS
maturity, Prometheus is an open-source monitoring and alerting toolkit Prometheus is a robust monitoring tool that uses a pull-based architecture
designed specifically for containers and microservices. Renowned for — as opposed to pushing metrics to the monitoring tool, it pulls metrics
being a top systems and service monitoring system, Prometheus offers from services. Prometheus also has a push gateway, meaning it does sup-
robust features that benefit all stakeholders involved in the development port "push" for certain metrics when the pull model doesn't work (though
pipeline, including cloud administrators and developers. It aggregates this method is discouraged).
metrics from configured targets at specified intervals, assesses rule ex-
In addition, Prometheus allows you to connect time series with metric
pressions, presents the results, and can provoke alerts if certain conditions
name and key-value pairs, simplifying monitoring in a complex, multi-
are observed to be true.
node cloud environment. As well as being able to see the big picture,
Prometheus delivers support for instrument application in multiple it's also possible to dig deep into an individual microservice easily with
languages. The tool also utilizes exporters to collect and expose telemetry Prometheus.
for services that can't be instrumented with the Prometheus instrumen-
tation client or library (PostgreSQL, MySQL, AWS CloudWatch, ETCD, and The tool also comes with PromQL, for easy processing of time-series data.

Kubernetes itself). You can use queries to manipulate data and generate insights relevant to
your situation. You can also use PromQL to create graphs, visualize data-
Prometheus does more than simply monitor predefined metrics. It is capa- sets, create tables, and generate alerts based on specific parameters.
ble of implementing dimensional data models for truly in-depth analysis,
creating relational insights based on multiple metrics. In other words, devel- Prometheus' web-based console lets you manage all features and

opers and administrators can get a clear view of Ops from different angles. tools available inside Prometheus. You can use regular expressions
and advanced PromQL queries for creating datasets and alerts, and it's
HOW PROMETHEUS WORKS accessible from the outside (when the cloud infrastructure is set up to
Prometheus is implemented as an additional layer into your Kubernetes enable remote access).

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 MONITORING KUBERNETES

BENEFITS OF THE PROMETHEUS APPROACH various sources, Sensu leverages a hybrid of push/pull with its publish/
Simplicity and flexibility are two of Prometheus' strongest suits. When us- subscribe (Pub/Sub) model — because Sensu uses a message bus for com-
ing Prometheus as a monitoring framework on Kubernetes, it's possible to munication, you can publish messages to a specific topic and consumers
implement a highly dimensional data model for monitoring. Prometheus can subscribe to one or more topics. This difference in approach makes
is also available as a Docker image, which means it is easy to set up the Sensu equally as robust, but for different reasons. Both Prometheus and
monitoring framework as part of a containerized cluster. The tool also Sensu allow you to collect the same metrics, although Sensu's service
integrates well with Grafana for increased monitoring visualization. health checking abilities (like the ability to monitor external resources
with proxy requests and entities) can help fill gaps in a purely teleme-
When used with Kubernetes, Prometheus can also collect node, pods,
try-focused approach. A better way forward, therefore, would be to run
and services metrics using the native service discovery configurations of
them in unison.
Kubernetes. There is no need to jump through hoops just to get a com-
prehensive monitoring system up and running; users can jump straight to HOW SENSU WORKS
defining expressions and creating alerts. Sensu uses native collection plugins to collect data for analysis from a
wide range of mainstream utilities (such as StatsD libraries, Prometheus
There is also Prometheus' simple and fluent nature. The scraping capability exporters, Nagios plugins, SNMP traps, CollectD metrics, and so on). It
of Prometheus can be easily integrated with Kube and other compatible differentiates itself from other tools, including Prometheus, by supporting
tools, including Docker and StatsD. In more complex environments, Pro- multi-cloud environments out of the box and providing high availability
metheus works well with kubernetes-service-endpoints and API servers. Even immediately from when users begin configuring the framework.
better, you can use the web GUI to configure alerts and graphs on the fly.
Sensu provides building blocks such as event filters, mutators, and
As with any pure telemetry monitoring solution, however, you miss handlers for operators to use to model workflows and automate them.
out on context with Prometheus' constrained data model. The fact that Via this method, data can be consolidated from external monitoring tools
Prometheus' data collection model defaults to time series presents a with existing operational ones to establish an event-based approach to
double-edged sword. On one hand, you can easily standardize collecting monitoring.(Sensu Go)
data in a particular way, which helps simplify things; on the other, you've
now limited yourself to a constrained data model that might be missing
some vital context.

The pull-based model also gives you limited granularity, as exporters only
provide summarized data that's only scraped periodically. This model
also requires that you punch holes in the firewalls of traditional networks,
and therefore is not suited for monitoring legacy or multi-generational
The Sensu Go open-source project is governed by an MIT license, with
infrastructure. Because Prometheus uses a set of discovery libraries to stay
source code that users are able to download and compile from github.
up to date with the platforms it monitors (like Kube), a degree of lag is in-
com/sensu/sensu-go. Supported (built and tested) packages are available
troduced as resources come and go, which in turn is exaggerated by metric
for download on the Sensu website, and are free to use up to 1,000 enti-
scraping intervals. Finally, Prometheus collection is unauthenticated and
ties. Organizations that require monitoring at scale with enterprise-level
unencrypted — anything that has access to the network can observe your
integrations will need an enterprise license.
telemetry data. (This includes metrics labels — you add labels for context,
but then that context would be out in the open.)
CORE COMPONENTS
For Kubernetes containers, Sensu optimizes its auto-discovery capabilities.
Sensu It's easy to configure monitoring checks and collectors for the Kubernetes
OVERVIEW OF SENSU components, and you can do the same in other containerized environments,
Sensu is a monitoring solution designed for multi-cloud and containerized including Docker. It's also easy to configure a handful of checks for monitor-
infrastructure. One of the benefits that Sensu brings to the table is a com- ing all Kubernetes components and the applications running on them.
posable monitoring event pipeline with an event-driven architecture. The
Sensu agent is a cross-platform event producer, enabling you to execute Similar to Prometheus, Sensu also comes with native support for integration
service checks to monitor system and service health as well as collect and and plugins. It works with logging tools and plays well with Prometheus
analyze unique metrics. Custom workflows enable you to go beyond alert itself. It's possible to use the two frameworks running alongside each other
or incident management (e.g., auto remediation), and monitoring APIs, and have them work together in processing different datasets. For example,
client libraries, and plugins written in any scripting or programming lan- Sensu can collect StatsD metrics and write them to Prometheus.
guage (including Nagios). Such customization offers a range of capabilities.
Sensu can also adapt to its Kubernetes environment. Let's say Sensu is
While Prometheus uses pull architecture for actively scraping data from deployed inside a container, and the decision is made to move the entire

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 MONITORING KUBERNETES

app to another containerized environment. Complete the app migration, metadata:

and Sensu will still work in a different environment. With its auto-native name: sensu-prometheus-collector
namespace: default
support, the Sensu agent will run at the new location and be discovered
labels: {}
by the Sensu system. You don't need another integration — it has its own
annotations: {}
auto-discovery mechanism.
spec:
url: https://github.com/sensu/sensu-prometheus-
KUBERNETES MONITORING WITH SENSU
collector/releases/download/1.1.5/sensu-prometheus-
Sensu is a highly scalable monitoring framework for Kubernetes, one that
collector_1.1.5_linux_amd64.tar.gz
will grow alongside deployed apps and cloud environments and provide sha512:
detailed functional monitoring. There is no upper limit to using Sensu, even b183e1262f13f427f8846758b339844812d6a802d5b98ef
in terms of the level of complexity needed for monitoring requirements. 28ba7959290aefdd4a62060bee867a3faffc9bad1427997
c4b50a69e1680b1cda205062b8d8f3be9
Sensu is a well-established, extensible solution with an active community filters:
of expert operators. - entity.system.os == linux
- entity.system.arch == amd64

Sensu and Prometheus

Using Sensu and Prometheus together can improve operational visibility sensuctl create -f sensu-sensu-prometheus-collector-
1.1.5-linux-amd64.yml
through monitoring. Both Sensu and Prometheus can run in tandem,
enhancing the other's capabilities and filling in the gaps.
And here's an example of a check using the Prometheus Collector:

HOW IT WORKS example-app-metrics.yml

The Sensu Prometheus Collector is a Sensu Check Plugin that aggregates ---
scraped data from a Prometheus exporter or the Prometheus query API. type: CheckConfig
The collected metrics are issued to STDOUT in one of three formats: api_version: core/v2
metadata:
• Influx (the default) name: example-app-metrics
• Graphite namespace: default

• JSON spec:
command: sensu-prometheus-collector -exporter-url
The Sensu Prometheus Collector combines the workflow automation http://localhost:8080/metrics

capabilities of Sensu and the power of Prometheus data scraping into a interval: 10
output_metric_format: influxdb_line
Prometheus metric poller. You can develop your own preferences for your
output_metric_handlers:
instrumented code and for when you receive alerts. Sensu will also deliver
- influxdb
the collected metrics to the external time-series database of your choice, publish: true
such as InfluxDB, Graphite, or Prometheus. runtime_assets:
- sensu-prometheus-collector
INSTALL THE SENSU PROMETHEUS COLLECTOR subscriptions:
You can discover, download, and share assets using Bonsai, the Sensu - metrics
asset index. To use the Sensu Prometheus Collector, select the Download
button on the asset page in Bonsai to download the asset definition for sensuctl create -f example-app-metrics.yml

your Sensu backend platform and architecture. Asset definitions tell Sensu
For the most up-to-date asset definition and usage examples, please check
how to download and verify the asset when required by a check, filter,
Bonsai: bonsai.sensu.io/assets/sensu/sensu-prometheus-collector
mutator, or handler.

BENEFITS
Once you've downloaded the asset definition, you can register the asset
The advantages of using Sensu and Prometheus in parallel through the
with Sensu using sensuctl, the command line tool for managing resources
Sensu Prometheus Collector include:
within Sensu, and create your own monitoring workflows.
• As well as monitoring the health of your Kubernetes clusters, the
For example, here's the Prometheus collector asset definition and associ- two will dynamically monitor and collect service health checks
ated sensuctl command for Linux: and metrics for the surrounding infrastructure.

sensu-sensu-prometheus-collector-1.1.5-linux-amd64.yml • Being able to go beyond problem detection for self-healing and

--- streamline the monitoring process through workflow automation.
type: Asset
• With the two together, you get more context (what/where the
api_version: core/v2
event is from) through context-rich notifications and reporting.

7 BROUGHT TO YOU IN PARTNERSHIP WITH

 MONITORING KUBERNETES

• The ability to achieve more granularity and flexibility to cover the Given how software-dependent we are today, availability is crucial for a
finer aspects of data scraping and analysis with more details. business' survival, and downtime can prove expensive as well as damag-
ing to an organization's reputation. Maintaining visibility into our systems
• Sensu supports and uses standard cryptography for communi-
is therefore essential to overcoming these challenges, and monitoring
cation. Sensu's model allows for a single agent to collect and
system infrastructure and applications has become integral to business
transmit data securely without having to compromise firewalls.
operations. In order to fully reap the rewards that Kubernetes has to offer,
• Being able to easily manage and configure your monitoring setup.
implementing a unified, future-proof monitoring solution (i.e., one that
• The capability to monitor your entire infrastructure from legacy to bridges the gap between past, current, and future technologies) is critical.
cloud-based infrastructure.

References
Conclusion • Prometheus Operator. (2019). Retrieved from coreos.com/opera-
Essentially, leveraging Sensu can allow you to monitor your entire infra-
tors/prometheus/docs/latest/user-guides/getting-started.html
structure — from Kubernetes to bare metal — with increased customiza-
tion and context. Prometheus' robust model is focused on scraping flat • Sensu Go. (n.d.). Sensu Go 5.5. Retrieved from docs.sensu.io/
telemetry data (at truly comprehensive levels), while Sensu offers visibility sensu-go/latest/
into your entire stack, working with industry standard technologies and
formats (like Nagios and StatsD). You can use Sensu to complement
Prometheus' extensive features for additional context from events, health
checks (i.e., not just metrics), more comprehensive capabilities around
processing monitoring data as a workflow, and a wholly secure solution.

Written by Stefan Thorpe, Head of DevOps and Security at Cherre and CTO at Caylent
Stefan is an IT professional with 20+ years management and hands-on experience providing technical and DevOps
solutions to support strategic business objectives.

Devada, Inc.
600 Park Offices Drive
Suite 150
Research Triangle Park, NC

DZone communities deliver over 6 million pages each month 888.678.0399 919.678.0300
to more than 3.3 million software developers, architects, and
decision makers. DZone offers something for everyone, including Copyright © 2019 DZone, Inc. All rights reserved. No part of this pub-
news, tutorials, cheat sheets, research guides, feature articles, lication may be reproduced, stored in a retrieval system, or transmit-
source code, and more. "DZone is a developer’s dream," says PC ted, in any form or by means electronic, mechanical, photocopying,
Magazine. or otherwise, without prior written permission of the publisher.

8 BROUGHT TO YOU IN PARTNERSHIP WITH

 BROUGHT TO YOU IN PARTNERSHIP WITH

CONTENTS

∠ Microservices Overview

∠ Orchestration Overview

∠ Key Characteristics of

Microservices on
Microservices With Orchestration

∠ Why Orchestration Matters in

Microservices

∠ Design Principles for Microservices

Kubernetes
in an Orchestrated World

∠ Monitoring in a Microservices/
Kubernetes World

∠ How to Handle Ephemeral

Containers

∠ Methods for Handling Microservice

Monitoring

∠ Summary Impact of Orchestration

on Microservice
WRITTEN CHRISTIAN MELENDEZ CLOUD ARCHITECT AT EQUINIX EMEA

Microservices is not a new architecture style. Back in 2014, Martin Having multiple services instead of just one brings increased levels
Fowler and James Lewis published a post defining microservices. of complexity. It adds such questions as: Which service should we
One year later, Kubernetes (the most popular container orchestration update first when it's time to update, especially when there are several
at the moment) was open-sourced by Google. In this Refcard, you services to update simultaneously? There are so many pieces now that
will find an introduction to these technologies, as well as the automating the deployment of each service becomes a requirement.
essential characteristics and design principles of microservices in If there are issues in production, which service is a problem or is
an orchestration world. This Refcard will also dive into monitoring causing a problem? Good observability is needed to spot the issue is
and logging, which are essential tools in distributed systems like an imminent need when working with microservices. It seems that
microservice architectures. with monolith systems, things were simpler. So, is it worth having
the additional complexity of decoupling a monolith into several
Microservices Overview microservices? Sometimes it is, especially when working with systems
Microservices is an architectural style where every piece of a system utilizing Kubernetes in the containers world.
lives separately, as an individual service — rather than condensing all
An advantage of having separate services is that it's possible to
the different parts of a system into a single, larger service, as happens
with traditional monolithic architectures. It is useful to keep in mind work with each microservice independently. Applying an update or

that microservices should not mean as small as possible, but rather patch, or even replacing an entire piece, means not having to notify

should be taken to mean as small as necessary. or change any other parts in the system. This independence provides
flexibility for developing each microservice in the appropriate
1456&78595:;<4= 1456&78;>?5@A?B;>A@= programming language — even different programming languages,
different frameworks, different database engines, or different
!"#$%&'$( !"#$%&'$( operative systems. As long as there's always a way of communicating
between services (like with REST, HTTP, or gRPC), all technology
)"#(*+%&+,%,-#. )"#(*+%&+,%,-#.
decisions will be reasonable.
+/'+0#*% +/'+0#*%
A good approach when working with different technologies and
,(&1'"23+' ,(&1'"23+' platforms is to use containers. Containers simplify the distribution
of a microservice, since the application and all its dependencies are

1
 MICROSERVICES ON KUBERNETES

packaged and ready to be used anywhere, as long as the container the state by orchestrating the storage, too; it doesn't matter if it's local
engine is running. But then, isn't it counterproductive having to or external (like a cloud service or a network storage system).
manually instantiate a container in a server cluster every time a
deployment happen? Yes — and that's where orchestration for Key Characteristics of Microservices With
containers comes into play. Orchestration
Based on all the features of Kubernetes, let's take a look at the key
Orchestration Overview characteristics that a microservice should have to integrate better in
Having to supervise the state of each different container manually, an orchestration ecosystem.
then restart it manually every time it goes down, won't scale as the
volume of containers increases. A number of questions arise: Where IDEMPOTENCE
Every time you deploy a microservice, its behavior should always
do we deploy the container? Are there enough resources available
be the same and should always be expected. One of the main
in the cluster of servers for our needs? An orchestrator assumes
reasons to work with containers in microservices is that you can
the burden of working with multiple containers in a microservices
port the application easily from one environment to another. In
architecture (or any containerized applications) in an automated
Kubernetes, the same portability principle applies — you can use
fashion. Other benefits come into play — like deploying with
the same commands to deploy in different environments. When a
minimum or zero downtime, the capability to automatically scale in
pod terminates due to a memory leak and then starts again, the pod
or scale out the containers or the hosts, or even having distinct ways
should continue working because a new deployment will happen.
to separate the work of developers and operations. (More about these
Think of it as if a VM restart has happened, but much faster, with less
topics later.)
overall impact — and it's done automatically by the orchestrator.

!1234536378*49:;6396143 FAULT TOLER ANCE

!"#$%&$'$( -&'$%&$'
Assume that a microservice can terminate unexpectedly. A node
)*('$% ./0
in the cluster can (and will) go down, and the microservice or the
*,- ./0 system won't suffer, or it will suffer a minimal disruption with a
downtime of only a few seconds. Rollouts in Kubernetes happen
./0
progressively. It starts by creating the new pod. When the new pod
!"#$%&$'$(8&+>$(
+,$%*'-+&( is ready (healthcheck pass), Kubernetes will terminate a pod from
the previous version and replace it with the new one. So, containers
-)*<$8%$<-('%=
are continually going down, and a microservice should expect that.
Ideally, a microservice is stateless to increase its fault tolerance.
There are different orchestration flavors; the three most popular
Kubernetes will make sure that the state of the cluster is the desired
today are Kubernetes, Swarm, and DC/OS. Kubernetes has gained
one, so it will self-recover applications automatically.
a lot of popularity and marketshare, mostly due to its built-in,
ready-to-use features. It's an open-source project written in Go, SINGLE PURPOSE
and it's based on several years of experience that Google has with A microservice should do only one thing, and do it well. In a
its Borg project. Kubernetes combines many of the best practices distributed system, there are going to be common problems for all
from companies that operate at scale, as well as different ideas and microservices — for example, transferring application logs. A library
practices from the larger community. could exist to abstract the logic of managing logs, but it won't be
transparent to the microservice. Every time a new version of the
In Kubernetes, every container is accessible through its DNS endpoint;
library comes, the microservice might need to be redeployed to
its consumers don't have to worry about losing the IP address
use the latest version. But that breaks the rule of having a single
when Kubernetes reschedules a container because of a failure in
purpose in a microservice. There are design patterns that help solve
the cluster or inside the container. Kubernetes will make sure all the
this problem, especially when working with Kubernetes where a
resources from the clusters are evenly distributed when scheduling
single pod could co-schedule multiple containers. More of this is
a container. And using metrics like CPU or memory, Kubernetes can
covered in the design principles section.
also auto-scale the containers or hosts horizontally. Kubernetes rolls
out new changes progressively and can roll them back if something SERVICE DISCOVERY
goes wrong using healthchecks. All these features work whether the When working with orchestrators like Kubernetes, service discovery
application is stateless or stateful. Kubernetes is capable of managing comes built-in. Kubernetes has a service object, which is the way

2 BROUGHT TO YOU IN PARTNERSHIP WITH

 MICROSERVICES ON KUBERNETES

to group pods that share the same group of labels. A consumer of a so clone it with git. From now on, we'll be referencing this application.
microservice won't have to keep a list of the pods to consume them; You can skip the step for building the application and deploy it. When
only having the service endpoint will be enough. Kubernetes will take needed, I'll reference portions of code or commands to run.
care of registering new pods and removing those that can't provide a
Let's start by deploying the application in a Kubernetes cluster.
service. Therefore, a microservice only needs to have the endpoints of
You could use the latest version of Docker for Windows or Mac,
its dependencies, ideally as a configuration variable.
or Minikube. Once the cluster is ready to use, run the following
commands:
CONFIGUR ABLE
Remember that one of the benefits of using containers is portability.
kubectl apply -f ./release/kubernetes-manifests.yaml
One way to make a microservice portable, while building the watch kubectl get pods
container only once, is to make use of configuration variables. In
Kubernetes, this is called a configmap. Configmaps allow for Wait a few minutes until all pods are in the "running" state. Then, to get

decoupling of the configuration from the application. The same the IP address of the frontend service, run the following command:

container image with Kubernetes YAML files can be used in different kubectl get service/frontend-external
environments. Kubernetes translates these configurations into
environment variables or as files inside a container. A microservice Visit the application on your browser to confirm that it's working. It's
should always make its configuration pluggable. that easy.

Lastly, more and more recently, there's been a lot of noise for the
Why Orchestration Matters in Microservices
service meshes in the microservices and Kubernetes ecosystem. The
It's true that microservices add more complexity, but when
term "service mesh," per Istio's documentation, is "used to describe
working with platforms like Kubernetes, much of that complexity
the network of microservices that make up such applications and
is orchestrated away. The state of the applications is defined
the interactions between them." In other words, it defines how
through YAML files. When doing a deployment of a microservice into
microservices can communicate. Istio is a popular open-source
Kubernetes, the operations team has to determine how the service
service mesh platform that runs on top of Kubernetes. It provides
will scale horizontally (at the pod or host level), or how to roll out
a uniform way to operate, control, and secure microservices.
changes in small portions (like 5 percent of the traffic first, then the
Kubernetes and Istio apply some design principles that the industry
rest). Kubernetes will watch the state all the time, and when there's
has been advocating for for years, so let's give them a look.
a mismatch, it will try to make the state match in the cluster. This
reconfiguration will happen even if it requires rescheduling pods or
replacing a Kubernetes worker because one was malfunctioning.
Design Principles for Microservices in an
Orchestrated World
Once you have defined the desired state, the aim should be to have In general, when designing software, simplicity matters. Even though
a simple workflow like the one in the following image. What is inside the system may be complex under the hood, its operation and
the files defining the state at any point could become complicated, development should be relatively simple. The aim when designing a
but it will be worth it if every deploy after development delivers the microservice is to be able to reuse code and avoid adding code that
intended (and same) results every time. it doesn't need. Nonetheless, there are certain design principles that
have become a prerequisite to success.

OBSERVABILIT Y
Having a microservice that emits enough information to make it
observable is key, especially when it's time to troubleshoot problems
in a live environment. For that reason, the microservice code
should have instrumentation to generate metrics that will help to

At the command line, using Docker will look like this: understand the error rate failure, or spot latency problems by sending
traces among all microservices involved in a request, then create a
docker build -t company/microservicea:1.0 .
correlation. This can be as simple as including a header to identify
docker push company/microservicea:1.0
a request that is forwarded through the entire application. A typical
kubectl -f ./kubernetes-manifests/
library for collecting and emitting telemetries like metrics and traces
There's a good microservices demo application developed by Google, is OpenCensus. With OpenCensus, it's possible to export metrics

3 BROUGHT TO YOU IN PARTNERSHIP WITH

 MICROSERVICES ON KUBERNETES

to an external service like Jaeger, Zipkin, Prometheus, and others. application and group them in a pod. Containers in a pod share the
OpenCensus provides libraries for Go, Java, C#, Node.js, C++, Ruby, same network and storage, allowing you to implement a log collector
Erlang/Elixir, Python, Scala, and PHP. (for example) as a sidecar pattern. The application container could
write logs to STDOUT and STDERR, and the collector container will
In the microservices demo, each microservice uses OpenCensus to
read the log data, apply transformations, and send it to a desired
send traces in every request. It's as simple as registering an exporter
centralized location. Log management then becomes transparent to
like this:
the application. If a change needs to happen regarding how to treat
logs, it doesn't interfere with the application.
exporter, err := jaeger.NewExporter(jaeger.Options{
Endpoint: fmt.Sprintf("http://%s", svcAddr),
"13
Process: jaeger.Process{
ServiceName: "productcatalogservice",
}, !""#$%!&$'(
%'(&!$()*
})
#'+
if err != nil { %'##)%&'*

log.Fatal(err)
,-./,012,
}
trace.RegisterExporter(exporter)
AMBASSADOR PATTERN
Another important aspect is that the microservice should provide The ambassador pattern is used as a proxy for the application
endpoints for its health status. In Kubernetes, a liveness probe can be container. It's commonly used for applications that are difficult to
configured to know when a pod should be restarted, perhaps due to a upgrade. An example of this pattern is present when working with
deadlock. And to make sure Kubernetes only sends traffic to healthy Istio. Every time a new pod has to be deployed, Kubernetes injects
pods, there's a readiness probe that can hit an endpoint in the API an Envoy container to route the pod's traffic. This container applies
and confirm that it gets an affirmative answer back. Ideally, these network security policies, circuit breakers, routes, and telemetry. The
endpoints should include logic to verify connections to the database or container application keeps its simplicity, and operators can adapt
run a smoke test for basic functionality. the proxy container as they need to.

Other patterns might be useful, but the ambassador and the sidecar
IMMUTABILIT Y
Once a container has been built, the expectation is that it won't patterns are the most common patterns used in an orchestrated world.

change while running. Containers should attempt to be immutable,

thus insuring that a container will be the same when it restarts. A "-.
microservice will be stateless, and if it depends on the state, it should
do so by using external storage like volumes or databases. Everything
"-.
!""#$%!&$'(
%'(&!$()*
that a container stores at runtime might (i.e. will) get lost. )(+', !""#$%!&$'(
)(+', %'(&!$()*

This native immutability in containers and orchestrators helps ensure

that every change is done through the desired state files definition
DEFINED RESOURCES
(Dockerfile or YAML). Promoting changes through these files, even if It's essential to have a clear definition of what resources a
it's to fix a bug in production, increases overall visibility for the team. microservice will need to run effectively. In Kubernetes, it's possible
No more surprises because someone forgot to apply a fix made in a to define how much memory and CPU a pod needs. When you
test server to a production server. Not knowing what exactly fixed the provide this information to Kubernetes, you will get predictable
problem is no longer an issue because it's now possible to compare behavior on how your pods will run. Furthermore, it's going to be
the current state to the previous state of the desired state files. easier for Kubernetes to determine where to schedule a pod or when
to scale out/in based on telemetry.
Every time a change is required, a new version is created. This provides
the option to do a rollback with a working version in case there's a Defining resources in a pod is as simple as this:
problem with the new one.
requests:
SIDECAR PATTERN cpu: 100m
The sidecar pattern is used to deploy components of the application to memory: 64Mi

extend its behavior as an isolated, decoupled and reusable container. limits:

cpu: 200m memory: 128Mi
Kubernetes will schedule this container in conjunction with the

4 BROUGHT TO YOU IN PARTNERSHIP WITH

 MICROSERVICES ON KUBERNETES

If you want to know more about this topic, Google published a blog Once you successfully collect error logs, you can define alerts and
post with a video explaining how to define the resources a pod will actions, like: "When more than 10% of the request has an error log,
normally need. notify." Initially, you might want to just notify yourself to tune the
threshold and choose a definitive action.
Monitoring in a Microservices/Kubernetes World
In distributed system architectures like microservices, having visibility FAILURE METRICS
from different perspectives will be critical at troubleshooting time. There might be times when errors logs exist even if the request
Many things could happen in a request when there are many parts succeeds. And there are other times when the failure is apparent just
constantly interacting at the same time. The most common method is from looking at the HTTP code in the response. For example, monitor
to write logs to the stdout and stderr streams. for all 5XX error codes, because that means that there has been a
failure in the microservice (like a timeout when calling a database).
For example, a latency problem in the system could exist because
4XX codes will tell you that clients are having problems either because
a microservice is not responding correctly. Maybe Kubernetes is
the request is malformed due to an attack or mistake, or because
restarting the pod too frequently, or perhaps the cluster is out of
there's still a call to a legacy endpoint. When using distributed tracing
capacity and can't schedule any more pods. But for this reason,
like OpenCensus, you can get default metrics for HTTP and gRPC
tools like Istio exist; by injecting a container in every pod, you can
services like latency, request count, and bytes received.
get a pretty good baseline of telemetry. Additionally, when you add
instrumentation with libraries like OpenCensus, you can deeply PERFORMANCE
understand what's happening with and within each service. Latency is a very important metric in microservices. Latency problems
in one service will impact the overall request latency when chaining
All this information will need a storage location, and as a good calls to different microservices. Every call to a microservice should
practice, you might want to have it a centralized location to provide record a trace, which is basically a record of how much time it took
access to anyone in the team — not just for the operations team. to respond. It's possible to add more details to the function level,
including the action, the result, and the pass to the next service. The
CENTR ALIZED LOGGING
You need to have a centralized logging solution due to the volatile hard part is triaging all traces in a request from a client. Usually, a trace

nature of containers and orchestrators. All the information that you ID header has to be sent in every request. If there isn't one, the logging

collect might (i.e. will) get lost. Moreover, a good approach will be to library creates it and it will represent the first trace in a request. As we

have this centralized logging solution isolated from the application saw in a previous section on microservice observability, adding traces

environment, e.g. in a different cluster of servers. You'll reuse this with OpenCensus is simply a matter of including the libraries and

solution for any upcoming system, and it will be easier to triage registering an exporter. Below is a screenshot of a trace in Zipkin.

information from various sources.

There are several storage solutions for logs, like Elasticsearch,

InfluxDB, cAdvisor, and Prometheus. There are also managed
solutions like Scalyr, Datadog, or New Relic.

WHAT TO MONITOR IN MICROSERVICES

Once data has been collected, it's time to keep an eye on it — while
LOGGING LEVELS IN KUBERNETES
not abusing monitoring without purpose. This means that every
Different components in Kubernetes architecture will emit logs in
monitor will have a clear threshold to alert, and this alert will
different ways and locations. Let's take a look at how to treat each of
trigger an action to solve the problem. Kubernetes will try to solve
the different logging levels that exist when working with Kubernetes.
most common problems, as I've explained in previous sections, like
recreating a pod if the liveness probe fails. But there is a certain group APPLICATION
of metrics that, in a microservices world, are mandatory to monitor. As I've said previously, you achieve this logging level by adding
enough instrumentation to the code with libraries like OpenCensus
ERROR LOGS and custom logging information (developer's responsibility). It's
It's important that (depending on the language of the microservice), better if the application writes logs to the stdout and stderr location,
any exception and its context are properly logged. This means that not through HTTP to reduce network traffic.
when adding instrumentation, the only reason a try/catch block will
exist is if an action needs to be taken, not to hide an error from the POD
All containers in a pod write their logs by default to the stdout and
end user. The language Go, for example, understands this problem
stderr . In Kubernetes, you can get the logs with the command
very well by not using exceptions.

5 BROUGHT TO YOU IN PARTNERSHIP WITH

 MICROSERVICES ON KUBERNETES

kubectl logs. This command will print the logs of a container in a stateless, specific workloads can require stateful behavior, such as a
pod; if there's more than one container, then a container name will database. That's also the case when containers in a pod need to share
need to be specified. data between them. A Kubernetes volume solves each of these problems.

Let's say you want the logs from a pod called nginx; you'd use this Volumes in Kubernetes are merely directories that are accessible to
command: the containers in a pod. The container sees this as a directory, but

kubectl logs nginx in reality, it could be a directory stored on the host or on a storage
provider in the cloud.
If you want to get the logs from all the containers, use this:
Let's say, for example, that the Kubernetes cluster lives in Google
kubectl logs nginx --all-containers=true
Cloud Platform (GCP). You first need to create a persistent disk:
Although, I’d recommend you to use cAdvisor to get the logs in a more
gcloud compute disks create --size=100GB --zone=us-
standard way by consuming its API.
central1-a k8s

CONTAINER
With Docker, you have the option to send logs to the stdout and Then, you'd use the persistent disk in a pod by including this:
stderr , too, but you also have the option to send them somewhere
- name: data
else by using logging drivers. You can send logs to syslog, Fluentd,
gcePersistentDisk:
AWS CloudWatch, or hosted services.
pdName: k8s

You can get logs using the Docker command: fsType: ext4

docker logs nginx

Methods for Handling Microservice Monitoring
As a general recommendation, avoid collecting information directly
Or you can get logs from a container using kubectl :
from microservices, as this impacts performance and availability. The
kubectl logs -c ruby microservice goal should be to send data in the background as a detached model.

You can also consume logs from cAdvisor for containers instead of And even if there are problems with the centralized logging storage

using the native tools for Docker or Kubernetes. With cAdvisor, you or with the process of collecting data, the capabillity for each service

have both a UI and an API to centralize logging. to continue providing value shouldn't get lost. There are two ways to
collect logs and metrics — one is as a daemonset and a second one is
ORCHESTR ATOR as a sidecar.
Kubernetes also logs data for itself. For containers that are part of
the kube-system namespace (e.g. kube-dns or kube-proxy), you get DAEMONSET
logs in the same way as with containers or pods. If systemd is present, A daemonset is a user “service” that can create a pod to run in
the rest of the logs (like the kubelet and the container runtime) are some or all of the nodes in a cluster. Every time a new node spins
written to journald. If systemd is not present, Kubernetes writes logs up, Kubernetes will create a pod based on what is defined in a
to the /var/log/ directory. If you're using a managed solution from a daemonset. This pod could include a logging agent with the primary
cloud provider, these logs might not be available directly. purpose of exposing logs or pushing them to a centralized logging
location. The approach to access logs from other containers or the
Managing logs in microservices is an important process; this is the
host is by using volumes.
data that will help the team to solve problems in the most effective
way. Logs are also going to be the source of truth needed to replicate a By utilizing a daemonset, you no longer need to change your
scenario that happened in the past. Therefore, data consistency matters application unless you need to directly change or specify a volume to
in this world, especially in microservices distributed architectures. write the logs.

After a container has been terminated, a cleanup process might get $"%&
!"#
rid of the data stored when it was running. Or a node could go down
because it is no longer needed after an auto-scaling action. Reasons
may vary, and that's why microservices shouldn't depend on state —
they should be stateless. !"#
'()*'+",'
-./&*)+01.#23",,0/,

How to Handle Stateful Containers

Even though microservices — especially with containers — should be

6 BROUGHT TO YOU IN PARTNERSHIP WITH

 MICROSERVICES ON KUBERNETES

SIDECAR Summary Impact of Orchestration on Microservices

A sidecar is about having a container co-scheduled with the Microservices are massively relevant. Existing tools and platforms
application container in the same pod. This new container could that are being developed on top of orchestrators like Kubernetes are
also include a logging agent and collect logs from a shared location helping with adoption, especially in a loosely coupled way. Docker and
between containers in the pod. It could expose the logs or push them Kubernetes help to automate the building, shipping, and running of
to a centralized logging location. microservices in a distributed system. Furthermore, the ecosystem of
tools like Jaeger and Prometheus, as well as libraries like OpenCensus,
It is also possible to combine a sidecar with a daemonset. The sidecar
are helping increase the quality of observability in microservices.
could read logs from the journald, a socket, or a file in a volume and
then share logs with the daemonset by writing logs to stdout or
stderr or any other volume.

!"#$
%"&

'((
4"2
)"*$+,*-. )"11-5$".
/0+./1"2/

%"&
)-*$.+1,3-&
'((
4"2 4"22,*2
)"*$+,*-. )"11-5$".
/0+./1"2/

Written by Christian Melendez, Cloud Architect at Equinix EMEA

Christian is a technologist who started as a software developer and has more recently become a cloud architect
focused on helping companies implement continuous delivery pipelines. Christian is also a technical writer
for topics around Kubernetes, containers, cloud, and DevOps. He’s contributed to the community and specific
companies with talks and workshops, too. You can find more about what he’s doing on cmelendeztech.com or on
Twitter @Christianhxc.

Devada, Inc.
600 Park Offices Drive
Suite 150
Research Triangle Park, NC
DZone communities deliver over 6 million pages each
month to more than 3.3 million software developers, 888.678.0399 919.678.0300
architects, and decision makers. DZone offers something for
Copyright © 2019 DZone, Inc. All rights reserved. No part of this publication
everyone, including news, tutorials, cheat sheets, research
may be reproduced, stored in a retrieval system, or transmitted, in any form
guides, feature articles, source code, and more. "DZone is a or by means electronic, mechanical, photocopying, or otherwise, without
developer’s dream," says PC Magazine. prior written permission of the publisher.

7 BROUGHT TO YOU IN PARTNERSHIP WITH