Kubernetes &

Microservices
Monitoring

20th August 2019

Christopher Chai
Solutions Architect, Dynatrace
Before we start…

• Setup Wifi
• SSID:
• Password
• Your Dynatrace environment
• Got the invitation email?
• Google Cloud Platform (GCP) account
• Sign up for free $300 free credit

Confidential 2
Agenda

• Introduction
• About Monoliths & Microservices
• Kubernetes concepts and architecture
• Hands-on Lab
• Install k8s cluster on GKE
• Install Dynatrace OneAgent Operator
• Setup Kubernetes UI integration
• Annotations & labels
• Process and service renaming
• Canary release – comparing the performance Confidential 3
Monolith vs Microservices

Confidential 4
Limitations of monolithic applications

UI Agility?
Scalability?
Ship. Acct.
DevOps cycle?
Mgmt. Billing Availability, fault tolerance, resiliency?

DB

Confidential 5
Limitations lead to the rise of Microservices

• Agility - Scope changes can be done in one microservice - other

UI
micro services are not impacted from these changes

Ship. Acct. • Scalability - Individual components can scale as needed

DB DB
• DevOps Cycle - Since each component operates independently,
continuous delivery cycle reduces

Mgmt. Billing
• Availability, Fault Tolerance, and Resiliency
DB DB

Confidential 6
From VMs to Containers

Containerized applications

Confidential 7
What’s Kubenetes?

• Open source container orchestration system

• Literally means helmsman in Greek

Confidential 10
Kubenetes Architecture

• Cluster
• Master nodes
• “Worker” nodes

• Pods
• Docker containers
• Docker registry
• Pod Manifest (YAML)

Confidential 11
Key Concepts - Nodes

• Worker machine in Kubernetes

• May be a VM or physical machine,
depending on the cluster.
• Managed by the control plane (master).
• Node components:
• kubelet
• kube-proxy
• Container Runtime Engine

Confidential 12
Key Concepts - Pods

• Atomic unit or smallest “unit of

work”of Kubernetes.

• Pods are one or MORE containers

that share volumes and
namespace.

• They are also ephemeral

Confidential 13
Key Concepts - Services

• Unified method of accessing the exposed

workloads of Pods.
• Think of it as an internal load balancer to
your pods.
• You have 3 HTTP pods scheduled? Create a
service, reference the pods, and (internally)
it will load balance across the three.
• How it is implemented depends on the cloud
provider or the on-prem config
• Not ephemeral <service name>.<namespace>.svc.cluster.local
Confidential 14
Let’s get started!

Confidential 19
Accessing your Dynatrace environment

• You should have received

an invitation email with
your environment details
• Check with the
Dynatracers around you if
you did not receive it

Confidential 20
 If you don’t have the invitation email…

1. Let’s sign up for a Dynatrace SaaS account.

2. If you do have an existing account, go to

• https://account.dynatrace.com
• Click on Start new trial

3. If you don’t have an existing account, go to

• Signup at https://www.dynatrace.com/trial/
• Choose AP Southeast Sydney

When chosen, your instance will spun up immediately and you will have access
to your instance.
Confidential 21
Tweaking a little bit…

• Go to Settings > Processes and Containers > Container Monitoring

• Enable Containerd containers

Confidential 22
Resources

• Lab GitHub - https://bit.ly/2HfxPcG

• Cheatsheet - https://bit.ly/2TN3HL0

• Some steps are only valid if you are registered for PerformGo! SG HOT
session.
• Please always refer to this deck for instructions.

Confidential 23
Lab 1

Confidential 24
Setting up your GKE

• If you don’t have an existing account, go to

• https://cloud.google.com/free
• Click on Get started for free
• Use your existing Google email account to sign in or create new one

• If you do have an existing account, go to

• https://console.cloud.google.com/home/
• Just login with your credential

Confidential 25
Setting up your GKE

• Enable Kubernetes Engine API

• Go to API & Services > Library
• Search for Kubernetes Engine API
• Click on ENABLE

Confidential 26
Setting up your GKE

• Activate Cloud Shell

• Click on the Terminal Icon on the
top right
• A Cloud-based Terminal will appear
at the bottom

Confidential 27
Setting up your GKE

• Set region zone to Australia Southeast

• gcloud config set compute/zone australia-southeast1-a

Confidential 28
Setting up your GKE

• Create your GKE Cluster named performgo2019 using COS with containerd
• gcloud container clusters create performgo2019 --image-
type=cos_containerd --cluster-version=1.13

Confidential 29
Setting up your GKE

• You have a running GKE Cluster!

• Run kubectl get nodes to check the number of nodes

Confidential 30
Lab 2

Confidential 31
Install Dynatrace OneAgent Operator

• Download codebase from Git repository

• git clone https://github.com/nodnarboen/HOT-k8s.git

Confidential 32
Install Dynatrace OneAgent Operator

• THIS STEP IS VALID ONLY FOR DO IT YOURSELF

• Follow the steps as outlined:
• https://www.dynatrace.com/support/help/technology-support/cloud-platforms/
google-cloud-platform/google-kubernetes-engine/installation-and-operation/fu
ll-stack/deploy-oneagent-on-google-kubernetes-engine-clusters/
• TIPS: Skip kubectl -n dynatrace logs -f
deployment/dynatrace-oneagent-operator
• TIPS: Create cr.yaml by copying the snippet provided in the link above
instead of downloading from GitHub

Confidential 33
Install Dynatrace OneAgent Operator

• THIS STEP IS VALID ONLY FOR PERFORMGO! HOT SESSION

• We have automated the steps with the use of a script (install-oneagent-
operator.sh)
• Before running it we just need 3 parameters
• AttendeeID
• API Token
• PaaS Token

• Open up a text document and copy them down!

Confidential 34
Install Dynatrace OneAgent Operator

• THIS STEP IS VALID ONLY FOR PERFORMGO! HOT SESSION

• AttendeeID
• Can be found from your Dynatrace environment URL. Look at the browser address
bar:
• Format is pgo19-SG-k8s-[AttendeeID]
• In the example above, it’s “238”

Confidential 35
Install Dynatrace OneAgent Operator

• API Token
• Go to your Dynatrace environment >
Settings > Integration > Dynatrace API
> Generate Token
• Enter a token name
• Make sure access scope enabled for
Access problem and event feed,
metrics and topology
• Click Generate
• Copy the token
Confidential 36
Install Dynatrace OneAgent Operator

• PaaS Token
• Go to your Dynatrace environment >
Settings > Integration > Platform as a
Service > Generate Token
• Enter a token name
• Click Generate
• Copy the token

Confidential 37
Install Dynatrace OneAgent Operator

• THIS STEP IS VALID ONLY FOR PERFORMGO! HOT SESSION

• Install OneAgent Operator
• cd HOT-k8s
• ./install-oneagent-operator.sh

• Enter the details as per screen prompt

• AttendeeID
• API Token
• PaaS Token

Confidential 38
Install Dynatrace OneAgent Operator

• Verify Installation
• Once installed, the cluster nodes
will appear straightaway!
• Click on Hosts
• Alternatively, run kubectl –n
dynatrace get pods

Confidential 39
Full stack monitoring, auto-discovery of topology

Confidential 40
Lab 3

Confidential 41
Setting up Kubernetes Dashboard

• THIS STEP IS VALID ONLY FOR PERFORMGO! HOT SESSION

• Install ActiveGate
• ./setup-activegate.sh
• When prompt for create directory, enter “y”
• No need to enter passphrase
• Look for “Installation finished successfully” message

Confidential 42
Setting up Kubernetes Dashboard

• THIS STEP IS VALID ONLY FOR DO IT YOURSELF

• Install ActiveGate
• Create a Linux VM on your Google Cloud
• Go to your Dynatrace SaaS > Deployment Status > Install ActiveGate > Linux
• Follow the instructions (We will install the ActiveGate on the Linux VM)
• Look for “Installation finished successfully” message

Confidential 43
Setting up Kubernetes Dashboard

• Verify ActiveGate
• Go to Dynatrace > Deployment Status > Dynatrace ActiveGates

Confidential 44
Setting up Kubernetes Dashboard

• Setup the K8S Overview Dashboard

• ./setup-k8s-ui.sh
• Copy the k8s API URL and Bearer token

Confidential 45
Setting up Kubernetes Dashboard

• Go to Dynatrace > Kubernetes

Confidential 46
Setting up Kubernetes Dashboard

Any name will do, as long as it makes

sense to you

k8s API URL here

Bearer token here

Last but not least, click Connect!

Confidential 47
Setting up Kubernetes Dashboard

• Go to Dynatrace > Kubernetes

Confidential 48
Setting up Kubernetes Dashboard

Confidential 49
Lab 4

Confidential 50
Setting up a demo application

• Run the sample app sock-shop

• ./deploy-sockshop.sh

Confidential 51
Kubernetes Labels and Annotation

• sock-shop metadata
• vi /HOT-k8s/manifests/sockshop-
app/production/front-end.yml

Confidential 52
Kubernetes Labels and Annotation

• Creating pod service account to get metadata

• kubectl create rolebinding serviceaccounts-view --clusterrole=view --
group=system:serviceaccounts:production --namespace=production

• kubectl create rolebinding serviceaccounts-view --clusterrole=view --

group=system:serviceaccounts:dev --namespace=dev

• ./recycle-sockshop-frontend.sh

Confidential 53
Kubernetes Labels and Annotation

• Go to Dynatrace > Transaction

& Services -> Click on front-
end (with label front-
Label
end.stable) -> Click on
Properties and tags -> Process
group …. server.js xxxxxx
• Validate the changes in
Dynatrace Annotation

Confidential 54
Accessing Sock Shop!

• Get the IP address and port

• Kubectl describe svc front-end –n production

Confidential 55
Accessing Sock Shop!

Check on Sock Shop on

your web browser by

http://
[YOUR_IP_ADDRESS]:8
080

Confidential 56
Full stack monitoring

Notice your user actions are

reflected in real time in the
topology

Confidential 57
Adding Environment Variables

Full stack monitoring from user actions

to database Confidential 58
 Problem history

Infographic and Service

overview Hotspot Analysis

Drilldown into Drilldown into

Service Details Dependencies

Event logger
Confidential 59
 Problem history
Process Groups – Single PGI View

Infographic and process Process availability

overview
Event logger
Log files for process

Dynamic measure display

Confidential 60
 Setting up Kubernetes Dashboard Problem history
Hosts – what do we get?

Infographic and host

Host availability
overview

Detected Processes

Dynamic measure display

Event logger

Log files for host

Confidential 61
Lab 5

Confidential 62
Adding Environment Variables

• Add some environment variables

• vi
/HOT-k8s/manifests/sockshop-
app/production/front-end.yml

• After saving, apply the change

• kubectl apply -f front-end.yml
• cd /HOT-k8s/
• ./recycle-sockshop-frontend.sh

Confidential 63
Adding Environment Variables

Confidential 64
Lab 6

Confidential 65
Process Renaming
• Go to Settings > Processes and containers
> Process group naming
• Provide a name to the rule
• Enter this format:
• k8s-
{ProcessGroup:Kubernetes:pipeline.projec
t}.{ProcessGroup:KubernetesNamespace}.
{ProcessGroup:KubernetesContainerName
}

• Conditions drop down > Kubernetes

namespace > exists
• Click on Create Rule > Save Confidential 66
Process Renaming

• Validate the change in Transactions &

Services

Confidential 67
Service Renaming

• Go to Settings > Server-side service

monitoring > Service naming rules
• Provide a name to rule
• Click Add a new rule
• Enter this format:
• {Service:DetectedName}.
{ProcessGroup:KubernetesNamespace}

• Conditions drop down > Kubernetes

namespace > exists
• Click on Create Rule > Save
Confidential 68
Service Renaming

• Validate the change in Transactions &

Services

Confidential 69
Lab 7

Confidential 70
Process Detection for Canary Deployment

• Deploy the canary release

• ./deploy-canary.sh

Confidential 71
Process Detection for Canary Deployment

• Verify that both stable and canary releases running for the front-end service
• kubectl get pods -n production -o wide

Confidential 72
Process Detection for Canary Deployment

• Wait 1-2 minutes, go to Transactions &

Services
• We should see 2 services in production,
one for stable and one for canary

Confidential 73
Process Detection for Canary Deployment

• Go to Settings > Processes and Containers

> Process group detection
• Click Add detection rule
• Select Use a process property

Confidential 74
Process Detection for Canary Deployment

Confidential 75
Process Detection for Canary Deployment

• Let’s apply the process detection rule:

• ./recycle-sockshop-frontend.sh
• kubectl get deployments -n production -l tier=frontend

Confidential 76
Process Detection for Canary Deployment

• Within Dynatrace, you can see the Process

Groups have been merged

Confidential 77
Merge the Services

• The services are still detected as individual services

• Let’s merge them:
• Settings > Server-side service monitoring > Merge Service monitoring > Create merged
service

Confidential 78
Merge the Services

• Validate via Transactions & services

Confidential 79
Compare Stable versus Canary Release

• With the services merged as one, you can now compare performance between two
• Go to Create Chart
• Choose Response time – Server and select Service Instance as Dimension splitting

Confidential 80
Compare Stable versus Canary Release

Confidential 81
dynatrace.com

Confidential 82