Ippevent : openshift Introduction

Docker/ Openshift introduction
Openshift Cluster in practice

Agenda
● Infrastructure
● Storage
● Monitoring
● Metrics
● Logs
● Backup
● Security
2

Infrastructure
3
Global Infrastructure

Infrastructure
3x Masters + etcd :
• 4 (v)CPUs
• 16 GB RAM
• 100 GB HDD
4
Masters, etcd cluster and nodes are VMWare ESX Virtual Machines
etcd performances issues since 3.4 !
• Separated from Masters
• Redhat recommends SSD drives
• Upgrade blocked ...
nodes : 2 to n per zone :
• 16 (v)CPUs
• 32 GB RAM
• 600 GB HDD
Infra nodes : 2 to n :
• 4 (v)CPUs
• 16 GB RAM
• 340 GB HDD
Logs :
• High memory, CPU and disk consumption !!!
• Move to Physical Servers (40 Threads, 30 TB HDD)
Global capacity to support a Rolling upgrade and reboot

Infrastructure : capacity planning
5
oc describe node node001
Name: node001
Role:
Labels: kubernetes.io/hostname=node001, ….
Taints: <none>
CreationTimestamp:Tue, 24 May 2016 16:58:10 +0200
Phase:
Conditions:
Type Status ... Reason
Message
---- ------ ----------------- ------------------ ------
Ready True ... ... KubeletReady kubelet is
posting ready status
OutOfDisk False ... ... KubeletHasSufficientDisk kubelet has sufficient disk space
available
MemoryPressure False ... ... KubeletHasSufficientMemory kubelet has sufficient memory
available
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
xxxxxxx zzzz-www-9-123ee 500m (3%) 1 (6%) 256Mi (0%) 1Gi (3%)
xxxxxxx zzzz-wcw-9-123ee 500m (3%) 1 (6%) 1024Mi (0%) 2Gi (3%)
xxxxxxx zzzz-waw-7-123ee 500m (3%) 1 (6%) 512Mi (0%) 1Gi (3%)
xxxxxxx zzzz-wsw-5-123ee 500m (3%) 1 (6%) 256Mi (0%) 4Gi (3%)
xxxxxxx zzzz-qww-4-123ee 500m (3%) 1 (6%) 256Mi (0%) 1Gi (3%)
Allocated resources:
(Total limits may be over 100 percent, i.e., overcommitted.
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
2500m (15%) 5 (31%) 2048Mi (6%) 9Gi (44%)
Capacity:
alpha.kubernetes.io/nvidia-gpu: 0
cpu:
16
memory:
32780448Ki
pods:
110
Allocatable: alpha.kubernetes.io/nvidia-
gpu: 0
cpu:
14
memory:
30731670Ki
pods:
110
System Info:
...
Remember to reserve resources for the node itself

Storage
Criteria :
• Type : NFS, GlusterFS, EBS, GCE, iSCSI, FC, Azure, …
• Access mode : RWO (for one pod), RWX (shared), ROX (read only)
• Capacity : volume size, IOPS, throughput.
StorageClass :
• Storage definition for dynamically provisioned on demand storage
• Can be associated to a specific “storage-admin” role
• Currently only for “cloud” resources
Phase :
• Available / Bound / Released  
/ Failed
Recycling Policy :
• Retain / Recycle
6
Persistent Storage

Storage
7
No SAN, no NAS, no cloud. So … What ? Build your own Storage Cluster !
Two big servers, Pacemaker, DRBD and XFS :
NFS HA Scalable Storage Cluster

Monitoring
Node liveness is done by kubernetes.
Two kinds of checks for pod status :
• Liveness probe : if not alive, pod will be restarted
• Readiness probe : if not ready, traffic won’t be sent to this pod. If
back, traffic will be back again.
8
Internal monitoring of pods and nodes status
SHAME ! Don’t use openshift to hide your
mistakes and incompetence
Openshift won’t do functional testings and monitoring for you.
Also associated to pod limits (see metrics), if limit is exceeded, pod is
killed (and restarted).

Monitoring
• Check the whole infrastructure : memory, cpu, load, storage, ...
• But also processus, docker thinpools, pods, persistent volumes, …
• Centreon + homemade scripts in shell and python with snmp and oc
9
External monitoring

Metrics
• Integrated realtime metrics : heapster, hawkular and Cassandra
• By default : Memory, CPU (millicores) and Network
• View for pods, and aggregated view for services
10
Pods and services metrics
• Metrics enable one great functionality : horizontal pod autoscaling !
• It can be associated with quotas (resource limits). If limits are
exceeded, pod is killed and restarted.
Number of
Nodes
Number of
Pods
Cassandra Storage growth
speed
Cassandra storage growth
per day
Cassandra storage growth per
week
210 10500 500 MB per hour 15 GB 75 GB
990 11000 1 GB per hour 30 GB 210 GB

Logs
Following best practice, pod log go to console/stdout
11
Log can be seen from the pod
oc log mymicser-31-9koc7 -n lovely-project
Don’t forget grep|sort|awk…
Kubetail can be useful
[Sat Jun 17 22:05:13.613952 2017] [:error] [pid 66452] [client 122.138.20.1:36114] xxxx xx xx xx
122.138.20.1 - - [17/Jun/2017:22:05:13 +0200] "GET / HTTP/1.1" 200 29601 xxxx xx xx xx

Logs / EFK
Aggregation
Elasticsearch
A highly-available environment requires at least three replicas of Elasticsearch; each on a different host. Elasticsearch
replicas require their own storage, but an OpenShift Container Platform deployment configuration shares storage
volumes between all its pods. So, when scaled up, the EFK deployer ensures each replica of Elasticsearch has its own
deployment configuration.
• Dedicate node using Node Selector
• Persistent Elasticsearch Storage (local storage)
Fluentd
Fluentd is deployed as a DaemonSet that deploys replicas according to a node label selector (which you can specify
with the deployer parameter fluentd-nodeselector; the default is logging-infra-fluentd).
You can configure Fluentd to send a copy of its logs to an external log aggregator, and not the default Elasticsearch,
using the secure-forward plug-in
Kibana
To access the Kibana console from the OpenShift Container Platform web console
Curator
Curator allows administrators to configure scheduled Elasticsearch maintenance operations to be performed
automatically on a per-project basis.
Ops Cluster
If you set enable-ops-cluster to true for the deployer, Fluentd is configured to split logs between the main
ElasticSearch cluster and another cluster reserved for operations logs
12
$ oc label node/xxx logging-infra-fluentd=true

Logs / EFK
Assumptions:
1. Which application: Apache
2. Bytes per line: 256
3. Lines per second load on application: 1
4. Raw text data → JSON
Baseline (256 characters per minute → 15KB/min)
The implication of keeping all those log need a lot of reflexion on :
• What to keep ?
• How long ?
• How to use them on the long run ?
You really need a project reflexion on the matter, because logs
grow fast and can rapidly become unmanageable.
13
Sometime size matter
Logging Infra Pods Storage Throughput
3 es 1 kibana 1 curator 1 fluentd 6 pods total: 90000 x 86400 = 7,7 GB/day

• Master / nodes backup
→Git / ansible
→etcd backup
• PV backup
→Backula, rsync...
• Project backup
→Images
Docker.io, artifactory, ….
→Object
oc export …,
Project level
https://github.com/vbehar/openshift-git
→PVC
https://github.com/getupcloud/backup
Backup
14
3 layers of back-up

Security
15
Dr. Egon Spengler: Don’t use root in your image
Dr. Peter Venkman: Why?
In Openshift Pods are launched with an arbitrary / random used ID.
Thanks to user namespaces.
Root is prevented by default.
You can’t escape :)
Your images must be adapted. Most docker hub images can’t be reused
as such.

Security
16
Project side : thanks to cgroups and limits
Remember : Petazzoni slides
• cgroups : limits how much you can use (and gauge)
• namespaces : limits what you can see (and therefore use)
• Security context constraints : allow administrators to control
permissions for pods.
• Segregation by namespaces : projects are isolated from each other
• Policies and roles at each level (cluster, project, resources, …)
• Images : Image vulnerability scanning, (Xray / BlackDuck), Base
images
• Network : segregation between projects by default (openvswith,
tags) : if you need to communicate, use services and routes EVEN in
the same project.
• Secrets : because you don’t want to spread hidden things in your
templates or configuration files (credentials, keys, …), openshift
handles them for you.
• selinux : RHEL and docker

Copy an object from project-1 to project-2
oc export xx -n project-1 | oc create -n project-2 -f -
Check container environment variable (first oc get dc) :
for proj in $(oc get project -l type=lamp); do
oc get po -n XXX -l app=mysql, version=5.6
done;
Check container environment variable (first oc get dc) :
oc get dc $DCTOINSPEC -n $DEV_PROJECT_NAME -o json | jq ".spec.template.spec.containers[]?.env"
Check user access within project
for PROJECTID in $PROJECTIDS;
do echo "$PROJECTID" &&
oc describe policyBindings :default -n $PROJECTID | grep -C 1 $USENAME_TO_ADD ;
done
Check image SHA tagged OK
for PROJECTID in $PROJECTIDS;
do echo "$PROJECTID" &&
oc get is $PROJECTID -n $PROJECTID -o json|
jq -r ".status.tags[]|select(.tag == "ok")|.items|
max_by(.created)|.dockerImageReference"
done
Tips
17
Always this scripting

Questions ?

Annexes

Ippevent : openshift Introduction

Recommended

More Related Content

What's hot (20)

Similar to Ippevent : openshift Introduction (20)

Recently uploaded (20)

Ippevent : openshift Introduction