What's new in Kubernetes

Google Cloud Platform
What’s new in Kubernetes
Docker & Bay Area OpenSource meetup
February 16, 2016
Daniel Smith <dbsmith@google.com>
Senior Software Engineer

Kubernetes
Greek for “Helmsman”; also the root of the
words “governor” and “cybernetic”
• Runs and manages containers
• Inspired and informed by Google’s experiences
and internal systems
• Supports multiple cloud and bare-metal
environments
• Supports multiple container runtimes
• 100% Open source, written in Go
Manage applications, not machines

Google has been developing
and using containers to
manage applications for
over 10 years.
Images by Connie Zhou

Review: What’s old in Kubernetes?

kubelet
UI
kubeletCLI
API
users master nodes
The 10000 foot view
etcd
kubelet
scheduler
controllers
apiserver

Pods
Small group of containers & volumes
Tightly coupled
The atom of scheduling & placement
Shared namespace
• share IP address & localhost
• share IPC, etc.
Managed lifecycle
• bound to a node, restart in place
• can die, cannot be reborn with same ID
Example: data puller & web server
Consumers
Content
Manager
File
Puller
Web
Server
Volume
Pod

Volumes
Very similar to Docker’s concept
Pod scoped storage
Share the pod’s lifetime & fate
Support many types of volume plugins
• Empty dir (and tmpfs)
• Host path
• Git repository
• GCE Persistent Disk
• AWS Elastic Block Store
• Azure File Storage
• iSCSI
• Flocker
• NFS
• GlusterFS
• Ceph File and RBD
• Cinder
• FibreChannel
• Secret, ConfigMap, DownwardAPI
• Flex (exec a binary)
• ...

ReplicationControllers

ReplicationControllers
A simple control loop
Runs out-of-process wrt API server
Has 1 job: ensure N copies of a pod
• if too few, start some
• if too many, kill some
• grouped by a selector
Cleanly layered on top of the core
• all access is by public APIs
Replicated pods are fungible
• No implied order or identity
ReplicationController
- name = “my-rc”
- selector = {“App”: “MyApp”}
- podTemplate = { ... }
- replicas = 4
API Server
How
many?
3
Start 1
more
OK
How
many?
4

Services

Services
A group of pods that work together
• grouped by a selector
Defines access policy
• “load balanced” or “headless”
Gets a stable virtual IP and port
• sometimes called the service portal
• also a DNS name
VIP is managed by kube-proxy
• watches all services
• updates iptables when backends change
Hides complexity - ideal for non-native apps
Client
Virtual IP

External Services
Services IPs are only available inside the
cluster
Need to receive traffic from “the outside
world”
Builtin: Service “type”
• NodePort: expose on a port on every node
• LoadBalancer: provision a cloud load-balancer
DiY load-balancer solutions
• socat (for nodePort remapping)
• haproxy
• nginx

What’s new in Kubernetes?

Ingress (L7)
Services are assumed L3/L4
Lots of apps want HTTP/HTTPS
Ingress maps incoming traffic to backend
services
• by HTTP host headers
• by HTTP URL paths
HAProxy, NGINX, AWS and GCE
implementations in progress
Now with SSL!
Status: BETA in Kubernetes v1.2
URL Map
Client

Service-foo: 10.0.0.1 Service-bar
10.0.0.2
api.company.com
24.7.8.9
http://api.company.com/foo http://api.company.com/bar
Ingress API
Ingress (L7)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: test
spec:
rules:
- host: k8s.io
http:
paths:
- path: /foo
backend:
serviceName: fooSvc
servicePort: 80
- path: /bar
backend:
serviceName: barSvc
servicePort: 80
fooSvc barSvc
http://k8s.io/foo http://k8s.io/bar
Ingress (L7)

kind: Ingress
metadata:
name: test
spec:
rules:
- host: asdf.io
http:
paths:
- backend:
serviceName:
qwertySvc
servicePort: 80
- host: aoeu.io
http:
paths:
- backend:
serviceName:
dvorakSvc
servicePort: 80
qwertySvc dvorakSvc
http://asdf.io/* http://aoeu.io/*
Ingress (L7)

Ingress Object Ingress Controller
● GCE
● HAProxy
● ...
Ingress (L7)

kube-proxy

iptables kube-proxy
iptables
kube-proxy apiserver
Node X

iptables
Node X
watch
services &
endpoints
iptables kube-proxy

iptables
Node X
kubectl run ...
watch
iptables kube-proxy

iptables
Node X
schedule
watch
iptables kube-proxy

iptables
Node X
watch
kubectl expose ...
iptables kube-proxy

iptables
Node X
new
service!
update
iptables kube-proxy

iptables
Node X
watch
configure
iptables kube-proxy

iptables
Node X
watch
VIP
iptables kube-proxy

iptables
Node X
new
endpoints!
update
VIP
iptables kube-proxy

iptables
Node X
VIP
watch
configure
iptables kube-proxy

iptables
Node X
VIP
watch
iptables kube-proxy

iptables
Node X
VIP
watch
Client
iptables kube-proxy

ConfigMaps (and Secrets)

ConfigMaps
Problem: how to manage app configuration
• ...without making overly-brittle container images
12-factor says config comes from the
environment
• Kubernetes is the environment
Manage config via the Kubernetes API
Inject config as a virtual volume into your Pods
• late-binding, live-updated (atomic)
• also available as env vars
Status: GA in Kubernetes v1.2
node
API
Pod Config
Map

Secrets
Problem: how to grant a pod access to a
secured something?
• don’t put secrets in the container image!
12-factor says config comes from the
environment
• Kubernetes is the environment
Manage secrets via the Kubernetes API
Inject secrets as virtual volumes into your Pods
• late-binding, tmpfs - never touches disk
• also available as env vars
node
API
Pod Secret

Rolling updates

Rolling Updates
- replicas: 3
- selector:
- app: MyApp
- version: v1
Service
- app: MyApp

Rolling Updates
- replicas: 3
- selector:
- app: MyApp
- version: v1
Service
- app: MyApp
# Update pods of frontend-v1 using new replication controller data in frontend-v2.json.
$ kubectl rolling-update frontend-v1 -f frontend-v2.json
# Update pods of frontend-v1 using JSON data passed into stdin.
$ cat frontend-v2.json | kubectl rolling-update frontend-v1 -f -
# Update the pods of frontend-v1 to frontend-v2 by just changing the image, and switching
the
# name of the replication controller.
$ kubectl rolling-update frontend-v1 frontend-v2 --image=image:v2
# Update the pods of frontend by just changing the image, and keeping the old name
$ kubectl rolling-update frontend --image=image:v2

- replicas: 3
- selector:
- app: MyApp
- version: v1
- replicas: 0
- selector:
- app: MyApp
- version: v2
Service
- app: MyAppRolling Updates

- replicas: 3
- selector:
- app: MyApp
- version: v1
- replicas: 1
- selector:
- app: MyApp
- version: v2
Service

- replicas: 2
- selector:
- app: MyApp
- version: v1
- replicas: 1
- selector:
- app: MyApp
- version: v2
Service

- replicas: 2
- selector:
- app: MyApp
- version: v1
- replicas: 2
- selector:
- app: MyApp
- version: v2
Service

- replicas: 1
- selector:
- app: MyApp
- version: v1
- replicas: 2
- selector:
- app: MyApp
- version: v2
Service

- replicas: 1
- selector:
- app: MyApp
- version: v1
- replicas: 3
- selector:
- app: MyApp
- version: v2
Service

- replicas: 0
- selector:
- app: MyApp
- version: v1
- replicas: 3
- selector:
- app: MyApp
- version: v2
Service

- replicas: 3
- selector:
- app: MyApp
- version: v2
Service

Deployments

Deployments
Rolling update is too imperative
Deployment manages RC changes for you
• stable object name
• updates are done server-side rather than client
• kubectl edit or kubectl apply is all you need
Aggregates stats
Can have multiple updates in flight
...

Deployments
...
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80

Jobs
Run-to-completion, as opposed to run-forever
• Express parallelism vs. required completions
• Workflow: restart on failure
• Build/test: don’t restart on failure
Aggregates success/failure counts
Built for batch and big-data work
...

Start Finish
kind: Job
metadata:
name: ffmpeg
spec:
selector:
matchLabels:
app: ffmpeg
template:
metadata:
labels:
app: ffmpeg
spec:
containers:
- name: ffmpeg
image: ffmpeg
restartPolicy: OnFailure
Jobs

Start Finish
kind: Job
metadata:
name: ffmpeg
spec:
selector:
matchLabels:
app: ffmpeg
# run 5 times before done
completions: 5
...
Jobs

Start Finish
kind: Job
metadata:
name: ffmpeg
spec:
selector:
matchLabels:
app: ffmpeg
# run 5 times before done
completions: 5
parallelism: 2
...
Jobs

DaemonSets

DaemonSets
Problem: how to run a Pod on every node
• or a subset of nodes
Similar to ReplicationController
• principle: do one thing, don’t overload
“Which nodes?” is a selector
Use familiar tools and patterns
Pod

Graceful Termination

Graceful Termination
Give pods time to clean up
• finish in-flight operations
• log state
• flush to disk
• 30 seconds by default
Catch SIGTERM, cleanup, exit ASAP
Pod status “Terminating”
Declarative: ‘DELETE’ manifests as an object
field in the API

HorizontalPodAutoscalers

HorizontalPodAutoScalers
Automatically scale ReplicationControllers to a
target utilization
• CPU utilization for now
• Probably more later
Operates within user-defined min/max bounds
Set it and forget it
...
Stats

Cluster Auto-Scaling

Cluster Scaling
Add nodes when needed
• e.g. CPU usage too high
• nodes self-register with API server
Remove nodes when not needed
• e.g. CPU usage too low
Status: Works on GCE, need other
implementations
...

New and coming soon
• Cron (scheduled jobs)
• Custom metrics
• “Apply” a config (even more declarative)
• Interactive containers
• Bandwidth shaping
• Third-party API objects
• Scalability: 1000 nodes, 100+ pods/node
• Performance
• Machine-generated Go clients (less deps!)
• Volume usage stats
• Multi-zone (AZ) support
• Multi-scheduler support
• Node affinity and anti-affinity
• Multi-cluster federation
• API federation
• More volume types
• Private Docker registry
• External DNS integration
• Volume classes and auto-provisioning
• Node fencing
• DiY Cloud Provider plugins
• More container runtimes (e.g. Hyper)
• Better auth{n,z}
• Network policy (micro-segmentation)
• Big data integrations
• Device scheduling (e.g. GPUs)

Kubernetes status & plans
Open sourced in June, 2014
• v1.0 in July, 2015
• v1.1 in November, 2015
• v1.2 ... soon!
Google Container Engine (GKE)
• hosted Kubernetes - don’t think about cluster setup
PaaSes:
• RedHat OpenShift, Deis, Stratos
Distros:
• CoreOS Tectonic, Mirantis Murano (OpenStack),RedHat
Atomic, Mesos
Hitting a ~3 month release cadence

The Goal: Read-write open source
Containers are a new way of working
Requires new concepts and new tools
Google has a lot of experience...
...but we are listening to users!
Your input does make a difference!

The Goal: Read-write open source

Kubernetes is Open
- open community
- open design
- open source
- open to ideas
http://kubernetes.io
https://github.com/kubernetes/kubernetes
slack: kubernetes
twitter: @kubernetesio

What's new in Kubernetes

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to What's new in Kubernetes (20)

Recently uploaded (20)

What's new in Kubernetes