Kubernetes Architecture - beyond a black box - Part 1

Kubernetes - Beyond a Black Box
A humble peek into one level below your running application in production

Copyright © 2017 Hao Zhang All rights reserved.
About The Author
• Hao (Harry) Zhang
• Currently: Software Engineer, LinkedIn, Team Helix @ Data
Infrastructure
• Previously: Member of Technical Staff, Applatix, Worked with
Kubernetes, Docker and AWS
• Previous blog series: “Making Kubernetes Production Ready”
• Part 1, Part 2, Part 3
• Or just Google “Kubernetes production”, “Kubernetes in production”, or similar
• Connect with me on LinkedIn

Motivation
• It’s one of the hottest cluster management project on Github
• It has THE largest developer community
• State-of-art architecture designed mainly by people from Google
who have been working on container orchestration for 10 years
• Collaboration of hundreds of engineers from tens of companies

Introduction
This set of slides is a humble dig into one level below your running
application in production, revealing how different components of
Kubernetes work together to orchestrate containers and present your
applications to the rest of the world.
The slides contains 80+ external links to Kubernetes documentations, blog
posts, Github issues, discussions, design proposals, pull requests,
papers, source code ﬁles I went through when I was working with
Kubernetes - which I think are valuable for people to understand how
Kubernetes works, Kubernetes design philosophies and why these design
came into places.

Outline - Part I
• A high level idea about what is Kubernetes
• Components, functionalities, and design choice
• API Server
• Controller Manager
• Scheduler
• Kubelet
• Kube-proxy
• Put it together, what happens when you do `kubectl create`

Outline - Part II
• An analysis of controlling framework design
• Micro-service Choreography
• Generalized Workload and Centralized Controller
• Interfaces for production environments
• High level workload abstractions
• Strength and limitations
• Conclusion

Overview
A high level idea about what is Kubernetes

– Kubernetes Ofﬁcial Website
“Kubernetes is an open-source system for
automating deployment, scaling, and
management of containerized
applications.”

Manage Apps, Not Machines
Manages Your Apps
• App Image Management
• Where to Run
• When to Get, when to Run, and when to Discard
• App Image Usage
• Image Service: Image lifecycle management
• Runtime: Run image as application

Manage Apps, Not Machines
Manages Things Around Your Apps
• High level workload abstractions
• Pod, Job, Deployment, StatefulSet, etc.
• “If container resembles atom, Kubernetes provides molecules” - Tim Hockin
• Storages
• Data volumes
• Network
• IP, Port mapping, DNS, …
• Monitoring, scaling, communicating, …

etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume
KubeletKube-proxy
Kubernetes Control Plane
KubeletKube-proxy KubeletKube-proxy

A Simpliﬁed Typical Web Service
Load Balancer
Web Server
MemCache
Object Store
(BLOB)
Data Volumes
Database
(SQL / NoSQL)
Backend Sync
Read Services
Backend Write
Sync Services
Backend Write
Async Services
Task Queue
Worker Pool
Browser Client

Web Service On Kube (AWS)
VPC (Virtual Private Cloud)
Subnet
EC2 Instance
EBS etcd
apiserver
Sched
Kubelet
CtlMgr
Docker
EBS
Auto Scaling Group
EC2 Instance
EBS etcd
apiserver
Sched
Kubelet
CtlMgr
Docker
EBS
ElasticLoadBalancer
Subnet
Elastic Load Balancer
EC2 Instance
KubeletDocker
Kube-proxy
EBS
SyncWrite AsyncW
Worker Pool MemCache
EC2 Instance
KubeletDocker
Kube-proxy
EBS
Web SvrSyncRead
AsyncW TaskQ EBS
MemCache
EC2 Instance
KubeletDocker
Kube-proxy
EBS
Web Svr Worker Pool
MemCache DB EBS
EC2 Instance
KubeletDocker
Kube-proxy
EBS
SyncRead SyncWrite
Worker PoolMemCache
DB EBS
EC2 Instance
KubeletDocker
Kube-proxy
EBS
SyncWrite TaskQ EBS
MemCache
DB EBS
EC2 Instance
KubeletDocker
Kube-proxy
EBS
Web SvrAsyncW
TaskQ EBSWorker Pool
DBEBS
Auto Scaling Group
VPC Routing Table
S3 (External Blob, Kubernetes Binaries, Docker Registry)
VPC Gateway
Note: This chart is a rough illustration about how different production components might sit in cloud.
It does not reﬂect any strict production conﬁgurations such as high-availability, fault tolerance, machine load balancing etc.

Kubernetes Architecture
Components, Functionalities and Design Choice

Kubernetes REST API
• Every piece of information in Kubernetes is an API object (a.k.a
Cluster metadata)
• Node, Pod, Job, Deployment, Endpoint, Service, Event,
PersistentVolume, ResourceQuota, Secrets, ConﬁgMap, ……
• Schema and data model explicitly deﬁned and enforced
• Every API object has a UID and version number
• Micro-services can perform CRUD on API objects through REST
• Desired state in cluster is achieved by collaborations of separate
autonomous entities reacting on changes of one or more API
object(s) they are interested in

Kubernetes REST API
• The API Space
• Different API Groups
• Usually a group represents a set of features, i.e. batch, apps, rbac, etc.
• Each group has its own version control
/apis/batch/v1/namespaces/default/jobs/myjob
Root Group
Name
Namespaced
Object
Namespace
Name
API Resource
Kind
API Object
Instance Name
Version
More Readings:
✤ Extending APIs design spec: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/api-machinery/extending-api.md
✤ Customer Resource Documentation: https://kubernetes.io/docs/tasks/access-kubernetes-api/extend-api-custom-resource-deﬁnitions/
✤ Extending core API with aggregation layer: https://kubernetes.io/docs/concepts/api-extension/apiserver-aggregation/
✤ Kubernetes API Spec: https://raw.githubusercontent.com/kubernetes/kubernetes/release-1.7/api/openapi-spec/swagger.json
✤ Kubernetes API Server Deep Dives:
✤ https://blog.openshift.com/kubernetes-deep-dive-api-server-part-1/
✤ https://blog.openshift.com/kubernetes-deep-dive-api-server-part-2/

Kubernetes REST API
• The API Object Deﬁnition
• 5 major ﬁelds: apiVersion, kind, metadata, spec and status
• A few exceptions such as v1Event, v1Binding, etc
• Metadata includes name, namespace, label, annotation, version, etc
• Spec describes desired state, while status describes current state
$ kubectl get jobs myjob --namespace default --output yaml
apiVersion: batch/v1
kind: job
metadata:
name: myjob
namespace: default
spec:
(Describing desired “job” object, i.e. Pod template, how many runs, etc.)
status:
(“job” object current status, i.e. ran # times already, etc.)

Kubernetes Control Plane
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
Persistent Data Volume (Usually SSD)
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node

Kube-apiserver & Etcd
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node

Etcd as metadata store:
• Watchable distributed K-V store
• Transaction and locking support
• Scalable and low memory consumption
• RBAC support
• Uses Raft for leader election
• R/W on all nodes of Etcd cluster
• …
General Functionalities:
• Authentication
• Admission control (RBAC)
• Rate limiting
• Connection pooling
• Feature grouping
• Pluggable storage backend
• Pluggable customer API deﬁnition
• Client tracing and API call stats
For API Objects:
• CRUD Operations
• Defaulting
• Validation
• Versioning
• Caching
For Containers:
• Application Proxying
• Stream Container Logs
• Execute Command in Container
• Metrics
WRITE API Call Process Pipeline:
1. Route through mux to the routine
2. Version conversions
3. Validation
4. Encode
5. Storage backend communication
READ API calls are process roughly
opposite of WRITE API calls.
etcd3
Kube-apiserver
gRPC + protobuf
REST APIs:
HTTP/HTTPS
Compute Node
Usually Co-locate
on same host
SSD

• API Server caching
• Decoding object from Etcd is a huge burden
• Without caching, 50% CPU spent on decoding objects read from etcd,
70% of which is on convert K/V (as of Etcd2)
• Also adds pressure on Golang’s GC (Lots of unused K/V)
• In-memory Cache for READs (Watch, Get and List)
• Decode only when etcd has more up-to-date version of the object

Citation and More Readings:
✤ Etcd! Etcd3!
✤ Discussion about pluggable storage backend (ZK, Consul, Redis, etc): https://github.com/kubernetes/kubernetes/issues/1957
✤ More on “Why etcd?” (Could be biased): https://github.com/coreos/etcd/blob/master/Documentation/learning/why.md
✤ Upgrade to etcd3: https://github.com/kubernetes/kubernetes/issues/22448
✤ Etcd3 Features: https://coreos.com/blog/etcd3-a-new-etcd.html
✤ Kubernetes & Etcd
✤ Kubernetes’ Etcd usage: http://cloud-mechanic.blogspot.com/2014/09/kubernetes-under-hood-etcd.html
✤ Raft - the consensus protocol behind etcd: https://speakerdeck.com/benbjohnson/raft-the-understandable-distributed-consensus-protocol/
✤ Protocol Buffer: https://developers.google.com/protocol-buffers/
✤ Setting up HA Kubernetes cluster: https://kubernetes.io/docs/admin/high-availability/#clustering-etcd
✤ Kubernetes API
✤ Kubernetes API Convention: https://github.com/kubernetes/community/blob/master/contributors/devel/api-conventions.md
✤ API Server Optimizations and Improvements
✤ Discussion about cache: https://github.com/kubernetes/kubernetes/issues/6678, Watch Cache Design Proposal
✤ Discussion about optimizing etcd connection: https://github.com/kubernetes/kubernetes/issues/7451
✤ Optimizing GET, LIST, and WATCH: https://github.com/kubernetes/kubernetes/issues/4817, https://github.com/kubernetes/kubernetes/issues/6564
✤ Serve LIST from memory: https://github.com/kubernetes/kubernetes/issues/15945
✤ Optimizing get many pods: https://github.com/kubernetes/kubernetes/issues/6514

• Kubernetes’ Performance SLO
• API Responsiveness: 99% API calls < 1s
• Pod Startup Time: 99% Pods and their containers start < 5s (With pre-pulled images)
• 5000 Nodes, 150,000 Pods, ~300,000 Containers (1~1.5M API objects)
• Resource (Single master set up for 1000 nodes as of v1.2): 32 Cores, 120G Memory
Citation and More Readings:
✤ Kubernetes 1.2 Scalability: http://blog.kubernetes.io/2016/03/1000-nodes-and-beyond-updates-to-Kubernetes-performance-and-scalability-in-12.html
✤ Kubernetes 1.6 Scalability: http://blog.kubernetes.io/2017/03/scalability-updates-in-kubernetes-1.6.html

Kube-controller-manager
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node

• Framework: Reflector, Store, SharedInformer, and Controller
• One of the optimizations in 1.6 to make it scale from 2k to 5k nodes
• ListAndWatch on a particular resource
• All instance of a resource
• List upon startup and connection broken
• List: get current states
• Watch: get updates
• Push ListAndWatch results into store
• Store is always up-to-date
• In memory read-only cache
• Shared among informers
• Optimizes GET and LIST
• Cache Update Broadcast
• Events: OnAdd, OnUpdate, OnDelete
• Store is at least as fresh as notification
Controller
Reflector
Store
SharedInformer
Event Handlers
Task Queue
• Blocking, MT Safe FIFO Queue
• Additional feature such as delayed
scheduling, failure counts, etc.
• Controller registers event handlers
to the shared informer of the
resource it is interested in
• A single SharedInformer has a pool
of event handlers registered by
different controllers
• Worker pool of a particular resource
• Reconciliation controlling loops
• Action is computed based on
DesiredState and CurrentState

• Reconciliation Controlling Loop: Things will ﬁnally go right
func (c *Controller) Run(poolSize int, stopCh chan struct{}) {
// start worker pool
for i := 0; i < poolSize; i++ {
// runWorker will loop until "something bad" happens. The .Until will
// then rekick the worker after one second
go wait.Until(c.runWorker, 1 * time.Second, stopCh)
}
// wait until we're told to stop
<-stopCh
}
func (c *Controller) runWorker() {
// hot loop until we're told to stop.
for c.processNextWorkItem() {}
}
func (c *Controller) processNextWorkItem() {
obj = c.queue.Get()
// Reconciliation between current and desired state
err = c.makeChange(obj.CurrentState, obj.DesiredState)
if !err {
return
}
// Cool down and retry using delayed scheduling upon error
c.queue.AddRateLimited(obj)
return
}

Shared Resource Informer Factory
Shared Cache
Pod Refl Job Refl Dep Refl Svc Refl Node Refl Ds Refl Rs Refl ……
Pod EventHdl Job EventHdl Dep EventHdl Svc EventHdl Node EventHdl Ds EventHdl Rs EventHdl ……
Pod Informer Job Informer Dep Informer Svc Informer Node Informer Ds Informer Dep Informer Other Informers
Shared Kubernetes Client
Handler Event Dispatching Mesh
Dep Worker Pool
Dep Controller
Queue
Job Worker Pool
Job Controller
Queue
Svc Worker Pool
Svc Controller
Queue
Rs Worker Pool
Rs Controller
Queue
Ds Worker Pool
Ds Controller
Queue
Node Worker Pool
Node Controller
Queue
XYZ Worker Pool
XYZ Controller
Queue

• 28 Controllers as of 1.8, usually deployed as a Pod
• All autonomous micro-services: easily turn on/off
• HA via leader election
• Shared Kubernetes client
• Limit QPS, manage session/HTTPConnection, auth, retry with jittered backoff
• InformerFactory (SharedInformer)
• Reduce API Call, ensure cache consistency, lighter GET/LIST
• Reﬂector / Store / Informer / Controller framework
• Easy, separated development of different controllers as micro services
• Worker Pool
• Individually conﬁgurable agility and resource consumption

More Readings:
✤ Controller Dev Guide: https://github.com/kubernetes/community/blob/master/contributors/devel/controllers.md
✤ Controller Manager Main: https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-controller-manager/app/controllermanager.go
✤ Shared Informer: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/shared_informer.go
✤ Controller Config: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/controller.go
✤ Reflector: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/reflector.go
✤ List and Watch: https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/client-go/tools/cache/listwatch.go
✤ All Controller Logics: https://github.com/kubernetes/kubernetes/tree/master/pkg/controller

Kube-scheduler
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node

Kube-scheduler
• Scheduling Algorithm
• Filter Out Impossible Node
• NodeIsReady: Node should be in Ready state
• CheckNodeMemoryPressure: Node can be over-provisioned,
filter out node with memory pressure
• CheckNodeDiskPressure: Node reporting host disk pressure
should not be considered
• MaxNodeEBSVolumeCount: If a Pod needs a volume, filter out
nodes reach max volume count
• MatchNodeSelector: If Pod has a nodeSelector, it should only
consider nodes with that label
• PodFitsResources: Node should have resource quota
• PodFitsHostPort: Pod’s HostPort should be available
• NoDiskConflict: We should have volume this Pod wants
• NoVolumeZoneConflict: Filter out nodes with zone conflict
• ……
• Give Every Node a Score
• LeastRequestedPriority: Nodes “more idle” is preferred
• BalancedResourceAllocation: Balance node utilization
• SelectorSpreadPriority: Replicas should be spread out
• Affinity/AntiAffinityPriority: “I prefer to stay away / co-locate
from some other categories of Pods”
• ImageLocalityPriority: Node is preferred if it has the image
• ……
Kubernetes Client
Node Filter
Node Ranker
Decision
Watch Unscheduled Pods
Node Candidates
Nodes Sorted By Ranking
POST/v1/bindings
Node with highest ranking

Kube-scheduler
• Default scheduler
• Sequential scheduler
• Some application / topology awareness
• User can define affinity, anti-affinity, node-selector, etc
• Schedule 30,000 Pods onto 1,000 Nodes within 587 Seconds (In v1.2)
• HA via leader election
• Supports multiple schedulers
• Pod can choose the scheduler that schedules it
• Schedulers share same pool of nodes, race condition is possible
• Kubelet can reject Pods in case of conflict, and triggers reschedule
• Easy to write user’s own scheduler or just extend algorithm provider for default
scheduler

Kube-scheduler
More Readings:
✤ Documentations:
✤ Scheduler Overview: https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler.md
✤ Guaranteed scheduling through re-scheduling: https://kubernetes.io/docs/tasks/administer-cluster/guaranteed-scheduling-critical-addon-pods/
✤ Default Scheduling Algorithm: https://github.com/kubernetes/community/blob/master/contributors/devel/scheduler_algorithm.md
✤ Blogs
✤ Scheduler Optimization by CoreOS: https://coreos.com/blog/improving-kubernetes-scheduler-performance.html
✤ Advanced Scheduling and Customized Scheduler After 1.6: http://blog.kubernetes.io/2017/03/advanced-scheduling-in-kubernetes.html
✤ Critical Code Snippets
✤ Scheduling Predicates: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithm/predicates/predicates.go
✤ Scheduling Defaults: https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/scheduler/algorithmprovider/defaults/defaults.go
✤ Designs, Specs, Discussions
✤ Scheduling Feature Proposals: https://github.com/kubernetes/community/tree/master/contributors/design-proposals/scheduling

Kubelet
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node

Kubelet
• Kubernetes’ node worker
• Ensure Pods run as described by spec
• Report readiness and liveness
• Node-level admission control
• Reserve resource, limite # of Pods, etc.
• Report node status to API server (v1Node)
• Machine info, resource usage, health, images
• Host stats (Cgroup metrics via cAdvisor)
• Communicate with container through runtime interface
• Stream logs, execute commands, port-forward, etc
• Usually not run as a container, but a service under Systemd

Kubelet
ContainerRuntime(i.e.Docker)
…
Pod Lifecycle Event Generator (PLEG)
(Pending, Running, Succeeded, Failed, Unknown)
Runtime Status
(RPC)
Pod Cache
Kubernetes
ClientWatched
PodsiNotifyPod Spec
Source
Pod Worker UID1
Pod Worker UID2
Pod Worker UIDN
Runtime Op
(RPC)
Pod Worker Pool
External Pod Event
Sources:
i.e. Container runtime
event stream, cAdvisor
events (Cgroup)
Pod Status
Report
Volume
Management
Container
Prober
Main Sync Loop
Pod Conﬁg Channel
Pod Re-sync Channel
(Clock Tick)
Housekeeping Channel
(Clock Tick)
Cleanup unwanted
Pods, pod workers,
release volumes, etc
PLEG
Channel

Kubelet
• Main Controller Loop
• Listen on events from channels and react
• PLEG (Pod Lifecycle Event Generation)
• Listen and List on external sources, i.e. container runtime, cAdvisor
• Translate container status changes to Pod Events
• Pod Worker
• Interact with container runtime
• Move Pod from current state (v1Pod.status) to desired state
(v1Pod.spec)

Kubelet
• Container Prober
• Probe container liveness and healthiness using methods deﬁned by
spec
• Publish event (v1Event) and put container in CrashLoopBackoff upon
probing failures
• Volume Manager
• Local directory management, device mount/unmount, device
initialization, etc
• cAdvisor
• Container matrix collection and reporting, events about kernel Cgroups

Kubelet
• How Pod is set up
• Sandbox: An isolated environment with resource constraints
• Impl. Based on runtime
• Could be VM, Linux Cgroup, etc
• Kubelet sets the following for docker impl.
• Cgroup, Port mapping, Linux security context
• Holds resolv.conf that is shared by all containers in Pod
• dockerService.makeSandboxDockerConfig()
• App Containers
• Use Sandbox as Cgroup parent
• Share same “localhost”
• dockerService.CreateContainer()
• kubeGenericRuntimeManager. startContainer()
Pod App
Container
Pod
Pod App
Container
Port: 80
curl localhost:80
Pod Sandbox
(Infra Container)
PodIP: 192.168.12.3
PortMapping: 80::33580
curl 192.168.12.3:33580

Kubelet
More Readings:
✤ Kubelet Main: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kubelet.go
✤ Kubelet main function: Kubelet.Run()
✤ Kubelet sync Loop channel event processing: Kubelet.syncLoopIteration()
✤ Kubelet sync Pod: Kubelet.syncPod()
✤ Pod Worker: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/pod_workers.go
✤ Container Runtime Sync Pod: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_manager.go
✤ Sync Pod using CRI: kubeGenericRuntimeManager.SyncPod()
✤ Container Runtime Start Container: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container.go
✤ How kubelet setup container: kubeGenericRuntimeManager.startContainer()
✤ Implementation of PodSandbox using Docker: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/dockershim/docker_sandbox.go
✤ Create Pod sandbox: dockerService.RunPodSandbox()
✤ PLEG Design Proposal: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-lifecycle-event-generator.md
✤ Pod Cgroup Hierarchy: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/node/pod-resource-
management.md#cgroup-hierachy
✤ All Kubelet related design docs: https://github.com/kubernetes/community/tree/master/contributors/design-proposals/node

Kube-proxy
etcd3
Kube-apiserver
Kube-controller-
manager
Kube-scheduler
KubeletKube-proxy
REST/protobuf
REST/protobuf
REST/protobuf
REST/protobuf
RPC/protobuf
Master Node
Minion Node

Kube-proxy
Kubernetes
Client
Kube-proxy
WATCH /api/v1/service
WATCH /api/v1/endpoint
Kernel iptables
Manages iptable rules
• Daemon on every node
• Watch on changes of Service and
Endpoint
• Manipulate iptable rules on host to
achieve service load balancing (one of
the implementations)
More Readings:
✤ v1Service object and different types of proxy implementations: https://kubernetes.io/
docs/concepts/services-networking/service/#virtual-ips-and-service-proxies
✤ iptable proxy implementation: https://github.com/kubernetes/kubernetes/blob/
master/pkg/proxy/iptables/proxier.go
✤ Proxier.syncProxyRules()
✤ Other built-in proxy implementations: https://github.com/kubernetes/kubernetes/
tree/master/pkg/proxy
✤ Some early discussions about Service object and implementations: https://
github.com/kubernetes/kubernetes/issues/1107

Put It Together
What will happen after `kubectl create`

Put it together
$ kubectl create -f myapp.yaml
Deployment nginx-deployment created
$
$ cat myapp.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 9376
• What will happen when you do the
following?

Kubernetes Reaction Chain
Kubenetes API Server & Etcd
Kubectl
Deployment Controller
WATCH v1Deployment, v1ReplicaSet
ReplicaSet Controller
WATCH v1ReplicaSet, v1Pod
Scheduler
WATCH v1Pod,
v1Node
Kubelet-node1
WATCH v1Pod on node1
CRI
CNI
Note: this is a rather high-level idea about Kubernetes reaction chain. It hides some details for illustration purpose and is different than actual behavior

1.POSTv1Deployment
Kubectl
v1Deployment
2. Persist v1Deployment API object
desiredReplica: 3
availableReplica: 0
3.ReﬂectorupdatesDep
Scheduler
WATCH v1Pod,
v1Node
Kubelet-node1
CRI
CNI

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
4. Deployment reconciliation loop
action: create ReplicaSet
5.POSTv1ReplicaSet
v1ReplicaSet
6. Persist v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0
7.ReﬂectorupdatesRS
Scheduler
WATCH v1Pod,
v1Node
Kubelet-node1
CRI
CNI

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
5.POSTv1ReplicaSet
v1ReplicaSet
desiredReplica: 3
availableReplica: 0
8. ReplicaSet reconciliation loop
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
10. Persist 3 v1Pod API objects
status: Pending
nodeName: none
11.ReﬂectorupdatesPod
Kubelet-node1
CRI
CNI

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
5.POSTv1ReplicaSet
v1ReplicaSet
desiredReplica: 3
availableReplica: 0
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
status: Pending
nodeName: none
12. Scheduler assign
node to Pod
13.POSTv1Binding*3
14. Persist 3 v1Binding API objects
name: PodName
targetNodeName: node1
Kubelet-node1
15.Podspecpushedtokubelet
CRI
16. Pull Image, Run Container,
PLEG, etc. Create v1Event and
update Pod status
CNI
v1Binding * 3

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
5.POSTv1ReplicaSet
v1ReplicaSet
desiredReplica: 3
availableReplica: 0
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
status: Pending
nodeName: none
node to Pod
13.POSTv1Binding*3
name: PodName
Kubelet-node1
CRI
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
17. Update 3 v1Pod API objects
status: PodInitializing -> Running
nodeName: node1
v1Event * N
17-1. Several v1Event objects created
e.g. ImagePulled, CreatingContainer,
StartedContainer, etc., controllers, scheduler
can also create events throughout the entire
reaction chain
CNI
v1Binding * 3

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
5.POSTv1ReplicaSet
v1ReplicaSet
desiredReplica: 3
availableReplica: 0
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
status: Pending
nodeName: none
node to Pod
13.POSTv1Binding*3
name: PodName
Kubelet-node1
CRI
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
nodeName: node1
v1Event * N
reaction chain
CNI
18.ReﬂectorupdatesPods
Update RS when ever a Pod enters
“Running” state
v1Binding * 3

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
5.POSTv1ReplicaSet
v1ReplicaSet
desiredReplica: 3
availableReplica: 0
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
status: Pending
nodeName: none
node to Pod
13.POSTv1Binding*3
name: PodName
Kubelet-node1
CRI
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
nodeName: node1
v1Event * N
reaction chain
CNI
“Running” state
20.PATCHv1ReplicaSet
21. Update v1ReplicaSet API object
desiredReplica: 3
availableReplica: 0 -> 3
v1Binding * 3

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
5.POSTv1ReplicaSet
v1ReplicaSet
desiredReplica: 3
availableReplica: 0
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
status: Pending
nodeName: none
node to Pod
13.POSTv1Binding*3
name: PodName
Kubelet-node1
CRI
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
nodeName: node1
v1Event * N
reaction chain
CNI
“Running” state
desiredReplica: 3
Update Dev status with RS status changes
v1Binding * 3

1.POSTv1Deployment
Kubectl
v1Deployment
desiredReplica: 3
availableReplica: 0
5.POSTv1ReplicaSet
v1ReplicaSet
desiredReplica: 3
availableReplica: 0
action: create Pod
9.POSTv1Pod*3
Scheduler
WATCH v1Pod,
v1Node
v1Pod *3
status: Pending
nodeName: none
node to Pod
13.POSTv1Binding*3
name: PodName
Kubelet-node1
CRI
update Pod status
17.PATCHv1Pod*3
17-1.PUTv1Event
nodeName: node1
v1Event * N
reaction chain
CNI
“Running” state
desiredReplica: 3
24.PATCHv1Deployment
Update Dev status with RS status changes
25. Update v1Deployment API object
desiredReplica: 3
v1Binding * 3

Kubernetes Architecture - beyond a black box - Part 1

Recommended

More Related Content

What's hot (20)

Viewers also liked (12)

Similar to Kubernetes Architecture - beyond a black box - Part 1 (20)

Recently uploaded (20)

Kubernetes Architecture - beyond a black box - Part 1