Continuous Deployment with Kubernetes, Docker and GitLab CI

Continuous Deployment
with Kubernetes, Docker
and GitLab CI
@alexander_kiel 
Clojure Berlin 2016

Outline
• Continuous Deployment why?
• Docker
• Kubernetes
• Sample Clojure Service
• Deploy with GitLabCI

Continuous Deployment
• What do we want?
• Increase responsiveness
• Decrease time to market
• Gain conﬁdence by deploying often in small amounts
• How to achieve that?
• Automate everything
• Always deploy the master into production
• Use feature toggles when needed

Simple Git Workflow
• Works for in-house apps
• not for libs or shipping apps
• No versions, no tags, just SHA’s
• Latest commit on master is always
deployed to production
• Feature/fix branches are merged
when ready
master
feature/fix
branches
1ebb95d
be61dda
6e4010d

Docker
• Like VM’s but much more light-weight and shippable
• Runs on Linux, executes processes in an isolated environment
(resource limitation, ﬁlesystem, network)
• Container principle: Can contain everything, but looks the
same from the outside
• A container platform can run every container
• Developers have max. freedom what to do
• In contrast: PaaS like Heroku - has to support the language

Kubernetes
• Container runtime platform
• Originally designed by Google - now Open Source
• One of the most active projects on GitHub - 20,000
stars, 40,000 commits, 15,000 issues, 200 releases
• Alternatives: Apache Mesos, Docker Swarm (lacks
features)

Kubernetes Architecture
k8s-master-1
k8s-master-2
k8s-master-3
load-balancer-1
load-balancer-2
DNS RR
k8s-worker-1
proxy
app-1
k8s-worker-2
proxy
app-2
k8s-worker-n
proxy
app-k
etcd cluster 
quorum
HAProxy
• Runs on VMware ESX
• CoreOS Linux
• Single YAML ﬁle as conﬁguration
• Everything in containers

Kubernetes - Pods
• A Pod is a deployable unit in
Kubernetes
• Pods can contain multiple
containers
• Containers inside a Pod share
on port space, can use
localhost and can
communicate via IPC and
shared memory
• Idea: one process per
container - many cooperating
processes in one Pod
apiVersion: v1 
kind: Pod 
metadata: 
name: <pod-name> 
labels: 
<key>: <value> 
spec: 
containers: 
- name: <container-name> 
image: <container-image> 
ports: 
- containerPort: 80 
env: 
- name: <key> 
value: <value>

Kubernetes - Deployments
• A Deployment ensures that
certain number of Pods are
always running
• It consists of a Pod template
and the number of replicas
• It supports hot-redeployments
by changing parts of the Pod
template
• Horizontal scaling is possible
apiVersion: extensions/v1beta1 
kind: Deployment 
metadata: 
name: <deployment-name> 
spec: 
replicas: 2 
template: 
metadata:
labels:
<key>: <value>
spec:
containers:
- name: <container-name>
image: <container-image>
ports:
- containerPort: 80
env:
- name: <key>
value: <value>

Kubernetes - Services
• Kubernetes uses an overlay
network to provide different address
spaces (we use ﬂannel)
• Every Pod has an IP address - but it
changes every time one is created
• Services provide a stable IP
address for groups of Pods
• Service names are resolvable by an
internal DNS
• Service selectors are used to match
Pods according to there labels
apiVersion: v1
kind: Service
metadata:
name: clojure-berlin-2016
labels:
app: lens
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
selector:
service: clojure-
berlin-2016

Kubernetes - External Access
• Kubernetes networks are internal
only
• External access through load
balancers necessary
• Certain Platforms like Google
Compute Engine provide load
balancer integration with Kubernetes
• We have our own solution as a
combination of HAProxy and
Kubernetes NodePort
• Kubernetes Services with type
NodePort are exposed on every
worker under a certain port
frontend http
bind 0.0.0.0:80
mode http
option httplog
acl host_clj hdr(host)
clj.<domain>
use_backend clj if host_clj
backend clj
mode http
balance roundrobin
option httplog
server worker-1 <ip>:32599 check
server worker-2 <ip>:32599 check

Deployment Lifecycle
GitLab CI
Source Code
build
test
Kubernetes
Test
Cluster
Kubernetes
Prod
Cluster
automatic deployment
manual
deployment
git
push

Sample Clojure Service
• .gitlab-ci.yml
• Like .travis.yml contains instructions for GitLabCI
how to test, build and deploy
• Dockerﬁle
• Instructions for Docker how to build the image of
the app
• Artifact of the build is a docker image - not
uberjar
• kube-deployment.yml
• Kubernetes deployment instructions
• kube-svc.yml
• Kubernetes service description
https://github.com/alexanderkiel/clojure-berlin-2016

The Core Namespace
(ns clojure-berlin-2016.core
(:require [aleph.http :as http]
[clojure.core.async :refer [<!! chan]]))
(defn -main [& args]
(-> (fn [_]
{:status 200
:body "Clojure Berlin 2016"})
(http/start-server {:port 8080}))
(<!! (chan)))
• A simple web server returning "Clojure Berlin 2016"

The Leiningen Project File
(defproject clojure-berlin-2016 "<VERSION>"
:dependencies [[aleph "0.4.1"]
[org.clojure/clojure "1.8.0"]
[org.clojure/core.async "0.2.395"]]
:main clojure-berlin-2016.core)
• <VERSION> is replaced at build time by the Git SHA
• :main is for lein run to work

.gitlab-ci.yml - test/build
image: clojure:lein-2.7.1
stages:
- test
- build
- deploy
test:
stage: test
tags:
- docker
script:
- lein test
build:
stage: build
tags:
- docker
script:
- sed -i "s/<VERSION>/$CI_BUILD_REF/" project.clj
- docker build -t clojure-berlin-2016:$CI_BUILD_REF .
- docker push clojure-berlin-2016:$CI_BUILD_REF

.gitlab-ci.yml - deploy branch
deploy-branch:
stage: deploy
environment: test
image: dreg.life.uni-leipzig.local/kubectl:0.4
tags:
- docker
script:
- sed -i "s/<VERSION>/$CI_BUILD_REF/" kube-deployment.yml
- kubectl config use-context gitlab-ci-test
- kubectl apply -f kube-deployment.yml
except:
- master
when: manual
• Used to test a feature/ﬁx branch in a full environment

.gitlab-ci.yml - deploy test
deploy-master:
stage: deploy
environment: test
tags:
- docker
script:
- kubectl config use-context gitlab-ci-test
only:
- master

.gitlab-ci.yml - deploy prod
deploy-prod:
stage: deploy
environment: prod
tags:
- docker
script:
- kubectl config use-context gitlab-ci-prod-a
only:
- master
when: manual

Docker ﬁle
FROM clojure:lein-2.7.1
COPY src /app/src
COPY project.clj /app/
WORKDIR /app
RUN lein with-profile production deps
EXPOSE 80
CMD ["lein", "with-profile", "production", "run"]
• Just copy the sources into the container
• Use Leiningen itself to run in production

kube-deployment.yml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
spec:
replicas: 2
template:
metadata:
labels:
app: lens
service: clojure-berlin-2016
spec:
containers:
- name: clojure-berlin-2016
image: dreg.life.uni-leipzig.local/clojure-berlin-2016:<VERSION>
ports:
- containerPort: 8080
resources:
requests:
cpu: "125m"
memory: "1Gi"
limits:
cpu: 1
memory: "2Gi"

kube-svc.yml
apiVersion: v1
kind: Service
metadata:
labels:
app: lens
spec:
type: NodePort
ports:
- port: 80
targetPort: 8080
protocol: TCP
selector:
service: clojure-berlin-2016

Steps to Follow
• Create the Kubernetes Service
• kubectl create -f kube-svc.yml
• Edit HAProxy Conﬁg
• add rules and backend for the service
• Push to GitLab
• git push

Environments in GitLabCI
• Very good visibility of wich commit is deployed in
which environment right now
• Manual deployment to prod possible

Environment History
• Easy to see when what commit was deployed
• Rollback possible

Numbers
• Our team has 4 developers
• We run 2 Kubernetes clusters (test and prod) with
about 96 GB RAM and and 24 vCPU’s each
• We run about 60 pods in production
• We have other services like central log aggregation
running using Fluentd and Elasticsearch/Kibana

Thank You
• Sample Project on Github 
https://github.com/alexanderkiel/clojure-berlin-2016
• Twitter 
@alexander_kiel
• Mail 
alexanderkiel@gmx.net

Continuous Deployment with Kubernetes, Docker and GitLab CI

Recommended

More Related Content

What's hot (20)

Similar to Continuous Deployment with Kubernetes, Docker and GitLab CI (20)

Recently uploaded (20)

Continuous Deployment with Kubernetes, Docker and GitLab CI