![Multi Machine Pipelines
What does it look like?
---
pipeline:
name: backend
steps:
- name: build
image: golang
commands:
- go build
- go test
...
---
depends_on: [ backend, frontend ]
pipeline:
name: notify
steps:
- name: slack
image: plugins/slack
---
pipeline:
name: frontend
steps:
- name: build
image: node
commands:
- npm install
- npm test
...](https://image.slidesharecdn.com/kubeci-k8smeetupfrankfurt-181016093951/85/KubeCI-Cloud-Native-Continuous-Delivery-for-Kubernetes-16-320.jpg)
![Jsonnet
● Generate config data (YAML)
● For really big pipelines
● Share config across projects with imports
{
pipeline: {
local go = 'golang:1.10.3',
'test': drone.step.new(go, group='build', commands=[
'cd api',
'make test',
]),
'build': drone.step.new(go, group='build', commands=[
'cd api',
'make build',
]),
...
}
}
pipeline:
test:
image: golang:1.10.3
group: go
commands:
- cd api
- make test
build:
image: golang:1.10.3
group: go
commands:
- cd api
- make build
...
$ drone jsonnet](https://image.slidesharecdn.com/kubeci-k8smeetupfrankfurt-181016093951/85/KubeCI-Cloud-Native-Continuous-Delivery-for-Kubernetes-17-320.jpg)
![Plugins - kubectl
● Wraps kubectl
● Decodes base64 secrets
● Some basic templating
○ {{ .DroneCommit }}
○ {{ .DroneBranch }}
pipeline:
kubectl:
image: kubeciio/kubectl
secrets: [ kubeconfig ]
kubectl: apply
files:
- /path/to/folder/foo.yaml
- /path/to/folder/bar.yaml
- /path/to/folder/baz.yaml](https://image.slidesharecdn.com/kubeci-k8smeetupfrankfurt-181016093951/85/KubeCI-Cloud-Native-Continuous-Delivery-for-Kubernetes-25-320.jpg)
![Plugins - helm
● Wraps helm
● Decodes base64 secrets
● Sets tiller namespace
● Iterates over charts array
● Fetches specific helm version
pipeline:
helm:
image: kubeciio/helm
secrets: [ kubeconfig ]
helm: upgrade --install
tiller_namespace: helm
charts:
- name: prometheus
path: monitoring/prometheues/
namespace: monitoring
- name: nginx
path: ingress/nginx/
namespace: ingress-nginx
- name: cert-manager
path: cert-manager/
namespace: cert-manager](https://image.slidesharecdn.com/kubeci-k8smeetupfrankfurt-181016093951/85/KubeCI-Cloud-Native-Continuous-Delivery-for-Kubernetes-26-320.jpg)
![Plugins - img
● Uses genuinetools/img
○ Daemon-less
○ unprivileged Dockerfile
○ OCI compatible
● Drop-in replacement for Drone’s
Docker plugin
○ Interesting for Kubernetes
pipelines:
img:
image: kubeciio/img
repo: prom/prometheus
secrets: [ docker_username, docker_password ]
tag: [ latest, 0.3, 0.3.1 ]
when:
event: tag](https://image.slidesharecdn.com/kubeci-k8smeetupfrankfurt-181016093951/85/KubeCI-Cloud-Native-Continuous-Delivery-for-Kubernetes-27-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to KubeCI - Cloud Native Continuous Delivery for Kubernetes (20)
![[Global logic] container runtimes and kubernetes](https://cdn.slidesharecdn.com/ss_thumbnails/globallogiccontainerruntimesandkubernetes-200910061008-thumbnail.jpg?width=560&fit=bounds)
Ad
More from Tobias Schneck (20)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
KubeCI - Cloud Native Continuous Delivery for Kubernetes
- 1. Click to edit place and date Name of presentation Subtitle of presentation | presentator KubeCI Cloud Native Continuous Delivery Tobias Schneck - @toschneck Loodse Frankfurt
- 3. Sebastian Scheele Headquarter in in Hamburg, Germany Founders Awards > 25 employees world wide Founded in 2016 Julian Hansert Cloud native and container technologies Software development K8s certified Multi & Hybrid Cloud Container Engine Partners Silver Member & Training Partner GCP & Training Partner
- 4. Drone
- 5. Drone ● Continuous Delivery ● Automates your testing and releases ● Open Source ● Written in Go ● Built on Docker ○ Each step is a new container
- 7. Drone Execution concept Docker Host Drone CI Server Docker volume Step 0 git clone Drone Agent source code Step 1 go test Step 2 go build Step 3 docker build & push Step 4 Kubectl create pod
- 8. Declarative Configuration pipeline: frontend: image: node commands: - npm install - npm test backend: image: golang commands: - go test -v - go install services: database: image: mysql
- 9. Plugins
- 10. Languages Everything that can be executed in a Container
- 11. Installation ● docker-compose.yml ● Uses SQLite by default ○ MySQL & Postgres are easy to enable server: image: drone/drone:0.8 ports: - 80:8000 - 9000 volumes: - /var/lib/drone:/var/lib/drone/ restart: always environment: - DRONE_OPEN=true - DRONE_HOST=${DRONE_HOST} - DRONE_GITHUB=true - DRONE_GITHUB_CLIENT=${DRONE_GITHUB_CLIENT} - DRONE_GITHUB_SECRET=${DRONE_GITHUB_SECRET} - DRONE_SECRET=${DRONE_SECRET} agent: image: drone/agent:0.8 command: agent restart: always depends_on: - drone-server volumes: - /var/run/docker.sock:/var/run/docker.sock environment: - DRONE_SERVER=drone-server:9000 - DRONE_SECRET=${DRONE_SECRET}
- 12. Installation ● Can be deployed to Kubernetes ● Helm chart available $ helm install stable/drone Still uses the Docker socket...
- 13. Demo
- 14. What’s next?
- 15. Multi Machine Pipelines ● Chain together multiple pipelines ● Multi-machine fan-in and fan-out ● Multi-operating system and multi-architecture builds ● Replaces matrix builds
- 16. Multi Machine Pipelines What does it look like? --- pipeline: name: backend steps: - name: build image: golang commands: - go build - go test ... --- depends_on: [ backend, frontend ] pipeline: name: notify steps: - name: slack image: plugins/slack --- pipeline: name: frontend steps: - name: build image: node commands: - npm install - npm test ...
- 17. Jsonnet ● Generate config data (YAML) ● For really big pipelines ● Share config across projects with imports { pipeline: { local go = 'golang:1.10.3', 'test': drone.step.new(go, group='build', commands=[ 'cd api', 'make test', ]), 'build': drone.step.new(go, group='build', commands=[ 'cd api', 'make build', ]), ... } } pipeline: test: image: golang:1.10.3 group: go commands: - cd api - make test build: image: golang:1.10.3 group: go commands: - cd api - make build ... $ drone jsonnet
- 18. Project KubeCI Cloud Native Continuous Delivery
- 19. Kubernetes Runtime ● Needs Kubernetes 1.10+ ● Implements the same interface as the Docker runtime ● Takes a kubeconfig to connect to Kubernetes ● Talks to the Kubernetes apiserver ● Schedules ○ Services ○ Pods ○ Persistent Volumes (local volume) DRONE_KUBERNETES=true DRONE_KUBERNETES_KUBECONFIG=/home/user/.kube/config DRONE_KUBERNETES_NAMESPACE=drone
- 20. Kubernetes Runtime Execution Concept Kubernetes Cluster Node affinity Drone CI Server local-storage Step 0 git clone Drone Agent source code Step 1 go test Step 2 go build Step 3 docker build & push Step 4 Kubectl create pod
- 21. Kubernetes Runtime Execution Concept Kubernetes Cluster Node affinity Drone CI Server local-storage Step 0 git clone Drone Agent source code Step 1 go test Step 2 go build Step 3 docker build & push Step 4 Kubectl create pod
- 22. Kubernetes Runtime Improvements ● At the moment we pick a random machine ○ Use NodeAffinity ● Building Docker containers on Kubernetes ○ Needs privileges ● No way to reference secrets ● No way to reference configmaps Expect a lot of fixes and improvements!
- 23. Demo
- 24. KubeCI ● We created a Drone Kubernetes Runtime ● We have a lot more ideas
- 25. Plugins - kubectl ● Wraps kubectl ● Decodes base64 secrets ● Some basic templating ○ {{ .DroneCommit }} ○ {{ .DroneBranch }} pipeline: kubectl: image: kubeciio/kubectl secrets: [ kubeconfig ] kubectl: apply files: - /path/to/folder/foo.yaml - /path/to/folder/bar.yaml - /path/to/folder/baz.yaml
- 26. Plugins - helm ● Wraps helm ● Decodes base64 secrets ● Sets tiller namespace ● Iterates over charts array ● Fetches specific helm version pipeline: helm: image: kubeciio/helm secrets: [ kubeconfig ] helm: upgrade --install tiller_namespace: helm charts: - name: prometheus path: monitoring/prometheues/ namespace: monitoring - name: nginx path: ingress/nginx/ namespace: ingress-nginx - name: cert-manager path: cert-manager/ namespace: cert-manager
- 27. Plugins - img ● Uses genuinetools/img ○ Daemon-less ○ unprivileged Dockerfile ○ OCI compatible ● Drop-in replacement for Drone’s Docker plugin ○ Interesting for Kubernetes pipelines: img: image: kubeciio/img repo: prom/prometheus secrets: [ docker_username, docker_password ] tag: [ latest, 0.3, 0.3.1 ] when: event: tag
- 28. Plugins What are your needs? What CI/CD challenges for Kubernetes do you have? Tell us!
- 29. Matthias Loib - @metalmatze POC Lead of KubeCI Brad Rydzewski - @bradrydzewski Creator of Drone Chris Bargmann - @chrisbargmann Support on KubeCI plugins Stefan Schimanski - @the_sttts Brainstorming at FOSDEM Jason Murray - @chaosaffe Initial hacking sessions Acknowledgements
- 30. Questions?
- 32. We are hiring! Hamburg, Berlin, Munich, San Francisco, Gdańsk, … www.loodse.com @Loodse
- 33. Thank you Tobias Schneck - @toschneck Loodse