SlideShare a Scribd company logo
RancherOS & Linux Kit
Sven Dowideit
Principal Software Engineer RancherOS
19 October 2017
Is a micro-Linux distro
• Linux Kernel,
• A custom init written in go,
• Uses Cloud-config to customise instances
• Has a System-Docker and a User-Docker
RancherOS
#cloud-init
rancher:
services_include:
http-proxy: true
registry-mirror: true
network:
interfaces:
eth1:
addresses:
- 10.11.11.1/24
...
Cloud-config
#cloud-init
rancher:
services:
rancher-agent-starter:
image: rancher/agent:v1.2.5
command:
http://10.10.10.168:8080/v1/scripts/22A906891A04AA99E831:1483142400000:U7MMVCDKpoYa3ngeh
fX6BPBt92k
privileged: true
environment:
- CATTLE_HOST_LABELS='magic=node'
labels:
io.rancher.os.scope: user
io.rancher.os.after: docker
restart: false
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/rancher:/var/lib/rancher
...
Cloud-config continued
One cut down System-Docker (v1.10-ish)
And the pluggable User-Docker
• v1.12.16 and up
• Can be selected using the cloud-config or on the
commandline.
Tale of two Docker daemons
First up, System-Docker - it’s from Q1 2016.
And worse, we start it 3 times to bootstrap the OS.
Each time we start a Docker daemon, we
docker load < stage/Images.tar
How RancherOS starts
Use the moby tool to compose the final runnable image or
ISO
RancherOS has been built by composing Container Images
into an Initrd since 2014
So there’s some legacy build tooling - which LinuxKit also
provides.
U
So… what can we modernise
Our 90M OS has turned into a 245M initrd.
[ 3.715450] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
[ 3.715450]
[ 3.717145] CPU: 0 PID: 1 Comm: init Not tainted 4.9.26-rancher #1
[ 3.718061] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-1ubuntu1 04/01/2014
[ 3.719478] ffff9fd6c031fe20 ffffffffb632793c ffff9060ed820000 ffffffffb6a8d098
[ 3.720933] ffff9fd6c031fea0 ffffffffb614624d ffff9fd600000010 ffff9fd6c031feb0
[ 3.722387] ffff9fd6c031fe48 ffffffffb6c5eb28 0000000000000100 ffff9060ed830010
[ 3.723840] Call Trace:
[ 3.724382] [<ffffffffb632793c>] dump_stack+0x61/0x7d
[ 3.725189] [<ffffffffb614624d>] panic+0xd8/0x221
[ 3.725943] [<ffffffffb606c67a>] do_exit+0x4d4/0x92c
[ 3.726717] [<ffffffffb608b8f2>] ? wake_up_state+0x10/0x12
[ 3.727539] [<ffffffffb6074485>] ? signal_wake_up_state+0x2a/0x3b
[ 3.728418] [<ffffffffb606cb49>] do_group_exit+0x41/0xa2
Step 1: So we’re huge, and….
Let’s replace System-Docker with runC and Containerd,
copying the init code from LinuxKit
Next up, rip out the Eye teeth
519M inittrd
15-20 seconds just to load initrd
boot2docker in 30 seconds
Step 2: even huger-er
LinuxKit’s service cfg doesn’t share images, it lays one
down per entry
Also means we need to add overlayfs
Share images between services
383M initrd
14s to starting initrd
boot2docker in 25 seconds
Step 3: baby steps
There’s a bunch of files that were needed by System
Docker that we can remove.
Remove cruft
190M initrd
5s to get to starting init
boot2docker in 12seconds
Step 4: ok, this is fine.
use the os-config.tmpl to generate the runtime spec
no actual change in size or speed
Generate containerd spec
client, err := containerd.New(config.DefaultContainerdSocket)
ctx := namespaces.WithNamespace(context.Background(), "default")
ctr, err := client.NewContainer(ctx, serviceName,
containerd.WithNewSpec(
withService(cfg, serviceSet, service),
removeRunTmpfsMount(),
withDevicesFromSpec(spec),
withOverlay(rootfs, rwDir, workDir),
dumpSpec(serviceName+".new"),
),
)
task, err := ctr.NewTask(ctx, io, WithNoPivotRoot())
err := task.Start(ctx)
Code
rancher:
services:
nginx:
image: nginx
command: nginx -g 'daemon off;'
labels:
io.rancher.os.scope: system
io.rancher.os.after: docker
restart: always
ports:
- "80:80"
volumes_from:
volumes:
- /usr/bin/ros:/usr/bin/ros
And finally, customise
about 11 seconds to serve the first HTTP request
adding the default nginx image adds ~30M to the initrd
$ docker images nginx
REPOSITORY TAG IMAGE ID CREATED SIZE
nginx latest 1e5ab59102ce 7 days ago 108MB
Step N: wash cycle
dhcp is taking 3s - hard-code the IP
start nginx after network
use the os-console image for all os-base services
smaller, simpler LinuxKit kernel image
Lets break all the things
boot2your-service in 7 seconds
This is still RancherOS
it still uses cloud-init to load host specific settings.
(Though it can go really badly due to conference wifi :) )
Step N+1: boot2your-service
Happy hacking!
@SvenDowideit
Sven@home.org.au

More Related Content

What's hot (20)

PDF
Continuous integration with Docker and Ansible
Dmytro Slupytskyi
 
PDF
Ansible docker
QNIB Solutions
 
PDF
Virtualization inside kubernetes
inwin stack
 
PDF
Running Docker with OpenStack | Docker workshop #1
dotCloud
 
PDF
Project Atomic-Nulecule
Lalatendu Mohanty
 
PDF
CoreOS @ summer meetup in Utrecht
Timo Derstappen
 
PDF
Containers in production with Docker, CoreOS, Kubernetes and Apache Stratos
Lakmal Warusawithana
 
PDF
AtlasCamp 2015: How to deliver radical architectural change without the custo...
Atlassian
 
PDF
Containerd: Building a Container Supervisor by Michael Crosby
Docker, Inc.
 
PDF
Breaking the RpiDocker challenge
Nicolas De Loof
 
PDF
An Introduction to Kubernetes
Imesh Gunaratne
 
PPTX
CoreOS Intro
Isaac Johnston
 
PPTX
Intro- Docker Native for OSX and Windows
Thomas Chacko
 
PDF
runC: The little engine that could (run Docker containers) by Docker Captain ...
Docker, Inc.
 
PDF
Kubernetes
Meng-Ze Lee
 
PDF
Quantifying Container Runtime Performance: OSCON 2017 Open Container Day
Phil Estes
 
PDF
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup
Stefan Schimanski
 
PDF
Kubernetes Basic Operation
Simon Su
 
PPTX
Deploy at scale with CoreOS Kubernetes and Apache Stratos
Chris Haddad
 
PDF
Net core, mssql, container und kubernetes
Thomas Fricke
 
Continuous integration with Docker and Ansible
Dmytro Slupytskyi
 
Ansible docker
QNIB Solutions
 
Virtualization inside kubernetes
inwin stack
 
Running Docker with OpenStack | Docker workshop #1
dotCloud
 
Project Atomic-Nulecule
Lalatendu Mohanty
 
CoreOS @ summer meetup in Utrecht
Timo Derstappen
 
Containers in production with Docker, CoreOS, Kubernetes and Apache Stratos
Lakmal Warusawithana
 
AtlasCamp 2015: How to deliver radical architectural change without the custo...
Atlassian
 
Containerd: Building a Container Supervisor by Michael Crosby
Docker, Inc.
 
Breaking the RpiDocker challenge
Nicolas De Loof
 
An Introduction to Kubernetes
Imesh Gunaratne
 
CoreOS Intro
Isaac Johnston
 
Intro- Docker Native for OSX and Windows
Thomas Chacko
 
runC: The little engine that could (run Docker containers) by Docker Captain ...
Docker, Inc.
 
Kubernetes
Meng-Ze Lee
 
Quantifying Container Runtime Performance: OSCON 2017 Open Container Day
Phil Estes
 
Kubernetes Architecture and Introduction – Paris Kubernetes Meetup
Stefan Schimanski
 
Kubernetes Basic Operation
Simon Su
 
Deploy at scale with CoreOS Kubernetes and Apache Stratos
Chris Haddad
 
Net core, mssql, container und kubernetes
Thomas Fricke
 

Similar to Using linuxKit to build custom rancherOS systems (20)

PDF
RancherOS - The perfect place to run Docker
Saputro Aryulianto
 
PPTX
RancherOS July 2017
Sebastiaan van Steenis
 
PDF
Docker for Developers: Dev, Test, Deploy @ BucksCo Devops at MeetMe HQ
Erica Windisch
 
PDF
Docker for developers
andrzejsydor
 
PPTX
Docker
Hussien Elhannan
 
PDF
Containers without docker | DevNation Tech Talk
Red Hat Developers
 
PPTX
Docker and kubernetes
Dongwon Kim
 
PDF
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
dotCloud
 
PDF
Docker Tutorial.pdf
MuhammadYusuf767705
 
PDF
JDO 2019: Tips and Tricks from Docker Captain - Łukasz Lach
PROIDEA
 
PDF
Docker and friends at Linux Days 2014 in Prague
tomasbart
 
PDF
DOCKER-PIAIC-SLIDES
MuhammadAhmed651877
 
PDF
Docker 0.11 at MaxCDN meetup in Los Angeles
Jérôme Petazzoni
 
PDF
Build and run applications in a dockerless kubernetes world
Jorge Morales
 
PDF
The Docker "Gauntlet" - Introduction, Ecosystem, Deployment, Orchestration
Erica Windisch
 
PDF
JOSA TechTalk: Taking Docker to Production
Jordan Open Source Association
 
PDF
Techtalks: taking docker to production
muayyad alsadi
 
PDF
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Jérôme Petazzoni
 
PDF
Docker Introduction + what is new in 0.9
Jérôme Petazzoni
 
PDF
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Jérôme Petazzoni
 
RancherOS - The perfect place to run Docker
Saputro Aryulianto
 
RancherOS July 2017
Sebastiaan van Steenis
 
Docker for Developers: Dev, Test, Deploy @ BucksCo Devops at MeetMe HQ
Erica Windisch
 
Docker for developers
andrzejsydor
 
Containers without docker | DevNation Tech Talk
Red Hat Developers
 
Docker and kubernetes
Dongwon Kim
 
Introduction to Docker at SF Peninsula Software Development Meetup @Guidewire
dotCloud
 
Docker Tutorial.pdf
MuhammadYusuf767705
 
JDO 2019: Tips and Tricks from Docker Captain - Łukasz Lach
PROIDEA
 
Docker and friends at Linux Days 2014 in Prague
tomasbart
 
DOCKER-PIAIC-SLIDES
MuhammadAhmed651877
 
Docker 0.11 at MaxCDN meetup in Los Angeles
Jérôme Petazzoni
 
Build and run applications in a dockerless kubernetes world
Jorge Morales
 
The Docker "Gauntlet" - Introduction, Ecosystem, Deployment, Orchestration
Erica Windisch
 
JOSA TechTalk: Taking Docker to Production
Jordan Open Source Association
 
Techtalks: taking docker to production
muayyad alsadi
 
Introduction to Docker, December 2014 "Tour de France" Bordeaux Special Edition
Jérôme Petazzoni
 
Docker Introduction + what is new in 0.9
Jérôme Petazzoni
 
Docker Introduction, and what's new in 0.9 — Docker Palo Alto at RelateIQ
Jérôme Petazzoni
 
Ad

More from Moby Project (11)

PDF
Libnetwork updates
Moby Project
 
PPTX
CRI-containerd
Moby Project
 
PDF
FaaS-and-Furious
Moby Project
 
PPTX
Notary - container signing
Moby Project
 
PDF
Declare your infrastructure: InfraKit, LinuxKit and Moby
Moby Project
 
PPTX
Moby Summit introduction
Moby Project
 
PPTX
Moby and kubernetes entitlements
Moby Project
 
PDF
Builder and BuildKit
Moby Project
 
PDF
OpenWhisk and IBM cloud functions
Moby Project
 
PDF
The State of containerd
Moby Project
 
PDF
LinuxKit and OpenOverlay
Moby Project
 
Libnetwork updates
Moby Project
 
CRI-containerd
Moby Project
 
FaaS-and-Furious
Moby Project
 
Notary - container signing
Moby Project
 
Declare your infrastructure: InfraKit, LinuxKit and Moby
Moby Project
 
Moby Summit introduction
Moby Project
 
Moby and kubernetes entitlements
Moby Project
 
Builder and BuildKit
Moby Project
 
OpenWhisk and IBM cloud functions
Moby Project
 
The State of containerd
Moby Project
 
LinuxKit and OpenOverlay
Moby Project
 
Ad

Recently uploaded (20)

PDF
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
PPTX
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
PDF
AI Agents in the Cloud: The Rise of Agentic Cloud Architecture
Lilly Gracia
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PDF
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
PDF
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PDF
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
PDF
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
PDF
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
PDF
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PPTX
Digital Circuits, important subject in CS
contactparinay1
 
PDF
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
Go Concurrency Real-World Patterns, Pitfalls, and Playground Battles.pdf
Emily Achieng
 
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
AI Agents in the Cloud: The Rise of Agentic Cloud Architecture
Lilly Gracia
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
What’s my job again? Slides from Mark Simos talk at 2025 Tampa BSides
Mark Simos
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
How do you fast track Agentic automation use cases discovery?
DianaGray10
 
Transforming Utility Networks: Large-scale Data Migrations with FME
Safe Software
 
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
 
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
Digital Circuits, important subject in CS
contactparinay1
 
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 

Using linuxKit to build custom rancherOS systems

  • 1. RancherOS & Linux Kit Sven Dowideit Principal Software Engineer RancherOS 19 October 2017
  • 2. Is a micro-Linux distro • Linux Kernel, • A custom init written in go, • Uses Cloud-config to customise instances • Has a System-Docker and a User-Docker RancherOS
  • 4. #cloud-init rancher: services: rancher-agent-starter: image: rancher/agent:v1.2.5 command: http://10.10.10.168:8080/v1/scripts/22A906891A04AA99E831:1483142400000:U7MMVCDKpoYa3ngeh fX6BPBt92k privileged: true environment: - CATTLE_HOST_LABELS='magic=node' labels: io.rancher.os.scope: user io.rancher.os.after: docker restart: false volumes: - /var/run/docker.sock:/var/run/docker.sock - /var/lib/rancher:/var/lib/rancher ... Cloud-config continued
  • 5. One cut down System-Docker (v1.10-ish) And the pluggable User-Docker • v1.12.16 and up • Can be selected using the cloud-config or on the commandline. Tale of two Docker daemons
  • 6. First up, System-Docker - it’s from Q1 2016. And worse, we start it 3 times to bootstrap the OS. Each time we start a Docker daemon, we docker load < stage/Images.tar How RancherOS starts
  • 7. Use the moby tool to compose the final runnable image or ISO RancherOS has been built by composing Container Images into an Initrd since 2014 So there’s some legacy build tooling - which LinuxKit also provides. U So… what can we modernise
  • 8. Our 90M OS has turned into a 245M initrd. [ 3.715450] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100 [ 3.715450] [ 3.717145] CPU: 0 PID: 1 Comm: init Not tainted 4.9.26-rancher #1 [ 3.718061] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.10.1-1ubuntu1 04/01/2014 [ 3.719478] ffff9fd6c031fe20 ffffffffb632793c ffff9060ed820000 ffffffffb6a8d098 [ 3.720933] ffff9fd6c031fea0 ffffffffb614624d ffff9fd600000010 ffff9fd6c031feb0 [ 3.722387] ffff9fd6c031fe48 ffffffffb6c5eb28 0000000000000100 ffff9060ed830010 [ 3.723840] Call Trace: [ 3.724382] [<ffffffffb632793c>] dump_stack+0x61/0x7d [ 3.725189] [<ffffffffb614624d>] panic+0xd8/0x221 [ 3.725943] [<ffffffffb606c67a>] do_exit+0x4d4/0x92c [ 3.726717] [<ffffffffb608b8f2>] ? wake_up_state+0x10/0x12 [ 3.727539] [<ffffffffb6074485>] ? signal_wake_up_state+0x2a/0x3b [ 3.728418] [<ffffffffb606cb49>] do_group_exit+0x41/0xa2 Step 1: So we’re huge, and….
  • 9. Let’s replace System-Docker with runC and Containerd, copying the init code from LinuxKit Next up, rip out the Eye teeth
  • 10. 519M inittrd 15-20 seconds just to load initrd boot2docker in 30 seconds Step 2: even huger-er
  • 11. LinuxKit’s service cfg doesn’t share images, it lays one down per entry Also means we need to add overlayfs Share images between services
  • 12. 383M initrd 14s to starting initrd boot2docker in 25 seconds Step 3: baby steps
  • 13. There’s a bunch of files that were needed by System Docker that we can remove. Remove cruft
  • 14. 190M initrd 5s to get to starting init boot2docker in 12seconds Step 4: ok, this is fine.
  • 15. use the os-config.tmpl to generate the runtime spec no actual change in size or speed Generate containerd spec
  • 16. client, err := containerd.New(config.DefaultContainerdSocket) ctx := namespaces.WithNamespace(context.Background(), "default") ctr, err := client.NewContainer(ctx, serviceName, containerd.WithNewSpec( withService(cfg, serviceSet, service), removeRunTmpfsMount(), withDevicesFromSpec(spec), withOverlay(rootfs, rwDir, workDir), dumpSpec(serviceName+".new"), ), ) task, err := ctr.NewTask(ctx, io, WithNoPivotRoot()) err := task.Start(ctx) Code
  • 17. rancher: services: nginx: image: nginx command: nginx -g 'daemon off;' labels: io.rancher.os.scope: system io.rancher.os.after: docker restart: always ports: - "80:80" volumes_from: volumes: - /usr/bin/ros:/usr/bin/ros And finally, customise
  • 18. about 11 seconds to serve the first HTTP request adding the default nginx image adds ~30M to the initrd $ docker images nginx REPOSITORY TAG IMAGE ID CREATED SIZE nginx latest 1e5ab59102ce 7 days ago 108MB Step N: wash cycle
  • 19. dhcp is taking 3s - hard-code the IP start nginx after network use the os-console image for all os-base services smaller, simpler LinuxKit kernel image Lets break all the things
  • 20. boot2your-service in 7 seconds This is still RancherOS it still uses cloud-init to load host specific settings. (Though it can go really badly due to conference wifi :) ) Step N+1: boot2your-service