Comparing Next-Generation Container Image Building Tools

Copyright©2018 NTT Corp. All Rights Reserved.
Akihiro Suda ( @_AkihiroSuda_ )
NTT Software Innovation Center
Comparing Next-Generation
Container Image Building Tools
Open Source Summit Japan (June 20-22, 2018)

2
• Software Engineer at NTT
• GitHub: @AkihiroSuda
• Twitter: @_AkihiroSuda_
• Docker Moby core maintainer
• In April 2017, Docker [ as a project ] transited into Moby
• Now Docker [ as a product ] has been developed as one of
downstreams of Moby
: ~ :
RHEL Fedora
Who am I

3
• BuildKit initial maintainer
• Next-generation `docker build`
• containerd maintainer
• Industry-standard container runtime
• Can be used as a Docker-replacement for Kubernetes
• Docker Tokyo Community Leader (meetup organizer)
• https://dockerjp.connpass.com/
Who am I

4
• Problems of `docker build`
• New image builder tools
• Comparison & Evaluation
• CBI: "Container Builder Interface"
Agenda
BuildKit img
Bazelkaniko
Buildah
Source-to-Image Metaparticle
umoci&orca

5
• Shell-script-like language for building Docker container
images
• Each of the lines is cached as a Copy-on-Write filesystem
layer, e.g. overlayfs
Introduction to Dockerfile
mount –t overlay
–o lowerdir=0,upperdir=1 ..
FROM golang:1.10
COPY . /go/src/github.com/foo/bar
RUN go build –o /bar github.com/foo/bar
mount –t overlay
–o lowerdir=1,upperdir=2 ..

6
• Supports transferring files between stages, starting with
Docker 17.05
• Effectively reduces the size of the final image
Introduction to Dockerfile
FROM golang:1.10 AS foobar
COPY . /go/src/github.com/foo/bar
RUN go build –o /bar github.com/foo/bar
FROM alpine:3.7
COPY –-from=foobar /bar /
copy "bar" to
the final stage

7
• Docker-integrated tool for building images using
Dockerfile
• Requires Docker daemon to be running
• Similar to `docker run`, but some features are intentionally
removed for security reason
• No volumes (`docker run -v`, `docker run --mount`)
• No privileged mode (`docker run –-privileged`)
Introduction to `docker build`

8
• Modifying a single line always invalidates the caches of
the subsequent lines
• N-th line is assumed to always depend on the (N-1)-th line
• A user needs to arrange the instructions carefully for
efficient caching
Problem: inefficient caching
FROM debian
EXPOSE 80
RUN apt update && apt install –y HEAVY-PACKAGES
Modifying this line always invalidates the apt cache
due to the false dependency

9
• A multi-stage Dockerfile has DAG structure
Problem: no concurrency
FROM golang AS stage0
...
RUN go build –o /foo ...
FROM clang AS stage1
...
RUN clang –o /bar ...
FROM debian AS stage2
COPY --from=stage0 /foo /usr/local/bin/foo
COPY --from=stage1 /bar /usr/local/bin/bar
0
2
1
Directed Acyclic Graph
has concurrency

10
• A multi-stage Dockerfile has DAG structure
Problem: no concurrency
...
...
0
2
1
0
1
2
Actual `docker build`
implementation (Sequential)

11
• No safe way to access private assets (e.g. Git repos, S3)
from build containers
• Copying credentials using `COPY` can leak the credential
accidentally
• Needs to be carefully used with either multi-stage or `--squash`
• Env vars are vulnerable to accidents as well
Problem: inaccessible to private assets
FROM ...
COPY id_rsa ~/.ssh
RUN git clone ssh://...
RUN rm –f ~/.ssh/id_rsa The key still remains in the layer!

12
• Cannot be executed without root privileges
• Important for building images on Kubernetes
• Cannot preserve compiler caches due to lack of volumes
• Unreproducible builds
• Non-deterministic command executions
• Left-pad issue
• Dockerfile can be too complex and hard to maintain
Other problems

13
• And more!
• FTL, Smith, Ansible Container...
• Some of them still use Dockerfile, others not
• No "silver bullet" solution
Solutions
BuildKit img
Bazelkaniko
Buildah
Source-to-Image Metaparticle
umoci & orca

14
• Uses DAG-style low-level intermediate language called LLB
• Accurate dependency analysis and cache invalidation
• Vertices can be executed in parallel
• LLB can be compiled from Dockerfile
• And also from 3rd party languages
BuildKit: next-generation `docker build`
Compile
Dockerfile
LLB DAG
3rd party languages
docker-image://alpine
Image
git://foo/bar
docker-image://gcc
Run("apk add ..")Run("make")
https://github.com/moby/buildkit
3 vertices can be executed in parallel
2

15
• DAG structure of LLB can be described using multi-stage
Dockerfile
...
...
0
2
1

16
Can be also used for building non-
container artifacts

17
• Distributed mode is also on plan (#224, #231)
• A worker tells the master its loadavg and LLB DAG vertex cache
info
• The master choose the worker for each of the LLB DAG vertices
using the info from the workers
Master
Master
Master
LBClient
Worker
Worker
Worker
"I can reproduce cache for vertex sha256:deadbeef!"

18
• Experimental support for rootless mode
• Runs everything including BuildKit itself as an unprivileged user,
using `user_namespaces(7)`
• Protect the system from potential bugs of BuildKit/containerd/runc.
• Also useful for HPC users
• Requires `newuidmap(1)` and `newgidmap(1)` with SUID bit for `apt`
• No patch for runc is needed since June 2018
• Don't confuse this with `dockerd --userns-remap`
• `dockerd –-userns-remap` still requires `dockerd` itself to be
executed as the root

19
• Rootless BuildKit can be executed inside Docker and
Kubernetes
• But requires `--privileged` for let `RUN` containers mount `/proc`
• Will be fixed soon via moby/moby#36644 and
kubernetes/kubernetes#64283
• Still safe because BuildKit works as an unprivileged user
...
USER penguin
ENTRYPOINT ["rootlesskit", "buildkitd"]
RootlessKit: shim for setting up user NS and mount NS
https://github.com/AkihiroSuda/rootlesskit

20
• Plan to support "privileged" build as well
• likely to use libentitlement (#238)
• e.g. `buildctl build --entitlements=security.unconfined`
for privileged build
• potential use-cases: GPU, FUSE, ...

21
• Supports non-standard Dockerfile "syntax",
e.g. `RUN –-mount`
• `RUN --mount` will also support SSH agent socket file and
secret files (#262)
# syntax = tonistiigi/dockerfile:runmount20180610
...
RUN --mount=target=/root/.cache,type=cache go build
Cache mount can be useful for compillers (e.g. Go)
and package managers (e.g. apt)

22
• Benchmark result (from Tonis's slide: https://t.co/aUKqQCVmXa )

23
• Will be integrated to Moby & Docker 18.06 (moby/moby#37151)
• No change on the `docker build` command line but you need to
set `DOCKER_BUILDKIT=1`
• Will be released by the end of this month
• Also adopted by OpenFaaS Cloud
• https://github.com/openfaas/openfaas-cloud
• "GitOps for your functions with native GitHub integrations"

24
• Developed under Moby's open governance
• But Dockerfile-to-LLB compiler is planned to be moved to Docker,
Inc.'s repo (#425)
• Dockerfile specification is maintained by Docker, Inc.
• LLB allows implementing non-Dockerfile languages
• Any idea for new language?

25
• Created by Jessie Frazelle (Microsoft)
• Uses BuildKit as a library but daemonless and has Docker-
like CLI
• Currently no support for running multiple `img` instances with the
same cache directory (#92)
• Rootless mode by default
img: daemonless BuildKit
$ img build –t example.com/foo .
$ img push example.com/foo
$ img save example.com/foo | docker load
https://github.com/genuinetools/img

26
• Created by Red Hat
• Officially included in RHEL since RHEL 7.5
• Supports Dockerfile, but `buildah run` and `buildah
commit` are supported as well
• as in `docker run` and `docker commit`, without Dockerfile
• Daemonless
• Can be used as a backend of `podman build`
• Podman: Red Hat's daemonless and swarmless Docker-like tool
Buildah: Red Hat's daemonless `docker build`
https://github.com/projectatomic/buildah

27
• Supports secret volume
• But configuration is globally scoped
• `/etc/containers/mounts.conf`
• e.g. `/usr/share/rhel/secrets:/run/secrets` for allowing all Buildah
containers to access RHEL subscriptions
• Seems to have usability and security concern for other use-cases
• Rootless mode is planned (#386)

28
• Cache for Dockerfile instructions is not supported but
planned (#601)
• Parallelization is also planned (#633)
• And distributed execution as well

29
• Created by Aleksa Sarai (SUSE)
• Umoci: Umoci modifies Open Container images
• Unpacks and repacks OCI Image Spec archives (tar+gz and
JSON) into/from OCI Runtime Spec bundles (directories)
• "Pure"-Rootless and daemonless
• Does not require setting up subuids/subgids (which require SUID
binary) for unpacking archives that have multiple UIDs/GIDs
• Uses `user.rootlesscontainers` xattr instead of `chown(2)`
Umoci & Orca: the first rootless and daemonless image builder
https://github.com/openSUSE/umoci
https://github.com/cyphar/orca-build

30
• Orca: Umoci-based image builder with support for Dockerfile
• Can be used with runROOTLESS for images that require multiple
UIDs/GIDs (typically Debian/Ubuntu apt)
• https://github.com/rootless-containers/runrootless
• Emulates several system calls using `ptrace(2)` and
`user.rootlesscontainers` xattr values (which are set by Umoci)
• No SUID binary is required (but slow)
• Multi-stage Dockerfile and caching are not supported at the moment
• Planned to be integrated into Umoci
• https://twitter.com/lordcyphar/status/987668301890207744
Umoci & Orca: the first rootless and daemonless image builder

31
• Created by Google
• Kaniko itself needs to be executed in a container, but does
not require `--privileged`
• Execute `RUN` instructions within Kaniko's rootfs and
namespaces
• i.e. `RUN` instructions are executed without creating containers
• Excludes kaniko itself's binary and configuration files on packing
the rootfs archives
• Seems inappropriate for malicious Dockerfiles due to lack of
isolation (#106)
kaniko: "containerless" rootless builder
https://github.com/GoogleCloudPlatform/kaniko

32
• Bazel: Google's generic build system
• Not specific to containers
• `rules_docker` can build Docker images, but equivalent of `RUN`
instruction is intentionally omitted due to poor reproducibility
Non-Dockerfile based tools
# https://github.com/bazelbuild/rules_docker#container_image
container_image(
name = "app",
base = "@java_base//image",
files = ["//java/com/example/app:Hello_deploy.jar"],
cmd = ["Hello_deploy.jar"]
)

33
• Source-to-Image: Red Hat OpenShift's build system
• Application developers don't need to write any file for building
images
• S2I base images contain scripts for building applications in the
language-specific way
• e.g. `centos/python-35-centos7` for Python 3.5
• Previous versions depended on Docker, but recent version can
produce Dockerfiles that can be built by other tools

34
• Metaparticle: library for cloud-native apps on Kubernetes
• Supports .NET, Go, Java, JS, Python, Ruby, Rust
• Hard to change the target repository without editing source
codes
• Or implementing a new library on top of Metaparticle
• Also provides service-related features
• e.g. sharding HTTP requests based on URL
from metaparticle import Containerize
@Containerize(package={'repo': 'foo/bar', ...)
def main():
...

35
• FTL
• Similar to S2I but only for Node.js, Python, and PHP
• Smith
• Supports Oracle's "Microcontainer Manifest"
• Ansible Container
• Supports Ansible Playbook
• README says "no longer under active development"

36
Comparison across Dockerfile-based tools
Docker BuildKit img Buildah Orca kaniko
Instruction cache Limited ✔ ✔
Parallelization
✔ ✔ Planned
Distributed
execution
Planned Planned
Daemonless As a
library ✔ ✔ ✔ ✔
Rootless
✔ ✔ Planned ✔ ✔
Requires SUID binary for apt
1 1 2 3
1

37
Parallelization
✔ ✔ Planned
Distributed
execution
Planned Planned
Daemonless As a
Rootless
No SUID required but slow
1 1 2 3
2

38
Parallelization
✔ ✔ Planned
Distributed
execution
Planned Planned
Daemonless As a
Rootless
Executable in containers without `--privileged` but still has security concern
1 1 2 3
3

39
Benchmark
Average time of Build #1 (5 times)
Average time of Build #2 (5 times)
Always
without cache
Some builders
use cache
Build #1
Build #2
Prune the state
Put a dummy file
Build #1
Build #2
Prune the state
Put a dummy file
...
Simulates
trivial code change

40
• Benchmark script is available
• https://github.com/AkihiroSuda/buildbench
• Supported tools: Docker, Buildkit, img, Buildah, Kaniko
• Everything is containerized
• Builders (except Kaniko) are configured to use overlayfs
• Tested on Travis CI (June 19, 2018)
• Logs (contains version info and raw data): https://travis-
ci.org/AkihiroSuda/buildbench/builds/393967682
• See also https://github.com/AkihiroSuda/buildbench/issues/5
• 2 bursted vCPUs, 7.5GB RAM
Benchmark

41
Benchmark: examples/ex01
FROM alpine AS buildc
RUN apk add --no-cache build-base
RUN echo ... > hello.c
COPY . /foo
RUN gcc -o /a.out /hello.c
FROM alpine AS buildgo
RUN apk add --no-cache build-base
RUN apk add --no-cache go
RUN echo ... > hello.go
RUN go build -o /a.out /hello.go
FROM alpine
COPY --from=buildc /a.out /hello1
COPY --from=buildgo /a.out /hello2
Only the cache for the next line
SHOULD be invalidated
on modification of the build ctx
`apk add build-base`
SHOULD NOT be executed twice

42
Benchmark result: examples/ex01
15.6s
7.9s
10.0s
15.3s
13.2s
1.2s 1.1s
1.9s
13.5s 13.1s
Docker BuildKit img Buildah Kaniko
#1 #2

43
• Dockerfile used for the development of Moby
• Good example of complex DAG
• 13 stages can be executed in parallel at maximum
• Buildah and Kaniko don't support this DAG at the moment
• `FROM base` results in attempt to pull `docker.io/library/base`
Another benchmark: moby/moby
golang:1.10.3
base
criu registry docker-py swagger frozen-images runtime-dev tomlv vndr containerd proxy gometalinter dockercli tini
dev
runc

44
Benchmark result: moby/moby
351.8s
278.8s
447.1s
6.4s 1.9s 7.6s
Docker BuildKit img
#1 #2

45
• My recommendation is BuildKit, but it is not the "silver
bullet"
• disclosure: I'm a maintainer of BuildKit
• Other tools are attractive as well
• Language-specific builders, e.g. S2I
• SUID-less rootless mode, e.g. Orca and Kaniko
• Enterprise support, e.g. Buildah
• Can we define the common interface for all of them?
So.. which one is the best?

46
• https://github.com/containerbuilding/cbi
• Defines "BuildJob" as a Kubernetes CRD
• Supports several backends
CBI: Container Builder Interface for Kubernetes
CBI
controller
Docker
BuildKit
img
Buildah
GCB
kubectl
Session
Manager
cbictl
Registry
CBI CRD ("buildjob")
CBI plugin API
OCI Distribution Spec
(Docker Registry API)
Note: not an official CNCF/Kubernetes project

47
apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
name: ex0
spec:
registry:
target: example.com/foo/bar
push: true
language:
dockerfile: {}
context:
git:
url: git://github.com/foo/bar
pluginSelector: plugin.name=buildkit
Most plugins accept Dockerfile,
but non-Dockerfile plugins are also supported.
e.g. Source-to-Image
The CBI controller converts "BuildJob" CRD objects
into Kubernetes batch/v1 Job objects

48
apiVersion: cbi.containerbuilding.github.io/v1alpha1
kind: BuildJob
metadata:
name: ex0
spec:
registry:
target: example.com/foo/bar
push: true
language:
dockerfile: {}
context:
git:
url: git://github.com/foo/bar
pluginSelector: plugin.name=buildkit
Also supports ConfigMap, HTTP, S3,
SFTP, and even Dropbox.. (using Rclone)
Registry and Git credentials can be
provided as Kubernetes secret objects

49
• Supported plugins:
• Docker
• BuildKit
• img
• Buildah
• kaniko
• OpenShift Source-to-Image
• Google Cloud Container Builder
• Managed service for `docker build`
• New plugin can be also added easily as a Kubernetes
service

50
• POC for Skaffold integration is available
(GoogleContainerTools/skaffold#596)
apiVersion: skaffold/v1alpha2
kind: Config
build:
artifacts:
- imageName: example.com/foo/bar
deploy:
kubectl:
manifests:
- k8s-pod.yaml
profiles:
- name: cbi
build:
cbi: {}
Deploy a Kubernetes pod using the image
By default the local Docker is used,
but can be easily switched to CBI
(`skaffold dev –p cbi`)

51
• My recommendation is BuildKit (disclosure: I'm a maintainer)
• Will be integrated to Docker 18.06 experimentally
(planned to be released by the end of this month)
• But other tools are promising as well
• Now is the time for standardization
• https://github.com/containerbuilding/cbi
Conclusion

Comparing Next-Generation Container Image Building Tools

Recommended

More Related Content

What's hot (20)

Similar to Comparing Next-Generation Container Image Building Tools (20)

More from Akihiro Suda (20)

Recently uploaded (20)

Comparing Next-Generation Container Image Building Tools