[Container Plumbing Days 2023] Why was nerdctl made?

Why was nerdctl made?
https://github.com/containerd/nerdctl
Akihiro Suda, NTT
Container Plumbing Days 2023 (March 23)

2
nerdctl: contaiNERD CTL
• Docker-compatible CLI for containerd (and buildkitd)
alias docker=nerdctl
• Made for facilitating new experiments in the containerd platform
• The experimental features can be safely turned off for production
environments (export NERDCTL_EXPERIMENTAL=0)
• Useful for debugging Kubernetes nodes too
https://github.com/containerd/nerdctl

3
Demo
$ nerdctl run ...
$ nerdctl compose up ...

• Lazy-pulling with eStargz/SOCI/Nydus/OverlayBD
• Registry-less P2P image distribution with IPFS *
• Image encryption with OCIcrypt *
• Image signing with Cosign / Notation *
• “Real” read-only mounts with mount_setattr
• Slirp-less rootless containers with bypass4netns *
• Interactive debugging of Dockerﬁles, with buildg *
4
Novel features of containerd & nerdctl
* = Experimental

• Lazy-pulling: pulling image contents on demand
• No need to pull an entire image
• Several formats are being proposed
5
Lazy-pulling with eStargz/SOCI/Nydus/OverlayBD
Format Implementation for containerd Description
eStargz github.com/containerd/stargz-snapshotter
Optimizes gzip granularity for seek()-ability;
Forward compatible with OCI v1 tar.gz
SOCI github.com/awslabs/soci-snapshotter
Captures a checkpoint of tar.gz decoder state;
Forward compatible with OCI v1 tar.gz
Nydus github.com/containerd/nydus-snapshotter
An alternate image format;
Not compatible with OCI v1 tar.gz
OverlayBD github.com/containerd/overlaybd
Block devices as container images;
Not compatible with OCI v1 tar.gz
https://github.com/containerd/nerdctl/blob/main/docs/stargz.md

6
Lazy-pulling with eStargz/SOCI/Nydus/OverlayBD
https://github.com/containerd/stargz-snapshotter/blob/v0.14.3/docs/images/benchmarking-result-ecdb227.png

7
Registry-less P2P image distribution with IPFS
Experimental
https://medium.com/nttlabs/nerdctl-ipfs-975569520e3d
https://github.com/containerd/nerdctl/blob/main/docs/ipfs.md
$ nerdctl pull ipfs://<CID>

• OCI Image Layers can be encrypted to protect secret ﬁles
(OCI Image Conﬁg, e.g., env vars, are not encrypted)
• Encrypted layers can be decrypted transparently
8
Image encryption with OCIcrypt
Experimental
$ nerdctl image encrypt --recipient=jwe:mypubkey.pem foo example.com/foo:encrypted
$ nerdctl push example.com/foo:encrypted
https://github.com/containerd/nerdctl/blob/main/docs/ocicrypt.md
$ nerdctl pull example.com/foo:encrypted

• Docker CLI supports DOCKER_CONTENT_TRUST (Notary v1), but it
didn’t see wide adoption
• DOCKER_CONTENT_TRUST is not implemented for nerdctl
• Instead, nerdctl supports Cosign and Notation (“Notary v2”)
9
Image signing with Cosign / Notation
Experimental
$ nerdctl pull --verify=(cosign|notation)
https://github.com/containerd/nerdctl/blob/main/docs/cosign.md
https://github.com/containerd/nerdctl/blob/main/docs/notation.md

• Surprisingly docker run -v /mnt:/mnt:ro doesn’t really make
/mnt inside the container read-only
• Submounts like /mnt/usbdisk are writable
• nerdctl supports RRO: recursively-read-only mounts
nerdctl run -v /mnt:/mnt:rro
– Implemented using MOUNT_ATTR_RDONLY (kernel >= 5.12)
10
“Real” read-only mounts with mount_setattr

• Bypass slirp4netns (usermode TCP/IP), by using SECCOMP_IOCTL_NOTIF_ADDFD
• Even faster than rootful
11
Slirp-less rootless containers with bypass4netns
Experimental
https://github.com/containerd/nerdctl/blob/main/docs/rootless.md#bypass4netns
https://speakerdeck.com/mt2naoki/ip-communications-in-rootless-containers-by-socket-switching?slide=4
$ nerdctl run --label=nerdctl/bypass4netns=1

• Breakpoints for Dockerﬁles
12
Interactive debugging of Dockerﬁles, with buildg
Experimental
https://github.com/containerd/nerdctl/blob/main/docs/builder-debug.md
https://github.com/ktock/buildg
$ nerdctl builder debug .
....
=> 1| FROM ubuntu AS dev
2| RUN echo hello > /hello
3| RUN echo world > /world
4|
(buildg) break 3
(buildg) continue
...
Breakpoint[0]: reached line: Dockerfile:3
(buildg) exec /bin/sh
#

• Namespacing: nerdctl --namespace=<NS> ps
• Connecting a container to multiple networks at once:
nerdctl run --net foo --net bar ...
• AppArmor for rootless containers
( Although still needs sudo nerdctl apparmor load )
• FreeBSD containers
• Windows containers
• And more...
13
Other features

• ctr: a CLI included in containerd
• ctr is often misbelieved to be the standard CLI for containerd
• But ctr is just a debugging tool written for containerd developers
• ctr lacks lots of high-level features that are present in Docker and nerdctl
• Exposing ports (nerdctl run -p <PORT>)
• Re-startable containers w/ networking (nerdctl run --restart=always)
• Support for ~/.docker/config.json , docker-credential-ecr-login, etc.
• Reading logs of background containers (nerdctl logs)
14
FAQ: Why not ctr?

• crictl is similar to ctr but uses a different API
• crictl has similar restrictions as ctr
• CRI API has very tight scope compared to the containerd native API
• Negotiation for modifying the CRI API is relatively hard
15
FAQ: Why not crictl?
containerd
CRI API Native API
kubelet crictl nerdctl ctr

• Docker doesn’t/didn’t use the containerd API for image management
• Can’t/Couldn’t be used for experimenting lazy-pulling, etc.
• Work is in progress toward Docker v24+
• Release cycle is/was slow
– Docker v19.03: Jul 2019
– Docker v20.10: Dec 2020
– Docker v23: Feb 2023
• Not trying to “defeat” Docker; Experiments in nerdctl are being ported to Docker too
16
FAQ: Why not Docker?

• Podman doesn’t work as a Kubernetes runtime
• CRI-O doesn’t support non-CRI API, and lacks Docker-like CLI
• podman ps , etc. works for CRI-O too, but still Podman != CRI-O
• Not trying to “defeat” Podman/CRI-O either;
Experiments in containerd/nerdctl are being ported to Podman/CRI-O too
– e.g., eStargz
17
FAQ: Why not Podman/CRI-O ?

• https://github.com/containerd/nerdctl/releases
• nerdctl-full-<VERSION>-linux-amd64.tar.gz contains all the
dependencies (containerd, runc, …)
18
Getting started
$ sudo systemctl enable --now containerd
$ sudo nerdctl run -d --name nginx -p 80:80 nginx:alpine
$ containerd-rootless-setuptool.sh install
$ nerdctl run -d --name nginx -p 8080:80 nginx:alpine

• macOS users should try Lima
• Lima: Linux Machine, originally designed as “nerdctl Machine”
• CNCF Sandbox Project
• Automatic port forwarding & host ﬁlesystem mounting
• nerdctl is also included in AWS Finch, Colima, Rancher Desktop, ...
19
Getting started
$ brew install lima
$ alias nerdctl=nerdctl.lima
$ nerdctl run -d --name nginx -p 80:80 nginx:alpine
Colima

• nerdctl = contaiNERD CTL
• Made for facilitating new experiments in the containerd platform
20
Wrap-up
https://github.com/containerd/nerdctl Slack: #containerd at slack.cncf.io
• Lazy-pulling with eStargz/SOCI/Nydus/OverlayBD
• Registry-less P2P image distribution with IPFS
• Image encryption with OCIcrypt
• Image signing with Cosign / Notation
• “Real” read-only mounts with mount_setattr
• Slirp-less rootless containers with bypass4netns
• Interactive debugging of Dockerﬁles, with buildg
• And more!

[Container Plumbing Days 2023] Why was nerdctl made?

Recommended

More Related Content

What's hot (20)

Similar to [Container Plumbing Days 2023] Why was nerdctl made? (20)

More from Akihiro Suda (20)

Recently uploaded (20)

[Container Plumbing Days 2023] Why was nerdctl made?