[FOSDEM 2020] Lazy distribution of container images
- 1. Lazy distribution of container images Current implementation status of containerd remote snapshotter Akihiro Suda FOSDEM (February 1, 2020) Credit to Kohei Tokunaga (NTT) for containerd impl. & benchmark scripts
- 2. • Run containers before completion of downloading the images • Lots of alternative image formats are proposed to support this • stargz is getting wide adoption (containerd & Podman) 2 Summary
- 4. The problems of the current Docker / OCI format
- 5. • Open Containers Initiative (OCI) defines the standard specifications for containers – Docker/Moby, Podman, Kubernetes (containerd, CRI-O, …), Singularity… • OCI Image Spec: defines the tar ball structure and the JSON metadata format – Based on Docker Image Manifest V2 Schema 2 • OCI Distribution Spec: defines the API for distributing images via HTTP – Based on Docker Registry HTTP API • Focuses on legacy rather than on innovation ☹ 5 Current Docker / OCI format
- 6. • Appeared in 1970s • Originally designed for magnetic tapes • No random access 6 TAR: Tape ARchiver https://en.wikipedia.org/wiki/PDP-11
- 7. • Without scanning the whole "tape“, file metadata cannot be listed up → Can't be mounted as a filesystem 7 Problem 1: Requires scanning the whole "tape" Metadata 0 File 0 Metadata 1 File 1 Metadata {n-1} File {n-1} Terminal zero bytes ... File name, permission, ... Content
- 8. • Having an external index file can solve the problem? → No, because gzip can’t be seek-ed (discussed later) 8 Problem 1: Requires scanning the whole "tape" Metadata 0 File 0 Metadata 1 File 1 Metadata {n-1} File {n-1} Terminal zero bytes ... Metadata 0 Metadata 1 Metadata {n-1} … Index file
- 9. • A registry might contain very similar images – Different versions – Different architectures – Different configuration files • Tar balls of these images are likely to waste the storage for identical/similar files • But not a serious issue when you have enough budget for the cloud storage 9 Problem 2: No deduplication
- 10. 1. Requires scanning the whole "tape" 2. No deduplication 10 Problems of Docker / OCI image format https://en.wikipedia.org/wiki/Magnetic_tape The main focus towards lazy distribution
- 11. • “pulling packages accounts for 76% of container start time, but only 6.4% of that data is read.” – Harter, Tyler, et al. "Slacker: Fast Distribution with Lazy Docker Containers." FAST 2016 11 Why do we want lazy distribution?
- 12. • “dev stage” images of multi-stage Dockerfiles – No need to consider tolerance against remote registry failures (because `RUN apt-get install` instructions are already flaky anyway) 12 Expected use-cases FROM example.com/heavy-dev-env:lazy AS dev RUN apt-get update && apt-get install -y some-additional-libs COPY src . RUN ./configure && make static && cp bin/foo /foo # the stage switches here FROM scratch COPY --from=dev /foo /foo ENTRYPOINT /foo
- 13. • Other use-cases are also valid, but mind fault tolerance (until the image gets 100% cached locally) – Kubernetes readinessProbe • FaaS • Web apps with huge number of HTML files and graphic files • Jupyter Notebooks with big data samples included • Full GNOME/KDE desktop – Will 2020 be the year of the containerized Linux desktop? 13 Expected use-cases
- 14. Our first attempt (2017)
- 15. Our first attempt (2017) … and post-mortem
- 16. • No tar balls • Composed of a protobuf index file (continuity manifest) + content-addressable blob files 16 Our first attempt : FILEgrain (2017)
- 17. • No tar balls • Composed of a protobuf index file (continuity manifest) + content-addressable blob files 17 Our first attempt : FILEgrain (2017) message Metadata { repeated string path; int64 uid; int64 gid; uint32 mode; uint64 size; repeated string sha256Digest; ... } Metadata 0 Metadata 1 Metadata {n-1} … blobs/sha256/deadbeef… blobs/sha256/cafebabe…
- 18. • Incompatibility with legacy tar balls • Chicken-and-egg: hard to finalize the spec when no implementation exists; hard to promote implementation when the spec is not finalized • Use-cases were unclear; didn’t need to focus on deduplication • Performance overhead due to huge numbers of HTTP requests for reading small files 18 FILEgrain post-mortem
- 19. The solution in 2020: stargz
- 20. • Proposed by Brad Fitzpatrick (Google, at that time) for accelerating the CI of the Go language project • No focus on data deduplication 20 stargz: seekable tar.gz Metadata 0 File 0 Metadata 1 File 1 Metadata {n-1} File {n-1} Terminal zero bytes ... gzip legacy tar.gz Metadata 0 File 0 gzip Metadata 1 File 1 gzip ... Metadata {n-1} File {n-1} gzip Metadata for s.i.j. stargz.index.json (Metadata 0…{n-1}) gzip Terminal zero bytes empty stream stargz gzip
- 21. • Fully compatible with legacy tar.gz • But contains extra “stargz.index.json” entry 21 stargz: seekable tar.gz Metadata 0 File 0 Metadata 1 File 1 Metadata {n-1} File {n-1} Terminal zero bytes ... gzip legacy tar.gz Metadata 0 File 0 gzip Metadata 1 File 1 gzip ... Metadata {n-1} File {n-1} gzip Metadata for s.i.j. stargz.index.json (Metadata 0…{n-1}) gzip Terminal zero bytes empty stream stargz gzip
- 22. • Only stargz.index.json is required for mounting the image • Actual files in the archive can be fetched on demand (when HTTP Range Requests are supported) 22 stargz: seekable tar.gz gzip This gzip header contains pointer for stargz.index.json stargz Metadata 1 File 1 gzip ... Metadata {n-1} File {n-1} gzip Metadata for s.i.j. stargz.index.json (Metadata 0…{n-1}) gzip Terminal zero bytes empty stream Metadata 0 File 0 gzip
- 23. • containerd: https://github.com/ktock/stargz-snapshotter – By Kohei Tokunaga (NTT) – Implemented as a containerd snapshotter plugin – stargz archives are mounted as read-only FUSE filesystems – OverlayFS is used for supporting writing – Supports more aggressive optimization (discussed later) • Podman: https://github.com/giuseppe/crfs-plugin – By Giuseppe Scrivano (Red Hat) – Implemented as a fuse-overlayfs plugin 23 stargz adoption in the ecosystem
- 24. • Profiles actual file access patterns by running an equivalent of docker run – Future: static analysis using ldd(-ish) ? Machine learning? • Reorders file entries in the archive so that relevant files can be prefetched in a single HTTP request 24 stargz optimizer for containerd /usr/bin/apt-get /bin/ls /bin/vi /lib/libc.so /lib/libjpeg.so /usr/bin/python3 .../usr/lib/python3/.../foo /usr/lib/python3/.../bar /app.py /bin/ls /app.py /usr/bin/python3 /lib/libc.so /usr/lib/python3/.../foo /usr/lib/python3/.../bar .../bin/vi /lib/libjpeg.so /usr/bin/apt-get
- 25. • Registry: Docker Hub (docker.io) • containerd host location: EC2 Oregon • Benchmark: execute typical base images with “compile hello world” command 25 Benchmark results
- 26. 26 Benchmark results Credit to Kohei Tokunaga (NTT) for containerd impl. & benchmark scripts
- 27. 27 Benchmark results Credit to Kohei Tokunaga (NTT) for containerd impl. & benchmark scripts
- 28. 28 Benchmark results Credit to Kohei Tokunaga (NTT) for containerd impl. & benchmark scripts
- 29. 29 Benchmark results Credit to Kohei Tokunaga (NTT) for containerd impl. & benchmark scripts
- 30. • Impl: Parallelize HTTP operations across image layers – https://github.com/ktock/stargz-snapshotter/issues/37 • Spec: Use zstd instead of gzip (“starzstd”?) – Proposed by Giuseppe https://github.com/golang/go/issues/30829#issuecomment-541532402 – Suitable for images with many small files – Not compatible with OCI Image Spec v1.0.1 – Compatible with OCI Image Spec v.Next 30 More optimizations are to come
- 31. • BuildKit: modern OCI image builder – Concurrent execution – Efficient caching – Rootless – (pseudo-)daemonless – Clustering on Kubernetes – And a lot of innovative features • stargz support is on our plan, stay tuned! – Producing stargz images – Consuming stargz images as base images 31 stargz integration for BuildKit
- 32. • CernVM-FS – Not compatible with OCI tar balls – Has been already widely deployed in CERN and their friends – Implementation available for containerd: https://github.com/ktock/remote-snapshotter/pull/27 • Unofficial “OCI v2” – Proposed by Aleksa Sarai (SUSE) – Not compatible with OCI v1 tarballs – Focuses on deduplication, using Restic algorithm – WIP implementation available for umoci (image manipulation tool): https://github.com/openSUSE/umoci/tree/experimental/ociv2 – No runtime implementation seems to exist 32 Other post-OCI formats
- 33. • IPCS – Proposed by Edgar Lee (Netflix) – Built on IPFS (P2P CAS) protocol – Not compatible with OCI tar balls – Implementation available for containerd: https://github.com/hinshun/ipcs • Azure Container Registry “Project Teleport” – Built on SMB protocol and VHD images – Not FLOSS 33 Other post-OCI formats
- 34. • Lots of alternative image formats are proposed for lazy distribution, but compatibility matters • stargz is getting wide adoption (containerd & Podman) • containerd supports sort+prefetch optimization for stargz https://github.com/ktock/stargz-snapshotter 34 Recap
- 35. • Valid & invalid use cases? • More efficient optimization techniques? • Issues/PRs are welcome at https://github.com/ktock/stargz-snapshotter (Expected to be moved under github.com/containerd soon) 35 Request for comments