OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orchestration by Jonathan Boulle
0 likes202 views
rkt and Kubernetes provide container runtimes and orchestration tools to seamlessly update operating systems without affecting application dependencies or uptime. rkt is a modern, secure container runtime that implements open standards and has a simple, modular architecture. It can be used as the container runtime for Kubernetes (rktnetes) or to run Kubernetes components directly. Both tools use the Container Networking Interface (CNI) plugin-based model for networking, allowing IP addresses to be assigned at the pod level. Integration between rkt and Kubernetes continues to improve to support features like TPM attestation and more seamless kubelet upgrades.
OSDC 2016 - rkt and Kubernentes what's new with Container Runtimes and Orchestration by Jonathan Boulle
- 1. rkt and Kubernetes What's new (and coming) with Container Runtimes and Orchestration
- 2. Jonathan Boulle github.com/jonboulle - @baronboulle
- 3. Why rkt and Kubernetes?
- 4. Why rkt and Kubernetes? Why container runtimes and orchestration?
- 5. CoreOS, Inc (2013 - today) Mission: "Secure the Internet" Started at the OS level: CoreOS Linux ● Modern, minimal operating system ● Self-updating (read-only) image ● Updates must be automatic and seamless
- 6. Automatic and seamless ● If the OS is always updating, what about applications running on it?
- 7. Automatic and seamless ● If the OS is always updating, what about applications running on it? ● Classic use case for containers and orchestration ○ containers decouple the application and OS update lifecycles (update at different cadences) ○ orchestration decouples application and OS uptime (services can remain unaffected during OS downtime)
- 8. Why container runtimes? ● So we can update the OS without affecting application dependencies
- 9. kernel systemd rkt ssh docker python java nginx mysql openssl app trodistrodistrodistrodistrodistro
- 10. python java nginx mysql openssl apptrodistrodistrodistrodistrodistro kernel systemd rkt ssh docker
- 14. Why orchestration? ● So we can update the OS without affecting application uptime
- 26. Why container runtimes and orchestration? So we can provide seamless updates and push forward the security of application servers
- 27. Why rkt?
- 28. A long time ago in an ecosystem far, far away.... (2014, to be precise)
- 29. ● Popular incumbent container tool (in CoreOS) ● Common practices, but few best practices ○ unsigned images (curl | sudo sh -) ○ inefficient/insecure images (FROM ubuntu:14.04) ○ PID1 or not to PID1 (zombie reaping problem) ● New platforms emerging, difficult to integrate ○ systemd + dockerd = sad times had by all 2014
- 30. ● Enter rkt (and appc) ○ Create an alternative container runtime (competition drives innovation) ○ Emphasise the importance of security and composability ○ Spur conversation around standards in the application container ecosystem 2014 (December)
- 31. a modern, secure container runtime a simple, composable tool an implementation of open standards
- 32. a modern, secure container runtime a simple, composable tool an implementation of open standards
- 33. a standard application container open specification associated tooling
- 34. appc spec in a nutshell ● Image Format (ACI) ○ what does an application consist of? ○ how can an image be located on the internet? ○ how can an image be securely signed & distributed? ● Pods ○ how can applications be grouped and run? ● Runtime format (ACE) ○ what does the execution environment look like?
- 35. ● grouping of applications executing in a shared context (network, namespaces, volumes) ● shared fate ● the only execution primitive: single applications are modelled as singleton pods appc pods
- 36. appc pods ≈ Kubernetes pods ● grouping of applications executing in a shared context (network, namespaces, volumes) ● shared fate ● the only execution primitive: single applications are modelled as singleton pods
- 37. a modern, secure container runtime a simple, composable tool an implementation of open standards (appc)
- 38. ● Docker and rkt both: ○ post 1.0 ○ "production ready" (and actively used in production) ● Kubernetes too! ● Container standards? ○ ongoing... 2016
- 39. ● appc (December 2014) ● OCI (June 2015) ● CNCF (December 2015) Container standards
- 40. appc ● Image format ○ Cryptographic identity ○ Signing ○ Discovery/federation ● Runtime format ● Pods 2015: appc vs OCI OCI ● Runtime format
- 41. ● New OCI project: image format ○ github.com/opencontainers/runtime-spec ○ github.com/opencontainers/image-spec ● Merging the best of Docker + appc ○ Container peace?! 2016: today +
- 42. OCI Image Format maintainers ● Vincent Batts, Red Hat ● Brandon Philips, CoreOS ● Brendan Burns, Google ● Jason Bouzane, Google ● John Starks, Microsoft ● Jonathan Boulle, CoreOS ● Stephen Day, Docker
- 43. Image Formats and standards Docker v1 appc Docker v2.2 OCI (in progress) Introduced 2013 December 2014 April 2015 April 2016 Content- addressable No Yes Yes Yes Signable No Yes, optional Yes, optional Yes, optional Federated namespace Yes Yes Yes Yes Delegatable DNS namespace No Yes No Yes
- 44. Container standards Why should you care? ● For users, things should "just work": ○ docker run example.com/org/app:v1.0.0 ○ rkt run example.com/org/app,version=v1.0.0 ● For administrators and operators: ○ Stronger security primitives built-in ○ Intercompatibility: mix and match tools, or write your own
- 45. a modern, secure container runtime a simple, composable tool an implementation of open standards (appc)
- 46. rkt architecture A quick introduction
- 47. no central daemon no (mandatory) API self-contained execution apps run directly under spawning process rkt - simple CLI tool
- 49. modular architecture execution divided into stages stage0 → stage1 → stage2 rkt internals
- 50. modular architecture take advantage of different technologies provide a consistent experience to users rkt internals
- 53. rkt (stage0) pod (stage1) bash/systemd/kubelet... (invoking process) app1 (stage2) app2 (stage2)
- 54. ● primary interface to rkt ● discover, fetch, manage application images ● set up pod filesystems ● manage pod lifecycle ○ rkt run ○ rkt image list ○ rkt gc ○ ... stage0 (rkt binary)
- 55. ● default implementation ○ based on systemd-nspawn+systemd ○ Linux namespaces + cgroups for isolation ● kvm implementation ○ based on lkvm+systemd ○ hardware virtualisation for isolation ● others? ○ e.g. xhyve (OS X), unc (unprivileged containers) stage1 (swappable execution engines)
- 56. ● actual app execution ● independent filesystems (chroot) ● shared namespaces, volumes, IPC, ... stage2 (inside the pod)
- 57. ● TPM, Trusted Platform Module ○ physical chip on the motherboard ○ cryptographic keys + processor ● Used to "measure" system state ● Historically just use to verify bootloader/OS (on proprietary systems) rkt TPM measurement (new!)
- 58. ● CoreOS added support to GNU Grub ● rkt can now record information about running pods in the TPM ● attestable record of what images and pods are running on a system rkt TPM measurement (new!)
- 59. rkt TPM measurement (new!)
- 60. ● optional, gRPC-based API daemon ● exposes read-only information on pods/images ● runs as unprivileged user ● easier integration with other projects rkt API service (new!)
- 62. +
- 63. rkt + Kubernetes rkt ♥ k8s in a few ways: ● using rkt as container runtime (aka "rktnetes") ● using rkt to run Kubernetes ("rkt fly") ● integrating with rkt networking (CNI)
- 64. Kubelet + Container Runtimes ● Kubelet code provides a Runtime interface ○ SyncPod() ○ GetPod() ○ KillPod() ○ ... ● in theory, anyone can implement this ● in practise, lots of Docker assumptions
- 65. Kubelet + Docker (default) kubelet dockerd container container container
- 66. Kubelet + Docker (default) Problems: ● Docker doesn't understand pods ○ kubelet must maintain pod<->container mapping ○ "infra container" to hold namespaces for pod ● dockerd = SPOF for node ○ if Docker goes down, so do all containers ● Docker doesn't interact well with systemd ○ References
- 67. Kubelet + Docker (before 1.11+) kubelet dockerd container container container
- 68. Kubelet + Docker (1.11+ with containerd) kubelet containerd containerd shim containerd shim containerd shim container container container dockerd
- 69. Kubelet + rkt (rktnetes) Using rkt as the kubelet's container runtime ● A pod-native runtime ● First-class integration with systemd hosts ● self-contained pods process model = no SPOF ● Multi-image compatibility (e.g. docker2aci) ● Transparently swappable
- 70. Kubelet + rkt (rktnetes - with systemd) kubelet systemd rkt rkt rkt rkt api service pods
- 71. Kubelet + rkt (rktnetes - without systemd) kubelet rkt rkt rkt rkt api service pods
- 72. Nearly complete! ~90% of end-to-end tests passing http://rktnetes.io/ P0 for Kubernetes 1.3 rktnetes today
- 73. Kubelet + Container Runtimes ● Kubelet's Runtime interface rework ○ Granular control over applications in containers ○ Dynamic resource management ○ Directly managing containers?
- 74. Using rkt to run Kubernetes ● Kubernetes components are largely self- hosting, but not entirely ○ Need a way to bootstrap kubelet on the host ○ kubelets can then host control plane components ● On CoreOS, this means in a container.. ○ ... but kubelet has some unique requirements (like mounting volumes on the host)
- 75. Using rkt to run Kubernetes ● rkt "fly" feature (new in 0.15.0+) ● unlike rkt run, does *not* execute pods ● execute a single application in an unconstrained environment ● all the other advantages of rkt (image discovery, signing/verification, management)
- 76. rkt (stage0) - without fly pod (stage1) bash/systemd/... (invoking process) app1 (stage2) app2 (stage2)
- 77. rkt (stage0) - without fly pod (stage1) bash/systemd/... (invoking process) app1 (stage2) app2 (stage2) Isolated mount (and PID, ...) namespace
- 78. rkt (stage0) - with fly bash/systemd/... (invoking process) application
- 79. rkt (stage0) - with fly bash/systemd/... (invoking process) application Host mount (and PID, ...) namespace
- 80. rkt (stage0) - with fly bash/systemd/... (invoking process) kubelet Host mount (and PID, ...) namespace
- 81. rkt networking Plugin-based IP(s)-per-pod Container Networking Interface (CNI)
- 82. Container Runtime (e.g. rkt) ptp macvlan ipvlan OVS Container Networking Interface (CNI)
- 83. CNI in a nutshell ● Container can join multiple networks ● Network described by JSON config ● Plugin supports two commands ○ ADD container to the network ○ REMOVE container from the network ● Plugins are responsible for all logic ○ allocating IPs, talking to backend components, ...
- 84. CNI: example configuration { "name": "mynet", "type": "ptp", "ipam": { "type": "host-local", "subnet": "10.1.1.0/24" } } $ rkt run --net=mynet coreos.com/etcd
- 85. exec() exec() create, join configure via setns + netlink How rkt uses CNI rkt network plugins (CNI) systemd-nspawn /var/lib/rkt/pods/run/$POD_UUID/netns network namespace
- 86. Kubernetes networking Plugin-based (but never left alpha) IP(s)-per-pod (sound familiar?)
- 87. Kubernetes and CNI Soon to be "the Kubernetes plugin model"
- 88. v0.2.0-rc1 Handles all networking in rkt Integrations with Project Calico, Weaveworks Hoping to donate to the CNCF CNI today
- 89. Looking ahead What's coming up for rkt and Kubernetes
- 90. rktnetes 1.0 2016Q2 Fully supported, full feature parity Automated end-to-end testing on CoreOS
- 91. LKVM backend by default Native support for OCI in Kubernetes API TPM up to the Kubernetes level rktnetes 1.0+
- 93. Try rktnetes today! https://gist.github.com/yifan-gu/091d258107f37ecdff48
- 94. rkt 1.0+ ● Loads of bugfixes ● app exit status propagation ● Discover and fetch stage1 images ○ e.g. from coreos.com
- 95. rkt 1.0+ ● rktnetes fixes ○ Hostname ○ Docker volume semantics ○ Improvements in the API Service ● SELinux support on Fedora ○ Fixes in nspawn, selinux-policy and rkt ● Concurrent image fetching
- 96. What’s next? ● rkt fly as a top-level command ● IPv6 support on CNI ● Full SELinux enforcing on Fedora ● Packaged in Debian (almost there!)
- 97. Stable configuration Stateless plugins (runtime responsibility) IPv6 <your suggestions here> CNI 1.0
- 98. Move all networking code into CNI plugins Kubernetes 1.3
- 99. Kubelet upgrades - Remember from CoreOS mission: "updates must be automatic and seamless" - If kubelet is in OS, must be upgraded in lock-step - But mixed-version clusters don't always work (e.g. upgrading from 1.07 - 1.1.1: https://github. com/kubernetes/kubernetes/issues/16961 )
- 100. Kubelet upgrades - Solution: API driven upgrades - Small agent living on host, invoking kubelet (using rkt fly) - Reading annotations from the kubelet API server - Follow along: https://github.com/coreos/bugs/issues/1051
- 101. Graceful kubelet shutdown ● When an update is ready, locksmith signals kubelet to gracefully shut down ● Kubernetes can then gracefully migrate apps before shutdown ● https://github.com/coreos/bugs/issues/1112 ● https://github.com/kubernetes/kubernetes/issues/7351
- 102. tl;dr: ● Use rkt ● Use Kubernetes ● Use rkt + Kubernetes (rktnetes) ● Get involved and help define the future of application containers and Kubernetes
- 103. May 9 & 10, 2016 - Berlin, Germany coreos.com/fest - @coreosfest
- 104. Questions? Join us! contribute: github.com/coreos/rkt careers: coreos.com/careers (now in Berlin!)
- 105. Extra slides Did I speak too fast?
- 106. rkt security ("secure by default") ● image security ○ cryptographic addressing ○ signature verification ● privilege separation ○ e.g. fetch images, expose API (new!) as non-root ● SELinux integration ● lkvm stage1 for true hardware isolation ● TPM attestation (new!)