How to Make Linux Microservice-aware with Cilium and eBPF
3 likes1,413 views
Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2Mb3oWl. Thomas Graf talks about a new efficient in-kernel programming language called eBPF. It allows everyone to extend existing kernel components or glue them together in new forms without requiring to change the kernel itself. Filmed at qconsf.com. Thomas Graf is co-founder & CTO at Covalent and creator of the Cilium project. Before this, he has been a Linux kernel developer at Red Hat for many years. Over the more than 15 years working on the Linux kernel, he was involved in a variety of networking and security subsystems. For the past couple of years, he has been involved in the development of BPF and XDP.
![Run BPF program on event
13
Process
NICDisk
Process
BPF
BPF
BPF
IORead
Sendnetwork
packet
connect()
Sockets
TCP/IP
Network Device
BPF
TCP
retrans
BPF
read()
File Descriptor
VFS
Block Device
Attachment points
● Kernel functions (kprobes)
● Userspace functions (uprobe)
● System calls
● Tracepoints
● Network devices (packet level)
● Sockets (data level)
● Network device (DMA level) [XDP]
● ...](https://image.slidesharecdn.com/untitled-190114125152/85/How-to-Make-Linux-Microservice-aware-with-Cilium-and-eBPF-15-320.jpg?cb=1547470526)
More Related Content
More from C4Media (20)
Recently uploaded (20)
How to Make Linux Microservice-aware with Cilium and eBPF
- 1. BPF Turning Linux into a Microservices-aware Operating System
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ linux-cilium-ebpf
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. About the Speaker Thomas Graf ● Linux kernel developer for ~15 years working on networking and security ● Helped write one of the biggest monoliths ever ● Worked on many Linux components over the years (IP, TCP, routing, netfilter/iptables, tc, Open vSwitch, …) ● Creator of Cilium to leverage BPF in a cloud native and microservices context ● Co-Founder & CTO of the company building Cilium 2
- 5. Agenda ● Evolution of running applications ○ From single task processes to microservices ● Problems of the Linux kernel ○ The kernel ● What is BPF? ○ Turning Linux into a modern, microservices-aware operating system ● Cilium - BPF-based networking security for microservices ○ What is Cilium? ○ Use Cases & Deep Dive ● Q&A 3
- 6. Evolution: Running applications Split the CPU and memory. Shared libraries, package management, Linux distributions. 4 Virtualization Microservices Containers Multi tasking Ship the OS together with application and run it in a VM for better resource isolation. Virtualized hardware and software defined infrastructure. Dark Age: Single tasking The simple age. Back to a shared operating system. Applications directly interact with the host operating system again.
- 7. Problems of the Linux Kernel in the age of microservices 5
- 8. Problem #1: Abstractions 6 ProcessProcess HW System Call Interface IPv4 Netdevice / Drivers Sockets Ethernet TCP IPv6 Netfilter UDP Raw Traffic Shaping Bridge OVS .. The Linux kernel is split into layers to provide strong abstractions. Pros: ● Strong userspace API compatibility guarantee. A 20 years old binary still works. ● Majority of Linux source code is not hardware specific. Cons: ● Every layer pays the cost of the layers above and below. ● Very hard to bypass layers.
- 9. Problem #2: Per subsystem APIs 7 ProcessProcess HW System Call Interface IPv4 Netdevice / Drivers Sockets Ethernet TCP IPv6 Netfilter UDP Raw Traffic Shaping Bridge OVS iptablesseccomp tcethtool .. ip brctl / ovsctl tcpdump
- 10. 8 Problem #3: Development Process The Good: ● Open and transparent process ● Excellent code quality ● Stability ● Available everywhere ● Almost entirely vendor neutral The Bad: ● Hard to change ● Shouting is involved (getting better) ● Large and complicated codebase ● Upstreaming code is hard, consensus has to be found. ● Upstreaming is time consuming ● Depending on the Linux distribution, merged code can take years to become generally available ● Everybody maintains forks with 100-1000s backports
- 11. 9 Problem #4: What is a container? What the kernel knows about: ● Processes & thread groups ● Cgroups ○ Limits and accounting of CPU, memory, network, … Configured by container runtime. ● Namespaces ○ Isolation of process, CPU, mount, user, network, IPC, cgroup, UTS (hostname). Configured by container ○ runtime ● IP addresses & port numbers ○ Configured by container networking ● System calls made & SELinux context ○ Optionally configured by container runtime What the kernel does not know: ● Containers or Kubernetes pods ○ There is no container ID in the kernel ● Exposure requirements ○ The kernel no longer knows whether an application should be exposed outside of the host or not. ● API calls made between containers/pods ○ Awareness stops at layer 4 (ports). While SELinux can control IPC, it can’t control service to service API calls. ● Servicemesh, huh?
- 12. What now? Alternatives? Linus was wrong. The app should provide its own OS. 10 Move OS to Userspace Rewrite Everything?Unikernel We don’t need kernel mode for most of the logic. Build on top of a minimal Linux. Examples: ClickOS, MirageOS, Rumprun, ... Give user space access to hardware Examples: User mode Linux, gVisor, ... Expose the hardware directly to user space. It will be fine. Examples: DPDK, UDMA, .. Total Estimated Cost to Develop Linux (average salary = $75,662.08/year, overhead = 2.40). $1,372,340,206
- 13. What is BPF? Highly efficient sandboxed virtual machine in the Linux kernel making the Linux kernel programmable at native execution speed. Jointly maintained by Cilium and Facebook with collaborations from Google, Red Hat, Netflix, Netronome, and many others. 11 $ clang -target bpf -emit-llvm -S 32-bit-example.c $ llc -march=bpf 32-bit-example.ll $ cat 32-bit-example.s cal: r1 = *(u32 *)(r1 + 0) r2 = *(u32 *)(r2 + 0) r2 += r1 *(u32 *)(r3 + 0) = r2 exit
- 14. The Linux kernel is event driven 12 Process Process CPU RAM MMU NIC Disk Disk System Call Interface USB Drivers 12M lines of source code Process Process System calls Interrupts
- 15. Run BPF program on event 13 Process NICDisk Process BPF BPF BPF IORead Sendnetwork packet connect() Sockets TCP/IP Network Device BPF TCP retrans BPF read() File Descriptor VFS Block Device Attachment points ● Kernel functions (kprobes) ● Userspace functions (uprobe) ● System calls ● Tracepoints ● Network devices (packet level) ● Sockets (data level) ● Network device (DMA level) [XDP] ● ...
- 16. Process BPF Maps 14 BPF BPF Maps BPF map use cases: ● Hold program state ● Share state between programs ● Share state with user space ● Export metrics & statistics ● Configure programs Map types: ● Hash tables ● Arrays ● LRU (Least recently used) ● Ring buffer ● Stack trace ● LPM (Longest prefix match)
- 17. BPF Helpers 15 bpf_get_prandom_u32() BPF BPF helpers: ● Stable kernel API exposed to BPF programs to interact with the kernel ● Includes ability to: ○ Get process/cgroup context ○ Manipulate network packets and forwarding ○ Access BPF maps ○ Access socket data ○ Send metrics to user space ○ ... bpf_skb_store_bytes() bpf_redirect() bpf_get_current_pid_tgid() bpf_perf_event_output()
- 18. BPF Tail Calls 16 BPF BPF BPF tail calls: ● Chain logical programs together ● Implement function calls ● Must be within same program type BPF BPF BPF
- 19. BPF JIT Compiler 17 JIT Compiler ● Ensures native execution performance without requiring to understand CPU ● Compiles BPF bytecode to CPU architecture specific instruction set Supported architectures: ● X86_64, arm64, ppc64, s390x, mips64, sparc64, arm Byte code Byte code x86_64 generic Byte code generic JIT
- 20. BPF Contributors 380 Daniel Borkmann (Cilium, Maintainer) 161 Alexei Starovoitov (Facebook, Maintainer) 160 Jakub Kicinski Netronome 110 John Fastabend (Cilium) 96 Yonghong Song (Facebook) 95 Martin KaFai Lau (Facebook) 94 Jesper Dangaard Brouer (Red Hat) 74 Quentin Monnet (Netronome) 45 Roman Gushchin (Facebook) 45 Andrey Ignatov (Facebook) Top contributors of the total 186 contributors to BPF from January 2016 to November 2018. 18
- 21. BPF Use Cases ● L3-L4 Load balancing ● Network security ● Traffic optimization ● Profiling https://code.fb.com/open-s ource/linux/ ● QoS & Traffic optimization ● Network Security ● Profiling ● Replacing iptables with BPF (bpfilter) ● NFV & Load balancing (XDP) ● Profiling & Tracing ● Performance Troubleshooting ● Tracing & Systems Monitoring ● Networking 19
- 22. Simple Kprobe Example 20 Example: BPF program using gobpf/bcc:
- 23. What is Cilium? At the foundation of Cilium is the new Linux kernel technology BPF, which enables the dynamic insertion of powerful security, visibility, and networking control logic within Linux itself. Besides providing traditional network level security, the flexibility of BPF enables security on API and process level to secure communication within a container or pod. Read More Cilium is open source software for transparently providing and securing the network and API connectivity between application services deployed using Linux container management platforms like Kubernetes, Docker, and Mesos. 21
- 24. Project Goals 22 Approachable BPF ● Make the efficiency and flexibility of BPF available in an approachable way ● Automate program creation and management ● Provide an extendable platform Microservices-aware Linux ● Use the flexibility of BPF to make the Linux kernel aware of cloud native concepts such as containers and APIs. Security ● Use the additional visibility of BPF to provide security for microservices including: ○ API awareness ○ Identity based enforcement ○ Process level context enforcement Performance ● Leverage the execution performance and JIT compiler to provide a highly efficient implementation.
- 25. Cilium Use Cases 23 Container Networking ● Highly efficient and flexible networking ● CNI and CMM plugins ● IPv4, IPv6, NAT46, direct routing, encapsulation ● Multi cluster routing Service Load balancing: ● Highly scalable L3-L4 load balancing implementation ● Kubernetes service implementation or API driven. Microservices Security ● Identity-based L3-L4 network security ● Accelerated API-aware security via Envoy (HTTP, gRPC, Kafka, Cassandra, memcached, ..) ● DNS aware policies ● SSL data visibility via kTLS Servicemesh acceleration: ● Minimize overhead when injecting servicemesh sidecar proxies
- 26. BPF-based servicemesh Acceleration 24 Service Container Sidecar proxy Service Container Sidecar proxy How it really looks:
- 27. BPF-based servicemesh Acceleration 25 Accelerate the service to sidecar communication ~3.5x performance improvement
- 28. Other BPF projects 26 Tracing / Profiling: ● BPFTrace - DTrace for Linux (Brendan Gregg, et al.) ● bpfd - Load BPF programs into entire clusters (Joel Fernandes, Google) Frameworks: ● gobpf - Go based framework to write BPF programs ● BCC - Python framework to write BPF programs Load balancing: ● Katran - Source code of Facebook’s primary L3-L4 LB (Facebook team) Security: ● Seccomp - Advanced BPF version of Seccomp (Kernel team) DDoS mitigation: ● bpftools - DDOS mitigation tool with iptables like syntax (Cloudflare) … and many more
- 29. Thank you! Source Code: https://github.com/cilium/cilium BPF reference guide: http://docs.cilium.io/en/stable/bpf/ Twitter: @ciliumproject Website: https://cilium.io/
- 30. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ linux-cilium-ebpf