eBPF in the view of a storage developer

eBPF from the view of a storage
developer
Richa’rd Kova’cs

© StorageOS, Inc. 2
Boring slide
• At work:
− Kubernetes Integration Engineer
− @StorageOS
− Operator, Scheduler, Automation
• At all:
− Many years of DevOps, cloud and
containerization.
− OSS devotee
− Known as @mhmxs
PHOTO

StorageOS is cloud native, software-defined
storage for running containerized applications
in production, running in the cloud, on-prem
and in hybrid/multi-cloud environments.
3

Agenda
Developer
experience Portability
and
debugging
Deep dive
Introduce kubectl
gadget plugin
Basics including
architecture,
performance, and
weaknesses

Agenda
Basics including
architecture,
performance, and
weaknesses

● What the heck is Extended Berkley Packet Filter (eBPF)
− Linux kernel feature since 4.1 - 🙀
− First it was an iptables replacement (BPF)
− It uses kernel events to do various things
− cat /proc/kallsyms | wc -l
● 185449 (and counting)
− eBPF has the capability to interact with userspace
− Script compiled to a special eBPF bytecode
− New attack vendor
● In short:
− Small, mostly C program, compiled to bytecode to hook up at almost anywhere in
the kernel.
Basics

How does it work?
Source: https://www.brendangregg.com/ebpf.html

Some projects based on eBPF
WeaveScope
Tracing TCP
connections
seccomp-bpf
Limiting syscalls
Calico
Network eBPF
dataplane
Inspector gadget
Kubectl plugin to work
with eBPF
Cilium
Networking,
Observability and
Security

Storage related options

● Tracing at the VFS layer level
− At this level eBPF plugin is able to catch file related events:
● CRUD of files or directories
● File system caches
● Mount points
● cat /proc/kallsyms | grep "t vfs" | wc -l
− 44
● Examples:
− vfsstat.py: Count VFS calls
− vfsreadlat.c: VFS read latency distribution

● Tracing at the file system layer level
− File system specific events:
● Ext4, NFS, BTRS, …
● CRUD operations
● Low level operations
● Performance related events
● cat /proc/kallsyms | grep "t ext4" | wc -l
− 397
● Examples:
− nfsslower.py: Trace slow NFS operations
− btrfsdist.py: Summarize BTRFS operation latency distribution

● Tracing at the block device / device driver layer levels
− A trace at this level gives insight on which areas of:
● Low level - near to HW – operations
● Physical disk devices
● Virtual block devices
● Block device read – write
● Examples:
− bitehist.py: Block I/O size
− disksnoop.py: Trace block device I/O latency

● Supported architectures are limited (arm, amd64 included)
● Not supported everywhere
− Needs CONFIG_BPF_SYSCALL during kernel build
− Container needs privileged mode
− In cloud it should be tricky, not widely supported
● Portability is tricky
● Limited size of MAPs
● Hard to debug
● Test matrix should be huge on case of a heterogeneous infrastructure
Weaknesses

● Small pre-built bytecode
● JIT compiled
− Depends on CONFIG_BPF_JIT
● Kernel changes observed function instruction order
− It is native
− No extra layer
− No exact or measurable overhead
Performance impact

Agenda
Deep dive

● Kprobe
− Kernel dynamic tracing
■ Kernel file write end
● Uprobe
− User level dynamic tracing
■ Return value of bash readline()
● Tracepoint
− Kernel static tracing
■ Trace sys_enter syscalls of a program
● Perf events
− Timed sampling Performance Monitoring Counter (PMC)
Hook points

Interacting with userspace

● Without interacting a user space program eBPF has just a limited use-cases
● EBPF uses a shared MAPs to gap the overlap the gap
● Read of MAP happens asynchronous
● There are several type of MAPs for different uses-cases

● BPF_MAP_TYPE_UNSPEC = 0,
● BPF_MAP_TYPE_HASH = 1,
● BPF_MAP_TYPE_ARRAY = 2,
● BPF_MAP_TYPE_PROG_ARRAY = 3,
● BPF_MAP_TYPE_PERF_EVENT_ARRAY = 4,
● BPF_MAP_TYPE_PERCPU_HASH = 5,
● BPF_MAP_TYPE_PERCPU_ARRAY = 6,
● BPF_MAP_TYPE_STACK_TRACE = 7,
● BPF_MAP_TYPE_CGROUP_ARRAY = 8,
● BPF_MAP_TYPE_LRU_HASH = 9,
● BPF_MAP_TYPE_LRU_PERCPU_HASH = 10,
● BPF_MAP_TYPE_LPM_TRIE = 11,
● BPF_MAP_TYPE_ARRAY_OF_MAPS = 12,
● BPF_MAP_TYPE_HASH_OF_MAPS = 13,
● BPF_MAP_TYPE_DEVMAP = 14,
● BPF_MAP_TYPE_SOCKMAP = 15,
● BPF_MAP_TYPE_CPUMAP = 16,
● BPF_MAP_TYPE_XSKMAP = 17,
● BPF_MAP_TYPE_SOCKHASH = 18,
● BPF_MAP_TYPE_CGROUP_STORAGE = 19,
● BPF_MAP_TYPE_REUSEPORT_SOCKARRAY = 20,
● BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE = 21,
● BPF_MAP_TYPE_QUEUE = 22,
● BPF_MAP_TYPE_STACK = 23,
● BPF_MAP_TYPE_SK_STORAGE = 24,
● BPF_MAP_TYPE_DEVMAP_HASH = 25,
● BPF_MAP_TYPE_STRUCT_OPS = 26,
● BPF_MAP_TYPE_RINGBUF = 27,
● BPF_MAP_TYPE_INODE_STORAGE = 28,

Agenda
Developer
experience

● BCC
− BCC is a toolkit for creating efficient kernel tracing and manipulation programs
− Contains lots of examples
− Kernel instrumentation is written in C
− Python and Lua frontends
● Dynamic generated C source in Python source looks really ugly
Frontends

● BPFTrace
− High level, fixed scope tracing language
− Solves portability
− Language is inspired by awk and C, and predecessor tracers such as Dtrace
− Many of the BCC examples have rewritten in BPFTrace
− Supports one liners
● bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("%s %sn", comm,
str(args->ﬁlename)); }
− Kubectl plugin exists: kubectl-trace
− Easy to learn:
● Trace all EXT4 reads in the given mount point
https://github.com/mhmxs/bpftrace/pull/1/files
Frontends

● Gobpf
− Provides Go binding for BCC Framework
− Low level utils to load and use eBPF programs
− The same as BCC:
● Kernel instrumentation is written in C
● Python - Go
Frontends

● Cilium/ebpf
− Pure Go library that provides utilities for loading, compiling, and debugging eBPF
programs
− Contains lots of examples
− Useful helper functions
− Kernel instrumentation is written in ASM
● Generated with Go code
− Kernel instrumentation is written in C
● Generates Go bindings
Frontends

Agenda
Portability
and
debugging

● By default eBPF program has to match with kernel
− Function signatures can change
− Data structures can change
● What options we have to increase portability
− Use BPFTrace if possible because it just works
− Deal with kernel version match
Portability

● Helpers to deal with it
● Use Cilium/ebpf because of it’s handy helpers
● Bpftool is able to dump kernel headers
● bpftool btf dump ﬁle /sys/kernel/btf/vmlinux format c > vmlinux.h
● High-level BPF CO-RE mechanics
● The CO-RE is a set of macros to generate memory accessors on the fly
● Read memory
● Field exists
● So on...
−
Portability

● Kernel memory is not readable directly
− bpf_core_read() function reads the memory
● Kernel structs are randomly ordered
● High-level BPF CO-RE mechanics
− BPF_CORE_READ(ﬁle, f_path.dentry, d_iname); // path of data
− With regular bpf_core_read() each f_path, dentry, d_name needs to read into a
separated variable
Portability

● Hard to debug
● Many times there is no error, just does nothing
● BPF calls are also traceable
− Needs to recompile the kernel
− Needs to disable JIT compiler
● Rbpf is a eBPF virtual machine in Rust
Debugging

Agenda
Introduce kubectl
gadget plugin

● I LOVE eBPF
● Lot’s of opportunities from AI driven storage miner detector to real-time file monitoring
● With a bit of kernel knowledge it is easy to react on almost any kind event
● Several frontends, helpers and other libraries
● Bunch of existing projects – real world experience
● Kubernetes integration depends on distribution/platform
● C is mandatory at the end of the day
● Really hard to debug
SUMM()

● eBPF for SRE with Reilably: https://dev.to/reliably/ebpf-for-sre-with-reliably-18dc
● Tracing Go function arguments in prod: https://blog.px.dev/ebpf-function-tracing/post/
● Tracing SSL/TLS connections: https://blog.px.dev/ebpf-openssl-tracing
Extra reading

eBPF in the view of a storage developer

Recommended

More Related Content

What's hot (20)

Similar to eBPF in the view of a storage developer (20)

More from Richárd Kovács (6)

Recently uploaded (20)

eBPF in the view of a storage developer