Linux kernel tracing

COSCUP 2016 –
Linux Kernel Tracing
Viller Hsiao <villerhsiao@gmail.com>
Aug. 21, 2016

02/09/2016 2
Who am I ?
Viller Hsiao
Embedded Linux / RTOS engineer
http://image.dfdaily.com/
2012/5/4/6347169311287512
50504b050c1_nEO_IMG.jpg

02/09/2016 3
What's Tracing
https://www.tnooz.com/wp-content/uploads/2010/12/tripadvisor-facebook-rampup1.jpg

02/09/2016 4
What's Tracing
●
Famous way in C: printf()

    void myfunc(int type)
    {
            if (type > 20) {
                 /* do some things */
                 printf (“I like it goes here!n”);
            } else if (type < 100) {
                /* do other things */
                 printf (“But it goes here!n”);
           } else {
                /* error handling */
                 printf (“Oh! I hate it's here! Wrong type is %dn”, type);
           }
    }

02/09/2016 5
What's tracing data used for?
Observe program behavior

02/09/2016 6
Debug program

02/09/2016 7
Debug program
Profile and get statistics
and so on

02/09/2016 8
Wellknown tool in kernel:
printk()
printk() is intuitive, but

02/09/2016 9
Issue of printk()
High overhead
“using printk(), especially when writing to the serial
console, may take several milliseconds per write.” ~ [1]

02/09/2016 10
Issue of printk()
High overhead
Lack of flexibility

02/09/2016 11
Topic today
Systematic tracing mechanisms in Linux kernel
How kernel exhausts compiler and CPU tricks to implement
flexible and low overhead system tracing

02/09/2016 12
Tracing in Linux
Tracing Implementations
Tracing Frameworks
Frontend Toolsuser
Interface for userspace
kernel

02/09/2016 14
ftrace
●
Linux2.6.27
●
Linux kernel internal tracer framework
– Function tracer
– Tracing data output
– Tracepoint
– hist triggers

02/09/2016 15
Function Tracer
  void Func ( … )
  {
      Line 1;
      Line 2;
      …
  }

  void Func ( … )
  {
      mcount (pc, ra);
      Line 1;
      Line 2;
      …
  }
gcc pg
Reuse gprof mechanism, then reimplement mcount()

02/09/2016 16
Function Tracer
  void Func ( … )
  {
      Line 1;
      Line 2;
      …
  }

  void Func ( … )
  {
      mcount (pc, ra);
      Line 1;
      Line 2;
      …
  }
gcc pg
Data recorded: function and its caller

02/09/2016 17
Dynamic Function Tracer

  void Func ( … )
  {
      nop;
      Line 1;
      Line 2;
      …
  }

  void Func ( … )
  {
      mcount (pc, ra);
      Line 1;
      Line 2;
      …
  }
Enabled
Disabled

02/09/2016 18
Tracing Data Output
●
trace_printk()
●
/sys/kernel/debug/tracing/
– tracefs (debugfs in the beginning)
“Writing into the ring buffer with trace_printk() only takes around a
tenth of a microsecond or so” ~ [1]

02/09/2016 19
Example: Function Tracer

02/09/2016 20
Example: Function Graph Tracer

02/09/2016 22
Tracepoint
●
Linux2.6.32
●
Define and insert hook in static point like
printk()

02/09/2016 23
Tracepoint – Declare Event
   #include <linux/tracepoint.h>

    TRACE_EVENT(mm_page_allocation,
TP_PROTO(unsigned long pfn, unsigned long free),
TP_ARGS(pfn, free),
TP_STRUCT__entry(
__field(unsigned long, pfn)
__field(unsigned long, free)
),
TP_fast_assign(
__entry>pfn = pfn;
__entry>free = free;
),
TP_printk("pfn=%lx zone_free=%ld", __entry>pfn, __entry>free)
);

02/09/2016 24
Tracepoint – Probe Event
       . . .
        trace_mm_page_allocation(page_to_pfn(page),
     zone_page_state(zone, NR_FREE_PAGES));
        . . .
Data recorded: custom defined data

02/09/2016 25
Example: Tracepoint

02/09/2016 26
tracecmd
  # tracecmd record e 'sched_wakeup*' e sched_switch yourapplication

  # kernelshark

02/09/2016 27
Kernelshark
https://static.lwn.net/images/2011/ks-fail1-open.png

02/09/2016 28
hist triggers
●
Introduced in Linux4.7
●
Create custom, efficient, inkernel histograms
# echo 'hist:key=common_pid.execname:values=ret:sort=ret if ret >= 0'
> /sys/kernel/tracing/events/syscalls/sys_exit_read/trigger

02/09/2016 29
Example hist triggers Logs
# cat /sys/kernel/tracing/events/syscalls/sys_exit_read/hist
[...]
{ common_pid: bash [ 16608] } hitcount: 4 ret: 11722
{ common_pid: irqbalance [ 1189] } hitcount: 36 ret: 21702
{ common_pid: snmpd [ 1617] } hitcount: 75 ret: 22078
{ common_pid: sshd [ 32745] } hitcount: 329 ret: 165710
[...]
http://www.brendangregg.com/blog/2016-06-08/linux-hist-triggers.html

02/09/2016 31
Kprobe
●
Linux2.6.9
●
Write probe hooks in kernel module
kernel
user
register_kprobe()
Insert
kprobe module
pre()
post()
addr

02/09/2016 32
Kprobe
INST BREAK
register_kprobe()
address
sym + offset

02/09/2016 33
Kprobe
BREAKBREAK INST
pre_handler()
post_handler()
exception
address
save regs
restore regs

02/09/2016 34
Kprobe
BREAKBREAK INST
pre_handler(pt_regs)
post_handler(pt_regs)
exception
address
save regs
restore regs
Data recorded: CPU register values

02/09/2016 35
Kprobe Variants
Kernel
user
Kprobe
Kretprobe
Jprobe
Uprobe

02/09/2016 36
Uprobe
echo 'p:myapp /bin/bash:0x4245c0' > /sys/kernel/tracing/uprobe_events
●
Linux3.5
●
userspace breakpoints in kernel

02/09/2016 37
jprobe
data: probed function
arguments

02/09/2016 38
jprobe
http://pds19.egloos.com/pds/201008/02/35/c0098335_4c55a764e1689.png

02/09/2016 40
kretprobe
http://cfile26.uf.tistory.com/image/1311D5455136D6AF3B7251

02/09/2016 41
Kprobe Overhead [7]
cycles per iteration
                AMD Athlon 1.7GH         Pentium III 860MHz
kprobe     0.99 us                            0.95 us
jprobe      0.82 us                            1.61 us

02/09/2016 42
Kprobebased Event Tracing
# echo 'r:myretprobe do_sys_open $retval' >> /sys/kernel/tracing/kprobe_events
# echo 1 > /sys/kernel/tracing/events/kprobes/myretprobe/enable
# cat /sys/kernel/tracing/trace
# tracer: nop
#
#           TASKPID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
              sh746   [000] d...   40.96: myretprobe: (SyS_open+0x2c/0x30 < do_sys_open) arg1=0x3
              sh746   [000] d...   42.19: myretprobe: (SyS_open+0x2c/0x30 < do_sys_open) arg1=0x3
…..

02/09/2016 43
Utilities for Kprobe
●
tracefs files
– perf probe
●
systemtap
– debuted in 2005 in Red Hat Enterprise Linux 4
– Probe by DSL script based on kprobe

02/09/2016 44
Userspace Scripts: systemtap
kernel
user
kprobe, ...
foo.stp systemtap
debuginfo
foo.ko
relayfs
output
kprobe
tracepoint
syscall
...

02/09/2016 45
perf + Tracing

02/09/2016 46
perf
●
Linux2.6.31
●
Statistics data
# perf stat myapp args
●
Sampling record
# perf record myapp args
●
Other sub cmds of perf tool
perftool
perf framework
kernel
user
perf_event
PMU
CPU
Performance Monitors

02/09/2016 47
perf Events
perftool
perf framework
kernel
user
HW event
perf_event syscall
SW event
PMU
trace
event
trace
point
dynamic
event
kprobe
uprobe
CPU
Counters

02/09/2016 48
perf Events
# perf record e 'syscalls:sys_enter_*' a g sleep 60

02/09/2016 49
Flame Graph
http://deliveryimages.acm.org/10.1145/2930000/2927301/gregg6.png

02/09/2016 50
Flame Graph
http://www.brendangregg.com/FlameGraphs/cpu-bash-flamegraph.png

02/09/2016 51
Flame Graph Tools
for perf Data
# perf record F 99 a g sleep 60
# perf script > out.perf
# /path/to/flamegraph/stackcollapseperf.pl out.perf > out.folded
# /path/to/flamegraph/flamegraph.pl out.kern_folded > kernel.svg

02/09/2016 53
LTTng
http://lttng.org/images/docs27/plumbing-27.png

02/09/2016 54
Eclipse LTTng Support
https://wiki.eclipse.org/images/e/ec/LTTngPerspective.png

02/09/2016 55
Disadvantage of
Previous Kernel Tracing
●
Components are isolated
●
Complex filters and scripts can be expensive
●
Need more comprehensive tools. Some solutions
– systemtap
– LTTng
– Dtrace
– ktap

02/09/2016 56
Tracing + eBPF

02/09/2016 57
network
stack
sniffer
kernel
user
net if
Applications
tcpdump nnnX port 3000
port 3000
VM filter
(BPF) http://www.ic
onsdb.com/ico
ns/download/g
ray/empty-fil
ter-512.png
BPF – Inkernel Packet Filter

02/09/2016 58
eBPF
●
(Linux3.15) Redesigned by Alexei Starovoitov
– Write programs in restricted C
●
compile to BPF with LLVM
– Justintime map to modern 64bit CPU with
minimal performance overhead

02/09/2016 59
Areas Use eBPF
more than a filter today
●
Seccomp filters of syscalls (chrome sandboxing)
●
Packet classifier for traffic contol
●
Actions for traffic control
●
Xtables packet filtering
●
Tracing
– (Linux4.1) attach to kprobe
– (Linux4.7) attach to tracepoint

02/09/2016 60
eBPF Architecture
BPF
binary
MAP
helper
subsys
Other
subsys
BPF_PROG_RUN
BPF
binary
kernel
user
BPF Interpreter/JIT
bpf syscall
verifier
Tracer

02/09/2016 61
Write Customized Tracing Script
Is Possible Now!

02/09/2016 62
eBPF Utilitiy – IO Visor BCC
Frontend
python, lua
llvm library
BPF bytecode
libbcc.so
BPF C text/code
BCC module
BCC
bpf syscallperf event / trace_fs
User
program

02/09/2016 63
Current Tracing Scripts
in BCC
https://raw.githubusercontent.com/iovisor/bcc/master/images/bcc_tracing_tools_2016.png
Tools for BPFbased Linux IO analysis, networking, monitoring, and
more

02/09/2016 64
perf + eBPF [8]
●
Linux4.8rc (?) by Wang Nan in Huawei
●
Ongoning staff and future plans
– Load BPF
– Tracing rare outliner
– Integrate LLVM and other frontend

02/09/2016 66
Linux Kernel Tracing
kprobe
uprobe
function
tracer
tracepoint
ftrace, hist trigger, perf, eBPF
systemtap
perftool
BCC
LTTng
Flamegraph
tracecmd
Kernelshark
eBPF
library

9/2/16 68/70
Reference
[1] Steven Rostedt (Dec. 2009), “Debugging the kernel using Ftrace part 1”, LWN
[2] Steven Rostedt (Feb. 2011), “Using KernelShark to analyze the realtime scheduler”, LWN
[3] 章亦春 , “ 动态追踪技术漫谈”
[4] Brendan Gregg, (Feb. 2016), "Linux 4.x Performance Using BPF Superpowers",
presented at Performance@ scale 2016
[5] Gary Lin (Mar. 2016), “eBPF: Trace from Kernel to Userspace ”, presented at OpenSUSE
Technology Sharing Day 2016
[6] Kernel documentation, “Using the Linux Kernel Tracepoints”
[7] William Cohen (Feb. 2005), “cost of kprobe and jprobe operations”, systemtap mailing list
[8] Wang Nan (Aug. 2016), “Performance Monitoring and AnalysisUsing perf+BPF” ,
LinuxCon North America 2016

9/2/16 69/70
● COSCUP is the Conference for Open Source Coders, Users and Promoters in Taiwan.
● iovisor is a project of Linux Foundation
● ARM are trademarks or registered trademarks of ARM Holdings.
● Linux Foundation is a registered trademark of The Linux Foundation.
● Linux is a registered trademark of Linus Torvalds.
● Other company, product, and service names may be trademarks or service marks
of others.
● The license of each graph belongs to each website listed individually.
● The others of my work in the slide is licensed under a CC-BY-SA License.
● License text: http://creativecommons.org/licenses/by-sa/4.0/legalcode
Rights to Copy
copyright © 2016 Viller Hsiao

Linux kernel tracing

Recommended

More Related Content

What's hot (20)

More from Viller Hsiao (9)

Recently uploaded (20)

Linux kernel tracing