EuroBSDcon 2017 System Performance Analysis Methodologies
24 likes15,817 views
keynote by Brendan Gregg. "Traditional performance monitoring makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. Modern BSD has advanced tracers and PMC tools, providing virtually endless metrics to aid performance analysis. It's time we really used them, but the problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems. There's a new way to approach performance analysis that can guide you through the metrics. Instead of starting with traditional metrics and figuring out their use, you start with the questions you want answered then look for metrics to answer them. Methodologies can provide these questions, as well as a starting point for analysis and guidance for locating the root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, chain graphs, and more. This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. Many methodologies will be discussed, from the production proven to the cutting edge, along with recommendations for their implementation on BSD systems. In general, you will learn to think differently about analyzing your systems, and make better use of the modern tools that BSD provides."
![History
• System Performance Analysis up to the '90s:
– Closed source UNIXes and applicaNons
– Vendor-created metrics and performance tools
– Users interpret given metrics
• Problems
– Vendors may not provide the best metrics
– ORen had to infer, rather than measure
– Given metrics, what do we do with them?
$ ps -auxw
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND
root 11 99.9 0.0 0 16 - RL 22:10 22:27.05 [idle]
root 0 0.0 0.0 0 176 - DLs 22:10 0:00.47 [kernel]
root 1 0.0 0.2 5408 1040 - ILs 22:10 0:00.01 /sbin/init --
[…]](https://image.slidesharecdn.com/eurobsdcon2017systemmethodology-170924184731/85/EuroBSDcon-2017-System-Performance-Analysis-Methodologies-6-320.jpg)
![TSA: Finding FreeBSD Thread States
# dtrace -ln sched:::
ID PROVIDER MODULE FUNCTION NAME
56622 sched kernel none preempt
56627 sched kernel none dequeue
56628 sched kernel none enqueue
56631 sched kernel none off-cpu
56632 sched kernel none on-cpu
56633 sched kernel none remain-cpu
56634 sched kernel none surrender
56640 sched kernel none sleep
56641 sched kernel none wakeup
[…]
struct thread {
[…]
enum {
TDS_INACTIVE = 0x0,
TDS_INHIBITED,
TDS_CAN_RUN,
TDS_RUNQ,
TDS_RUNNING
} td_state;
[…]
#define KTDSTATE(td)
(((td)->td_inhibitors & TDI_SLEEPING) != 0 ? "sleep" :
((td)->td_inhibitors & TDI_SUSPENDED) != 0 ? "suspended" :
((td)->td_inhibitors & TDI_SWAPPED) != 0 ? "swapped" :
((td)->td_inhibitors & TDI_LOCK) != 0 ? "blocked" :
((td)->td_inhibitors & TDI_IWAIT) != 0 ? "iwait" : "yielding")
probes
thread flags](https://image.slidesharecdn.com/eurobsdcon2017systemmethodology-170924184731/85/EuroBSDcon-2017-System-Performance-Analysis-Methodologies-33-320.jpg)
![TSA: FreeBSD
# ./tstates.d
Tracing scheduler events... Ctrl-C to end.
^C
Time (ms) per state:
COMM PID CPU RUNQ SLP SUS SWP LCK IWT YLD
irq14: ata0 12 0 0 0 0 0 0 0 0
irq15: ata1 12 0 0 0 0 0 0 9009 0
swi4: clock (0) 12 0 0 0 0 0 0 9761 0
usbus0 14 0 0 8005 0 0 0 0 0
[...]
sshd 807 0 0 10011 0 0 0 0 0
devd 474 0 0 9009 0 0 0 0 0
dtrace 1166 1 4 10006 0 0 0 0 0
sh 936 2 22 5648 0 0 0 0 0
rand_harvestq 6 5 38 9889 0 0 0 0 0
sh 1170 9 0 0 0 0 0 0 0
kernel 0 10 13 0 0 0 0 0 0
sshd 935 14 22 5644 0 0 0 0 0
intr 12 46 276 0 0 0 0 0 0
cksum 1076 929 28 0 480 0 0 0 0
cksum 1170 1499 1029 0 0 0 0 0 0
cksum 1169 1590 1144 0 0 0 0 0 0
idle 11 5856 999 0 0 0 0 0 0
DTrace proof of concept
hjps://github.com/brendangregg/DTrace-tools/blob/master/sched/tstates.d](https://image.slidesharecdn.com/eurobsdcon2017systemmethodology-170924184731/85/EuroBSDcon-2017-System-Performance-Analysis-Methodologies-34-320.jpg)
![CPU Flame Graphs: FreeBSD
• Use either DTrace or pmcstat. Eg, kernel CPU with DTrace:
• Both user & kernel CPU:
git clone https://github.com/brendangregg/FlameGraph; cd FlameGraph
dtrace -n 'profile-99 /arg0/ { @[stack()] = count(); } tick-30s { exit(0); }' > stacks01
stackcollapse.pl < stacks01 | sed 's/kernel`//g' | ./flamegraph.pl > stacks01.svg
hjp://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#DTrace
dtrace -x ustackframes=100 -x stackframes=100 -n '
profile-99 { @[stack(), ustack(), execname] = sum(1); }
tick-30s,END { printa("%k-%k%sn%@dn", @); trunc(@); exit(0); }' > stacks02](https://image.slidesharecdn.com/eurobsdcon2017systemmethodology-170924184731/85/EuroBSDcon-2017-System-Performance-Analysis-Methodologies-37-320.jpg)
![Off-CPU Profiling: FreeBSD
#!/usr/sbin/dtrace -s
#pragma D option ustackframes=100
#pragma D option dynvarsize=32m
sched:::off-cpu /execname == "bsdtar"/ { self->ts = timestamp; }
sched:::on-cpu
/self->ts/
{
@[stack(), ustack(), execname] = sum(timestamp - self->ts);
self->ts = 0;
}
dtrace:::END
{
normalize(@, 1000000);
printa("%k-%k%sn%@dn", @);
}
offcpu.d
Uses DTrace
Warning: can have significant overhead
(scheduler events can be frequent)
Change/remove as desired
eg, add /curthread->td_state <= 1/ to exclude preempt, otherwise sees iCsw
# ./offcpu.d > out.stacks
# git clone https://github.com/brendangregg/FlameGraph; cd FlameGraph
# stackcollapse.pl < ../out.stacks | sed 's/kernel`//g' |
./flamegraph.pl --color=io --title="Off-CPU Flame Graph" --countname=ms > out.svg](https://image.slidesharecdn.com/eurobsdcon2017systemmethodology-170924184731/85/EuroBSDcon-2017-System-Performance-Analysis-Methodologies-42-320.jpg)
![Wakeup Profiling: FreeBSD
#!/usr/sbin/dtrace -s
#pragma D option quiet
#pragma D option ustackframes=100
#pragma D option dynvarsize=32m
sched:::sleep /execname == "bsdtar"/ { ts[curlwpsinfo->pr_addr] = timestamp; }
sched:::wakeup
/ts[arg0]/
{
this->delta = timestamp - ts[arg0];
@[args[1]->p_comm, stack(), ustack(), execname] = sum(this->delta);
ts[arg0] = 0;
}
dtrace:::END
{
normalize(@, 1000000);
printa("n%s%k-%k%sn%@dn", @);
}
wakeup.d
Uses DTrace
Warning: can have significant overhead
(scheduler events can be frequent)
Change/remove as desired](https://image.slidesharecdn.com/eurobsdcon2017systemmethodology-170924184731/85/EuroBSDcon-2017-System-Performance-Analysis-Methodologies-45-320.jpg)
![Ye Olde BPF
Berkeley Packet Filter
# tcpdump host 127.0.0.1 and port 22 -d
(000) ldh [12]
(001) jeq #0x800 jt 2 jf 18
(002) ld [26]
(003) jeq #0x7f000001 jt 6 jf 4
(004) ld [30]
(005) jeq #0x7f000001 jt 6 jf 18
(006) ldb [23]
(007) jeq #0x84 jt 10 jf 8
(008) jeq #0x6 jt 10 jf 9
(009) jeq #0x11 jt 10 jf 18
(010) ldh [20]
(011) jset #0x1fff jt 18 jf 12
(012) ldxb 4*([14]&0xf)
(013) ldh [x + 14]
[...]
User-defined bytecode
executed by an in-kernel
sandboxed virtual machine
Steven McCanne and Van Jacobson, 1993
2 x 32-bit registers
& scratch memory
OpNmizes packet filter
performance](https://image.slidesharecdn.com/eurobsdcon2017systemmethodology-170924184731/85/EuroBSDcon-2017-System-Performance-Analysis-Methodologies-47-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to EuroBSDcon 2017 System Performance Analysis Methodologies (20)
Ad
More from Brendan Gregg (20)
Ad
Recently uploaded (20)
EuroBSDcon 2017 System Performance Analysis Methodologies
- 1. EuroBSDcon 2017 System Performance Analysis Methodologies Brendan Gregg Senior Performance Architect
- 3. ERASABLE MEMORY CORE SET AREA VAC SETS FIXED MEMORY Apollo Lunar Module Guidance Computer performance analysis
- 5. Background
- 6. History • System Performance Analysis up to the '90s: – Closed source UNIXes and applicaNons – Vendor-created metrics and performance tools – Users interpret given metrics • Problems – Vendors may not provide the best metrics – ORen had to infer, rather than measure – Given metrics, what do we do with them? $ ps -auxw USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 99.9 0.0 0 16 - RL 22:10 22:27.05 [idle] root 0 0.0 0.0 0 176 - DLs 22:10 0:00.47 [kernel] root 1 0.0 0.2 5408 1040 - ILs 22:10 0:00.01 /sbin/init -- […]
- 7. Today 1. Open source – OperaNng systems: Linux, BSD, etc. – ApplicaNons: source online (Github) 2. Custom metrics – Can patch the open source, or, – Use dynamic tracing (open source helps) 3. Methodologies – Start with the quesNons, then make metrics to answer them – Methodologies can pose the quesNons Biggest problem with dynamic tracing has been what to do with it. Methodologies guide your usage.
- 10. Street Light An2-Method 1. Pick observability tools that are – Familiar – Found on the Internet – Found at random 2. Run tools 3. Look for obvious issues
- 12. Blame Someone Else An2-Method 1. Find a system or environment component you are not responsible for 2. Hypothesize that the issue is with that component 3. Redirect the issue to the responsible team 4. When proven wrong, go to 1
- 13. Traffic Light An2-Method 1. Turn all metrics into traffic lights 2. Open dashboard 3. Everything green? No worries, mate. • Type I errors: red instead of green – team wastes Nme • Type II errors: green instead of red – performance issues undiagnosed – team wastes more Nme looking elsewhere Traffic lights are suitable for objec2ve metrics (eg, errors), not subjec2ve metrics (eg, IOPS, latency).
- 14. Methodologies
- 15. Performance Methodologies System Methodologies: – Problem statement method – FuncNonal diagram method – Workload analysis – Workload characterizaNon – Resource analysis – USE method – Thread State Analysis – On-CPU analysis – CPU flame graph analysis – Off-CPU analysis – Latency correlaNons – Checklists – StaNc performance tuning – Tools-based methods … • For system engineers: – ways to analyze unfamiliar systems and applicaNons • For app developers: – guidance for metric and dashboard design Collect your own toolbox of methodologies
- 16. Problem Statement Method 1. What makes you think there is a performance problem? 2. Has this system ever performed well? 3. What has changed recently? – soRware? hardware? load? 4. Can the problem be described in terms of latency? – or run Nme. not IOPS or throughput. 5. Does the problem affect other people or apps? 6. What is the environment? – soRware, hardware, instance types? versions? config?
- 17. FuncNonal Diagram Method 1. Draw the funcNonal diagram 2. Trace all components in the data path 3. For each component, check performance Breaks up a bigger problem into smaller, relevant parts Eg, imagine throughput between the UCSB 360 and the UTAH PDP10 was slow… ARPA Network 1969
- 18. Workload Analysis • Begin with applicaNon metrics & context • A drill-down methodology • Pros: – ProporNonal, accurate metrics – App context • Cons: – Difficult to dig from app to resource – App specific ApplicaNon System Libraries System Calls Kernel Hardware Workload Analysis
- 19. Workload CharacterizaNon • Check the workload, not resulNng performance • Eg, for CPUs: 1. Who: which PIDs, programs, users 2. Why: code paths, context 3. What: CPU instrucNons, cycles 4. How: changing over Nme Target Workload
- 20. Workload CharacterizaNon: CPUs Who How What Why top CPU profile CPU flame graphs monitoring PMCs CPI flame graph
- 22. Resource Analysis • Typical approach for system performance analysis: begin with system tools & metrics • Pros: – Generic – Aids resource perf tuning • Cons: – Uneven coverage – False posiNves ApplicaNon System Libraries System Calls Kernel Hardware Workload Analysis
- 23. The USE Method • For every resource, check: 1. Utilization: busy time 2. Saturation: queue length or time 3. Errors: easy to interpret (objective) Starts with the questions, then finds the tools Eg, for hardware, check every resource incl. busses:
- 27. ERASABLE MEMORY CORE SET AREA VAC SETS FIXED MEMORY Apollo Lunar Module Guidance Computer performance analysis
- 28. USE Method: SoRware • USE method can also work for soRware resources – kernel or app internals, cloud environments – small scale (eg, locks) to large scale (apps). Eg: • Mutex locks: – uNlizaNon à lock hold Nme – saturaNon à lock contenNon – errors à any errors • EnNre applicaNon: – uNlizaNon à percentage of worker threads busy – saturaNon à length of queued work – errors à request errors Resource UNlizaNon (%) X
- 29. RED Method • For every service, check these are within SLO/A: 1. Request rate 2. Error rate 3. Dura=on (distribuNon) Another exercise in posing quesNons from funcNonal diagrams By Tom Wilkie: hjp://www.slideshare.net/weaveworks/monitoring-microservices Load Balancer Web Proxy Web Server User Database Payments Server Asset Server Metrics Database
- 33. TSA: Finding FreeBSD Thread States # dtrace -ln sched::: ID PROVIDER MODULE FUNCTION NAME 56622 sched kernel none preempt 56627 sched kernel none dequeue 56628 sched kernel none enqueue 56631 sched kernel none off-cpu 56632 sched kernel none on-cpu 56633 sched kernel none remain-cpu 56634 sched kernel none surrender 56640 sched kernel none sleep 56641 sched kernel none wakeup […] struct thread { […] enum { TDS_INACTIVE = 0x0, TDS_INHIBITED, TDS_CAN_RUN, TDS_RUNQ, TDS_RUNNING } td_state; […] #define KTDSTATE(td) (((td)->td_inhibitors & TDI_SLEEPING) != 0 ? "sleep" : ((td)->td_inhibitors & TDI_SUSPENDED) != 0 ? "suspended" : ((td)->td_inhibitors & TDI_SWAPPED) != 0 ? "swapped" : ((td)->td_inhibitors & TDI_LOCK) != 0 ? "blocked" : ((td)->td_inhibitors & TDI_IWAIT) != 0 ? "iwait" : "yielding") probes thread flags
- 34. TSA: FreeBSD # ./tstates.d Tracing scheduler events... Ctrl-C to end. ^C Time (ms) per state: COMM PID CPU RUNQ SLP SUS SWP LCK IWT YLD irq14: ata0 12 0 0 0 0 0 0 0 0 irq15: ata1 12 0 0 0 0 0 0 9009 0 swi4: clock (0) 12 0 0 0 0 0 0 9761 0 usbus0 14 0 0 8005 0 0 0 0 0 [...] sshd 807 0 0 10011 0 0 0 0 0 devd 474 0 0 9009 0 0 0 0 0 dtrace 1166 1 4 10006 0 0 0 0 0 sh 936 2 22 5648 0 0 0 0 0 rand_harvestq 6 5 38 9889 0 0 0 0 0 sh 1170 9 0 0 0 0 0 0 0 kernel 0 10 13 0 0 0 0 0 0 sshd 935 14 22 5644 0 0 0 0 0 intr 12 46 276 0 0 0 0 0 0 cksum 1076 929 28 0 480 0 0 0 0 cksum 1170 1499 1029 0 0 0 0 0 0 cksum 1169 1590 1144 0 0 0 0 0 0 idle 11 5856 999 0 0 0 0 0 0 DTrace proof of concept hjps://github.com/brendangregg/DTrace-tools/blob/master/sched/tstates.d
- 35. On-CPU Analysis 1. Split into user/kernel states – /proc, vmstat(1) 2. Check CPU balance – mpstat(1), CPU uNlizaNon heat map 3. Profile soRware – User & kernel stack sampling (as a CPU flame graph) 4. Profile cycles, caches, busses – PMCs, CPI flame graph CPU UNlizaNon Heat Map
- 36. CPU Flame Graph Analysis 1. Take a CPU profile 2. Render it as a flame graph 3. Study largest "towers" first Discovers issues by their CPU usage - Directly: CPU consumers - Indirectly: iniNalizaNon of I/O, locks, Nmes, ... Narrows target of study Flame Graph
- 37. CPU Flame Graphs: FreeBSD • Use either DTrace or pmcstat. Eg, kernel CPU with DTrace: • Both user & kernel CPU: git clone https://github.com/brendangregg/FlameGraph; cd FlameGraph dtrace -n 'profile-99 /arg0/ { @[stack()] = count(); } tick-30s { exit(0); }' > stacks01 stackcollapse.pl < stacks01 | sed 's/kernel`//g' | ./flamegraph.pl > stacks01.svg hjp://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html#DTrace dtrace -x ustackframes=100 -x stackframes=100 -n ' profile-99 { @[stack(), ustack(), execname] = sum(1); } tick-30s,END { printa("%k-%k%sn%@dn", @); trunc(@); exit(0); }' > stacks02
- 38. Java Mixed-Mode CPU Flame Graph Java Kernel (C) JVM (C++) User (C) By sampling stack traces with: • -XX:+PreserveFramePointer • Java perf-map-agent
- 39. CPI Flame Graph: BSD A CPU flame graph (cycles) colored using instructions/stall profile data eg, using FreeBSD pmcstat: red == instrucNons blue == stalls hjp://www.brendangregg.com/blog/2014-10-31/cpi-flame-graphs.html
- 42. Off-CPU Profiling: FreeBSD #!/usr/sbin/dtrace -s #pragma D option ustackframes=100 #pragma D option dynvarsize=32m sched:::off-cpu /execname == "bsdtar"/ { self->ts = timestamp; } sched:::on-cpu /self->ts/ { @[stack(), ustack(), execname] = sum(timestamp - self->ts); self->ts = 0; } dtrace:::END { normalize(@, 1000000); printa("%k-%k%sn%@dn", @); } offcpu.d Uses DTrace Warning: can have significant overhead (scheduler events can be frequent) Change/remove as desired eg, add /curthread->td_state <= 1/ to exclude preempt, otherwise sees iCsw # ./offcpu.d > out.stacks # git clone https://github.com/brendangregg/FlameGraph; cd FlameGraph # stackcollapse.pl < ../out.stacks | sed 's/kernel`//g' | ./flamegraph.pl --color=io --title="Off-CPU Flame Graph" --countname=ms > out.svg
- 44. Wakeup Time Flame Graph: FreeBSD Who did the wakeup: waker wakee user-stack kernel-stack
- 45. Wakeup Profiling: FreeBSD #!/usr/sbin/dtrace -s #pragma D option quiet #pragma D option ustackframes=100 #pragma D option dynvarsize=32m sched:::sleep /execname == "bsdtar"/ { ts[curlwpsinfo->pr_addr] = timestamp; } sched:::wakeup /ts[arg0]/ { this->delta = timestamp - ts[arg0]; @[args[1]->p_comm, stack(), ustack(), execname] = sum(this->delta); ts[arg0] = 0; } dtrace:::END { normalize(@, 1000000); printa("n%s%k-%k%sn%@dn", @); } wakeup.d Uses DTrace Warning: can have significant overhead (scheduler events can be frequent) Change/remove as desired
- 47. Ye Olde BPF Berkeley Packet Filter # tcpdump host 127.0.0.1 and port 22 -d (000) ldh [12] (001) jeq #0x800 jt 2 jf 18 (002) ld [26] (003) jeq #0x7f000001 jt 6 jf 4 (004) ld [30] (005) jeq #0x7f000001 jt 6 jf 18 (006) ldb [23] (007) jeq #0x84 jt 10 jf 8 (008) jeq #0x6 jt 10 jf 9 (009) jeq #0x11 jt 10 jf 18 (010) ldh [20] (011) jset #0x1fff jt 18 jf 12 (012) ldxb 4*([14]&0xf) (013) ldh [x + 14] [...] User-defined bytecode executed by an in-kernel sandboxed virtual machine Steven McCanne and Van Jacobson, 1993 2 x 32-bit registers & scratch memory OpNmizes packet filter performance
- 50. Latency CorrelaNons 1. Measure latency histograms at different stack layers 2. Compare histograms to find latency origin Even bejer, use latency heat maps • Match outliers based on both latency and Nme
- 51. Checklists: eg, BSD Perf Analysis in 60s 1. uptime 2. dmesg -a | tail 3. vmstat 1 4. vmstat -P 5. ps -auxw 6. iostat -xz 1 7. systat -ifstat 8. systat -netstat 9. top 10. systat -vmstat load averages kernel errors overall stats by Nme CPU balance process usage disk I/O network I/O TCP stats process overview system overview adapted from hjp://techblog.neylix.com/2015/11/linux-performance-analysis-in-60s.html
- 52. 1. RPS, CPU 2. Volume 6. Load Avg 3. Instances 4. Scaling 5. CPU/RPS 7. Java Heap 8. ParNew 9. Latency 10. 99th Nle Checklists: eg, Neylix perfvitals Dashboard
- 56. Other Methodologies • ScienNfic method • 5 Why's • Process of eliminaNon • Intel's Top-Down Methodology • Method R
- 57. What You Can Do
- 58. What you can do 1. Know what's now possible on modern systems – Dynamic tracing: efficiently instrument any soRware – CPU faciliNes: PMCs, MSRs (model specific registers) – VisualizaNons: flame graphs, latency heat maps, … 2. Ask quesNons first: use methodologies to ask them 3. Then find/build the metrics 4. Build or buy dashboards to support methodologies
- 59. Dynamic Tracing: Efficient Metrics send receive tcpdump Kernel buffer file system 1. read 2. dump Analyzer 1. read 2. process 3. print disks Old way: packet capture New way: dynamic tracing Tracer 1. configure 2. read tcp_retransmit_skb() Eg, tracing TCP retransmits
- 60. Dynamic Tracing: Instrument Most SoRware My Solaris/DTrace tools (many already work on BSD/DTrace):
- 63. Summary • It is the crystal ball age of performance observability • What majers is the quesNons you want answered • Methodologies are a great way to pose quesNons Who How What Why
- 64. References & Resources • FreeBSD @ Neylix: – hjps://openconnect.itp.neylix.com/ – hjp://people.freebsd.org/~scojl/Neylix-BSDCan-20130515.pdf – hjp://www.youtube.com/watch?v=FL5U4wr86L4 • USE Method – hjp://queue.acm.org/detail.cfm?id=2413037 – hjp://www.brendangregg.com/usemethod.html • TSA Method – hjp://www.brendangregg.com/tsamethod.html • Off-CPU Analysis – hjp://www.brendangregg.com/offcpuanalysis.html – hjp://www.brendangregg.com/blog/2016-01-20/ebpf-offcpu-flame-graph.html – hjp://www.brendangregg.com/blog/2016-02-05/ebpf-chaingraph-prototype.html • StaNc Performance Tuning, Richard Elling, Sun blueprint, May 2000 • RED Method: hjp://www.slideshare.net/weaveworks/monitoring-microservices • Other system methodologies – Systems Performance: Enterprise and the Cloud, PrenNce Hall 2013 – hjp://www.brendangregg.com/methodology.html – The Art of Computer Systems Performance Analysis, Jain, R., 1991 • Flame Graphs – hjp://queue.acm.org/detail.cfm?id=2927301 – hjp://www.brendangregg.com/flamegraphs.html – hjp://techblog.neylix.com/2015/07/java-in-flames.html • Latency Heat Maps – hjp://queue.acm.org/detail.cfm?id=1809426 – hjp://www.brendangregg.com/HeatMaps/latency.html • ARPA Network: hjp://www.computerhistory.org/internethistory/1960s • RSTS/E System User's Guide, 1985, page 4-5 • DTrace: Dynamic Tracing in Oracle Solaris, Mac OS X, and FreeBSD, PrenNce Hall 2011 • Apollo: hjp://www.hq.nasa.gov/office/pao/History/alsj/a11 hjp://www.hq.nasa.gov/alsj/alsj-LMdocs.html
- 65. Thank You • hjp://slideshare.net/brendangregg • hjp://www.brendangregg.com • [email protected] • @brendangregg EuroBSDcon 2017