Optimizing Servers for High-Throughput and Low-Latency at Dropbox
0 likes976 views
I'm going to discuss the efficiency/performance optimizations of different layers of the system. Starting from the lowest levels like hardware and drivers: these tunings can be applied to pretty much any high-load server. Then we’ll move to Linux kernel and its TCP/IP stack: these are the knobs you want to try on any of your TCP-heavy boxes. Finally, we’ll discuss library and application-level tunings, which are mostly applicable to HTTP servers in general and nginx/envoy specifically. For each potential area of optimization I’ll try to give some background on latency/throughput tradeoffs (if any), monitoring guidelines, and, finally, suggest tunings for different workloads. Also, I'll cover more theoretical approaches to performance analysis and the newly developed tooling like `bpftrace` and new `perf` features.
![[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/12skt-160226171513-thumbnail.jpg?width=560&fit=bounds)
![[232] 성능어디까지쥐어짜봤니 송태웅](https://cdn.slidesharecdn.com/ss_thumbnails/232-161025013504-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Optimizing Servers for High-Throughput and Low-Latency at Dropbox (20)
![CETH for XDP [Linux Meetup Santa Clara | July 2016]](https://cdn.slidesharecdn.com/ss_thumbnails/ceth5overview1-160801192921-thumbnail.jpg?width=560&fit=bounds)
Ad
More from ScyllaDB (20)
Ad
Recently uploaded (20)
Optimizing Servers for High-Throughput and Low-Latency at Dropbox
- 1. Brought to you by Optimizing Servers for High Throughput and Low Latency Alexey Ivanov Software Engineer at Dapper Labs
- 2. Alexey Ivanov Software Engineer, Dapper Labs ■ Previously: Traffic, Networking, and Databases @Dropbox ■ Performance: Hardware. OS. Application. RUM.
- 3. Optimizing (web-)Servers 5 Years Later… This is an updated version of the nginx.conf’17 talk. Changelog: ■ New hardware features are available. AMD EPYCs and ARM64 are a thing. ■ New linux kernel features. Especially around observability. ■ Replace nginx with a generic HTTP-server/-client focus. ● (Most of the clients and servers nowadays are HTTP- or HTTP/2-based)
- 4. The biggest performance gains are usually gained via high-level optimizations: load-balancing, algorithms,data structures, and (especially) business logic. A few examples from large scale production systems. ■ The lower the variance in backend load – the better. ● Applying “Two Random Choices” load-balancing greatly reduced latencies. ■ The fastest code is “no code”. ● E.g. at Dropbox we’ve pre-compressed static files for web so we spent 0% CPU on it while maintaining the best possible compression ratio. ■ Algorithm improvements. ● Switching from zlib to brotli saved us both CPU and storage. ■ Data locality improvements. ● Switching from B+tree to LSM-based storage improved compression efficiency and reduced database sizes by ~2.5x. High-level vs Low-level Optimizations
- 5. Hardware
- 6. CPU and Memory Generally, picking the newest processor is the best choice since it will have the most hardware offloads: ■ AVX2, BMI, ADX, AVX-512, AES-NI, SHA-NI (x86) ● (Symmetric/Asymmetric encryption, signatures, hashing, MACs) ■ PMUL, PMULL2, SHA256H, SHA3 (ARMv8.2+) ● (finite field arithmetic, hashing, MACs) Many of the things that previously were prohibitively expensive now are almost free due to hardware offloads: mTLS, crypto-hashing, storage encryption.
- 7. CPU and Memory (Cont’d) What if budget is limited? Rules of thumb: ■ Low-latency: single NUMA-node, bigger caches, disabled SMT, more Ghz, more memory channels. ■ High-throughput: more cores, enabled SMT, more memory. Frequently, in production, high CPU usage does not mean a CPU bottleneck but a “CPU pipeline stall” problem, i.e.: cache, TLB, or memory-bandwidth limitation.
- 9. github.com/andikleen/pmu-tools # toplev.py -l1 --single-thread --force-events ./app BE Backend_Bound: 60.34% This category reflects slots where no uops are being delivered due to a lack of required resources for accepting more uops in the Backend of the pipeline...
- 10. github.com/andikleen/pmu-tools # toplev.py -l3 --single-thread --force-events ./app BE Backend_Bound: 60.42% BE/Mem Backend_Bound.Memory_Bound: 32.23% BE/Mem Backend_Bound.Memory_Bound.L1_Bound: 32.44% This metric represents how often CPU was stalled without missing the L1 data cache... BE/Core Backend_Bound.Core_Bound: 45.93% BE/Core Backend_Bound.Core_Bound.Ports_Utilization: 45.93% This metric represents cycles fraction application was stalled due to Core computation issues (non divider- related)...
- 11. NICs Relevant only for real hardware, not clouds. ■ 25Gbits or more, older NICs would likely have misc bottlenecks. ■ Open-source drivers, small firmwares, active community. ● In case if (but most likely, “when”) issues occur.
- 12. Pressure Stall Information (PSI) “PSI provides for the first time a canonical way to see resource pressure increases as they develop, with new pressure metrics for three major resources— memory, CPU, and IO.” Source: https://facebookmicrosites.github.io/psi/docs/overview
- 13. PSI: global and Per-cgroup (v2) $ cat /proc/pressure/io some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0 $ cat /sys/fs/cgroup/cg1/io.pressure some avg10=0.00 avg60=0.00 avg300=0.00 total=0 full avg10=0.00 avg60=0.00 avg300=0.00 total=0
- 14. Understanding Software Dynamics by Richard L. Sites
- 15. Linux Kernel
- 16. Kernel Optimizations The best Linux optimization is the recent kernel version. New kernel versions bring improvements to networking, memory management, io, and the rest of linux subsystems. But most importantly they bring improvements to observability tooling.
- 17. CPU and Memory After you’ve picked the best CPU for your workload, you’ll need to utilize it to the max: ■ For Intel/AMD you would want to use intel_pstate or amd-pstate driver. ● If you want to be more energy efficient you may consider using schedutil governor. Use performance otherwise. ■ Set NUMA affinity for your application. ■ Use transparent huge pages. ● Careful here, this may lead to reduction in performance on some workloads.
- 18. Networking The main goal of low-level tuning is to parallelize packet processing, add affinities, increase buffer sizes, and enable hardware offloads. ■ ethtool is your friend here: # of queues, ring buffers, offloads, coalescing. ● -L, -G, -K, -C, etc. ● -S is your friend to keep track of drops/misses/errors/overruns/etc. ■ Mellanox and Intel cards come with set_irq_affinity/mlnx_affinity. ● Do not forget to turn off irqbalance. ■ After RSS is enabled it is generally a good idea to turn on XPS and xps_rxqs. ■ Avoid RPS. RFS can also have negative consequences. ■ For low latency: try to stay within the NUMA node PCIe NIC is attached to.
- 19. The main goal of high-level tuning is to remove transport-level bottlenecks. ■ Enabling BBR congestion control is generally a good idea. ■ Enabling FQ scheduler w/ pacing is always a good idea. ■ Your friends here are RUM metrics and ss -n --extended --info or getsockopt(TCP_INFO/TCP_CC_INFO) Networking (Cont’d)
- 20. iproute2 $ ss -tie … ts sack bbr rto:220 rtt:16.139/10.041 ato:40 mss:1448 pmtu:1500 rcvmss:1269 advmss:1428 cwnd:106 ssthresh:52 bytes_sent:9070462 bytes_retrans:3375 bytes_acked:9067087 bytes_received:5775 segs_out:6327 segs_in:551 data_segs_out:6315 data_segs_in:12 bbr:(bw:99.5Mbps,mrtt:1.912,pacing_gain:1,cwnd_gain:2) send 76.1Mbps lastsnd:9896 lastrcv:10944 lastack:9864 pacing_rate 98.5Mbps delivery_rate 27.9Mbps delivered:6316 busy:3020ms rwnd_limited:2072ms(68.6%) retrans:0/5 dsack_dups:5 rcv_rtt:16.125 rcv_space:14400 rcv_ssthresh:65535 minrtt:1.907 …
- 21. It is impossible to talk about network tuning w/o mentioning sysctls. Here is a couple of a relatively safe ones. ■ net.ipv4.tcp_slow_start_after_idle=0 ● Should be safe if FQ w/ pacing is enabled. ■ net.ipv4.tcp_mtu_probing=1 ● Must have on the edge (along with a slightly reduced advmss) ■ net.ipv4.tcp_rmem, net.ipv4.tcp_wmem ● Should be big enough for connections to not be rcv/snd window limited. ■ net.ipv4.tcp_notsent_lowat=262144 ● Or even lower if HTTP/2 prioritization is used. Sysctl Cargo Culting
- 23. Application
- 24. Compiler Flags, Toolchains, and Runtimes Keeping you compiler/runtime up-to-date is generally a good idea. ■ Compiler upgrade, -O2, and -mtune can visibly affect performance. ● You can also try keeping -march/GOAMD64 in sync with your (cloud) hardware. ■ Link time optimization (LTO) can give a measurable perf boost. ■ Runtime upgrade can frequently give you single to double digit perf improvements. ● For example, Go runtime upgrades frequently deliver memory/cpu usage improvements. ■ (Toolchain upgrades are also great from the security perspective)
- 25. Profile-guided Optimization and Beyond Most compilers are capable of PGO based on `perf record` profiles. ■ Clang has AutoFDO. ■ Golang would likely have Feedback-Guided Optimization in 1.20. You can go beyond compile-time optimization and use post-link optimizer: ■ Facebook’s BOLT is now a part of LLVM: https://github.com/llvm/llvm-project/tree/main/bolt
- 26. Any modern application consists of a myriad of libraries. Most servers nowadays would have allocator, TLS, compression, and serialization libraries. These are the main candidates for tuning. For example in case of C/C++ servers: ■ Keeping libraries up-to-date is important. ● It doesn’t matter whether CPU supports AVX2 if your library can’t use it. ■ Changing malloc implementation is an option. ● Both jemalloc and tcmalloc have excellent tuning guides. ■ BoringSSL can (mostly) be used as a drop–in replacement for OpenSSL. ● Often switching from RSA to ECDSA, or from AES to ChaCha (or back) can improve perf. ■ zlib has multiple performance-oriented forks. ● Intel, Cloudflare, zlib-ng. ● Sometimes more efficient algorithms like brotli or zstd can be used instead. Libraries
- 28. Site Reliability Engineering Chapter 19. Load Balancing at the Frontend Chapter 20. Load Balancing in the Datacenter Chapter 21. Handling Overload Chapter 22. Addressing Cascading Failures
- 29. Brought to you by Alexey Ivanov [email protected] @SaveTheRbtz