Achieving the ultimate performance with KVM
2 likes579 views
This document summarizes an presentation about achieving ultimate performance with KVM. It discusses optimizing hardware, CPU, memory, networking, and storage for virtual machines. The goal is the lowest cost per delivered resource while meeting performance targets. Specific optimizations mentioned include CPU pinning, huge pages, SR-IOV networking, virtio drivers, and bypassing the host for storage. It cautions that many performance claims use unrealistic benchmarks and hardware configurations unlike real-world usage.
![[Container Runtime Meetup] runc & User Namespaces](https://cdn.slidesharecdn.com/ss_thumbnails/publicruncusernamespacescontainerruntimemeetup1-190924085316-thumbnail.jpg?width=560&fit=bounds)
![[KubeCon NA 2020] containerd: Rootless Containers 2020](https://cdn.slidesharecdn.com/ss_thumbnails/kubeconna2020rootlesscontainers-201117203340-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Achieving the ultimate performance with KVM (20)
Ad
More from ShapeBlue (20)
Ad
Recently uploaded (20)
Achieving the ultimate performance with KVM
- 1. Achieving the ultimate performance with KVM Venko Moyankov at European Cloud Infrastructure & CloudStack User Day London, 2019-10
- 2. Why performance ● Better application performance -- e.g. time to load a page, time to rebuild, time to execute specific query ● Happier customers (in cloud / multi-tenant environments) ● ROI, TCO - Lower cost per delivered resource (per VM) through higher density
- 4. Agenda ● Hardware ● Compute - CPU & Memory ● Networking ● Storage
- 5. Usual optimization goal - lowest cost per delivered resource - fixed performance target - calculate all costs - power, cooling, space, server, network, support/maintenance Example: cost per VM with 4x dedicated 3 GHz cores and 16 GB RAM Unusual - Best single-thread performance I can get at any cost - 5 GHz cores, yummy :) Compute node hardware
- 7. Compute node hardware Intel lowest cost per core: - Xeon Gold 6222V - 20 cores @ 2.4 GHz lowest cost per 3GHz+ core: - Xeon Gold 6210U - 20 cores @ 3.2 GHz - Xeon Gold 6240 - 18 cores @ 3.3 GHz - Xeon Gold 6248 - 20 cores @ 3.2 GHz AMD - EPYC 7702P - 64 cores @ 2.0/3.35 GHz - lowest cost per core - EPYC 7402P - 24 cores / 1S - low density - EPYC 7742 - 64 cores @ 2.2/3.4GHz x 2S - max density
- 8. Compute node hardware Form factor from to
- 9. Compute node hardware ● firmware versions and BIOS settings ● Understand power management -- esp. C-states, P-states, HWP and “bias” ○ Different on AMD EPYC: "power-deterministic", "performance-deterministic" ● Think of rack level optimization - how do we get the lowest total cost per delivered resource?
- 10. Agenda ● Hardware ● Compute - CPU & Memory ● Networking ● Storage
- 11. Tuning KVM RHEL7 Virtualization_Tuning_and_Optimization_Guide link https://pve.proxmox.com/wiki/Performance_Tweaks https://events.static.linuxfound.org/sites/events/files/slides/CloudOpen2013_Khoa_Huynh_v3.pdf http://www.linux-kvm.org/images/f/f9/2012-forum-virtio-blk-performance-improvement.pdf http://www.slideshare.net/janghoonsim/kvm-performance-optimization-for-ubuntu … but don’t trust everything you read. Perform your own benchmarking!
- 12. CPU and Memory Recent Linux kernel, KVM and QEMU … but beware of the bleeding edge E.g. qemu-kvm-ev from RHEV (repackaged by CentOS) tuned-adm virtual-host tuned-adm virtual-guest
- 13. CPU Typical ● (heavy) oversubscription, because VMs are mostly idling ● HT ● NUMA ● route IRQs of network and storage adapters to a core on the NUMA node they are on Unusual ● CPU Pinning
- 14. Understanding oversubscription and congestion Linux scheduler statistics: linux-stable/Documentation/scheduler/sched-stats.txt Next three are statistics describing scheduling latency: 7) sum of all time spent running by tasks on this processor (in jiffies) 8) sum of all time spent waiting to run by tasks on this processor (in jiffies) 9) # of timeslices run on this cpu 20% CPU load with large wait time (bursty congestion) is possible 100% CPU load with no wait time, also possible Measure CPU congestion!
- 15. Understanding oversubscription and congestion
- 16. Memory Typical ● Dedicated RAM ● huge pages, THP ● NUMA ● use local-node memory if you can Unusual ● Oversubscribed RAM ● balloon ● KSM (RAM dedup)
- 17. Agenda ● Hardware ● Compute - CPU & Memory ● Networking ● Storage
- 18. Networking Virtualized networking Use virtio-net driver regular virtio vs vhost_net Linux Bridge vs OVS in-kernel vs OVS-DPDK Pass-through networking SR-IOV (PCIe pass-through)
- 20. Networking - vhost Qemu VM Kernel Kernel User space vhost
- 21. Networking - vhost-user Qemu VM Kernel Kernel User space vhost
- 22. ● Direct exclusive access to the PCI device ● SR-IOV - one physical device appears as multiple virtual functions (VF) ● Allows different VMs to share a single PCIe hardware Host NIC VF1 Hypervisor / VMM VM Host driver driver VM driver VM driver VF2 VF3PF PCIe IOMMU / VT-d Networking - PCI Passthrough and SR-IOV
- 23. Discussion
- 24. Agenda ● Hardware ● Compute - CPU & Memory ● Networking ● Storage
- 25. Storage - virtualization Virtualized cache=none -- direct IO, bypass host buffer cache io=native -- use Linux Native AIO, not POSIX AIO (threads) virtio-blk vs virtio-scsi virtio-scsi multiqueue iothread vs. Full bypass SR-IOV for NVMe devices
- 26. Storage - vhost Virtualized with qemu bypass vhost before: guest kernel -> host kernel -> qemu -> host kernel -> storage system after: guest kernel -> storage system
- 27. storpool_server instance 1 CPU thread 2-4 GB RAM NIC storpool_server instance 1 CPU thread 2-4 GB RAM storpool_server instance 1 CPU thread 2-4 GB RAM • Highly scalable and efficient architecture • Scales up in each storage node & out with multiple nodes 25GbE . . . 25GbE storpool_block instance 1 CPU thread NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD NVMe SSD KVM Virtual Machine KVM Virtual Machine
- 28. Storage benchmarks Beware: lots of snake oil out there! ● performance numbers from hardware configurations totally unlike what you’d use in production ● synthetic tests with high iodepth - 10 nodes, 10 workloads * iodepth 256 each. (because why not) ● testing with ramdisk backend ● synthetic workloads don't approximate real world (example)
- 30. Latency opspersecond best service lowest cost per delivered resource 30
- 31. Latency opspersecond best service lowest cost per delivered resource only pain 31
- 32. Latency opspersecond best service lowest cost per delivered resource only pain 32 benchmarks
- 33. example1: 90 TB NVMe system - 22 IOPS per GB capacity example2: 116 TB NVMe system - 48 IOPS per GB capacity
- 34. Real load
- 35. ?
- 36. ?
- 37. Discussion