SlideShare a Scribd company logo
Life at 700us
Nick Fisk
Who Am I
• Nick Fisk
• Ceph user since 2012
• Author of Mastering Ceph
• Technical manager at SysGroup
• Managed Service Provider
• Use Ceph for providing tier-2 services to customers (Backups, standby
replicas) - Veeam
• Ceph RBD to ESXi via NFS
What is Latency?
• What the user feels when he clicks the button
• Buffer IO probably not affected though
• Traditional 10G iSCSI storage array will service a 4KB IO in around 300us.
• Local SAS SSD ~20us
• NVME ~2us
• Software defined storage will always have higher latency due to replication
across nodes and a fatter software stack.
• Latency heavily affects single threaded operations that can’t run in parallel.
• Eg. SQL transaction logs
• Or in the case of Ceph PG Contention
PG Contention
• PG serialises distributed workload in Ceph
• Each operation takes a lock on that PG, can lead to contention
• Multiple requests to a single Object will be hitting same PG
• Or if you are unlucky 2 hot objects may share the same PG
• Latency defines how fast a PG can process that operation, 2nd
operation has to wait
• If you dump slow ops from the OSD admin socket and see a lot of
delay “Waiting For PG”, you are likely hitting PG Contention
The theory behind minimising latency
• Ceph is software
• Each step of the Ceph “software” will run through faster with faster CPU’s (Ghz)
• Generally, CPU’s with more cores = lower Ghz
• High CPU Ghz = $$$ ?
• Try and avoid dual socket systems, adds latency and can introduce complications
on high disk count boxes (thread counts, thread pinning, interrupts)
• Every write has to go to journal, so make journal as fast as reasonably possible
• Bluestore – Only small IO’s
• Blessing or a curse?
• 10G networking is a must
• So…..less faster cores + NVME journal = Ceph Latency Nirvana
• Lets come up with a hardware design that takes this into account…
Bluestore – deferred writes
• For spinning disks
• IO<64K write to WAL, ACK, async commit later to disk
• IO>64K sync commit to disk
• This is great from a double write perspective, WAL doesn’t need to be
stupidly fast or have massive write endurance
• But a NVME will service a 128kb write a lot faster than a 7.2k disk
• May need to tune cutover for your use case
Ceph CPU Frequency Scaling
CPU
Mhz 4Kb Write IO
Min Latency
(us)
Avg Latency
(us)
1600 797 886 1250
2000 815 746 1222
2400 1161 630 857
2800 1227 549 812
3300 1320 482 755
4300 1548 437 644
• Ever wondered how Ceph performs at different clockspeeds?
• Using manual CPU governor on unlocked desktop CPU, ran fio QD=1 on a RBD
at different clock speeds
Networking Latency
• Sample ping test with 4KB payload over 1G and 10G networks
• 25Gb networking is interesting in potentially further reducing latency
• Even still, networking latency makes up a large part of the overall latency
due to Ceph replication between nodes.
• Client -> Primary OSD -> Replica OSD’s
• If using a NFS/iSCSI gateway/proxy, extra network hop added again
• RDMA will be the game changer!!
The Hardware
• 1U server
• Xeon E3 4x3.5Ghz (3.9Ghz Turbo)
• 10GB-T Onboard
• 8 SAS onboard
• 8 SATA onboard
• 64GB Ram
• 12x8TB He8’s (Not pictured)
• Intel P3700 400GB for Journal + OS
• 96TB node = ~£5k (Brexit!!)
• 160W Idle
• 180W average Ceph Load
• 220W Disks + CPU maxed out
How much CPU does Ceph require?
• Please don’t take this as a “HW
requirements” guide
• Use it to make informed
decisions, instead of 1 core per
OSD
• If latency is important, work out
total required Ghz and find CPU
with highest Ghz per core that
meets that total. Ie 3.5Ghz*4 =
14
0.1
1
10
100
1000
4KB 8KB 16KB 32KB 64KB 1MB 4MB
Mhz per Ceph IO
Mhz Per IO Mhz Per MB/s
Initial Results
• I was wrong!!!! - 4kb Average latency 2.4ms
write: io=115268KB, bw=1670.1KB/s, iops=417, runt= 68986msec
slat (usec): min=2, max=414, avg= 4.41, stdev= 3.81
clat (usec): min=966, max=27116, avg=2386.84, stdev=571.57
lat (usec): min=970, max=27120, avg=2391.25, stdev=571.69
clat percentiles (usec):
| 1.00th=[ 1480], 5.00th=[ 1688], 10.00th=[ 1912], 20.00th=[ 2128],
| 30.00th=[ 2192], 40.00th=[ 2288], 50.00th=[ 2352], 60.00th=[ 2448],
| 70.00th=[ 2576], 80.00th=[ 2704], 90.00th=[ 2832], 95.00th=[ 2960],
| 99.00th=[ 3312], 99.50th=[ 3536], 99.90th=[ 6112], 99.95th=[ 9536],
| 99.99th=[22400]
But Hang On, what’s this?
Real Current Frequency 900.47 MHz [100.11 x 8.99] (Max of below)
Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % Temp VCore
Core 1 [0]: 900.38 (8.99x) 10.4 44.2 3.47 49.7 27 0.7406
Core 2 [1]: 900.16 (8.99x) 8.46 66.7 1.18 29.9 27 0.7404
Core 3 [2]: 900.47 (8.99x) 10.5 73.8 1 22.5 27 0.7404
Core 4 [3]: 900.12 (8.99x) 8.03 58.6 1 38.3 27 0.7404
• Core’s are spending a lot of their time in C6 and below
• And only running at 900Mhz
Intel Cstate Wake Up Latency (us)
• POLL
• 0
• C1-SKL
• 2
• C1E-SKL
• 10
• C3-SKL
• 70
• C6-SKL
• 85
• C7s-SKL
• 124
• C8-SKL
• 200
From the previous slide, a large proportion of threads could be waiting for up to 200us for the CPU to wakeup to be serviced!!!
4kb Seq Write – Replica x3
• That’s more like it
write: io=105900KB, bw=5715.7KB/s, iops=1428, runt= 18528msec
slat (usec): min=2, max=106, avg= 3.50, stdev= 1.31
clat (usec): min=491, max=32099, avg=694.16, stdev=491.91
lat (usec): min=494, max=32102, avg=697.66, stdev=492.04
clat percentiles (usec):
| 1.00th=[ 540], 5.00th=[ 572], 10.00th=[ 588], 20.00th=[ 604],
| 30.00th=[ 620], 40.00th=[ 636], 50.00th=[ 652], 60.00th=[ 668],
| 70.00th=[ 692], 80.00th=[ 716], 90.00th=[ 764], 95.00th=[ 820],
| 99.00th=[ 1448], 99.50th=[ 2320], 99.90th=[ 7584], 99.95th=[11712],
| 99.99th=[24448]
Questions?
Ad

More Related Content

What's hot (20)

A crash course in CRUSH
A crash course in CRUSHA crash course in CRUSH
A crash course in CRUSH
Sage Weil
 
Seastore: Next Generation Backing Store for Ceph
Seastore: Next Generation Backing Store for CephSeastore: Next Generation Backing Store for Ceph
Seastore: Next Generation Backing Store for Ceph
ScyllaDB
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Odinot Stanislas
 
Disaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoFDisaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoF
ShapeBlue
 
Ceph issue 해결 사례
Ceph issue 해결 사례Ceph issue 해결 사례
Ceph issue 해결 사례
Open Source Consulting
 
Ceph and RocksDB
Ceph and RocksDBCeph and RocksDB
Ceph and RocksDB
Sage Weil
 
Proxmox Clustering with CEPH
Proxmox Clustering with CEPHProxmox Clustering with CEPH
Proxmox Clustering with CEPH
FahadIbrar5
 
2021.02 new in Ceph Pacific Dashboard
2021.02 new in Ceph Pacific Dashboard2021.02 new in Ceph Pacific Dashboard
2021.02 new in Ceph Pacific Dashboard
Ceph Community
 
Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM
ShapeBlue
 
BlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for CephBlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for Ceph
Sage Weil
 
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
Ceph Object Storage Performance Secrets and Ceph Data Lake SolutionCeph Object Storage Performance Secrets and Ceph Data Lake Solution
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
Karan Singh
 
Understanding DPDK
Understanding DPDKUnderstanding DPDK
Understanding DPDK
Denys Haryachyy
 
Boosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uringBoosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uring
ShapeBlue
 
Ceph RBD Update - June 2021
Ceph RBD Update - June 2021Ceph RBD Update - June 2021
Ceph RBD Update - June 2021
Ceph Community
 
Intel dpdk Tutorial
Intel dpdk TutorialIntel dpdk Tutorial
Intel dpdk Tutorial
Saifuddin Kaijar
 
Ceph Block Devices: A Deep Dive
Ceph Block Devices:  A Deep DiveCeph Block Devices:  A Deep Dive
Ceph Block Devices: A Deep Dive
Red_Hat_Storage
 
Fast Userspace OVS with AF_XDP, OVS CONF 2018
Fast Userspace OVS with AF_XDP, OVS CONF 2018Fast Userspace OVS with AF_XDP, OVS CONF 2018
Fast Userspace OVS with AF_XDP, OVS CONF 2018
Cheng-Chun William Tu
 
resource on openstack
 resource on openstack resource on openstack
resource on openstack
jieun kim
 
Revisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS SchedulerRevisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS Scheduler
Yongseok Oh
 
MinIO January 2020 Briefing
MinIO January 2020 BriefingMinIO January 2020 Briefing
MinIO January 2020 Briefing
Jonathan Symonds
 
A crash course in CRUSH
A crash course in CRUSHA crash course in CRUSH
A crash course in CRUSH
Sage Weil
 
Seastore: Next Generation Backing Store for Ceph
Seastore: Next Generation Backing Store for CephSeastore: Next Generation Backing Store for Ceph
Seastore: Next Generation Backing Store for Ceph
ScyllaDB
 
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Ceph: Open Source Storage Software Optimizations on Intel® Architecture for C...
Odinot Stanislas
 
Disaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoFDisaggregating Ceph using NVMeoF
Disaggregating Ceph using NVMeoF
ShapeBlue
 
Ceph and RocksDB
Ceph and RocksDBCeph and RocksDB
Ceph and RocksDB
Sage Weil
 
Proxmox Clustering with CEPH
Proxmox Clustering with CEPHProxmox Clustering with CEPH
Proxmox Clustering with CEPH
FahadIbrar5
 
2021.02 new in Ceph Pacific Dashboard
2021.02 new in Ceph Pacific Dashboard2021.02 new in Ceph Pacific Dashboard
2021.02 new in Ceph Pacific Dashboard
Ceph Community
 
Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM Achieving the ultimate performance with KVM
Achieving the ultimate performance with KVM
ShapeBlue
 
BlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for CephBlueStore: a new, faster storage backend for Ceph
BlueStore: a new, faster storage backend for Ceph
Sage Weil
 
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
Ceph Object Storage Performance Secrets and Ceph Data Lake SolutionCeph Object Storage Performance Secrets and Ceph Data Lake Solution
Ceph Object Storage Performance Secrets and Ceph Data Lake Solution
Karan Singh
 
Boosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uringBoosting I/O Performance with KVM io_uring
Boosting I/O Performance with KVM io_uring
ShapeBlue
 
Ceph RBD Update - June 2021
Ceph RBD Update - June 2021Ceph RBD Update - June 2021
Ceph RBD Update - June 2021
Ceph Community
 
Ceph Block Devices: A Deep Dive
Ceph Block Devices:  A Deep DiveCeph Block Devices:  A Deep Dive
Ceph Block Devices: A Deep Dive
Red_Hat_Storage
 
Fast Userspace OVS with AF_XDP, OVS CONF 2018
Fast Userspace OVS with AF_XDP, OVS CONF 2018Fast Userspace OVS with AF_XDP, OVS CONF 2018
Fast Userspace OVS with AF_XDP, OVS CONF 2018
Cheng-Chun William Tu
 
resource on openstack
 resource on openstack resource on openstack
resource on openstack
jieun kim
 
Revisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS SchedulerRevisiting CephFS MDS and mClock QoS Scheduler
Revisiting CephFS MDS and mClock QoS Scheduler
Yongseok Oh
 
MinIO January 2020 Briefing
MinIO January 2020 BriefingMinIO January 2020 Briefing
MinIO January 2020 Briefing
Jonathan Symonds
 

Similar to Nick Fisk - low latency Ceph (20)

ceph-barcelona-v-1.2
ceph-barcelona-v-1.2ceph-barcelona-v-1.2
ceph-barcelona-v-1.2
Ranga Swami Reddy Muthumula
 
Ceph barcelona-v-1.2
Ceph barcelona-v-1.2Ceph barcelona-v-1.2
Ceph barcelona-v-1.2
Ranga Swami Reddy Muthumula
 
Your 1st Ceph cluster
Your 1st Ceph clusterYour 1st Ceph cluster
Your 1st Ceph cluster
Mirantis
 
Oracle Performance On Linux X86 systems
Oracle  Performance On Linux  X86 systems Oracle  Performance On Linux  X86 systems
Oracle Performance On Linux X86 systems
Baruch Osoveskiy
 
Stabilizing Ceph
Stabilizing CephStabilizing Ceph
Stabilizing Ceph
Ceph Community
 
System Capa Planning_DBA oracle edu
System Capa Planning_DBA oracle eduSystem Capa Planning_DBA oracle edu
System Capa Planning_DBA oracle edu
엑셈
 
V mware2012 20121221_final
V mware2012 20121221_finalV mware2012 20121221_final
V mware2012 20121221_final
Web2Present
 
Couchbase live 2016
Couchbase live 2016Couchbase live 2016
Couchbase live 2016
Pierre Mavro
 
OSS Presentation VMWorld 2011 by Andy Bennett & Craig Morgan
OSS Presentation VMWorld 2011 by Andy Bennett & Craig MorganOSS Presentation VMWorld 2011 by Andy Bennett & Craig Morgan
OSS Presentation VMWorld 2011 by Andy Bennett & Craig Morgan
OpenStorageSummit
 
Memory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and VirtualizationMemory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and Virtualization
Bigstep
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
HungWei Chiu
 
Aerospike & GCE (LSPE Talk)
Aerospike & GCE (LSPE Talk)Aerospike & GCE (LSPE Talk)
Aerospike & GCE (LSPE Talk)
Sayyaparaju Sunil
 
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...
Ceph Community
 
Azure VM 101 - HomeGen by CloudGen Verona - Marco Obinu
Azure VM 101 - HomeGen by CloudGen Verona - Marco ObinuAzure VM 101 - HomeGen by CloudGen Verona - Marco Obinu
Azure VM 101 - HomeGen by CloudGen Verona - Marco Obinu
Marco Obinu
 
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...
ervogler
 
Super scaling singleton inserts
Super scaling singleton insertsSuper scaling singleton inserts
Super scaling singleton inserts
Chris Adkin
 
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on CephBuild an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Rongze Zhu
 
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack CloudJourney to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Ceph Community
 
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack CloudJourney to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Patrick McGarry
 
Accelerating HBase with NVMe and Bucket Cache
Accelerating HBase with NVMe and Bucket CacheAccelerating HBase with NVMe and Bucket Cache
Accelerating HBase with NVMe and Bucket Cache
Nicolas Poggi
 
Your 1st Ceph cluster
Your 1st Ceph clusterYour 1st Ceph cluster
Your 1st Ceph cluster
Mirantis
 
Oracle Performance On Linux X86 systems
Oracle  Performance On Linux  X86 systems Oracle  Performance On Linux  X86 systems
Oracle Performance On Linux X86 systems
Baruch Osoveskiy
 
System Capa Planning_DBA oracle edu
System Capa Planning_DBA oracle eduSystem Capa Planning_DBA oracle edu
System Capa Planning_DBA oracle edu
엑셈
 
V mware2012 20121221_final
V mware2012 20121221_finalV mware2012 20121221_final
V mware2012 20121221_final
Web2Present
 
Couchbase live 2016
Couchbase live 2016Couchbase live 2016
Couchbase live 2016
Pierre Mavro
 
OSS Presentation VMWorld 2011 by Andy Bennett & Craig Morgan
OSS Presentation VMWorld 2011 by Andy Bennett & Craig MorganOSS Presentation VMWorld 2011 by Andy Bennett & Craig Morgan
OSS Presentation VMWorld 2011 by Andy Bennett & Craig Morgan
OpenStorageSummit
 
Memory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and VirtualizationMemory, Big Data, NoSQL and Virtualization
Memory, Big Data, NoSQL and Virtualization
Bigstep
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
HungWei Chiu
 
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...
CEPH DAY BERLIN - 5 REASONS TO USE ARM-BASED MICRO-SERVER ARCHITECTURE FOR CE...
Ceph Community
 
Azure VM 101 - HomeGen by CloudGen Verona - Marco Obinu
Azure VM 101 - HomeGen by CloudGen Verona - Marco ObinuAzure VM 101 - HomeGen by CloudGen Verona - Marco Obinu
Azure VM 101 - HomeGen by CloudGen Verona - Marco Obinu
Marco Obinu
 
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...
Big Data Hadoop Briefing Hosted by Cisco, WWT and MapR: Cisco UCS For Big Dat...
ervogler
 
Super scaling singleton inserts
Super scaling singleton insertsSuper scaling singleton inserts
Super scaling singleton inserts
Chris Adkin
 
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on CephBuild an High-Performance and High-Durable Block Storage Service Based on Ceph
Build an High-Performance and High-Durable Block Storage Service Based on Ceph
Rongze Zhu
 
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack CloudJourney to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Ceph Community
 
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack CloudJourney to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Journey to Stability: Petabyte Ceph Cluster in OpenStack Cloud
Patrick McGarry
 
Accelerating HBase with NVMe and Bucket Cache
Accelerating HBase with NVMe and Bucket CacheAccelerating HBase with NVMe and Bucket Cache
Accelerating HBase with NVMe and Bucket Cache
Nicolas Poggi
 
Ad

More from ShapeBlue (20)

Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
ShapeBlue
 
VM-HA with CloudStack and Linstor, Rene Peinthor
VM-HA with CloudStack and Linstor, Rene PeinthorVM-HA with CloudStack and Linstor, Rene Peinthor
VM-HA with CloudStack and Linstor, Rene Peinthor
ShapeBlue
 
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIOHow We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
ShapeBlue
 
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.OnlineInternet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
ShapeBlue
 
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
ShapeBlue
 
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlueWhat’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
ShapeBlue
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
ShapeBlue
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
ShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
ShapeBlue
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
ShapeBlue
 
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHubHow We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
ShapeBlue
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
ShapeBlue
 
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
ShapeBlue
 
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIOHow We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
ShapeBlue
 
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
ShapeBlue
 
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
ShapeBlue
 
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue
 
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
ShapeBlue
 
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
ShapeBlue
 
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
ShapeBlue
 
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
Development of an Оbject Storage Plugin for CloudStack, Christian Reichert, s...
ShapeBlue
 
VM-HA with CloudStack and Linstor, Rene Peinthor
VM-HA with CloudStack and Linstor, Rene PeinthorVM-HA with CloudStack and Linstor, Rene Peinthor
VM-HA with CloudStack and Linstor, Rene Peinthor
ShapeBlue
 
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIOHow We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
How We Use CloudStack to Provide Managed Hosting, Swen Brüseke, proIO
ShapeBlue
 
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.OnlineInternet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
Internet Facing VMs and the DDoS Problem, Wido den Hollander, Your.Online
ShapeBlue
 
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
Transitioning from VMware to Apache CloudStack: A Path to Profitability and C...
ShapeBlue
 
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlueWhat’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
What’s New and What’s Upcoming in Apache CloudStack, Giles Sirett, ShapeBlue
ShapeBlue
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
ShapeBlue
 
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlueCloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
CloudStack Tooling Ecosystem – Kiran Chavala, ShapeBlue
ShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
ShapeBlue
 
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlueVM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
VM Migration from VMware to CloudStack and KVM – Suresh Anaparti, ShapeBlue
ShapeBlue
 
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHubHow We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
How We Grew Up with CloudStack and its Journey – Dilip Singh, DataHub
ShapeBlue
 
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
What’s New in CloudStack 4.19, Abhishek Kumar, Release Manager Apache CloudSt...
ShapeBlue
 
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
CloudStack 101: The Best Way to Build Your Private Cloud – Rohit Yadav, VP Ap...
ShapeBlue
 
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIOHow We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
How We Use CloudStack to Provide Managed Hosting - Swen Brüseke - proIO
ShapeBlue
 
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
Enabling DPU Hardware Accelerators in XCP-ng Cloud Platform Environment - And...
ShapeBlue
 
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
ShapeBlue
 
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue
 
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
ShapeBlue
 
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
Use Existing Assets to Build a Powerful In-house Cloud Solution - Magali Perv...
ShapeBlue
 
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
Import Export Virtual Machine for KVM Hypervisor - Ayush Pandey - University ...
ShapeBlue
 
Ad

Recently uploaded (20)

Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
Asthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdfAsthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdf
VanessaRaudez
 
Leading AI Innovation As A Product Manager - Michael Jidael
Leading AI Innovation As A Product Manager - Michael JidaelLeading AI Innovation As A Product Manager - Michael Jidael
Leading AI Innovation As A Product Manager - Michael Jidael
Michael Jidael
 
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
Lynda Kane
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfLinux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdf
RHCSA Guru
 
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your UsersAutomation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Lynda Kane
 
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...
Vishnu Singh Chundawat
 
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath MaestroDev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
UiPathCommunity
 
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
Alan Dix
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5..."Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
Fwdays
 
Network Security. Different aspects of Network Security.
Network Security. Different aspects of Network Security.Network Security. Different aspects of Network Security.
Network Security. Different aspects of Network Security.
gregtap1
 
Buckeye Dreamin' 2023: De-fogging Debug Logs
Buckeye Dreamin' 2023: De-fogging Debug LogsBuckeye Dreamin' 2023: De-fogging Debug Logs
Buckeye Dreamin' 2023: De-fogging Debug Logs
Lynda Kane
 
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
SOFTTECHHUB
 
Technology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data AnalyticsTechnology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data Analytics
InData Labs
 
Learn the Basics of Agile Development: Your Step-by-Step Guide
Learn the Basics of Agile Development: Your Step-by-Step GuideLearn the Basics of Agile Development: Your Step-by-Step Guide
Learn the Basics of Agile Development: Your Step-by-Step Guide
Marcel David
 
Image processinglab image processing image processing
Image processinglab image processing  image processingImage processinglab image processing  image processing
Image processinglab image processing image processing
RaghadHany
 
Role of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered ManufacturingRole of Data Annotation Services in AI-Powered Manufacturing
Role of Data Annotation Services in AI-Powered Manufacturing
Andrew Leo
 
Cyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of securityCyber Awareness overview for 2025 month of security
Cyber Awareness overview for 2025 month of security
riccardosl1
 
Asthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdfAsthma presentación en inglés abril 2025 pdf
Asthma presentación en inglés abril 2025 pdf
VanessaRaudez
 
Leading AI Innovation As A Product Manager - Michael Jidael
Leading AI Innovation As A Product Manager - Michael JidaelLeading AI Innovation As A Product Manager - Michael Jidael
Leading AI Innovation As A Product Manager - Michael Jidael
Michael Jidael
 
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
#AdminHour presents: Hour of Code2018 slide deck from 12/6/2018
Lynda Kane
 
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdfSAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
SAP Modernization: Maximizing the Value of Your SAP S/4HANA Migration.pdf
Precisely
 
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptxSpecial Meetup Edition - TDX Bengaluru Meetup #52.pptx
Special Meetup Edition - TDX Bengaluru Meetup #52.pptx
shyamraj55
 
Linux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdfLinux Professional Institute LPIC-1 Exam.pdf
Linux Professional Institute LPIC-1 Exam.pdf
RHCSA Guru
 
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your UsersAutomation Dreamin' 2022: Sharing Some Gratitude with Your Users
Automation Dreamin' 2022: Sharing Some Gratitude with Your Users
Lynda Kane
 
What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...What is Model Context Protocol(MCP) - The new technology for communication bw...
What is Model Context Protocol(MCP) - The new technology for communication bw...
Vishnu Singh Chundawat
 
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath MaestroDev Dives: Automate and orchestrate your processes with UiPath Maestro
Dev Dives: Automate and orchestrate your processes with UiPath Maestro
UiPathCommunity
 
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
AI Changes Everything – Talk at Cardiff Metropolitan University, 29th April 2...
Alan Dix
 
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager APIUiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPath Community Berlin: Orchestrator API, Swagger, and Test Manager API
UiPathCommunity
 
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5..."Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
"Client Partnership — the Path to Exponential Growth for Companies Sized 50-5...
Fwdays
 
Network Security. Different aspects of Network Security.
Network Security. Different aspects of Network Security.Network Security. Different aspects of Network Security.
Network Security. Different aspects of Network Security.
gregtap1
 
Buckeye Dreamin' 2023: De-fogging Debug Logs
Buckeye Dreamin' 2023: De-fogging Debug LogsBuckeye Dreamin' 2023: De-fogging Debug Logs
Buckeye Dreamin' 2023: De-fogging Debug Logs
Lynda Kane
 
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
AI EngineHost Review: Revolutionary USA Datacenter-Based Hosting with NVIDIA ...
SOFTTECHHUB
 
Technology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data AnalyticsTechnology Trends in 2025: AI and Big Data Analytics
Technology Trends in 2025: AI and Big Data Analytics
InData Labs
 
Learn the Basics of Agile Development: Your Step-by-Step Guide
Learn the Basics of Agile Development: Your Step-by-Step GuideLearn the Basics of Agile Development: Your Step-by-Step Guide
Learn the Basics of Agile Development: Your Step-by-Step Guide
Marcel David
 
Image processinglab image processing image processing
Image processinglab image processing  image processingImage processinglab image processing  image processing
Image processinglab image processing image processing
RaghadHany
 

Nick Fisk - low latency Ceph

  • 2. Who Am I • Nick Fisk • Ceph user since 2012 • Author of Mastering Ceph • Technical manager at SysGroup • Managed Service Provider • Use Ceph for providing tier-2 services to customers (Backups, standby replicas) - Veeam • Ceph RBD to ESXi via NFS
  • 3. What is Latency? • What the user feels when he clicks the button • Buffer IO probably not affected though • Traditional 10G iSCSI storage array will service a 4KB IO in around 300us. • Local SAS SSD ~20us • NVME ~2us • Software defined storage will always have higher latency due to replication across nodes and a fatter software stack. • Latency heavily affects single threaded operations that can’t run in parallel. • Eg. SQL transaction logs • Or in the case of Ceph PG Contention
  • 4. PG Contention • PG serialises distributed workload in Ceph • Each operation takes a lock on that PG, can lead to contention • Multiple requests to a single Object will be hitting same PG • Or if you are unlucky 2 hot objects may share the same PG • Latency defines how fast a PG can process that operation, 2nd operation has to wait • If you dump slow ops from the OSD admin socket and see a lot of delay “Waiting For PG”, you are likely hitting PG Contention
  • 5. The theory behind minimising latency • Ceph is software • Each step of the Ceph “software” will run through faster with faster CPU’s (Ghz) • Generally, CPU’s with more cores = lower Ghz • High CPU Ghz = $$$ ? • Try and avoid dual socket systems, adds latency and can introduce complications on high disk count boxes (thread counts, thread pinning, interrupts) • Every write has to go to journal, so make journal as fast as reasonably possible • Bluestore – Only small IO’s • Blessing or a curse? • 10G networking is a must • So…..less faster cores + NVME journal = Ceph Latency Nirvana • Lets come up with a hardware design that takes this into account…
  • 6. Bluestore – deferred writes • For spinning disks • IO<64K write to WAL, ACK, async commit later to disk • IO>64K sync commit to disk • This is great from a double write perspective, WAL doesn’t need to be stupidly fast or have massive write endurance • But a NVME will service a 128kb write a lot faster than a 7.2k disk • May need to tune cutover for your use case
  • 7. Ceph CPU Frequency Scaling CPU Mhz 4Kb Write IO Min Latency (us) Avg Latency (us) 1600 797 886 1250 2000 815 746 1222 2400 1161 630 857 2800 1227 549 812 3300 1320 482 755 4300 1548 437 644 • Ever wondered how Ceph performs at different clockspeeds? • Using manual CPU governor on unlocked desktop CPU, ran fio QD=1 on a RBD at different clock speeds
  • 8. Networking Latency • Sample ping test with 4KB payload over 1G and 10G networks • 25Gb networking is interesting in potentially further reducing latency • Even still, networking latency makes up a large part of the overall latency due to Ceph replication between nodes. • Client -> Primary OSD -> Replica OSD’s • If using a NFS/iSCSI gateway/proxy, extra network hop added again • RDMA will be the game changer!!
  • 9. The Hardware • 1U server • Xeon E3 4x3.5Ghz (3.9Ghz Turbo) • 10GB-T Onboard • 8 SAS onboard • 8 SATA onboard • 64GB Ram • 12x8TB He8’s (Not pictured) • Intel P3700 400GB for Journal + OS • 96TB node = ~£5k (Brexit!!) • 160W Idle • 180W average Ceph Load • 220W Disks + CPU maxed out
  • 10. How much CPU does Ceph require? • Please don’t take this as a “HW requirements” guide • Use it to make informed decisions, instead of 1 core per OSD • If latency is important, work out total required Ghz and find CPU with highest Ghz per core that meets that total. Ie 3.5Ghz*4 = 14 0.1 1 10 100 1000 4KB 8KB 16KB 32KB 64KB 1MB 4MB Mhz per Ceph IO Mhz Per IO Mhz Per MB/s
  • 11. Initial Results • I was wrong!!!! - 4kb Average latency 2.4ms write: io=115268KB, bw=1670.1KB/s, iops=417, runt= 68986msec slat (usec): min=2, max=414, avg= 4.41, stdev= 3.81 clat (usec): min=966, max=27116, avg=2386.84, stdev=571.57 lat (usec): min=970, max=27120, avg=2391.25, stdev=571.69 clat percentiles (usec): | 1.00th=[ 1480], 5.00th=[ 1688], 10.00th=[ 1912], 20.00th=[ 2128], | 30.00th=[ 2192], 40.00th=[ 2288], 50.00th=[ 2352], 60.00th=[ 2448], | 70.00th=[ 2576], 80.00th=[ 2704], 90.00th=[ 2832], 95.00th=[ 2960], | 99.00th=[ 3312], 99.50th=[ 3536], 99.90th=[ 6112], 99.95th=[ 9536], | 99.99th=[22400]
  • 12. But Hang On, what’s this? Real Current Frequency 900.47 MHz [100.11 x 8.99] (Max of below) Core [core-id] :Actual Freq (Mult.) C0% Halt(C1)% C3 % C6 % Temp VCore Core 1 [0]: 900.38 (8.99x) 10.4 44.2 3.47 49.7 27 0.7406 Core 2 [1]: 900.16 (8.99x) 8.46 66.7 1.18 29.9 27 0.7404 Core 3 [2]: 900.47 (8.99x) 10.5 73.8 1 22.5 27 0.7404 Core 4 [3]: 900.12 (8.99x) 8.03 58.6 1 38.3 27 0.7404 • Core’s are spending a lot of their time in C6 and below • And only running at 900Mhz
  • 13. Intel Cstate Wake Up Latency (us) • POLL • 0 • C1-SKL • 2 • C1E-SKL • 10 • C3-SKL • 70 • C6-SKL • 85 • C7s-SKL • 124 • C8-SKL • 200 From the previous slide, a large proportion of threads could be waiting for up to 200us for the CPU to wakeup to be serviced!!!
  • 14. 4kb Seq Write – Replica x3 • That’s more like it write: io=105900KB, bw=5715.7KB/s, iops=1428, runt= 18528msec slat (usec): min=2, max=106, avg= 3.50, stdev= 1.31 clat (usec): min=491, max=32099, avg=694.16, stdev=491.91 lat (usec): min=494, max=32102, avg=697.66, stdev=492.04 clat percentiles (usec): | 1.00th=[ 540], 5.00th=[ 572], 10.00th=[ 588], 20.00th=[ 604], | 30.00th=[ 620], 40.00th=[ 636], 50.00th=[ 652], 60.00th=[ 668], | 70.00th=[ 692], 80.00th=[ 716], 90.00th=[ 764], 95.00th=[ 820], | 99.00th=[ 1448], 99.50th=[ 2320], 99.90th=[ 7584], 99.95th=[11712], | 99.99th=[24448]