Disaggregated Container Attached Storage - Yet Another Topology with What Purpose?
0 likes33 views
The storage topology in vogue seems to cycle every few years. Internal storage is followed by centralized Storage Area Networks only to be superseded by one-size-fits-all Hyperconverged models - until scalability constraints led to distributed storage. Then comes NVMe, offering blistering speeds that all of these storage stacks struggle with. Kubernetes inspires Container Attached Storage aspiring to be the perfect model, so why is disaggregated storage now making an appearance? This talk considers the motivations behind yet another storage topology and examines a modern, flexible architecture for delivering high-performance storage under Kubernetes. This talk was given by Nick Connolly for DoK Day Europe @ KubeCon 2022.
Ad
More Related Content
Similar to Disaggregated Container Attached Storage - Yet Another Topology with What Purpose? (20)
More from DoKC (20)
Ad
Recently uploaded (20)
Ad
Disaggregated Container Attached Storage - Yet Another Topology with What Purpose?
- 1. Disaggregated Container Attached Storage - Yet Another Technology with What Purpose? Nick Connolly, Chief Scientist, DataCore Software
- 2. Agenda • A brief history of storage! • Disaggregated Storage • Container Native Storage • OpenEBS • Conclusion
- 3. Local Storage • Disks inside servers • Connectivity • SCSI / ATA / SAS • Magnetic media performance • Throughput: ~5-200MB/sec • Latency: ~10msec • Data Services • Operating system • RAID controller
- 4. RAID Controller • Offloads host system • Aggregates local storage • Striping for performance - RAID 0 • Mirroring for protection - RAID 1 • Parity for cost reduction - RAID 5 / RAID 6 • Presents logical disks
- 5. JBOD • Server disk slots are limited • External expansion chassis • Just a Bunch Of Disks • No RAID controller • Can be Dual Ported • Redundancy - path or controller failure • Connecting multiple hosts • Storage Area Network
- 6. Storage Area Network (SAN) • Fast networks enable sharing • FibreChannel / iSCSI • Low latency and high throughput relative to storage • Minimal impact on storage traffic • Enables centralised pooling of storage • Easier management • Improves capacity utilisation • Centralised data services • Storage Arrays
- 7. Storage Area Network (SAN) Host OS App App App Controller Host OS App App App Host OS App App App FC / iSCSI Array
- 8. Storage Array • Designed to be highly available and reliable • Redundant paths and controllers • Data protection with RAID • Independently managed • Has its own control plane • Data services, e.g. • Caching • Thin provisioning • Hierarchical Storage Management • Deduplication, compression and encryption • Snapshots, clones
- 9. Software Defined Storage Host OS App App App Software Defined Storage Host OS App App App Host OS App App App FC / iSCSI
- 10. Software Defined Storage • Storage is now a software problem! • It’s just another application • Variety of platforms • Physical • Virtual • Containerised
- 11. Hyperconverged • Combines • Software Defined Storage • Virtualisation • Software Defined Networking • Management plane • Often identical nodes • Storage is local for easy scalability • One-size-fits-all limits flexibility
- 12. NVMe • Paradigm shift • Faster protocol designed for high speed flash • Simpler storage commands • Memory based command and response queues • Multiple queues for lock free access • Minimises contention for high core counts • Optane performance • Throughput: ~3-7GB/sec • Latency: ~10usec • Network latency is significant • Local storage - back to silos of capacity
- 13. NVMe over Fabrics • NVMe-oF extends NVMe across the network • No more than 10usec additional latency • Supports several transports • FibreChannel • RDMA • Infinband • TCP • Access remote NVMe disk as if local • Storage arrays starting to support NVMe-oF
- 14. Disaggregated Storage • New model emerging due to NVMe-oF • Dynamic mapping between hosts and storage • Allows per-workload design and scalability • No I/O blender effect • Capacity utilisation and manageability • Data services • Either implemented at the host, or … • Requires full virtualisation of the storage with a high performance storage stack
- 15. Disaggregated Storage NVMe-oF Host OS App App App Host OS App App App Host OS App App App Remote Storage via NVMe-oF
- 16. IPU / DPU • Offload disaggregated storage stack • PCIe board • Similar to RAID controller • Array of Arm cores • On-board RAM • High speed networking • Presents local PCIe NVMe disk • Data services
- 17. Storage Performance Development Kit (SPDK) • Well regarded storage stack • Can deliver full potential of Optane • Tools and libraries for writing: • High performance, scalable • User-mode storage applications • Cutting Edge • Leverage the latest NVMe features • Poll-mode and event-loop for performance • Lockless, thread-per-core design
- 18. Kubernetes
- 19. Container Native Storage • Software Defined Storage in Kubernetes • Designed for Kubernetes • Managed by Kubernetes • Disaggregated NVMe storage • Performance • Scalability • Replicas are essential • Persistent data on ephemeral infrastructure
- 20. OpenEBS • Leading open source Container Native Storage • CNCF Sandbox with vibrant community • https://github.com/openebs/openebs • https://openebs.io • Mayastor 1.0 • Production ready • NVMe based
- 21. OpenEBS Mayastor Engine • High performance NVMe / NVMe-oF TCP • Control plane designed for Kubernetes • N-way synchronous replicas (disaggregated) • Controller per volume • Supports heterogeneous storage • Basic pooling – multiple volumes per disk • Designed for running at scale • Portable – no kernel mode components • Implemented in Rust for memory safety
- 22. OpenEBS Mayastor Engine NVMe-oF TCP Mayastor App App App Mayastor App App App Mayastor App App App PV Remote Replica Target via NVMe-oF TCP Node 1 Node 3 Node 2
- 23. Conclusion • Disaggregated storage with NVMe-oF as part of a Container Native Storage solution offers significant scalability and performance advantages for Kubernetes • The OpenEBS Mayastor engine provides a good way to get started with the technology