Learning from ZFS to Scale Storage on and under Containers
1 like1,398 views
Evan Powell presented this deck at the MSST 2107 Mass Storage Conference. "What is so new about the container environment that a new class of storage software is emerging to address these use cases? And can container orchestration systems themselves be part of the solution? As is often the case in storage, metadata matters here. We are implementing in the open source OpenEBS.io some approaches that are in some regards inspired by ZFS to enable much more efficient scale out block storage for containers that itself is containerized. The goal is to enable storage to be treated in many regards as just another application while, of course, also providing storage services to stateful applications in the environment." Watch the video: http://wp.me/p3RLHQ-gPs Learn more: blog.openebs.io and http://storageconference.us Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
![[OpenStack Days Korea 2016] Track1 - All flash CEPH 구성 및 최적화](https://cdn.slidesharecdn.com/ss_thumbnails/12skt-160226171513-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (20)
Similar to Learning from ZFS to Scale Storage on and under Containers (20)
Ad
More from inside-BigData.com (20)
Ad
Recently uploaded (20)
Learning from ZFS to Scale Storage on and under Containers
- 1. Let’s decompose storage (again) Why? How? Huh? MSST May 17 2017 Evan Powell
- 5. CERTIFIED SYSADMIN(S) Layering Storage Medium (Disk, Flash, RAM) XFS, ZFS, LVM, EXT4 POSIX (IO, Checksum, Snapshots) NFS,iSCSI Enterprise Storage (SAN, NAS, ScaleOut, DP, DR, Backup, Compression, Dedup) (3-TIER APPs) Format Disks and Use. Manage Few Disks Supervised Provisioning Manage Storage Upgrades! Manage 100s of Volumes Managed Upgrades GEEK(LOCAL APPs)
- 6. CERTIFIED SYSADMIN(S) Layering Storage Medium (Disk, Flash, RAM) XFS, ZFS, LVM, EXT4 POSIX (IO, Checksum, Snapshots) NFS,iSCSI Enterprise Storage (SAN, NAS, ScaleOut, DP, DR, Backup, Compression, Dedup) OpenStack Cinder, VASA Cloud Storage - SDS (Public, Private, Hybrid) (3-TIER APPs) (CLOUD SERVICE) OpenEBS Containerized Storage (Hyper-Converged, Auto-Scaling, Auto-Upgradable, Yaml Driven) (MICRO APPS) Format Disks and Use. Manage Few Disks Supervised Provisioning Manage Storage Upgrades! Manage 100s of Volumes Managed Upgrades Auto-Provisioned Delegated Administration Manage Shared Infra Manage 1000s of Volumes Scheduled Upgrades Auto-magically Provisioned Auto-managed Storage(ML) Data Mobility Manage Millions of Volumes Seamless Upgrades GEEK DEVOPS AT SPEED OpenSource Technology Stack (LOCAL APPs) PAAS ADMINs
- 8. Painful persistence Manifests express intent K8S used rarely for apps requiring persistence because they require brittle tight coupling. Container 1 Container 2 Container 2 Hard wired connections via plug-in stateless stateful NAS SAN S3 NAS SAN S3
- 9. Manifests express intent stateless stateful No changes to DevOps workflow even for containers requiring persistence. Users manifest their intent and the storage and storage controllers adjust automatically as needed. OpenEBS Policy engine Remote Storage Local Storage OpenEBS Storage Hosts OpenEBS VSMs OpenEBS Storage Cluster EBSAPIs Containers and underlying storage, local on host or dedicated storage pods OR remote S3 or EBS storage all grouped into a storage cloud that just works. Manifests express intent Desired state of state
- 10. Architecture and Design - Powered by Linux, Go and OpenSource - Built and Delivered as Containers / Micro-services - Longhorn, Gotgt, Kubernetes, Consul
- 11. Fault tolerant and secure by default Low entry barrier, easy to setup Storage optimized for Containerized Applications Horizontally scalable to millions of Containers Seamless integration into existing private and public cloud environments Non-disruptive upgrades Design Goals and Constraints Developer and Operators Friendly Completely OpenSource (Apache license) Microservices based DevOps architecture
- 12. Remote Storage Local Storage OpenEBS Storage Hosts Data Storage Driver HTTPS (manage) Overview & Terminology K8s master K8s minions OpenEBS Maya master OpenEBS VSMs / Storage Pod Pod Network (Flannel*) Network (Flannel)
- 13. Network (Flannel) TCP OpenEBS Maya-K8s Adaptors Deployment - Hyper-Converged K8s master K8s minions Storage Pods(3)Pod TCMU OpenEBS Maya Storage Orchestrator
- 14. VSM - Storage in Containers Local Disks NAS or SAN Cloud Storage NVMe Flash Maya Storage Orchestrator Backend Containers to Persist Data (Cached, Protected) Frontend ContainersVSM / Storage Pod OpenEBS Storage Hosts Storage Data (iSCSI/TCMU) Container (Docker) Inline Replication Multiple Storage Backends
- 15. Jiva - Containerized Storage Image OpenEBS Container runtime (Maya) Compute Network Storage fapi-server qorum-net fe-iscsi fe-tcmu stats-db IO Processor meta-cache IO Store Replication/Multiplex Backend Store snap r/w snap-s3 snap-local NVF RWCTTA Compression/Encryption syncer snap control bapi-server monitor snap-qcow Hot Data Deduplication key-vault Front-end Container Backend Container Container Image https://hub.docker.com/r/openebs/jiva/ key-vault QoS QoS TTA Pull/Push
- 16. Maya - Container Storage Orchestration maya mSCH mDB mAgent mAPI mCluster mTelmetry mAIEngine maya mConnect mCluster mStorageInterface Integrations OpenEBS Maya Master OpenEBS Storage Host mTerrafor m mAWS-MPmDriver mJRunner mGUI mAnalytics
- 17. Storage Internals - Capacity Management - QoS - Access - iSCSI, TCMU - Snapshot / Restore (S3) - Backup / Migration - Caching/Tiering - Replication / Rebuild
- 18. OpenEBS - Core differentiations ● The block storage software is made into a micro service ● The ‘micro service’ has its own block protocol stack, tiering engine, QoS engine and ML prediction capability ● The block storage knowledge is maintained on a per-volume basis. The data of each volume is divided into cold-data and hot-data. Cold-data resides on NVMe-Flash or on 3DX-Memory. Hot-data resides in slower disks / SAN/ Cloud-Storage/S3 ● The metadata knowledge also is maintained at a volume level (not the entire storage). This saves us from the issue of huge-metadata-sifting at scale. The traversal through meta-data depends on the “size of the volume” and not on the “number of volumes”. ● Within the volume the meta data is not managed at “block-level” but at “chunk-level”. Typical block-size is 4KB and Typical chunk-size is 4 MB. This results in the huge reduction of metadata size of the block-volume that needs to be maintained. ● Checksum - One of the important metadata is checksum. OpenEBS guarantees bit-rot protection through the use of checksums. The checksums are managed at a chunk level only on Cold-Data. The checksums are not managed on hot-data, the blocks go in and out of chunks on the hot-data without the need of checksum calculation on the fly. ● Deduplication-while-tiering: Deduplication has capacity benefits but kills performance (either inline or offline). But in OpenEBS, we do this while moving the data from hot-to-cold tiers. In effect, the benefits of deduplication without the performance penalty.
- 19. Cchchchcunking LVM RAID XFSNvme - RAID KernelUserland lun1.1.metabook lun1.2.metabook lun1 data chunks lun2 data chunks lun3 data chunks Read - uncompress Write Dedup and compress DatainnvmeFlash Data blocks in NVRAM (NVDIMM) Blocks to chunks (coalescing) lun chunkbooks Data block to chunkbook mapper
- 20. OpenEBS - Metadata at scale is not an issue Global Metadata at a system level Logical volumes Lun Application Metadata is managed at block level IO Blocks IO Blocks 100TB Raw 8 TB Meta 100TB Raw XFS Meta data XFS Meta data2 TB Meta 2 TB Meta Logical Volumes in XFS files Volume meta Lun Application IO Blocks IO Chunks Metadata of the volume is managed at chunk level 100 GB Meta The volume IO processing has to deal with the global metadata of 8TB The volume IO processing has to deal with the volume metadata of 100GB
- 21. Storage Data format /xfs/outside/lun1 (sparse file) Block layout of the lun inside the xfs file Meta table of the lun (Fixed size) Two dimensional array indexed chunk number. Chunk number (computed with block byte range) ● Location (fast memory/chunk) ● A ● B ● C ● D ● ? Chunk table1 (Fixed size) Chunk table2 (Fixed size)
- 22. Storage Interface - HardDisks - SAS/SATA Flash - NVMe Flash - PCIe Flash - S3 - Cloud Block Storage
- 23. VSM Network Interface - Host Networking - VLANs / IPSpaces
- 24. Ease of Configuration - VSM Configuration Spec - Infra Spec - Integrate into K8s / EBS Compatible
- 25. Integration to Orchestration Options to consume the storage by containers: - iSCSI Driver ( Pre-provisioned) - Maya Volume Driver - Integrated Orchestration
- 26. Network (Flannel*) Network (Flannel) Remote Storage Local Storage OpenEBS Storage Hosts iSCSI iSCSI Driver HTTPS (manage) Storage Connectivity - iSCSI Claims K8s master K8s minions OpenEBS Maya master OpenEBS VSMs / Storage Pod Pod
- 27. Network (Flannel*) Network (Flannel) Remote Storage Local Storage OpenEBS Storage Hosts iSCSI Maya Driver HTTPS (manage) Storage Connectivity - Maya Driver K8s master K8s minions OpenEBS Maya master OpenEBS VSMs / Storage Pod Pod
- 28. Network (Flannel) Remote Storage Local Storage OpenEBS Storage Hosts TCP Maya Driver HTTPS (manage) Storage Connectivity - Shared Orchestration K8s master K8s minions OpenEBS Maya master OpenEBS VSMs / Storage Pod Pod TCMU
- 29. Resiliency and Fault Tolerance - Scaleout - Blue-Green Upgrades - Infra - Rolling Upgrades - VSMs - High-Availability
- 30. Security - Data Security - Encryption - Secure Delete
- 31. Telemetry - Monitoring and Troubleshooting - Analytics
- 32. Performance - IO Latency - Provisioning - Analytics
- 33. Scale - Capacity - Number of Volumes
- 34. Deployment Flexibility OpenEBS Deployment Options for: - Dedicated Storage (External) - Hyper-converged - Hybrid-Cloud (AWS)
- 35. OpenEBS Roadmap 0.1 0.4 1.0 0.3 0.2 ● Soft launch / Basic version ● Containerized controller ● Longhorn integration basics 0.2 - Basic k8s integration 0.3 - Full k8s integration (Hyper-Converged) 0.4 - Tiering and distributed storage 1.0 - First enterprise edition (Full functionality for basic production usage) ● K8s Provisioning via EBS-like driver ● S3 building blocks ● Complete longhorn integration ● First usable release for community consumption ● Complete K8s integration ● Fully hyper-converged ● Support for local backing store ● Building blocks of Q0S ● Building blocks of Tiering ● Tiering and QoS demonstrated ● Building blocks of ML for storage analytics ● DevOps operator use cases demonstrated ● Integration into Enterprise LDAP w/ role based access control ● All features supportable at scale Nov, 2017 Sep, 2017 Jun, 2017 Mar, 2017 Jan, 2017