Introducing MetalK8s, An Opinionated Kubernetes Implementation
0 likes682 views
Scality Architect Nicolas Trangez introduces MetalK8s, an opinionated Kubernetes distribution with a focus on long-term on-prem deployments, launched by Scality to deploy its Zenko solution in customer data centers. Nicolas presented this at the OpenStack Summit in Vancouver, on May 22, 2018.
Ad
More Related Content
What's hot (20)
Similar to Introducing MetalK8s, An Opinionated Kubernetes Implementation (20)
Ad
More from Scality (13)
Ad
Recently uploaded (20)
Introducing MetalK8s, An Opinionated Kubernetes Implementation
- 1. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS Nicolas Trangez - Technical Architect [email protected] @eikke
- 3. ONE PURPOSE GIVING FREEDOM & CONTROL TO PEOPLE WHO CREATE VALUE WITH DATA
- 4. 8 60+ 120+ 20+ ~10 GLOBAL CLIENT BASEGLOBAL PRESENCE 20+ OFFICES 200+ PEOPLE NATIONALITIES EUROPEAMERICAS AUSTRALIA JAPAN
- 5. OUR JOURNEY TO KUBERNETES Scality RING, S3 Connector & Zenko
- 6. Scality RING - Physical servers, some VMs - Only the OS available (incl. ‘Legacy’ like CentOS 6) - Static resource pools - Static server roles / configurations - Solution distributed as RPM packages, deployed using SaltStack - De-facto taking ownership of host, difficult to run multiple instances - Fairly static post-install On-premise Distributed Object & File Storage
- 7. Scality S3 Connector On-premise S3-compatible Object Storage - Physical servers, sometimes VMs - Static resource pools - “Microservices” architecture - Solution distributed as Docker container images, deployed using Ansible playbooks - No runtime orchestration - Log management, monitoring,... comes with solution
- 8. Scality Zenko - Deployed on-prem or ‘in the Cloud’: major paradigm shift - New challenges, new opportunities - Multi-Cloud Data Controller, must run on multiple Cloud platforms Multi-Cloud Data Controller
- 9. Scality Zenko - Embraced Docker as distribution mechanism - Some shared with Scality S3 Connector - For Cloud deployments, started with Docker Swarm - Ran into scaling, reliability and other technical issues - Decided to move to Kubernetes - Managed platforms for Cloud deployments, where available (GKE, AKS, EKS one day) - On-prem clusters Deployment Model
- 10. Scality Zenko - Homogenous deployment between in-cloud and on-prem - Various services provided by cluster: - Networking & policies - Service restart, rolling upgrades - Service log capturing & storage - Service monitoring & metering - Load-balancing - TLS termination - Flexible resource management - If needed, easily add resources to cluster by adding some (VM) nodes - HorizontalPodAutoscaler Kubernetes Benefits
- 11. OUR JOURNEY TO KUBERNETES MetalK8s
- 12. On-prem Kubernetes - Can’t expect a Kubernetes to be available, provided by Scality customer - Looked into various existing offerings, but in the ends needs to be supported by/through Scality (single offering) - Decided to roll our own
- 13. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS
- 14. OPINIONATED We offer an out-of-the-box experience, no non-trivial choices to be made by users
- 15. LONG-TERM Zenko solution is mission-critical, can’t spawn a new cluster to upgrade and use ELB (or similar) in front
- 16. ON-PREM Can’t expect anything to be available but (physical) servers with a base OS
- 17. Scality MetalK8s - “Stand on the shoulders of giants” - Scope: 5-20 physical machine, pre-provisioned by customer or partner - Built on top of the excellent Kubespray Ansible playbook - Use Kubespray to lay out a base Kubernetes cluster - Also: etcd, CNI - Add static & dynamic inventory validation pre-checks, OS tuning, OS security - Based on experience from large-scale Scality RING deployments - Augment with various services, deployed using Helm - Operations - Ingress - Cluster services - Take care of on-prem specific storage architecture
- 18. Scality MetalK8s: Cluster Services - “Stand on the shoulders of giants” - Heapster for dashboard graphs, `kubectl top`,... - metrics-server for HorizontalPodAutoscaler - Looking into k8s-prometheus-adapter - Ingress & TLS termination: nginx-ingress-controller - Cluster monitoring & alerting: Prometheus, prometheus-operator, Alertmanager, kube-prometheus, Grafana - Host-based node_exporter on all servers comprising the cluster, including etcd - Host & container logs: ElasticSearch, Curator, fluentd, Kibana - Considering switch to fluent-bit - All of the above gives a great out-of-the-box experience for operators
- 22. Scality MetalK8s: Storage - On-prem: no EBS, no GCP Persistent Disks, no Azure Storage Disk,... - Also: can’t rely on NAS (e.g. through OpenStack Cinder) to be available - Lowest common denominator: local disks in a node - PVs bound to a node, hence PVCs bound, hence Pods bound - Thanks PersistentLocalVolumes & VolumeScheduling! - Decided not to use LocalVolumeProvisioner, but static approach (for now) - Based on LVM2 Logical Volumes for flexibility - PV, VG, LVs defined in inventory, created/formatted/mounted by playbook - K8s PV objects created by playbook - May support whole partitions/drives depending on application need - Working with community on Dynamic Local Volume provisioning - Also using LVM2
- 23. Scality MetalK8s: Deployment - Based on years of years of experience deploying Scality RING at enterprise customers, service providers,... - Constraints in datacentra often very different from ‘VMs on EC2’ - No direct internet access: everything through HTTP(S) proxy, no non-HTTP traffic - Dynamic server IP assignment - Security rules requiring services to bind to specific IPs only - Fully air gapped systems: requires 100% offline installation - Non-standard OS/kernel - Integration with corporate authn/authz systems - Not all of the above supported yet, tackling one by one - Relevant patches to be upstreamed to Kubespray - Only support RHEL/CentOS family of Linux distributions - Support for Ubuntu and others can be community-driven, Kubespray supports them - RHEL/CentOS sometimes difficult targets for containers/Docker/Kubernetes
- 24. Scality MetalK8s: Ease of Deployment $ # Requirements: a Linux or OSX machine with Python and coreutils-like $ # Create inventory $ vim inventory/... $ make shell # Launches a ‘virtualenv’ with Ansible & deps, ‘kubectl’, ‘helm’ $ # Demo @ https://asciinema.org/a/9kNIpBWg4KiwjT5mNSrH0tmj9 $ ansible-playbook -i inventory -b metal-k8s.yml $ # Grab a coffee, and done
- 25. Scality MetalK8s: The road forward - Documentation: Install guides, Operations guides, Troubleshooting guides,... - Forward & backward compatibility requirements - Sizing numbers - Hardware & software compatibility testing - Security auditing & testing - Testing/CI: install, upgrade, downgrade, ‘monkey’,... - Also in very constrained environments - Delivery of fully-offline installation package - ...
- 26. SCALITY METALK8S AN OPINIONATED KUBERNETES DISTRIBUTION WITH A FOCUS ON LONG-TERM ON-PREM DEPLOYMENTS https://zenko.io https://github.com/scality/metal-k8s @Scality | @Zenko