Introduction to KubeDirector - SF Kubernetes Meetup
Download as PPT, PDF3 likes655 views
Presentation from San Francisco Kubernetes Meetup on October 30, 2018 https://www.meetup.com/San-Francisco-Kubernetes-Meetup/events/255431002 What is KubeDirector? - Tom Phelan & Joel Baxter, Bluedata Kubernetes is clearly the container orchestrator of choice for cloud-native stateless applications. And with the introduction of StatefulSets and Persistent Volumes it is becoming possible to run stateful applications on Kubernetes. Now the new KubeDirector project allows users to manage complex stateful clusters for AI, machine learning, and big data analytics on Kubernetes without writing a single line of GO code. KubeDirector is an open source Apache project that uses the standard Kubernetes custom resource functionality and API extensions to deploy and manage complex stateful scale-out application clusters. This session will provide an overview of the KubeDirector architecture, show how to author the metadata and artifacts required for an example stateful application (e.g. with Spark, Jupyter, and Cassandra), and demonstrate the deployment and management of the cluster on Kubernetes using KubeDirector. https://github.com/bluek8s/kubedirector
Ad
More Related Content
What's hot (20)
Similar to Introduction to KubeDirector - SF Kubernetes Meetup (20)
Ad
More from BlueData, Inc. (18)
Ad
Recently uploaded (20)
![Get & Download Wondershare Filmora Crack Latest [2025]](https://cdn.slidesharecdn.com/ss_thumbnails/revolutionizingresidentialwi-fi-250422112639-60fb726f-250429170801-59e1b240-thumbnail.jpg?width=560&fit=bounds)
Introduction to KubeDirector - SF Kubernetes Meetup
- 1. KubeDirector:KubeDirector: New Open Source ProjectNew Open Source Project for Stateful Applicationsfor Stateful Applications on Kuberneteson Kubernetes
- 2. Today’s Speakers Joel Baxter @joel_k_baxter Distinguished Engineer BlueData Tom Phelan @tapbluedata Co-Founder and Chief Architect BlueData
- 3. Agenda • Complex Stateful Applications – What Makes Large-Scale Big Data Analytics, AI, Machine Learning, and Deep Learning Applications Different • BlueK8s • KubeDirector • Demo • Q & A
- 4. Complex Stateful Applications • Big Data / AI / Machine Learning / Deep Learning • What do all these applications have in common? – Require large amounts of data – Use distributed processing, multiple tools / services – When on-prem, typically deployed on bare-metal – Do not have a cloud native architecture • No microservices • Application instance-specific state
- 5. Kubernetes – Pod • Ideally: Each application service could be deployed in its own container running in a Pod – Microservices architecture • Current reality: All services of each node for a complex stateful application must run in the same container – The Pod ordering feature does not help in the ordering of services (which is key for complex stateful apps)
- 6. Kubernetes – Operators • Operator • A way of packaging, deploying, and managing a given application • Operator Framework • A set of developer and runtime tools to help accelerate the writing of a Operator • Operator SDK • An SDK that further hides the complexities of the Kubernetes API Source: https://coreos.com/operators
- 7. Kubernetes – Operators • Still best suited when application is decomposed into independent services – Primarily in the realm of the application vendor or OSS community to change/re- architect apps (e.g. Spark)
- 8. • Application-specific means a new operator needs to be written for each application Kubernetes – Operators
- 9. BlueK8s and KubeDirector • BlueK8s is an Apache open source initiative focused on bringing enterprise support for complex stateful applications to Kubernetes • A series of projects will be rolled out under the BlueK8s umbrella – The first major project is “KubeDirector” Source: www.bluedata.com/blog/2018/07/operation-stateful-bluek8s-and-kubernetes-director
- 10. KubeDirector Overview • KubeDirector (KD) is a K8s “custom controller” • Watches for custom resources (CRs) to appear/change • Creates/modifies standard K8s resources (StatefulSets etc.) in response, to implement specs from CRs • Differs from normal operator pattern: – No app-specific logic in KubeDirector code – App deployment is data-driven from external app definitions – Supports interactions among different apps + other objects
- 11. Deploy KubeDirector to K8s Learn more at: https://github.com/bluek8s/kubedirector/wiki kubectl create -f kubedirector/deployment.yaml
- 12. Separation of Concerns • Application experts (on-site or elsewhere) – Responsible for making app images/metadata/configscripts – No need to write go code or understand operator concepts • Administrators (on-site) – Select which apps are available to end users – Change app versions independently of KubeDirector upgrade • End users – Pick from menu of applications and config choices
- 14. Custom Resource Definitions • Primary CRD: KubeDirectorCluster – Models any kind of application instance launchable by KubeDirector • Other CRDs for related objects, e.g. – App definitions (KubeDirectorApp) – DataTaps and other shared storage – Config sets for AD/LDAP integration for containers – Machine Learning models • This talk will concentrate on KubeDirectorCluster/App
- 16. Deployment • Create custom resource definitions (CRDs) in your K8s cluster • Deploy KubeDirector – Normally runs in a pod on same K8s cluster – Authenticates to K8s API w/ privileged service account • Configure KubeDirector global settings – E.g. supply app definitions, set types of service & storage
- 18. App Definition Metadata • App identifier/description/version • Service endpoints • Available “roles”, and container image per role • Available deploy-time choices, and their effects on services per role • Info for optional runtime setup package • And more!
- 19. App Definition Example 1/3 apiVersion: kubedirector.bluedata.io/v1alpha1 kind: KubeDirectorApp metadata: name: spark221e2 spec: label: name: Spark 2.2.1 on centos7x with Jupyter image: repoTag: docker.io/bluedata/sparkbase:2.0 setup_package: import_url: https://s3.amazonaws.com/mybucket/spark221e2/appconfig.tgz
- 20. App Definition Example 2/3 - id: spark_master_ui label: name: Spark master (web UI) endpoint: port: 8080 is_dashboard: true url_scheme: http - id: spark_worker_ui label: name: Spark worker (web UI) endpoint: port: 8081 is_dashboard: true url_scheme: http roles: - id: controller cardinality: 1 - id: worker cardinality: 0+ services: - id: spark label: name: Spark master endpoint: port: 7077
- 21. App Definition Example 3/3 config: selected_roles: - controller - worker role_services: - role_id: controller service_ids: - spark - spark_master_ui - role_id: worker service_ids: - spark_worker_ui
- 22. Application Setup Package • Optional tgz injected into each container, contains: – Entrypoint script – Standard script functions for reading deployment info – Any artifacts (config file templates etc.) required for setup • Entrypoint script will be invoked at lifecycle events: – This container has just been created – Some other member(s) added to or removed from the cluster
- 23. Setup Script Actions • Perform any setup that requires runtime info – E.g. FQDNs of other member(s) of the cluster • Enable and start appropriate services – Can query the role of current node – Services-to-start depend on role and deploy-time choices • Can use features of KubeDirector Agent, if deployed
- 25. Custom Resource Creation apiVersion: "kubedirector.bluedata.io/v1alpha1" kind: "KubeDirectorCluster" metadata: name: "spark-instance" spec: app: spark221e2 roles: - id: controller resources: limits: memory: "4Gi“ - id: worker members: 2
- 26. Cluster Creation Sequence 1/2
- 27. Cluster Creation Sequence 2/2
- 28. Other Operations • Shrink & expand of role member count is handled similarly • All resources are automatically cleaned up if CR is deleted (because CR is their “owner”) • End user can read the CR to see current status, service objects, etc.
- 29. KubeDirector Demonstration on GCP
- 30. Join the KubeDirector Community! Joel Baxter @joel_k_baxter Tom Phelan @tapbluedata https://github.com/bluek8s
Editor's Notes
- #6: Too many interdependencies in current monolithic design In this deployment model, the Pod is equivalent to a single container
- #8: Client (Helm) / Server (Tiller) Functions: package mgmt, app lifecycle mgmt, rendering Functionality permissions are “all or nothing” With Tiller, Helm cannot support per function user privileges Tiller scheduled to be dropped in the next release of Helm Without Tiller, Helm will have difficulty managing long-running tasks and monitoring
- #14: Each container in a deployed cluster will take on a role, which will determine: - the container image - the group of services running on that container - the resources (CPU/GPU/memory/disk) available to the container Functionality provided by optional agent: - Advanced service management - Track and advertise service statuses - Service watchdogging (restart) - Advanced configuration coordination - Token wait/wake between cluster members - Remote file copy within cluster as particular user - Remote command exec within cluster as particular user - Advanced job/script submission - Run on one or multiple nodes - Track status and output; - Advertise status & optionally send output to log collection
- #15: When to model things through CRDs? If an end user can create/edit the thing. The “related objects” CRDs can be used in conjunction with multiple clusters of multiple application types. CRDs are not necessary for administrative configuration of KD settings.
- #16: Largely a standard custom controller deployment so we will skim over this.
- #17: KD authenticates to k8s as a service account - has all privs in its own custom API space - has necessary privs (to deploy app clusters) for k8s resources in some k8s namespace(s) - pods, services, PVCs, etc.
- #19: Setup package is injected into containers. Catalog entry identifies URL for acquiring package, and whether the package’s operation requires that KubeDirector Agent be deployed into the containers. Backup slides have an example catalog entry.
- #20: Somewhat more complicated example on bluek8s github wiki.
- #22: (example of “config choices” not shown)
- #23: Edited/simple example script on bluek8s github wiki.
- #24: Configuration-related functionality provided by optional agent: - Token wait/wake between cluster members - Remote file copy within cluster as particular user - Remote command exec within cluster as particular user
- #26: To deploy an app cluster, the end user creates a CR with their tool of choice (e.g. kubectl). Slightly more complex example on bluek8s github wiki.
- #27: 1. An end user submits a CR spec to the K8s API. 2. K8s admission control consults the validator module in KubeDirector. 3. The validator consults the app catalog. If the request is valid for that app (e.g. appropriate choices, number of replicas, minimum memory for role, etc.) then the request is OK'ed. 4. The CR appears in the K8s KV store. 5. KubeDirector notices the CR and starts its reconciliation handler. 6. The handler consults the app catalog for appropriate images, services, persistent storage volumes, etc. It asks the K8s API to create stateful sets, services, etc. and to update the CR object's status properties. 7. Requested new and updated objects appear in the K8s KV store. 8. K8s begins creating pods to comply with stateful set replica counts.
- #28: 9. KubeDirector periodically polls the CR and runs the reconciliation handler again. 10. The handler checks on the current state of all related objects in the K8s store. Once all desired pods have become running pods, it proceeds to next steps. 11. The handler injects configuration data into the pods, and then an invocation that will fetch and run the application setup package. 12. Within each pod, the setup package is fetched from an artifact store and executed. 13. Once the application setup completes, the handler updates the CR status properties. 14. New status properties appear on the CR in the K8s KV store.
- #29: See backup slides for more reconciliation logic details.