Scaling notebooks for Deep Learning workloads
3 likes364 views
The document discusses the use of Jupyter notebooks for deep learning workloads, emphasizing the importance of Kubernetes for resource management and scalability. It highlights the transition to using Jupyter enterprise gateway for kernel isolation and efficient resource utilization, enabling smoother deep learning application lifecycles. The presentation outlines the benefits of using lightweight containerized environments and the integration of various tools in the deep learning infrastructure to support extensive data science applications.
![The Secret Sauce:
Process Proxies mixed with Kernel Launchers
© 2018 IBM Corporation
Process Proxy:
• Abstracts kernel process represented by Jupyter
framework
• Pluggable class definition identified in kernelspec
(kernel.json)
• Manages kernel lifecycle
Kernel Launcher:
• Embeds target kernel
• Listens on gateway communication port
• Conveys interrupt requests (via local signal)
• Could be extended for additional communications
{
"language": "python",
"display_name": "Spark - Python (Kubernetes Mode)",
"process_proxy": {
"class_name":
"enterprise_gateway.services.processproxies.k8s.KubernetesProcessP
roxy",
"config": {
"image_name": "elyra/kubernetes-kernel-py:dev",
"executor_image_name": "elyra/kubernetes-kernel-py:dev”,
"port_range" : "40000..42000"
}
},
"env": {
"SPARK_HOME": "/opt/spark",
"SPARK_OPTS": "--master k8s://https://${KUBERNETES_SERVICE_HOST
--deploy-mode cluster --name …",
…
},
"argv": [
"/usr/local/share/jupyter/kernels/spark_python_kubernetes/bin/run.
sh",
"{connection_file}",
"--RemoteProcessProxy.response-address",
"{response_address}",
"--RemoteProcessProxy.spark-context-initialization-mode",
"lazy"
]
}](https://image.slidesharecdn.com/scalingnotebooksfordeeplearningworkloads-180823155615/85/Scaling-notebooks-for-Deep-Learning-workloads-25-320.jpg)
Scaling notebooks for Deep Learning workloads
- 1. Scaling Notebooks for Deep Learning workloads Luciano Resende JupyterCon - 2018 1© 2018 IBM Corporation
- 2. About me - Luciano Resende 2 Open Source AI Platform Architect – IBM – CODAIT • Senior Technical Staff Member at IBM, contributing to open source for over 10 years • Currently contributing to : Jupyter Notebook ecosystem, Apache Bahir, Apache Toree, Apache Spark among other projects related to AI/ML platforms [email protected] https://www.linkedin.com/in/lresende @lresende1975 https://github.com/lresende © 2018 IBM Corporation
- 3. Jupyter Notebooks Overview 3© 2018 IBM Corporation
- 4. Jupyter Notebooks © 2018 IBM Corporation 4 Notebooks are interactive computational environments, in which you can combine code execution, rich text, mathematics, plots and rich media.
- 5. Jupyter Notebooks © 2018 IBM Corporation 5 • Notebook UI runs on the browser • The Notebook Server serves the ’Notebooks’ • Kernels interpret/execute cell contents – Are responsible for code execution – Abstracts different languages – 1:1 relationship with Notebook – Runs and consume resources as long as notebook is running
- 6. Jupyter Notebooks and Deep Learning 6© 2018 IBM Corporation
- 7. Deep Learning workloads © 2018 IBM Corporation 7 Resource Intensive workloads Requires expensive hardware (GPUs) Long Running training jobs - Simple MNIST takes over one hour WITHOUT a decent GPU - Other non complex deep learning model training can easily take over a day WITH GPUs
- 8. Jupyter & Kubernetes © 2018 IBM Corporation 8 Kubernetes Platform - Containers provides a flexible way to deploy applications and are here to stay - Kubernetes enables easy management of containerized applications and resources with the benefit of Elasticity and Quality of Services Source: https://github.com/Langhalsdino/Kubernetes-GPU-Guide
- 9. Enterprise Gateway & Kubernetes © 2018 IBM Corporation Supported Platforms FfDL Before Enterprise Gateway After Enterprise Gateway Before Jupyter Enterprise Gateway … • Resources required for all kernels needs to be allocated during Notebook Server pod creation • Resources limited to what is physically available on the host node that runs all kernels and associated Spark drivers After Jupyter Enterprise Gateway … • Gateway pod very lightweight • Kernels in their own pod, isolation • Kernel pods built from community images: Spark-on-K8s, TensorFlow, Keras, etc.
- 10. Jupyter Enterprise Gateway - Kubernetes © 2018 IBM Corporation 10 Container images defined in kernelspec Community image Kernel Spark on K8 Kernel Distributed File System Vanilla Kernels Spark based kernels Gateway nb2kg nb2kg
- 11. Deep Learning Platforms © 2018 IBM Corporation 11 Prohibited costs - Deep Learning resources are prohibitive in costs to be locked/idle during interactive development Deep Learning Platforms - We have seen the rise of Deep Learning platforms that leverage containers and Kubernetes as the basis of their infrastructure - Kubernetes enables Deep Learning platforms to easily share and restrict accelerated hardware Fabric for Deep Learning IBM Watson Studio Deep Learning as a service Batch oriented developm ent
- 12. Deep Learning Workspace March 30 2018 / © 2018 IBM Corporation 12 Streamline Data Science user experience when coming from Notebook/Interactive development interfaces • Current process include multiple steps, one being decomposing the notebook into an application that needs to be submitted as a zip to the deep learning runtime which becomes a show stopper for data scientists to adopt FfDL and DLaas
- 13. March 30 2018 / © 2018 IBM Corporation 13 Streamline the Deep Learning application lifecycle • Run local notebook experiments, with small data samples and seamlessly validate experiments on Deep Learning environments • IBM Cloud DLaaS, FfDL (open source), KubeFlow (open source) Simplify productionalization of Model training and serving from Notebooks • Enable running/scheduling notebooks on production environments as batch jobs • Results can be made available via updated notebook, or exported to html, pdf and a few other formats. Interactive development lifecycle done on commodity hardware with sampled data Training on full dataset gets scheduled as batch jobs on deep learning infrastructure Deep Learning Workspace
- 14. 14March 30 2018 / © 2018 IBM Corporation
- 15. March 30 2018 / © 2018 IBM Corporation 15 • User select where to run the experiment • Job is packaged and submitted on behalf of user • User has access to Job Console to monitor experiment Deep Learning Workspace
- 16. March 30 2018 / © 2018 IBM Corporation Deep Learning Extensions Notebook Extension (button) https://github.com/lresende/enterprise_scheduler_extension Notebook Scheduler (backend) https://github.com/lresende/enterprise_scheduler 16 Enable interactive development on commodity hardware while easily enabling model training on Deep Learning infrastructure with GPU based hardware. Deep Learning Workspace
- 17. 17March 30 2018 / © 2018 IBM Corporation
- 18. BACKUP SLIDES 18© 2018 IBM Corporation
- 19. Jupyter Enterprise Gateway 19© 2018 IBM Corporation
- 20. Jupyter Enterprise Gateway © 2018 IBM Corporation Jupyter Enterprise Gateway at IBM Code https://developer.ibm.com/code/openprojects/jupyter-enterprise-gateway/ Jupyter Enterprise Gateway source code at GitHub https://github.com/jupyter-incubator/enterprise_gateway Jupyter Enterprise Gateway Documentation http://jupyter-enterprise-gateway.readthedocs.io/en/latest/ Supported Kernels Supported Platforms 20 A lightweight, multi-tenant, scalable and secure gateway that enables Jupyter Notebooks to share resources across an Apache Spark or Kubernetes cluster for Enterprise/Cloud use cases Spectrum Conductor +
- 21. Jupyter Enterprise Gateway Features © 2018 IBM Corporation Gather Data Analyze Data Machine Learning Deep Learning Deploy Model Maintain Model Python Data Science Stack Fabric for Deep Learning (FfDL) Mleap + PFA Scikit-LearnPandas Apache Spark Apache Spark Jupyter Model Asset eXchange Keras + Tensorflow 21 16 32 48 64 0 10 20 30 40 50 60 70 80 4 Nodes 8 Nodes 12 Nodes 16 NodesMaxKernels(4GBHeap) Cluster Size (32GB Nodes) MAXIMUM NUMBER OF SIMULTANEOUS KERNELS Optimized Resource Allocation – Utilize resources on all cluster nodes by running kernels as Spark applications in YARN Cluster Mode. – Pluggable architecture to enable support for additional Resource Managers Enhanced Security – End-to-End secure communications • Secure socket communications • Encrypted HTTP communication using SSL Multiuser support with user impersonation – Enhance security and sandboxing by enabling user impersonation when running kernels (using Kerberos). – Individual HDFS home folder for each notebook user. – Use the same user ID for notebook and batch jobs. Kernel Kernel Kernel Kernel Kernel Kernel Kernel Kernel Kernel
- 22. Jupyter Enterprise Gateway – YARN © 2018 IBM Corporation 22 YARN Cluster YARN Workers Gateway Node Jupyter Enterprise Gateway • Multitenancy • Remote kernel lifecycle management via process proxies Spark Executors Spark Executors Spark Executors Yarn Container Jupyter Kernel Spark Driver Impersonation: Alice’s kernel runs under Alice’s user ID. Spark Executors Spark Executors Spark Executors Yarn Container Jupyter Kernel Spark Driver SecurityLayer nb2kg nb2kg Spark Executors Spark Executors Spark Executors Yarn Container Jupyter Kernel Spark Driver Bob Alice
- 23. Enterprise Gateway & Kubernetes © 2018 IBM Corporation Supported Platforms FfDL Before Enterprise Gateway After Enterprise Gateway Before Jupyter Enterprise Gateway … • Resources required for all kernels needs to be allocated during Notebook Server pod creation • Resources limited to what is physically available on the host node that runs all kernels and associated Spark drivers After Jupyter Enterprise Gateway … • Gateway pod very lightweight • Kernels in their own pod, isolation • Kernel pods built from community images: Spark-on-K8s, TensorFlow, Keras, etc.
- 24. Jupyter Enterprise Gateway - Kubernetes © 2018 IBM Corporation 24 Container images defined in kernelspec Community image Kernel Spark on K8 Kernel Distributed File System Vanilla Kernels Spark based kernels Gateway nb2kg nb2kg
- 25. The Secret Sauce: Process Proxies mixed with Kernel Launchers © 2018 IBM Corporation Process Proxy: • Abstracts kernel process represented by Jupyter framework • Pluggable class definition identified in kernelspec (kernel.json) • Manages kernel lifecycle Kernel Launcher: • Embeds target kernel • Listens on gateway communication port • Conveys interrupt requests (via local signal) • Could be extended for additional communications { "language": "python", "display_name": "Spark - Python (Kubernetes Mode)", "process_proxy": { "class_name": "enterprise_gateway.services.processproxies.k8s.KubernetesProcessP roxy", "config": { "image_name": "elyra/kubernetes-kernel-py:dev", "executor_image_name": "elyra/kubernetes-kernel-py:dev”, "port_range" : "40000..42000" } }, "env": { "SPARK_HOME": "/opt/spark", "SPARK_OPTS": "--master k8s://https://${KUBERNETES_SERVICE_HOST --deploy-mode cluster --name …", … }, "argv": [ "/usr/local/share/jupyter/kernels/spark_python_kubernetes/bin/run. sh", "{connection_file}", "--RemoteProcessProxy.response-address", "{response_address}", "--RemoteProcessProxy.spark-context-initialization-mode", "lazy" ] }
- 26. 26March 30 2018 / © 2018 IBM Corporation