Jupyter Notebooks for machine learning on Kubernetes & OpenShift | DevNation Tech Talk
1 like3,045 views
In this session you will see how to take your machine model from development to production by watching the steps involved, which include: 1) Developing a ML model crafted via a Jupyter Notebook directly on top of Kubernetes/OpenShift; 2) Publishing that model as a service to be shared with your team or even the world; and 3) Monitoring the RESTful service via Grafana.
![In [ ]:
In [ ]:
In [ ]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
Exploring my data set
| Python 3
In [ ]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-29-320.jpg)
![In [*]:
In [ ]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
Exploring my data set
| Python 3
In [ ]: new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
In [ ]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-30-320.jpg)
![In [1]:
In [ ]:
Out [1]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
120000
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
Exploring my data set
| Python 3
In [ ]: new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
In [ ]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-31-320.jpg)
![In [1]:
In [*]:
Out [1]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
120000
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
Exploring my data set
| Python 3
In [ ]: new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
In [ ]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-32-320.jpg)
![In [1]:
In [2]:
Out [1]:
Out [2]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
120000
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
Exploring my data set
| Python 3
In [ ]: new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
In [ ]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-33-320.jpg)
![In [1]:
In [2]:
Out [1]:
Out [2]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
120000
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
Exploring my data set
| Python 3
In [3]: new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
In [ ]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-34-320.jpg)
![In [1]:
In [2]:
Out [1]:
Out [2]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
120000
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
Exploring my data set
| Python 3
In [3]: new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
In [*]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-35-320.jpg)
![In [1]:
In [2]:
Out [1]:
Out [2]:
import pandas as pd
data = pd.read_parquet("data.parquet")
data.size
120000
from plot_tool import plot
plot_df = pd.DataFrame(data, columns=["x", "y"])
plot(plot_df)
Exploring my data set
| Python 3
In [3]: new_data = pd.read_parquet("new_data.parquet")
data = data.append(pd.DataFrame(new_data))
So far we have loaded in the data and plotted it. Then we appended our new
data. Now we save it to file.
In [4]: data.to_parquet("data2.parquet")](https://image.slidesharecdn.com/sophwats-devnation-jupyter-notebooks-for-ml-on-kubernetes-and-openshift6-4-20-200604172418/85/Jupyter-Notebooks-for-machine-learning-on-Kubernetes-OpenShift-DevNation-Tech-Talk-36-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Jupyter Notebooks for machine learning on Kubernetes & OpenShift | DevNation Tech Talk (20)
Ad
More from Red Hat Developers (20)
Ad
Recently uploaded (20)
Jupyter Notebooks for machine learning on Kubernetes & OpenShift | DevNation Tech Talk
- 1. Sophie Watson [email protected] @sophwats Jupyter Notebooks for Machine learning on Kubernetes and Openshift
- 2. Sophie Watson [email protected] @sophwats Jupyter Notebooks for Machine learning on Kubernetes and Openshift
- 3. Agenda - ML workflow - Kubernetes and OpenShift for Machine Learning - Demo
- 4. codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation
- 5. feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning codifying problem and metrics codifying problem and metrics
- 6. codifying problem and metrics feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning codifying problem and metrics
- 7. feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning codifying problem and metrics codifying problem and metrics data collection and cleaning
- 8. feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning codifying problem and metrics codifying problem and metrics feature engineering data collection and cleaning
- 9. codifying problem and metrics feature engineering data collection and cleaning feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning codifying problem and metrics
- 10. codifying problem and metrics feature engineering data collection and cleaning feature engineering model training and tuning model validation model deployment monitoring, validation data collection and cleaning codifying problem and metrics
- 11. data collection and cleaning feature engineering model training and tuning model validation model deployment monitoring, validation codifying problem and metrics codifying problem and metrics feature engineering model training and tuning data collection and cleaning legit
- 12. data collection and cleaning feature engineering model training and tuning model validation model deployment monitoring, validation codifying problem and metrics codifying problem and metrics feature engineering model training and tuning data collection and cleaning legit
- 13. data collection and cleaning feature engineering model training and tuning model validation model deployment monitoring, validation codifying problem and metrics data collection and cleaning codifying problem and metrics feature engineering model training and tuning model validation
- 14. feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation codifying problem and metrics data collection and cleaning codifying problem and metrics feature engineering model training and tuning model validation model deployment monitoring, validation
- 15. codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation
- 16. codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation
- 17. codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation Robust, repeatable pipelines Model monitoring and alerting Scale up Scale out Self service environments Reproducible environments Multi-tenant environments Access to specialised hardware Scale out
- 18. base image configuration and installation recipes user application code 979229b9 33721112 e8cae4f6 2bb6ab16 a8296f7e a6afd91e 6b8cad3e
- 19. base image configuration and installation recipes user application code 979229b9 33721112 e8cae4f6 2bb6ab16 a8296f7e a6afd91e 6b8cad3e
- 20. base image configuration and installation recipes user application code 979229b9 33721112 e8cae4f6 2bb6ab16 a8296f7e a6afd91e 6b8cad3e
- 21. base image configuration and installation recipes user application code 979229b9 33721112 e8cae4f6 2bb6ab16 a8296f7e a6afd91e 6b8cad3e
- 22. base image configuration and installation recipes user application code 979229b9 33721112 e8cae4f6 2bb6ab16 a8296f7e a6afd91e 6b8cad3e model in production on 16 July 2019
- 28. codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation Robust, repeatable pipelines Model monitoring and alerting Scale up Scale out Self service environments Reproducible environments Multi-tenant environments Access to specialised hardware Scale out
- 29. In [ ]: In [ ]: In [ ]: import pandas as pd data = pd.read_parquet("data.parquet") data.size from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. Exploring my data set | Python 3 In [ ]: data.to_parquet("data2.parquet")
- 30. In [*]: In [ ]: import pandas as pd data = pd.read_parquet("data.parquet") data.size from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) Exploring my data set | Python 3 In [ ]: new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. In [ ]: data.to_parquet("data2.parquet")
- 31. In [1]: In [ ]: Out [1]: import pandas as pd data = pd.read_parquet("data.parquet") data.size 120000 from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) Exploring my data set | Python 3 In [ ]: new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. In [ ]: data.to_parquet("data2.parquet")
- 32. In [1]: In [*]: Out [1]: import pandas as pd data = pd.read_parquet("data.parquet") data.size 120000 from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) Exploring my data set | Python 3 In [ ]: new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. In [ ]: data.to_parquet("data2.parquet")
- 33. In [1]: In [2]: Out [1]: Out [2]: import pandas as pd data = pd.read_parquet("data.parquet") data.size 120000 from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) Exploring my data set | Python 3 In [ ]: new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. In [ ]: data.to_parquet("data2.parquet")
- 34. In [1]: In [2]: Out [1]: Out [2]: import pandas as pd data = pd.read_parquet("data.parquet") data.size 120000 from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) Exploring my data set | Python 3 In [3]: new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. In [ ]: data.to_parquet("data2.parquet")
- 35. In [1]: In [2]: Out [1]: Out [2]: import pandas as pd data = pd.read_parquet("data.parquet") data.size 120000 from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) Exploring my data set | Python 3 In [3]: new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. In [*]: data.to_parquet("data2.parquet")
- 36. In [1]: In [2]: Out [1]: Out [2]: import pandas as pd data = pd.read_parquet("data.parquet") data.size 120000 from plot_tool import plot plot_df = pd.DataFrame(data, columns=["x", "y"]) plot(plot_df) Exploring my data set | Python 3 In [3]: new_data = pd.read_parquet("new_data.parquet") data = data.append(pd.DataFrame(new_data)) So far we have loaded in the data and plotted it. Then we appended our new data. Now we save it to file. In [4]: data.to_parquet("data2.parquet")
- 37. Identity and RBAC Project Isolation Sharing of resources Resource Isolation Resource Quotas Priority
- 38. codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation Robust, repeatable pipelines Model monitoring and alerting Self service environments Reproducible environments Multi-tenant environments Access to specialised hardware
- 40. @sophwats Demo
- 41. codifying problem and metrics feature engineering model training and tuning model validation data collection and cleaning model deployment monitoring, validation OpenShift Pipelines
- 42. codifying problem and metrics model validation data collection and cleaning model deployment monitoring, validation 2 3 OpenShift Pipelines