Deploying deep learning models with Kubernetes and Kubeflow

Model Deployment
Alexey Grigorev
Principal Data Scientist — OLX Group
Founder — DataTalks.Club

Plan
● Different options to deploy a model (Lambda, Kubernetes, SageMaker)
● Kubernetes 101
● Deploying an XGB model with Flask and Kubernetes
● Deploying a Keras model with TF-Serving and Kubernetes
● Deploying a Keras model with Kubeflow

Ways to deploy a model
● Flask + AWS Elastic Beanstalk
● Serverless (AWS Lambda)
● Kubernetes (EKS)
● Kubeflow (EKS)
● AWS SageMaker
● ...
(or their alternatives in other cloud providers)

{
"tshirt": 0.9993,
"pants": 0.0005,
"shoes": 0.00004
}
AWS Lambda

Kubernetes
Ingress
Client
Node1
Node2
Pod A
Pod B Pod C
Pod D
Pod E
Pod F
Service
1
Service
2
Deployment 1
Deployment 2
Kubernetes Cluster

Kubeflow / KFServing
Ingress
Client
Node1
Node2
Pod A
Pod B Pod C
Pod D
Pod E
Pod F
Service
1
Service
2
Deployment 1
Deployment 2
Kubernetes Cluster
InferenceService

SageMaker
Client
Model
Endpoint
AWS SageMaker

Lambda vs SageMaker vs Kubernetes
● Lambda
○ Cheap for small load
○ Easy to manage
○ Not always transparent

● Lambda
○ Easy to manage
● SageMaker (serving)
○ Easy to use/manage
○ Needs wrappers
○ Expensive

● Lambda
○ Easy to manage
● SageMaker (serving)
○ Easy to use/manage
○ Needs wrappers
○ Expensive
● Kubernetes
○ Complex (for me)
○ More flexible
○ Cloud-agnostic *
○ Requires support
○ Cheaper for high load
* sort of

Kubernetes glossary
● Pod ~ one instance of your service
● Deployment - a bunch of pods
● HPA - horizontal pod autoscaler
● Node - a server (e.g. EC2 instance)
● Service - an interface to the deployment
● Ingress - an interface to the cluster

import xgboost as xgb
# load the model from the pickle file
@app.route('/predict', methods=['POST'])
def predict():
data = request.get_json()
result = apply_model(data)
return jsonify(result)
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=9696)

FROM python:3.7.5-slim
RUN pip install flask gunicorn xgboost
COPY "model.py" "model.py"
EXPOSE 9696
ENTRYPOINT ["gunicorn", "--bind", "0.0.0.0:9696", "model:app"]

apiVersion: apps/v1
kind: Deployment
metadata:
name: xgb-model
labels:
app: xgb-model
spec:
replicas: 1
selector:
matchLabels:
app: xgb-model
template:
metadata:
labels:
app: xgb-model
spec:
containers:
- name: xgb-model
image: XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/xgb-model:v100500
ports:
- containerPort: 9696
env:
- name: MODEL_PATH
value: "s3://models-bucket-pickle/xgboost.bin"

apiVersion: v1
kind: Service
metadata:
name: xgb-model
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 9696
protocol: TCP
name: http
selector:
app: xgb-model

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

import tensorflow as tf
from tensorflow import keras
model = keras.models.load_model('keras-model.h5')
tf.saved_model.save(model, 'tf-model')

$ ls -lhR
.:
total 3,1M
4,0K assets
3,1M saved_model.pb
4,0K variables
./assets:
total 0
./variables:
total 83M
83M variables.data-00000-of-00001
15K variables.index

saved_model_cli show --dir tf-model --all
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
...
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_8'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 299, 299, 3)
name: serving_default_input_8:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense_7'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 10)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict

docker run -it --rm
-p 8500:8500
-v "$(pwd)/tf-model:/models/tf-model/1"
-e MODEL_NAME=tf-model
tensorflow/serving:2.3.0
2021-09-07 21:03:58.579046: I tensorflow_serving/model_servers/server.cc:367]
Running gRPC ModelServer at 0.0.0.0:8500 ...
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
2021-09-07 21:03:58.582097: I tensorflow_serving/model_servers/server.cc:387]
Exporting HTTP/REST API at:localhost:8501 ...

pip install grpcio==1.32.0
tensorflow-serving-api==2.3.0

https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-09-kubernetes/09-image-preparation.ipynb
def np_to_protobuf(data):
return tf.make_tensor_proto(data, shape=data.shape)
pb_request = predict_pb2.PredictRequest()
pb_request.model_spec.name = 'tf-model'
pb_request.model_spec.signature_name = 'serving_default'
pb_request.inputs['input_8'].CopyFrom(np_to_protobuf(X))
pb_result = stub.Predict(pb_request, timeout=20.0)
pred = pb_result.outputs['dense_7'].float_val

Gateway
(Resize and
process image)
Flask
Model
(Make predictions)
TF-Serving
Pants
Raw
predictions
Pre-processed
image

Not so fast
def np_to_protobuf(data):
return tf.make_tensor_proto(data, shape=data.shape)
pb_request = predict_pb2.PredictRequest()
pb_request.model_spec.name = 'tf-model'
pb_request.model_spec.signature_name = 'serving_default'
pb_request.inputs['input_8'].CopyFrom(np_to_protobuf(X))
pb_result = stub.Predict(pb_request, timeout=20.0)
pred = pb_result.outputs['dense_7'].float_val

2,0 GB dependency?
Get only the things you need!
https://github.com/alexeygrigorev/tensorflow-protobuf

from tensorflow.keras.applications.xception import preprocess_input
https://github.com/alexeygrigorev/keras-image-helper
from keras_image_helper import create_preprocessor
preprocessor = create_preprocessor('xception', target_size=(299, 299))
url = 'http://bit.ly/mlbookcamp-pants'
X = preprocessor.from_url(url)

Next steps...
● Bake in the model into the TF-serving image
● Wrap the gRPC calls in a Flask app for the Gateway
● Write a Dockerfile for the Gateway
● Publish the images to ERC

Okay!
We’re ready to deploy to K8S

apiVersion: apps/v1
kind: Deployment
metadata:
name: tf-serving-model
labels:
app: tf-serving-model
spec:
replicas: 1
selector:
matchLabels:
template:
metadata:
labels:
spec:
containers:
- name: tf-serving-model
image: X.dkr.ecr.eu-west-1.amazonaws.com/model-serving:tf-serving-model
ports:

apiVersion: v1
kind: Service
metadata:
name: tf-serving-model
labels:
spec:
ports:
- port: 8500
targetPort: 8500
protocol: TCP
name: http
selector:

kubectl apply -f tf-serving-deployment.yaml
kubectl apply -f tf-serving-service.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
name: serving-gateway
labels:
app: serving-gateway
spec:
replicas: 1
selector:
matchLabels:
template:
metadata:
labels:
spec:
containers:
- name: serving-gateway
image: X.dkr.ecr.eu-west-1.amazonaws.com/model-serving:serving-gateway
ports:
env:
- name: TF_SERVING_HOST
value: "tf-serving-model.default.svc.cluster.local:8500"

apiVersion: v1
kind: Service
metadata:
name: serving-gateway
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 9696
protocol: TCP
name: http
selector:

kubectl apply -f gateway-deployment.yaml
kubectl apply -f gateway-service.yaml

gRPC load balancing
https://kubernetes.io/blog/2018/11/07/grpc-load-balancing-on-kubernetes-without-tears/

Deploying deep learning models with Kubernetes and Kubeflow

Kubeflow
● Open-source ML platform with wany services (notebooks, pipelines, serving)
● KFServing makes it easier to deploy models compared to plain Kubernetes

Installing
https://mlbookcamp.com/article/kfserving-eks-install
git clone git@github.com:alexeygrigorev/kubeflow-deep-learning.git
cd kubeflow-deep-learning/install
./install.sh

Next...
● Upload the saved_model to S3
● Allow KFServing to access S3
https://mlbookcamp.com/article/kfserving-eks-install

apiVersion: "serving.kubeflow.org/v1beta1"
kind: "InferenceService"
metadata:
name: "tf-model"
spec:
default:
predictor:
serviceAccountName: sa
tensorflow:
storageUri: "s3://models-bucket-tf/tf-model"

kubectl apply -f tf-inference-service.yaml

$ kubectl get inferenceservice
NAME URL
flowers-sample http://tf-model.default.kubeflow.mlbookcamp.com/v1/models/tf-model ...

url = f'https://{model_url}:predict'
data = {
"instances": X.tolist()
}
resp = requests.post(url, json=data)
results = resp.json()

Pre-processing
(resize/process
images)
Post-processing
(transform
predictions)
Transformer
Model
(Make predictions)
KFServing
Pants
Raw
predictions
Pre-processed
image

apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
name: "tf-model"
spec:
default:
predictor:
serviceAccountName: sa
tensorflow:
storageUri: "s3://models-bucket-tf/tf-model"
transformer:
custom:
container:
image: "agrigorev/kfserving-keras-transformer:0.0.1"
name: user-container
env:
- name: MODEL_INPUT_SIZE
value: "299,299"
- name: KERAS_MODEL_NAME
value: "xception"
- name: MODEL_LABELS
value: "dress,hat,longsleeve,outwear,pants,shirt,shoes,shorts,skirt,t-shirt"

https://github.com/alexeygrigorev/kfserving-keras-transformer

url = f'https://{model_url}:predict'
data = {
"instances": [
{"url": "http://bit.ly/mlbookcamp-pants"},
]
}
resp = requests.post(url, json=data)
results = resp.json()

What’s the catch?
● Kubeflow runs on Kubernetes
● Not easy to run the whole thing locally
● Not easy to debug
● Istio

Summary
● AWS SageMaker vs AWS Lambda vs Kubernetes vs Kubeflow

Summary
● Deploying models with Kubernetes: deployment + service

Summary
● Deploying Keras models: TF-Serving + Gateway (over gRPC)

Summary
● KFServing: transformers + model

Summary
● KFServing: transformers + model
● No size fits all

mlbookcamp.com
● Learn Machine Learning by doing
projects
● http://bit.ly/mlbookcamp
● Get 40% off with code “grigorevpc”
Machine Learning
Bookcamp

mlzoomcamp.com
Learn Machine Learning Engineering in 4
months
Machine Learning
Zoomcamp
Zoom

@Al_Grigor
agrigorev
DataTalks.Club

Deploying deep learning models with Kubernetes and Kubeflow

Recommended

More Related Content

Similar to Deploying deep learning models with Kubernetes and Kubeflow (20)

More from DataPhoenix (6)

Recently uploaded (20)

Deploying deep learning models with Kubernetes and Kubeflow