RESTful Machine Learning with Flask and TensorFlow Serving - Carlo Mazzaferro
5 likes2,192 views
Those of us who use TensorFlow often focus on building the model that's most predictive, not the one that's most deployable. So how to put that hard work to work? In this talk, we'll walk through a strategy for taking your machine learning models from Jupyter Notebook into production and beyond.
![Takeaways
● To make great products
○ Do machine learning like the great engineer you are, not like the great machine learning expert
you aren’t.
[Google’s best practices for ML]
○ Provide DS with the right tooling, and the entire organization will benefit](https://image.slidesharecdn.com/restfulmachinelearningwithflaskandtensorflowserving-190104023422/85/RESTful-Machine-Learning-with-Flask-and-TensorFlow-Serving-Carlo-Mazzaferro-7-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to RESTful Machine Learning with Flask and TensorFlow Serving - Carlo Mazzaferro (20)
Ad
More from PyData (20)
Ad
Recently uploaded (20)
RESTful Machine Learning with Flask and TensorFlow Serving - Carlo Mazzaferro
- 1. RESTful Machine Learning with Flask and TensorFlow Serving Carlo Mazzaferro Data Scientist ByteCubed PyData DC 2018
- 2. Goals ● Explain some common problems that arise when developing machine learning products ● Describe some of the solutions that are leveraged to solve those problems, alongside with their limitations ● Open source a library
- 3. Background ● BS Biomed Eng/Bioinformatics ● DNA stuff ○ Lots of data (wrangling) ○ Data pipelines ○ Data management
- 4. Background ● BS Biomed Eng/Bioinformatics ● DNA stuff ○ Lots of data (wrangling) ○ Data pipelines ○ Data management ○ Data Science aspirant ● Currently ○ ML Engineering @ ByteCubed ○ Some frontend work
- 5. Background ● Currently ○ ML Engineering @ ByteCubed ○ Some frontend work ○ Putting models in the client’s hands ● Since always ○ Workflow optimization for ML/DS
- 6. Takeaways ● To make great products ○ Repeatability matters ○ Ease of deployment matters ○ Quick iteration is key ○ 80-20 rule Ultimately, reduce friction between R&D and the rest of the Software Practice
- 7. Takeaways ● To make great products ○ Do machine learning like the great engineer you are, not like the great machine learning expert you aren’t. [Google’s best practices for ML] ○ Provide DS with the right tooling, and the entire organization will benefit
- 8. TensorFlow Serving A high-performance serving system for machine learning models, designed for production environments Capabilities ● High-performance inference ● Model discovery ● RPC interface (gRPC and REST) ● Much more: tensorflow.org/serving
- 9. How? A high-performance serving system for machine learning models, designed for production environments Capabilities ● Keeping models loaded without needing to restore the dataflow graph and session ● gRPC interface for high-performance, low-latency inference ● Low-level control on how models are loaded ○ C++ API
- 10. ● Relies on a C++ API for low-level control ● Undocumented Python API ○ Small subset of features implemented ● gRPC API only ○ REST API (for inference only) introduced in August Limitations A high-performance serving system for machine learning models, designed for production environments
- 11. ● Pythonic API ✅ ● JSON + REST ✅ ● Test different models, architectures, configurations ✅ ● Track models over time ✅ Needs A high-performance serving system for machine learning models, designed for production environments
- 12. Current state Existing solutions ● Clipper.ai → great, but not very active (last commit in June) ● P/SaaS platforms (Algorithmia, Rekognition, etc.) ● Mlflow.org → R&D, tracking of model performance ● Custom solutions of varied quality Model deployment frameworks
- 13. Introducing racket Minimalistic framework for ML model deployment & management ● Access to most of the TensorFlow Serving functionality in a Pythonic way ● Model exploration and deployment, consolidated
- 14. racket Motivation ● Automation ○ Quick iteration ● Reduce boilerplate ○ Versioning & serving ● Exploration & visibility ○ Loading a different version ○ Ability to track & query model performance ● Flexibility ○ Multiple models can be loaded and are accessible through a simple API
- 15. racket Features ● Automated model version ● RESTful interface with rich capabilities ○ Loading a different model ○ Loading a different version ○ Ability to track & query model performance ● Automatic API documentation with Swagger ● Train, explore, and deploy with a single tool ● CLI access ● Static typing
- 16. ML For Humans Enabling integration with non-ml experts: Developer perspective ● REST access ○ Ease of integration ● Ease of deployment ○ Containers not just an afterthought ● Visibility: discover shapes of inputs needed, shapes of outputs +
- 17. ML For Humans Enabling integration with non-ml experts: Data Scientist perspective from racket import KerasLearner from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense from tensorflow.keras.callbacks import TensorBoard class KerasModel(KerasLearner): VERSION = '1.1.1' MODEL_TYPE = 'regression' MODEL_NAME = 'keras-simple-regression' def build_model(self): optimizer = tf.train.RMSPropOptimizer(0.001) model = Sequential() model.add(Dense(24, input_dim=5, kernel_initializer='normal', activation='relu')) model.add(Dense(48, kernel_initializer='normal', activation='relu')) model.add(Dense(1, kernel_initializer='normal')) model.compile(loss='mean_absolute_error', optimizer=optimizer) return model def fit(self, x, y, x_val=None, y_val=None, epochs=2, batch_size=20): self.model.fit(x, y, epochs=epochs, verbose=0, validation_data=(x_val, y_val)) if __name__ == '__main__': X_train, X_test, y_train, y_test = train_test_split(x, y) kf = KerasModel() kf.fit(X_train, y_train, x_val=X_test, y_val=y_test) kf.store(autoload=True) ● Define model, rest is taken care of ● Scoring done automatically ○ Including user-defined/multiple metrics ● Access to previous runs’ data ● Multiple models & versions
- 18. Quick demo Demo link: https://asciinema.org/a/xxoebEfyu1bzO84hWWAams577 Docs: https://r-racket.readthedocs-hosted.com/en/latest/ Code: https://github.com/carlomazzaferro/racket
- 19. Moving Forward ● Needs ○ Support for batching requests ○ Better testing ○ Deployment guide ■ Kubernetes & co. ○ Feature parity on CLI vs REST access
- 20. Questions ?