"Managing the Complete Machine Learning Lifecycle with MLflow"
1 like2,554 views
Machine Learning development brings many new complexities beyond the traditional software development lifecycle. Unlike in traditional software development, ML developers want to try multiple algorithms, tools, and parameters to get the best results, and they need to track this information to reproduce work. In addition, developers need to use many distinct systems to productionize models. To address these problems, many companies are building custom “ML platforms” that automate this lifecycle, but even these platforms are limited to a few supported algorithms and to each company’s internal infrastructure. In this session, we introduce MLflow, a new open-source project from Databricks that aims to design an open ML platform where organizations can use any ML library and development tool of their choice to reliably build and share ML applications. MLflow introduces simple abstractions to package reproducible projects, track results, and encapsulate models that can be used with many existing tools, accelerating the ML lifecycle for organizations of any size.
Ad
More Related Content
What's hot (20)
Similar to "Managing the Complete Machine Learning Lifecycle with MLflow" (20)
Ad
More from Databricks (20)
Ad
Recently uploaded (20)
![Download Wondershare Filmora Crack [2025] With Latest](https://cdn.slidesharecdn.com/ss_thumbnails/neo4j-howkgsareshapingthefutureofgenerativeaiatawssummitlondonapril2024-240426125209-2d9db05d-250419-250428115407-a04afffa-thumbnail.jpg?width=560&fit=bounds)
![Pixologic ZBrush Crack Plus Activation Key [Latest 2025] New Version](https://cdn.slidesharecdn.com/ss_thumbnails/fashionevolution2-250322112409-f76abaa7-250428124909-b51264ff-250504160528-fc2bb1c5-thumbnail.jpg?width=560&fit=bounds)
"Managing the Complete Machine Learning Lifecycle with MLflow"
- 1. Montreal| March 27, 2020 Platform for Complete Machine Learning Lifecycle Jules S. Damji @2twitme PyData Montreal
- 2. Apache Spark Developer & Community Advocate @ Databricks Developer Advocate @ Hortonworks Software engineering @ Sun Microsystems, Netscape, @Home, Excite@Home, VeriSign, Scalix, Centrify, LoudCloud/Opsware, ProQuest Program Co-chair Spark + AI Summit https://www.linkedin.com/in/dmatrix @2twitme $ whoami
- 3. Accelerate innovation by unifying data science, engineering and business to solve data problems • Original creators of • 2000+ global companies use our platform across big data & machine learning lifecycle VISION WHO WE ARE Unified Data Analytics PlatformSOLUTION Koalas
- 4. Outline • Overview of ML development challenges • How MLflow tackles these • MLflow Components • MLflow Tracking, Projects, Models & Registry • Managed MLflow Demo • Q & A
- 5. Machine Learning Development is Complex
- 6. Traditional Software Machine Learning Goal: Optimize a metric (e.g., accuracy) • Constantly experiment to improve it Quality depends on input data and tuning parameters Compare + combine many libraries, models & algorithms for the same task Goal: Meet a functional specification Quality depends only on code Typically pick one software stack
- 7. ML Lifecycle 7 Delta Data Prep Training Deploy Raw Data μ λ θ Tuning Scale μ λ θ Tuning Scale Scale Scale Model Exchange Governance
- 8. Custom ML Platforms Some Big Data Companies +Standardize the data prep / training / deploy loop: if you work with the platform, you get these! –Limited to a few algorithms or frameworks –Tied to one company’s infrastructure –Out of luck if you left the company…. Can we provide similar benefits in an open manner?
- 9. Introducing Open machine learning platform • Works with popular ML library & language • Runs the same way anywhere (e.g., any cloud or locally) • Designed to be useful for 1 or 1000+ person orgs • Simple. Modular.Easy-to-use. • Offers positive developer experience to get started!
- 10. 10 “API-first” Modular design MLflow Design Philosophy • Allow different components individually (e.g., use MLflow’s project format but not its deployment tools) • Not monolithic • But Distinctive and Selective Key enabler: distinct components (Tracking/Projects/Models/Registry) • Submit runs, log models, metrics, etc. from popular library & language • Abstract “model” lambda function that MLflow can then deploy in many places (Docker, Azure ML, Spark UDF) • Open interface allows easy integration from the community Key enabler: built around Programmatic APIs, REST APIs & CLI
- 11. MLflow Components Tracking Record and query experiments: code, data, config, and results Projects Package data science code in a format that enables reproducible runs on any platform Models Deploy machine learning models in diverse serving environments environments Model Registry Store, annotate and manage models in a central repository new mlflow.org github.com/mlflow twitter.com/MLflow databricks.com/ mlflow
- 12. Key Concepts in MLflow Tracking Parameters: key-value inputs to your code Metrics: numeric values (can update over time) Tags and Notes: information about a run Artifacts: files, data, and models Source: what code ran? Version: what of the code?
- 13. Model Development without MLflow data = load_text(file) ngrams = extract_ngrams(data, N=n) model = train_model(ngrams, learning_rate=lr) score = compute_accuracy(model) print(“For n=%d, lr=%f: accuracy=%f” % (n, lr, score)) pickle.dump(model, open(“model.pkl”)) For n=2, lr=0.1: accuracy=0.71 For n=2, lr=0.2: accuracy=0.79 For n=2, lr=0.5: accuracy=0.83 For n=2, lr=0.9: accuracy=0.79 For n=3, lr=0.1: accuracy=0.83 For n=3, lr=0.2: accuracy=0.82 For n=4, lr=0.5: accuracy=0.75 ... What if I expand the input data? What if I tune this other parameter? What if I upgrade my ML library? What version of my code was this result from?
- 14. MLflow Tracking API: Simple & Pythonic! 14 Tracking Record and query experiments: code, configs, results, …etc import mlflow import mflow.tensorflow # log model’s tuning parameters with mlflow.start_run() as run: mlflow.log_param("layers", layers) mlflow.log_param("alpha", alpha) # log metrics and model mlflow.log_metric("mse", model.mse()) mlflow.log_artifact("plot", model.plot(test_df)) mlflow.tensorflow.log_model(model)
- 15. $ mlflow ui Model Development with MLflow is Simple! data = load_text(file) ngrams = extract_ngrams(data, N=n) model = train_model(ngrams, learning_rate=lr) score = compute_accuracy(model) with mlflow.start_run() as run: mlflow.log_param(“data_file”, file) mlflow.log_param(“n”, n) mlflow.log_param(“learn_rate”, lr) mlflow.log_metric(“score”, score) mlflow.sklearn.log_model(model) Track parameters, metrics, output files & code version Search using UI or API
- 16. Notebooks Local Apps Cloud Jobs MLflow Tracking Python, Java, R or REST API $ export MLFLOW_TRACKING_URI <URI> mlflow.set_tracking_uri(URI) UI API Tracking Server Parameters Metrics Artifacts ModelsMetadata Spark Data Source
- 17. MLflow Tracking Backend Stores 1. Entity (Metadata) Store • FileStore (local filesystem) • SQLStore (via SQLAlchemy) • PostgreSQL, MySQL, SQLlite 2. Artifact Store • S3 backed store • Azure Blob storage • Google Cloud storage • DBFS artifact repo
- 18. MLflow Components Tracking Record and query experiments: code, data, config, and results Projects Package data science code in a format that enables reproducible runs on any platform Models Deploy machine learning models in diverse serving environments Model Registry Store, annotate and manage models in a central repository new mlflow.org github.com/mlflow twitter.com/MLflow databricks.com/ mlflow
- 19. MLflow Projects Motivation Diverse set of tools Diverse set of environments Challenge: ML results difficult to reproduce Projects Package data science code in a format that enables reproducible runs on any platform
- 20. Project Spec Code Data Config Local Execution Remote Execution MLflow Projects Dependencies
- 21. Example MLflow Project my_project/ ├── MLproject │ │ │ │ │ ├── conda.yaml ├── main.py └── model.py ... conda_env: conda.yaml entry_points: main: parameters: training_data: path lambda: {type: float, default: 0.1} command: python main.py {training_data} {lambda} $ mlflow run git://<my_project> -P lambda=0.2 mlflow.run(“git://<my_project>”, ...) mlflow run . –e main –P lambda=0.2
- 22. Example MLflow Project my_project/ ├── MLproject │ │ │ │ │ ├── conda.yaml ├── main.py └── model.py ... channels: - defaults dependencies: - python=3.7.3 - scikit-learn=0.20.3 - pip: - mlflow - cloudpickle==0.8.0 name: mlflow-env
- 23. MLflow Components Tracking Record and query experiments: code, data, config, and results Projects Package data science code in a format that enables reproducible runs on any platform Models Deploy machine learning models in diverse serving environments Model Registry Store, annotate and manage models in a central repository new mlflow.org github.com/mlflow twitter.com/MLflow databricks.com/ mlflow
- 24. Inference Code Batch & Stream Scoring Serving Tools MLflow Model Motivation ML Frameworks
- 25. Model Format Flavor 2Flavor 1 ML Frameworks Inference Code Batch & Stream Scoring Serving Tools Standard for ML models MLflow Models
- 27. Example MLflow Model my_model/ ├── MLmodel │ │ │ │ │ └── estimator/ ├── saved_model.pb └── variables/ ... Usable by tools that understand TensorFlow model format Usable by any tool that can run Python (Docker, Spark, etc!) run_id: 769915006efd4c4bbd662461 time_created: 2018-06-28T12:34 flavors: tensorflow: saved_model_dir: estimator signature_def_key: predict python_function: loader_module: mlflow.tensorflow mlflow.tensorflow.log_model(...)
- 28. Model Flavors Example Train a model mlflow.keras.log_model(…) Model Format Flavor 1: Pyfunc Flavor 2: Keras predict = mlflow.pyfunc.load_model(…) predict(pandas.input_dataframe) model = mlflow.keras.load_model(…) model.predict(keras.Input(…))
- 29. Model Flavors Example predict = mlflow.pyfunc.load_model(model_uri) predict(pandas.input_dataframe)
- 30. MLflow Components Tracking Record and query experiments: code, data, config, and results Projects Package data science code in a format that enables reproducible runs on any platform Models Deploy machine learning models in diverse serving environments Model Registry Store, annotate and manage models in a central repository new mlflow.org github.com/mlflow twitter.com/MLflow databricks.com/ mlflow
- 31. The Model Management Problem When you’re working on one ML app alone, storing your models in files is manageable MODEL DEVELOPER classifier_v1.h5 classifier_v2.h5 classifier_v3_sept_19.h5 classifier_v3_new.h5 …
- 32. The Model Management Problem When you work in a large organization with many models, many data teams, management becomes a major challenge: • Where can I find the best version of this model? • How was this model trained? • How can I track docs for each model? • How can I review models? MODEL DEVELOPER REVIEWER MODEL USER ???
- 33. Model Registry VISION: Centralized and collaborative model lifecycle management Tracking Server Parameters Metrics Artifacts Staging Production Archived ModelsMetadata Model Registry Data Scientists Deployment Engineers
- 34. MLflow Model Registry Repository of named, versioned models with comments & tags Track each model’s stage: none, staging, production, or archived Easily load a specific version Provides Model lineage & activities
- 35. Model Registry Workflow API Model Registry MODEL DEVELOPER DOWNSTREAM USERS AUTOMATED JOBS REST SERVING REVIEWERS, CI/CD TOOLS mlflow.register_model("runs:/e148278107274..832/artifacts/model", " WeatherForecastModel") client = mlflow.tracking.Mlflowclient() client.transition_model_version_stage(name=”WeatherForecastModel”, version=5, stage= "Production") model_uri= "models:/{model_name}/production".format( model_name=" WeatherForecastModel") model_prod = mlflow.sklearn.load_model(model_uri)
- 36. What Did We Talk About? Modular Components greatly simplify the ML lifecycle • Available APIs: Python, Java & R (Soon Scala) • Easy to install and use • Develop & Deploy locally and track locally or remotely • Visualize experiments and compare runs • Centrally register and mange model lifecycle
- 37. 0 20 40 60 80 100 120 140 0 6 12 18 24 30 36 42 #ofContributors Months since Project Launch Project Contributors over Time MLflow Apache Spark Today at 179 contributors
- 38. Learning More About MLflow • pip install mlflow to get started • Find docs & examples at mlflow.org • https://github.com/mlflow/mlflow • tinyurl.com/mlflow-slac • dbricks.co/mlflow-tutorials 38
- 39. MLflow Demo Fun & Fake Demo Serious & Real Demo
- 40. Merci Montreal J Q & A [email protected] @2twitme https://www.linkedin.com/in/dmatrix/