Reproducibility and experiments management in Machine Learning
0 likes118 views
Machine Learning becomes more and more common practice in many companies. ML teams size is growing and collaboration goes out of office and personal laptops. The complexity of ML projects leads to adopting distributed team collaboration, cloud based infrastructure and distributed machine learning. Well defined and manageable process for ML experiments becomes a central issue. Practices to apply automated pipelines, models and data set versioning helps to establish a good manageable process in project and provide reproducible results. This speech helps to start with handling models and datasets versioning using open source tools: DVC, mlflow, Luigi, etc.
![ML reproducibility is a dimension of quality
9
What is Reproducibility?
● using the original methods applied to
the original data to produce the
original results [Gardner]
Why should you care?
● Trust
● Consistent Results
● Versioned History
● Team Performance
● Pain Less Production
Josh Gardner, Yuming Yang, Ryan S. Baker, Christopher Brooks. Enabling End-To-End Machine
Learning Replicability: A Case Study in Educational Data Mining](https://image.slidesharecdn.com/rozhkovreproducibilityandexperimentsmanagementinmlprojectsv42019-04-26-190728080053/85/Reproducibility-and-experiments-management-in-Machine-Learning-9-320.jpg)
Ad
More Related Content
Similar to Reproducibility and experiments management in Machine Learning (20)
More from Mikhail Rozhkov (15)
Ad
Recently uploaded (20)
Ad
Reproducibility and experiments management in Machine Learning
- 1. Simple. Precise. Competent. Reproducibility and experiments management in ML projects D1 EXPO 2019, Alicante Rozhkov Mikhail Senior Data Scientist Raiffeisen Bank Russia
- 2. Topics • Difference of ML projects from IT projects • ML experiments management • Agile ML • ML reproducibility and why it’s important? • Approaches and tools 2
- 3. 1. Different from IT projects 2. Longer dev cycle 3. Experiments driven 4. Not easy to test and validate 3 ML projects
- 4. IT projects development processes 4 Source: https://online.husson.edu/software-development-cycle/ Source: https://en.wikipedia.org/wiki/Software_development_process
- 5. ML project workflow is experiments driven 5 Problem Statement MVP design Get data Prepare data Train model Evaluate modelTest & Integrate Serve / Predict Monitor 1. Analyze & Plan 2. Prototype 4. Monitor & Maintain 3. Productionize Inspired by Uber’s workflow of a machine learning project diagram. Scaling Machine Learning at Uber with Michelangelo https://eng.uber.com/scaling-michelangelo/ Solution development
- 6. Experiment = code + dataset + outputs 6 Algorithm Data Hyperpara meters Evaluation Measure Model ETL tasks test dataset train dataset evaluate train Experiment config - artifacts - pipelines - code - configs
- 7. ML project requires more factors to take into account 7 Software ML Architecture design tasks, UI/UX integrations + nature and quality of data Quality measures working code + model quality metrics + performance in production Version control code environment + pipelines + datasets + models & artifacts Testing code + data and features + model development methods + ML infrastructure + ML systems Inspired by Dmitry Petrov, Ivan Shcheklein. Open source tools for machine learning model and dataset versioning.
- 8. 1. Satisfy customer 2. Early fail 3. Fail safe 4. Frequent code updates 5. Constant changing requirements 6. Team = Business + DS/ML 7. Frequent team meetings/statuses 8. Measure of progress=working code 9. Technical excellence and good design 10. Reproducibility 8 Agile ML Inspired by: Andrew Kelleher, Adam Kelleher. Machine Learning in Production: Developing and Optimizing Data Science Workflows and Applications. 2019
- 9. ML reproducibility is a dimension of quality 9 What is Reproducibility? ● using the original methods applied to the original data to produce the original results [Gardner] Why should you care? ● Trust ● Consistent Results ● Versioned History ● Team Performance ● Pain Less Production Josh Gardner, Yuming Yang, Ryan S. Baker, Christopher Brooks. Enabling End-To-End Machine Learning Replicability: A Case Study in Educational Data Mining
- 10. ● Code, models and data version control ● Automated pipelines ● Tests ● Control environment ● Experiments management ● Methodology and procedures documentation 10 Reproducible ML
- 11. 11 Use case
- 12. Onsite Recommendation System 12 Purpose: improve conversion rate on landing page send online user data get promo User History DataCV prediction model Promo recommendati on model Promo DB {uid, cv_pred, promo_id}
- 13. Tracking project statuses and issues, documentation 13
- 14. Code 14 ● Version control ● Re-usable .py modules ● Tests... Source: https://www.bitbull.it/en/blog/how-git-flow-works/
- 15. Data and artifacts 15 ● Version Control ● Store / share ● Access
- 16. Pipelines 16 ● One button run ● End-to-end or selected steps ● Configs (i.e. random seeds) load/ transform raw data evaluate train split train/test prepare train dataset select best model prepare test dataset predict
- 17. Experiments Management 17 ● Browse history ● Compare results ● Share results ● Methodology and procedures Data Model pipelines MetricsConfig pipelines pipelines Data Model pipelines MetricsConfig pipelines pipelines experiment X (dd.mm.2018) experiment Y (dd.mm.2019)
- 18. Environment 18 ● Libraries ● OS ● Hardware Gronenschild EHBM, Habets P, Jacobs HIL, Mengelers R, Rozendaal N, et al. (2012) The Effects of FreeSurfer Version, Workstation Type, and Macintosh Operating System Version on Anatomical Volume and Cortical Thickness Measurements. PLOS ONE 7(6): e38234. https://doi.org/10.1371/journal.pone.0038234 https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0038234 Example: Effects of data processing conditions on the voxel volumes for a subsample of (sub)cortical structures
- 19. Test ML ● Tests for data and features ● Tests for model development ● Tests for ML infrastructure ● Test for running ML systems 19
- 20. Conclusions 1. ML projects requires different approach 2. Data and experiments are crucial 3. Agility is driven by fast experimenting and reproducibility 4. Experiments are versioned, browsable, comparable, documented, reproducible 5. Reproducibility is a dimension of quality and maturity 20
- 21. raiffeisen.ru Simple. Precise. Competent. Thank you 21 Rozhkov Mikhail Senior Data Scientist Raiffeisen Bank Russia [email protected]