2019 GDRR: Blockchain Data Analytics - QuTrack: Model Life Cycle Management for AI and ML models using a Blockchain approach - Srikanth Krishnamurthy, October 7, 2019
2 likes438 views
The document presents Qutrack, a blockchain-based approach to model lifecycle management, addressing challenges in machine learning workflows such as reproducibility and the tracking of complex processes. It discusses various elements like data ingestion, model evaluation, and AI governance, emphasizing the importance of metadata and experiment lineage for reproducible pipelines. Additionally, it highlights integration with existing frameworks and tools to enhance model management and provide secure access to archived pipelines.
2019 GDRR: Blockchain Data Analytics - QuTrack: Model Life Cycle Management for AI and ML models using a Blockchain approach - Srikanth Krishnamurthy, October 7, 2019
- 1. QuTrack: A Blockchain-based approach to Model Lifecycle Management 2019 Copyright QuantUniversity LLC. Presented By: Sri Krishnamurthy [email protected] • Northeastern University • QuantUniversity : A NEU IDEAS startup
- 2. 2 • Model Life-cycle Management challenges • Approaches • QuTrack: A Blockchain-based approach to Model Lifecycle Management • Demo Agenda
- 3. 3
- 4. Machine Learning Workflow Data Scraping/ Ingestion Data Exploration Data Cleansing and Processing Feature Engineering Model Evaluation & Tuning Model Selection Model Deployment/ Inference Supervised Unsupervised Modeling Data Engineer, Dev Ops Engineer Data Scientist/QuantsSoftware/Web Engineer • AutoML • Model Validation • Interpretability Robotic Process Automation (RPA) (Microservices, Pipelines ) • SW: Web/ Rest API • HW: GPU, Cloud • Monitoring • Regression • KNN • Decision Trees • Naive Bayes • Neural Networks • Ensembles • Clustering • PCA • Autoencoder • RMS • MAPS • MAE • Confusion Matrix • Precision/Recall • ROC • Hyper-parameter tuning • Parameter Grids Risk Management/ Compliance(All stages) Analysts& DecisionMakers
- 5. 5 • Developing ML applications involves: ▫ Heuristics ▫ Best practices (templates) ▫ Lots of experimentation ▫ Many moving pieces ▫ No “right” answer ▫ Error creep: Even small untracked errors can through off results AI/ML application development => Design of Experiments
- 6. 6 Source: Sculley et al., 2015 "Hidden Technical Debt in Machine Learning Systems" Challenges
- 8. 8 • Repeatability (Same team, same experimental setup) ▫ The measurement can be obtained with stated precision by the same team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same location on multiple trials. For computational experiments, this means that a researcher can reliably repeat her own computation. • Replicability (Different team, same experimental setup) ▫ The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author’s own artifacts. • Reproducibility (Different team, different experimental setup) ▫ The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently. Repeatable or Reproducible or Replicable https://www.acm.org/publications/policies/artifact-review-badging
- 10. 10 Elements of Model Risk Management
- 11. 11 AI Governance is gaining focus https://legalinstruments.oecd.org/ en/instruments/OECD-LEGAL-0449
- 12. 12 AI Governance is gaining focus https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449
- 13. 13 NLP pipeline Data Ingestion from Edgar Pre-Processing Invoking APIs to label data Compare APIs Build a new model for sentiment Analysis Stage 1 Stage 2 Stage 3 Stage 4 Stage 5 • Amazon Comprehend API • Google API • Watson API • Azure API
- 14. 14 • Programming environment • Execution environment • Hardware specs • Cloud • GPU • Dependencies • Lineage/Provenance of individual components • Model params • Hyper parameters • Pipeline specifications • Model specific • Tests • Data versions Data Model EnvironmentProcess Components that needs to be tracked
- 15. 15 Source: T. van derWeide, O. Smirnov, M. Zielinski, D. Papadopoulos, and T. van Kasteren. Versioned machine learning pipelines for batch experimentation. In ML Systems, Workshop NIPS 2016, 2016. Provenance and Lineage of pipelines
- 16. 16 Schemas proposed Sebastian Schelter, Joos-Hendrik Boese, Johannes Kirschnick, Thoralf Klein, and Stephan Seufert. Automatically Tracking Metadata and Provenance of Machine Learning Experiments. NIPS Workshop on Machine Learning Systems, 2017.
- 17. 17 Schemas proposed G. C. Publio, D. Esteves, and H. Zafar, “ML-Schema : Exposing the Semantics of Machine Learning with Schemas and Ontologies,” in Reproducibility in ML Workshop, ICML’18, 2018.
- 18. 18 MLFlow
- 19. 19 DVC
- 20. 20 GoCD
- 21. 21
- 22. 22 I. Altintas, O. Barney, and E. Jaeger-Frank. Provenance collection support in the Kepler scientific workflow system. In Provenance and annotation of data, pages 118–132. Current approaches
- 23. 23 Miao, Hui & Chavan, Amit & Deshpande, Amol. (2016). ProvDB: A System for Lifecycle Management of Collaborative Analysis Workflows. Current approaches
- 24. 24 Related work Xueping Liang, Sachin Shetty, Deepak Tosh, Charles Kamhoua, Kevin Kwiat, and Laurent Njilla. 2017. ProvChain: A Blockchain-based Data Provenance Architecture in Cloud Environment with Enhanced Privacy and Availability. In Proceedings of the 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid '17). IEEE Press, Piscataway, NJ, USA, 468-477. DOI: https://doi.org/10.1109/CCGRID.2017.8 Focus on Cloud data provenance using Blockchain
- 25. 25 Related work Ramachandran, Aravind & Kantarcioglu, Dr. (2017). Using Blockchain and smart contracts for secure data provenance management. DataProv: Built on top of Ethereum, the platform utilizes smart contracts and open provenance model (OPM) to record immutable data trails.
- 26. 26 Related work Sarpatwar, Kanthi & Vaculín, Roman & Min, Hong & Su, Gong & Heath, Terry & Ganapavarapu, Giridhar & Dillenberger, Donna. (2019). Towards Enabling Trusted Artificial Intelligence via Blockchain. 10.1007/978-3-030-17277-0_8. Trusted AI and provenance of AI models
- 27. 27
- 28. 28 QuSandbox research suite Model Analytics Studio QuSandboxQuTrack QuResearchHub Prototype, Iterate and tune Standardize workflowsProductionize and share Track experiments
- 29. 29 The four components that need to be encapsulated for reproducible pipelines Code Data Environment Process
- 30. 30 QuSandbox
- 31. 31
- 33. 33 • JDF: Job Definition File; A DSL for representing Model Pipelines • Stage • Entity ▫ Model ▫ Data ▫ Environment • Version format ▫ M:m:p -> Major Version: Minor Version: Patch Terms
- 34. 34 JDF- DSL
- 35. 35 QuResearchHub
- 36. 36 • Ability to track the evolution of experiments • Snapshot and store the lineage of pipelines • Ensure auditability and secure access to archived pipelines • Track experiments and their associated parameters • Track all aspects of experiments to ensure reproducibility • For high-impact (regulated, critical) applications, ensure experiment traces are immutable and verifiable QuTrack Design requirements
- 37. 37 Metadata • Data about the information to be tracked • Includes version number, timestamps, user information, MD5 of the artifacts and high-level notes Data • Pipelines, custom DSL, standard formats for representing models • Events (Updates, rollbacks • JSON, Amazon ION, YAML, Artifacts • Model Pickle files, ONYX, COREML, Model params • Data, blobs etc. Architecture : What’s tracked ?
- 38. 38 Blockchain-based: • QLDB • Ethereum Non-Blockchain-based: • MongoDB Architectures supported
- 39. 39 Amazon Quantum Ledger Database (QLDB) • Fully managed ledger database that provides a transparent, immutable, and cryptographically verifiable transaction log. • SQL-like API • Cost-effective Amazon QLDB
- 40. 40 Amazon QLDB
- 43. 43
- 44. 44 • Support for ONYX, CoreML • Integration with: ▫ MLFlow, DVC, GoCD • Integration with SCM systems ▫ Github, SVM • Tracking Back tests • Push Architecture -> Event-Driven Architecture • Enriched Analytics • Roles and Authorization Future work
- 45. Thank You! If you are interested in trying out QuSandbox, Please sign up for updates at: www.qusandbox.com Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC. 45