Stories from the Financial Service AI Trenches: Lessons Learned from Building AI Models in EY
0 likes356 views
The document discusses lessons learned from building AI models in the financial services industry. It highlights two main challenges - data is often scarce or restricted due to privacy concerns, and data comes from multiple jurisdictions with different regulations. It provides examples of using synthetic data and country-specific iterative modeling to address these challenges. The key takeaways are that rich yet difficult to acquire data is appealing but challenging for AI in financial services, and multi-jurisdiction modeling requires an end-to-end MLOps framework to develop machine learning at scale.
Stories from the Financial Service AI Trenches: Lessons Learned from Building AI Models in EY
- 1. Stories from the Financial Service AI Trenches Lessons learned from building AI models in EY 18 November 2020 Tim Santos, Assistant Director, Client Technology AI Mustafa Somalya , Assistant Director , Client Technology AI
- 2. 18 November 2020Page 2 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY 1 AI in Financial Services Overview 2 Use Cases and Learnings Agenda
- 3. 18 November 2020Page 3 AI in Financial Services How does an experiment-driven disruptive technology such as AI look like in a highly-regulated industry? Sources: https://www.fca.org.uk/publication/research/research-note-on-machine-learning-in-uk-financial-services.pdf https://ec.europa.eu/digital-single-market/en/high-level-expert-group-artificial-intelligence http://rms.koenig-solutions.com/Sync_data/Trainer/QMS/1752-2020328106-AuditingArtificialIntelligencereseng1218(1).pdf Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
- 4. 18 November 2020Page 4 Review data sourcing, profiling, processing, as well as data quality and ethical issues Assess approach and models are fit for purpose, explainable, reproducible, and robust, with supporting evidence Confirm outcomes achieve desired level of precision and consistency, and are aligned with ethical, lawful, and fair design criteria Ensure solution is scalable and deployable with the right tech infrastructure, and continuously monitored Ensure business purpose, governance and stakeholder engagement are properly identified and aligned Solution Lifecycle Modelling Outcome Analysis Deployment and Monitoring Data and Processing Business and Governance Source: https://www.ukfinance.org.uk/system/files/Trust%2C%20Context%20and%20Regulation%20-%20Achieving%20more%20explainable%20AI%20in%20financial%20services.pdf Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY AI in Financial Services How does an experiment-driven disruptive technology such as AI look like in a highly-regulated industry?
- 5. How do you train models for rich yet highly restricted data that could be difficult to acquire? 18 November 2020Page 5 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
- 6. Use Case : Know Your Customer (KYC) Page 6 KYC Social Network Employment Information Self-certification Forms Biometric Data Legal Documents Open Banking Proof of Identity Digital Footprint KYC requires a lot of time consuming repetitive manual work that involves the processing of a variety of data sources. Ubiquity, variety of data sources, and complexity involved in cognitive tasks make it a very attractive use case for AI. Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY18 November 2020
- 7. Use Case : Know Your Customer (KYC) Page 7 Form Field Detection 2 Handwritten Text Recognition 3 Data Synthesis 1 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY18 November 2020
- 8. Use Case : Know Your Customer (KYC) ► Data is scarce or highly restricted because of sensitive and personally-identifiable information ► SDLC and DevOps can be inadequate for ML development, consider MLOps ► Treat the scarcity of data as a technological and scientific problem ► When using synthetic or generic datasets, ensure that there’s a feedback mechanism for when live data becomes available 22 November 2020 Presentation titlePage 8
- 9. How do you develop models when data from clients come from different geographies, have different legislations and cross- border restrictions? 18 November 2020Page 9 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
- 10. Use Case: Multi-Jurisdiction Models Generic models and pipelines are reused, iterations produce bespoke models by incorporating country-specific data 22 November 2020 Presentation titlePage 10 Reusable Components Standard ML Pipeline: Base Model ► Common laws and treaties ► Similar industry trends and treatments ► Transactional trends ► Language models ► Common data model ► Generic dataset ► Regional market ► Cross regional market ► National market Base Model Country X Country Y Model Y v1Model X v2Model X v1 Model X v3 Model Y v2 retrain increment Country Z retrain Model Z v1 Model Y v3 MODELS XYZ ML Pipeline Iteration XYZ: Bespoke Model ► Hyperparameter Tuning ► Country-specific datasets and enrichment ► Additional categories and features
- 11. 18 November 2020Page 11 Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY Multi-Jurisdiction Models Example – MLOps and AutoML Modelling Outcome Analysis Deployment and MonitoringData and Processing Data Prep Exploratory Data Analysis Feature Engineering Feature Selection Benchmark Metrics Model Serving (Inference) Drift Monitoring (inference) Model Build and AutoML Pipeline Hyperparameter Tuning Training CI/CD Model Serving (Train Pipeline) Retraining/ Rollback / Increment Data slicing Model Serving (Training Pipeline) Experimentation Feature Importance Drift Monitoring (Training) Model Serving (Inference) model is stale make predictions Human in the loop Consume/ Interface High confidence Low confidence Model Drift Monitoring (Data Signature) model is good • Create training (baseline) and inference dataset signatures from features • Create signatures from predictions, also called theories • Measure the distance of signatures • Population Stability Index : 𝑃𝑆𝐼 = ∑!(𝐴! − 𝐵!) ln "! #! {𝐴!, 𝐵! − 𝑓𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 𝑐𝑜𝑢𝑛𝑡 𝑝𝑒𝑟 𝑓𝑒𝑎𝑡𝑢𝑟𝑒 𝑏𝑢𝑐𝑘𝑒𝑡} • Numerical Parametric (should pass normality, homoskedasticity): T-test • Numerical Non-parametric: Kruskal-Wallis, Wilcoxon, Kolmogorov, Mann-Whitney-U • Categorical Features and theory testing: Pearson’s Chi-squared test • Provide pass/warning/fail logic to trigger retraining, rollback, AutoML, reinforcement learning Training Dataset Inference Dataset Model features features predictions Inference Signature Score (Distance) Training Signature Data Augmentation
- 12. Transfer learning and Model Finetuning 18 November 2020Page 12 Use Case: Multi-Jurisdiction Models Data drift monitoring and MLOps tools Reproducible end-to-end ML pipelines and AutoML Leveraging “human in the loop” with MLOps framework and online learning Enabling components for Multi-Jurisdiction and ML at scale Time from Technical and Business SMEs are valuable, a complementing operating model and tooling would be necessary to maximise value Building and deploying bespoke models for each jurisdiction is difficult to scale without an end-to-end MLOps platform Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
- 13. 18 November 2020Page 13 The appeal of using AI in FS lies in very rich data, the same reason that makes data very challenging to acquire. AI in FS usually involve clients in multiple jurisdictions, it is imperative to have MLOps framework and platform to develop ML at scale. Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY Key Takeaways
- 14. Q&A 18 November 2020Page 14 Tim Santos Assistant Director | Global IT ► Global Client Technology AI ► MLOps Lead ► [email protected] Mustafa Somalya Assistant Director | Global IT ► Global Client Technology AI ► ML Experimentation Lead ► [email protected] Stories from the Financial Service AI Trenches: Lessons learned from building AI models in EY
- 15. EY | Assurance | Tax | Strategy and Transactions | Consulting About EY EY is a global leader in assurance, tax, strategy, transaction and consulting services. The insights and quality services we deliver help build trust and confidence in the capital markets and in economies the world over. We develop outstanding leaders who team to deliver on our promises to all of our stakeholders. In so doing, we play a critical role in building a better working world for our people, for our clients and for our communities. EY refers to the global organization, and may refer to one or more, of the member firms of Ernst & Young Global Limited, each of which is a separate legal entity. Ernst & Young Global Limited, a UK company limited by guarantee, does not provide services to clients. Information about how EY collects and uses personal data and a description of the rights individuals have under data protection legislation are available via ey.com/privacy. For more information about our organization, please visit ey.com. This news release has been issued by EYGM Limited, a member of the global EY organization that also does not provide any services to clients. © 2020 EYGM Limited. All Rights Reserved. EYG no. ED MMYY This material has been prepared for general informational purposes only and is not intended to be relied upon as accounting, tax or other professional advice. Please refer to your advisors for specific advice.