Building an AI Assistant Factory - Rasa Summit 2021
0 likes151 views
The document outlines the journey of implementing AI technology in a major Canadian bank, detailing various initiatives from 2018 to 2020 to enhance chatbot capabilities. It highlights the establishment of a standardized dialog technology using the Rasa platform, the creation of an AI factory team, and the bank's unique expertise in dialog technology. The document emphasizes the importance of operational excellence and DevOps practices in successfully deploying enterprise-level chatbots.
Building an AI Assistant Factory - Rasa Summit 2021
- 1. Building an AI Assistant Factory Dominique Boucher, Ph.D. Chief Solutions Architect Eric Charton, Ph.D. Senior AI Director
- 2. 2 A journey on the introduction of a state of the art technology in one of the oldest bank of Canada https://www.bnc.ca/fr/a-propos-de-nous/notre-organisation/la-banque/histoire.html
- 3. 3 History : Changing paradigm, making things differently & creating a new mindset January 2018 : • 9 bots projects • 9 different technologies • 9 different vendors • No performances evaluation Summer 2018 : proposal to standardize dialog technology using RASA Platform – Proof of concept FAQ BNC September 2018 : - Proof of concept of FAQ Bots for online banking - Creation of a specific algorithm to generate FAQ dialog system from FAQ Q3 2019 à Q4 2020: R&D work on robustness (paraphrasing, disambiguation) December 2018: Patent registration System and methods for performing automated interactive conversation with a user Poor technology gives poor satisfaction November 2018 : Multiple Proof of concepts using the technology : legal bot, acquisition tools … June 2019 : platform integration using RASA and creation of AI Factory team February 2020 : Online Banking Chatbot during the COVID pandemic April 2020 : pre-login chatbot Q1 2020: R&D work on automation for model update (Ongoing) November 2020 : Executive presentation for a bank wide dialog systems strategy December 2020 : Validation of the strategy
- 4. Leader in dialog technology applied to banking industry in Canada Unique chatbots expertise in Canada banking environment: • One patent on disambiguation (unique patent on dialog by a bank in Canada) • Innovative technologies published in 4 tier one AI and NLP conferences • Highly qualified team of Dialog specialists qualified in NLP and Computational Linguistic (Ph.D, Masters) • Dialog Platform fully based on open source • In house Analytic solution to handle the full life cycle and maintain the highest level of robustness Conference communications CompSci publications Patents
- 5. 5 The solution : technological components on catalog supported by 2 teams Dialog engine Cloud architecture, multi canal Reporting system to measure performances and collect feedback Log export in the analytic system with user ID to make deep analysis Gateways to expose the dialog engine on various mediums (web, online banking, TEAM …) Automatic model generation tools (based on question answering) AI Practice: Product Ownership, includes Business analysis, model design, technological innovations, model update automation, bank wide strategy AI Factory (IT): Solution integration, industrialization, implementation of technologies, operational support Automatic model update New channels Other's components in preparation Available components
- 6. 6 Chat Bot lifecycle Experimental model Proof of concept: • Measures • Tuning • Demos Cloud Logs generation - Data Collection - Business analysis Production version deployment Performance's tracking Feedback from users' interactions Updating production model - Publishing Development Production • During the COVID Crisis, time to market for changes in the model of 72 hours • From 100 questions to 324 in 3 months
- 7. 7 THE ROAD TO INDUSTRIALIZATION The IT side
- 8. 8 Building enterprise virtual assistants 1 An enterprise level chatbot is an IT project of which AI is a small component 2 Operational excellence of which DevOps is a major part is crucial for the success of a chatbot project 3 Projects must first focus on infrastructure & performance aspects (NLU, etc.) 3 Truths
- 9. 9 Operational excellence is needed to succeed The DevOps Way • Every asset should be version-controlled (git) • Code (actions, policies, etc.) • Multimedia resources (images, etc.) • Training data (NLU, stories) • Configurations, thresholds • Infrastructure (as code) • CI / CD pipelines Use of project templates to streamline the creation of new projects!
- 10. 10 CI / CD Automated Tests Build Images Deployment Scans
- 11. 11 CI / CD Automated Tests Build Images Deployment Scans Coding Standards Vulnerabilities Code quality metrics Bugs (code smells)
- 12. 12 CI / CD Automated Tests Build Images Deployment Scans Build Docker images (Rasa + actions) Fetch trained model from storage bucket
- 13. 13 CI / CD Automated Tests Build Images Deployment Scans Unit Tests NLU Performance - Ensure minimal performance - Teams notification on failure Rasa end-to-end - No regression on the channel - Coverage of basic flows
- 14. 14 End-to-end Rasa Testing Small Python library - Integrated with pytest - Makes HTTP requests to a deployed bot - Used on local machines and in pipelines (docker-compose)
- 15. 15 The « Ops » in DevOps – monitoring and alerting
- 17. 17 The reality of large organizations (and financial institutions) 1 Security 2 UI Tax 3 Integration in the target application