Craftworkz at InterConnect 2017 - Creating a Highly Scalable Chatbot in a Microservices Architecture Using IBM Bluemix
Download as PPTX, PDF2 likes1,000 views
The document discusses the design of a highly scalable chatbot using IBM Bluemix, highlighted at the IBM Interconnect 2017 conference. It addresses challenges such as scaling, message loss, and the need for external data management, while also emphasizing the utilization of various technologies like MongoDB, Redis, and Node.js. The chat serves as a dialogue about enhancing the chatbot's capabilities and summarizes contact details for Craftworkz.
Craftworkz at InterConnect 2017 - Creating a Highly Scalable Chatbot in a Microservices Architecture Using IBM Bluemix
- 1. Session ID: 3322A Company: Craftworkz Sam Hendrickx & Michiel Vandendriessche IBM InterConnect 2017 Conference Creating a Highly Scalable Chatbot in a Microservices Architecture Using IBM Bluemix
- 3. Hi handsome! How are you doing today? Feeling great! Just visiting San Francisco. How r u? Not yet, we just came back from a conference. I’m doing fine. Any plans for tonight? Want to go for a drink? Yeah why not! Awesome Michiel! Can you please verify your identity here so I know you’re not a creep: http://bit.ly/abc123
- 4. OMG, IT’S A BOT!
- 6. THE ERA OF BOTS
- 8. BELGIUM
- 18. What can you tell me about IBM InterConnect 2017? InterConnect is a conference where you can tap into the most advanced cloud technology in the market today! Where is it being held? The location of InterConnect is Mandalay Bay, Las Vegas, NV
- 23. NLP REST SESSION RESPONSENLP REST SESSION RESPONSENLP REST SESSION RESPONSE PYTHON RUNTIMES . . .
- 25. What can you tell me about IBM InterConnect 2017? InterConnect is a conference where you can tap into the most advanced cloud technology in the market today! Where is it being held? I don’t know how to answer that question…
- 27. NLP REST SESSION RESPONSENLP REST SESSION RESPONSENLP REST SESSION RESPONSE PYTHON RUNTIMES . . .
- 29. NLP REST SESSION RESPONSENLP REST SESSION RESPONSENLP REST SESSION RESPONSE PYTHON RUNTIMES . . .
- 30. STORE SESSION INFO IN DATABASE
- 31. NLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSE PYTHON RUNTIMES . . . SESSION COMPOSE FOR MONGODB
- 32. OUR BOT IS NOT ON SOCIAL MEDIA!
- 34. NLP REST SESSION MANAGER RESPONSE WE DON’T WANT TO MANAGE CONNECTIONS HERE
- 35. ADD API MANAGER TO ARCHITECTURE
- 36. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES NLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSE PYTHON RUNTIMES . . .
- 37. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES NLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSE PYTHON RUNTIMES . . .
- 38. SOME MESSAGES ARE GETTING LOST!
- 39. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES NLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSENLP REST SESSION MANAGER RESPONSE PYTHON RUNTIMES . . . MESSAGE LOSS!
- 41. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES COMPOSE FOR REDIS NLP SESSION MANAGER RESPONSE PYTHON RUNTIMES . . . NLP SESSION MANAGER RESPONSE NLP SESSION MANAGER RESPONSE
- 42. OUR AI NEEDS EXTERNAL DATA!
- 45. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES COMPOSE FOR REDIS PYTHON RUNTIMES . . . NLP SESSION MANAGER RESPONSE CONNECTION MANAGER NLP SESSION MANAGER RESPONSE CONNECTION MANAGER NLP SESSION MANAGER RESPONSE CONNECTION MANAGER
- 46. OOPS, THIS I/O IS QUITE SLOW!
- 48. WITH I/O NLP CONNECTION MANAGER SESSION MANAGER RESPONSE Q A ~ 200 ms
- 51. CPU USAGE
- 54. CPU USAGE
- 55. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES COMPOSE FOR REDIS PYTHON RUNTIMES . . . NLP SESSION MANAGER RESPONSE CONNECTION MANAGER NLP SESSION MANAGER RESPONSE CONNECTION MANAGER NLP SESSION MANAGER RESPONSE CONNECTION MANAGER
- 57. RESULTS
- 58. THESE INSTANCES SHOULD BE SCALING…
- 59. BLUEMIX AUTO-SCALING, TAKE IT AWAY!
- 61. FINAL RESULTS
- 64. WATSON CARES!
- 65. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES COMPOSE FOR REDIS PYTHON RUNTIMES . . . NLP SESSION MANAGER RESPONSE CONNECTION MANAGER NLP SESSION MANAGER RESPONSE CONNECTION MANAGER NLP SESSION MANAGER RESPONSE CONNECTION MANAGER
- 66. SESSION COMPOSE FOR MONGODB API MANAGER API MANAGER API MANAGER . . . NODEJS RUNTIMES COMPOSE FOR REDIS PYTHON RUNTIMES . . .SESSION MANAGER RESPONSE CONNECTION MANAGER SESSION MANAGER RESPONSE CONNECTION MANAGER SESSION MANAGER RESPONSE CONNECTION MANAGER
- 67. Get Connected @craftworkz_co Let’s stay in touch! We can meet you in one of our offices in Leuven or Kontich. Or we can come to you. Nothing beats a coffee and an interesting conversation ;-) @craftworkz @craftworkzco @craftworkz_co www.craftworkz.be Veldkant 33A, 2550 Kontich [email protected] +32 498 44 50 61 Headquarter Gaston Geenslaan 11 B4 BE-3001 Leuven LEUVEN (BE) OFFICE Veldkant 33A BE-2550 Kontich ANTWERP (BE) OFFICE Pedro de Medinalaan 81, NL-1086 XP Amsterdam AMSTERDAM (NL) OFFICE
Editor's Notes
- #2: SAM
- #3: SAM a couple of months ago -> San Francisco When we came back from the conference -> Michiel Tinder Michiel was swiping left & right Came across beautiful girl Match Converstation went like this
- #4: SAM
- #5: SAM
- #6: SAM - We we’rent the only ones
- #7: SAM
- #8: MICHIEL Sam & Michiel from Belgium Idea of where Belgium is -> animation Now -> Las Vegas
- #9: MICHIEL Very small country with only 11 million inhabitants Very complex country 6 different governments in this very small country 3 different official languages Over 1000 kinds of beer ** DOES ANYBODY HAVE AN IDEA WHICH LANGUAGES? **
- #10: MICHIEL … Era of chatbots Build chatbots A lot of cool tools for developers from different vendors on the market But one problem…
- #11: MICHIEL … Those developer tools do not support the Dutch language Very simple solution
- #12: MICHIEL … Let’s build this chatbot framework ourselves
- #13: MICHIEL Founders of Craftworkz Build innovative prototypes for other companies Strong focus on artificial intelligence Build an engine that can be trained to interpret sentences in a specific domain Started with a very modest and humble approach
- #14: MICHIEL Let’s take a look at this first approach
- #15: MICHIEL REST API for handling HTTP requests
- #16: MICHIEL NLP component for interpreting the sentences and invoking the right response NLP in a chatbot context uses machine learning to learn how to interpret short sentences Train the engine by giving examples of sentences and their corresponding interpretations
- #17: MICHIEL Component that handles the actual responses
- #18: MICHIEL Session component for keeping context We deployed this onto Bluemix and we built a first chatbot that could have looked something like this
- #19: MICHIEL We will do a live demo later in the session
- #20: SAM
- #21: SAM
- #22: SAM
- #23: SAM
- #24: SAM
- #25: MICHIEL With this new architecture A new problem arose …
- #26: MICHIEL Let’s take a look at the previous conversation … This only happened some times but not all the times
- #27: MICHIEL ** DOES ANYONE HAVE AN IDEA WHAT HAPPENED? **
- #28: MICHIEL Take a look at the architecture
- #29: MICHIEL Context is stored in memory in the instances
- #30: MICHIEL Every engine has its own version of the context in memory and they are not in sync What can you tell me about IBM InterConnect? -> handled by one engine The follow up question: Where is it being held? -> handled by another engine not knowing about the context
- #31: MICHIEL Solution: … Looked at the Bluemix catalog
- #32: MICHIEL Added a compose for MongoDB database Shared across all the engines Every engine now has a session manager for communicating with the MongoDB
- #33: SAM
- #34: SAM
- #35: SAM
- #36: SAM
- #37: SAM
- #38: SAM
- #39: SAM
- #40: SAM
- #41: SAM
- #42: SAM
- #43: MICHIEL Chatbots need to integrate with external datasources API’s, databases, SQL, NoSQL, CRM’s and so on The responses can be enriched with data from these data sources
- #44: MICHIEL …
- #45: MICHIEL Added a connection manager to our engine …
- #46: MICHIEL Complete architecture looked something like this Message comes from social media Sent to the API Pushed on the queue Pulled from the queue Analyzed Response is generated, with external data Keeping context in mind Sent back to the frontend
- #47: MICHIEL We encoutered a new problem …
- #48: MICHIEL Let’s take a look at how a message was processed before Pulled from the queue to the engine, processed, a response in generated and sent back
- #49: MICHIEL Now Pulled from the queue Processed Response is enriched with external data -> communication takes some time Sent back to the frontend 10 times as long
- #50: MICHIEL 10.000 users sending 10 messages in parallel This was getting slow Up to minutes response time We didn’t really know how to solve this
- #51: MICHIEL Installed an activity monitor plugin for bluemix This is what it looked like under heavy load Disk usage -> don’t really care Memory -> seems okay CPU is very idle
- #52: MICHIEL Only 5% of CPU usage under heavy load Most of the time, the CPU is waiting for I/O Waiting for external data
- #53: MICHIEL Multithreading to the rescue Added 100 threads per instance working in parallel for handling messages
- #54: MICHIEL If we now take a look at the activity monitor
- #55: MICHIEL CPU usage of 93% A lot more efficient
- #56: MICHIEL Let’s take a look at the complete testing architecture Jmeter for simulating 10.000users talking in parallel Making requests to the API
- #57: MICHIEL This is what it looked like in a testing environment
- #58: MICHIEL Average response time of around …
- #59: SAM
- #60: SAM
- #61: SAM
- #62: SAM
- #63: MICHIEL
- #64: SAM
- #65: SAM
- #66: SAM
- #67: SAM
- #68: SAM