API Workshop Series Part 2: The Future of Intelligent User Interactions
- 2. The Intelligence Problem Information explosion Human attention span is limited Technology wasn’t designed to be natural….until now.
- 3. TYPE TALK TAP Bot Builder Platform Services Intelligent Tools
- 7. { Your Code } REST Endpoint Direct Line Protocol Conversational and Business Logic Canvas Aware Context Sensitive SDK Bot Builder SDK Platform Platform Services HTTP REST Endpoint AI Intelligent Tools
- 8. Bot web service Your bot code Entity Extraction Speech Vision/Face Natural Language Translation + Microsoft Cognitive Services Search Emotion Knowledge API … Message input <> output State Management Bot Connector Service Conversation Canvas/Channels …… … Other services, APIs, Databases, Azure Machine Learning, Azure Search, etc… Bot Builder SDK Web Chat Direct Line… Email Facebook GroupMe Kik Skype Slack Telegram Twilio (SMS) An x-ray of a typical bot Bot Builder SDK Your bot code goes here
- 9. Your Bot Framework Bot Bot Connector Bot DirectoryBot Builder SDKs Connect your bot(s) to text/sms, Office 365 mail, Skype, Slack, and other services. Build great dialogs within your Node.js or C# bot Try, use, and add published bots to the world’s top conversation experiences • Register, connect, publish and manage your bot through the bot dashboard • Message routing • Automatic translation to 30+ languages • User and state management • Embeddable web chat control • Debugging tools • Open source SDK on Github • From simple built-in prompts and command dialogs to simple to use yet sophisticated ‘FormFlow’ dialogs • Libraries, samples and tools to make a great conversationalist • Chat emulator • Leverage related services available in Cognitive Services • Public directory of bots registered and approved with Bot Framework • Users can try your bot from the directory via the web chat control • Users can discover and add your bot to the channels on which it is configured
- 11. O N E B O T M U L T I P L E C A N V A S E S
- 12. Adaptive Cards O P E N F R A M E W O R K M U L T I P L E C A N V A S E S Notification Microsoft Teams Skype Android iOS
- 13. Emotion Speaker Recognition Speech Custom Recognition Computer Vision Face Video microsoft.com/cognitive Linguistic Analysis Language Understanding Bing Spell Check Entity Linking Knowledge Exploration Academic Knowledge Bing Image Search Bing Video Search Bing Web Search WebLM Text Analytics Recommendations Bing Autosuggest Bing News Search Translator https://www.microsoft.com/cognitive-services/
- 14. QnA Maker Build, train and publish Q&A bots based on existing FAQs https://www.qnamaker.ai
- 15. The problem Language understanding in human-computer interaction is: Technically challenging It’s exceedingly difficult to enable a computer to understand what a person wants and to find the pieces of information that are relevant to their intent. Costly to implement Building and maintaining machine learning systems requires a large investment of time, money and engineering resources Often domain specific In the past, building your own machine learned models often required assistance of a team of data scientists that would customize the models to the specific domain.
- 16. Speech Intent Recognition Text to Speech Conversion Convert text to spoken audio Speech Recognition Convert spoken audio to text 450 6th St. San Francisco Convert audio to intent "450 sixth St in San Francisco." "For 50 six St San Francisco." "456th St San Francisco." • State of the art algorithms to process spoken language • Voice Recognition – Converts spoken audio to text • Support for 18 languages (28 locales) • Voice Output – Synthesize audio from Text • Support for 9 languages (17 locales) • Powered by Bing, which is the same backend for Xbox and Cortana on Windows Phone 8.1 and Windows 10 Speech APIs
- 17. Determine intent and detects entities Seamless integration with Speech Recognition Learns over time Use pre-built, world class models from Bing and Cortana Models work across devices Intent: TextToSpeech Content: headlines Read me the headlines Play yesterday’s Daily Show Intent: PauseDevice DateTime.duration: 5 Minutes Pause for 5 minutes Intent: PlayEpisode Content: Daily Show DateTime.date: T-1 Day
- 18. LUIS is designed to enable you to quickly deploy an HTTP endpoint that will take the sentences you send it and interpret them in terms of the intention they convey and the key entities that are present. After your endpoint has processed a few dozen interactions, LUIS begins active learning. LUIS examines all the utterances that have been sent to it, and calls to your attention the ones that it would like you to label. LUIS relies on the cutting-edge technologies developed and incubated directly by Microsoft Research that have been tried and tested with hundreds of internal and external customers all over the globe. In addition to allowing you to build your own applications, LUIS also provides selected models from the Microsoft Cortana personal assistant as a pre-built application. It learns and adaptsIt is fast and easy It is innovative It uses pre-built applications Why LUIS?
- 19. LUIS offers a set of user-friendly tools that completely abstract technical complexity and get you up and running with just a few steps. This breakthrough feature enables LUIS to examine all the utterances that have been sent to it, critique its own work, and ask for any corrections in order to continually improve accuracy. LUIS is able to fulfill actions that were triggered through a set of channels (HTTP, news, stocks, etc.) LUIS provides a way to add a prompt question if user query is missing any of the required parameters Active learningEasy creation of LU models Action fulfillment Dialog support Key features
- 20. How LUIS works
- 21. How LUIS works Go to luis.ai: Sign in with your Microsoft account (MSA). If you don’t have MSA, you will have an option to create one. Get started by creating a New App and entering some basic information. Next, you will be presented with the Application Editor Workspace that will allow you to create and train your own language understanding model. Create a New App1
- 22. How LUIS works Intents: Intents are actions that a user wants your app to take or the information they would like to obtain. Example intents could include getting weather, booking tickets, adding a calendar entry or operating a light fixture. Add one or more of user intents that you expect your app to handle by clicking + next to Intents item in the in the left-hand panel of the Editor Workspace. Add intents and entities2 X
- 23. How LUIS works Entities: Entities are real world objects such as persons, locations, organizations, products, etc. that can be denoted with a proper name. Entities can be abstract or have a physical existence. Entities can be generic (location, celebrity, datetime) or more specific (Seattle, Satya Nadella, June) Add one or more entities that you expect your app to recognize by clicking + next to Entities item in the left- hand panel of the Editor Workspace. Several commonly used pre-built entities (e.g. datetime, number) are also available to be added to the app by clicking + next to Pre-built Entities. Add intents and entities2 X
- 24. How LUIS works Seed the system with more examples of utterances: Enter more examples of queries that you expect your users to make. As you enter each one, you will need to: • select the name of the correct intent from the dropdown • label your entities that appear in each utterance by clicking on the entity and choosing corresponding label from the list • pre-built entities get automatically labeled in grey The more examples you provide, the more accurate the predictions. Provide more labeled examples3 X
- 25. Train your model4 Last train completed: 9/10/2016, 3:33:38 PM How LUIS works As you click Train at left bottom corner of the page, LUIS: Generalizes from the examples you provided. Uses logistic regression classifiers to recognize intents. Uses conditional random field to determine the entities.
- 26. How LUIS works Deploy the model to an HTTP endpoint: Click the Publish button in the upper left-hand corner. The URL that you see appear after a few moments makes your model available as a web service. Publish your model5
- 27. Activate model from your application on any device: Update the URL with the parameter for the user query. The response received from LUIS will contain the list of detected intents and entities together with the confidence scores. You can now use this information in your app. For our example, we could next call the weather service and display the response in our app UI: Use the model in your application6 How LUIS works X What’s the weather in Berlin The weather is sunny with a temperature of 88°F. (data provided by Foreca http://www.foreca.com)
- 28. Bot web service Your bot code Entity Extraction Speech Vision/Face Natural Language Translation + Microsoft Cognitive Services Search Emotion Knowledge API … Message input <> output State Management Bot Connector Service Conversation Canvas/Channels …… … Other services, APIs, Databases, Azure Machine Learning, Azure Search, etc… Bot Builder SDK Web Chat Direct Line… Email Facebook GroupMe Kik Skype Slack Telegram Twilio (SMS) An x-ray of a typical bot Bot Builder SDK Your bot code goes here
- 29. Questions