Natural Language Processing
1 like171 views
This document provides an overview of natural language processing (NLP) and the use of deep learning for NLP tasks. It discusses how deep learning models can learn representations and patterns from large amounts of unlabeled text data. Deep learning approaches are now achieving superior results to traditional NLP methods on many tasks, such as named entity recognition, machine translation, and question answering. However, deep learning models do not explicitly model linguistic knowledge. The document outlines common NLP tasks and how deep learning algorithms like LSTMs, CNNs, and encoder-decoder models are applied to problems involving text classification, sequence labeling, and language generation.
![– Question answering
– Summarization
– Dialog systems
Langauge Representation:
How to make text
computable?
Document Representation & Language Model
ˆ How to represent a document?
ˆ Make it computable
ˆ How to infer the relationship among documents or identify
the structure within a document?
ˆ Knowledge discovery
ˆ Language Model and N-Grams
Bag-of-Words representation
Term as the basis for vector space
ˆ Doc1: Text mining is to identify useful information.
ˆ Doc2: Useful information is mined from text.
ˆ Doc3: Apple is delicious.
What are Word Vectors/Embeddings?
ˆ Word Embeddings are the texts converted into numbers
ˆ There may be different numerical representations of same
text.
ˆ Many Machine Learning algorithms and almost all Deep
Learning Architectures are incapable of processing strings
or plain text in their raw form.
ˆ They require numbers as inputs to perform any sort of job,
be it classification, regression etc. in broad terms.
ˆ So, for the computer to be able to ”understand” a vector
representation of a word is required.
Different types of Word Vectors
ˆ (Traditional) Frequency based Embedding:
– One-hot
– Count Vector
– TF-IDF Vector
– Co-Occurrence Vector
ˆ (Modern) Prediction based Embedding:
– Word2vec (Google)
– Global Vector Representations (GloVe) (Stanford)
One Hot
One-hot: Suppose our vocabulary has only five words: King,
Queen, Man, Woman, and Child. We could encode the word
‘Queen’ as:
No meaningful comparison possible.
Good Vector Representation
ˆ To have ”Semantic” (meaning-wise) representation, the
Similar words should be close to each other in the hyper
dimensional space.
ˆ Non-similar words should be far apart from each other in
the hyper dimensional space.
Good Vector Representation
ˆ Traditional One Hot Encoding:
– Apple = [1, 0, 0]
– Orange = [0, 1, 0]
– Plane = [0, 0, 1]
ˆ Very few cells participate in the representation.
Word2Vec
Word2vec (Google): a distributed representation of a word is used
and not sparse like One-Hot.
Represent in some abstract way the ‘meaning’ of a word.](https://image.slidesharecdn.com/mainseminardeepnaturallanguageprocessingcheatsheet-180903113326/85/Natural-Language-Processing-5-320.jpg)
![Word Distributed Representation - Word2Vec
ˆ All vector cells participate in representing each word.
ˆ Words are represented by real valued dense vectors of sig-
nificantly smaller dimensions (e.g. 100 - 1000).
ˆ Intuition: consider each vector cell as a representative of
some feature.
Word Representations Comparison
Traditional Method - Bag of
Words Model
ˆ Uses one hot encoding
ˆ Each word in the
vocabulary is rep-
resented by one bit
position in a HUGE
vector.
ˆ For example, with a
vocabulary of 10000
words, and ”Hello” is
the 4th word in the
dictionary: 0 0 0 1 0 0
. . . . . . . 0 0 0 0
ˆ Context information is
not utilized
Modern - Word Vectors
ˆ Stores each word in
as a point in space,
represented by a vec-
tor of fixed number of
dimensions (generally
300)
ˆ Unsupervised, built
just by reading huge
corpus
ˆ For example, ”Hello”
might be represented
as : [0.4, −0.11, 0.55,
0.3 . . . 0.1, 0.02]
ˆ Context information is
utilized
Examples
Vectors for King, Man, Queen, & Woman:
Examples
Gender relation:
Plural relation:
Examples
Word pair relationships:
Examples
Country-capital city relationship:
The Power of Word2Vecs
ˆ They provide a fresh perspective to ALL problems in NLP,
and not just solve one problem.
ˆ Technological Improvement
ˆ Rise of deep learning since 2006 (Big Data + GPUs + Work
done by Andrew Ng, Yoshua Bengio, Yann Lecun and Geoff
Hinton)](https://image.slidesharecdn.com/mainseminardeepnaturallanguageprocessingcheatsheet-180903113326/85/Natural-Language-Processing-6-320.jpg)
Ad
More Related Content
What's hot (20)
Similar to Natural Language Processing (20)
Ad
More from punedevscom (10)
Ad
Recently uploaded (20)
Natural Language Processing
- 1. Using Deep Learning for Natural Language Processing (DNLP) Yogesh Kulkarni Introduction to NLP Use case: Bank Call Center Calling the Call Center ˆ Calling to an IVR (Integrated voice response) ˆ A prerecorded menu selection. ˆ “Please press 1 for Account Details, Please press 2 for . . . ” ˆ till it comes to your option. ˆ towards end, somewhere, given access to a person to talk to. Boring? Annoying? (Ref: Deep Learning and NLP A-Z - Kirill Eremenko) Instead, how about typing/saying your query directly and getting the answer right away? Solution Chatbots ˆ Which problem of IVR it is solving? ˆ Advantages? ˆ Disadvantages? ˆ Gaining popularity . . . ˆ Many platforms ˆ Companies in Pune? The Giants are at it . . . (Ref: Deep Learning and NLP A-Z - Kirill Eremenko) ˆ Chatbots or QA systems, predominantly voice based, ˆ Underlying processing is primarily Natural Language Pro- cessing (NLP). ˆ You can have your own chatbot, specific to you!! ˆ NLP is the core skill needed. Why so much popularity? Chatbots are: ˆ Autonomous and Always Available ˆ Drive Conversation ˆ Able to handle millions of requests, scalable. Its hard to master language, and thus NLP. NLP is AI-complete ˆ “The most difficult problems in AI manifest themselves in human language phenomena.” ˆ Use of language is the touchstone of intelligent behavior. ˆ Test for Intelligence - Turing Test ˆ Alan Turing (1950) proposed a test of a machine’s capabil- ity to perform human-like conversation. Turing Test A human judge engages in a natural language conversation with two other parties, one a human and the other a machine; if the judge cannot reliably tell which is which, then the machine is said to pass the test. Early Conversational Programs ˆ ELIZA (by Joseph Weizenbaum), 1966. ˆ A psychotherapist, but NO real understanding; ˆ Simple pattern-matching to respond to user input to canned responses
- 2. Loebner Prize ˆ In 1990, Hugh Loebner started Turing Test competition ˆ $100,000 will be awarded to the first bot that judges can- not distinguish from a real human in a Turing test that includes text, visual, and auditory input. ˆ Nobody has won the grand prize yet. ˆ 2016 (and 2013) year-wise top winner - Mitsuku. https://www.facebook.com/mitsukubot Why can’t we win the Grand Prize? What are the challenges? Why Language is hard? What is Language? What is Language? Language Types Natural Language Artificial Language Differences? Language, simplistically ˆ A vocabulary consists of a set of words ˆ A text is composed of a sequence of words from a vocabu- lary ˆ A language is constructed of a set of all possible texts NLP ˆ NLP is Natural Language Processing, ie processing Natural Langauge for some end-purpose in mind. ˆ Inspite of usage of Natural Language for thousands of years, why are we not able to process it well? NLP Challenges Paraphrasing Paraphrasing: Different words/sentences express the same mean- ing ˆ Season of the year: Fall/Autumn ˆ Book delivery time – When will my book arrive? – When will I receive my book? Ambiguity Ambiguity: One word/sentence can have different meanings ˆ Fall – The third season of the year – Moving down towards the ground or towards a lower position ˆ The door is open – Expressing a fact – A request to close the door Syntax and ambiguity “I saw the man with a telescope.” - Who had the telescope? Semantics The astronomer loves the star. ˆ Star in the sky ˆ Celebrity NLP Applications
- 3. Grammar Spell and Grammar Checking ˆ Checking spelling and grammar ˆ Suggesting alternatives for the errors Word Prediction Word Prediction: Predicting the next word that is highly probable to be typed by the user ˆ Mobile typing ˆ Search Engines Information Retrieval Information Retrieval: Finding relevant information to the user’s query Text Categorization Text Categorization: Assigning one (or more) pre-defined cate- gory to a text Text Categorization Summarization Summarization: Generating a short summary from one or more documents, sometimes based on a given query Question answering Question answering: Answering questions with a short answer Question answering Question answering: IBM Watson in Jeopardy
- 4. Information Extraction Information Extraction: Extracting important concepts from texts and assigning them to slot in a certain template Information Extraction Information Extraction: Includes named-entity recognition Machine Translation Machine Translation: Translating a text from one language to another Sentiment Analysis Sentiment Analysis: Identifying sentiments and opinions stated in a text Sentiment Analysis Restaurant/hotel recommendation, Product reviews Sentiment Analysis Text analytics in financial services NLP in today’s time Trends: ˆ An enormous amount of information is now available in machine readable form as natural language text (newspa- pers, web pages, medical records, financial filings, product reviews, discussion forums, etc.) ˆ Conversational agents are becoming an important form of human-computer communication ˆ Much of human-human interaction is now mediated by computers via social media Collectively, this means that copious data is available to be used in the development of NLP systems. Level of difficulties ˆ Easy (mostly solved) – Spell and grammar checking – Some text categorization tasks – Some named-entity recognition tasks ˆ Intermediate (good progress) – Information retrieval – Sentiment analysis – Machine translation – Information extraction ˆ Difficult (still hard)
- 5. – Question answering – Summarization – Dialog systems Langauge Representation: How to make text computable? Document Representation & Language Model ˆ How to represent a document? ˆ Make it computable ˆ How to infer the relationship among documents or identify the structure within a document? ˆ Knowledge discovery ˆ Language Model and N-Grams Bag-of-Words representation Term as the basis for vector space ˆ Doc1: Text mining is to identify useful information. ˆ Doc2: Useful information is mined from text. ˆ Doc3: Apple is delicious. What are Word Vectors/Embeddings? ˆ Word Embeddings are the texts converted into numbers ˆ There may be different numerical representations of same text. ˆ Many Machine Learning algorithms and almost all Deep Learning Architectures are incapable of processing strings or plain text in their raw form. ˆ They require numbers as inputs to perform any sort of job, be it classification, regression etc. in broad terms. ˆ So, for the computer to be able to ”understand” a vector representation of a word is required. Different types of Word Vectors ˆ (Traditional) Frequency based Embedding: – One-hot – Count Vector – TF-IDF Vector – Co-Occurrence Vector ˆ (Modern) Prediction based Embedding: – Word2vec (Google) – Global Vector Representations (GloVe) (Stanford) One Hot One-hot: Suppose our vocabulary has only five words: King, Queen, Man, Woman, and Child. We could encode the word ‘Queen’ as: No meaningful comparison possible. Good Vector Representation ˆ To have ”Semantic” (meaning-wise) representation, the Similar words should be close to each other in the hyper dimensional space. ˆ Non-similar words should be far apart from each other in the hyper dimensional space. Good Vector Representation ˆ Traditional One Hot Encoding: – Apple = [1, 0, 0] – Orange = [0, 1, 0] – Plane = [0, 0, 1] ˆ Very few cells participate in the representation. Word2Vec Word2vec (Google): a distributed representation of a word is used and not sparse like One-Hot. Represent in some abstract way the ‘meaning’ of a word.
- 6. Word Distributed Representation - Word2Vec ˆ All vector cells participate in representing each word. ˆ Words are represented by real valued dense vectors of sig- nificantly smaller dimensions (e.g. 100 - 1000). ˆ Intuition: consider each vector cell as a representative of some feature. Word Representations Comparison Traditional Method - Bag of Words Model ˆ Uses one hot encoding ˆ Each word in the vocabulary is rep- resented by one bit position in a HUGE vector. ˆ For example, with a vocabulary of 10000 words, and ”Hello” is the 4th word in the dictionary: 0 0 0 1 0 0 . . . . . . . 0 0 0 0 ˆ Context information is not utilized Modern - Word Vectors ˆ Stores each word in as a point in space, represented by a vec- tor of fixed number of dimensions (generally 300) ˆ Unsupervised, built just by reading huge corpus ˆ For example, ”Hello” might be represented as : [0.4, −0.11, 0.55, 0.3 . . . 0.1, 0.02] ˆ Context information is utilized Examples Vectors for King, Man, Queen, & Woman: Examples Gender relation: Plural relation: Examples Word pair relationships: Examples Country-capital city relationship: The Power of Word2Vecs ˆ They provide a fresh perspective to ALL problems in NLP, and not just solve one problem. ˆ Technological Improvement ˆ Rise of deep learning since 2006 (Big Data + GPUs + Work done by Andrew Ng, Yoshua Bengio, Yann Lecun and Geoff Hinton)
- 7. ˆ Application of Deep Learning to NLP - led by Yoshua Ben- gio, Christopher Manning, Richard Socher, Tomas Mikalov ˆ The need for unsupervised learning . (Supervised learning tends to be excessively dependent on hand-labeled data and often does not scale) NLP Activities: How to process text? Document/Section splitting Document/Section splitting: Splitting a text into sections Sentence splitting Sentence splitting: Splitting a text into sentences Tokenization Tokenization ˆ Process of breaking a stream of text up into tokens ( = words, phrases, symbols, or other meaningful elements) ˆ Typically performed at the “word” level ˆ Not easy: Hewlett-Packard, U.S.A., in some languages there is no “space” between words! Stemming Stemming ˆ Reduces similar words to a given “stem” ˆ E.g. detects, detected, detecting, detect : detect (stem). ˆ Usually set of rules for suffix stripping ˆ Most popular for English: Porter’s Algorithm ˆ 36% reduction in indexing vocabulary (English) ˆ Linguistic correctness of resulting stems not necessary (sen- sitivities : sensit) Lemmatization Lemmatization ˆ Uses a vocabulary and full morphological analysis of words ˆ Aims to remove inflectional endings only ˆ Return the base or dictionary form of a word, which is known as the lemma. ˆ E.g. saw : see, been, was : be Part-of-speech tagging Part-of-speech tagging: Assigning a syntatic tag to each word in a sentence Parsing Parsing: Building the syntactic tree of a sentence Parsing ((DaimlerChryslersshares)NP (rose(threeeights)NUMP (to22)P P −NU
- 8. Syntax Tree Syntax: Sample English grammar Named-entity recognition Named-entity recognition: Identifying pre-defined entity types in a sentence Topic modelings Topic modeling: Identifying structures in the text corpus Word embeddings Word embeddings: Compute a vector representing the distributed representation for every word Introduction to Deep NLP Use of Deep Learning in NLP Let’s discuss ˆ What is Deep Learning? ˆ Why is it important? ˆ How to apply Deep Learning to Natural Language Process- ing? (Ref (next few slides): Deep Learning for Natural Langauge Processing - Sihem Romd- hani) Machine Learning (Ref: Deep Learning for Natural Language Processing - Sihem Romdhani) Hand crafted features are needed in Machine Learning. E.g. for Spam Detection, features could be presennce of BIG $ amounts, FROM country, etc. Deep Learning (Ref: Deep Learning for Natural Language Processing - Sihem Romdhani) Hand crafted features are NOT needed in Deep Learning. E.g. for Object Detection, CNNs are come up with own features like, edges, parts, etc.
- 9. Reasons for Applying Deep Learning to NLP Automatic Representation Learning (Ref: A not-so-short introduction to Deep Learning NLP - Francesco Gadaleta, PhD) Reasons for Applying Deep Learning to NLP Learning from unlabeled data ˆ Typical traditional Machine Leaning based NLP requires labeled training data. ˆ DL based methods like Skip Gram, CBOW generate word2vec by making labels from unlabeled data. Reasons for Applying Deep Learning to NLP Human language is seqeuntial and contextual. (Ref: A not-so-short introduction to Deep Learning NLP - Francesco Gadaleta, PhD) RNNs serve the purpose well. Traditional NLP under threat? ˆ Deep learning models have taken NLP by storm, achieving superior results across many applications. ˆ Many DL approaches do not model any linguistic knowl- edge. They view language as a sequence of strings. ˆ Is this the end of NLP as a separate discipline? NLP ˆ Rule based systems (since 1960s): Regex ˆ Machine Learning (since late 1980s): Naive Bayes, SVM, HMM ˆ Deep Learning (since 2000) The Promise of Deep NLP (Ref: Deep Learning for NLU - Dr. David Talby) Deep NLP Opportunities (Ref: Deep Learning and NLP A-Z - Kirill Eremenko) Deep NLP Algorithms LSTM for sequence labelling (Ref: Deep Learning for NLP - Yves Peirsman) Application: named entity recognition Encoder-Decoder Architecture (Ref: Deep Learning for NLP - Yves Peirsman) Applications: machine translation, text summarization, dialogue modelling, etc
- 10. Traditional Named Entity Recognition The problem (Ref: Deep Learning for NLU - Dr. David Talby) Traditional Named Entity Recognition Conditional Random Fields (CRFs), “Classic” machine learning approach (Ref: Deep Learning for NLU - Dr. David Talby) Deep Named Entity Recognition LSTM (Ref: Deep Learning for NLU - Dr. David Talby) Deep Named Entity Recognition Bi-LSTM (Ref: Deep Learning for NLU - Dr. David Talby) Summary of DL algos for NLP (Ref: Engineering Intelligent NLP Applications Using Deep Learning Part 2 Saurabh Kaushik) Conclusion ˆ Deep learning has simplified feature engineering in many cases (it certainly hasn’t removed it) ˆ Less feature engineering is leading to more complex ma- chine learning architectures ˆ Most of the time, these model architectures are as specific to a given task as feature engineering used to be. ˆ The job of the data scientist will stay sexy for a while (keep your fingers crossed on this one). What next? ˆ Coursera : Dr Radev’s NLP course (https://www.coursera.org/l language-processing) ˆ Course: Deep NLP By Richard Sochar (Stanford) ˆ Book: Natural Language Processing with Python
- 11. References References Many publicly available resources have been refereed for making this presentation. Some of the notable ones are: ˆ Introduction to Natural Language Processing - Dr. Mari- ana Neves, SoSe 2016 ˆ Machine Learning for Natural Language Processing - Tra- ian Rebedea, Stefan Ruseti - LeMAS 2016 - Summer School ˆ CSC 594 Topics in AI - Natural Language Processing - De Paul ˆ Deep Learning for Natural Language Processing - Sihem Romdhani ˆ Notebooks and Material @ https://github.com/rouseguy/DeepL Copyleft « Send suggestions to [email protected]