Ad
More Related Content
What's hot (20)
Similar to L1 nlp intro (20)
Ad
Recently uploaded (20)
Ad
L1 nlp intro
- 1. NLP – Chap1
- 2. Text Book • Text Books : • Daniel Jurafsky & James H. Martin, Speech and Language Processing, Second Edition, 2008.
- 3. Background
- 4. Background • How the syllabus is framed? – Starts with word and its components – Then how the word fits together – syntax – To the meaning of words, phrases, sentences – semantics – Issues of coherent texts, dialog and translation • Technologies to cover theory: – Regular expressions, – IR, – Context Free Grammar, – unification, First order predicate calculus, – Hidden Markov and other probabilistic models
- 5. 5 Background - Goal of NLP • Develop techniques and tools to build practical and robust systems that can communicate with users in one or more natural language Natural Lang. Artificial Lang. Lexical >100 000 words ~100 words Syntax Complex Simple Semantic 1 word --> several meanings 1 word --> 1 meaning
- 6. Background • What we mean by “natural language” ? • A language that is used for everyday communication by humans; Ex: English, Hindi • In contrast to programming languages and mathematical notations: – natural languages have evolved as they pass from generation to generation – and are hard to pin down with explicit rules.
- 7. Background • What we mean by “Natural Language Processing” ? • Natural Language Processing—or NLP for short—in a wide sense covers any kind of computer manipulation of natural language. – NLP is the branch of computer science focused on developing systems that allow computers to communicate with people using everyday language. – Also called Computational Linguistics – Also concerns how computational methods can aid the understanding of human language – it could be as simple as counting word frequencies to compare different writing styles. – At the other extreme, NLP involves “understanding” complete human utterances, at least to the extent of being able to give useful responses to them.
- 8. Background • Technologies based on NLP are becoming increasingly widespread. – Ex: phones and handheld computers support predictive text and handwriting recognition; – web search engines give access to information locked up in unstructured text; – Machine translation allows to retrieve texts written in Chinese and read them in Spanish. – providing more natural human-machine interfaces, – and more sophisticated access to stored information, • language processing has come to play a central role in the multilingual information society.
- 9. Background • Linguistics • 100 years of history as a scientific discipline • Computational Linguistics • 40 year history as a part of CS • Language understanding • Since last 15 years, emerged as an industry reaching millions of people with – IR and ML available on the internet – Speech recognition on computers
- 10. Background • How the course is related to other Dept Courses? • traditionally taught in different courses in different departments: – speech recognition in electrical engineering depts – parsing, semantic interpretation, and pragmatics in natural language processing courses in computer science departments, – Computational morphology and phonology in computational linguistics courses in linguistics departments
- 11. Forms of Natural Language • The input/output of a NLP system can be: – written text: newspaper articles, letters, manuals, prose, … – Speech: read speech (radio, TV, dictations), conversational speech, commands, … • To process written text, we need: – lexical, – syntactic, – Semantic knowledge about the language – discourse information, – real world knowledge
- 12. Forms of Natural Language • To process written text, we need: – lexical, syntactic, semantic knowledge about the language – discourse information, real world knowledge • To process spoken language, we need – everything above plus – speech recognition – speech synthesis
- 13. Technologies • Speech recognition – Spoken language is recognized and transformed in into text as in dictation systems, into commands as in robot control systems, or into some other internal representation. • Speech synthesis – Utterances in spoken language are produced from text (text-to-speech systems) or from internal representations of words or sentences (concept-to-speech systems)
- 14. Components of NLP • Natural Language Understanding – Mapping the given input in the natural language into a useful representation. – Different level of analysis required: morphological analysis, syntactic analysis, semantic analysis, discourse analysis, … • Natural Language Generation – Producing output in the natural language from some internal representation. – Different level of synthesis required: • deep planning (what to say), • syntactic generation
- 15. Introduction
- 16. Introduction NLP Applications 1. Mundane applications (Simple) such as word counting and automatic hyphenation, spelling correction, text categorization 2. Cutting edge applications (Complex) such as automated question answering on the Web, and real-time spoken language translation, speech recognition, machine translation, information extraction, sentiment analysis What distinguishes these language processing applications from other data processing systems ? • is their use of knowledge of language
- 18. Introduction • Language processing applications vs. other data processing systems • Ex: Unix wc program, to count the total number of bytes, words, and lines in a text file. • When used to count bytes and lines, wc is an ordinary data processing application. • However, when it is used to count the words in a file it requires knowledge about what it means to be a word, • and thus becomes a language processing system.
- 19. 1.1 Knowledge in Speech and Language Processing Natural Language Processing Tasks
- 20. Natural Language Processing Tasks • Processing natural language text involves various syntactic, semantic and pragmatic tasks in addition to other problems.
- 21. 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge Acoustic/ Phonetic Syntax Semantics Pragmatics words parse trees literal meaning meaning (contextualized) sound waves
- 22. 22 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge 1. Phonetics and Phonology — The study of linguistic sounds 2. Morphology —The study of the meaningful components of words 3. Syntax —The study of the structural relationships between words 4. Semantics — The study of meaning 5. Pragmatics — The study of how language is used to accomplish goals 6. Discourse—The study of linguistic units larger than a single utterance
- 23. 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • The tasks of analyzing an incoming audio signal and • recovering the exact sequence of words and generating its response • require knowledge about phonetics and phonology, • which can help model how words are pronounced in colloquial (used in ordinary or familiar conversation; not formal or literary ) speech (Chapters 4 and 5).
- 24. 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • Producing and recognizing the variations of individual words (e.g., recognizing that doors is plural) • requires knowledge about morphology, • which captures information about the shape and behavior of words in context (Chapters 2 and 3).
- 25. 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • Syntax: the knowledge needed to order and group words together HAL, the pod bay door is open. HAL, is the pod bay door open? I’m I do, sorry that afraid Dave I’m can’t. (Dave, I’m sorry I’m afraid I can’t do that.)
- 26. 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • Lexical semantics: knowledge of the meanings of the component words • Compositional semantics: knowledge of how these components combine to form larger meanings – To know that Dave’s command is actually about opening the pod bay door, rather than an inquiry about the day’s lunch menu.
- 27. Word Sense Disambiguation (WSD) • Words in natural language usually have a fair number of different possible meanings. – Ellen has a strong interest in computational linguistics. – Ellen pays a large amount of interest on her credit card. • For many tasks (question answering, translation), the proper sense of each ambiguous word in a sentence must be determined.
- 28. 28 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • Pragmatics: the appropriate use of the kind of polite and indirect language No or No, I won’t open the door. I’m sorry, I’m afraid, I can’t. I won’t.
- 29. 29 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • discourse conventions: knowledge of correctly structuring these such conversations – HAL chooses to engage in a structured conversation relevant to Dave’s initial request. HAL’s correct use of the word that in its answer to Dave’s request is a simple illustration of the kind of between-utterance device common in such conversations. Dave, I’m sorry I’m afraid I can’t do that.
- 30. 30 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • Phonology – concerns how words are related to the sounds that realize them. • Morphology – concerns how words are constructed from more basic meaning units called morphemes. A morpheme is the primitive unit of meaning in a language. • Syntax – concerns how words can be put together to form correct sentences and determines what structural role each word plays in the sentence and what phrases are subparts of other phrases. • Semantics – concerns what words mean and how these meaning combine in sentences to form sentence meaning. The study of context-independent meaning.
- 31. 31 1.1 Knowledge in Speech and Language Processing- 6 Categories of Linguistic Knowledge • Pragmatics – concerns how sentences are used in different situations and how use affects the interpretation of the sentence. • Discourse – concerns how the immediately preceding sentences affect the interpretation of the next sentence.For example, interpreting pronouns and interpreting the temporal aspects of the information. • World Knowledge – includes general knowledge about the world. What each language user must know about the other’s beliefs and goals.
- 32. 1.2 Ambiguity
- 33. 33 1.2 Ambiguity • A perhaps surprising fact about the six categories of linguistic knowledge is that most or all tasks in speech and language processing can be viewed as resolving ambiguity at one of these levels. • We say some input is ambiguous – if there are multiple alternative linguistic structures than can be built for it. • The spoken sentence, I made her duck, has five different meanings. – (1.1) I cooked waterfowl for her. – (1.2) I cooked waterfowl belonging to her. – (1.3) I created the (plaster?) duck she owns. – (1.4) I caused her to quickly lower her head or body. – (1.5) I waved my magic wand and turned her into undifferentiated waterfowl.
- 34. 34 1.2 Ambiguity • These different meanings are caused by a number of ambiguities. – Duck can be a verb or a noun, while her can be a dative pronoun or a possessive pronoun. – The word make can mean create or cook. – Finally, the verb make is syntactically ambiguous in that it can be transitive (1.2), or it can be ditransitive (1.5). – Finally, make can take a direct object and a verb (1.4), meaning that the object (her) got caused to perform the verbal action (duck). – In a spoken sentence, there is an even deeper kind of ambiguity; the first word could have been eye or the second word maid.
- 35. Why NL Understanding is hard? • Natural language is extremely rich in form and structure, and very ambiguous. – How to represent meaning, – Which structures map to which meaning structures. • One input can mean many different things. Ambiguity can be at different levels. – Lexical (word level) ambiguity -- different meanings of words – Syntactic ambiguity -- different ways to parse the sentence – Interpreting partial information -- how to interpret pronouns – Contextual information -- context of the sentence may affect the meaning of that sentence. • Many input can mean the same thing. • Interaction among components of the input is not clear. • Noisy input (e.g. speech)
- 36. 1.2 Ambiguity • Ways to resolve or disambiguate these ambiguities: – Deciding whether duck is a verb or a noun can be solved by part- of-speech tagging . – Deciding whether make means “create” or “cook” can be solved by word sense disambiguation. – Resolution of part-of-speech and word sense ambiguities are two important kinds of lexical disambiguation. • A wide variety of tasks can be framed as lexical disambiguation problems. – For example, a text-to-speech synthesis system reading the word lead needs to decide whether it should be pronounced as in lead pipe or as in lead me on. • Deciding whether her and duck are part of the same entity (as in (1.1) or (1.4)) or are different entity (as in (1.2)) is an example of syntactic disambiguation and can be addressed by probabilistic parsing. • Ambiguities that don’t arise in this particular example (like whether a given sentence is a statement or a question) will also be resolved, for example by speech act interpretation.
- 37. Why is Language Ambiguous? • Having a unique linguistic expression for every possible conceptualization that could be conveyed would make language overly complex and linguistic expressions unnecessarily long. • Allowing resolvable ambiguity permits shorter linguistic expressions, i.e. data compression. • Language relies on people’s ability to use their knowledge and inference abilities to properly resolve ambiguities. • Infrequently, disambiguation fails, i.e. the compression is lossy.
- 38. Natural Languages vs. Computer Languages • Ambiguity is the primary difference between natural and computer languages. • Formal programming languages are designed to be unambiguous, i.e. they can be defined by a grammar that produces a unique parse for each sentence in the language. • Programming languages are also designed for efficient (deterministic) parsing, i.e. they are deterministic context-free languages (DCFLs). – A sentence in a DCFL can be parsed in O(n) time where n is the length of the string.
- 39. 1.3 Models and Algorithms
- 40. 1.3 Models and Algorithms • The most important model: – state machines, – formal rule systems, – logic, – probability theory and – other machine learning tools • The most important algorithms of these models: – state space search algorithms and – dynamic programming algorithms
- 41. 1.3 Models and Algorithms • State machines are – formal models that consist of states, transitions among states, and an input representation. • Some of the variations of this basic model: – Deterministic and non-deterministic finite-state automata, – finite-state transducers, which can write to an output device, – weighted automata, Markov models, and hidden Markov models, which have a probabilistic component.
- 42. 1.3 Models and Algorithms • Closely related to the above procedural models are their declarative counterparts: formal rule systems. – regular grammars and regular relations, context-free grammars, feature-augmented grammars, as well as probabilistic variants of them all. • State machines and formal rule systems are the main tools used when dealing with knowledge of phonology, morphology, and syntax. • The algorithms associated with both state-machines and formal rule systems typically involve a search through a space of states representing hypotheses about an input. • Representative tasks include – searching through a space of phonological sequences for a likely input word in speech recognition, or – searching through a space of trees for the correct syntactic parse of an input sentence. • Among the algorithms that are often used for these tasks are well-known graph algorithms such as depth-first search, as well as heuristic variants such as best-first, and A* search. • The dynamic programming paradigm is critical to the computational tractability of many of these approaches by ensuring that redundant computations are avoided.
- 43. 1.3 Models and Algorithms • The third model that plays a critical role in capturing knowledge of language is logic. • We will discuss – first order logic, also known as the predicate calculus, as well as – such related formalisms as feature-structures, – semantic networks, and – conceptual dependency. • These logical representations have traditionally been the tool of choice when dealing with knowledge of semantics, pragmatics, and discourse (although, as we will see, applications in these areas are increasingly relying on the simpler mechanisms used in phonology, morphology, and syntax).
- 44. 1.3 Models and Algorithms • Each of the other models (state machines, formal rule systems, and logic) can be augmented with probabilities. • One major use of probability theory is to solve the many kinds of ambiguity problems that we discussed earlier; – almost any speech and language processing problem can be recast as: “given N choices for some ambiguous input, choose the most probable one”. • Another major advantage of probabilistic models is that – they are one of a class of machine learning models. • Machine learning research has focused on ways to automatically learn the various representations described above; – automata, rule systems, search heuristics, classifiers. • These systems can be trained on large corpora and can be used as a powerful modeling technique, especially in places where we don’t yet have good causal models.