Slides
Uploaded by
Luiz Felipe Niedermaier CustódioSlides
Uploaded by
Luiz Felipe Niedermaier CustódioNatural Language Processing
Natural Language Processing
Ann Copestake
Computer Laboratory
University of Cambridge
October 2007
Natural Language Processing
Outline of today’s lecture
Lecture 1: Introduction
Overview of the course
Why NLP is hard.
Scope of NLP.
A sample application: sentiment classification
More NLP applications
NLP components.
Natural Language Processing
Lecture 1: Introduction
Overview of the course
NLP and linguistics
NLP: the automatic processing of human language.
1. Morphology — the structure of words: lecture 2.
2. Syntax — the way words are used to form phrases:
lectures 3, 4 and 5.
3. Semantics
I Compositional semantics — the construction of meaning
based on syntax: lecture 6.
I Lexical semantics — the meaning of individual words:
lecture 6.
4. Pragmatics — meaning in context: lecture 7.
Natural Language Processing
Lecture 1: Introduction
Overview of the course
Also note:
I Exercises: pre-lecture and post-lecture
I Glossary
I Recommended Book: Jurafsky and Martin (2000) (and
second edition draft)
Natural Language Processing
Lecture 1: Introduction
Why NLP is hard.
Querying a knowledge base
User query:
I Has my order number 4291 been shipped yet?
Database:
ORDER
Order number Date ordered Date shipped
4290 2/2/02 2/2/02
4291 2/2/02 2/2/02
4292 2/2/02
USER: Has my order number 4291 been shipped yet?
DB QUERY: order(number=4291,date_shipped=?)
RESPONSE: Order number 4291 was shipped on 2/2/02
Natural Language Processing
Lecture 1: Introduction
Why NLP is hard.
Why is this difficult?
Similar strings mean different things, different strings mean the
same thing:
1. How fast is the 505G?
2. How fast will my 505G arrive?
3. Please tell me when I can expect the 505G I ordered.
Ambiguity:
I Do you sell Sony laptops and disk drives?
I Do you sell (Sony (laptops and disk drives))?
I Do you sell (Sony laptops) and disk drives)?
Natural Language Processing
Lecture 1: Introduction
Why NLP is hard.
Why is this difficult?
Similar strings mean different things, different strings mean the
same thing:
1. How fast is the 505G?
2. How fast will my 505G arrive?
3. Please tell me when I can expect the 505G I ordered.
Ambiguity:
I Do you sell Sony laptops and disk drives?
I Do you sell (Sony (laptops and disk drives))?
I Do you sell (Sony laptops) and disk drives)?
Natural Language Processing
Lecture 1: Introduction
Why NLP is hard.
Why is this difficult?
Similar strings mean different things, different strings mean the
same thing:
1. How fast is the 505G?
2. How fast will my 505G arrive?
3. Please tell me when I can expect the 505G I ordered.
Ambiguity:
I Do you sell Sony laptops and disk drives?
I Do you sell (Sony (laptops and disk drives))?
I Do you sell (Sony laptops) and disk drives)?
Natural Language Processing
Lecture 1: Introduction
Why NLP is hard.
Why is this difficult?
Similar strings mean different things, different strings mean the
same thing:
1. How fast is the 505G?
2. How fast will my 505G arrive?
3. Please tell me when I can expect the 505G I ordered.
Ambiguity:
I Do you sell Sony laptops and disk drives?
I Do you sell (Sony (laptops and disk drives))?
I Do you sell (Sony laptops) and disk drives)?
Natural Language Processing
Lecture 1: Introduction
Why NLP is hard.
Why is this difficult?
Similar strings mean different things, different strings mean the
same thing:
1. How fast is the 505G?
2. How fast will my 505G arrive?
3. Please tell me when I can expect the 505G I ordered.
Ambiguity:
I Do you sell Sony laptops and disk drives?
I Do you sell (Sony (laptops and disk drives))?
I Do you sell (Sony laptops) and disk drives)?
Natural Language Processing
Lecture 1: Introduction
Why NLP is hard.
Why is this difficult?
Similar strings mean different things, different strings mean the
same thing:
1. How fast is the 505G?
2. How fast will my 505G arrive?
3. Please tell me when I can expect the 505G I ordered.
Ambiguity:
I Do you sell Sony laptops and disk drives?
I Do you sell (Sony (laptops and disk drives))?
I Do you sell (Sony laptops) and disk drives)?
Natural Language Processing
Lecture 1: Introduction
Scope of NLP.
Some NLP applications
I spelling and grammar I information extraction
checking
I question answering
I optical character
recognition (OCR)
I summarization
I screen readers
I text segmentation
I augmentative and
I exam marking
alternative communication I report generation
I machine aided translation I machine translation
I lexicographers’ tools I natural language interfaces
I information retrieval to databases
I document classification
I email understanding
I document clustering
I dialogue systems
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Sentiment classification: finding out what people think
about you.
I Task: scan documents for positive and negative opinions
on people, products etc.
I Find all references to entity in some document collection:
list as positive, negative (possibly with strength) or neutral.
I Summaries plus text snippets.
I Fine-grained classification:
e.g., for phone, opinions about: overall design, keypad,
camera.
I Still often done by humans . . .
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Motorola KRZR (from the Guardian)
Motorola has struggled to come up with a worthy
successor to the RAZR, arguably the most influential
phone of the past few years. Its latest attempt is the
KRZR, which has the same clamshell design but has
some additional features. It has a striking blue finish
on the front and the back of the handset is very tactile
brushed rubber. Like its predecessors, the KRZR has
a laser-etched keypad, but in this instance Motorola
has included ridges to make it easier to use.
. . . Overall there’s not much to dislike about the phone,
but its slightly quirky design means that it probably
won’t be as huge or as hot as the RAZR.
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Sentiment classification: the research task.
I Full task: information retrieval, cleaning up text structure,
named entity recognition, identification of relevant parts of
text. Evaluation by humans.
I Research task: preclassified documents, topic known,
opinion in text along with some straightforwardly
extractable score.
I Movie review corpus, with ratings.
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
IMDb: An American Werewolf in London (1981)
Rating: 9/10
Ooooo. Scary.
The old adage of the simplest ideas being the best is
once again demonstrated in this, one of the most
entertaining films of the early 80’s, and almost
certainly Jon Landis’ best work to date. The script is
light and witty, the visuals are great and the
atmosphere is top class. Plus there are some great
freeze-frame moments to enjoy again and again. Not
forgetting, of course, the great transformation scene
which still impresses to this day.
In Summary: Top banana
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Bag of words technique.
I Treat the reviews as collections of individual words.
I Classify reviews according to positive or negative words.
I Could use word lists prepared by humans, but machine
learning based on a portion of the corpus (training set) is
preferable.
I Use star rankings for training and evaluation.
I Pang et al, 2002: Chance success is 50% (movie database
was artifically balanced), bag-of-words gives 80%.
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Some sources of errors.
I Negation:
Ridley Scott has never directed a bad film.
I Overfitting the training data:
e.g., if training set includes a lot of films from before 2005,
Ridley may be a strong positive indicator, but then we test
on reviews for ‘Kingdom of Heaven’?
I Comparisons and contrasts.
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Contrasts in the discourse
This film should be brilliant. It sounds like a great plot,
the actors are first grade, and the supporting cast is
good as well, and Stallone is attempting to deliver a
good performance. However, it can’t hold up.
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
More contrasts
AN AMERICAN WEREWOLF IN PARIS is a failed
attempt . . . Julie Delpy is far too good for this movie.
She imbues Serafine with spirit, spunk, and humanity.
This isn’t necessarily a good thing, since it prevents us
from relaxing and enjoying AN AMERICAN
WEREWOLF IN PARIS as a completely mindless,
campy entertainment experience. Delpy’s injection of
class into an otherwise classless production raises the
specter of what this film could have been with a better
script and a better cast . . . She was radiant,
charismatic, and effective . . .
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Sample data
http://www.cl.cam.ac.uk/ aac10/sentiment/
(linked from http://www.cl.cam.ac.uk/ aac10/stuff.html)
See test data texts in:
http://www.cl.cam.ac.uk/ aac10/sentiment/test/
classified into positive/negative.
Natural Language Processing
Lecture 1: Introduction
A sample application: sentiment classification
Doing sentiment classification properly?
I Morphology, syntax and compositional semantics:
who is talking about what, what terms are associated with
what, tense . . .
I Lexical semantics:
are words positive or negative in this context? Word
senses (e.g., spirit)?
I Pragmatics and discourse structure:
what is the topic of this section of text? Pronouns and
definite references.
I But getting all this to work on arbitrary text is very hard.
I Ultimately the problem is AI-complete, but can we do well
enough for NLP to be useful?
Natural Language Processing
Lecture 1: Introduction
More NLP applications
IR, IE and QA
I Information retrieval: return documents in response to a
user query (Internet Search is a special case)
I Information extraction: discover specific information from a
set of documents (e.g. company joint ventures)
I Question answering: answer a specific user question by
returning a section of a document:
What is the capital of France?
Paris has been the French capital for many centuries.
Much more about these in Simone Teufel’s course.
Natural Language Processing
Lecture 1: Introduction
More NLP applications
MT
I Earliest attempted NLP application
I Quality depends on restricting the domain
I Utility greatly increased with increase in availability of
electronic text
I Good applications for bad MT . . .
I Spoken language translation is viable for limited domains
Natural Language Processing
Lecture 1: Introduction
More NLP applications
Natural language interfaces and dialogue systems
All rely on a limited domain:
I LUNAR: classic example of a natural language interface to
a database (NLID): 1970–1975
I SHRDLU: (text-based) dialogue system: 1973
I Current spoken dialogue systems: e.g., BA flight
information
Limited domain allows disambiguation: e.g., in LUNAR, rock
had one sense.
Natural Language Processing
Lecture 1: Introduction
NLP components.
Generic NLP modules
I input preprocessing: speech recogniser, text preprocessor
or gesture recogniser.
I morphological analysis
I part of speech tagging
I parsing: this includes syntax and compositional semantics
I disambiguation
I context module
I text planning
I tactical generation
I morphological generation
I output processing: text-to-speech, text formatter, etc.
Natural Language Processing
Lecture 1: Introduction
NLP components.
Natural language interface to a knowledge base
KB
*
j
KB INTERFACE KB OUTPUT
6
?
PARSING TACTICAL GENERATION
6
?
MORPHOLOGY MORPHOLOGY GENERATION
6
?
INPUT PROCESSING OUTPUT PROCESSING
6
?
user input output
Natural Language Processing
Lecture 1: Introduction
NLP components.
General comments
I Even ‘simple’ applications might need complex knowledge
sources
I Applications cannot be 100% perfect
I Applications that are < 100% perfect can be useful
I Aids to humans are easier than replacements for humans
I NLP interfaces compete with non-language approaches
I Shallow processing on arbitrary input or deep processing
on narrow domains
I Limited domain systems require extensive and expensive
expertise to port
I External influences on NLP are very important
Natural Language Processing
Lecture 1: Introduction
NLP components.
Outline of the next lecture
Lecture 2: Morphology and finite state techniques.
A brief introduction to morphology.
Using morphology.
Spelling rules.
Finite state techniques.
More applications for finite state techniques.
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Outline of today’s lecture
Lecture 2: Morphology and finite state techniques.
A brief introduction to morphology.
Using morphology.
Spelling rules.
Finite state techniques.
More applications for finite state techniques.
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
A brief introduction to morphology.
Some terminology.
I morpheme: the minimal information carrying unit
I affix: morpheme which only occurs in conjunction with
other morphemes
I words are made up of a stem (more than one in the case
of compounds) and zero or more affixes. e.g., dog plus
plural suffix +s
I affixes: prefixes, suffixes, infixes and circumfixes
I in English: prefixes and suffixes (prefixes only for
derivational morphology)
I productivity: whether affix applies generally, whether it
applies to new words
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
A brief introduction to morphology.
Inflectional morphology
I e.g., plural suffix +s, past participle +ed
I sets slots in some paradigm
I e.g., tense, aspect, number, person, gender, case
I inflectional affixes are not combined in English
I generally fully productive (modulo irregular forms)
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
A brief introduction to morphology.
Derivational morphology
I e.g., un-, re-, anti-, -ism, -ist etc
I broad range of semantic possibilities, may change part of
speech
I indefinite combinations
e.g., antiantidisestablishmentarianism
anti-anti-dis-establish-ment-arian-ism
I generally semi-productive
I zero-derivation (e.g. tango, waltz)
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
A brief introduction to morphology.
Internal structure and ambiguity
Morpheme ambiguity: stems and affixes may be individually
ambiguous: e.g. dog (noun or verb), +s (plural or 3persg-verb)
Structural ambiguity: e.g., shorts/short -s
unionised could be union -ise -ed or un- ion -ise -ed
Bracketing:
I un- ion is not a possible form
I un- is ambiguous:
I with verbs: means ‘reversal’ (e.g., untie)
I with adjectives: means ‘not’ (e.g., unwise)
I internal structure of un- ion -ise -ed
has to be (un- ((ion -ise) -ed))
Temporarily skip 2.3
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
A brief introduction to morphology.
Internal structure and ambiguity
Morpheme ambiguity: stems and affixes may be individually
ambiguous: e.g. dog (noun or verb), +s (plural or 3persg-verb)
Structural ambiguity: e.g., shorts/short -s
unionised could be union -ise -ed or un- ion -ise -ed
Bracketing:
I un- ion is not a possible form
I un- is ambiguous:
I with verbs: means ‘reversal’ (e.g., untie)
I with adjectives: means ‘not’ (e.g., unwise)
I internal structure of un- ion -ise -ed
has to be (un- ((ion -ise) -ed))
Temporarily skip 2.3
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Using morphology.
Applications of morphological processing
I compiling a full-form lexicon
I stemming for IR (not linguistic stem)
I lemmatization (often inflections only): finding stems and
affixes as a precursor to parsing
NB: may use parsing to filter results (see lecture 5)
e.g., feed analysed as fee-ed (as well as feed)
but parser blocks (assuming lexicon does not have fee as a
verb)
I generation
Morphological processing may be bidirectional: i.e.,
parsing and generation.
sleep + PAST_VERB <-> slept
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Using morphology.
Morphology in a deep processing system (cf lec 1)
KB
*
j
KB INTERFACE KB OUTPUT
6
?
PARSING TACTICAL GENERATION
6
?
MORPHOLOGY MORPHOLOGY GENERATION
6
?
INPUT PROCESSING OUTPUT PROCESSING
6
?
user input output
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Using morphology.
Lexical requirements for morphological processing
I affixes, plus the associated information conveyed by the
affix
ed PAST_VERB
ed PSP_VERB
s PLURAL_NOUN
I irregular forms, with associated information similar to that
for affixes
began PAST_VERB begin
begun PSP_VERB begin
I stems with syntactic categories (plus more)
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Using morphology.
Mongoose
A zookeeper was ordering extra animals for his zoo. He started
the letter:
“Dear Sir, I need two mongeese.”
This didn’t sound right, so he tried again:
“Dear Sir, I need two mongooses.”
But this sounded terrible too. Finally, he ended up with:
“Dear Sir, I need a mongoose, and while you’re at it,
send me another one as well.”
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Using morphology.
Mongoose
A zookeeper was ordering extra animals for his zoo. He started
the letter:
“Dear Sir, I need two mongeese.”
This didn’t sound right, so he tried again:
“Dear Sir, I need two mongooses.”
But this sounded terrible too. Finally, he ended up with:
“Dear Sir, I need a mongoose, and while you’re at it,
send me another one as well.”
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Using morphology.
Mongoose
A zookeeper was ordering extra animals for his zoo. He started
the letter:
“Dear Sir, I need two mongeese.”
This didn’t sound right, so he tried again:
“Dear Sir, I need two mongooses.”
But this sounded terrible too. Finally, he ended up with:
“Dear Sir, I need a mongoose, and while you’re at it,
send me another one as well.”
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Spelling rules.
Spelling rules (sec 2.3)
I English morphology is essentially concatenative
I irregular morphology — inflectional forms have to be listed
I regular phonological and spelling changes associated with
affixation, e.g.
I -s is pronounced differently with stem ending in s, x or z
I spelling reflects this with the addition of an e (boxes etc)
I in English, description is independent of particular
stems/affixes
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Spelling rules.
e-insertion
e.g. boxˆs to boxes
s
ε → e/ x ˆ s
z
I map ‘underlying’ form to surface form
I mapping is left of the slash, context to the right
I notation:
position of mapping
ε empty string
ˆ affix boundary — stem ˆ affix
I same rule for plural and 3sg verb
I formalisable/implementable as a finite state transducer
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Spelling rules.
e-insertion
e.g. boxˆs to boxes
s
ε → e/ x ˆ s
z
I map ‘underlying’ form to surface form
I mapping is left of the slash, context to the right
I notation:
position of mapping
ε empty string
ˆ affix boundary — stem ˆ affix
I same rule for plural and 3sg verb
I formalisable/implementable as a finite state transducer
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Spelling rules.
e-insertion
e.g. boxˆs to boxes
s
ε → e/ x ˆ s
z
I map ‘underlying’ form to surface form
I mapping is left of the slash, context to the right
I notation:
position of mapping
ε empty string
ˆ affix boundary — stem ˆ affix
I same rule for plural and 3sg verb
I formalisable/implementable as a finite state transducer
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Finite state automata for recognition
day/month pairs:
0,1,2,3 digit / 0,1 0,1,2
1 2 3 4 5 6
digit digit
I non-deterministic — after input of ‘2’, in state 2 and state 3.
I double circle indicates accept state
I accepts e.g., 11/3 and 3/12
I also accepts 37/00 — overgeneration
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Recursive FSA
comma-separated list of day/month pairs:
0,1,2,3 digit / 0,1 0,1,2
1 2 3 4 5 6
digit digit
,
I list of indefinite length
I e.g., 11/3, 5/6, 12/04
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Finite state transducer
e:e
other : other
ε: ˆ s:s
1 2 3
e:e s:s
other : other x:x e: ˆ s
z:z
ε → e/ x ˆ s
z
4
surface : underlying
s:s cakes↔cakeˆs
x:x
z:z boxes↔boxˆs
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
b:b
ε: ˆ
1 2 3
Input: b
4 Output: b
(Plus: . ˆ)
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
b:b
ε: ˆ
1 2 3
Input: b
4 Output: b
(Plus: . ˆ)
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
o:o
1 2 3
4 Input: b o
Output: b o
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
1 2 3
x:x
4 Input: b o x
Output: b o x
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
1 2 3
e:e
e: ˆ
Input: b o x e
4 Output: b o x ˆ
Output: b o x e
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
ε: ˆ
1 2 3
Input: b o x e
Output: b o x ˆ
Output: b o x e
4 Input: b o x e
Output: b o x e ˆ
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
s:s
1 2 3
s:s Input: b o x e s
Output: b o x ˆ s
Output: b o x e s
4 Input: b o x e s
Output: b o x e ˆ s
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Analysing b o x e s
e:e
other : other
ε: ˆ s:s
1 2 3
e:e s:s Input: b o x e s
other : other x:x e: ˆ
z:z Accept output: b o x ˆ s
Accept output: b o x e s
4 Input: b o x e s
Accept output: b o x e ˆ s
s:s
x:x
z:z
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
Finite state techniques.
Using FSTs
I FSTs assume tokenization (word boundaries) and words
split into characters. One character pair per transition!
I Analysis: return character list with affix boundaries, so
enabling lexical lookup.
I Generation: input comes from stem and affix lexicons.
I One FST per spelling rule: either compile to big FST or run
in parallel.
I FSTs do not allow for internal structure:
I can’t model un- ion -ize -d bracketing.
I can’t condition on prior transitions, so potential redundancy
(cf 2006/7 exam q)
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
More applications for finite state techniques.
Some other uses of finite state techniques in NLP
I Grammars for simple spoken dialogue systems (directly
written or compiled)
I Partial grammars for named entity recognition
I Dialogue models for spoken dialogue systems (SDS)
e.g. obtaining a date:
1. No information. System prompts for month and day.
2. Month only is known. System prompts for day.
3. Day only is known. System prompts for month.
4. Month and day known.
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
More applications for finite state techniques.
Example FSA for dialogue
mumble
1
mumble mumble
month day
2 day & 3
month
day month
4
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
More applications for finite state techniques.
Example of probabilistic FSA for dialogue
0.1
1
0.1 0.2
0.5 0.1
2 0.3 3
0.9 0.8
4
Natural Language Processing
Lecture 2: Morphology and finite state techniques.
More applications for finite state techniques.
Next lecture
Lecture 3: Prediction and part-of-speech tagging.
Corpora in NLP
Word prediction
Part-of-speech (POS) tagging
Evaluation in general, evaluation of POS tagging
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Outline of today’s lecture
Lecture 3: Prediction and part-of-speech tagging.
Corpora in NLP
Word prediction
Part-of-speech (POS) tagging
Evaluation in general, evaluation of POS tagging
First of three lectures that concern syntax (i.e., how words fit
together). This lecture: ‘shallow’ syntax: word sequences and
POS tags. Next lectures: more detailed syntactic structures.
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Corpora in NLP
Corpora
Changes in NLP research over the last 15-20 years are largely
due to increased availability of electronic corpora.
I corpus: text that has been collected for some purpose.
I balanced corpus: texts representing different genres
genre is a type of text (vs domain)
I tagged corpus: a corpus annotated with POS tags
I treebank: a corpus annotated with parse trees
I specialist corpora — e.g., collected to train or evaluate
particular applications
I Movie reviews for sentiment classification
I Data collected from simulation of a dialogue system
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Word prediction
Prediction
Guess the missing words:
Illustrations produced by any package can be transferred with
consummate to another.
Wright tells her story with great .
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Word prediction
Prediction
Guess the missing words:
Illustrations produced by any package can be transferred with
consummate ease to another.
Wright tells her story with great .
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Word prediction
Prediction
Guess the missing words:
Illustrations produced by any package can be transferred with
consummate ease to another.
Wright tells her story with great professionalism .
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Word prediction
Prediction
Prediction is relevant for:
I language modelling for speech recognition: e.g., using
N-grams. This is an alternative to finite state grammars,
suitable for large-scale recognition
I word prediction for communication aids (augmentative and
alternative communication). e.g., to help enter text that’s
input to a synthesiser
I text entry on mobile phones and similar devices
I OCR, spelling correction, text segmentation
I estimation of entropy
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Word prediction
bigrams (N-gram with N=2)
A probability is assigned to a word based on the previous word:
P(wn |wn−1 )
where wn is the nth word in a sentence.
Probability of a sequence of words (assuming independence):
n
P(W1n ) ≈ P(wk |wk −1 )
Y
k =1
Probability is estimated from counts in a training corpus:
C(wn−1 wn ) C(wn−1 wn )
≈
w C(wn−1 w) C(wn−1 )
P
i.e. count of a particular bigram in the corpus divided by the
count of all bigrams starting with the prior word.
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Word prediction
Calculating bigrams
hsi good morning hsi good afternoon hsi good afternoon hsi it
is very good hsi it is good hsi
sequence count bigram probability
hsi 5
hsi good 3 .6
hsi it 2 .4
good 5
good morning 1 .2
good afternoon 2 .4
good hsi 2 .4
morning 1
morning hsi 1 1
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Word prediction
Practical application
I word prediction: guess the word from initial letters (user
confirms each word)
I speech recognition: maximize likelihood of a sequence
(implemented using the Viterbi algorithm)
Problems because of sparse data:
I smoothing: distribute ‘extra’ probability between rare and
unseen events
I backoff: approximate unseen probabilities by a more
general probability, e.g. unigrams
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Part of speech tagging
They can fish .
I They_PNP can_VM0 fish_VVI ._PUN
I They_PNP can_VVB fish_NN2 ._PUN
I They_PNP can_VM0 fish_NN2 ._PUN no full parse
POS lexicon fragment:
they PNP
can VM0 VVB VVI NN1
fish NN1 NN2 VVB VVI
tagset (CLAWS 5) includes:
NN1 singular noun NN2 plural noun
PNP personal pronoun VM0 modal auxiliary verb
VVB base form of verb VVI infinitive form of verb
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Part of speech tagging
I They_PNP can_VM0 fish_VVI ._PUN
I They_PNP can_VVB fish_NN2 ._PUN
I They_PNP can_VM0 fish_NN2 ._PUN no full parse
POS lexicon fragment:
they PNP
can VM0 VVB VVI NN1
fish NN1 NN2 VVB VVI
tagset (CLAWS 5) includes:
NN1 singular noun NN2 plural noun
PNP personal pronoun VM0 modal auxiliary verb
VVB base form of verb VVI infinitive form of verb
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Part of speech tagging
I They_PNP can_VM0 fish_VVI ._PUN
I They_PNP can_VVB fish_NN2 ._PUN
I They_PNP can_VM0 fish_NN2 ._PUN no full parse
POS lexicon fragment:
they PNP
can VM0 VVB VVI NN1
fish NN1 NN2 VVB VVI
tagset (CLAWS 5) includes:
NN1 singular noun NN2 plural noun
PNP personal pronoun VM0 modal auxiliary verb
VVB base form of verb VVI infinitive form of verb
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Why POS tag?
I Coarse-grained syntax / word sense disambiguation: most
approaches are quick to run, so applicable to very large
corpora.
I Potentially useful for some linguistic research and for
lexicography: e.g., how often is tango used as a verb?
dog?
I Basis for named entity recognition and similar tasks (finite
state patterns over POS tagged data).
I Features for machine learning e.g., sentiment
classification. (e.g., stink_V vs stink_N)
I Preliminary processing for full parsing: cut down search
space or provide guesses at unknown words.
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Stochastic part of speech tagging
1. Start with untagged text.
2. Assign all possible tags to each word in the text on the
basis of a lexicon that associates words and tags.
3. Find the most probable sequence (or n-best sequences) of
tags, based on probabilities from the training data.
I lexical probability: e.g., is can most likely to be VM0, VVB,
VVI or NN1?
I and tag sequence probabilities: e.g., is VM0 or NN1 more
likely after PNP?
Note: tags are more fine-grained than conventional part of
speech. Different possible tagsets: i.e., sets of POS tags.
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Training stochastic POS tagging
They_PNP used_VVD to_TO0 can_VVI fish_NN2 in_PRP
those_DT0 towns_NN2 ._PUN But_CJC now_AV0 few_DT0
people_NN2 fish_VVB in_PRP these_DT0 areas_NN2
._PUN
sequence count bigram probability
NN2 4
NN2 PRP 1 0.25
NN2 PUN 2 0.5
NN2 VVB 1 0.25
Also lexicon: fish NN2 VVB
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Training stochastic POS tagging
They_PNP used_VVD to_TO0 can_VVI fish_NN2 in_PRP
those_DT0 towns_NN2 ._PUN But_CJC now_AV0 few_DT0
people_NN2 fish_VVB in_PRP these_DT0 areas_NN2
._PUN
sequence count bigram probability
NN2 4
NN2 PRP 1 0.25
NN2 PUN 2 0.5
NN2 VVB 1 0.25
Also lexicon: fish NN2 VVB
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Training stochastic POS tagging
They_PNP used_VVD to_TO0 can_VVI fish_NN2 in_PRP
those_DT0 towns_NN2 ._PUN But_CJC now_AV0 few_DT0
people_NN2 fish_VVB in_PRP these_DT0 areas_NN2
._PUN
sequence count bigram probability
NN2 4
NN2 PRP 1 0.25
NN2 PUN 2 0.5
NN2 VVB 1 0.25
Also lexicon: fish NN2 VVB
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Assigning probabilities
More complex than word prediction, because looking at words
and tags. P(T |W ) is prob of tag sequence T, given word
sequence W, but can’t estimate this directly.
So, applying Bayes theorem:
P(T )P(W |T )
P(T |W ) =
P(W )
P(W ) is constant (tagging a known sequence)
estimate P(T ) as P(ti |ti−1 ) (bigram assumption)
estimate P(W |T ) as P(wi |ti ) (i.e., the probability of each word
given its tag)
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Example
Tagging: they fish
Assume P(PNP|they) = 1.
Then sequence probability depends on:
P(NN1|PNP)P(fish|NN1)
P(NN2|PNP)P(fish|NN2)
P(VVB|PNP)P(fish|VVB)
etc
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Part-of-speech (POS) tagging
Assigning probabilities, more details
I Maximise the overall tag sequence probability — e.g., use
Viterbi.
they_PNP can_VVB fish_NN2
they_PNP can_VM0 fish_VVI
P(VVI|VM0)P(fish|VVI) may be lower than
P(NN2|VVB)P(fish|NN2) but P(VVB|PNP)P(can|VVB)
versus P(VM0|PNP)P(can|VM0)
I Actual systems use trigrams — smoothing and backoff are
critical.
I Unseen words: these are not in the lexicon, so use all
possible open class tags, possibly restricted by
morphology.
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Evaluation in general, evaluation of POS tagging
Evaluation of POS tagging
I percentage of correct tags
I one tag per word (some systems give multiple tags when
uncertain)
I over 95% for English (but note punctuation is
unambiguous)
I baseline of taking the most common tag gives 90%
accuracy
I different tagsets give slightly different results: utility of tag
to end users vs predictive power (an open research issue)
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Evaluation in general, evaluation of POS tagging
Evaluation in general
I Training data and test data Test data must be kept unseen,
often 90% training and 10% test data.
I Baseline
I Ceiling Human performance on the task, where the ceiling
is the percentage agreement found between two
annotators (interannotator agreement)
I Error analysis Error rates are nearly always unevenly
distributed.
I Reproducibility
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Evaluation in general, evaluation of POS tagging
Representative corpora and data sparsity
I test corpora have to be representative of the actual
application
I POS tagging and similar techniques are not always very
robust to differences in genre
I balanced corpora may be better, but still don’t cover all text
types
I communication aids: extreme difficulty in obtaining data,
text corpora don’t give good prediction for real data
Natural Language Processing
Lecture 3: Prediction and part-of-speech tagging.
Evaluation in general, evaluation of POS tagging
Outline of next lecture
Lecture 4: Parsing and generation.
Generative grammar
Simple context free grammars
Random generation with a CFG
Simple chart parsing with CFGs
More advanced chart parsing
Why not finite state?
Natural Language Processing
Lecture 4: Parsing and generation.
Parsing (and generation)
Syntactic structure in analysis:
I as a step in assigning semantics
I checking grammaticality
I corpus-based investigations, lexical acquisition etc
Lecture 4: Parsing and generation.
Generative grammar
Simple context free grammars
Random generation with a CFG
Simple chart parsing with CFGs
More advanced chart parsing
Why not finite state?
Next lecture — beyond simple CFGs
Natural Language Processing
Lecture 4: Parsing and generation.
Generative grammar
Generative grammar
a formally specified grammar that can generate all and only the
acceptable sentences of a natural language
Internal structure:
the big dog slept
can be bracketed
((the (big dog)) slept)
constituent a phrase whose components ‘go together’ . . .
weak equivalence grammars generate the same strings
strong equivalence grammars generate the same strings with
same brackets
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
Context free grammars
1. a set of non-terminal symbols (e.g., S, VP);
2. a set of terminal symbols (i.e., the words);
3. a set of rules (productions), where the LHS (mother) is a
single non-terminal and the RHS is a sequence of one or
more non-terminal or terminal symbols (daughters);
S -> NP VP
V -> fish
4. a start symbol, conventionally S, which is a non-terminal.
Exclude empty productions, NOT e.g.:
NP ->
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
A simple CFG for a fragment of English
lexicon
rules V -> can
V -> fish
S -> NP VP
NP -> fish
VP -> VP PP
NP -> rivers
VP -> V
NP -> pools
VP -> V NP
NP -> December
VP -> V VP
NP -> Scotland
NP -> NP PP
NP -> it
PP -> P NP
NP -> they
P -> in
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
Analyses in the simple CFG
they fish
(S (NP they) (VP (V fish)))
they can fish
(S (NP they) (VP (V can) (VP (V fish))))
(S (NP they) (VP (V can) (NP fish)))
they fish in rivers
(S (NP they) (VP (VP (V fish))
(PP (P in) (NP rivers))))
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
Analyses in the simple CFG
they fish
(S (NP they) (VP (V fish)))
they can fish
(S (NP they) (VP (V can) (VP (V fish))))
(S (NP they) (VP (V can) (NP fish)))
they fish in rivers
(S (NP they) (VP (VP (V fish))
(PP (P in) (NP rivers))))
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
Analyses in the simple CFG
they fish
(S (NP they) (VP (V fish)))
they can fish
(S (NP they) (VP (V can) (VP (V fish))))
(S (NP they) (VP (V can) (NP fish)))
they fish in rivers
(S (NP they) (VP (VP (V fish))
(PP (P in) (NP rivers))))
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
Structural ambiguity without lexical ambiguity
they fish in rivers in December
(S (NP they)
(VP (VP (V fish))
(PP (P in) (NP (NP rivers)
(PP (P in) (NP December))))))
(S (NP they)
(VP (VP (VP (V fish))
(PP (P in) (NP (NP rivers))))
(PP (P in) (NP December))))
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
Structural ambiguity without lexical ambiguity
they fish in rivers in December
(S (NP they)
(VP (VP (V fish))
(PP (P in) (NP (NP rivers)
(PP (P in) (NP December))))))
(S (NP they)
(VP (VP (VP (V fish))
(PP (P in) (NP (NP rivers))))
(PP (P in) (NP December))))
Natural Language Processing
Lecture 4: Parsing and generation.
Simple context free grammars
Parse trees
S
NP VP
they V VP
can VP PP
V P NP
fish in December
(S (NP they)
(VP (V can)
(VP (VP (V fish))
(PP (P in)
(NP December)))))
Natural Language Processing
Lecture 4: Parsing and generation.
Random generation with a CFG
Using a grammar as a random generator
Expand cat category sentence-record:
Let possibilities be all lexical items matching category
and all rules with LHS category
If possibilities is empty,
then fail
else
Randomly select a possibility chosen from possibilities
If chosen is lexical,
then append it to sentence-record
else
expand cat on each rhs category in chosen
(left to right) with the updated sentence-record
return sentence-record
Natural Language Processing
Lecture 4: Parsing and generation.
Random generation with a CFG
Random generator example
Expand cat S ()
possibilities = {S -> NP VP}, chosen = S -> NP VP
Expand cat NP ()
possibilities = {it, they, fish}
chosen = fish
sentence-record = (fish)
Expand cat VP (fish)
possibilities ={VP -> V, VP -> V VP, VP -> V NP}
chosen = VP -> V
Expand cat V (fish)
possibilities = {fish, can}
chosen = fish
sentence-record = (fish fish)
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Chart parsing
A dynamic programming algorithm (memoisation):
chart store partial results of parsing
edge representation of a rule application
Edge data structure:
[id,left_vtx, right_vtx,mother_category, dtrs]
. they . can . fish .
0 1 2 3
Fragment of chart:
id l r ma dtrs
5 2 3 V (fish)
6 2 3 VP (5)
7 1 3 VP (3 6)
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
A bottom-up passive chart parser
Parse:
Initialize the chart
For each word word, let from be left vtx,
to right vtx and dtrs be (word)
For each category category
lexically associated with word
Add new edge from, to, category, dtrs
Output results for all spanning edges
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Inner function
Add new edge from, to, category, dtrs:
Put edge in chart: [id,from,to, category,dtrs]
For each rule lhs → cat1 . . . catn−1 ,category
Find sets of contiguous edges
[id1 ,from1 ,to1 , cat1 ,dtrs1 ] . . .
[idn−1 ,fromn−1 ,from, catn−1 ,dtrsn−1 ]
(such that to1 = from2 etc)
For each set of edges,
Add new edge from1 , to, lhs, (id1 . . . id)
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Bottom up parsing: edges
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
word = they, categories = {NP}
Add new edge 0, 1, NP, (they)
Matching grammar rules: {VP→V NP, PP→P NP}
No matching edges corresponding to V or P
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
word = can, categories = {V}
Add new edge 1, 2, V, (can)
Matching grammar rules: {VP→V}
recurse on edges {(2)}
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 1, 2, VP, (2)
Matching grammar rules: {S→NP VP, VP→V VP}
recurse on edges {(1,3)}
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 0, 2, S, (1, 3)
No matching grammar rules for S
Matching grammar rules: {S→NP VP, VP→V VP}
No edges for V VP
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
word = fish, categories = {V, NP}
Add new edge 2, 3, V, (fish) NB: fish as V
Matching grammar rules: {VP→V}
recurse on edges {(5)}
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 2, 3, VP, (5)
Matching grammar rules: {S →NP VP, VP →V VP}
No edges match NP
recurse on edges for V VP: {(2,6)}
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 1, 3, VP, (2, 6)
Matching grammar rules: {S→NP VP, VP→V VP}
recurse on edges for NP VP: {(1,7)}
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 0, 3, S, (1, 7)
No matching grammar rules for S
Matching grammar rules: {S→NP VP, VP →V VP}
No edges matching V
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 2, 3, NP, (fish) NB: fish as NP
Matching grammar rules: {VP→V NP, PP→P NP}
recurse on edges for V NP {(2,9)}
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 1, 3, VP, (2, 9)
Matching grammar rules: {S→NP VP, VP→V VP}
recurse on edges for NP VP: {(1, 10)}
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Parse construction
11:S
10:VP
8:S 9:NP
4:S 7:VP
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Add new edge 0, 3, S, (1, 10)
No matching grammar rules for S
Matching grammar rules: {S→NP VP, VP→V VP}
No edges corresponding to V VP
Matching grammar rules: {VP→V NP, PP→P NP}
No edges corresponding to P NP
Natural Language Processing
Lecture 4: Parsing and generation.
Simple chart parsing with CFGs
Output results for spanning edges
Spanning edges are 8 and 11:
Output results for 8
(S (NP they) (VP (V can) (VP (V fish))))
Output results for 11
(S (NP they) (VP (V can) (NP fish)))
Note: sample chart parsing code in Java is downloadable from
the course web page.
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Packing
I exponential number of parses means exponential time
I body can be cubic time: don’t add equivalent edges as
whole new edges
I dtrs is a set of lists of edges (to allow for alternatives)
about to add: [id,l_vtx, right_vtx,ma_cat, dtrs]
and there is an existing edge:
[id-old,l_vtx, right_vtx,ma_cat, dtrs-old]
we simply modify the old edge to record the new dtrs:
[id-old,l_vtx, right_vtx,ma_cat, dtrs-old ∪ dtrs]
and do not recurse on it
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Packing example
1 0 1 NP {(they)}
2 1 2 V {(can)}
3 1 2 VP {(2)}
4 0 2 S {(1 3)}
5 2 3 V {(fish)}
6 2 3 VP {(5)}
7 1 3 VP {(2 6)}
8 0 3 S {(1 7)}
9 2 3 NP {(fish)}
Instead of edge 10 1 3 VP {(2 6)}
7 1 3 VP {(2 6), (2 9)}
and we’re done
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Packing example
8:S 9:NP
4:S 7:VP+
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Both spanning results can now be extracted from edge 8.
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Packing example
10:VP
8:S 9:NP
4:S 7:VP+
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Both spanning results can now be extracted from edge 8.
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Packing example
8:S 9:NP
4:S 7:VP+
3:VP 6:VP
1:NP 2:V 5:V
they can fish
Both spanning results can now be extracted from edge 8.
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Active chart parsing
2:S/VP
1:NP
they can fish
id l r ma exp dtrs
1 0 1 NP (they)
2 0 1 S VP (1 ?)
I store partial rule applications
I record expected input as well as seen
I one active edge can create more than one passive edge.
e.g., they fish in Scotland — edge 2 completed by fish and
fish in Scotland. NP is combined with rule once not twice.
I active edges can be packed
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Active chart parsing
2:S/VP
1:NP
they can fish
id l r ma exp dtrs
1 0 1 NP (they)
2 0 1 S VP (1 ?)
I store partial rule applications
I record expected input as well as seen
I one active edge can create more than one passive edge.
e.g., they fish in Scotland — edge 2 completed by fish and
fish in Scotland. NP is combined with rule once not twice.
I active edges can be packed
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Active chart parsing
2:S/VP
1:NP
they can fish
id l r ma exp dtrs
1 0 1 NP (they)
2 0 1 S VP (1 ?)
I store partial rule applications
I record expected input as well as seen
I one active edge can create more than one passive edge.
e.g., they fish in Scotland — edge 2 completed by fish and
fish in Scotland. NP is combined with rule once not twice.
I active edges can be packed
Natural Language Processing
Lecture 4: Parsing and generation.
More advanced chart parsing
Ordering the search space
I agenda: order edges in chart by priority
I top-down parsing: predict possible edges
Producing n-best parses:
I manual weight assignment
I probabilistic CFG — trained on a treebank
I automatic grammar induction
I automatic weight assignment to existing grammar
I beam-search
Natural Language Processing
Lecture 4: Parsing and generation.
Why not finite state?
Why not FSA?
centre-embedding:
A → αAβ
generate grammars of the form an bn .
For instance:
the students the police arrested complained
However:
? the students the police the journalists criticised arrested
complained
Limits on human memory / processing ability
Natural Language Processing
Lecture 4: Parsing and generation.
Why not finite state?
Why not FSA practically?
More importantly for practical application:
1. FSM grammars are very redundant: difficult to build and
maintain
2. FSM grammars don’t support composition of semantics
but FSMs useful for:
1. tokenizers (dates, times etc)
2. named entity recognition in information extraction etc
3. approximating CFGs in speech recognition
Natural Language Processing
Lecture 4: Parsing and generation.
Why not finite state?
Outline of next lecture
Providing a more adequate treatment of syntax than simple
CFGs: replacing the atomic categories by more complex data
structures.
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisatio
Feature structures (informally)
Encoding agreement
Parsing with feature structures
Feature stuctures more formally
Encoding subcategorisation
Interface to morphology
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Outline of today’s lecture
Providing a more adequate treatment of syntax than simple
CFGs: replacing the atomic categories by more complex data
structures.
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisatio
Feature structures (informally)
Encoding agreement
Parsing with feature structures
Feature stuctures more formally
Encoding subcategorisation
Interface to morphology
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisation, long distance dependencies.
Overgeneration in atomic category CFGs
I agreement: subject verb agreement. e.g., they fish, it
fishes, *it fish, *they fishes. * means ungrammatical
I case: pronouns (and maybe who/whom) e.g., they like
them, *they like they
S -> NP-sg-nom VP-sg NP-sg-nom -> he
S -> NP-pl-nom VP-pl NP-sg-acc -> him
VP-sg -> V-sg NP-sg-acc NP-sg-nom -> fish
VP-sg -> V-sg NP-pl-acc NP-pl-nom -> fish
VP-pl -> V-pl NP-sg-acc NP-sg-acc -> fish
VP-pl -> V-pl NP-pl-acc NP-pl-acc -> fish
BUT: very large grammar, misses generalizations, no way of
saying when we don’t care about agreement.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisation, long distance dependencies.
Overgeneration in atomic category CFGs
I agreement: subject verb agreement. e.g., they fish, it
fishes, *it fish, *they fishes. * means ungrammatical
I case: pronouns (and maybe who/whom) e.g., they like
them, *they like they
S -> NP-sg-nom VP-sg NP-sg-nom -> he
S -> NP-pl-nom VP-pl NP-sg-acc -> him
VP-sg -> V-sg NP-sg-acc NP-sg-nom -> fish
VP-sg -> V-sg NP-pl-acc NP-pl-nom -> fish
VP-pl -> V-pl NP-sg-acc NP-sg-acc -> fish
VP-pl -> V-pl NP-pl-acc NP-pl-acc -> fish
BUT: very large grammar, misses generalizations, no way of
saying when we don’t care about agreement.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisation, long distance dependencies.
Intuitive solution for case and agreement
I Separate slots (features) for CASE and AGR
I Slot values for CASE may be nom (e.g., they), acc (e.g.,
them) or unspecified (i.e., don’t care)
I Slot values for AGR may be sg, pl or unspecified
I Subjects have the same value for AGR as their verbs
ISubjects have CASE nom, objects have CASE acc
CASE [ ] CASE [ ]
can (n) fish (n)
AGR sg
AGR [ ]
CASE nom CASE acc
she them
AGR sg AGR pl
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisation, long distance dependencies.
Subcategorization
I intransitive vs transitive etc
I verbs (and other types of words) have different numbers
and types of syntactic arguments:
*Kim adored
*Kim gave Sandy
*Kim adored to sleep
Kim liked to sleep
*Kim devoured
Kim ate
I Subcategorization is correlated with semantics, but not
determined by it.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisation, long distance dependencies.
Overgeneration because of missing subcategorization
Overgeneration:
they fish fish it
(S (NP they) (VP (V fish) (VP (V fish) (NP it))))
I Informally: need slots on the verbs for their syntactic
arguments.
I intransitive takes no following arguments (complements)
I simple transitive takes one NP complement
I like may be a simple transitive or take an infinitival
complement, etc
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Problems with simple CFG encoding: agreement, subcategorisation, long distance dependencies.
Long-distance dependencies
1. which problem did you say you don’t understand?
2. who do you think Kim asked Sandy to hit?
3. which kids did you say were making all that noise?
‘gaps’ (underscores below)
1. which problem did you say you don’t understand _?
2. who do you think Kim asked Sandy to hit _?
3. which kids did you say _ were making all that noise?
In 3, the verb were shows plural agreement.
* what kid did you say _ were making all that noise?
The gap filler has to be plural.
I Informally: need a ‘gap’ slot which is to be filled by
something that itself has features.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Feature structures (informally)
Feature structures
CASE [ ]
AGR sg
1. Features like AGR with simple values: atomic-valued
2. Unspecified values possible on features: compatible with
any value.
3. Values for features for subcat and gap themselves have
features: complex-valued
4. path: a sequence of features
5. Method of specifying two paths are the same: reentrancy
6. Unification: combining two feature structures, retaining all
information from each, or fail if information is incompatible.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Feature structures (informally)
Feature structures, continued
I Feature structures are singly-rooted directed acyclic
graphs, with arcs labelled by features and terminal nodes
associated with values.
CASE
-
CASE [ ]
AGR
AGR sg
sg
j
I In grammars, rules relate FSs — i.e. lexical entries and
phrases are represented as FSs
I Rule application by unification
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Feature structures (informally)
Graphs and AVMs
Example 1: CAT -NP
CAT NP
AGR
sg
jsg
AGR
Here, CAT and AGR are atomic-valued features. NP and sg are
values.
Example 2:
HEAD- CAT -NP
HEAD CAT NP
AGR
AGR []
j
HEAD is complex-valued, AGR is unspecified.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Feature structures (informally)
Reentrancy
: a
F
F a
a
G
G - a
F F 0 a
G G 0
a z
-
Reentrancy indicated by boxed integer in AVM diagram:
indicates path goes to the same node.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding agreement
CFG with agreement
S -> NP-sg VP-sg
S -> NP-pl VP-pl
VP-sg -> V-sg NP-sg
VP-sg -> V-sg NP-pl
VP-pl -> V-pl NP-sg
VP-pl -> V-pl NP-pl
V-pl -> like
V-sg -> likes
NP-sg -> it
NP-pl -> they
NP-sg -> fish
NP-pl -> fish
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding agreement
FS grammar fragment encoding agreement
CAT S CAT NP CAT VP
subj-verb rule → ,
1 1 1
AGR AGR AGR
CAT VP CAT V CAT NP
verb-obj rule → ,
AGR 1 AGR 1 AGR []
Root structure: S
h i
CAT
CAT NP NP V
they it CAT likes CAT
AGR pl sg sg
AGR AGR
CAT NP V
fish like CAT
pl
AGR [ ] AGR
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Parsing with feature structures
Parsing ‘they like it’
I The lexical structures for like and it are unified with the
corresponding structures on the right hand side of the
verb-obj rule (unifications succeed).
I The structure corresponding to the mother of the rule is
then:
CAT VP
AGR pl
I This unifies with the rightmost daughter position of the
subj-verb rule.
I The structure for they is unified with the leftmost daughter.
I The result unifies with root structure.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Parsing with feature structures
Rules as FSs
Rules have features MOTHER , DTR 1, DTR 2 . . . DTRN.
CAT VP CAT V CAT NP
informally: → ,
AGR 1 AGR 1 AGR [ ]
MOTHER VP CAT
AGR 1
CAT V
actually: DTR 1
1
AGR
DTR 2 CAT NP
AGR [ ]
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Parsing with feature structures
Verb-obj rule application
Feature structure for like unified with the value of DTR 1:
CAT VP
MOTHER AGR 1 pl
DTR 1 CAT V
AGR 1
CAT NP
DTR 2
AGR [ ]
Feature structure for it unified with the value for DTR 2:
CAT VP
MOTHER AGR 1 pl
DTR 1 CAT V
AGR 1
CAT NP
DTR 2
AGR sg
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Parsing with feature structures
Subject-verb rule application 1
MOTHER value from the verb-object rule acts as the DTR 2 of the
subject-verb rule:
CAT S
MOTHER AGR 1
CAT VP CAT NP
unified with the DTR 2 of: DTR 1
pl AGR 1
AGR
CAT VP
DTR 2
AGR 1
Gives:
S CAT
MOTHER AGR 1 pl
DTR 1 CAT NP
1
AGR
CAT VP
DTR 2
AGR 1
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Parsing with feature structures
Subject rule application 2
CAT NP
FS for they:
AGR pl
Unification of this with the value of DTR 1 succeeds (but adds no
new information):
CAT S
MOTHER AGR 1 pl
DTR 1 CAT NP
AGR 1
CAT VP
DTR 2
AGR 1
Final structure unifies with the root structure: CAT S
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Feature stuctures more formally
Properties of FSs
Connectedness and unique root A FS must have a unique root
node: apart from the root node, all nodes have
one or more parent nodes.
Unique features Any node may have zero or more arcs leading
out of it, but the label on each (that is, the feature)
must be unique.
No cycles No node may have an arc that points back to the
root node or to a node that intervenes between it
and the root node.
Values A node which does not have any arcs leading out
of it may have an associated atomic value.
Finiteness A FS must have a finite number of nodes.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Feature stuctures more formally
Subsumption
Feature structures are ordered by information content — FS1
subsumes FS2 if FS2 carries extra information.
FS1 subsumes FS2 if and only if the following conditions hold:
Path values For every path P in FS1 there is a path P in FS2. If
P has a value t in FS1, then P also has value t in
FS2.
Path equivalences Every pair of paths P and Q which are
reentrant in FS1 (i.e., which lead to the same node
in the graph) are also reentrant in FS2.
Unification
The unification of two FSs FS1 and FS2 is the most general FS
which is subsumed by both FS1 and FS2, if it exists.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Grammar with subcategorisation
1 HEAD 1
HEAD
Verb-obj rule: OBJ filled → OBJ 2 , 2 OBJ filled
SUBJ 3 SUBJ 3
verb CAT
HEAD AGR pl
can (transitive verb): HEAD CAT noun
OBJ
OBJ filled
SUBJ HEAD CAT noun
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Grammar with subcategorisation (abbrev for slides)
1 HEAD 1
HEAD
Verb-obj rule: OBJ fld → OBJ 2 , 2 OBJ fld
SUBJ 3 SUBJ 3
v CAT
HEAD AGR pl
can (transitive verb): HEAD CAT n
OBJ
fld
OBJ
SUBJ HEAD CAT n
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
S
NP VP
V VP
VP PP
V P NP
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
S+
NP VP+
V VP
VP PP
V P NP
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
S
NP VP+
V+ VP
VP PP
V P NP
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
S
NP VP
V VP+
VP+ PP
V P NP
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
S
NP VP
V VP
VP+ PP
V+ P NP
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
S
NP VP
V VP
VP PP+
V P+ NP
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
I SUBJ:
The subject-verb rule unifies the first daughter of the rule
with the SUBJ value of the second. (‘the first dtr fills the
SUBJ slot of the second dtr in the rule’)
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Concepts for subcategorisation
I HEAD: information shared between a lexical entry and the
dominating phrases of the same category
I SUBJ:
The subject-verb rule unifies the first daughter of the rule
with the SUBJ value of the second. (‘the first dtr fills the
SUBJ slot of the second dtr in the rule’)
I OBJ:
The verb-object rule unifies the second dtr with the OBJ
value of the first. (‘the second dtr fills the OBJ slot of the
first dtr in the rule’)
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Example rule application: they fish 1
CAT v
HEAD
AGR pl
Lexical entry for fish:
OBJ fld
SUBJ HEAD CAT n
subject-verb rule:
HEAD 1 HEAD AGR 3 1
HEAD AGR 3
OBJ fld → 2 OBJ fld , OBJ fld
SUBJ fld SUBJ fld SUBJ 2
unification with second dtr position gives:
CAT v CAT n
HEAD 1 HEAD AGR 3 HEAD 1
AGR 3 pl
→ 2 , OBJ fld
OBJ fld OBJ fld
2
SUBJ
SUBJ fld SUBJ fld
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
CAT n
HEAD AGR pl
Lexical entry for they:
OBJ fld
SUBJ fld
unify this with first dtr position:
CAT v CAT n
HEAD 1 HEAD AGR 3 HEAD 1
AGR 3 pl
→ 2 , OBJ fld
OBJ fld OBJ fld
2
SUBJ
SUBJ fld SUBJ fld
HEAD CAT v
Root is: OBJ fld
SUBJ fld
Mother structure unifies with root, so valid.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Encoding subcategorisation
Parsing with feature structure grammars
I Naive algorithm: standard chart parser with modified rule
application
I Rule application:
1. copy rule
2. copy daughters (lexical entries or FSs associated with
edges)
3. unify rule and daughters
4. if successful, add new edge to chart with rule FS as
category
I Efficient algorithms reduce copying.
I Packing involves subsumption.
I Probabilistic FS grammars are complex.
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Interface to morphology
Templates
Capture generalizations in the lexicon:
fish INTRANS_VERB
sleep INTRANS_VERB
snore INTRANS_VERB
HEAD CAT v
AGR pl
INTRANS_VERB
OBJ fld
i
n
h
SUBJ HEAD CAT
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Interface to morphology
Interface to morphology: inflectional affixes as FSs
HEAD CAT n
s
AGR pl
HEAD CAT n
AGR []
if stem is:
OBJ fld
SUBJ fld
stem unifies with affix template.
But unification failure would occur with verbs etc, so we get
filtering (lecture 2).
Natural Language Processing
Lecture 5: Parsing with constraint-based grammars.
Interface to morphology
Outline of next lecture
Compositional semantics: the construction of meaning
(generally expressed as logic) based on syntax.
Lexical semantics: the meaning of individual words.
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Logical forms
Meaning postulates
Lexical semantics: semantic relations
Polysemy
Word sense disambiguation
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Outline of today’s lecture
Compositional semantics: the construction of meaning
(generally expressed as logic) based on syntax.
Lexical semantics: the meaning of individual words.
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Logical forms
Meaning postulates
Lexical semantics: semantic relations
Polysemy
Word sense disambiguation
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Simple compositional semantics in feature structures
I Semantics is built up along with syntax
I Subcategorization ‘slot’ filling instantiates syntax
I Formally equivalent to logical representations (below:
predicate calculus with no quantifiers)
I Alternative FS encodings possible
Objective: obtain the following semantics for they like fish:
pron(x) ∧ (like_v(x, y ) ∧ fish_n(y ))
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Feature structure encoding of semantics
and
PRED
pron
PRED
ARG 1
ARG 1 1
PRED and
PRED like_v
ARG 1
ARG 1 1
ARG 2
ARG 2 2
PRED fish_n
ARG 2
ARG 1 2
pron(x) ∧ (like_v(x, y ) ∧ fish_n(y ))
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Noun entry
HEAD CAT
n
AGR []
OBJ fld
fish
SUBJ fld
INDEX 1
SEM PRED fish_n
ARG 1 1
I Corresponds to fish(x) where the INDEX points to the
characteristic variable of the noun (that is x).
The INDEX is unambiguous here, but
e.g., picture(x, y ) ∧ sheep(y )
picture of sheep
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Noun entry
HEAD CAT
n
AGR []
OBJ fld
fish
SUBJ fld
INDEX 1
SEM PRED fish_n
ARG 1 1
I Corresponds to fish(x) where the INDEX points to the
characteristic variable of the noun (that is x).
The INDEX is unambiguous here, but
e.g., picture(x, y ) ∧ sheep(y )
picture of sheep
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Verb entry
HEAD CAT v
AGR pl
CAT n
h i
HEAD
OBJ fld
OBJ
h i
INDEX 2
SEM
like
CAT n
h i
HEAD
SUBJ h
i
SEM INDEX 1
PRED like_v
ARG 1 1
SEM
ARG 2 2
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Compositional semantics in feature structures
Verb-object rule
HEAD 1
OBJ fld
HEAD 1
SUBJ 3 → OBJ 2 , 2 OBJ fld
PRED and
SUBJ 3 SEM 5
SEM ARG 1 4 SEM 4
ARG 2 5
I As last time: object of the verb (DTR2) ‘fills’ the OBJ slot
I New: semantics on the mother is the ‘and’ of the semantics
of the dtrs
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Logical forms
Logic in semantic representation
I Meaning representation for a sentence is called the logical
form
I Standard approach to composition in theoretical linguistics
is lambda calculus, building FOPC or higher order
representation.
I Representation in notes is quantifier-free predicate
calculus but possible to build FOPC or higher-order
representation in FSs.
I Theorem proving.
I Generation: starting point is logical form, not string.
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Meaning postulates
Meaning postulates
I e.g.,
∀x[bachelor0 (x) → man0 (x) ∧ unmarried0 (x)]
I usable with compositional semantics and theorem provers
I e.g. from ‘Kim is a bachelor’, we can construct the LF
bachelor0 (Kim)
and then deduce
unmarried0 (Kim)
I OK for narrow domains, but ‘classical’ lexical semantic
relations are more generally useful
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Lexical semantics: semantic relations
Lexical semantic relations
Hyponymy: IS-A:
I (a sense of) dog is a hyponym of (a sense of) animal
I animal is a hypernym of dog
I hyponymy relationships form a taxonomy
I works best for concrete nouns
Meronomy: PART-OF e.g., arm is a meronym of body, steering
wheel is a meronym of car (piece vs part)
Synonymy e.g., aubergine/eggplant
Antonymy e.g., big/little
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Lexical semantics: semantic relations
WordNet
I large scale, open source resource for English
I hand-constructed
I wordnets being built for other languages
I organized into synsets: synonym sets (near-synonyms)
Overview of adj red:
1. (43) red, reddish, ruddy, blood-red, carmine,
cerise, cherry, cherry-red, crimson, ruby,
ruby-red, scarlet - (having any of numerous
bright or strong colors reminiscent of the color
of blood or cherries or tomatoes or rubies)
2. (8) red, reddish - ((used of hair or fur)
of a reddish brown color; "red deer";
reddish hair")
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Lexical semantics: semantic relations
Hyponymy in WordNet
Sense 6
big cat, cat
=> leopard, Panthera pardus
=> leopardess
=> panther
=> snow leopard, ounce, Panthera uncia
=> jaguar, panther, Panthera onca,
Felis onca
=> lion, king of beasts, Panthera leo
=> lioness
=> lionet
=> tiger, Panthera tigris
=> Bengal tiger
=> tigress
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Lexical semantics: semantic relations
Some uses of lexical semantics
I Semantic classification: e.g., for selectional restrictions
(e.g., the object of eat has to be something edible) and for
named entity recognition
I Shallow inference: ‘X murdered Y’ implies ‘X killed Y’ etc
I Back-off to semantic classes in some statistical
approaches
I Word-sense disambiguation
I Machine Translation: if you can’t translate a term,
substitute a hypernym
I Query expansion: if a search doesn’t return enough
results, one option is to replace an over-specific term with
a hypernym
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Polysemy
Polysemy
I homonymy: unrelated word senses. bank (raised land) vs
bank (financial institution)
I bank (financial institution) vs bank (in a casino): related but
distinct senses.
I bank (N) (raised land) vs bank (V) (to create some raised
land): regular polysemy. Compare pile, heap etc
I vagueness: bank (river vs snow vs cloud)?
No clearcut distinctions.
Dictionaries are not consistent.
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
Word sense disambiguation
Needed for many applications, problematic for large domains.
Assumes that we have a standard set of word senses (e.g.,
WordNet)
I frequency: e.g., diet: the food sense (or senses) is much
more frequent than the parliament sense (Diet of Wurms)
I collocations: e.g. striped bass (the fish) vs bass guitar:
syntactically related or in a window of words (latter
sometimes called ‘cooccurrence’). Generally ‘one sense
per collocation’.
I selectional restrictions/preferences (e.g., Kim eats bass,
must refer to fish
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
WSD techniques
I supervised learning: cf. POS tagging from lecture 3. But
sense-tagged corpora are difficult to construct, algorithms
need far more data than POS tagging
I unsupervised learning (see below)
I Machine readable dictionaries (MRDs): e.g., look at
overlap with words in definitions and example sentences
I selectional preferences: don’t work very well by
themselves, useful in combination with other techniques
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
WSD by (almost) unsupervised learning
Disambiguating plant (factory vs vegetation senses):
1. Find contexts in training corpus:
sense training example
? company said that the plant is still operating
? although thousands of plant and animal species
? zonal distribution of plant life
? company manufacturing plant is in Orlando
etc
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
Yarowsky (1995): schematically
Initial state
? ?? ? ?
? ?
? ??
? ? ?
?? ?
? ? ?
?? ? ? ?
? ?
? ? ?
? ?
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
2. Identify some seeds to disambiguate a few uses. e.g., ‘plant
life’ for vegetation use (A) ‘manufacturing plant’ for factory use
(B):
sense training example
? company said that the plant is still operating
? although thousands of plant and animal species
A zonal distribution of plant life
B company manufacturing plant is in Orlando
etc
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
Seeds
? ?? ? B
? B
? ?? manu.
? ? ?
?? life ?
A ? ?
?? A A ?
? A
? A ?
? ?
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
3. Train a decision list classifier on the Sense A/Sense B
examples.
reliability criterion sense
8.10 plant life A
7.58 manufacturing plant B
6.27 animal within 10 words of plant A
etc
Decision list classifier: automatically trained if/then statements.
Experimenter decides on classes of test by providing definitions
of features of interest: system builds specific tests and provides
reliability metrics.
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
4. Apply the classifier to the training set and add reliable
examples to A and B sets.
sense training example
? company said that the plant is still operating
A although thousands of plant and animal species
A zonal distribution of plant life
B company manufacturing plant is in Orlando
etc
5. Iterate the previous steps 3 and 4 until convergence
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
Iterating:
? ?? ? B
B B
? ??
animal ? ? ?
company
AA B
A ? ?
?? A A ?
? A
? A ?
? ?
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
Final:
A AA B B
B B
A BB
A B B
AA B
A B B
AA A A B
A A
A A B
A B
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
6. Apply the classifier to the unseen test data
‘one sense per discourse’: can be used as an additional
refinement
e.g., once you’ve disambiguated plant one way in a particular
text/section of text, you can assign all the instances of plant to
that sense
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
Evaluation of WSD
I SENSEVAL competitions
I evaluate against WordNet
I baseline: pick most frequent sense — hard to beat (but
don’t always know most frequent sense)
I human ceiling varies with words
I MT task: more objective but sometimes doesn’t
correspond to polysemy in source language
Natural Language Processing
Lecture 6: Compositional and lexical semantics.
Word sense disambiguation
Outline of next lecture
Putting sentences together (in text).
Lecture 7: Discourse.
Relationships between sentences.
Coherence
Anaphora (pronouns etc)
An algorithm for anaphora resolution
Natural Language Processing
Lecture 7: Discourse.
Outline of today’s lecture
Putting sentences together (in text).
Lecture 7: Discourse.
Relationships between sentences.
Coherence
Anaphora (pronouns etc)
An algorithm for anaphora resolution
Natural Language Processing
Lecture 7: Discourse.
Relationships between sentences.
Rhetorical relations
Max fell. John pushed him.
can be interpreted as:
1. Max fell because John pushed him.
EXPLANATION
or
2 Max fell and then John pushed him.
NARRATION
Implicit relationship: discourse relation or rhetorical relation
because, and then are examples of cue phrases
Natural Language Processing
Lecture 7: Discourse.
Coherence
Coherence
Discourses have to have connectivity to be coherent:
Kim got into her car. Sandy likes apples.
Can be OK in context:
Kim got into her car. Sandy likes apples, so Kim thought she’d
go to the farm shop and see if she could get some.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Coherence in generation
Strategic generation: constructing the logical form. Tactical
generation: logical form to string.
Strategic generation needs to maintain coherence.
In trading yesterday: Dell was up 4.2%, Safeway was down
3.2%, HP was up 3.1%.
Better:
Computer manufacturers gained in trading yesterday: Dell was
up 4.2% and HP was up 3.1%. But retail stocks suffered:
Safeway was down 3.2%.
So far this has only been attempted for limited domains: e.g.
tutorial dialogues.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Coherence in interpretation
Discourse coherence assumptions can affect interpretation:
Kim’s bike got a puncture. She phoned the AA.
Assumption of coherence (and knowledge about the AA) leads
to bike interpreted as motorbike rather than pedal cycle.
John likes Bill. He gave him an expensive Christmas present.
If EXPLANATION - ‘he’ is probably Bill.
If JUSTIFICATION (supplying evidence for first sentence), ‘he’
is John.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Factors influencing discourse interpretation
1. Cue phrases.
2. Punctuation (also prosody) and text structure.
Max fell (John pushed him) and Kim laughed.
Max fell, John pushed him and Kim laughed.
3. Real world content:
Max fell. John pushed him as he lay on the ground.
4. Tense and aspect.
Max fell. John had pushed him.
Max was falling. John pushed him.
Hard problem, but ‘surfacy techniques’ (punctuation and cue
phrases) work to some extent.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Rhetorical relations and summarization
Analysis of text with rhetorical relations generally gives a binary
branching structure:
I nucleus and satellite: e.g., EXPLANATION,
JUSTIFICATION
I equal weight: e.g., NARRATION
Max fell because John pushed him.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Rhetorical relations and summarization
Analysis of text with rhetorical relations generally gives a binary
branching structure:
I nucleus and satellite: e.g., EXPLANATION,
JUSTIFICATION
I equal weight: e.g., NARRATION
Max fell because John pushed him.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Summarisation by satellite removal
If we consider a discourse relation as a
relationship between two phrases, we get a binary branching
tree structure for the discourse. In many relationships,
such as Explanation, one phrase depends on the other:
e.g., the phrase being explained is the main
one and the other is subsidiary. In fact we can get rid of the
subsidiary phrases and still have a reasonably coherent
discourse.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Summarisation by satellite removal
If we consider a discourse relation as a
relationship between two phrases, we get a binary branching
tree structure for the discourse. In many relationships,
such as Explanation, one phrase depends on the other:
e.g., the phrase being explained is the main
one and the other is subsidiary. In fact we can get rid of the
subsidiary phrases and still have a reasonably coherent
discourse.
Natural Language Processing
Lecture 7: Discourse.
Coherence
Summarisation by satellite removal
If we consider a discourse relation as a
relationship between two phrases, we get a binary branching
tree structure for the discourse. In many relationships,
such as Explanation, one phrase depends on the other:
e.g., the phrase being explained is the main
one and the other is subsidiary. In fact we can get rid of the
subsidiary phrases and still have a reasonably coherent
discourse.
We get a binary branching tree structure for the discourse. In
many relationships one phrase depends on the other. In fact we
can get rid of the subsidiary phrases and still have a reasonably
coherent discourse.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Referring expressions
Niall Ferguson is prolific, well-paid and a snappy dresser.
Stephen Moss hated him — at least until he spent an hour
being charmed in the historian’s Oxford study.
referent a real world entity that some piece of text (or
speech) refers to. the actual Prof. Ferguson
referring expressions bits of language used to perform
reference by a speaker. ‘Niall Ferguson’, ‘he’, ‘him’
antecedant the text evoking a referent. ‘Niall Ferguson’
anaphora the phenomenon of referring to an antecedant.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Pronoun resolution
Pronouns: a type of anaphor.
Pronoun resolution: generally only consider cases which refer
to antecedant noun phrases.
Niall Ferguson is prolific, well-paid and a snappy dresser.
Stephen Moss hated him — at least until he spent an hour
being charmed in the historian’s Oxford study.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Pronoun resolution
Pronouns: a type of anaphor.
Pronoun resolution: generally only consider cases which refer
to antecedant noun phrases.
Niall Ferguson is prolific, well-paid and a snappy dresser.
Stephen Moss hated him — at least until he spent an hour
being charmed in the historian’s Oxford study.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Pronoun resolution
Pronouns: a type of anaphor.
Pronoun resolution: generally only consider cases which refer
to antecedant noun phrases.
Niall Ferguson is prolific, well-paid and a snappy dresser.
Stephen Moss hated him — at least until he spent an hour
being charmed in the historian’s Oxford study.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Pronoun resolution
Pronouns: a type of anaphor.
Pronoun resolution: generally only consider cases which refer
to antecedant noun phrases.
I hard constraints (e.g., agreement)
I soft preferences / salience (depend on discourse structure)
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Hard constraints: Pronoun agreement
I A little girl is at the door — see what she wants, please?
I My dog has hurt his foot — he is in a lot of pain.
I * My dog has hurt his foot — it is in a lot of pain.
Complications:
I The team played really well, but now they are all very tired.
I Kim and Sandy are asleep: they are very tired.
I Kim is snoring and Sandy can’t keep her eyes open: they
are both exhausted.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Hard constraints: Reflexives
I Johni cut himselfi shaving. (himself = John, subscript
notation used to indicate this)
I # Johni cut himj shaving. (i 6= j — a very odd sentence)
Reflexive pronouns must be coreferential with a preceeding
argument of the same verb, non-reflexive pronouns cannot be.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Hard constraints: Pleonastic pronouns
Pleonastic pronouns are semantically empty, and don’t refer:
I It is snowing
I It is not easy to think of good examples.
I It is obvious that Kim snores.
I It bothers Sandy that Kim snores.
Natural Language Processing
Lecture 7: Discourse.
Anaphora (pronouns etc)
Soft preferences: Salience
Recency Kim has a fast car. Sandy has an even faster one.
Lee likes to drive it.
Grammatical role Subjects > objects > everything else: Fred
went to the Grafton Centre with Bill. He bought a
CD.
Repeated mention Entities that have been mentioned more
frequently are preferred.
Parallelism Entities which share the same role as the pronoun
in the same sort of sentence are preferred: Bill
went with Fred to the Grafton Centre. Kim went
with him to Lion Yard. Him=Fred
Coherence effects (mentioned above)
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Lappin and Leass
Discourse model: referring NPs in equivalence classes with
global salience value (incrementally updated).
For example:
N Niall Ferguson, him 435
S Stephen Moss 310
H the historian 100
O Oxford study 100
Resolve each pronoun to the entity with the highest weight in
the discourse model.
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Lappin and Leass’s algorithm
For each sentence:
1. Divide by two the global salience factors
2. Identify referring NPs
3. Calculate global salience factors for each NP (see below)
4. Update the discourse model with the referents and their
global salience scores.
5. For each pronoun:
5.1 Collect potential referents
5.2 Filter referents
5.3 Calculate the per pronoun adjustments for each referent
(see below).
5.4 Select the referent with the highest salience value for its
equivalence class plus its per-pronoun adjustment.
5.5 Add the pronoun into the equivalence class for that referent,
and increment the salience factor.
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Weights
Global salience factors:
recency 100 (current sentence)
subject 80
existential 70 there is a cat
direct object 50
indirect object 40 give Sandy a present
oblique complement 40 put the cat on a mat
non-embedded noun 80
non-adverbial 50
(i.e., embedded -80 and adverbial -50 but no negative weights)
Per pronoun salience factors:
cataphora -175 pronoun before NP
same role 35 e.g., pronoun and NP both subject
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Example
Niall Ferguson is prolific, well-paid and a snappy dresser.
Stephen Moss hated him
— at least until he spent an hour being charmed in the
historian’s Oxford study.
Possible antecedants:
N Niall Ferguson, him 435
S Stephen Moss 310
N has score 155 + 280 ((subject + non-embedded +
non-adverbial + recency)/2 + (direct object + non-embedded +
non-adverbial + recency))
S has score 310 (subject + non-embedded + non-adverbial +
recency) + same role per-pronoun 35
So we resolve he to N (wrongly . . . )
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Example, continued
Add he to the discourse referent equivalence class.
Update weights: add 80 because he is subject
N Niall Ferguson, him, he 515
Note: no duplicate factors for the same sentence (e.g., no
weight added because he is non-embedded)
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Anaphora for everyone, Kennedy and Boguraev
Modification of Lappin and Leass that doesn’t require a parser.
1. POS tag input text (Lingsoft tagger)
2. Regular expressions to identify NPs (NP chunking), mark
expletive it
3. Regular expressions for grammatical role
4. Text segmentation: don’t cross document boundaries etc.
5. Heuristics for reflexives
6. Otherwise much as Lappin and Leass
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Evaluation
1. LL quoted 86% (computer manuals), KB 75% (mix genres)
2. much less standardized than POS tagging: datasets,
metrics
3. results are genre-dependent
4. replication is difficult
Natural Language Processing
Lecture 7: Discourse.
An algorithm for anaphora resolution
Outline of next lecture
Applications and (perhaps) demos.
Natural Language Processing
Lecture 8: Applications.
Outline of today’s lecture
Lecture 8: Applications.
Spoken dialogue systems
Question Answering
Wrapping up
Natural Language Processing
Lecture 8: Applications.
Spoken dialogue systems
Spoken dialogue systems
1. Single initiative systems (also known as system initiative
systems): system controls what happens when.
System: Do you have your customer reference number?
Please say yes or no. User: Yes
Limited mixed-initiative:
System: When do you want to leave?
User: the twenty-third
OR
User: the morning of the twenty-third
2. Mixed initiative dialogue. Both participants can control the
dialogue to some extent.
System: Which station do you want to leave from?
User: I don’t know, tell me which station I need for
Cambridge.
Natural Language Processing
Lecture 8: Applications.
Spoken dialogue systems
Approaches to SDS
I Custom grammars: FSAs or simple CFGs (compiled to
FSAs) at each point in a dialogue controlled by an FSA.
VoiceXML
Feature structure grammar compiled to CFG/FSA: e.g.,
Clarissa (International Space Station).
I Statistical language modelling plus robust customised
grammars or keyword spotting
I Statistical language modelling plus grammar induction
I Statistical language modelling plus general purpose
grammar
Natural Language Processing
Lecture 8: Applications.
Spoken dialogue systems
Spoken dialogue system architecture
DIALOGUE MANAGER
6
DB
*
slot filling
R
j j
string withrecorded
PARSING: turn-specific grammars
Y filled slots speech
grammar6
constraints constraints
?
R
TEXT-TO-
SPEECH RECOGNITION
SPEECH
6
R
user input speech output
Natural Language Processing
Lecture 8: Applications.
Spoken dialogue systems
Dialogue management in single-initiative SDS
I Finite-state dialogue manager
I Tightly controls the dialogue: prompts user for specific
information
I Separate recognition grammar for every state
I DB may help specify the grammars: e.g.,
1. prompt for post code
2. get 100 items on n-best list from recogniser
3. use first line of addresses from these to build a FS grammar
4. prompt for ‘first line’ of address
5. disambiguate post code
I Confirmation strategy is important
Natural Language Processing
Lecture 8: Applications.
Question Answering
QA with parsed corpus
PARSED CORPUS
*
j
query match response selection
6
?
PARSING output
6
MORPHOLOGY
6
INPUT PROCESSING
6
user input
Natural Language Processing
Lecture 8: Applications.
Question Answering
Questions and answers: QA, NLID etc
A valid answer should entail the query (with suitable
interpretation of wh-terms etc).
Is a dog barking?
∃x[dog0 (x) ∧ bark0 (x)]
A dog is barking entails A dog is barking
Rover is barking and Rover is a dog entails A dog is barking.
bark0 (Rover ) ∧ dog0 (Rover ) entails ∃x[dog0 (x) ∧ bark0 (x)]
which dog is barking?
bark0 (Rover ) ∧ dog0 (Rover ) entails ∃x[dog0 (x) ∧ bark0 (x)]
Bind query term to answer.
Natural Language Processing
Lecture 8: Applications.
Question Answering
QA example 1
Example
What eats jellyfish?
Simplified semantics:
[ a:eat(e), ARG1(a,x), ARG2(a,y), jellyfish(y) ]
So won’t match on jellyfish eat fish.
Natural Language Processing
Lecture 8: Applications.
Question Answering
What eats jellyfish?
Example
Turtles eat jellyfish and they have special hooks in their throats
to help them swallow these slimy animals.
Natural Language Processing
Lecture 8: Applications.
Question Answering
What eats jellyfish?
Example
Turtles eat jellyfish and they have special hooks in their throats
to help them swallow these slimy animals.
Match on [ a:eat(e), ARG1(a,x), ARG2(a,y), jellyfish(y) ]
A logically valid answer which entails the query since the
conjunct can be ignored.
Natural Language Processing
Lecture 8: Applications.
Question Answering
What eats jellyfish?
Example
Sea turtles, ocean sunfish (Mola mola) and blue rockfish all are
able to eat large jellyfish, seemingly without being affected by
the nematocysts.
Natural Language Processing
Lecture 8: Applications.
Question Answering
What eats jellyfish?
Example
Sea turtles, ocean sunfish (Mola mola) and blue rockfish all are
able to eat large jellyfish, seemingly without being affected by
the nematocysts.
Pattern matching on semantics:
[ a:eat(e), ARG1(a,x), ARG2(a,y), large(y), jellyfish(y) ]
eat large jellyfish entails eat jellyfish (because large is
intersective)
Natural Language Processing
Lecture 8: Applications.
Question Answering
What eats jellyfish?
Example
Also, open ocean-dwelling snails called Janthina and even
some seabirds have been known to eat jellyfish.
Natural Language Processing
Lecture 8: Applications.
Question Answering
What eats jellyfish?
Example
Also, open ocean-dwelling snails called Janthina and even
some seabirds have been known to eat jellyfish.
[ a1:know(e), ARG2(a1,h1), qeq(h1,lb), lb:a:eat(e), ARG1(a,x),
ARG2(a,y), jellyfish(y) ]
Logically valid if know is taken as truth preserving.
∀P∀y [know(y , P) =⇒ P]
Axioms like this required for logically valid entailment: missing
axiom would cause failure to match.
Natural Language Processing
Lecture 8: Applications.
Question Answering
What eats jellyfish?
Example
Also, open ocean-dwelling snails called Janthina and even
some seabirds have been known to eat jellyfish.
[ a1:know(e), ARG2(a1,h1), qeq(h1,lb), lb:a:eat(e), ARG1(a,x),
ARG2(a,y), jellyfish(y) ]
Logically valid if know is taken as truth preserving.
∀P∀y [know(y , P) =⇒ P]
Axioms like this required for logically valid entailment: missing
axiom would cause failure to match.
Natural Language Processing
Lecture 8: Applications.
Question Answering
QA processing
0. Corpus collection. Parse corpus to semantic
representation.
1. Process query: (i.e., separate words, detect non-words),
run automatic spelling correction (maybe).
2. Morphology and lexical lookup
3. Parse with general grammar. Stochastic parse selection
(trained on treebank).
4. Match question against parsed corpus.
5. Return snippets/documents for best match.
Natural Language Processing
Lecture 8: Applications.
Question Answering
Morphology and lexical lookup
What eats aardvarks?
What eat+3SG aardvark+PL?
Natural Language Processing
Lecture 8: Applications.
Question Answering
Morphology and lexical lookup, 2
spelling rule
plur_noun_orule :=
%suffix (!s !ss) (!ss !ssses) (ss sses)
(!ty !ties) (ch ches) (sh shes) (x xes) (z zes)
lex_rule_infl_affixed &
[ ND-AFF +,
SYNSEM mass_or_count_synsem &
[ LOCAL plur_noun ]].
Natural Language Processing
Lecture 8: Applications.
Question Answering
Semantics
What eats aardvarks?
<mrs>
<label vid=’1’/><var vid=’2’/>
<ep cfrom=’0’ cto=’4’><pred>THING_REL</pred><label vid=’3’/>
<fvpair><rargname>ARG0</rargname><var vid=’4’ sort=’x’>
<extrapair><path>PERS</path><value>3</value></extrapair>
<extrapair><path>NUM</path><value>SG</value></extrapair>
<extrapair><path>SF</path><value>PROP</value></extrapair></var></fvpair></ep>
<ep cfrom=’0’ cto=’4’><pred>WHICH_Q_REL</pred><label vid=’5’/>
<fvpair><rargname>ARG0</rargname><var vid=’4’ sort=’x’>
<extrapair><path>PERS</path><value>3</value></extrapair>
<extrapair><path>NUM</path><value>SG</value></extrapair>
<extrapair><path>SF</path><value>PROP</value></extrapair></var></fvpair>
<fvpair><rargname>RSTR</rargname><var vid=’6’ sort=’h’></var></fvpair>
<fvpair><rargname>BODY</rargname><var vid=’7’ sort=’h’></var></fvpair></ep>
<ep cfrom=’5’ cto=’9’><spred>_eat_v_1_rel</spred><label vid=’8’/>
<fvpair><rargname>ARG0</rargname><var vid=’2’ sort=’e’>
<extrapair><path>TENSE</path><value>PRES</value></extrapair>
<extrapair><path>MOOD</path><value>INDICATIVE</value></extrapair>
<extrapair><path>PROG</path><value>-</value></extrapair>
<extrapair><path>PERF</path><value>-</value></extrapair>
<extrapair><path>SF</path><value>QUES</value></extrapair></var></fvpair>
<fvpair><rargname>ARG1</rargname><var vid=’4’ sort=’x’>
<extrapair><path>PERS</path><value>3</value></extrapair>
<extrapair><path>NUM</path><value>SG</value></extrapair>
<extrapair><path>SF</path><value>PROP</value></extrapair></var></fvpair>
<fvpair><rargname>ARG2</rargname><var vid=’9’ sort=’x’>
<extrapair><path>PERS</path><value>3</value></extrapair>
Natural Language Processing
Lecture 8: Applications.
Question Answering
DELPH-IN: Deep Linguistic Processing using HPSG
I Informal collaboration on tools and grammars: see
http://www.delph-in.net/
I Large grammars for English, German and Japanese;
medium/growing for Spanish, Norwegian, Portuguese,
Korean, French. Many small grammars.
I Common semantic framework: Minimal Recursion
Semantics (MRS) and Robust MRS. RMRS also from
shallower parsing, chunking, POS tagging.
I Parsing and generation (realization), integrated shallower
processing.
I Grammar Matrix: framework/starter kit for the development
of grammars for diverse languages.
Natural Language Processing
Lecture 8: Applications.
Question Answering
Generation with the ERG
I Bidirectional grammar, but want to recognise multiple
dialects and generate consistently in an appropriate one
I Full generation so far only used in MT
I Needs further work on speed, selection of realisation (i.e.,
the generated string) and implementation in a runtime
system
Natural Language Processing
Lecture 8: Applications.
Wrapping up
Themes: ambiguity
I levels: morphology, syntax, semantic, lexical, discourse
I resolution: local ambiguity, syntax as filter for morphology,
selectional restrictions.
I ranking: parse ranking, WSD, anaphora resolution.
I processing efficiency: chart parsing
Natural Language Processing
Lecture 8: Applications.
Wrapping up
Themes: evaluation
I training data and test data
I reproducibility
I baseline
I ceiling
I module evaluation vs application evaluation
I nothing is perfect!
Natural Language Processing
Lecture 8: Applications.
Wrapping up
Themes: evaluation
I training data and test data
I reproducibility
I baseline
I ceiling
I module evaluation vs application evaluation
I nothing is perfect!
Natural Language Processing
Lecture 8: Applications.
Wrapping up
Conclusion
I different processing modules
I different applications blend modules differently
I many different styles of algorithm:
1. FSAa and FSTs
2. Markov models and HMMs
3. CFG (and probabilistic CFGs)
4. constraint-based frameworks
5. inheritance hierarchies (WordNet), decision trees (WSD)
6. mixing hard and soft constraints (Lappin and Leass)
Natural Language Processing
Lecture 8: Applications.
Wrapping up
More about speech and language processing
I Simone Teufel’s Part II course
I MPhil course: CSTIT
I jointly taught with Dept of Engineering
I see lab web page for details
I financial support available for some applicants
Natural Language Processing
Lecture 8: Applications.
Wrapping up
More about speech and language processing
I Simone Teufel’s Part II course
I MPhil course: CSTIT
I jointly taught with Dept of Engineering
I see lab web page for details
I financial support available for some applicants
You might also like
- Hourglass Workout Program by Luisagiuliet 276% (21)Hourglass Workout Program by Luisagiuliet 251 pages
- Read People Like A Book by Patrick King-Edited57% (83)Read People Like A Book by Patrick King-Edited12 pages
- Livingood, Blake - Livingood Daily Your 21-Day Guide To Experience Real Health77% (13)Livingood, Blake - Livingood Daily Your 21-Day Guide To Experience Real Health260 pages
- Donald Trump & Jeffrey Epstein Rape Lawsuit and Affidavits83% (1016)Donald Trump & Jeffrey Epstein Rape Lawsuit and Affidavits13 pages
- The 36 Questions That Lead To Love - The New York Times91% (35)The 36 Questions That Lead To Love - The New York Times3 pages
- The 36 Questions That Lead To Love - The New York Times95% (21)The 36 Questions That Lead To Love - The New York Times3 pages
- 14 Easiest & Hardest Muscles To Build (Ranked With Solutions)100% (8)14 Easiest & Hardest Muscles To Build (Ranked With Solutions)27 pages
- Jeffrey Epstein39s Little Black Book Unredacted PDF77% (13)Jeffrey Epstein39s Little Black Book Unredacted PDF95 pages
- A First Dictionary of Cultural Literacy PDF100% (9)A First Dictionary of Cultural Literacy PDF298 pages
- The 4 Hour Workweek, Expanded and Updated by Timothy Ferriss - Excerpt23% (954)The 4 Hour Workweek, Expanded and Updated by Timothy Ferriss - Excerpt38 pages
- Natural Language Processing Inside Pages 2No ratings yetNatural Language Processing Inside Pages 2159 pages
- Introduction To NLP: Natural Language ProcessingNo ratings yetIntroduction To NLP: Natural Language Processing21 pages
- Introduction to Natural Language ProcessingNo ratings yetIntroduction to Natural Language Processing31 pages
- Raymond S. T. Lee - Natural Language Processing. A Textbook With Python Implementation-Springer (2024)No ratings yetRaymond S. T. Lee - Natural Language Processing. A Textbook With Python Implementation-Springer (2024)454 pages
- Introducing Natural Language ProcessingNo ratings yetIntroducing Natural Language Processing13 pages
- CSE4022 Natural-Language-Processing ETH 1 AC41No ratings yetCSE4022 Natural-Language-Processing ETH 1 AC416 pages
- Principles of Natural Language ProcessingNo ratings yetPrinciples of Natural Language Processing264 pages
- Natural Language Processing Report (By Sandeep Kumar Dash)No ratings yetNatural Language Processing Report (By Sandeep Kumar Dash)25 pages
- Natural Language Processing GRADE 10 - 2021No ratings yetNatural Language Processing GRADE 10 - 202182 pages
- "I'm Sorry Dave, I'm Afraid I Can't Do That": Linguistics, Statistics, and Natural Language Processing Circa 2001No ratings yet"I'm Sorry Dave, I'm Afraid I Can't Do That": Linguistics, Statistics, and Natural Language Processing Circa 20016 pages
- Natural Language Processing: MIT 6.8610-6.8611 / Fall 2023No ratings yetNatural Language Processing: MIT 6.8610-6.8611 / Fall 2023122 pages
- Introduction To Natural Language Processing (NLP)No ratings yetIntroduction To Natural Language Processing (NLP)87 pages
- CCPS521-WIN2023-Week10 Natural Language Processing FinalNo ratings yetCCPS521-WIN2023-Week10 Natural Language Processing Final56 pages
- Get Syntactic Gradience The Nature of Grammatical Indeterminacy First Edition Bas Aarts PDF ebook with Full Chapters Now100% (4)Get Syntactic Gradience The Nature of Grammatical Indeterminacy First Edition Bas Aarts PDF ebook with Full Chapters Now71 pages
- English 8 q2 Mod1 Explaining Visual Verbal Relationships v2No ratings yetEnglish 8 q2 Mod1 Explaining Visual Verbal Relationships v235 pages
- 19CSE453 - Natural Language Processing: Part of Speech TaggingNo ratings yet19CSE453 - Natural Language Processing: Part of Speech Tagging59 pages
- (G David Morley) Syntax in Functional GrammarNo ratings yet(G David Morley) Syntax in Functional Grammar257 pages
- G. V. Chulanova: Ministry of Education and Science of Ukraine Sumy State UniversityNo ratings yetG. V. Chulanova: Ministry of Education and Science of Ukraine Sumy State University75 pages
- Lesson Plan (1) Course Code: 8608: Arshia Manzoor BY670514 B.ED. (1.5 YEARS)No ratings yetLesson Plan (1) Course Code: 8608: Arshia Manzoor BY670514 B.ED. (1.5 YEARS)13 pages
- Teaching and Assessment of Grammar LessonNo ratings yetTeaching and Assessment of Grammar Lesson8 pages
- hake-7th-grade-writing-lessons-workbook-previewNo ratings yethake-7th-grade-writing-lessons-workbook-preview32 pages
- Soal PSTS Bahasa Inggris Kelas 7 2024-2025No ratings yetSoal PSTS Bahasa Inggris Kelas 7 2024-20252 pages
- The Basic Sentence: The Four Most Serious ErrorsNo ratings yetThe Basic Sentence: The Four Most Serious Errors14 pages
- 3 Lang Review Q1 (Conjunctions, Prepositions, Interjections)No ratings yet3 Lang Review Q1 (Conjunctions, Prepositions, Interjections)3 pages
- Parts of Speech Tagset: Kannada: S.N TAG Description Example (English)No ratings yetParts of Speech Tagset: Kannada: S.N TAG Description Example (English)5 pages
