text summarization using amr

Text summarization using Abstract Meaning Representation
Text summarization using Abstract Meaning
Representation
AMR
Amit Nagarkoti
Supervisor: Dr. Harish Karnick
Department of Computer Science and Engineering
Indian Institute of Technology Kanpur

Table of contents
1. Introduction
2. Extractive Summarization Methods
3. Seq2Seq Learning
4. Results

Introduction
Outline
1 Introduction
2 Extractive Summarization Methods
3 Seq2Seq Learning
4 Results

Introduction
Introduction
Deﬁnition
Text summarization is the process of reducing the size of original
document to a much more concise form such that the most relevant facts
in the original document are retained.
Summaries are always lossy.
Summarization methodologies
Extractive summarization : extract import words and sentences
Abstractive summarization harder : rephrase, generate similar words

Introduction
Why Important?
user point of view: Evaluate importance of article.
linguistic/scientiﬁc/philosophical view: Solving summarization is
equivalent to solving the problem of language understanding.

Introduction
AMR : welcome to amr
(w / welcome-01
:ARG2 (a / amr))
represent sentence as a DAG
captures “who is doing what to whom”
nodes : verb sense (see-01), objects (boy, marble)
edges : relations (ARG1, ARG0)

Introduction
Abstraction in AMR
Figure: AMR example
The boy sees the white marble.
The boy saw the marble that was white.

Introduction
Propbank Lexicon
white-03 : refers to the color white
ARG1: thing that is white in color (marble)
ARG2: speciﬁc part of ARG1, if also mentioned
see-01 : to see or view
ARG0: viewer (boy)
ARG1: thing viewed (marble)
ARG2: attribute of ARG1, further description (white - but we have
diﬀerent amr here)

Introduction
How to get an AMR?
Use JAMR : 84% accuracy for concept node identiﬁcation.
gives AMR for a sentence
(d / discover-01 | 0
:ARG0 (r / rope | 0.0)
:ARG1 (n / noose | 0.1)
:location (c / campus | 0.2))
the noose made of rope was discovered on campus

Introduction
word-node Alignment
node alignment word
discover-01 6-7—0 discovered
rope 4-5—0.0 rope
noose 1-2—0.1 noose
campus 8-9—0.2 campus
Table: word-node alignment
These alignments will be used to generate summaries from AMRs

Extractive Summarization Methods
Outline
1 Introduction
3 Seq2Seq Learning
4 Results

Using AMRs
Figure: Steps to generate Document graph from sentence AMR (modiﬁed from
[Liu et al., 2015])

Document Graph
Figure: Dense Document Graph

Finding the summary sub-graph
maximize
n
i=1
ψi θT
f (vi ) (1)
here ψi is a binary variable indicating node i is selected or not
θ = [θ1, . . . , θm] are model parameters
f (vi ) = [f1(vi ), . . . , fm(vi )] are node features

Constraints for a valid sub-graph
Figure: vi − eij ≥ 0 vj − eij ≥ 0
Figure: i f0i − vi = 0
Figure: i fij − k fjk − vj = 0
Neij − fij ≥ 0, ∀i, j sanity constraint
i,j eij ≤ L size constraint

Complete Algorithm
f o r cur doc in corpus :
c r e a t e doc graph
add ILP c o n s t r a i n t s
s o l v e o b j e c t i v e to minimize l o s s
c a l c u l a t e g r a d i e n t s
update model parameters

LSA for extractive summarization
Figure: term-sentence matrix
term vector tT
i = [xi1 . . . xin]
sentence vector sT
j = [x1j . . . xmj ]
term-sentence matrix can be huge
no relation between terms
sparsity

Figure: SVD over term-sentence matrix source:Wikipedia
Figure: low rank approximation

Finally a score is calculated for each sentence vector given by
Sl =
n
i=1
v2
l,i · σ2
i (2)
where Sl is score for sentence l
choose L sentences with highest scores

Seq2Seq Learning
Outline
1 Introduction
3 Seq2Seq Learning
4 Results

Seq2Seq Learning
s2s
Figure: Sequence to Sequence Model

Seq2Seq Learning
Figure: RNN cell st = f (Uxt + Wst−1) ot = softmax(Vst ) source:
nature [LeCun et al., 2015]
Figure: Chat client using s2s model based on LSTM source: [Christopher, ]

Seq2Seq Learning
S2S continued
During training model tries to minimize the negative log likelihood of the
target word
lossD =
T
t=1
− log(P(wt)) (3)
here wt is the target word at step t.
Problems in s2s
slow training with long sequences - limit sequence lengths
limited context for decoder critical for s2s
large vocabularies

Seq2Seq Learning
Attention to rescue
Attend or Re-visit critical information when making decisions
Figure: Attention in RNNs source:distill.pub

Seq2Seq Learning
Attention complete view
Figure: Attention at step t

Seq2Seq Learning
Attention Heatmap
X − axis is the input sequence, Y − axis is the generated output
Figure: Attention Heatmap during Decoding

Seq2Seq Learning
Byte Pair Encoding
Deﬁnition
BPE is a compression technique where the most frequent pair of
consecutive bytes is replaced by a byte not in the document.
BPE has been adapted for NMT [Sennrich et al., 2015] using the
idea of subword unit.
“lower” will be represented as “l o w e r @”.
vocabsize = numOfIterations + numOfChars.
BPE merge operations learned from dictionary low, lowest, newer, wider.
using 4 merge operations.
r @ r@
l o lo
lo w low
e r@ er@

Seq2Seq Learning
Pointer Generator Model for OOVs
Figure: Pointer Gen model [See et al., 2017]

Seq2Seq Learning
Coverage
“Coverage” is used in MT to control over and under production of target
words.
Some words may never get enough attention resulting in poor
translation/summaries.
The solution is to use coverage to guided attention [Tu et al., 2016]
and [See et al., 2017].
Accumulate all attention weights and penalize for extra attention.
ct =
t−1
t =1
αt
coverage vector (4)
covlosst =
encsteps
i=1
min(αi , ct
i ) (5)

Seq2Seq Learning
AMR Linearization
Figure: AMR DFS Traversal gives the linearization -TOP-( eat-01 ARG1(
bone )ARG1 ARG0( dog )ARG0 )-TOP-

Seq2Seq Learning
Data Augmentation/Extension with POS
POS sequence p1 · · · pn is obtained using the Stanford Parser
Figure: s2s model generating multiple output distributions

Seq2Seq Learning
Hierarchical Sequence Encoder for AMRs
sentence vector Si is obtained by using attention on word encoder
states
document vector Di is obtained by using attention on sentence
encoder states
Figure: hierarchical models learn the document vector using level-wise learning

Seq2Seq Learning
Using Dependency Parsing
“A Dependency Parse of a sentence is tree with labelled edges such that
the main verb or the focused noun is the root and edges are the relations
between words in the sentence.”
To reduce document size we used a context of size L around the
root word, reducing the sentence size to at max 2L + 1.
Fig below has “capital” as root with L = 3 we are able to extract
crux of the sentence i.e “Delhi is the capital of India”.
Figure: Dependency Parse :: Delhi is the capital of India , with lots of people.

Results
Outline
1 Introduction
3 Seq2Seq Learning
4 Results

Results
Rouge
RougeN-Precision MatchingN−grams
Count candidateN−grams
RougeN-Recall MatchingN−grams
Count referenceN−grams
RougeN-F1 2RougeNre∗RougeNpre
RougeNre+RougeNpre
RougeL-Precision lenLCS(ref ,can)
lencandidate
RougeL-Recall lenLCS(ref ,can)
lenreference
RougeL-F1 2RougeLre∗RougeLpre
RougeLre+RougeLpre
Table: Rouge Formulas

Results
Examples
Rouge-N
candidate :: the cat was found under the bed
reference :: the cat was under the bed
has recall 6
6 = 1 and precision 6
7 = 0.86
Rouge-L
reference :: police killed the gunman
candidate1 :: police kill the gunman
candidate2 :: the gunman kill police
candidate1 has RougeL − F1 of 0.75 and candidate2 has
RougeL − F1 of 0.5

Results
Dataset Description
We used CNN/Dailymail dataset. The distribution of dataset is as follows
Train 287,226
Test 11,490
Validation 13,368
Table: CNN/Dailymail split
average number of sentences per article 31
average number of sentences per summary 3
average number of words per article 790
average number of words per summary 55
average number of words per article (BPE vsize 50k) 818
Table: CNN/Dailymail average stats for Training sets

Results
Results onn CNN/Dailymail
Model R1-f1 R2-f1 RL-f1
bpe-no-cov 35.39 15.53 32.31
bpe-cov 36.31 15.69 33.63
text-no-cov 31.4 11.6 27.2
text-cov 33.19 12.38 28.12
text-200 34.66 16.23 30.96
text-200-cov 37.18 17.41 32.68
pos-full* 35.76 16.9 31.86
pos-full-cov* 37.96 17.71 33.23
Table: pos-full minimizes loss for word generation and POS tag generation, all
models use pointer copying mechanism as default except the bpe model,
text-200 uses glove vectors for word vector initialization

Results
AMR S2S
AMR as augmented data using 25k training examples
aug-no-cov 30.18 11 26.52
aug-cov 34.53 13.22 29.21
Table: AMRs as Augmented Data
AMR to AMR using s2s
s2s-cnn-1 17.97 6.31 17.35
s2s-cnn-2 25.60 8.31 25.01
s2s-cnn-3 31.96 10.71 29.12
Table: 1: cnn no pointer gen 2: cnn with pointer gen 3: cnn with cov and
pointer gen

Results
AMR using Doc Graph
Model R1-f1
number of edges L <= nodes/2 - 1 (ref gold) 18.59
number of edges L <= nodes/3 -1 (ref gold) 19.72
number of edges L <= nodes/4 -1 (ref gold) 19.57
number of edges L <= nodes/2 -1 (ref *generated) 38.05
Table: Graph based AMR summaries *reference summaries were generated
using word alignment in summary graph

Results
Extractive + Abstractive
We used LSA and the dependency parse for the extractive phase
dep-no-cov 25.89 8.72 23.81
dep-cov 30.53 10.26 28.06
lsa-10-no-cov 29.75 11.27 27.3
lsa-10-cov 32.9 12.17 30.41
lsa-15-no-cov 31.64 12.34 28.74
lsa-15-cov 33.22 12.86 30.11
Table: Comparing the mixed approach for summarization, Dependency based
method used L = 7 as context window value

Results
Sample Outputs
repetition
Reference: roseanne barr told the daily beast that she is slowly
going blind barr said she has macular degeneration and glaucoma
Generated: roseanne barr said she is slowly going blind in interview
with the daily beast barr said she suffers from macular degeneration
and glaucoma [she suffers from macular degeneration and glaucoma]
[she suffers from macular degeneration and glaucoma]
less abstraction
Reference: batman superman dawn justice trailer leaked thursday
yanked offline film will released march 25 2016 stars ben affleck
henry cavill
Generated: batman superman dawn justice [leaked handheld camera
phone user uploaded handheld camera phone capture trailer spanish]
superman dawn justice will released march 25 2016 [stars affleck
batman henry cavill superman film also stars amy adams] much costs

Results
Sample Outputs
facts redundancy
Reference: coast guard says about 50 people were rescued from
mobile bay more than 100 sailboats took part in the dauphin island
race an annual event
Generated: [sailboats took part in the dauphin island race] and as
many as 50 [sailboats took part in the dauphin island race] and as
many as 50 people in all were rescued from water [the coast guard
says this is the 57th year] for the event [the club says this is the 57th
year] for the event

Results
Conclusion and Future Work
In this work we walked through some of the techniques for text
summarization and suggested some changes to them.
Extractive methods always degrades the summaries due to lack of
complete information but improves on the training times.
We looked into AMRs and their applicability to the task of
summarization and their eﬀectiveness on a smaller data set.
Finding Global graph embeddings for AMR type structures.
Using co-referencing in AMRs.
Data Augmentation without increasing the model complexity.
Expanding Memory Networks for Summarization.
Reinforcement Learning for Summarization.
Better Extraction using Dependency Trees.

Results
Thank You!
(t / thank-01
:ARG1 (y / you))

Results
References I
Christopher, O.
Understanding lstm networks.
LeCun, Y., Bengio, Y., and Hinton, G. (2015).
Deep learning.
Nature, 521(7553):436–444.
Insight.
Liu, F., Flanigan, J., Thomson, S., Sadeh, N., and Smith, N. A.
(2015).
Toward abstractive summarization using semantic representations.
See, A., Liu, P. J., and Manning, C. D. (2017).
Get to the point: Summarization with pointer-generator networks.
arXiv preprint arXiv:1704.04368.

Results
References II
Sennrich, R., Haddow, B., and Birch, A. (2015).
Neural machine translation of rare words with subword units.
Tu, Z., Lu, Z., Liu, Y., Liu, X., and Li, H. (2016).
Modeling coverage for neural machine translation.

text summarization using amr

Recommended

More Related Content

What's hot (20)

Similar to text summarization using amr (20)

Recently uploaded (20)

text summarization using amr