Seed Selection for Distantly Supervised Web-Based Relation Extraction

Seed Selection for
Distantly Supervised Web-Based
Relation Extraction
Isabelle Augenstein
Department of Computer Science, University of Sheffield, UK
i.augenstein@dcs.shef.ac.uk
August 24, 2014
Semantic Web for Information Extraction (SWAIE) Workshop, COLING 2014

2
Motivation
•  Goal: extraction of relations in text on Web pages (e.g.
Mashable) with respect to a knowledge base (e.g. Freebase)
Seed Selection for Distantly Supervised Web-Based
Relation Extraction (SWAIE Workshop)
Isabelle Augenstein

3
Motivation
•  What are possible methodologies?
•  Supervised learning: manually annotate text, train machine learning
classifier
•  Unsupervised learning: extract language patterns, cluster similar ones
•  Semi-supervised learning: start with a small number of language
patterns, iteratively learn more (bootstrapping)
•  Distant supervision: automatically label text with relations from
knowledge base, train machine learning classifier
Isabelle Augenstein

4
Motivation
•  What are possible methodologies?
•  Supervised learning: manually annotate text, train machine learning
classifier -> manual effort
•  Unsupervised learning: extract language patterns, cluster similar ones
-> difficult to map to KB, lower precision than supervised method
•  Semi-supervised learning: start with a small number of language
patterns, iteratively learn more (bootstrapping)
-> still manual effort, semantic drift (unwanted shift in meaning)
•  Distant supervision: automatically label text with relations from
knowledge base, train machine learning classifier
-> allows to extract relations with respect to KB, reasonably high
precision, no manual effort

5
Distant Supervision
Creating positive &
negative training
examples
Feature
Extraction
Classifier
Training
Prediction of
New
Relations

6
Distant Supervision
Creating positive &
negative training
examples
Feature
Extraction
Classifier
Training
Prediction of
New
Relations
Supervised learning
Automatically generated
training data
+

7
Generating training data
“If two entities participate in a relation, any sentence that contains those two
entities might express that relation.” (Mintz, 2009)
Amy Jade Winehouse was a
singer and songwriter known
for her eclectic mix of musical
genres including R&B, soul
and jazz.!

Blur helped to popularise the
Britpop genre.!

Beckham rose to fame with the
all-female pop group Spice
Girls.!

Name Genre …
Amy Winehouse
Amy Jade Winehouse
Wino
…
R&B
soul
jazz
…
…
Blur
…
Britpop
…
…
Spice Girls
…
pop
…
…
different
lexicalisations

8
Generating training data: is it that easy?
Let It Be is the twelfth album
by The Beatles which contains
their hit single Let It Be.
Name Album Track
The Beatles
…
Let It Be
…
Let It Be
…

9
Generating training data: is it that easy?
Let It Be is the twelfth album
by The Beatles which contains
their hit single Let It Be.
Name Album Track
The Beatles
…
Let It Be
…
Let It Be
…
•  Use ‘Let It Be’ mentions as positive training examples for
album or for track?
•  Problem: if both mentions of ‘Let It Be’ are used to extract
features for both album and track, wrong weights are learnt
•  How can such ambiguous examples be detected?
•  Develop methods to detect, then automatically discard
potentially ambiguous positive and negative training data

10
Seed Selection: ambiguity within an entity
•  Example: Let It Be is the twelfth album by The Beatles
which contains their hit single Let It Be.
•  Let It Be can be both an album and a track of the musical artist
The Beatles
•  For every relation, consisting of a subject, a property and an
object (s, p, o), is the subject related to (at least) two different
objects with the same lexicalisation which express two
different relations?
•  Unam:
•  Retrieve the number of such senses using the Freebase API
•  Discard the lexicalisation of the object as positive training data if it has at
least two different senses within an entity

11
Seed Selection: ambiguity across classes
•  Example: common names of book authors or common genres,
e.g. “Jack mentioned that he read On the Road”, in which
Jack is falsely recognised as the author Jack Kerouac.
•  Stop: remove common words that are stopwords
•  Stat: Estimate how ambiguous a lexicalisation of an object is
compared to other lexicalisations of objects of the same relation
•  For every lexicalisation of an object of a relation, retrieve the number of
senses using the Freebase API (example: for Jack n=1066)
•  Compute frequency distribution per relation with min, max, median (50th
percentile), lower (25th percentile) and upper quartile (75th percentile)
(example: for author: min=0, max=3059, median=10, lower=4, upper=32)
•  For every lexicalisation of an object of a relation, if the number of senses >
upper quartile (or the lower quartile, or median, depending on the model),
discard it (example: 1066 > 32 -> Jack will be discarded)

12
Seed Selection: discarding negative seeds
•  Creating negative training data: all entities which appear in the
sentence with the subject, but are not in a relation with it, will
be used as negative training data
•  Problem: knowledge bases are incomplete
•  Idea: object lexicalisations are often shared across entities,
e.g. for the relation genre
•  Check if an unknown lexicalisation is a lexicalisation of a
different relation
•  Incomp: for every lexicalisation l of a property, discard it as
negative training data if any of the properties of the class we
examine has an object lexicalisation l
Isabelle Augenstein

13
Distant Supervision system: corpus
•  Web crawl corpus, created using entity-specific search
queries, consisting of 450k Web pages
Isabelle Augenstein
Class Property / Relation
Book author, characters, publication date, genre,
ISBN, original language
Musical Artist album, active (start), active (end), genre,
record label, origin, track
Politician birthdate, birthplace, educational institution,
nationality, party, religion, spouses

14
Distant Supervision system: relation candidate identification
•  Two step process: recognise entities, then check if they
appear in Freebase
•  Use Stanford 7-class NERC to identify named entities (NEs)
•  Problem: domain-specific entities (e.g. album, track) are often
not recognised
•  Solution: use heuristic to recognise (but not classify) more
NEs
•  Get all sequences of capitalised words, and noun sequences
•  Every subsequence of those sequences is a potential NE,
“pop music” -> “pop music”, “pop”, “music”
Isabelle Augenstein

15
Distant Supervision system
•  Features: standard relation extraction features, including BOW
features, POS-based features, NE class
•  Classifier: first-order CRF
•  Model: one model per Freebase class to classify a relation
candidate (occurrence) into one of the different properties or
NONE, apply to respective part of corpus
•  Seed Selection: apply different seed selection methods Unam,
Stop, Stat75, Stat50, Stat25, Incomp
•  Merging and Ranking: aggregate predictions of occurrences
with same surface form
•  E.g.: Dublin could have predictions MusicalArtist:album, origin and NONE
•  Compute mean average of confidence values, select highest ranked one

16
Evaluation
•  Split corpus equally for training and test
•  Hand-annotate the portion of the test corpus which has NONE
prediction (no representation in Freebase)
Isabelle Augenstein

17
Results: precision per model, ranked by confidence
y-axis: precision,
x-axis: min confidence
(e.g. 0.1 are all
occurrences with
confidence >= 0.1)
Overall precision:
unam_stop_stat25: 0.896
unam_stop: 0.842
stop: 0.84
baseline: 0.825
incomp: 0.74
0.7
0.75
0.8
0.85
0.9
0.95
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
unam_stop_stat25
unam_stop_stat50
unam_stop_stat75
unam_stop
stop
baseline
incompl

18
Results Summary
•  Best-performing model (unam_stop_stat25) has a precision of
0.896, compared to baseline model with precision of 0.825,
reducing the error rate by 35%
•  However, those seed selection methods all come at a small
loss of the number of extractions (20%) because they reduce
the amount of training data
•  Removing potentially false negative training data (incomp)
does not perform well
•  Too many training examples are removed
•  The training examples which are removed are lexicalisations which
have the same types of values, those are crucial for learning
•  Especially poor performance for numerical values
Isabelle Augenstein

19
Related work on dealing with noise for distant supervision
•  At least one models:
•  Relaxed distant supervision assumption, assume that just one of the
relation mentions is a true relation mention
•  Graphical models, inference based on ranking
•  Very challenging to train
•  Hierarchical topic models:
•  Only learn from positive training examples
•  Pre-processing with multi-layer topic model to group extraction patterns
to determine which ones are specific for each relation and which are not
•  Pattern correlations:
•  Probabilistic graphical model to group extraction patterns
•  Information Retrieval approach:
•  Pseudo relevance feedback, re-rank extraction patterns

20
Comparison of our approach to deal with ambiguity with
related approaches
•  Related approaches all try to solve the problem of ambiguity
within the machine learning model
•  Our approach deals with ambiguity as a pre-processing step for
creating training data
•  While related approaches try to address the problem of noisy
data by using more complicated models, we explored how to
exploit background data from the KB even further
•  We explored how simple, statistical statistical methods based on
data already present in the knowledge base can help to filter
unreliable training data
Isabelle Augenstein

21
Conclusions
•  Simple, statistical methods based on background knowledge
present in KB perform well at detecting ambiguous training
examples
•  Error reduction of up to 35% can be achieved by strategically
selecting seed data
•  Increase in precision is encouraging, however, this comes at
the expense of the number of extractions (20% fewer
extractions)
•  Higher recall could be achieved by increasing the number of
training instances initially
•  Use a bigger corpus
•  Make better use of knowledge contained in corpus

22
Future Work
•  Distantly supervised named entity classification for relation
extraction to improve performance for non-standard entities,
joint models for NER and RE
•  Relax distant supervision assumption to achieve a higher
number of extractions: extract relations across sentence
boundaries, coreference resolution
•  Combined extraction models for information from text, lists
and tables on Web pages to improve precision and recall

23Seed Selection for Distantly Supervised Web-Based
Thank you for
your attention!
Questions?
Isabelle Augenstein

Seed Selection for Distantly Supervised Web-Based Relation Extraction

Recommended

More Related Content

Viewers also liked (19)

Similar to Seed Selection for Distantly Supervised Web-Based Relation Extraction (17)

More from Isabelle Augenstein (17)

Recently uploaded (20)

Seed Selection for Distantly Supervised Web-Based Relation Extraction