D3.1 Multilingual content processing methods

www.comrades-project.eu
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under
grant agreement No 687847
D3.1 Multilingual content processing methods
Version Date Modification reason Modified by
0.1 30-11-2016 Initial draft Diana Maynard
Project acronym: COMRADES
Project full title:
Collective Platform for Community
Resilience and Social Innovation during
Crises
Grant agreement no.: 687847
Responsible: Diana Maynard
Contributors:
Diana Maynard, Johann Petrak, Adam Funk,
Leon Derczynski
Document Reference: D3.1
Dissemination Level: <PU >
Version: Final
Date: 30/12/16
Disclaimer: This document reflects only the author's view and the Commission is not
responsible for any use that may be made of the information it contains.

2 | P a g e
© Copyright 2016 Tina Comes, Stina Torjesen, Kenny Meesters

3 | P a g e
Table of contents
Table of contents .............................................................................................................3
List of Figures...................................................................................................................5
Executive summary..........................................................................................................6
1 Introduction .............................................................................................................7
1.1 Relationship with other work packages.......................................................................8
1.2 Structure of the deliverable .........................................................................................8
2 TwitIE: a tool for named entity recognition from social media....................................9
2.1 Language Identification..............................................................................................10
2.2 Tokeniser ....................................................................................................................12
2.3 Twitter Normaliser .....................................................................................................12
2.4 Hashtag Tokeniser......................................................................................................14
2.5 Gazetteers ..................................................................................................................15
2.6 POS tagger..................................................................................................................15
2.7 Named entity recognizer and co-reference...............................................................17
2.8 Evaluation...................................................................................................................18
3 TwitIE for French ....................................................................................................19
3.1 French Tokeniser ........................................................................................................19
3.2 French Hashtag Tokeniser..........................................................................................19
3.3 French normalizer ......................................................................................................20
3.4 French POS tagger......................................................................................................21
3.5 Gazetteers ..................................................................................................................24
3.6 Named Entity recogniser............................................................................................24
3.7 Orthographic co-reference.........................................................................................24
3.8 Evaluation of French TwitIE........................................................................................25
4 TwitIE for German ..................................................................................................26
4.1 German tokenizer.......................................................................................................26
4.2 German hashtag tokenizer.........................................................................................26
4.3 German normalizer ....................................................................................................27
4.4 German POS tagger....................................................................................................28
4.5 German gazetteer ......................................................................................................29
4.6 German named entity recognizer ..............................................................................29
4.7 Evaluation of German TwitIE......................................................................................30
5 YODIE: Named Entity Disambiguation and Linking ...................................................31
5.1 Multilingual YODIE......................................................................................................32

4 | P a g e
5.1.1 Preparation Phase ..............................................................................................33
5.1.2 Processing Pipeline.............................................................................................36
5.2 Deployment................................................................................................................39
5.3 Performance evaluation of YODIE..............................................................................40
5.3.1 English Tweet Pipeline........................................................................................41
5.3.2 French YODIE Pipeline........................................................................................42
5.3.3 German YODIE Pipeline......................................................................................43
5.4 Free Open Source Components .................................................................................44
6 Summary................................................................................................................45
7 References .............................................................................................................45

5 | P a g e
List of Figures
Figure 1: TwitIE pipeline in GATE ......................................................................................10
Figure 2: Example of text normalisation in TwitIE..............................................................13
Figure 3: Example of hashtags split into tokens in TwitIE...................................................15
Figure 4: Hashtag tokenisation in French ..........................................................................20
Figure 5: Example of a normalised French abbreviation in GATE ........................................20
Figure 6: Mapping of FTB tags to Universal POS tags.........................................................23
Figure 7: Example of German hashtag tokeniser in GATE...................................................27
Figure 8: Example of a normalised German abbreviation in GATE......................................28
Figure 9: Entities in text linked to their respective entries in Wikipedia ..............................31
Figure 10: Information from an IRI/label pair for “NASA” ..................................................36
Figure 11: Demo of YODIE in GATE Cloud ..........................................................................40

6 | P a g e
Executive summary
This report describes the tools developed for multilingual text processing of social media. It
gives details about the linguistic approaches used, the scope of the tools, and some results of
performance evaluation. WP3 focuses on developing methods to detect relevant and
informative posts, as well as content with clear validity, so that these can be dealt with
efficiently without the clutter of uninformative or irrelevant posts obscuring the important
facts. In order to achieve these objectives, low-level linguistic processing components are first
required in order to generate lexical, syntactic and semantic features of the text required by
the informativeness and trustworthiness components to be developed later in the project.
These tools are also required by components developed in WP4 for the detection of
emergency events, modelling and matchmaking. They take as input social media and other
kinds of text-based messages and posts, and produce as output additional information about
the language of the message, the named entities, and syntactic and semantic information.
In this report, we first describe the suite of tools we have developed for Information
Extraction from social media for English, French and German. While English is the main
language of messages dealt with by the tools in this project, it is very useful to be able to both
recognise and deal with messages in other languages. French and German are therefore used
as examples to show the adaptability of our tools to other languages, and our multilingual
components thus serve as a testbed for new language adaptation techniques with which we
have experimented during the project. Various aspects of these tools are evaluated for
accuracy.
Second, we describe the tools we have developed for entity disambiguation and linking from
social media, for English, French and German. These ensure not only that we extract relevant
instances of locations, names of people and organisations, but that we know which particular
instance we are talking about since these names may potentially refer to different things. By
linking to a semantic knowledge base, we ensure both disambiguation and also that we have
additional knowledge (for example, the coordinates of a location). The tools are evaluated for
accuracy as well as speed, since traditionally these techniques are extremely slow and
cumbersome to use in real world scenarios. The tools are all made available as GATE Cloud
services.1
1
See https://gate.ac.uk/projects/comrades/index.html for details.

7 | P a g e
1 Introduction
The aim of this work package is to develop multi-lingual language analysis tools to process
social media messages and other text-based content. These tools include components for
basic language processing, and tools for detecting relevant and informative posts, as well as
assessing message trustworthiness and veracity. The low-level linguistic processing
components (Task 3.1) are required in order to generate lexical, syntactic and semantic
features of the text required by the informativeness and trustworthiness components (Tasks
3.2 and 3.3) as well as by the tools for event detection, modelling and matchmaking
developed in WP4.
This deliverable describes the tools for basic language processing in English, French and
German (Task 3.1). As discussed in D2.1, communities need to be able to provide and receive
information in their local languages, and hence COMRADES’ tools must demonstrate their
ability to cater for multiple languages. To this end, the tools described in this deliverable were
designed for English, French, and German, to showcase their multilingual power. As already
explained in D1.2, one of the communities we are currently heavily engaged with is in the
Kathmandu area in Nepal, which was recently hit by a devastating earthquake. Many of the
local inhabitants understand or speak English, and hence they tend to use English when
posting about their situation, to facilitate communication with foreign relief agencies and
other NGOs operating in their area.
We thus build on the existing language processing tools developed in GATE, a toolkit for
language processing which has been in continuous development at the University of Sheffield
over the last 20 years. GATE already contains a plugin for information extraction from tweets
for English, and some very basic information extraction tools from news texts for French and
German. We have improved some of the components of the English tool, adapted the French
and German tools to deal specifically with social media texts, and improved and extended
their components substantially. We have also continued development of the tool for entity
disambiguation and linking in GATE, YODIE, and adapted it to French and German. This tool
relies on the output of the information extraction tools (TwitIE and its French and German
counterparts) as input to the learning.
Where possible, we reuse existing components from elsewhere, rather than recreating new
tools unnecessarily. We have considered and evaluated a number of linguistic processing
components from other sources, such as components from the Stanford CoreNLP toolkit,
various language identification tools, and POS taggers for French and German, as well as
reusing components we have developed in previous EU projects such as ARCOMEM
(components for linguistic processing of English tweets) and DecarboNet (tools for German
linguistic processing).
The tools all take as input social media and other kinds of text-based messages and posts, and
produce as output additional information about the language of the message, the named
entities, and syntactic and semantic information. They are made available as web services via

8 | P a g e
the GATE Cloud2
and can also be used directly by project partners as Java APIs. Ultimately,
they will be connected to the Ushahidi platform.
1.1 Relationship with other work packages
The lexical, syntactic, and semantic information extracted from the social media content is
required as input for the informativeness and validity assessment algorithms (T3.2 and T3.3
respectively). It is also required for the tasks in WP4 on emergency event detection,
modelling, and matchmaking. The detected entities and other relevant information extracted
from the text in WP3 tasks will also be stored in the ontological models in WP4 (T4.1).
Conversely, multilingual lexical and formal semantic knowledge from these models is used to
help refine the multilingual NLP algorithms here.
1.2 Structure of the deliverable
The rest of the deliverable is organized as follows. In Section 2, 3 and 4, we describe the tools
for social media analysis from English, French and German respectively, giving details about
the linguistic methodology, and some evaluation of individual components where
appropriate. In Section 5, we describe the tool for named entity disambiguation and linking,
focusing on how we have extended and improved it from its original basic form since the start
of the project, and adapted it to French and German, and giving some details of performance
evaluation. We conclude in Section 6 with a summary and outlook.
2
See https://gate.ac.uk/projects/comrades/index.html for details.

9 | P a g e
2 TwitIE: a tool for named entity recognition from social media
In terms of Information Extraction (IE), and language processing in general, social media and
microblogs are probably the hardest kind of content to process. First, their brevity in the case
of microblogs (maximum 140 characters for tweets) makes them hard to interpret.
Consequently, ambiguity is a major problem, since IE methods tend to require contextual
information. Unlike longer news articles, there is a low amount of discourse information per
microblog document, and threaded structure is fragmented across multiple documents,
flowing in multiple directions. Second, social media also exhibits much greater language
variation, tends to be less grammatical, contains unorthodox capitalisation, and makes
frequent use of emoticons, abbreviations and hashtags, which can form an important part of
the meaning.
Tools for language processing of social media thus require adaptation to the specifics of social
media, in order to address the genre-specific challenges of noisiness, brevity, idiosyncratic
language, and social context. General-purpose tools (e.g. POS taggers and entity recognisers)
do particularly badly on such texts (Bontcheva et al., 2013).
The open-source GATE NLP framework (Cunningham et al., 2013) comes pre-packaged with
the ANNIE general purpose IE pipeline (Cunningham et al., 2002). This consists of the
following main processing resources: tokeniser, sentence splitter, part-of-speech tagger,
gazetteer lists, named entity recogniser, and coreference resolver. The ANNIE components
can be used individually or coupled together with new modules in order to create new
applications. For social media texts, most of the components in ANNIE (other than its
gazetteers and sentence splitter, which are generic) need to be re-trained and/or adapted to
deal with the special nature of the language. TwitIE (Bontcheva et al., 2013) is GATE’s social
media-adapted version of ANNIE, which has been tested extensively on microblog messages
(Derczynski et al., 2015), and is distributed as a plugin in GATE.
Figure 1 shows the TwitIE pipeline and its components. Re-used ANNIE components are
shown in blue boxes, whereas the ones in red boxes are new and specific to the microblog
genre. TwitIE is not new, but we have made some improvements to the original tool in the
context of this task, both to improve some performance issues and to enable greater
flexibility for its adaptation to other languages, which is a major part of the effort in this task.
We summarise briefly below the main components of the English TwitIE pipeline and any
improvements we have carried out, before describing the adaptation of the tool for French
and German in the subsequent sections. More detailed information about TwitIE is available
on the GATE website.3
3 https://gate.ac.uk/wiki/twitie.html

10 | P a g e
Figure 1: TwitIE pipeline in GATE
As depicted in Figure 1, the TwitIE pipeline comprises the following main components:
 Language identification
 Tokeniser
 Normaliser
 Gazetteer
 Sentence Splitter
 Part-of-Speech Tagger
 Named entity recogniser (including co-reference resolution)
We describe briefly these components below, and explain how we have improved or
experimented with them in this project, where relevant. The main improvements to the
English version of TwitIE made during the project, as will be described, comprise
experimentation with and development of:
 New language identification tool (langid.py)
 Improvements to the normaliser
 New POS tagger (BiLSTM)
2.1 Language Identification
One of the first steps in language processing is often to determine which language has been
used for a target document. This is harder than one would expect, and many methods have
been developed for language identification as a result. However, the majority of these have
been developed and evaluated over newswire text, and break down when applied to social
media.

11 | P a g e
Typically, language identification tools rely on consistent repeating subsequences within
target documents. For example, the five-character sequence “ the ” (including spaces) could
be an indicator of English. The longer the sequences, the higher the confidence of a match;
sometimes, these sequences can be long enough to include multiple words. And indeed, data-
intensive approaches to language processing have typically involve training over long texts
with fairly regular use of surface forms and phrases – especially in newswire. This means that
most language ID tools perform better on long, formal documents.
Social media text on the other hand is typically short. In addition, while the whole words are
generally spelled correctly, the space pressure in some venues – like Twitter, with its 140
character limit – leads to the dropping of some words. For example, prepositions might be
ditched in favour of verbs and nouns (Halliday (2004)’s “lexical words”). In addition, people
tend to compose their social media messages quite carefully when they are out of space,
involving unusual ordering and choice of words to convey a concept. This compressed
phrasing means that word and letter sequences on social media may be novel, and so will not
overlap very well with those in newswire, which traditional language identification tools rely
on.
The TwitIE system uses the TextCat (Cavnar and Trenkle, 1994) language identification
algorithm, which relies on n-gram frequency models to discriminate between languages.
More specifically, it integrates the TextCat adaptation to Twitter (Carter et al., 2013) which
works currently on five languages. It is 97.4% accurate overall, with per language accuracy
ranging between 95.2% for French and 99.4% for English (Derczynski et al., 2013a).
However, although TextCat achieves excellent results when determining between only a small
number of known languages, it struggles when the set of possible languages is wider. To
address this shortfall, in this project we have adopted the langid.py tool (Lui et al. 2012). This
has good performance on social media text, as shown in competitive evaluations (Derczynski
et al. 2015). Additionally, because it decides between almost 100 languages, misclassification
and the results of decisions on documents in languages not found in the training data are less
harmful than, for example, the five-class TextCat-based tool currently in TwitIE.
This tool is designed to be robust to domain changes. It is trained by taking documents from
multiple domains, and then determining which features are stable indicators of a particular
language across all input domains. This reduces the chance of domain-specific transient false
positive indicators, and is achieved by means of the LD feature selection algorithm (Lui et al.
2011), which is essentially information gain-based and requires explicit per-domain
separation. This approach mirrors well the situation with social media: there are many
authors with no set unifying style or genre – in fact, the direct opposite. Each author,
community, or dimension of homophily functions as a micro-domain which is represented
linguistic (as well as in other ways). A domain-robust approach therefore has a stronger
chance of rising above this noise and avoiding the many potential false positives.
To use langid.py, its web-service application is started; this is then accessed by GATE via API.
Language identification is fast, and also scalable by starting as many processes as required in
parallel and then load-balancing the access port. The tool has not yet been integrated into the
GATE applications, but will be in the near future, so that the tools in future tasks can benefit
from it.

12 | P a g e
2.2 Tokeniser
Tokenisation aims to segment a text into words, numbers, symbols etc (typically separated in
English by white space). Tokens may also be categorized further, for instance, words may
have orthography information (uppercase, lowercase, mixed case) associated with them.
Tokenising well-written text is generally reliable and reusable, since it tends to be domain-
independent, e.g. the Unicode tokeniser bundled with the ANNIE.
However, such general purpose tokenisers need to be adapted to work correctly on social
media, in order to handle specific tokens like URLs, hashtags, user @mentions, special
abbreviations (e.g. RT, LOL), and emoticons. These elements, which are highly prevalent in
tweets, prove particularly disruptive to conventional text processing tools (Derczynski et al.,
2013a). Therefore, tokenising these accurately is important.
The TwitIE tokeniser is an adaptation of the ANNIE English tokenizer, which pre-categorises
abbreviations and URLs as a single token each. Hashtags, emoticons and user mentions also
have special treatment. Other issues, such as normalization, and the tokenization of text
within hashtags, are dealt with in optional separate modules, so that tokenization itself is fast
and generic, while tailored to the needs of information extraction.
2.3 Twitter Normaliser
Noisy environments such as microblog text pose challenges to existing tools, being rich in
previously unseen tokens, elision of words, and unusual grammar. Normalisation is commonly
proposed as a solution for overcoming or reducing linguistic noise, such as unknown words,
elision, unusual grammar and so on (Sproat et al., 2001). For example, POS taggers may not
operate correctly if a word is misspelt or unknown. The task is generally approached in two
stages: first, the identification of orthographic errors in an input discourse, and second, the
correction of these errors.
The Twitter Normaliser aims to create normalised versions of misspellings, slang, and
acronyms and abbreviations specific to social media, such as 2moro (tomorrow), lol (laughing
out loud), flud (flood) etc. It is a rather contentious issue which words should be “corrected”
and which should be left as they are because they were written that way deliberately. In
general, we only make normalized versions of words where it would be beneficial for the
processing tools or in order to clarify the meaning. For example, misspelt words may not
recognized by a POS tagger. However, slang terms for common nouns could be left intact. We
always retain the original word(s), however, since this information may also be useful (for
example, in detecting reliability/trustworthiness.
The TwitIE Normaliser combines two spelling correction dictionaries – one which is generic,
and one which is specific to social media. The latter contains entries such as 2moro and brb,
similar to Han et al. (2012). Figure 2: Example of text normalisation in TwitIE shows an
example tweet, where tho has been normalized to though (shown by the value of the
origString and string features respectively).

13 | P a g e
Figure 2: Example of text normalisation in TwitIE
The original Twitter Normaliser had a number of issues and was not terribly effective,
generating many erroneous normalisations. There were also various technical problems with
the design. We have therefore refactored the normalizer to resolve some of these issues.
From a technical point of view, the encoding of the various lexicons it uses were originally
hard-coded to the platform default encoding, which could create problems for lexicons
involving accents etc. (as required for French and German). This has been changed so that
each lexicon can have its encoding explicitly set. Second, there was no possibility to load a
lexicon from a source other than a file on disk, which could create problems in future due to
the way in which resources will be managed in forthcoming versions of GATE.
Second, there were some issues with case-insensitive matching. The original tool worked in
such a way that while the matching of the lexicon was case-insensitive, the return value is
made to match the casing of the input (i.e. making the return value upper case if the input is
upper case). This makes sense as we generally do not want to change the case information
(since even if incorrectly cased, this might bring useful additional information – for example,
opinionated words in all uppercase tend to convey stronger sentiment; requests for help in all
uppercase might indicate urgency; case information can also be critical in helping to
determine the validity/trustworthiness of a tweet (Zubiaga et al., 2014). However, for the
slang lexicon, the initial string in the text was always converted to lowercase before matching,
which meant that any upper case entry would be ignored. This has been modified to allow the
case-sensitivity of matching to be specified explicitly as a parameter, enabling more control of
matching (in some cases, we might want to only match case-sensitively, and we should not
try to constrain unnecessarily how a user might want to match).
Third, the lexicon files were restricted to a very specific format (a very restricted kind of csv
file), and only one slang lexicon was possible. The possibilities have now been expanded so
that more than one slang lexicon can be used (e.g. so that one can choose between an
abbreviation lexicon, a slang lexicon, and any other kinds of lexicon one might wish to add).
This enables the possibility for testing different configurations of lexicon and also for opening
up the possibilities for what kinds of normalization can be done with the tool (e.g. only

14 | P a g e
acronym expansion), thus providing much greater flexibility. The tool has been modified so
that either a single-column CSV can be specified, where each line is the relative path to
another file to load, or a multi-column CSV can be specified as before. Additionally, the
parsing of this CSV is now performed by a proper CSV parser library, which means that it can
handle things like commas within values if the entry is double-quoted.
In terms of the performance of the normalizer, we have made several improvements. First,
the set of slang terms has been cleaned and expanded. Second, it has been separated into
separate lexicons for acronyms (e.g. lol – laughing out loud), slang (e.g. bcuz – because) etc.
for future experimentation with accuracy, and for flexibility. Third, a specific misspelling
lexicon has been constructed (e.g. recieve – receive) from some lists found on the internet
and some manual additions. We might also envisage adding emoticons and their
interpretations as a further list, since knowing the meaning of a smiley face could be
important.
2.4 Hashtag Tokeniser
Hashtags on Twitter and in other social media often contain much useful information, but this
information is not always easily accessible because hashtags may contain not just a single
word but multiple words squashed together into a single Token. The hashtag tokenizer
attempts to separate the relevant words out from the hashtag. For example #hurricanesandy
should be split into the individual components Hurricane and Sandy. For a person, it is usually
quite easy to identify what these words should be, but for a machine, it is much harder,
unless these words are camelcased (#HurricaneSandy). Indeed, even humans may struggle to
correctly identify the relevant words: a famous example of confusion occurred when
Margaret Thatcher died and a hashtag #nowthatcherisdead was popular on Twitter.
Unfortunately, some people interpreted this as “now that Cher is dead” rather than “now
Thatcher is dead” and rumours quickly spread that the famous pop star had died.
Having access to the correct individual words in the hashtag is useful for tasks such as entity
and event recognition, topic recognition, and sentiment analysis, among other tasks, and may
also be useful for trustworthiness and validity classification. For example, off-topic posts are
less likely to have the standard event hashtag: a tweet about a different hurricane would be
unlikely to have either #hurricanesandy or #sandy as a hashtag, though it might have a
different hurricane-related hashtag such as #hurricane or #hurricanematthew.
Full details of the hashtag tokenizer can be found in (Maynard and Greenwood, 2014). The
module essentially uses a Viterbi-like algorithm to match from left to right against a dictionary
of known words, plus some additional rules to prioritise certain elements. The Linux
dictionary for English is used for this, along with gazetteer lists of known English named
entities (locations, organisations etc) and a dictionary of slang words (as used in the
normalization module). Figure 3 shows an example of retokenized hashtags in GATE: we can
see for example that #hurricanesandy has been retokenised as # - hurricane – sandy, while
#neverforget has been retokenised as #-never-forget.
The hashtag tokenizer was evaluated in (Maynard and Greenwood, 2014), and the results
discussed further. On a gold standard set containing 2010 hashtags and 4538 tokens, the
system achieved 98.12% Precision and 96.41% Recall, and an F1 of 97.25%.

15 | P a g e
Figure 3: Example of hashtags split into tokens in TwitIE
2.5 Gazetteers
In GATE, gazetteers contain lists of things which assist with entity finding, as well as other
low-level tasks such as identifying numbers, slang words etc. The gazetteers used for TwitIE
are pretty much identical to those used in ANNIE, with some small variations. We have made
some minor improvements to these to update with additional words and terms, such as some
extra location and organization names.
2.6 POS tagger
The lack of high quality part-of-speech taggers tailored specifically to social media, and
particularly Twitter, impairs the accuracy of key downstream NLP techniques (e.g. named
entity recognition, term extraction), and by extension, overall application results. Microblogs,
such as tweets, are characterised by: short messages; inclusion of URIs; username mentions;
topic markers; and threaded conversations. They often include colloquial content containing
abbreviations and errors, as discussed above. Some of these phenomena comprise linguistic
noise which, when coupled with message brevity and the lack of labelled corpora, make
microblog part-of-speech tagging very challenging. Alongside the genre’s informal nature,
such limits encourage “compressed” utterances, with authors omitting not only needless
words but also those with grammatical or contextualising function.
The POS tagger in TwitIE consists of an adapted Stanford tagger (Toutanova et al., 2003),
trained on tweets tagged with the Penn TreeBank (PTB) tagset. It is more fully described in
(Bontcheva et al, 2013). Extra tag labels have been added for retweets, URLs, hashtags and
user mentions. We trained this tagger using hand-annotated tweets (Ritter et al., 2011), the
NPS IRC corpus (Forsyth and Martell, 2007), and news text from PTB (Marcus et al., 1993).
The resulting model achieves 83.14% token accuracy, which is still, however, below that
achieved on news content.
Part-of-speech tagging is a central problem in NLP, and a key step early in many NLP pipelines.
Machine learning-based POS taggers can exploit labeled training data to adapt to new genres
or even languages, through supervised learning. Algorithm sophistication apart, the
performance of these taggers is reliant upon the quantity and quality of available training
data. Consequently, lacking large POS-annotated resources and faced with prevalent noise,
state-of-the-art POS taggers perform poorly on microblog text (Derczynski et al., 2013b), with
error rates up to ten times higher than on newswire. This is likely to be particularly

16 | P a g e
problematic in tweets emerging from situations where authors will not prioritise formal
presentation of a message as much as its general content and timeliness - e.g., in crises.
One core use of POS tags is to help identify which parts of a sentence form phrases and which
parts are linked to each other. In the context of event identification, it helps determine which
words form the head event in a sentence -- typically a verb or verb phrase describing some
action or occurrence (e.g. "We have run out of water"), or a nominative event ("There has
been an explosion.").
Conventional part-of-speech taggers were developed using datasets over text from the 1980s
and 1990s; for English, the Penn Treebank. These datasets have specific biases. Being
newswire, they have been filtered and linguistically constrained through application of
Associated Press guidelines. However, other subtler biases also affect the data, particularly
regarding lexical choice. For example, for content to appear in newswire, it must have been
deemed newsworthy, which may only apply to a restricted set of named entities (e.g. person,
organisation and location names). This means that other named entities (NEs) will almost
always constitute new, unseen words when encountered by a system trained on newswire.
Tagging of unknown words forces the tagger to rely on contextual clues. Errors on these words
make up a large part of the mis-tagged tokens. For example, in English tweets, approximately
one in five words are unknown, and accuracy on these is typically below 30% (Derczynski et
al., 2013b). The impact of this is that it becomes impossible to correctly chunk or parse
sentences, so that locations, expressions of need, and relations between resources and
statuses all risk being lost.
In fact, spelling variation is very broad in social media, and demonstrably decoupled from
spelling ability (Drouin 2009). Rather, it is related to individual dialects and deliberate stylistic
choices. Therefore, this variation is something we should expect to encounter in any situation
across any part of the world, and from sources that are both reliable and unreliable. In
addition, the nature of crises is such that they will often bring to prominence places and
names that have not occurred in the data or public discourse before (e.g. “Banda Aceh” was
not a broadly-covered term prior to the 2004 Christmas earthquakes). Therefore, robustness
against this kind of noise is paramount to the NLP pipeline.
While the existing Twitter POS tagger is an improvement over more generic taggers, it still has
performance issues. Furthermore, for other languages, we do not have Twitter-adapted POS
taggers readily available. Therefore, for this task, we have adapted technology that
demonstrates excellent performance on part-of-speech tagging across domains, but also
achieves much improved accuracy on out-of-vocabulary terms. For this, we compared existing
state-of-the-art taggers, on unknown words, for English tweets, as shown in Table 1.
Tagger Word accuracy
Stanford 83.14%
TnT 79.17%
TwitIE 88.69%
Table 1: Performance of existing taggers for English tweets

17 | P a g e
One good option for our new tagger is to leverage unsupervised data. Most POS taggers
require labelled data, which is not readily available for social media for most languages –
certainly not to the extent required for deep learning, which really excels with datasets in the
order of 106
examples of training data, whereas POS tagsets are lucky to reach the order of
105
. To this end, we adapt the CMU part of speech tagger (Owoputi et al., 2013) – geared for
tweets, one of the toughest forms of social media to process – by training it on universal
dependency data for the two new target languages, French and German (Nivre et al., 2016).
Additionally, we build monolingual sample corpora, taken from a seven-year archive of tweets
maintained by the Sheffield team and filtered by language using langid.py (Lui and Baldwin,
2013), for each of these languages, and proceed to use this to generate clusters of social
media terms using a generalised Brown clusterer (Derczynski and Chester, 2016). These
clusters have been shown before to be effective for resolving social media spelling variations.
For English, we maintain the state-of-the-art tagger (Derczynski et al., 2013b) with updated
normalisations.
This tagger is made portable and scalable, then connected to GATE. Portability is achieved by
wrapping it as a REST web service, that takes and receives JSON-format requests, with a list as
input plus the original whole sentence, and list as output.
Machine learning algorithms tend to excel at overcoming noise in data given enough volume –
similar to many earlier tools, like CRF, given time and parameter flexibility. For French and
German, as no prior gold standard data exists, we created a small corpus for evaluation using
the universal tagset (Petrov et al., 2011). Results are given in Table 2.
Language Word accuracy Sequence accuracy
French 76.15% 80.77%
German 66.67% 72.48%
Table 2: Accuracy of the new Twitter POS tagger for French and German
From these results, we can see that despite using a tool tailored to social media alongside
large amounts of unsupervised data, with the correct feature extractors, the Stanford tool
actually excelled at French and German. To this end, we release the test corpus as well as
social-media adapted French and German taggers, the first of their kind.4
2.7 Named entity recognizer and co-reference
The TwitIE named entity recognizer and co-reference modules are also very similar to the
ones used for ANNIE. The main difference is that because social media tends to have many
proper names which do not have correct capitalization, and also often tends to use only first
names instead of full names for people, some additional precautions are put in place to avoid
overgeneration when capitalization rules are relaxed. For example, many entity names will
not be annotated as proper nouns by the POS tagger, especially if they are ambiguous with
common nouns (e.g. will, mark). First names are therefore only recognized in TwitIE when
appearing as part of a longer name or, if in isolation, in certain very specific contexts. There
are also some rules to deal with entities occurring as part of hashtags and @mentions. We
4 https://gate.ac.uk/projects/comrades/index.html

18 | P a g e
have not modified these beyond a few minor improvements, as for the gazetteers. The co-
reference module is identical to the one used in ANNIE, as this is not really affected by the
problems of social media.
2.8 Evaluation
The TwitIE named entity recognition tool has been evaluated previously on a corpus of tweets
and compared with ANNIE, which is designed for general-purpose text and not specifically
social media. TwitIE outperformed ANNIE in this evaluation considerably (Derczynski et al.,
2013a): overall, TwitIE achieved 77% Precision, 83% Recall and 80% F1, while ANNIE achieved
47% Precision, 83% Recall and 60% F1.
In another evaluation (Greenfield et al., 2016), TwitIE was compared with various state-of-
the-art tools for named entity recognition on Twitter: the Stanford tools (Finkel et al., 2005),
2 flavours of MITIE5, TwitterNLP (Ritter et al., 2011), and a fused system of their own. In this
experiment, TwitIE scored top for Organisations, second for Locations, and last for Persons.
We suspect that the issue with the Persons in this evaluation was due to the fact that we had
not seeded our gazetteers with current personalities (for example, current pop star, reality TV
stars and the like). For a discussion of this issue, see (Derczynski 2016). Note that these
evaluations were on the version of TwitIE before the enhancements were carried out; we
plan additional evaluations in future.
5 https://github.com/mit-nlp/MITIE

19 | P a g e
3 TwitIE for French
The TwitIE system for French has been developed entirely within the course of this task, and
consists of the following components:
 language identification (as for English)
 French tokeniser
 French hashtag tokeniser
 French normaliser
 Sentence splitter (as for English)
 French POS tagger
 French gazetteer
 French named entity recogniser
 Orthographic co-reference (as for English)
We now describe the new components in more detail; the sentence splitter and co-reference
modules are not language-specific and so can be re-used directly from the English version
with no modification.
3.1 French Tokeniser
The French tokenizer was developed by combining components of tokenization from the
English TwitIE tokenizer and the French standard tokenizer. The English TwitIE tokenizer has
some elements specific to social media processing such as the recognition of URLs, hashtags,
emoticons, smileys, @mentions, separators and so on, which are also required for French. On
the other hand, the French standard tokenizer has some elements specific to French such as
hyphenated words (e.g. apprend-on) and cases where two words are joined by an apostrophe
without a space (e.g. l’homme) which need to be identified as two separate Tokens rather
than a single one. The latter case is similar to the English case (e.g. didn’t) but involves
different characters and words, so the rules need to be altered accordingly.
3.2 French Hashtag Tokeniser
While the concept of hashtag tokenization is identical for French and English, this module
requires a dictionary of known words against which it compares elements in the hashtag in
order to find a match. For the English version, the Linux dictionary for English is used for this,
along with gazetteer lists of known English named entities (locations, organisations etc) and a
dictionary of slang words (as used in the normalization module). These need to be adapted
for French. For the French version, we used a dictionary of 208,000 words downloaded from
JUSTWORDS!6
for the generic wordlist, and we used the relevant entity and slang lists from
our French gazetteer and normalization components described in the following sections. An
example of a tokenized French hashtag in GATE is shown in Figure 4: looking at the hashtag
#risquenaturel in the bottom part of the screenshot, we can see the red Hashtag block
denoting the hashtag span, and below it three pink Token blocks, denoting the hashtag
symbol and the words risque and naturel respectively.
6 http://www.gwicks.net/dictionaries.htm

20 | P a g e
Figure 4: Hashtag tokenisation in French
We evaluated the hashtag tokenizer on a set of 1000 tweets about the environment,
described in Section 3.8. The tweets were annotated with the hashtag tokenizer and the
hashtags examined. Accuracy was established as 95.6%, which is not much lower than the
accuracy of the English hashtag tokenizer. The errors were mostly due to either the use of
English words (sometimes mixed with French) such as #BeforetheFlood (which was tokenized
as Beforethe + Flood according to the rules of camelcasing). There were also some cases
where it is hard to know what a correct tokenization should even be, such as
#weAAAre (tokenized as # we AAA re). #weAAAre is a campaign launched by the initiative
“Adaptation of African Agriculture to Climate Change” – the graphic on their website seems to
depict it with the same tokenization as our system produced, so perhaps this is as good as
any. For use on a specific dataset (e.g. about a particular crisis) it would be very simple to add
any known hashtag words to our lexicon before annotating.
3.3 French normalizer
Figure 5: Example of a normalised French abbreviation in GATE
The French Twitter normalizer is adapted from the English version, but requires a dictionary
of French words and a set of slang words/abbreviations and their extension/regularized form.
We used the same dictionary as for the French hashtag tokenizer, since jaspell (used for the
English normalizer, does not have a French dictionary. We compiled a list of 189 French slang

21 | P a g e
words and abbreviations used commonly on Twitter and in SMS messages from the internet.
Table 3 shows some examples: the first column shows the slang word; the second column
the normalized form; and the third column the English equivalent normalized form (not used
in the tool, just for clarity here). Figure 4 shows a screenshot from GATE of an example of a
French tweet where the abbreviation “pkoi” has been expanded into its normalized form
“pourquoi” (why), as can be seen by the feature “string”. Note that “origString” denotes the
original string in the text before normalization.
French slang Normalised French form English translation
Bjr Bonjour Hello
Bsr Bonsoir Good evening
C C’est It is
C mal1 C’est malin That’s clever
Keske Qu’est-ce que What
OQP Occupé Busy
A12C4 À un de ces quatre See you one of these days
Table 3: Examples of French Twitter slang
On initial testing, it was found that the French dictionary was missing many words, and there
were some errors due to encoding problems, so some effort was spent in cleaning this up.
Similarly, some test French tweets produced some new slang words and abbreviations which
we added to the slang dictionary manually.
We then evaluated the normalizer on a collection of 260 French tweets about climate change
that we extracted from the Media Watch for Climate Change tool7
(since we did not have a
specific collection of disaster tweets in French in the project at this time). Out of these 260
tweets, our normalizer changed 83 words, of which manual inspection showed that 75 were
correct (90.36% accuracy). The 8 incorrect ones were mostly due to abbreviations in capital
letters which the tool had attempted to resolve to the nearest real word component, or due
to uncommon but real words missing from our dictionary. We also found 1 occurrence of a
word which should have been normalized but was missed (lislam should have been corrected
to l’islam). However, there may have been a few more missed normalisations as it is
sometimes hard to tell the intended meaning of a word when incorrectly written, especially
with the capitalized abbreviations which without context were sometimes impossible to
decipher. The results are thus very promising, though this was only a small-scale evaluation,
and it is clear that more effort could be spent improving the lexicon and slang dictionary,
something easily done on an ad hoc basis as necessary, or with further iterations of testing
and improving.
3.4 French POS tagger
We added a generic POS tagger by taking a pre-trained model for French (not specifically for
Twitter) from the Stanford CoreNLP toolkit. This model uses a different set of POS tags
(French TreeBank set – FTB) from the one we use for English (Penn TreeBank set – PTB). It is a
7 http://www.ecoresearch.net/climate/

22 | P a g e
much simpler set than the PTB one, having only 15 lexical categories, while the PTB has 36
categories. The difference is in the level of detail; for example, the PTB has 3 categories for
adjectives (normal, comparative and superlative) and 6 categories of verbs (base form, past
tense, participle, 3rd
person singular present, non-3rd
person singular present). It also has tags
for things like interjections and symbols, while the FTB just classifies these as common nouns.
Because the FTB tagset has fewer categories than the PTB, we cannot map the former to the
latter, and thus we have to adapt any components which make use of the POS tags (such as
our Named Entity recognizer) to make use of this simpler set.
We decided to make a more generic solution to the POS tag mapping problem, since typically
for each new language, a different tag set is used. The solution adopted therefore was the
Universal POS Tagset8
, shown in Table 4, which enables mapping from many different sets
into a single simple representation.
Universal Tag Meaning
ADJ adjective
ADP adposition
ADV adverb
AUX auxiliary verb
CONJ co-ordinating conjunction
DET determiner
INTJ interjection
NOUN common noun
NUM numeral
PART particle
PRON pronoun
PROPN proper noun
PUNCT punctuation
SCONJ subordinating conjunction
SYM symbol
VERB verb
X other
Table 4:Universal POS tag set
We do not replace the original POS tags with these, because the original set may contain
functionality we wish to make use of, so we simply add an extra feature to each word giving
details of the Universal category as well as the original category. This allows the writing of
generic rules for other components (such as the Named Entity grammars and for YODIE)
involving categories which will hold across different languages. If needed, however, more
precise rules can also be added (for example, if we wish to distinguish between singular and
plural nouns). This makes our toolkit more future-proof in the case of adding new languages.
The mapping is implemented by simply specifying a list of mappings from the original set to
the universal set for each language, and then selecting which mapping we want to use for
each language-specific application. Some mappings already exist9
; however, we discovered
that some of these were erroneous or incomplete, so after some testing on a corpus, we
improved the mappings. Table 5 shows our final mapping. The mapping is implemented in the
form of a feature gazetteer, which adds a new feature to the existing Token. Figure 6 depicts
8 http://universaldependencies.org/u/pos/index.html
9 https://github.com/slavpetrov/universal-pos-tags

23 | P a g e
an example sentence in French, where the noun changement (change) was originally given
the category NC, and now also has the upos value NOUN.
FTB Tag Universal Tag
ADJ, ADJWH ADJ
ADV, ADVWH ADV
CC, CS CONJ
CLO, CLR, CLS PRON
DET, DETWH DET
ET X
N, NC NOUN
NPP PROPN
P ADP
PRO, PROREL, PROWH PRON
PUNC PUNCT
V, VIMP, VINF, VPP, VPR, VS V
I X
Table 5: Mapping from French tagset to Universal tagset
Figure 6: Mapping of FTB tags to Universal POS tags
The Stanford tagger for French (Green et al., 2013) has been evaluated previously on the test
portion of the Crabbé and Candito version of the French Treebank (Crabbé and Candito,
2008). This consisted of 1234 sentences and 37934 words, of which 1377 were unknown
words. Table 6: Evaluation of Stanford POS Tagger for French shows the results.
Correct Incorrect
Sentences 379 (30.71%) 855 (69.29%)
Tokens 35,790 (94.35%) 2144 (5.65%)
Unknown words 1190 (86.42%) 187 (13.58%)
Table 6: Evaluation of Stanford POS Tagger for French

24 | P a g e
We have also experimented with implementing a French POS tagger specifically for tweets, as
described in Section 2.6. We plan to replace the Stanford tagger with this model in the near
future.
3.5 Gazetteers
In GATE, gazetteers contain lists of things which assist with entity finding, as well as other
low-level tasks such as identifying numbers, slang words etc. We have developed a set of
gazetteer lists for NER in French through a combination of downloading relevant lists from
the Web (e.g. lists of major French cities, common French first names) and translating some
of the English ones. In some cases, we also retain the English lists, for example currency and
measurement abbreviations, lists of English first names, since these may also be found in
French texts. There are currently 127 lists, comprising around 66,000 entries (with some
duplicates, since some are ambiguous and could occur in multiple lists). For comparison, the
English NE tool has 165 lists, and 398,000 entries. Note that the English gazetteers have been
developed gradually over 17 years, so they are unsurprisingly more complete than the French
ones. However, the French gazetteers will certainly be extended over time.
3.6 Named Entity recogniser
The Named Entity recogniser comprises a set of hand-crafted grammar rules which take as
input annotations created by the previous components in the pipeline, and attempt to predict
which words in the text are named entities. While some of the rules in the English system are
language-independent and therefore can be reused for French, some of them must be
adapted due to issues such as different POS tags used and different grammatical
constructions. For example, in French, adjectives typically follow a noun (with some notable
exceptions), while in English, they typically precede a noun. These grammatical differences
have an impact on the way the rules are written.
The French grammar rules have been modified in order to take account of the universal POS
tags used in the tool, and extended considerably from the original very small set of rules.
They will continue to be extended and modified during the life of the project, in a continuous
process and testing and development. The results of the Named Entity recognition are
evaluated in Section 3.8, since the orthographic co-reference module also has an effect on
the final Named Entities extracted.
3.7 Orthographic co-reference
The orthographic co-reference module attempts to link together (co-refer) different instances
relating to the same physical entity, e.g. different mentions of the same person or location.
These might have different surface forms, which must also be matched together (e.g. Mr
Smith and John Smith). The matching between different lexical forms is performed by another
set of rules. Mostly, these rules are the same for different languages, or at least those which
have similar grammatical structures such as English and French, so they will work quite well
without adaptation. Initially, therefore, we do not alter this module for French; however,
there are some small improvements which could be made later. In addition to linking
different instances of the same entity, the module also enables matching between a known
entity and an unclassified entity (i.e. something which has been recognized as an entity but
whose type is unknown). If a match is found according to the rules, the unclassified entity is
then classified as Person, Organisation or Location according to the class of its co-referring

25 | P a g e
entity. The module therefore can improve the performance of the Named Entity recognition
module described in the previous section.
3.8 Evaluation of French TwitIE
In order to evaluate the performance of the Named Entity recognition as a whole, we created
a small corpus of 1,000 French tweets about the environment, downloaded from the Media
Watch for Climate Change10
and de-duplicated, resulting in 385 unique tweets. Although
these are not specifically about a disaster or crisis, they form a better set for evaluation
because they are not focused on a single specific event, which would be biased towards a
small set of organisations and locations in particular. An expert linguist knowledgeable about
named entity recognition annotated the set of tweets manually, followed by some discussion
about difficult cases with the developers of TwitIE and corrected following the establishment
of tighter guidelines. A sample of the annotations was then verified by a second expert
linguist.
The original French NER tool (before adaptation to Twitter and the improvements described
above) achieved 75.18% F1-measure, as shown in Table 7. The new French TwitIE tool
improved Precision slightly, and Recall considerably, with an F1-measure of 89.30%. A
breakdown is shown in Table 8.
Class Total Precision Recall F1
Location 622 88.08 84.98 86.50
Organisation 161 97.77 65.96 78.78
Person 377 71.03 39.07 50.41
Total 1160 85.32 67.19 75.18
Table 7: Evaluation of original generic French IE system
Location 622 95.07 94.63 94.85
Organisation 161 89.66 76.47 82.54
Person 377 81.38 87.16 84.17
Total 1160 88.58 90.02 89.30
Table 8: Evaluation of French TwitIE system
10 http://www.ecoresearch.net/climate/

26 | P a g e
4 TwitIE for German
The TwitIE system for German has been developed almost entirely within this task (except
where indicated), and consists of the following components:
 Language identification (as for English)
 German tokeniser
 German hashtag tokeniser
 German normaliser
 Sentence splitter (as for English)
 German POS tagger
 German gazetteer
 German named entity recogniser
 Orthographic co-reference (as for English)
We now describe the new components in more detail; as for French, the sentence splitter and
co-reference modules are not language-specific and so can be re-used directly from the
English version with no modification. We do not describe the language identification,
sentence splitting and co-reference modules as the rationale for not adapting them has
already been explained in Section 3.
4.1 German tokenizer
The German tokenizer was developed by combining components of tokenization from the
English TwitIE tokenizer and the German standard tokenizer. As for French, the German
TwitIE tokenizer has some elements specific to social media processing such as the
recognition of URLs, hashtags, emoticons, smileys, @mentions, separators and so on, which
are also required for German. On the other hand, the German standard tokenizer has some
elements specific to German – mainly, it does not require the specific post-processing
elements for dealing with apostrophes that English and French do, but it requires some
generic components used by those tools to deal with numbers and spaces.
4.2 German hashtag tokenizer
As for French, the concept of hashtag tokenization is identical to that for English. The
adaptation to German therefore requires a dictionary of known words against which it
compares elements in the hashtag in order to find a match. For the English version, the Linux
dictionary for English is used for this, along with gazetteer lists of known English named
entities (locations, organisations etc) and a dictionary of slang words (as used in the
normalization module). These need to be adapted for German. For the German version, we
used a dictionary of approximately 166,000 words from JUSTWORDS! for the generic wordlist,
and the entity and slang lists from our German gazetteer and normalization components
described below.
An example of a tokenized German hashtag in GATE is shown in Figure 7: looking at the quite
complex hashtag #hunderttausendfreundeeinverein (which could be translated as “a hundred
thousand friends, one club”) in the bottom part of the screenshot, we can see the red Hashtag
block denoting the hashtag span, and below it five pink Token blocks, denoting the hashtag

27 | P a g e
symbol and the words hunderttausend, freunde, ein and verein respectively. We should note,
however, that due to the nature of German compounding, the hashtag tokenizer does often
segment compound nouns into their respective components. It is often not clear whether
these should be considered as a single compound noun (one Token) or two individual nouns
joined together to form the hashtag (because these compound nouns often do not exist in
the dictionary). However, for practical purposes it is often immaterial, as it does not actually
matter if they are classed as two separate Tokens or one single one.
Figure 7: Example of German hashtag tokeniser in GATE
The hashtag tokenizer was evaluated on a set of 100 German tweets, described in Section 4.7.
The tweets were annotated and the hashtags were manually evaluated by a German-speaking
linguistic expert. Accuracy was assessed at XX%. Most of the errors found were quite obscure.
For example, #FCBWOB refers to two football teams, FCB & WOB, which were not recognized.
As discussed above, some compound nouns were split unnecessarily, for example
#Lebensqualität was tokenized as Lebens + qualität, and #Hausbau as Haus + bau. This
compound noun splitting does not alter the meaning and is unlikely to affect any other stage
in the pipeline.
4.3 German normalizer
The idea behind the German Twitter normalizer was to adapt it from the English version, in
the same way that the French Twitter normalizer was created. This required a dictionary of
German words and a set of slang words/abbreviations and their extension/regularized form.
We used the same dictionary as for the German hashtag tokenizer, and compiled a list of 64
German slang words and abbreviations used commonly on Twitter and in SMS messages from
the internet. It seems that in German, English slang words and abbreviations tend to be used
very frequently, which explains why the list is much shorter than for French. Table 9 shows
some examples: the first column shows the slang word or abbreviation; the second column
the normalized form; and the third column the English equivalent normalized form (not used
in the tool, just for clarity here).
German slang Normalised German form English translation
BM Bis Morgen See you tomorrow
GSD Gott sei Dank! Thank goodness!
kk OK OK
kP kein Problem No problem
LG Liebe Grüße Lots of love

28 | P a g e
pwned besiegt defeated
SuFu Suchfunktion search function
Table 9: Examples of German Twitter slang
Following some experimentation, we found that the methodology for replacing unknown
words using edit distance did not work well, and so we decided not to use this. Instead, we
kept only the replacement of slang words and abbreviations from the known list. We
implemented an extra facility in the GATE feature gazetteer, which enables not only the
replacement of one feature value for another, according to a list of replacement words (as in
the table above), but also the creation of a new feature which stores the original value of the
replaced word (i.e. the slang word). This means that the replacement table can simply be
updated with any new combinations as needed, and this can be run independently from the
full normalizer component.
Figure 8: Example of a normalised German abbreviation in GATE
Figure 8 shows a screenshot from GATE of an example of a German tweet where the
abbreviation “kP” has been expanded into its normalized form “kein Problem” (no problem),
as can be seen by the feature “string”. Note that “origString” denotes the original string in the
text before normalization.
In our evaluation corpus (described in Section 4.7), we actually did not find any examples of
normalization, so we did not evaluate the normalizer (the example was taken from a small set
of tweets collected for system development). However, we plan to evaluate this on another
dataset in future.
4.4 German POS tagger
For the German POS tagging, we used the Stanford CoreNLP model for German, in the same
way as we did for French. This model is trained on the TIGRE corpus, and has a different POS
tag set. Again, we mapped this set to the Universal Tagset using the same methodology as for
French. Shows the mappings: we can see that the TIGRE tagset is quite complex compared
with the French and Universal tagsets; however, most of these very precise distinctions are
not necessary for our tools.
We have also experimented with implementing a German POS tagger specifically for tweets,
as described in Section 2.6. We plan to replace the Stanford tagger with this model in the
near future.

29 | P a g e
TIGRE Universal
all punctuation .
ADJA, ADJD ADJ
ADV ADV
APPO, APPR, APPRART, APZR ADP
ART DET
CARD NUM
FM, ITJ X
KOKOM, KON, KOUI, KOUS CONJ
NE, NN, NNE NOUN
PDAT, PDS, PIAT, PIS, PPER, PPOSAT,
PPOSS, PRELAT, PRELS, PRF, PROAV
PRON
PTKA, PTKANT, PTKNEG, PTKVZ, PTKZU PRT
PWAT, PWAV, PWS PRON
TRUNC X
VAFIN, VAIMP, VAINF, VAPP, VMFIN, VMINF,
VMPP, VVFIN, VVIMP, VVINF, VVIZU, VVPP
VERB
XY X
Table 10: Mapping of German TIGRE POS tags to Universal POS tags
4.5 German gazetteer
As for French, we have developed a set of gazetteer lists for NER in German through a
combination of downloading relevant lists from the Web (e.g. lists of major German
companies, cities and regions, common German first names) and translating some of the
English ones. In some cases, we also retain the English lists, for example currency and
measurement abbreviations, lists of English first names, since these may also be found in
French texts. There are currently 125 lists, comprising around 66,000 entries (the same
number as for French), with some duplicates, since some are ambiguous and could occur in
multiple lists. The gazetteers will continue to be extended over time, as for French.
4.6 German named entity recognizer
As for French, while some of the rules in the English system are language-independent and
therefore can be reused for German, some of them must be adapted due to issues such as
different POS tags used and different grammatical constructions. The German grammar rules
have been modified in order to take account of the universal POS tags used in the tool (since
the previous version did not make use of these), and extended considerably from the original
very small set of rules. They will continue to be extended and modified during the life of the
project, in a continuous process and testing and development. The results of the Named
Entity recognition are evaluated in 4.7.

30 | P a g e
4.7 Evaluation of German TwitIE
The full German TwitIE system has been evaluated on a small corpus of 100 German tweets,
randomly selected from a de-duplicated larger set of 60,000 German tweets about the
environment, downloaded from the Media Watch for Climate Change. As with the French
evaluation corpus, although these are not specifically about a disaster or crisis, they form a
better set for evaluation because they are not focused on a single specific event, which would
be biased towards a small set of organisations and locations in particular. An expert linguist
knowledgeable about named entity recognition annotated the set of tweets manually,
followed by some discussion about difficult cases with the developers of TwitIE, and some
corrections following the establishment of tighter guidelines. A sample of the annotations
was then verified by a second expert linguist.
The original German NER tool (before adaptation to Twitter and the improvements described
above) achieved 58.55% F1-measure, as shown in Table 11. The new German TwitIE tool
improved both Precision and Recall, as shown in Table 12. Although the F1-measure is still a
bit low at 62.91%, this is largely because of low Recall for Organisations and Persons, which
further attention to the gazetteers and rules will improve. The development work will
continue during the project, and we plan to create a larger evaluation dataset and re-
evaluate.
Location 34 65.48 78.57 71.43
Organisation 44 78.95 32.61 46.15
Person 5 40.00 40.00 40.00
Total 83 67.42 51.74 58.55
Table 11: Evaluation of original generic German NE system
Location 34 89.39 84.29 86.76
Organisation 44 53.45 33.70 41.33
Person 5 83.33 50.00 62.50
Total 83 73.08 55.23 62.91
Table 12: Evaluation of German TwitIE system

31 | P a g e
5 YODIE: Named Entity Disambiguation and Linking
Named entity disambiguation and linking (NED/NEL) goes one step beyond named entity
recognition: in addition to finding the mention of an entity of interest, the task is to also find
out which of several possible entities the mention refers to. This is usually done by identifying
entities from a knowledge base (KB) or ontology. The purpose of NED is thus to associate or
link a mention to the correct entity in the KB.
Figure 8 illustrates the purpose of NEL with an example where mentions of entities (people,
organisations, etc.) are linked to the addresses of Wikipedia pages describing the entity.
Mentions of named entities are often very ambiguous: the KB may contain dozens or
hundreds of entries for a name. The purpose of NED is to choose the correct link for the
particular mention with high probability. Since a knowledge base does not contain entries for
all possible entities, it is also important for the NED process to detect mentions of such
unknown entities, which are commonly called NIL entities: while the mention does refer to an
entity (e.g. a Person) the concrete person is not included in the KB (e.g. because it does not
satisfy the necessary condition of being notable enough to be included in Wikipedia).
NED adds valuable semantics to the result of information extraction and thus enables many
additional downstream uses like semantic search, relation extraction, recommendation,
filtering, semantic analysis and others. In the context of this project, YODIE is useful because
it enables to know which particular entity is being mentioned. This is critical when there are
multiple geographical locations with the same name – if for example a village is flooded, we
want to be sure we get the right one. Furthermore, linking to information in a knowledge
base enables us to aggregate queries over multiple entities, e.g. to search for all villages or
towns in a certain region, or all organisations of a certain type or in a certain location. This
information is also particularly helpful in the tasks concerned with establishing relevance and
veracity of information: if a tweet is talking about a location in France and we are concerned
with a disaster in Africa, then the tweet is unlikely to be relevant.
Figure 9: Entities in text linked to their respective entries in Wikipedia
YODIE is currently being developed as the multi-lingual named entity recognition and
disambiguation (NERD) tool of the University of Sheffield GATE group. It serves two purposes:
to act as a framework for carrying out research into the various approaches of multilingual
NERD, and as a reference implementation using GATE which will be made available to the
community as open source software (see Section 5.4). As part of this work package, we have

32 | P a g e
extended an earlier version of our NERD system (LODIE) to work better with social media text,
with the languages other than English relevant in this project, and to make it easier to update
to more recent versions of the underlying knowledge base (DBpedia). We also worked on
methods for decreasing the heavy demand on processing and memory resources of the
original version, although this is still an ongoing process.
For clarity, we present the whole YODIE system here. Specifically, in the context of this
project, the main contributions have been:
 implementation of the German and French processing pipelines for Tweets;
 enabling the YODIE annotation pipelines for English, German and French to run on
the GATE Cloud infrastructure;
 improving the preparation scripts and implementation of the pre-calculation of new
features: static page rank based on the graph from infoboxes, static page rank based
on the graph of intra-wiki links and reciprocal intra-wiki links;
 upgrading the preparation scripts to the more recent version of DBpedia 2015/10;
 improvements to the machine learning component used by YODIE, the GATE Learning
Framework: support for more learning tools (Weka via command line interface and
RESTful server, Keras, Scikit Learn), support of cost-sensitive learning and instance
weighting, creation of more features, support of missing values;
 implementation of a plugin gateplugin-CorpusStats for the calculation of term
statistics (tf, idf etc) on large corpora;
 implementation of a plugin and library for the use of word embeddings for the
calculation of similarity scores which will be used as features in updated versions of
YODIE;
 implementation of a PR (MapdbLookup in plugin JdbcLookup) for fast lookup of
embedding vectors and scores for tokens in a document;
 creation of additional features in the annotation pipeline (ambiguity and other
features calculated from the complete candidate list);
 preparation of additional corpora for experimentation;
 experiments to improve the annotation pipelines;
 updating the Simple Manual Annotation plugin to facilitate manual annotation of the
French and German manually annotated corpora.
YODIE is similar to other NED and NERD Systems like Dbpedia Spotlight (Daiber et al., 2013),
AIDA (Hoffart et al., 2011), and AGDISTIS (Usbeck et al., 2014). TagMe/WAT (Piccino and
Ferragina, 2014) was designed specifically for linking short texts like Tweets.
5.1 Multilingual YODIE
YODIE is implemented as a number of GATE pipelines which take arbitrary textual input and
produce annotations for mentions of known entities. The mention annotations contain
features which provide the assigned entity URL and additional information about the entity
and for debugging. The pipeline can be run within the GATE GUI (GATE Developer), or can be
used together with multi-threaded GATE GCP for batch processing of a corpus of documents
or can be deployed as a RESTful service as part of GATE Cloud.
YODIE currently supports the languages English, French, and German. For each language,
there are two possible processing modes: one for tweets and texts from social media, and
one for generic documents. Depending on the language, some steps in the processing

33 | P a g e
pipeline are different for social media texts and generic documents (e.g. the importance of
case for POS tagging or the recognition and processing of hashtags). In the remainder of this
section, we will concentrate on describing the social-media-specific processing mode.
The YODIE processing pipelines use a rich set of pre-computed resources. For example, to
recognize potential n-grams of mentions of known entities, a gazetteer list compiled from all
known entity labels is used, while to recognize the potential coherence between possible
entity candidates of several mentions in a text, a graph of relationships between entities is
used. In theory, all the information needed is present in the complete multilingual DBpedia
ontology which can be made available through a SPARQL endpoint. However, this
representation is not usable for the processing pipeline due to performance constraints.
Instead, we compile optimized representations for each purpose in advance, and then use
those representations, such as databases, compressed trie data structures, optimized graph
representations etc. in the processing pipeline during runtime. The YODIE implementation
therefore consists of two major parts, representing two phases of processing: the preparation
phase which takes the various knowledge sources as input, precomputes as many static
features, models, and statistics as possible, and produces task-optimized representations of
the knowledge source; and the annotation phase which uses these representations to carry
out the actual annotation of documents from a GATE document processing pipeline.
5.1.1 Preparation Phase
In the preparation phase, the following main tasks are carried out:
1. convert the information from knowledge base into data structures which can be
readily used by the annotation pipelines;
2. pre-calculate as many features as possible for the mentions and for the entity
candidates which can match a mention;
3. train machine learning models and tune parameters for some of the steps in the
processing pipeline.
For YODIE, DBpedia (Lehmann et al., 2012) is used as the main knowledge source. DBpedia is
automatically extracted and built from Wikipedia. We use the raw download files available
from the DBpedia download page.11
Most of these files are in Turtle format. In addition, we
use the original Wikipedia articles as a corpus -- for this we use the XML dump available from
Wikipedia and most recently also from the DBpedia download page. However, YODIE is
implemented in a way that tries to make it relatively flexible with respect to the knowledge
base used: a modified version is used for the annotation of bio-medical texts and uses UMLS
as its knowledge base (Bio-YODIE12
).
For NERD, and especially for the easy evaluation of NERD, it is important to use a unique
identifier for each entity. DBpedia uses URIs to refer to entities, relations and types (classes)
in the knowledge base. However, within the RDF and OWL representations as used for the
DBpedia knowledge base, many different URIs can possibly refer to the same entity: there
could be completely different URIs, for which owl:sameAs mappings exist, or URIs which
originate from different language versions of Wikipedia which refer to the same entity.
However, it is also possible to represent the same URI by different strings, depending on
which characters are percent-encoded or depending on whether non-ASCII characters are
11 http://wiki.DBpedia.org/Downloads2015-10
12 https://gate.ac.uk/applications/bio-yodie.html

34 | P a g e
represented as UTF-8 characters (in which case the representation is called an IRI). The first
step in the preparation phase is therefore to map all URIs/IRIs to a canonical representation:
 all URIs with percent-encoded UTF-8 characters or percent-encoded ASCII characters
which do have a non-percent-encoded representation are converted to IRIs
 if IRIs from a non-English language and from English exist, the English IRI is used as
the canonical form (this is also important because relationships or type information in
DBpedia are more frequently available for English than for other languages)
Each entity IRI in DBpedia is associated with a label, which can be seen as the name or
description of the entity. In the preparation phase, the labels are processed, filtered, and
normalized (e.g. alternate ways of representing accented characters or umlauts) and used to
generate a list of potential entity mention n-grams.
Wikipedia represents synonymy and sometimes common spelling variations by establishing
redirect links. In DBpedia, these redirected links are represented as separate URIs. For YODIE,
the label for each redirected URL is used as an alternate way to refer to the entity and is
associated with the canonical IRI the URL is redirecting to.
In addition, homonymy is often represented in Wikipedia by disambiguation pages. DBpedia
provides information about which original URL can have multiple interpretations or entities
associated with it. For YODIE, the label of the disambiguation page is used as a possible
additional alternate way of referring to each of the canonical IRIs that correspond to the
disambiguated URLs.
DBpedia also provides the information for Wikipedia “Infoboxes”: this information is used to
find additional possible ways of referring to entities. For example, for persons, the infobox
information may provide a nickname.
Finally, the anchor texts of links within Wikipedia articles can provide additional ways of
referring to the entity the link points to. In the preparation phase, we process the whole
corpus of Wikipedia articles to gather all anchor texts of such links, and use them as
additional, alternate names for the entities.
All labels gathered from DBpedia and Wikipedia are currently folded to lower-case, however
for each label the original form of the label is kept. Labels which originate from Wikipedia
articles often also contain explanatory text in parentheses (e.g. “Aberdeen (band)”) which can
give information about the type of entity, the type of article, e.g. “disambiguation”, context
information, e.g. the province in which a city is located, and many other kinds of information.
This part is removed from the label but for each label and entity combination, any parenthesis
information is kept in a database.
Ultimately the following resources are created from these steps:
 a database mapping every known URI or IRI for an entity to its canonical IRI;
 for each language, a gazetteer of known names of entities;
 information about which n-grams can be which kind of name for which IRI (the same
n-gram could be a preferred label for one entity, an alternate label for another entity
and a nickname for a third entity).
Once the gazetteer list of possible labels has been created, the whole Wikipedia corpus is
processed as a document corpus to gather statistics about how often a label occurs anywhere

35 | P a g e
in the text, and how often it occurs as an anchor text for each of potentially several possible
entities it could link to (for example “Tree” could link to the entity describing that form of
plant, or an entity describing a data structure, or an entity of a song or novel of that name).
This is used to pre-calculate the following features:
 mention frequency for the label;
 link frequency for the label;
 link probability for the label;
 commonness for the label, given each possible entity IRI.
DBpedia also provides class/type information for some of the entities, mainly based on
Infobox information. The original data provides all classes that apply for each entity, this
includes classes which can be inferred (parent classes) and multiple classes which cannot be
inferred. From this we calculate all most specific classes, as well as the classes closest to one
of the NE types we are interested in (e.g. DBpedia-owl:Person or DBpedia-
owl:PopulatedPlace). Since classes can also have many different representations, each class
URI is mapped to its canonical IRI. The class information of several languages is combined in
order to make use of the class information originating from one version of Wikipedia, if it is
missing from another version and there exists a language mapping between the entities. This
results in the original approximately 78.5M triples involving 363 classes getting converted to
8.3M mappings of entities to most specific or interesting classes. We also generate the known
category IRIs for each entity, although this information is not yet actually used in the
processing pipeline.
The Wikipedia Infoboxes not only contain literal information but can also point to other
entities, which are represented as relation triples in DBpedia. Together, these triples
represent a graph of how entities are linked to other entities. We combine these triples from
all processed languages since the use of Infobox templates varies between languages and the
relations from one language could still be used for those entities for which an inter-language
mapping is available. The graph contains about 32M directed edges. Since the edges are
labeled, entity pairs can be connected through many different edges. We convert the original
representation to a very compact representation which allows us to store the whole graph in
memory and to lookup the number of edges between two entities in a very efficient way in
the processing pipeline.
In addition to the graph generated from the Infobox fields, we also generate graphs from the
Wikipedia link structure: each link from one Wikipedia page to another Wikipedia page
represents a directed edge between the two corresponding entities. This graph is converted
to the same compact representation as for the relations graph. In addition, we create an
undirected version of the link-based graph restricted to pairs for pages where links exist in
both directions. The relations-based graph and the two link-based graphs are used to pre-
calculate, for each entity, a static page rank.
The various pre-calculated features for each label, IRI, or label-IRI pair are joined into one
large database table for each language. Each row in this table, uniquely identified by the
combination of the normalized label and the canonical IRI, represents an entity candidate for
use with a specific mention in the processing pipeline. Figure 10 shows part of the
information for one specific label/IRI pair as displayed in the GATE GUI.

36 | P a g e
Figure 10: Information from an IRI/label pair for “NASA”
We also create a bag-of-words based vector space model for abstracts associated with each
entity. This is used during runtime to estimate the textual similarity between text describing
the entity and the context of a mention in a document.
All in all, the data prepared for English is about 106GB, for German 45GB and for French
38GB. However, this data currently contains the information for all types of entities, not just
the ones which will later get annotated in the pipeline (note that for a significant part of the
entities, the type is unknown since Wikipedia does not have an Infobox for them and hence
DBpedia does not contain any type information). One reason for this is that we are planning
to use the information for entities of some type other than PER/LOC/ORG to better detect
spurious mentions.
In addition to the preparation of data mentioned above, we already generate data which is
not getting used in the current version of the pipeline, but is planned to get used in the next
version, most importantly language-specific embeddings for entities (based on entity
abstracts and entity link contexts).
5.1.2 Processing Pipeline
YODIE provides one annotation pipeline for each language and mode (social media
documents or generic documents). The pipelines accept a text-only document and generate
annotations for each mention that was disambiguated to an entity in the knowledge base. For

37 | P a g e
each disambiguated entity, the canonical IRI of the entity is added as a feature of the
annotation.
The pipelines for the processing modes and for the different languages are very similar: some
processing steps are shared between all tweet processing pipelines, other processing steps
are shared between different languages, and sometimes a processing resource needs
different parameter settings for the different versions. This has been accomplished by
creating the pipelines in a modular and parametrizable way using the ModularPipelines
plugin13
: all pipelines are split up into sub-pipelines of which several are shared between
languages and modes, and all pipelines and sub-pipelines can have their parameters
configured from a separate configuration file.
The following main steps are represented as individual sub-pipelines: pre-processing, label n-
gram matching, mention detection and filtering, entity candidate lookup, candidate pre-
filtering, candidate scoring, candidate disambiguation and post-processing.
Preprocessing: this sub-pipeline is a slightly modified version (in order to make it
parametrizable and avoid some steps not needed for YODIE) of the language-specific NER
pipeline described in Section 5.1.2. The annotations used by YODIE from this step are tokens,
sentences, the POS tags of tokens, named entity annotations of type Person, Location and
Organization, and co-reference chains.
Label n-gram matching: this step uses the language-specific gazetteer of label n-grams to find
all potential matches in the document text. Since the matching is done in a case-insensitive
fashion and anywhere in the document, a large number of potentially overlapping matches is
produced.
Mention detection and filtering: this sub-pipeline uses various heuristics to filter possible
mentions of relevant entities from DBpedia. The n-grams detected in the preprocessing step
often do not match the desired locations of mentions very well, depending on the processing
mode, the corpus and the types the YODIE pipeline is configured to find. Instead, we start
with the n-gram matches and use various features on each n-gram match for subsequent
filtering and ranking of mention candidates. For each n-gram match, a feature indicates if the
match overlaps or coextends any of the NE annotations from the preprocessing stage.
Additional features indicate the length of the match in tokens, if the match contains a token
with a POS tag that indicates a proper noun, or one that is a noun, as well as features pre-
calculated in the preparation phase, for example the frequency of the label in all Wikipedia
articles and the relative frequency of the label being used as a link anchor text in Wikipedia
(link probability). We use empirical weights determined on the training set to calculate a
score from all features, which is compared with a threshold tuned on the training set. The
annotations created in this step are the n-gram matches with a final score that exceeds the
threshold, indicating matches which are likely to be true mentions.
Candidate lookup: this sub-pipeline uses a query string created from the n-gram match to
look up all candidates in the language-specific candidate database which was created in the
preparation phase. The candidate database contains for each query string (representing a
label) the whole array of candidates, each candidate corresponds to an entity and IRI from
13https://github.com/johann-petrak/gateplugin-modularpipelines/wiki

38 | P a g e
the knowledge base and is represented as a map of key/value pairs representing the pre-
calculated features of that candidate. Some features contain arrays of values (for example,
entity types since an entity can belong to several DBpedia ontology classes, or the original
labels with their original spelling). In addition, a number of additional features are calculated
for the mention itself: these features reflect various aspects of the ambiguity of the mention
(e.g. the number of candidates found, the difference in commonness between the most
common and 2nd
common candidate and others).
Candidate pre-filtering: this is a highly configurable step that includes various heuristics to
reduce the number of candidates in each list of potential entity candidates for a mention. The
most important heuristics currently are:
 remove candidates where the DBpedia type indicates that we are not interested, e.g.
a biological species.
 Identify candidates for a label which could be an abbreviation, if followed by the full
name for the abbreviation.
 Process the lists for all overlapping mentions detected for a location name of the
form “Washington, D.C.” – this uses DBpedia relations (“Washington_DC lies in D.C.”)
to limit the number of candidates which could match the full (longest) mention.
 Reduce overlapping mentions: the general strategy is to use the longest span that
matches a label from Wikipedia, but this strategy sometimes produces wrong spans
or wrong entity lists: for example “The White House” is a valid label but may
incorrectly match “in [the White House]”. We reduce the span for mentions that
include determiners or possessive constructs.
 Finally, the candidate list is sorted by a score that reflects the likelihood of the
candidate being a valid match (currently commonness is used) and only the
candidates with the best N scores are kept. N has been tuned in previous experiments
on the training corpora (which included mostly generic documents) and set to 5, this
will get re-tuned for Twitter corpora.
Candidate scoring: Each candidate at this point already has a large number of pre-calculated
features: some from the preparation stage, some inherited from features of the mention
annotation (created in the mention detection and filtering stage) and some representing the
ambiguity of the mention (created in the candidate lookup stage). In this stage, additional
features are calculated to reflect two aspects of how well the candidate may fit the mention:
similarity to the text surrounding the mention, and similarity to other potential candidates in
the vicinity of the mention (coherence). We calculate several features for either aspect:
1. Cosine similarity of the context words with the DBpedia abstract of the candidate,
based on the bag-of-words vector space model created in the preparation phase;
2. Cosine similarity of the context words with the DBpedia literals from any infobox
properties of the entity;
3. Four scores calculated from the relationships that exist between the candidate and
candidates for other mentions in the context. These scores are based on the number
of different relationships that exist directly between two candidates or indirectly with
any single entity in between. Since the relations are directed, we use three ways of
how the current entity candidate c can be related to the other candidate o via some
additional entity a:
i. c→a→o or c←a←o (path)
ii. c←a→o (child)
iii. c→a←o (parent)

39 | P a g e
In addition to the raw scores, we also calculate the rank of the candidate for each
score: this is done since the absolute values may not be comparable between
candidate lists or their order may be more important for a decision than the absolute
value.
Candidate disambiguation: In this step, a pre-trained machine learning model is used to
assign to each candidate, based on the features calculated for the candidate, the class false
(do not pick that candidate) or true (pick this candidate as the correct one). Currently the
model used is a Random Forest of 100 trees which has been found to perform best based on
cross validation evaluations carried out on the training set. The algorithm used also generates
a probability estimate for the classification. This probability estimate is currently used in two
ways: if more than one candidate is assigned a label “true” then the candidate with the
highest probability estimate is used for linking the mention. Also, a threshold (tuned on the
training set) on the probability estimate is used to decide that a mention should really be a
“NIL”, i.e. should not get linked to any of the candidates.
Post-processing: this stage can be configured to remove some or all of the intermediate
annotations and features from the document. For experimenting and debugging, it is
important to keep as much information as possible, but the amount of information can be
huge: even a small document can potentially contain dozens of initial mention candidates,
and each mention can contain hundreds of matching entity candidates. The current default
for the deployed version is to keep only the final candidate annotation with two features:
“inst” containing the full IRI to which the mention is linked, and “confidence” containing the
probability estimate of the model for the assignment.
5.2 Deployment
The YODIE pipeline can be used from within the GATE GUI, but the demand for system
resources (memory, processing power) are currently still rather big. In addition, the pipeline
can be run on a corpus in batch mode using Gate GCP, or deployed as a RESTful service within
the GATE Cloud.
For the scope of this project, the annotation pipelines for the supported languages are
available as GATE Cloud services.14
This service can be accessed from a GATE pipeline using
the GATE Cloud client15
or any other client that follows the REST protocol.16
GATE Cloud also
provides a demo GUI for testing the pipeline, as shown in Figure 11.
14 Available from https://gate.ac.uk/projects/comrades/index.html
15https://github.com/GateNLP/cloud-client
16https://cloud.gate.ac.uk/info/help/rest-api.html

40 | P a g e
Figure 11: Demo of YODIE in GATE Cloud
5.3 Performance evaluation of YODIE
Since YODIE is meant to be a complete NERD pipeline which should not just disambiguate
known mentions of entities but also recognize those mentions, we evaluate it by calculating
the precision, recall and F1.0 measure of the complete task of finding only the correct
mentions and finding the correct IRI for each of the mentions. This means that spurious
mentions, missing mentions or mentions where the location has been detected correctly but
the wrong IRI has been assigned will decrease the F1.0 measure.
Evaluation of a complete NERD pipeline that links to DBpedia poses a number of challenges
(see also van Erp et al., 2016):
 Different corpora, both for training and evaluation and different use cases may have
different definitions of which types of entities should get included and how exactly
those types are defined. For example, for the entity type “Location” should landmarks
or notable buildings get included? Does “Location” include or extend to what is called
“Geopolitical Entity” in other corpora?
 Different evaluation corpora and different use cases may follow different annotation
guidelines. For example, if an entity name is part of a longer entity name, which parts
should get annotated, depending on the type of the containing and contained entity?
The entity “Treaty of Versailles” contains a place name which usually is not annotated
separately. On the other hand, the labels “Death of Yasser Arafat” and “Cause of
Yasser Arafat's death” exist in Wikipedia but should not prevent the “Person” entity
from being annotated. Place names of the form “Vienna, Georgia” are sometimes
fully annotated as one “Location” and sometimes as two (with the first linking to
“Vienna,_Georgia” and the second to “Georgia_(U.S._state)”.
 Different ways of indirectly referring to entities of a specific type may or may not be
acceptable, for example country adjectivals (German) to refer to a country.
 Different ways of (co-)referring to an entity may be acceptable, for example “The
British Monarch” for referring specifically to Elizabeth II.

41 | P a g e
 The corpus reflects the version of the knowledge base when the corpus was created,
but the system to evaluate uses a much more recent version of the knowledge base
where links may have changed, been deleted or added. Even if an entity in the
knowledge base is the same as in the old version, the representation of the URI may
have changed (e.g. the percent-encoding).
 For the evaluation of non-English pipelines, the language specific URI, or the
corresponding English URI if an inter-language mapping exists, could be chosen as the
correct target link.
We attempt to deal with at least some of the technical problems by updating all evaluation
(and training) corpora whenever we use a new version of the knowledge base: all IRIs are
converted to the canonical representation used for that version. If we compare with another
NERD system, then both the evaluation corpus and the predictions from the NERD system are
converted using the same strategy.
5.3.1 English Tweet Pipeline
The following compares the English YODIE disambiguation pipeline for tweets on an unseen
test set of 191 tweets from the corpus we have created previously (Gorrell et al., 2015). The
training set from the same corpus was used together with the Aida training, TAC 2009, TAC
2011 and TAC2012 corpora for training the candidate selection model. The corpus contains
191 documents with a total of 5100 tokens, of which 3333 are word tokens, and with 225
linkable (non-NIL) target annotations. The results are shown in Table 13.
The following systems were used for comparison:
 YODIE
 Aida 201417
: this evaluation was carried out using a local installation of the Aida 2014
software [Hoffart et al., 2014), and a plugin for annotating documents using the AIDA
service endpoint. For AIDA 2014, all settings were used with their default values.
 Spotlight18
: this evaluation was carried out in 2015 using the DBpedia Spotlight
(Daiber et al., 2013) disambiguation service endpoint (the service was accessible at
the time this evaluation was carried out in 2015, but is not accessible at that address
any more at the time of this writing), the following parameter values, documented to
be the default settings at that time, were used: confidence=0.2, support=20)
 TagMe19
: this evaluation was carried out using the TagMe (Ferragina and Scaiella,
2010) service endpoint for short texts as of 2015, with all parameters left to their
default values.
 TextRazor20: evaluation carried out using the REST service endpoint of TextRazor as of
2015, with all settings left to their default values.
 Zemanta21
: evaluation carried out using the REST service endpoint of Zemanta (Tori
and Tomaž, 2008) as of 2015, with all settings left to their default values.
17 https://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-
naga/aida/
18 https://github.com/dbpedia-spotlight/dbpedia-spotlight
19 http://acube.di.unipi.it/tagme/
20 https://www.textrazor.com/
21 no longer available

42 | P a g e
System Precision Recall F1.0
YODIE 0.50 0.61 0.55
Aida 2014 0.59 0.38 0.46
Spotlight 0.09 0.51 0.15
TagMe 0.10 0.67 0.17
TextRazor 0.19 0.44 0.26
Zemanta 0.48 0.56 0.52
Table 13: Comparison of different NED systems on an English Twitter Corpus
Note that we expect some further improvements in the performance of YODIE in the coming
months. This is due to two reasons. First, we have detected some target annotations in the
gold standard corpus which are not quite correct and need to be fixed. Second, the model is
trained on the previous release of DBpedia, while the application itself uses the newer
version, so the features are slightly different. We plan to retrain the model on the latest
release of DBpedia.
5.3.2 French YODIE Pipeline
The French YODIE pipeline uses all the language-independent modules and settings of the
English pipeline. The gazetteers, graphs, candidate databases and other language-specific
resources have been created from the French DBpedia files in the preparation phase. Since
there were no French training corpora available, we also use the machine learning model for
candidate selection which was trained on the English training corpora. The preprocessing
stage uses the resources created for the French TwitIE pipeline as described in Section 3.
For evaluation, a corpus with 385 French tweets was used, as described in Section 3. This
corpus was created from an original selection of 1000 French tweets, from which all re-tweets
have been removed. The corpus was first automatically annotated to find named entities of
type Person, Organization and Location; then the documents were manually checked and
corrected. The corpus with the corrected named entity annotations was then pre-annotated
with a sorted list of linking candidates using a modified YODIE pipeline. The pre-annotated
corpus was then again manually checked using the SimpleManualAnnotator tool.22
The result
of the manual correction was then manually checked by another person and automatically
updated to conform to the format needed for evaluation and to canonicalize any links which
were manually modified. The final corpus contains 385 documents, of which 170 do not
contain any linked named entities, and a total of 393 linked named entities.
Table [XXX] shows the result of evaluating the French YODIE pipeline on this corpus without
any language-specific tuning or training. This is a surprisingly good result: the machine
learning model for candidate selection generalizes rather well to this specific corpus. This is
not likely to generalize equally well to other kinds of tweets, however, and future work will
tune some parts of the YODIE pipeline in a language-specific way based on using the French
Wikipedia articles as a silver standard training corpus (see future work).
22 https://github.com/GateNLP/gate-SimpleManualAnnotator

43 | P a g e
P-strict R-strict F1-strict P-lenient R-lenient F1-lenient
French
YODIE
0.75 0.87 0.80 0.76 0.87 0.81
Table 14: Evaluation of French YODIE
5.3.3 German YODIE Pipeline
As for the French YODIE pipeline, the German YODIE pipeline uses all the language-
independent modules and settings of the English pipeline, and the gazetteers, graphs,
candidate databases and other language-specific resources have been created from the
German DBpedia files in the preparation phase. Again, due to lack of a training corpus, the
machine learning model for candidate selection which was trained on the English training
corpora is used. The pre-processing stage uses the resources created for the German TwitIE
pipeline, as described in Section 4.
For evaluation, a corpus of 1000 German tweets (see Section 4) was created and processed
and manually corrected in the same way as the French corpus. During manual annotation,
some Tweets had to be removed, and the final corpus contains 977 tweets.
For this corpus, an initial evaluation on the complete corpus like it was done for French
showed very low performance (lenient F1.0 < 0.4). This was due to several factors: it turned
out that corpus was much more challenging both for automatic linking and for manual
annotation. The initial evaluation showed that there were a number of incorrect manual
annotations, but also a number of cases where our initial annotation guidelines produced
results which are counted as incorrect but not entirely wrong. For example, the annotation
guidelines demanded that if the name of an entity is part of a longer name of another entity
which is not of a type that should be included in the corpus, then this entity should not be
annotated. For example in "the treaty of Versailles", Versailles should not be annotated
because "Treaty of Versailles" is the name of an entity of type "international treaty". This
corpus contains a large number of cases where this rule was applied, for example:
 "AEG PerfectMix sB 2400 Mini Mixer": here YODIE links "AEG" to the company (ORG);
 "Feuerwehr Dresden" where YODIE links Dresden to the city;
 "UN-Gipfel" where YODIE links UN to the organization.
A number of other cases were revealed where the annotation guidelines may have led to
undesirable manual annotations, for example mentions of online media like "derStandard.at"
which were not intended to be annotated, but are (correctly) linked by YODIE. Another factor
was that, unlike for French, the model for candidate selection did not generalize as well to
German and most of candidate lists did not get any candidate selected. Finally, there are
many cases where YODIE annotates entities correctly, but the entities are not of one of the
types Person, Location or Organization. This is partly because YODIE has been configured to
also include certain brand names, car models and other entity types (similar to the "MISC"
class of the Stanford 4-class named entity models). Another reason is that for some matches
of n-grams we find, the best candidate does not have any usable type information (the
DBpedia type is owl:Thing).

44 | P a g e
Because of all this, the German corpus, which is significantly larger than the French one, was
split into two parts: a training/tuning set of 450 tweets/documents, and a testing/evaluation
set with 527 tweets. The training set was used to tune the part of YODIE which finds initial
mention locations and to tune the strategy for choosing the best linking candidate from the
entity candidate list. Both were done manually based on an analysis of the behaviour of the
pipeline on the training set. The locating module was slightly changed to put more emphasis
on the named entities detected in the preprocessing stage when matching a 1-gram label
from the DBpedia gazetteer. The disambiguation module was changed to combine confidence
of the machine learning model and the commonness feature of a candidate for ranking all
candidates to find the best one and a threshold tuned on the training set for that combined
score for creating NILs.
Table 15 shows the result of evaluating the tuned pipeline on the test-set. As for the French
pipeline, future work will try to significantly improve on this by using the German Wikipedia
articles as a language-specific silver standard training corpus.
P-strict R-strict F1-strict P-lenient R-lenient F1-lenient
French
YODIE
0.56 0.52 0.54 0.68 0.63 0.66
Table 15: Evaluation of German YODIE
5.4 Free Open Source Components
The following free and open source components have been created or updated as part of the
work on this work package:
 GATE plugin Lang_French
 GATE plugin Lang_German
 gate-SimpleManualAnnotator23
: Software to manually inspect and correct entity
linking annotations for the creation of gold-standard corpora for the named entity
link tasks
 gateplugin-LearningFramework24
: GATE plugin to make it easy to use various machine
learning libraries with GATE annotation pipelines.
 Gateplugin-StringAnnotation25
: A GATE plugin that provides a very compact and
flexible implementation of a gazetteer based n-gram matcher.
 Gateplugin-CorpusStats26
: A GATE plugin that allows the efficient calculation of term
frequency statistics over very large corpora (e.g. Wikipedia)
 gateplugin-JdbcLookup27: A GATE plugin for looking up information in JDBC databases
or large serialized Map stores and create annotations or features from that
23https://github.com/GateNLP/gate-SimpleManualAnnotator
24https://github.com/GateNLP/gateplugin-LearningFramework
25https://github.com/johann-petrak/gateplugin-StringAnnotation
26https://github.com/johann-petrak/gateplugin-CorpusStats
27https://github.com/johann-petrak/gateplugin-JdbcLookup

45 | P a g e
information. This is used e.g. to create the annotations that represent linking
candidates in YODIE.
In addition, we are planning to release both the YODIE preparation software and the YODIE
annotation pipeline as open source software. However, this is currently not possible due to
licensing issues: both components use software from the Mallet project (currently licensed
under CPL) and from the Stanford CoreNLP project (licensed under GPL3) which have licenses
that cannot be combined for distribution. However, it is planned that Mallet will change its
license to one that is compatible with GPL, at which point we will be able to make YODIE
available under a compatible license. A variation of YODIE for processing bio-medical texts,
Bio-YODIE has already been released as open source software.28
6 Summary
In this deliverable, we have described the tools developed in GATE for analysing social media
text in English, French and German. This includes tools for linguistic pre-processing, named
entity recognition, and disambiguation/linking of named entities. The tools have been
adapted specifically to work on social media such as tweets, which is a challenging genre due
to the noisy nature of the text (incorrect spelling, punctuation, capitalization etc). The English
system is more fully-fledged, since English will be the primary language of the datasets used
in the project. However, the adaptation to English and German demonstrates the flexibility of
our approach and tools, and shows how they can be modified for new languages. In the
course of this adaptation work, we have faced new research challenges, such as integrating
new 3rd
party components and making them compatible with our existing tools, and adapting
other components to the complexities of different language structures. Evaluation results are
promising, but the tools will continue to be developed through the course of the project
nevertheless.
7 References
[1] K. Bontcheva, L. Derczynski, A. Funk, M.A. Greenwood, D. Maynard, N. Aswani. TwitIE: An
Open-Source Information Extraction Pipeline for Microblog Text. Proceedings of the
International Conference on Recent Advances in Natural Language Processing (RANLP
2013).
[2] S. Carter, W. Weerkamp, and E. Tsagkias. 2013. Microblog language identification:
Overcoming the limitations of short, unedited and idiomatic text. Language Resources
and Evaluation Journal.
[3] B. Crabbé and M. Candito. 2008. Expériences d’analyse syntaxique statistique du français.
In Proc. of TALN’08, pages 45–54, Senlis, France.
[4] W. Cavnar and J. Trenkle. 1994. N-gram-based text categorization. In Proceedings of the
Annual Symposium on Document Analysis and Information Retrieval, pages 161–175.
[5] J. Daiber, M. Jakob, C. Hokamp and P.N. Mendes. Improving Efficiency and Accuracy in
Multilingual Entity Extraction Proceedings of the 9th International Conference on
Semantic Systems, ACM, 2013, 121-124
[6] L. Derczynski, D. Maynard, N. Aswani, and K. Bontcheva. 2013a. Microblog-Genre Noise
and Impact on Semantic Annotation Accuracy. In Proceedings of the 24th ACM Conference
on Hypertext and Social Media. ACM.
28https://gate.ac.uk/applications/bio-yodie.html

46 | P a g e
[7] L. Derczynski, A. Ritter, S. Clark, K. Bontcheva. 2013b. Twitter Part-of-Speech Tagging for
All: Overcoming Sparse and Noisy Data. In Proceedings of the International Conference on
Recent Advances in Natural Language Processing (RANLP 2013).
[8] L. Derczynski, D. Maynard, G. Rizzo, M. van Erp, G. Gorrell, R. Troncy, J. Petrak, K.
Bontcheva. Analysis of Named Entity Recognition and Linking for Tweets. In Information
Processing and Management, 51(2) 32-49, 2015.
[9] L. Derczynski and S. Chester, 2016. Generalised Brown Clustering and Roll-Up Feature
Generation. In Proceedings of the Association for Advancement of Artificial Intelligence,
2016.
[10] M. Drouin and C. Davis, 2009. R u txting? Is the use of text speak hurting your literacy?.
Journal of Literacy Research, 41(1), pp.46-67.
[11] P. Ferragina and U. Scaiella. TAGME: On-the-fly Annotation of Short Text Fragments (by
Wikipedia Entities) Proceedings of the 19th ACM International Conference on Information
and Knowledge Management, ACM, 2010, 1625-1628
[12] J. R. Finkel, T. Grenager and C. Manning, "Incorporating non-local Information into
Information Extraction Systems by Gibbs Sampling," in Proceedings of the 43rd Annual
Meeting of the Association for Computational Linguistics (ACL), 2005.
[13] G. Gorrell, J. Petrak and K. Boncheva. Using @twitter Conventions to #enrich Content for
Named Entity Disambiguation, 2015
[14] S. Green, M.C. de Marneffe and C.D. Manning, 2013. Parsing models for identifying
multiword expressions. Computational Linguistics, 39(1), pp.195-227.
[15] K. Greenfield, R. Caceres, M. Coury, K. Geyer, Y. Gwon, J. Matterer, A. Mensch, C. Sahin,
and O. Simek. "A reverse approach to named entity extraction and linking”. In
#Microposts (2016): 67-69.
[16] M. Halliday, C.M. Matthiessen, and C. Matthiessen, 2014. An introduction to functional
grammar. Routledge.
[17] B. Han, P. Cook, and T. Baldwin. 2012. Automatically constructing a normalisation
dictionary for microblogs. In Proceedings of the conference on Em- pirical Methods in
Natural Language Processing, pages 421–432. ACL.
[18] J. Hoffart, M.A. Yosef, I. Bordino, H, Fürstenau, M. Pinkal, M. Spaniol, B. Taneva, S. Thater
and G. Weikum. Robust Disambiguation of Named Entities in Text Proceedings of the
Conference on Empirical Methods in Natural Language Processing, Association for
Computational Linguistics, 2011, 782-792
[19] J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P.N. Mendes, S. Hellmann,
M. Morsey, P. van Kleef, S. Auer and C. Bizer. DBpedia - A Large-scale, Multilingual
Knowledge Base Extracted from Wikipedia Semantic Web Journal, 2012
[20] M. Lui and T. Baldwin, 2011. Cross-domain feature selection for language identification.
In In Proceedings of 5th International Joint Conference on Natural Language Processing.
[21] M. Lui and T. Baldwin, 2012, July. langid.py: An off-the-shelf language identification tool.
In Proceedings of the ACL 2012 system demonstrations (pp. 25-30). Association for
Computational Linguistics.
[22] J. Nivre, J., M.C. de Marneffe, F. Ginter, Y. Goldberg, J. Hajic, C. D. Manning, R. Tsarfaty,
2016. Universal Dependencies v1: A Multilingual Treebank Collection. In Proceedings of
the Conference on Language Resources and Evaluation, 2016.
[23] O. Owoputi, B. O'Connor, C. Dyer, K. Gimpel, N. Schneider, N.A. Smith, 2013. Improved
part-of-speech tagging for online conversational text with word clusters. In Proceedings
of the conference of the North American Association for Computational Linguistics, 2013.
[24] S. Petrov, D. Das, R. McDonald, 2011. A universal part-of-speech tagset. ArXiv preprint

47 | P a g e
arXiv:1104.2086, 2011.
[25] F. Piccinno and P. Ferragina. From TagME to WAT: A New Entity Annotator Proceedings
of the First International Workshop on Entity Recognition &38; Disambiguation, ACM,
2014, 55-62
[26] B. Plank, A. Søgaard and Y, Goldberg, 2016. Multilingual Part-of-Speech Tagging with
Bidirectional Long Short-Term Memory Models and Auxiliary Loss. arXiv preprint
arXiv:1604.05529.
[27] A. Ritter, S. Clark, S. Mausam and O. Etzioni, "Named Entity Recognition in Tweets: An
Experimental Study," in EMNLP, 2011.
[28] R. Sproat, A. Black, S. Chen, S. Kumar, M. Ostendorf, and C. Richards. 2001. Normalization
of non-standard words. Computer Speech & Language, 15(3):287–333.
[29] A. Tori and S. Tomaž "Zemanta service."
https://web.archive.org/web/20140215095047/http://developer.zemanta.com/docs/Ze
manta_API_companion/
[30] R. Usbeck, A.-C. Ngonga Ngomo, S. Auer, D. Gerber and A. Both. AGDISTIS - Graph-Based
Disambiguation of Named Entities using Linked Data International Semantic Web
Conference, 2014
[31] M. van Erp, P.N. Mendes, H. Paulheim, F. Ilievski, J. Plu, G. Rizzo and J. Waitelonis, 2016.
Evaluating Entity Linking: An Analysis of Current Benchmark Datasets and a Roadmap for
Doing a Better Job Proceedings of Language Resources and Evaluation Conference (LREC
2016).
[32] A. Zubiaga and H. Ji. "Tweet, but verify: epistemic study of information verification on
twitter." Social Network Analysis and Mining 4.1 (2014): 1-12.

D3.1 Multilingual content processing methods

More Related Content

Similar to D3.1 Multilingual content processing methods (20)

More from COMRADES project (19)

Recently uploaded (20)

D3.1 Multilingual content processing methods