Key2Vec: Automatic Ranked Keyphrase Extraction from Scientific

Articles using Phrase Embeddings

Debanjan Mahata John Kuriakose Rajiv Ratn Shah Roger Zimmermann

Bloomberg Infosys Limited IIIT-Delhi NUS-Singapore,
New York, U.S.A Pune, India New Delhi, India Singapore
[email protected] John [email protected] [email protected] [email protected]

Abstract (Hasan and Ng, 2014), whereas the unsupervised

methods are mostly based on TF-IDF, cluster-
Keyphrase extraction is a fundamental task
ing, and graph-based ranking (Hasan and Ng,
in natural language processing that facilitates
mapping of documents to a set of representa- 2010; Mihalcea and Tarau, 2004). On the pres-
tive phrases. In this paper, we present an un- ence of domain-specific data, supervised methods
supervised technique (Key2Vec) that leverages have shown better performance. The unsupervised
phrase embeddings for ranking keyphrases methods have the advantage of not requiring any
extracted from scientific articles. Specifi- training data and can produce results in any do-
cally, we propose an effective way of pro- main.
cessing text documents for training multi-word
With recent advancements in deep learning
phrase embeddings that are used for thematic
representation of scientific articles and rank- techniques applied to natural language processing
ing of keyphrases extracted from them using (NLP), the trend is to represent words as dense
theme-weighted PageRank. Evaluations are real-valued vectors, popularly known as word em-
performed on benchmark datasets producing beddings. These representations of words have
state-of-the-art results. been shown to equal or outperform other methods
(e.g. LSA, SVD) (Baroni et al., 2014). The em-
1 Introduction and Background
bedding vectors, are supposed to preserve the se-
Keyphrases are single or multi-word linguistic mantic and syntactic similarities between words.
units that represent the salient aspects of a doc- They have been shown to be useful for several
ument. The task of ranked keyphrase extraction NLP tasks, like part-of-speech tagging, chunk-
from scientific articles is of great interest to sci- ing, named entity recognition, semantic role la-
entific publishers as it helps to recommend arti- beling, syntactic parsing, and speech processing,
cles to readers, highlight missing citations to au- among others (Collobert et al., 2011). Some of the
thors, identify potential reviewers for submissions, most popular approaches for training word embed-
and analyze research trends over time (Augenstein dings are Word2Vec (Mikolov et al., 2013), Glove
et al., 2017). Due to its widespread use, keyphrase (Pennington et al., 2014) and Fasttext (Bojanowski
extraction has received significant attention from et al., 2016).
researchers (Kim et al., 2010; Augenstein et al.,
Title: Identification of states of complex systems with estimation of
2017). However, the task is far from solved and admissible measurement errors on the basis of fuzzy information.
Abstract: The problem of identification of states of complex systems on the
the performances of the present systems are worse basis of fuzzy values of informative attributes is considered. Some estimates
in comparison to many other NLP tasks (Liu et al., of a maximally admissible degree of measurement error are obtained that
make it possible, using the apparatus of fuzzy set theory, to correctly identify
2010). Some of the major challenges are the var- the current state of a system.
Automatically identified keywords: complex systems, fuzzy information,
ied length of the documents to be processed, their admissible measurement errors, fuzzy values, informative attributes
measurement error, maximally admissible degree, fuzzy set theory
structural inconsistency and developing strategies Manually assigned keywords: complex system state identification,
that can perform well in different domains (Hasan admissible measurement errors, informative attributes, measurement errors
fuzzy set theory
and Ng, 2014).
Methods for automatic keyphrase extraction are Table 1: Keyphrases extracted by using Key2Vec from
mainly divided into two categories: supervised a sample research article abstract.
and unsupervised. Supervised methods approach
the problem as a binary classification problem Word embeddings have already shown promis-
ing results in the process of keyphrase extraction
from scientific articles (Wang et al., 2015, 2014).
However, Wang et al. did not use domain-specific
word embeddings and had suggested that train-
ing them might lead to improvements. This mo-
tivated us to experiment with domain-specific em-
beddings on scientific articles.
In this work, we represent candidate keyphrases
extracted from a scientific article by domain- Figure 1: Text processing pipeline for preparing text
specific phrase embeddings and rank them using samples used for training word embedding models.
a theme-weighted PageRank algorithm (Langville
and Meyer, 2004), such that the thematic weight named entity extraction models. For this work we
of a candidate keyphrase indicate how similar use Spacy1 as our NLP toolkit along with its de-
it is to the thematic representation or the main fault models. The choice of Spacy is just for con-
theme of the article, which is also constructed us- venience and is not driven by any other factor. We
ing the same embeddings. Due to extensive use split a text document into sentences, tokenize a
of phrase embeddings for representing the can- sentence into unigram tokens, as well as identify
didate keyphrases and ranking them, we name noun phrases and named entities from it. During
our method as Key2Vec. To our knowledge, us- this process if a named entity is detected at a par-
ing multi-word phrase embeddings for construct- ticular offset in the sentence then a noun phrase
ing thematic representation of a given document appearing at the same offset is not considered.
and to assign thematic weights to phrases have not We take steps in cleaning the individual sin-
been used for ranked keyphrase extraction, and gle word and multi-word tokens that we obtain.
this work is the first preliminary attempt to do so. Specifically, we filter out the following tokens.
Table 1. shows ranked keyphrases extracted using
Key2Vec from a sample research abstract. Next, • Noun phrases and named entities that are
we present our methodology. fully numeric.

2 Methodology • Named entities that belong to the following

categories are filtered out : DATE, TIME,
Our methodology primarily uses three steps: can- PERCENT, MONEY, QUANTITY, ORDI-
didate selection, candidate scoring, and candidate NAL, CARDINAL. Refer, Spacy’s named
ranking, similar to other popular frameworks of entity documentation2 for details of the tags.
ranked keyphrase extraction (Kim et al., 2013).
All the steps depend on the choice of our text pro- • Standard stopwords are removed.
cessing steps and a phrase embedding model that
• Punctuations are removed except ‘-’.
we train on a large corpus of scientific articles. We
explain them next and give a detailed description We also take steps to clean leading and ending
of their implementations. tokens of a multi-word noun phrase and named en-
tity.
2.1 Text Processing
It has been shown (Mikolov et al., 2013), that the • Common adjectives and reporting verbs are
presence of multi-word phrases intermixed with removed if they occur as the first or last token
unigram words increases the performance and ac- of a noun phrase/named entity.
cruacy of the embedding models trained using • Determiners are removed from the first token
techniques such as Word2Vec. However, in our of a noun phrase/named entity.
framework we take a different approach in de-
tecting meaningful and cohesive chunk of phrases • First or last tokens of noun phrases/named
while preparing the text samples for training. In- entities belonging to following parts of
stead of relying on measures considering how of- 1
https://spacy.io
ten two or more words co-occur with each other, 2
https://spacy.io/usage/linguistic-features#section-
we rely on already trained dependency parsing and named-entities
 speech: INTJ Interjection, AUX Aux-
iliary, CCONJ Coordinating Conjunction,
ADP Adposition, DET Interjection, NUM
Numeral, PART Particle, PRON Pronoun,
SCONJ Subordinating Conjunction, PUNCT
Punctutation, SYM Symbol, X Other, are
removed. For a detailed reference of each
of these POS tags please refer Spacy’s doc-
umentation3 .
Figure 2: Frequency distribution of topics in the arxiv
• Starting and ending tokens of a noun dataset used for training phrase embeddings.
phrase/named entity is removed if they be-
long to a standard list of english stopwords
and functional words. refrain from using them in this preliminary work.
We would like to use them in the future.
Apart from relying on Spacy’s parser we use The main aim of the underlying embedding
hand crafted regexes for cleaning the final list model is to capture semantic and syntactic simi-
of tokens obtained after the above data cleaning larities between textual units comprising of both
steps. single word and multi-word phrases. We chose
Fasttext over other embedding techniques because
• Get rid of leading/trailing junk characters. it captures both the semantic and morphological
similarities5 between words. For example, if we
• Handle dangling/backwards parentheses. We have breast cancer in the content of the theme of a
don’t allow ’(’ or ’)’ to appear without the document then intuitively phrases like breast can-
other. cer, breast cancer treatment, should be assigned
higher thematic weight than prostrate cancer or
• Handle oddly separated hyphenated words.
lung cancer, even though the document might
• Handle oddly separated apostrophe’d words. mention other forms of cancer as well. Embed-
ding techniques like Word2Vec and Glove, only
• Normalize whitespace. takes into account the semantic similarity between
words based on their occurrences in a similar con-
The resultant unigram tokens and multi-word text and will not display the desired property that
phrases are merged in the order they appeared in we want to leverage. We would like to study the
the original sentence. Figure 1, shows an example effects of other types of embeddings in the future.
of how the text processing pipeline works on an Dataset: Since this work deals with the domain
example sentence for preparing the training sam- of scientific articles we train the embedding model
ples that act as an input to the embedding algo- on a collection of more than million scientific doc-
rithm. uments. We collect 1,147,000 scientific abstracts
related to different areas (Fig 2) from arxiv.org6 .
2.2 Training Phrase Embedding Model
For collecting data we use the API provided by
The methodology to a great extent relies on the arxiv.org that allows bulk access7 to the articles
underlying embeddings. We directly train multi- uploaded in their portal. We also add the scien-
word phrase embeddings using Fasttext4 , rather tific documents present in the benchmark datasets
than first training embedding models for unigram (Sections 3), increasing the total number of docu-
words and then combining their dense vectors to ments to 1,149,244.
obtain vectors for multi-word phrases. Our train- After processing the text of the documents as
ing vocabulary consists of both unigram as well mentioned above, we train a Fasttext-skipgram
as multi-word phrases. We are aware of the ex- model using negative sampling with a context win-
isting procedures for training phrase embeddings
5
(Yin and Schütze, 2014; Yu and Dredze, 2015), but https://rare-technologies.com/fasttext-and-gensim-
word-embeddings/
3 6
https://spacy.io/api/annotation#pos-tagging http://arxiv.org
4 7
https://fasttext.cc/ https://arxiv.org/help/bulk data
dow size of 5, dimension of 100 and number of a given document (di ), the candidate scores are
epochs set to 10. We would like to experiment fur- scaled again between 0 and 1 with a score of 1 as-
ther in the future on the selection of optimal pa- signed to the candidate semantically closest to the
rameters for the embedding model. main theme of the document and 0 to the farthest.
Candidate Selection: This step aids in chos- Candidate Ranking: In order to perform fi-
ing candidate keyphrases from the set of all nal ranking of the candidate keyphrases we use
phrases that can be extracted from a document, weighted personalized PageRank algorithm. A di-
and is commonly used in most of the automated rected graph Gdi is constructed for a given doc-
ranked keyphrase extraction systems. Not all the ument (di ) with Cdi as the vertices and Edi as
phrases are considered as candidates. Generally, the edges connecting two candidate keyphrases if
unwanted and noisy phrases are eliminated in this they co-occur within a window size of 5. The
process by using different heuristics (Section 2.1). edges are bidirectional. Weights sr(cdj i , cdki ) are
We split a given document into sentences and to calculated for the edges using the semantic simi-
extract noun phrases and named entities as de- larity between the candidate keyphrases obtained
scribed previously. As an output of this step we get from the phrase embedding model and their fre-
a set of unique phrases (Cdi = {c1 , c2 , ..., cn }di ) quency of co-occurrence, as used by Wang et
for a document di to be used later for scoring and al. (Wang et al., 2015), and shown in equation
1
ranking in the next two steps. 3. We use cosine distance ( di di ) and
1−cosine(cj ,ck )
Candidate Scoring: In this step we assign Point-wise Mutual Information (P M I(cdj i , ckdi ))
a theme vector (τˆdi ) to a document (di ). The for calculating semantic(cdj i , cdki ) (equation 1)
theme vector can be tuned according to the type
and cooccur(cdj i , cdki ) (equation 2), respectively.
of documents that are being processed and the
The main intuition behind calculating semantic re-
type of keyphrases that we want to get in our fi-
latedness by using a phrase embedding model is
nal results. In this work, we extract a theme ex-
to capture how well two phrases are related to
cerpt from a given document and further extract
each other in general. Whereas, the co-occurrence
a unique set of thematic phrases comprising of
score captures the local relationship between the
named entities, noun phrases and unigram words
phrases within the context of the given document.
(Tdi = {t1 , t2 , ..., tm }di ) from it. For the Inspec
dataset we use the first sentence of the document
which consists its title, and for the SemEval dataset semantic(cdj i , cdki ) =
1
(1)
we use the title and the first ten sentences extracted 1 − cosine(cdj i , cdki )
from the beginning of the document, as the theme
excerpts, respectively (see Section 3). The first ten cooccur(cdj i , cdki ) = P M I(cdj i , cdki ) (2)
sentences of a document from the SemEval dataset
essentially captures the abstract and sometimes
first few sentences of the introduction of a scien-
sr(cdj i , cdki ) = semantic(cdj i , cdki ) × cooccur(cdj i , cdki )
tific article. We get the vector representation (tˆj ) (3)
of each thematic phrase extracted from the theme Given graph G, if ε(cdj i ) be the set of all edges
excerpt using the phrase embedding model that we
incident on the vertex cdj i , and wcdji is the thematic
trained and perform vector addition in order to get
the final theme vector (τˆdi = m
P
ˆ weight of cdj i as calculated in the candidate scor-
j=1 tj ) of the doc-
ument. The phrase embedding model is then used ing step, then the final PageRank score R(cdj i ) of a
to get the vector representation (cˆk ; k ∈ {1...n}) candidate keyphrase cdj i is calculated using equa-
for each candidate keyphrase in Cdi . tion 4, where d = 0.85 is the damping factor and
We calculate the cosine distance between the out(cdki ) is the out-degree of the vertex cdki .
theme vector (τˆdi ) and vector for each candidate
keyphrase (cˆk ) and assign a score (κ(x̂, ŷ) → X sr(cdj i , cdki )
[0, 1]) to each candidate, with 1 indicating a com- R(cdj i ) = (1 − d)wcdji + d × ( )R(cdki )
out(cdki )
plete similarity with the theme vector and 0 in- d d
c i ∈ε(cj i )
k
dicating a complete dissimilarity. To get the fi- (4)
nal thematic weight (wcdji ) for each candidate w.r.t. Next, we evaluate the performance of Key2Vec.
 Micro Micro Micro Micro Micro Micro Micro Micro Micro
Avg. Avg. Avg. Avg. Avg. Avg. Avg. Avg. Avg.
Precision Recall F1 Precision Recall F1 Precision Recall F1
@5 @5 @5 @10 @10 @10 @15 @15 @15
Inspec 61.78 % 25.67 % 36.27 % 57.58 % 42.09 % 48.63 % 55.90 % 50.06 % 52.82 %
SemEval 41 % 14.37 % 21.28 % 35.29 % 24.67 % 29.04 % 34.39 % 32.48 % 33.41 %

Table 2: Performance of Key2Vec over combined controlled and uncontrolled annotated keyphrases for Inspec and
SemEval 2010 datasets.

SGRank TopicRank
Inspec Wang et al., Liu et al.,
Key2Vec (Danesh et al., (Bougouin et al.,
(Combined) 2015 2010
2015) 2013)
Micro Avg. F1@10 48.63 % 44.7 % 45.7 % 33.95 % 27.9 %

Table 3: Comparison of Key2Vec with some state-of-the-art systems (Liu et al., 2009; Danesh et al., 2015; Bougouin
et al., 2013; Wang et al., 2015) for Avg. F1@10 on Inspec dataset.

SGRank HUMB TopicRank

SemEval 2010
Key2Vec (Danesh et al., (Lopez and Romary, (Bougouin et al.,
(Combined)
2015) 2010) 2013)
Micro Avg. F1@10 29.04 % 26.07 % 22.50 % 12.1 %

Table 4: Comparison of Key2Vec with some state-of-the-art systems (Danesh et al., 2015; Bougouin et al., 2013;
Lopez and Romary, 2010) for Avg. F1@10 on SemEval 2010 dataset.

3 Experiments and Results respectively. In the evaluation, we check the per-

formance over the top 5, 10 and 15 candidates re-
The final ranked keyphrases obtained using the
turned by Key2Vec. The performance of Key2Vec
Key2Vec methodology as described in the previous
on the metrics is shown in Table 2. Tables 3 and 4
section is evaluated on the popular Inspec and Se-
shows a comparison of Key2Vec with some of the
mEval 2010 datasets. The Inspec dataset (Hulth,
state-of-the-art systems giving best performances
2003) is composed of 2000 abstracts of scientific
on the Inspec and SemEval 2010 datasets, respec-
articles divided into sets of 1000, 500, and 500,
tively.
as training, validation and test datasets respec-
tively. Each document has two lists of keyphrases
assigned by humans - controlled, which are as-
4 Conclusion and Future Work
signed by the authors, and uncontrolled, which
are freely assigned by the readers. The controlled
keyphrases are mostly abstractive, whereas the un- In this paper, we proposed a framework for auto-
controlled ones are mostly extractive (Wang et al., matic extraction and ranking of keyphrases from
2015). The Semeval 2010 dataset (Kim et al., scientific articles. We showed an efficient way of
2010) consists of 284 full length ACM articles di- training phrase embeddings, and showed its effec-
vided into a test set of size 100, training set of size tiveness in constructing thematic representation of
144 and trial set of size 40. Each article has two scientific articles and assigning thematic weights
sets of human assigned keyphrases: the author- to candidate keyphrases. We also introduced
assigned and reader-assigned ones, equivalent to theme-weighted PageRank to rank the candidate
the controlled and uncontrolled categories, respec- keyphrases. Experimental evaluations confirm
tively of the Inspec dataset. We only use the test that our proposed technique of Key2Vec produces
datasets for our evaluations and combine the an- state-of-the-art results on benchmark datasets. In
notated controlled and uncontrolled keyphrases. the future, we plan to use other existing proce-
The ranked keyphrases are evaluated using ex- dures for training phrase embeddings and study
act match evaluation metric as used in SemEval their effects. We also plan to use Key2Vec in
2010 Task 5. We match the keyphrases in the an- other domains such as news articles and extend the
notated documents in the benchmark datasets with methodology for other related tasks like summa-
those generated by Key2Vec, and calculate micro- rization.
averaged precision, recall and F-score (β = 1),
References Amy N Langville and Carl D Meyer. 2004. Deeper in-
side pagerank. Internet Mathematics 1(3):335–380.
Isabelle Augenstein, Mrinal Das, Sebastian Riedel,
Lakshmi Vikraman, and Andrew McCallum. Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and
2017. Semeval 2017 task 10: Scienceie-extracting Maosong Sun. 2010. Automatic keyphrase extrac-
keyphrases and relations from scientific publica- tion via topic decomposition. In Proceedings of
tions. arXiv preprint arXiv:1704.02853 . the 2010 conference on empirical methods in nat-
ural language processing. Association for Compu-
Marco Baroni, Georgiana Dinu, and Germán tational Linguistics, pages 366–376.
Kruszewski. 2014. Don’t count, predict! a
systematic comparison of context-counting vs. Zhiyuan Liu, Peng Li, Yabin Zheng, and Maosong
context-predicting semantic vectors. In ACL (1). Sun. 2009. Clustering to find exemplar terms for
pages 238–247. keyphrase extraction. In Proceedings of the 2009
Conference on Empirical Methods in Natural Lan-
Piotr Bojanowski, Edouard Grave, Armand Joulin, guage Processing: Volume 1-Volume 1. Association
and Tomas Mikolov. 2016. Enriching word vec- for Computational Linguistics, pages 257–266.
tors with subword information. arXiv preprint
arXiv:1607.04606 . Patrice Lopez and Laurent Romary. 2010. Humb: Au-
tomatic key term extraction from scientific articles
Adrien Bougouin, Florian Boudin, and Béatrice Daille. in grobid. In Proceedings of the 5th international
2013. Topicrank: Graph-based topic ranking for workshop on semantic evaluation. Association for
keyphrase extraction. In International Joint Con- Computational Linguistics, pages 248–251.
ference on Natural Language Processing (IJCNLP).
pages 543–551. Rada Mihalcea and Paul Tarau. 2004. Textrank: Bring-
ing order into texts. Association for Computational
Ronan Collobert, Jason Weston, Léon Bottou, Michael Linguistics.
Karlen, Koray Kavukcuoglu, and Pavel Kuksa.
2011. Natural language processing (almost) from Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Cor-
scratch. Journal of Machine Learning Research rado, and Jeff Dean. 2013. Distributed representa-
12(Aug):2493–2537. tions of words and phrases and their compositional-
ity. In Advances in neural information processing
Soheil Danesh, Tamara Sumner, and James H Martin. systems. pages 3111–3119.
2015. Sgrank: Combining statistical and graphical
methods to improve the state of the art in unsuper- Jeffrey Pennington, Richard Socher, and Christopher D
vised keyphrase extraction. Lexical and Computa- Manning. 2014. Glove: Global vectors for word
tional Semantics (* SEM 2015) page 117. representation. In EMNLP. volume 14, pages 1532–
1543.
Kazi Saidul Hasan and Vincent Ng. 2010. Conundrums
Rui Wang, Wei Liu, and Chris McDonald. 2014.
in unsupervised keyphrase extraction: making sense
Corpus-independent generic keyphrase extraction
of the state-of-the-art. In Proceedings of the 23rd In-
using word embedding vectors. In Software Engi-
ternational Conference on Computational Linguis-
neering Research Conference. volume 39.
tics: Posters. Association for Computational Lin-
guistics, pages 365–373. Rui Wang, Wei Liu, and Chris McDonald. 2015. Using
word embeddings to enhance keyword identification
Kazi Saidul Hasan and Vincent Ng. 2014. Automatic for scientific publications. In Australasian Database
keyphrase extraction: A survey of the state of the art. Conference. Springer, pages 257–268.
In ACL (1). pages 1262–1273.
Wenpeng Yin and Hinrich Schütze. 2014. An explo-
Anette Hulth. 2003. Improved automatic keyword ex- ration of embeddings for generalized phrases. In
traction given more linguistic knowledge. In Pro- ACL (Student Research Workshop). pages 41–47.
ceedings of the 2003 conference on Empirical meth-
ods in natural language processing. Association for Mo Yu and Mark Dredze. 2015. Learning composition
Computational Linguistics, pages 216–223. models for phrase embeddings. Transactions of the
Association for Computational Linguistics 3:227–
Su Nam Kim, Olena Medelyan, Min-Yen Kan, and 242.
Timothy Baldwin. 2010. Semeval-2010 task 5: Au-
tomatic keyphrase extraction from scientific articles.
In Proceedings of the 5th International Workshop
on Semantic Evaluation. Association for Computa-
tional Linguistics, pages 21–26.

Su Nam Kim, Olena Medelyan, Min-Yen Kan, and

Timothy Baldwin. 2013. Automatic keyphrase ex-
traction from scientific articles. Language resources
and evaluation 47(3):723–742.