Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6804–6811

Marseille, 11–16 May 2020

c European Language Resources Association (ELRA), licensed under CC-BY-NC

Aspect-Based Sentiment Analysis as Fine-Grained Opinion Mining

Gerardo Ocampo Diaz1 , Xuanming Zhang2 , Vincent Ng1

1
Human Language Technology Research Institute, University of Texas at Dallas
2
University of Nottingham, Ningbo China
[email protected], [email protected], [email protected]
Abstract
We show how the general fine-grained opinion mining concepts of opinion target and opinion expression are related to aspect-based
sentiment analysis (ABSA) and discuss their benefits for resource creation over popular ABSA annotation schemes. Specifically, we
first discuss why opinions modeled solely in terms of (entity, aspect) pairs inadequately captures the meaning of the sentiment originally
expressed by authors and how opinion expressions and opinion targets can be used to avoid the loss of information. We then design a
meaning-preserving annotation scheme and apply it to two popular ABSA datasets, the 2016 SemEval ABSA Restaurant and Laptop
datasets. Finally, we discuss the importance of opinion expressions and opinion targets for next-generation ABSA systems. We make
our datasets publicly available for download.

Keywords: opinion mining, sentiment analysis, text mining

1. Introduction According to the annotation scheme, one opinion is listed

For almost two decades, researchers in the text mining for each sentence:
and natural language processing (NLP) communities have (1)<OTE="sea urchin"
worked to improve the state of aspect-based sentiment anal- category="FOOD#QUALITY"
ysis (ABSA); a task that, roughly speaking, involves ex- polarity="positive"
tracting sentiment/opinions from text in terms of targets from="42" to="52"/>
they address. For example, given the sentence: (2)<OTE="NULL"
The soup here is very expensive. category="FOOD#QUALITY"
polarity="positive"
The goal of ABSA is to extract the subject matter of the from="0" to="0"/>
opinion, which is typically represented by an (entity,aspect)
pair along with the sentiment expressed towards it (fre- Here, each opinion’s subject matter is labeled under the cat-
quently in terms of polarity). For example, the previous egory field; each category is an (entity, aspect) pair, while
sentence expresses a positive polarity towards (food, price). the opinion target expression (OTE) field labels the explicit
The entity and the aspect are both chosen from predefined reference in the sentence to the entity in the category field.
sets. So, if an opinion is expressed on an entity that does In sentence (1), the author describes the sea urchin as heav-
not appear in the predefined set of entities, then an ABSA enly. The value of the category field in the annotation is as
system is not expected to output any opinion for that entity. food/quality, while sea urchin is marked as a reference to
The important role this task can play in decision making the entity food. This roughly seems to encode the original
for governments, companies, and individuals has resulted meaning of the sentiment expressed by the author.
in a large body of research on the subject being devel- In sentence (2), however, the author expresses an opin-
oped over the years, along with the organization of mul- ion directly towards the consistency of the sea urchin, but
tiple shared tasks in SemEval (Pontiki et al., 2014; Pontiki the category FOOD#QUALITY does not reflect this. More
et al., 2015; Pontiki et al., 2016), TASS (Martı́nez Cámara specifically, this annotation is by no means wrong: sea
et al., 2018), and GermanEval (Wojatzki et al., 2017). The urchin is a dish in the restaurant (which leads to FOOD),
datasets published in the SemEval ABSA tasks (Pontiki et and consistency is an attribute closely related to QUAL-
al., 2016), which are composed of user-generated reviews ITY). However, by relying on predefined entities and as-
of products and services such as those found on Amazon pects, consistency is being abstracted to QUALITy, result-
and Yelp, have heavily influenced work in the area, as re- ing in loss of information. Moreover, the OTE field is null
searchers frequently rely on them for system training and because by definition it cannot be filled with a pronoun.
testing. However, these datasets follow a task-specific an- Overall, the annotation for this sentence does not precisely
notation scheme, under which opinion annotations do not encode the author’s original intended meaning.
always capture the full meaning of the sentiment expressed In this paper, we make the argument for a different annota-
in text. Consider the following pair of sentences taken from tion scheme for ABSA; one that aims to more closely pre-
the SemEval 2016 Restaurants dataset: serve the semantic information of the opinions originally
expressed in text.
(1) I tend to judge a sushi restaurant by its sea Our contributions in this paper are three-fold. First, we
urchin, which was heavenly at sushi rose. (2) make a connection between key ideas in the general opin-
It melted in my little mouth and the perfect ion mining community (e.g., opinion target1 and opinion
consistency-not too fishy, creamy, and slightly
1
buttery. As we will explain later, the notion of opinion targets in the

6804
expression) and ABSA, which uses the notion of aspects, other hand, does not express an opinion on self-driving fea-
and explore how opinion targets and opinion expressions tures in general, but specifically the self-driving features of
are used by authors in ABSA datasets to express sentiment. the new Tesla Model 3. The fact that the target is not fully
Second, we present our annotation scheme and reannotate specified in the sentence is considered particularly impor-
two popular ABSA datasets from the SemEval 2016 ABSA tant in the domain of online user-generated reviews, where
shared task (laptop and restaurants), which we make pub- most of the initial work on ABSA was originally carried out
licly available. Finally, we discuss how our datasets rep- (Hu and Liu, 2004a; Hu and Liu, 2004b; Liu et al., 2005).
resent a first step towards building next-generation ABSA In this domain, the objects of interest (e.g., a laptop, restau-
systems. rant, etc) can be modeled as a hierarchy of “parts” where
every part can have attributes or dimensions and sub-parts,
2. Opinion Mining and ABSA: A Critical and opinion targets can be mapped to a specific leaf node
Overview in the hierarchy. For example (Liu, 2012):
Before discussing an annotation scheme for ABSA, it is A particular model of camera is an entity, e.g.,
sensible to try to understand the core ideas behind it. Here, Canon G12. It has a set of attributes, e.g., picture
we aim to cover key concepts such as opinion models, opin- quality, size, and weight, and a set of parts, e.g.,
ion targets, and opinion expressions, as well as provide lens, viewfinder, and battery. Battery also has its
readers with an intuitive understanding of the breadth of own set of attributes, e.g., battery life and battery
ways in which authors express sentiment in text and the weight. A topic can be an entity too, e.g., tax in-
information that might be useful for understanding it. We crease, with its parts “tax increase for the poor,”
also introduce our own concepts, examples, and explana- “tax increase for the middle class,” and “tax in-
tions to clear up ambiguity and enable a deeper discussion crease for the rich.”
on key ideas, as the well-established literature is sometimes
lacking in this sense (Liu, 2012; Liu, 2015; Liu, 2017). From this observation, work on ABSA frequently uses a
simplified notion of flat targets. Here, target g from M 1 is
2.1. Opinion Models and ABSA now represented by a pair (e,a), where e represents the en-
In this section, we examine two well-known opinion mod- tity or topic of interest, and a represents an aspect: any at-
els. The first one, which we refer to as M 1, models an tribute/dimension or part/component of e. This representa-
opinion conceptually as a quadruple (g, s, h, t) (Liu, 2015), tion of targets corresponds to another opinion model, which
where g represents the target of the opinion (i.e., the topic, we refer to as M 2. Specifically, M 2 is a quintuplet model
event, object, etc... towards which sentiment is expressed), of opinions (e,a,s,h,t). Consider the sentence The ink of
s represents whatever sentiment is expressed, h represents the printer is expensive in the context of a printer review.
the holder of the opinion/sentiment, and t represents the Since the main entity being described is the printer and ink
time when the opinion/sentiment is expressed/held. Al- is an aspect, we could represent this opinion’s target (e,a)
though the sentiment s can be represented in terms of dif- as (printer,ink); alternatively, if we choose to model the ink
ferent attitudes (Wilson, 2008), researchers in ABSA fre- as a separate entity, the target could be (ink, price), but then
quently extract only sentiment polarity. We say that an one needs to separately store the relation between ink and
opinion can be modeled as a quadruple conceptually, be- printer.
cause coming up with a string that fully encodes the target It is important to point out that the targets under M 2 are
g of an opinion is not always straightforward. For exam- in a sense open to interpretation. Consider again the sen-
ple, in the sentence I love ice cream, it is easy to identify tence The ink of the printer is expensive. Here, the target
the target ice cream, but this is not the case for These books the author has in mind is clearly the price of the ink of the
are inappropriate for children; this is not a general opinion printer, but under M 2, the valid targets for this opinion
on these books, but on books in relation to children reading include (printer,ink), (printer, cost of operation), or (ink,
them (hence, inappropriate. Further, even when a string price). Intuitively, only (ink, price) matches the meaning
can fully describe the target g, it may not be fully contained intended by the author, but all of the targets correspond to
in the sentence that expresses sentiment (Liu, 2012). Con- valid interpretations from a reader’s perspective, each with
sider the following interaction between two speakers: a different focus. In other words, modeling opinion tar-
gets as (entity, aspect) pairs involves an arbitrary level of
S1: (1) I really like the new AI-centric granularity, frequently defined by the scope of the task at
cars companies are coming up with.
hand, and does not necessarily preserve authors’ originally
S2: (2) Me too, I love the new Tesla
intended meaning.
Model 3.
(3) The self-driving features are
2.2. Opinion Types and General Opinion Mining
great!
Opinion models describe desired system output, not how
(1) expresses a general opinion on the new AI-centric cars opinions are expressed. Unfortunately, work on ABSA
companies are coming up with. Here the target of the opin- rarely touches on this subject, and even when it does it is
ion is completely contained in the sentence. (3), on the frequently lacking in depth. In this subsection, we aim to
share our perspectives on how opinions are expressed, bas-
general fine-grained opinion mining community is not the same as ing our discussion on already-established ABSA and opin-
opinion target expression in ABSA datasets. ion mining concepts but using our own examples to illus-

6805
trate how opinions are expressed within the scope of the – Adjectives that are used to describe internal
SemEval ABSA Laptop and Restaurant 2014-2016 datasets states, such as happy, sad, angry, joyous, can
(Pontiki et al., 2014)2 . also be used like this, although the syntactic re-
One of the more common ways of explaining how opinions lations between them and targets can be more
are expressed is to distinguish between explicit and implicit complicated than examples using transitive verbs
opinions (Liu, 2012). Intuitively, an explicit opinion is ex- (above). Examples include I am happy that I
pressed through a subjective statement: one that a) explic- came here and I am angry because you did not
itly states an opinion holder’s attitude or feelings towards a buy me ice cream.
target, such as I love ice cream, or b) uses language that in-
herently describes an opinion holder’s attitude or feelings, • Using language that implies sentiment on a target
such as The ice cream is delicious. On the other hand, im- – Simple adjectives like good, bad, great, tasty are
plicit opinions are expressed through objective or factual frequently found in opinionated text, such as This
statements, such as I bought this phone a month ago and it restaurant is awful, but they can also appear in
died today or Repairing an iPhone costs almost as much as slightly more complex statements like You are
buying a new one. an amazing person, where syntactically speak-
Conceptually, extracting implicit and explicit opinions in- ing, amazing describes person, but because of the
volves very different tasks: we can say that explicit opin- copula relation between You and person, the en-
ions are extracted, as they are already indicated in the text, tire phrase amazing person can be considered an
while implicit opinions are inferred due to the fact that they opinion expression on the target You.
require accounting in some way for context and/or domain-
– Nouns like genius, hero, champion can be used in
knowledge.
simple copula relations to describe targets, such
Although this distinction is fairly important, researchers
as in I finally realized, you are a genius.
frequently fail to explain opinions further. There are many
different ways in which one can use subjective or objective – Adverbs, like adjectives can be used in sim-
statements to express opinions. Below we attempt to allevi- ple phrases like The steak tasted delicious or in
ate this problem by providing a quick overview of the types more complex ones like The fried shrimp was
of opinions that we have found during our survey, as well as too spicy, where too spicy describes The fried
a discussion of the importance of different lexical elements shrimp: even though spicy might not necessarily
from which opinions can be inferred. imply an opinion, the adverb too does.

2.2.1. Explicit Opinions and Opinion Expressions 2.2.2. Implicit Opinions

Explicit opinions are characterized by containing an ex- Implicit opinions are inferred from objective statements.
plicit opinion expression: a word or phrase that describes One of the explanations provided in existing ABSA lit-
an opinion holder’s sentiment or attitude towards a particu- erature is that implicit opinions are derived from desir-
lar target. Adjectival phrases are the most common type of able facts (Liu, 2012): common-sense or domain-specific
explicit opinion expression, but adverbial and noun phrases knowledge. For example, the sentence We initially ordered
can be used too, along with verbs. Examples include ad- the kung pao chicken and wanted to cancel it so we’d have
jectives such as beautiful, great, nouns such as hero, idiot, room for dessert, but the waiter ignored us when we called
and adverbs such as beautifully, effectively. Although there him. implies a negative opinion on The waiter. It is com-
are words or phrases that can be considered opinion expres- monsense knowledge that restaurant staff should be atten-
sions related to the same attitude or polarity in multiple do- tive to customer’s requests and therefore ignoring a cus-
mains (e.g., fantastic, good), there might be others that are tomer is considered inappropriate. However, explaining
domain or even target-specific. For example, describing a implicit opinions simply as desirable facts is a significant
pillow as hard might imply negative sentiment, while de- step back from explicit opinion expressions discussed in
scribing a hammer as hard might not imply any sentiment the previous subsection. From the point of view of general
at all. In practice, explicit opinion expressions frequently fine-grained opinion mining, however, one can still iden-
serve one of the following functions: tify implicit opinion expressions: specific words, phrases,
or clauses in an objective statement from which the opinion
• Explicitly describing an opinion holder’s atti- holder’s sentiment can be inferred3 . In practice, implicit
tude/sentiment opinion expressions describe the target itself (through noun
phrases) or the target’s behavior (what the target does/ is
– Transitive verbs like love, hate, agree, enjoy can
done to the target):
be used to explicitly describe sentiment towards a
target such as in I love the dress you bought me, or • Describing the target
I agree that we should have stricter immigration
laws. – Nouns or noun phrases may be used in copula
relations with targets to describe them. No sin-
2
The concepts and explanations provided here (both from the gle part of the phrase needs to imply sentiment,
literature and our own) are not meant to be exhaustive, but roughly
3
cover the breadth of opinions contained in the SemEval datasets. In general opinion mining literature, the term opinion expres-
For further reading on traditional explanations of the task, we refer sion is used to refer to both implicit and explicit opinion expres-
the reader to Liu (2015). sions (Wiebe et al., 2005).

6806
 but the phrase should imply some sentiment as and (printer, ink). Simply put, the combination of lexi-
a whole. For example, in the sentence The new cal target and opinion expression sufficiently encodes the
president is truly someone who has no idea of original meaning of the sentiment intended by the author,
what he’s doing, the noun phrase someone who and can therefore be used to infer all valid (entity, attribute)
has no idea of what he’s doing describes the new pairs in M 2. Specifically, from the lexical target ink of the
president. printer and opinion expression too expensive, it is easy to
see that among the three valid targets, (ink, price) matches
• Describing target behaviour most closely the meaning of the original text. In addition,
it can be used to infer the other two valid targets, which
– Verb phrases can imply sentiment on their sub-
are less fine-grained. In other words, the combination of
jects or objects. Consider the sentence The waiter
lexical target and opinion expression encodes the sentiment
took our order while holding a garbage bag in
expressed in the text without any loss of information, so it
a restaurant review. Here, took our order while
can be useful regardless of the level of target granularity
holding a garbage bag implies negative senti-
one is interested in.
ment on the waiter, who performs the action.
Similarly, in I could fit the carry on bag in the
overhead bin easily, the verb phrase could fit in
3. Dataset
the overhead bin easily implies a positive senti- In this section we provide an overview of related fine-
ment on the carry on bag. grained sentiment analysis datasets, detail our meaning-
preserving annotation scheme and procedure, and detail
Nevertheless, less common, implicit opinions can also inter-annotator agreement and dataset statistics.
sometimes be inferred from events, such as I took the pill
and then I felt much better. Here, the entire sentence can be 3.1. Related Resources
viewed as the opinion expression. Conceptually, extracting Over the years, researchers have produced datasets for
the opinion from this example involves inferring first that ABSA of different sizes and domains. However, not all
The pill caused me to feel better and then that cause me to datasets have been made publicly available and few have
feel better implies positive sentiment (like in explicit opin- received as much attention as the SemEval ABSA Restau-
ions). rant and Laptop datasets (Pontiki et al., 2014; Pontiki et al.,
2015; Pontiki et al., 2016), which consist of 440 and 530
2.3. Lexical Targets, Opinion Expressions, and user-generated reviews mined from online websites respec-
Semantic Targets tively. The datasets aim to capture sentiment in terms of
To facilitate our discussion, in this subsection we introduce pre-defined opinion targets, where a target is defined as a
the concept of lexical and semantic targets. Lexical targets specific (entity, attribute) pair, such as FOOD#QUALITY
are phrases found in the text which the author uses to refer or LAPTOP#PRICE. These categories are assigned if an
to the opinion target, while semantic targets are the “real”, opinion directly references an ENTITY#ATTRIBUTE pair,
fully-specified targets in the opinion holder’s mind4 . For or if the opinion itself can be generalized to describe one.
example, in I went to Feng Cha. The tea was so cheap!, For each sentence, every opinion in the sentence is anno-
the lexical target is The tea, while the semantic target is the tated, with each set of annotations consisting of 1) the opin-
price of Feng Cha’s tea. Note that semantic targets are the ion target as an entity,attribute pair, 2) the opinion polarity
same as g in M 1. (positive or negative), and 3) the opinion target expression
In practice, what really matters is the semantic target, so (OTE), which is defined as the explicit reference to the en-
why are lexical targets important? To answer this ques- tity in the opinion target (if any). For example, in the con-
tion, we make the following observation: the combination text of a restaurant review, there would be one opinion an-
of lexical target + opinion expression can be used to infer notation for The shrimp was delicious!, with the opinion
an opinion’s semantic target. In the previous example, the target (FOOD,QUALITY), OTE shrimp, and polarity pos-
price of Feng Cha’s tea can be inferred from the The tea itive. Opinion holders and the time at which opinions are
(lexical target) and expensive (opinion expression). This is held (from M 1) are not annotated, as it is assumed that the
also true even when the semantic target can not be described opinions expressed in a review are in line with the author’s
in a straightforward manner, such as in The way the waiter own.
looked at me was totally inappropriate, where the lexical It is important to point out that OTEs are not equivalent to
target is The way the waiter looked at me and the opinion our notion of lexical targets. To see why, consider the ex-
expression is totally inappropriate. ample The way the waiter looked at me was totally inappro-
To better understand the significance of our observation, priate.. Under the SemEval annotation scheme, the corre-
consider its implication for opinion model M 2. Recall sponding target is (SERVICE,GENERAL), since the author
that The ink of the printer is too expensive. has three is expressing an opinion towards the customer service (the
valid targets that correspond to different levels of granu- aspect is GENERAL because SERVICE does not have any
larity under M 2: (ink, price), (printer, cost of operation), specific attributes under the SemEval annotation scheme),
and the corresponding OTE is waiter, with opinion polarity
4
In the general fine-grained opinion mining literature (Yang negative. However, in reality, this statement directly ex-
and Cardie, 2013), lexical targets are simply referred to as opinion presses an opinion on the waiter’s behavior, specifically,
targets, while the notion of semantic targets is absent. its appropriateness. Because of this, we can interpret this

6807
as an opinion on the waiter, and as a consequence, as an the best in town!, it is annotated as the lexical target,
opinion on customer service. This information is lost un- but crab cake is also labeled as the resolution of it.
der the SemEval annotation scheme due to the fact that the
lexical target and opinion expression are not annotated. As far as we know, none of the existing ABSA datasets has
Another relevant well-known resource is the Multi- pronominal targets resolved.
Perspective Question Answering (MPQA) dataset (Wiebe In addition, for each lexical target we annotate its semantic
et al., 2005; Wilson, 2008; Deng and Wiebe, 2015), which head:
is composed of 535 political news articles spanning 10 dif-
• The views around the mountain are beautiful → The
ferent topics and annotated with opinion expressions, opin-
opinion target is The views around the mountain and
ion holders, and lexical opinion targets. The MPQA dataset
its head is views.
introduces a general framework for fine-grained sentiment
analysis, and contains annotations for opinions in terms of • I love swimming in the ocean → The opinion target is
lexical targets, opinion expressions, and opinion holders. swimming in the ocean and its head is swimming
Given that it is composed of political news articles, there
is no notion of targets as entity, aspect pairs, and the lan- Our annotation of semantic heads is motivated by the
guage, topics, and opinion holders can be quite different scheme used for annotating the MUC-6 and MUC-7
from those found in user-generated reviews. datasets (Grishman and Sundheim, 1995; Chinchor, 1998),
which were designed for information extraction tasks such
3.2. Annotation Scheme and Procedure as named entity recognition and entity coreference resolu-
Recall that the SemEval datasets a) annotate targets as (en- tion; Complex phrases are typically hard for a system to
tity, aspect) pairs, leading to annotations that lose infor- extract, so in order not to penalize a system for its failure to
mation from authors’ originally intended sentiment, and extract a phrase because of its complexity, one can consider
b) ignore opinion expressions that are useful for inferring that it extracts the phrase correctly as long as it extracts the
the semantic target and sentiment polarity of the opinion. simpler (and thus arguably easier to extract) phrase denoted
Therefore, we reannotate both the Laptop and Restaurant by the semantic head. As far as we know, none of the ex-
datasets from the SemEval 2016 ABSA shared task. For isting fine-grained sentiment analysis corpora has semantic
each opinion in the dataset, we annotate the lexical opinion heads annotated.
target, opinion expression and opinion polarity. By annotat-
ing lexical opinion targets and opinion expressions, 1) au- 3.2.2. Opinion Expressions
thors’ originally intended sentiment is preserved, and 2) we We annotate both explicit and implicit opinion expressions,
provide useful information to infer semantic targets, valid which can be broadly divided into five categories:
(entity, aspect) pairs at different levels of granularity, and
the sentiment of the opinions. • Type 1: Noun or adjectival phrases inherently associ-
In the following subsections, we describe the different types ated with sentiment
of annotations found in the dataset. The different types of
opinion targets and expressions are exhaustive; no opinions – The fried chicken was far too oily → far too oily
found in the dataset are considered out of scope. – This place is the best → the best
3.2.1. Opinion Targets • Type 2: Verbs or verb phrases that imply sentiment on
We annotate three different types of opinion targets: their subject
• Lexical Targets: Targets appearing as phrases in the
original text – These pills got me to stop throwing up! → got
me to stop throwing up
– The screen of the laptop is great → The screen of – The waiter took our order while holding a
the laptop garbage bag → took our order while holding the
• Implicit Targets: Opinion targets that do not appear garbage bag
in the original text but can be inferred from context/
opinion expressions. • Type 3: Verb or verb phrases that imply sentiment on
their object
– Not a great place for family dining → restau-
rant. The opinion is given towards the entity it- – My dog ripped it [this leash] apart in less than a
self - the restaurant. Implicit targets are annotated month → ripped it apart in less than a month
based on the corresponding SemEval entity cata- – I had to ask the waiter because she was not pay-
log. Here, the entity is annotated in place of the ing attention → had to ask the waiter three times
lexical target, and the semantic target is encoded because she was not paying attention
by the combination of the implicit target and the
opinion expression. • Type 4: Explicit expressions of sentiment
• Resolved Targets: Pronominal targets that can be re- – I am so happy I came here. → happy
solved to noun phrases based on neighboring sen-
tences. For example, in I ate the crab cake. It was • Type 5: Miscellaneous

6808
 – Since I started eating here, my life changed. → Finally, pronominal target resolution, implicit targets, se-
The whole phrase is marked as an opinion expres- mantic heads of subjective opinion expressions and lexical
sion. targets, and opinion expression types are annotated by the
– I bought this laptop and now I can play all the meta-annotator.
games I want! → The whole phrase is marked as To measure agreement we use both Cohen’s kappa (Cohen,
an opinion expression. 1960) and recall-based agreement. Recall-based agreement
is calculated as follows
For each opinion expression, we also annotate its polarity
(positive/negative): # of spans annotated by A and B
agr(A, B) =
# of spans annotated by A
• The laptop is nice but kind of expensive. → Two opin-
ion expressions are labeled: nice with polarity positive
agr(A,B) agr(b,a) avg.
and kind of expensive with polarity negative
0.92 0.90 0.91
Further, each opinion expression is labeled as either subjec-
tive or objective: Table 1: Recall-based agreement for opinion target spans

• I had an amazing time here! → had an amazing time

is labeled as subjective agr(A,B) agr(B,A) avg.
0.99 0.96 0.975
• We had to wait 45 minutes for our food → had to wait
45 minutes for my food is labeled as objective Table 2: Recall-based agreement for opinion expressions
Finally, just like opinion targets, subjective opinion expres- Cohen’s kappa (Cohen, 1960) is calculated over every word
sions have their semantic heads labeled: in the corpus. Each word is tagged as no label, opinion
• The fish was far too oily! → the opinion expression is expression word or opinion target word.
far too oily and semantic head is oilyh
Span Cohen’s Kappa
• Applebee’s is my favorite restaurant in the whole Opinion Target 0.90
world! → the opinion expression is my favorite restau- Opinion Expression 0.97
rant in the whole world and the semantic head is fa-
vorite Table 3: Cohen’s K for opinion targets and opinion expres-
sions annotation
• My dog broke this leash in less than a week → the
opinion expression is my dog broke this leash in less
than a week and the semantic head is broke (the lexical 3.4. Dataset Statistics
target is this leash). We reannotate the restaurant and laptop datasets from Se-
mEval 2016 (Pontiki et al., 2016), consisting of 440 restau-
Note that objective opinion expressions do not have their rant reviews and 530 laptop reviews. The restaurant dataset
semantic heads annotated since the entirety of the expres- contains 2,687 sentences in the dataset, with 2,420 opinion
sion is needed to infer sentiment polarity and the semantic annotations. The laptop dataset contains 3,308 sentences
target. with 1,735 opinion annotations5 .
3.3. Annotation Procedure and Inter-annotator
Restaurants Laptops
Agreement
Opinions 2,420 1,735
The dataset is labeled by two human annotators and a meta- Subjective Opinion Expressions 2,089 1,456
annotator. First, annotators are given a small set of 20 re- Objective Opinion Expressions 339 279
views to familiarize themselves with the annotation scheme
and procedure. After discussion with the meta-annotator, Table 4: Dataset statistics
both annotators separately label opinion target and opin-
ion expression spans for 30% of laptop and restaurant re-
views (159 and 132 reviews, respectively). This includes 4. Implications for New Applications
pronominal target resolution, implicit targets, and seman- The original motivation behind ABSA was to summarize
tic head annotations for subjective opinion expressions and user-generated online reviews (Hu and Liu, 2004a; Hu and
lexical targets. Inter-annotator agreement is calculated and Liu, 2004b). The idea is that if all opinion targets can be
disagreements are resolved with the meta-annotator. Due modeled in terms of entity, aspect pairs, then all of the
to the high agreement between annotators for pronominal opinions in a given review can be reported in a structured
and implicit target resolution, as well as semantic head an- manner. Current work on ABSA is based on pre-defining
notations (.98−1.0). It is agreed to let the meta-annotator aspects, but there are inherent limitations to handling opin-
label these fields over the final lexical target and opinion ion targets in this way. First, targets defined in this way
expression annotations. The two annotators are then each
assigned 50% of the remaining laptop and restaurant re- 5
The datasets are publicly available for download at
views and annotate opinion target and expression spans. https://godiaz01.github.io/resources/

6809
are frequently highly dependent on domains (e.g., laptops 6. Acknowledgments
and restaurants do not have many aspects in common under We thank the three anonymous reviewers for their detailed
the SemEval annotation schemes), making it complicated and insightful comments on an earlier draft of the pa-
to apply ABSA to a domain for which one does not have per. This work was supported in part by NSF Grants IIS-
annotated data. Second, working at the aspect level can 1528037 and CCF-1848608. Any opinions, findings, con-
result in the loss of important information. Consider The clusions or recommendations expressed in this paper are
charge capacity is excellent! and The battery gets really those of the authors and do not necessarily reflect the views
hot when charging in the context of a laptop review. Both or official policies, either expressed or implied, of NSF.
sentences can be reported under the target (battery, oper-
ating performance), but they do not address the same di-
7. Bibliographical References
mension of the operating performance. Attempting to solve
this problem by pre-specifying more aspects increases the Chinchor, N. A. (1998). Overview of MUC-7/MET-2.
number of needed annotations and exacerbates the previ- In Proceedings of the Seventh Message Understanding
ous problem. Finally, a system can only report opinions in Conference.
terms of the fixed aspects and aspect hierarchy that are de- Cohen, J. (1960). A coefficient of agreement for nom-
fined by its training data; in particular, there is no way for inal scales. Educational and Psychological Measure-
a system to report opinions in terms of dimensions that are ment, 20(1):37–46.
of interest to a specific user. Deng, L. and Wiebe, J. (2015). Mpqa 3.0: An entity/event-
level sentiment corpus. In Proceedings of the 2015 Con-
We envision that more advanced fine-grained sentiment
ference of the North American Chapter of the Associa-
analysis systems should be able to learn meaningful ways
tion for Computational Linguistics: Human Language
to report opinions without requiring humans to specify as-
Technologies, pages 1323–1328.
pect hierarchies, prevent information loss that may make
Grishman, R. and Sundheim, B. (1995). Design of the
two different opinions be reported under the same target
MUC-6 evaluation. In Proceedings of the Sixth Message
when they in fact do not talk about the same thing, and re-
Understanding Conference, pages 1–11.
port opinions under different aspects to satisfy the needs
of different users. Such systems would be able to provide Hu, M. and Liu, B. (2004a). Mining and summarizing
shopping assistance for example, providing side by side customer reviews. In Proceedings of the Tenth ACM
comparisons of two unseen products based entirely on their SIGKDD International Conference on Knowledge Dis-
reviews, identifying meaningful axes around which to cat- covery and Data Mining, pages 168–177.
egorize and order opinions independently, or reporting re- Hu, M. and Liu, B. (2004b). Mining opinion features in
viewers’ opinions in terms of dimensions specified by the customer reviews. In Proceedings of the 19th National
user. Given that the combination of lexical targets and opin- Conference on Artificial Intelligence, pages 755–760.
ion expressions encodes sentiment as expressed by writes Liu, B., Hu, M., and Cheng, J. (2005). Opinion observer:
without loss of information, we believe we will need re- Analyzing and comparing opinions on the web. In Pro-
sources based around them to work towards more advance ceedings of the 14th International Conference on World
sentiment analysis systems in this domain. Wide Web, pages 342–351.
Liu, B. (2012). Sentiment analysis and opinion mining.
Synthesis Lectures on Human Language Technologies,
5(1):1–167.
5. Conclusion Liu, B. (2015). Sentiment Analysis: Mining Opinions, Sen-
timents, and Emotions. Cambridge University Press.
In this paper we attempted to bridge the gap between gen- Liu, B. (2017). Many facets of sentiment analysis. In
eral opinion mining and aspect-based sentiment analysis by A Practical Guide to Sentiment Analysis, pages 11–39.
showing the relationship between ABSA concepts such as Springer.
opinion models and aspects and those used in opinion min- Martı́nez Cámara, E., Almeida Cruz, Y., Dı́az Galiano,
ing, opinion expressions and opinion targets. We showed M. C., Estévez-Velarde, S., Garcı́a Cumbreras, M. Á.,
how traditional ABSA annotation schemes based on rep- Garcı́a Vega, M., Gutiérrez, Y., Montejo Ráez, A., Mon-
resenting targets as (entity, aspect) pairs can result in los- toyo, A., Muñoz, R., et al. (2018). Overview of TASS
ing the original opinion target intended by the author and 2018: Opinions, health and emotions. Proceedings of
how the combination of lexical opinion targets and opin- TASS 2018: Workshop on Semantic Analysis at SEPLN.
ion expressions solves this problem while at the same time Pontiki, M., Galanis, D., Pavlopoulos, J., Papageorgiou,
enabling the inference of valid (entity, aspect) pairs at dif- H., Androutsopoulos, I., and Manandhar, S. (2014).
ferent levels of granularity. We reannotated the popular SemEval-2014 task 4: Aspect based sentiment analysis.
SemEval 2016 ABSA restaurant and laptop datasets based In Proceedings of the 8th International Workshop on Se-
on this notion, annotating opinion expressions, opinion tar- mantic Evaluation (SemEval 2014), pages 27–35.
gets, and opinion polarities, as well as the semantic heads Pontiki, M., Galanis, D., Papageorgiou, H., Manandhar, S.,
of opinion expressions and opinion targets. Finally, we dis- and Androutsopoulos, I. (2015). SemEval-2015 task 12:
cussed how these datasets could be useful for building next- Aspect based sentiment analysis. In Proceedings of the
generation ABSA systems. To stimulate work on this prob- 9th International Workshop on Semantic Evaluation (Se-
lem, we make our datasets publicly available. mEval 2015), pages 486–495.

6810
Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopou-
los, I., Manandhar, S., AL-Smadi, M., Al-Ayyoub, M.,
Zhao, Y., Qin, B., De Clercq, O., Hoste, V., Apidi-
anaki, M., Tannier, X., Loukachevitch, N., Kotelnikov,
E., Bel, N., Jiménez-Zafra, S. M., and Eryiğit, G. (2016).
SemEval-2016 task 5: Aspect based sentiment analysis.
In Proceedings of the 10th International Workshop on
Semantic Evaluation (SemEval-2016), pages 19–30.
Wiebe, J., Wilson, T., and Cardie, C. (2005). Annotating
expressions of opinions and emotions in language. Lan-
guage Resources and Evaluation, 39(2-3):165–210.
Wilson, T. A. (2008). Fine-grained subjectivity and senti-
ment analysis: recognizing the intensity, polarity, and at-
titudes of private states. Ph.D. thesis, University of Pitts-
burgh.
Wojatzki, M., Ruppert, E., Holschneider, S., Zesch, T., and
Biemann, C. (2017). Germeval 2017: Shared task on
aspect-based sentiment in social media customer feed-
back. Proceedings of the GermEval, pages 1–12.
Yang, B. and Cardie, C. (2013). Joint inference for fine-
grained opinion extraction. In Proceedings of the 51st
Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1640–1649.

6811