Expert Systems With Applications 72 (2017) 221–230

Contents lists available at ScienceDirect

Expert Systems With Applications

journal homepage: www.elsevier.com/locate/eswa

Improving sentiment analysis via sentence type classification using

BiLSTM-CRF and CNN
Tao Chen a, Ruifeng Xu a,b,∗, Yulan He c, Xuan Wang a
a
Shenzhen Engineering Laboratory of Performance Robots at Digital Stage, Shenzhen Graduate School, Harbin Institute of Technology, Shenzhen, China
b
Guangdong Provincial Engineering Technology Research Center for Data Science, Guangzhou, China
c
School of Engineering and Applied Science, Aston University, Birmingham, UK

a r t i c l e i n f o a b s t r a c t

Article history: Different types of sentences express sentiment in very different ways. Traditional sentence-level senti-
Received 9 July 2016 ment classification research focuses on one-technique-fits-all solution or only centers on one special type
Revised 1 October 2016
of sentences. In this paper, we propose a divide-and-conquer approach which first classifies sentences
Accepted 21 October 2016
into different types, then performs sentiment analysis separately on sentences from each type. Specif-
Available online 9 November 2016
ically, we find that sentences tend to be more complex if they contain more sentiment targets. Thus,
Keywords: we propose to first apply a neural network based sequence model to classify opinionated sentences into
Natural language processing three types according to the number of targets appeared in a sentence. Each group of sentences is then
Sentiment analysis fed into a one-dimensional convolutional neural network separately for sentiment classification. Our ap-
Deep neural network proach has been evaluated on four sentiment classification datasets and compared with a wide range of
baselines. Experimental results show that: (1) sentence type classification can improve the performance
of sentence-level sentiment analysis; (2) the proposed approach achieves state-of-the-art results on sev-
eral benchmarking datasets.
© 2016 The Authors. Published by Elsevier Ltd.
This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).

1. Introduction positive; for the interrogative sentence “Is it good?”, the sentiment
polarity is obscure, and it slightly inclined to the negative; for the
Sentiment analysis is the field of study that analyzes people’s comparative sentence “A is better than B.”, we even cannot decide
opinions, sentiments, appraisals, attitudes, and emotions toward its sentiment polarity, because it is dependent on which opinion
entities and their attributes expressed in written text (Liu, 2015). target we focus on (A or B).
With the rapid growth of social media on the web, such as re- Unlike factual text, sentiment text can often be expressed in a
views, forum discussions, blogs, news, and comments, more and more subtle or arbitrary manner, making it difficult to be identi-
more people share their views and opinions online. As such, this fied by simply looking at each constituent word in isolation. It is
fascinating problem is increasingly important in business and soci- argued that there is unlikely to have a one-technique-fits-all so-
ety. lution (Narayanan, Liu, & Choudhary, 2009). A divide-and-conquer
One of the main directions of sentiment analysis is sentence- approach may be needed to deal with some special sentences
level sentiment analysis. Much of the existing research on this with unique characteristics, that is, different types of sentences
topic focused on identifying the polarity of a sentence (e.g. posi- may need different treatments on sentence-level sentiment anal-
tive, negative, neutral) based on the language clues extracted from ysis (Liu, 2015).
the textual content of sentences (Liu, 2012; Pang & Lee, 2004; Tur- There are many ways in classifying sentences in sentiment
ney, 2002). They solved this task as a general problem without analysis. Sentences can be classified as subjective and objec-
considering different sentence types. However, different types of tive which is to separate opinions from facts (Wiebe & Wilson,
sentences express sentiment in very different ways. For example, 2002; Wiebe, Bruce, & O’Hara, 1999; Yu & Hatzivassiloglou, 2003).
for the sentence “It is good.”, the sentiment polarity is definitely Some researchers focused on target-dependent sentiment classi-
fication, which is to classify sentiment polarity for a given tar-
get on sentences consisting of explicit sentiment targets (Dong
∗
Corresponding author. Fax.: 8618988759558. et al., 2014; Jiang, Yu, Zhou, Liu, & Zhao, 2011; Mitchell, Aguilar,
E-mail addresses: [email protected] (T. Chen), [email protected] (R. Wilson, & Durme, 2013; Tang, Qin, Feng, & Liu, 2015a; Vo &
Xu), [email protected] (Y. He), [email protected] (X. Wang).

http://dx.doi.org/10.1016/j.eswa.2016.10.065
0957-4174/© 2016 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
222 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230

Zhang, 2015). Others dealt with mining opinions in comparative state-of-the-art results on several benchmarking datasets. This
sentences, which is to determinate the degree of positivity sur- shows that sentence type classification can improve the perfor-
round the analysis of comparative sentences (Ganapathibhotla & mance of sentence-level sentiment analysis.
Liu, 2008; Jindal & Liu, 2006b; Yang & Ko, 2011). There has also The main contributions of our work are summarized below:
been work focusing on sentiment analysis of conditional sentences
• We propose a novel two-step pipeline framework for sentence-
(Narayanan et al., 2009), or sentences with modality, which have
level sentiment classification by first classifying sentences into
some special characteristics that make it hard for a system to de-
different types based on the number of opinion targets they
termine sentiment orientations (Liu, Yu, Chen, & Liu, 2013).
contain, and then training 1d-CNNs separately for sentences in
In this paper, we propose a different way in dealing with dif-
each type for sentiment detection;
ferent sentence types. In particular, we investigate the relation-
• While conventional sentiment analysis methods largely ignore
ship between the number of opinion targets expressed in a sen-
different sentence types, we have validated in our experiments
tence and the sentiment expressed in this sentence; propose a
that learning a sentiment classifier tailored to each sentence
novel framework for improving sentiment analysis via sentence
type would result in performance gains in sentence-level senti-
type classification. Opinion target (hereafter, target for short) can
ment classification.
be any entity or aspect of the entity on which an opinion has been
expressed (Liu, 2015). An opinionated sentence can express senti- The rest of this article is organized as follows: we review
ments without a mention of any target, or towards one target, two related work in Section 2; and then present our approach in
or more targets. We define three types of sentences: non-target Section 3; experimental setup, evaluation results and discussions
sentences, one-target sentences and multi-target sentences, re- are reported in Section 4; finally, Section 5 concludes the paper
spectively. Consider the following examples from the movie review and outlines future research directions.
sentence polarity dataset v1.0 (hereafter, MR dataset for short)
(Pang & Lee, 2005)1 : 2. Related work

Example 1. A masterpiece four years in the making.

2.1. Sentence type classification for sentiment analysis
Example 2. If you sometimes like to go to the movies to have fun,
Wasabi is a good place to start. Since early 20 0 0, sentiment analysis has grown to be one
of the most active research areas in natural language processing
Example 3. Director Kapur is a filmmaker with a real flair for epic (NLP) (Liu, 2015; Ravi & Ravi, 2015). It is a multifaceted prob-
landscapes and adventure, and this is a better film than his earlier lem with many challenging and interrelated sub-problems, includ-
English-language movie, the overpraised Elizabeth. ing sentence-level sentiment classification. Many researchers real-
ized that different type of sentence need different treatment for
Example 1 is a non-target sentence. In order to infer its tar-
sentiment analysis. Models of different sentence types, including
get, we need to know its context. Example 2 is a one-target sen-
subjective sentences, target-dependent sentences, comparative sen-
tence, in which the sentiment polarity of the target Wasabi is posi-
tences, negation sentences, conditional sentences, sarcastic sen-
tive. Example 3 is a multi-target sentence, in which there are three
tences, have been proposed for sentiment analysis.
targets: Director Kapur, film and his earlier English-language movie,
Subjectivity classification distinguishes sentences that express
the overpraised Elizabeth. We can observe that sentences tend to be
opinions (called subjective sentences) from sentences that express
more complex with more opinion targets, and sentiment detection
factual information (called objective sentences) (Liu, 2015). Al-
is more difficult for sentences containing more targets.
though some objective sentences can imply sentiments or opin-
Based on this observation, we apply a deep neural network se-
ions and some subjective sentences may not express any opinion
quence model, which is a bidirectional long short-term memory
or sentiment, many researchers regard subjectivity and sentiment
with conditional random fields (henceforth BiLSTM-CRF) (Lample,
as the same concept (Hatzivassiloglou & Wiebe, 20 0 0; Wiebe et al.,
Ballesteros, Subramanian, Kawakami, & Dyer, 2016), to extract tar-
1999), i.e., subjective sentences express opinions and objective sen-
get expressions in opinionated sentences. Based on the targets ex-
tences express fact. Riloff and Wiebe (2003) presented a boot-
tracted, we classify sentences into three groups: non-target, one-
strapping process to learn linguistically rich extraction patterns for
target and multi-target. Then, one-dimensional convolutional neu-
subjective expressions from a large unannotated data. Rill, Reinel,
ral networks (1d-CNNs) (Kim, 2014) are trained for sentiment clas-
Scheidt, and Zicari (2014) presented a system to detect emerging
sification on each group separately. Finally, the sentiment polarity
political topics on twitter and the impact on concept-level senti-
of each input sentence is predicted by one of the three 1d-CNNs.
ment analysis. Appel, Chiclana, Carter, and Fujita (2016) proposed
We evaluate the effectiveness of our approach empirically on
a hybrid approach using SentiWordNet (Baccianella, Esuli, & Se-
various benchmarking datasets including the Stanford sentiment
bastiani, 2010) and fuzzy sets to estimate the semantic orienta-
treebank (SST)2 (Socher et al., 2013) and the customer reviews
tion polarity and intensity of sentiment words, before computing
dataset (CR)3 (Hu & Liu, 2004). We compare our results with a
the sentence level sentiments. Muhammad, Wiratunga, and Loth-
wide range of baselines including convolutional neural networks
ian (2016) introduced a lexicon-based sentiment classification sys-
(CNN) with multi-channel (Kim, 2014), recursive auto-encoders
tem for social media genres, which captures contextual polarity
(RAE) (Socher, Pennington, Huang, Ng, & Manning, 2011), recur-
from both local and global context. Fernández-Gavilanes, Álvarez-
sive neural tensor network (RNTN) (Socher et al., 2013), dynamic
López, Juncal-Martínez, Costa-Montenegro, and González-Castaño
convolutional neural network (DCNN) (Kalchbrenner, Grefenstette,
(2016) proposed a novel approach to predict sentiment in online
& Blunsom, 2014), Naive Bayes support vector machines (NBSVM)
texts based on an unsupervised dependency parsing-based text
(Wang & Manning, 2012), dependency tree with conditional ran-
classification method.
dom fields (tree-CRF) (Nakagawa, Inui, & Kurohashi, 2010) et al.
Most previous target related works assumed targets have been
Experimental results show that the proposed approach achieves
given before performing sentiment classification (Dong et al., 2014;
Jiang et al., 2011; Mitchell et al., 2013; Vo & Zhang, 2015). Little
1
Available at: https://www.cs.cornell.edu/people/pabo/movie-review-data/. research has been conducted on classifying sentence by the target
2
http://nlp.stanford.edu/sentiment/. number although there is a large body of work focusing on opinion
3
https://www.cs.uic.edu/∼liub/FBS/sentiment-analysis.html#datasets. target extraction from text.
 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230 223

A comparative opinion sentence expresses a relation of similar- 2.2. Opinion target detection
ities or differences between two or more entities and/or a prefer-
ence of the opinion holder based on some shared aspects of the Hu and Liu (2004) used frequent nouns and noun phrases as
entities. Jindal and Liu (2006a) showed that almost every com- feature candidates for opinion target extraction. Qiu, Liu, Bu, and
parative sentence had a keyword (a word or phrase) indicating Chen (2011) proposed a bootstrapping method where a depen-
comparison, and identified comparative sentences by using class dency parser was used to identify syntactic relations that linked
sequential rules based on human compiled keywords as features opinion words and targets for opinion target extraction. Popescu
for a naive Bayes classifier. Ganapathibhotla and Liu (2008) re- and Etzioni (2005) considered product features to be concepts
ported they were the first work for mining opinions in compara- forming certain relationships with the product and sought to iden-
tive sentences. They solved the problem by using linguistic rules tify the features connected with the product name by comput-
and a large external corpus of Pros and Cons from product re- ing the point wise mutual information (PMI) score between the
views to determine whether the aspect and sentiment context phrase and class-specific discriminators through a web search.
were more associated with each other in Pros or in Cons. Kessler Stoyanov and Cardie (2008) treated target extraction as a topic co-
and Kuhn (2014) presented a corpus of comparison sentences from reference resolution problem and proposed to train a classifier to
English camera reviews. Park and Yuan (2015) proposed two lin- judge if two opinions were on the same target. Liu, Xu, and Zhao
guistic knowledge-driven approaches for Chinese comparative ele- (2014) constructed a heterogeneous graph to model semantic rela-
ments extraction. tions and opinion relations, and proposed a co-ranking algorithm
Negation sentences occur fairly frequently in sentiment anal- to estimate the confidence of each candidate. The candidates with
ysis corpus. Many researchers considered the impact of negation higher confidence would be extracted as opinion targets. Poria,
words or phrases as part of their works (Hu & Liu, 2004; Pang, Cambria, and Gelbukh (2016) presented the first deep learning ap-
Lee, & Vaithyanathan, 2002); a few researchers investigated nega- proach to aspect extraction in opinion mining using a 7-layer CNN
tion words identification and/or negative sentence processing as a and a set of linguistic patterns to tag each word in sentences.
single topic. Jia, Yu, and Meng (2009) studied the effect of nega- Mitchell et al. (2013) modeled sentiment detection as a se-
tion on sentiment analysis, including negation term and its scope quence tagging problem, extracted named entities and their sen-
identification, by using a parse tree, typed dependencies and spe- timent classes jointly. They referred this kind of approach open
cial linguistic rules. Zhang, Ferrari, and Enjalbert (2012) proposed domain targeted sentiment detection. Zhang, Zhang, and Vo
a compositional model to detect valence shifters, such as nega- (2015) followed Mitchell et al.’s work, studied the effect of word
tions, which contribute to the interpretation of the polarity and embeddings and automatic feature combinations on the task by
the intensity of opinion expressions. Carrillo-de Albornoz and Plaza extending a CRF baseline using neural networks.
(2013) studied the effect of modifiers on the emotions affected by
negation, intensifiers and modality. 2.3. Deep learning for sentiment classification
Conditional sentences are another commonly used language
constructs in text. Such a sentence typically contains two clauses: Deep learning approaches are able to automatically capture, to
the condition clause and the consequent clause. Their relationship some extent, the syntactic and semantic features from text without
has significant impact on the sentiment orientation of the sen- feature engineering, which is labor intensive and time consuming.
tence (Liu, 2015). Narayanan et al. (2009) first presented a lin- They attract much research interest in recent years, and achieve
guistic analysis of conditional sentences, and built some super- state-of-the-art performances in many fields of NLP, including sen-
vised learning models to determine if sentiments expressed on timent classification.
different topics in a conditional sentence are positive, negative Socher et al. (2011) introduced semi-supervised recursive au-
or neutral. Liu (2015) listed a set of interesting patterns in con- toencoders for predicting sentiment distributions without using
ditional sentences that often indicate sentiment, which was par- any pre-defined sentiment lexica or polarity shifting rules. Socher
ticularly useful for reviews, online discussions, and blogs about et al. (2013) proposed a family of recursive neural network, in-
products. cluding recursive neural tensor network (RNTN), to learn the
Sarcasm is a sophisticated form of speech act widely used in compositional semantic of variable-length phrases and sentences
online communities. In the context of sentiment analysis, it means over a human annotated sentiment treebank. Kalchbrenner et al.
that when one says something positive, one actually means neg- (2014) and Kim (2014) proposed different CNN models for senti-
ative, and vice versa. Tsur, Davidov, and Rappoport (2010) pre- ment classification, respectively. Both of them can handle the in-
sented a novel semi-supervised algorithm for sarcasm identifi- put sentences with varying length and capture short and long-
cation that recognized sarcastic sentences in product reviews. range relations. Kim (2014)’s model has little hyper parameter tun-
González-Ibánez, Muresan, and Wacholder (2011) reported on a ing and can be trained on pre-trained word vectors. Irsoy and
method for constructing a corpus of sarcastic Twitter messages, Cardie (2014a) presented a deep recursive neural network (DRNN)
and used this corpus to investigate the impact of lexical and prag- constructed by stacking multiple recursive layers for composi-
matic factors on machine learning effectiveness for identifying sar- tionality in Language and evaluated the proposed model on sen-
castic utterances. Riloff et al. (2013) presented a bootstrapping al- timent classification tasks. Tai, Socher, and Manning (2015) in-
gorithm for sarcasm recognition that automatically learned lists troduced a tree long short-term memory (LSTM) for improving
of positive sentiment phrases and negative situation phrases from semantic representations, which outperforms many existing sys-
sarcastic tweets. tems and strong LSTM baselines on sentiment classification. Tang
Adversative and concessive structures, as another kind of lin- et al. (2015c) proposed a joint segmentation and classification
guistical feature, are constructions express antithetical circum- framework for sentence-level sentiment classification. Liu, Qiu, and
stances (Crystal, 2011). A adversative or a concessive clause is Huang (2016) used a recurrent neural network (RNN) based mul-
usually in clear opposition to the main clause about the fact titask learning framework to jointly learn across multiple related
or event commented. Fernández-Gavilanes et al. (2016) treated tasks. Chaturvedi, Ong, Tsang, Welsch, and Cambria (2016) pro-
the constructions as an extension of intensification propaga- posed a deep recurrent belief network with distributed time de-
tion, where the sentiment formulated could be diminished or lays for learning word dependencies in text which uses Gaussian
intensified, depending on both adversative/concessive and main networks with time-delays to initialize the weights of each hidden
clauses. neuron. Tang, Qin, and Liu (2015b) gave a survey on this topic.
224 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230

Fig. 1. Framework of sentence type classification based sentiment analysis using BiLSTM-CRF and 1d-CNN.

Table 1 consisting of a hidden unit h and an optional output y. T is the

An example sentence with labels in IOB format. The target is the act,
last time step. It is also the length of input sentence in this text
the label B indicates the beginning of a target, I indicates that the word
is inside a target, and O indicates a word belongs to no target. sequence learning task. At each time step t, the hidden state ht of
the RNN is computed based on the previous hidden state ht−1 and
Words: Yet the act is still charming here .
the input at the current step xt :
Labels: O B I O O O O O
ht = g(Uxt + W ht−1 ) (1)

3. Methodology where U and W are weight matrices of the network; g( · ) is

a non-linear activation function, such as an element-wise logis-
We present our approach for improving sentiment analysis via tic sigmoid function. The output at time step t is computed as
sentence type classification in this section. An overview of the yt = softmax(V ht ), where V is another weight parameter of the
approach is shown in Fig. 1. We first introduce the BiLSTM-CRF network, softmax is an activation function often implemented at
model which extracts target expressions from input opinionated the final layer of a network.
sentences, and classifies each sentence according to the number LSTM is a variant of RNN designed to deal with vanishing gra-
of target explicitly expressed in it (Section 3.1). Then, we describe dients problem (Hochreiter & Schmidhuber, 1997). The LSTM used
the 1d-CNN sentiment classification model which predicts senti- in the BiLSTM-CRF (Lample et al., 2016) has two gates (an input
ment polarity for non-target sentences, one-target sentences and gate it , an output gate ot ) and a cell activation vectors ct .
multi-target sentences, separately (Section 3.2). BiLSTM uses two LSTMs to learn each token of the sequence
based on both the past and the future context of the token. As
3.1. Sequence model for sentence type classification shown in Fig. 2, one LSTM processes the sequence from left to
right, the other one from right to left. At each time step t, a hidden
−
→
We describe our approach for target extraction and sentence forward layer with hidden unit function h is computed based on
−
→
type classification with BiLSTM-CRF. Target extraction is similar the previous hidden state h t−1 and the input at the current step xt
to the classic problem of named entity recognition (NER), which ← −
and a hidden backward layer with hidden unit function h is com-
views a sentence as a sequence of tokens usually labeled with IOB ←−
puted based on the future hidden state h t+1 and the input at the
format (short for Inside, Outside, Beginning). Table 1 shows an ex-
current step xt . The forward and backward context representations,
ample sentence with the appropriate labels in this format. −
→ ←−
Deep neural sequence models have shown promising success generated by h t and h t respectively, are concatenated into a long
in NER (Lample et al., 2016), sequence tagging (Huang, Xu, & Yu, vector. The combined outputs are the predictions of teacher-given
2015) and fine-grained opinion analysis (Irsoy & Cardie, 2014b). target signals.
BiLSTM-CRF is one of deep neural sequence models, where a bidi- As another widely used sequence model, conditional random
rectional long short-term memory (BiLSTM) layer (Graves, Mo- fields (CRF) is a type of discriminative undirected probabilistic
hamed, & Hinton, 2013) and a conditional random fields (CRF) graphical model, which represents a single log-linear distributions
layer (Lafferty, McCallum, & Pereira, 2001) are stacked together for over structured outputs as a function of a particular observation
sequence learning, as shown in Fig. 2. BiLSTM incorporates a for- input sequence.
ward long short-term memory (LSTM) layer and a backward LSTM Given observations variables X whose values are observed, ran-
layer in order to learn information from preceding as well as fol- dom variables Y whose values the task requires the model to pre-
lowing tokens. LSTM (Hochreiter & Schmidhuber, 1997) is a kind dict, and a undirected graph G where Y are connected by undi-
of recurrent neural network (RNN) architecture with long short- rected edges indicating dependencies. CRF defines the conditional
term memory units as hidden units. Next we briefly describe RNN, probability of a set of output values y ∈ Y given a set of input val-
LSTM, BiLSTM and BiLSTM-CRF. ues x ∈ X to be proportional to the product of potential functions
RNN (Elman, 1990) is a class of artificial neural sequence model, on cliques of the graph (McCallum, 2003),
where connections between units form a directed cycle. It takes
1 
arbitrary embedding sequences x = (x1 , . . . , xT ) as input, uses its p( y | x ) = s (ys , xs ) (2)
internal memory network to exhibit dynamic temporal behavior. It Zx
s∈S(y,x )
 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230 225

Fig. 2. An illustration of BiLSTM-CRF for target extraction and sentence type classification. BiLSTM layer incorporates a forward LSTM layer and a backward LSTM layer.

where Zx is a normalization factor overall output values, S(y, x) is are generated. Many feature sequences are combined into a fea-
the set of cliques of G, s (ys , xs ) is the clique potential on clique s. ture map. In the pooling layer, a max-overtime pooling operation
Afterwards, in the BiLSTM-CRF model, a softmax over all possi- (Collobert et al., 2011) is applied to capture the most useful local
ble tag sequences yields a probability for the sequence y. The pre- features from feature maps. Activation functions are added to in-
diction of the output sequence is computed as follows: corporate element-wise non-linearity. The outputs of multiple fil-
ters are concatenated in the merge layer. After another dropout
y∗ = argmaxy∈Y σ (X, y ) (3)
process, a fully connected softmax layer output the probability dis-
where σ (X, y) is the score function defined as follows: tribution over labels from multiple classes.

n 
n CNN is one of most commonly used connectionism model for
σ (X, y ) = Ayi ,yi+1 + Pi,yi (4) classification. Connectionism models focus on learning from envi-
i=0 i=1 ronmental stimuli and storing this information in a form of con-
where A is a matrix of transition scores, Ayi ,yi+1 represents the nections between neurons. The weights in a neural network are
score of a transition from the tag yi to yi+1 . n is the length of a adjusted according to the training data by some learning algo-
sentence, P is the matrix of scores output by the BiLSTM network, rithm. It is the greater the difference in the training data, the
Pi,yi is the score of the yth tag of the ith word in a sentence. more difficult for the learning algorithm to adapt the training data,
i
As shown in Fig. 2, dropout technique is used after the input and the worse classification results. Dividing opinionated sentences
layer of BiLSTM-CRF to reduce overfitting on the training data. This into different types according to the number of targets expressed
technique is firstly introduced by Hinton, Srivastava, Krizhevsky, in them can reduce the differences of training data in each group,
Sutskever, and Salakhutdinov (2012) for preventing complex co- therefore, improve overall classification accuracy.
adaptations on the training data. It has given big improvements on
many tasks. 4. Experiment
After target extraction by BiLSTM-CRF, all opinionated sentences
are classified into non-target sentences, one-target sentences and We conduct experiments to evaluate the performance of the
multi-target sentences, according to the number of targets ex- proposed approach for sentence-level sentiment classification on
tracted from them. various benchmarking datasets. In this section, we describe the ex-
perimental setup and baseline methods followed by the discussion
3.2. 1d-CNN for sentiment classification on each sentence type of results.

1d-CNN, firstly proposed by Kim (2014), takes sentences of

4.1. Experimental setup
varying lengths as input and produces fixed-length vectors as out-
put. Before training, word embeddings for each word in the glos-
For training BiLSTM-CRF for target extraction and sentence type
sary of all input sentences are generated. All the word embeddings
classification, we use the MPQA opinion corpus v2.0 (MPQA dataset
are stacked in a matrix M. In the input layer, embeddings of words
for short) provided by Wiebe, Wilson, and Cardie (2005)4 since
comprising current training sentence are taken from M. The max-
it contains a diverse range of sentences with various numbers of
imum length of sentences that the network handles is set. Longer
opinion targets. It contains 14,492 sentences from a wide variety
sentences are cut; shorter sentences are padded with zero vectors.
of news sources manually annotated with opinion target at the
Then, dropout regularization is used to control over-fitting.
phrase level (7,026 targets). All the sentences are used to train
In the convolution layer, multiple filters with different window
BiLSTM-CRF.
size move on the word embeddings to perform one-dimensional
convolution. As the filter moves on, many sequences, which cap-
ture the syntactic and semantic features in the filtered n-gram, 4
http://mpqa.cs.pitt.edu/.
226 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230

For sentiment classification with 1d-CNN, we test our approach Table 2

Example sentences in each target class of Stanford sentiment treebank. T0, T1
on different datasets:
and T2+ refer to non-target sentences, one-target sentences and multi-target sen-
• MR: Movie review sentence polarity dataset v1.0. It contains tences recognized by BiLSTM-CRF, respectively. In each target class, we show
3 example sentences (one positive, one neutral, one negative sentence, respec-
5331 positive snippets and 5331 negative snippets extracted tively), s1 to s9 are the order numbers of the examples.
from Rotten Tomatoes web site pages where reviews marked
with “fresh” are labeled as positive, and reviews marked with Class Example sentences

“rotten” are labeled as negative. 10-fold cross validation was T0 s1: ...very funny, very enjoyable ...
used for testing. s2: Dark and disturbing, yet compelling to watch.
s3: Hey, who else needs a shower?
• SST-1: Stanford sentiment treebank contains 11,855 sentences
T1 s4: Yet the act is still charming here.
also extracted from the original pool of Rotten Tomatoes s5: As a director, Mr. Ratliff wisely rejects the temptation to make fun
page files. These sentences are split into 8544/1101/2210 for of his subjects.
train/dev/test. Each of them is fine-grained labeled (very pos- s6: Notorious C.H.O. has oodles of vulgar highlights.
itive, positive, neutral, negative, very negative). T2+ s7: Singer/composer Bryan Adams contributes a slew of songs – a few
potential hits, a few more simply intrusive to the story – but the
• SST-2: Binary labeled version of Stanford sentiment treebank, whole package certainly captures the intended, er, spirit of the piece.
in which neutral reviews are removed, very positive and posi- s8: You Should Pay Nine Bucks for This: Because you can hear about
tive reviews are labeled as positive, negative and very negative suffering Afghan refugees on the news and still be unaffected.
reviews are labeled as negative (Kim, 2014). It contains 9613 s9: ...while each moment of this broken character study is rich in
emotional texture, the journey doesn’t really go anywhere.
sentences split into 6920/872/1821 for train/dev/test.
• CR: Customer reviews of 5 digital products contains 3771 sen-
tences extracted from amazon.com, including 2405 positive
sentences and 1366 negative sentences. 10-fold cross validation • RNTN: Recursive deep neural network for semantic composi-
was used for testing. tionality over a sentiment treebank using tensor-based feature
function proposed by Socher et al. (2013).
Following Kim (2014)’s work, we use accuracy as the evaluation • Paragraph-Vec: An unsupervised algorithm learning distributed
metric to measure the overall sentiment classification performance.
feature representations from sentences and documents pro-
During training a BiLSTM-CRF for target extraction in a sen-
posed by Le and Mikolov (2014).
tence, the input sequence xt is set to the t-th word embedding • DCNN: Dynamic convolutional neural network with dynamic k-
(a distributed representation for a word (Bengio, Ducharme, Vin-
max pooling operation proposed by Kalchbrenner et al. (2014).
cent, & Jauvin, 2003)) in a input sentence. Publicly available word • CNN-non-static: 1d-CNN with pre-trained word embeddings
vectors trained from Google News5 are used as pre-trained word
and fine-tuning optimizing strategy proposed by Kim (2014).
embeddings. The size of these embeddings is 300. U, W, V and h0 • CNN-multichannel: 1d-CNN with two sets of pre-trained word
are initialized to a random vector of small values, ht+1 are initial-
embeddings proposed by Kim (2014).
ized to a copy of ht recursively. A back-propagation algorithm with • DRNN: Deep recursive neural networks with stacked multiple
Adam stochastic optimization method is used to train the network
recursive layers proposed by Irsoy and Cardie (2014a).
through time with learning rate of 0.05. After each training epoch, • Multi-task LSTM: A multi-task learning framework using LSTM
the network is tested on validation data. The log-likelihood of val-
to jointly learn across multiple related tasks proposed by Liu
idation data is computed for convergence detection.
et al. (2016).
For training CNN, we use: CNN-non-static model, ReLU as ac- • Tree LSTM: A generalization of LSTM to tree structured network
tivation function, Adadelta decay parameter of 0.95, dropout rate
topologies proposed by Tai et al. (2015).
of 0.5, the size of initial word vectors of 300. We use different fil- • Sentic patterns: A concept-level sentiment analysis approach
ter windows and feature maps for different target classes. For non-
using dependency-based rules proposed by Poria, Cambria,
target sentences, we use filter windows of 3, 4, 5 with 100 feature
Winterstein, and Huang (2014).
maps each; For one-target sentences, we use filter windows of 3,
4, 5, 6 with 100 feature maps each; For multi-target sentences, we
use filter windows of 3, 4, 5, 6, 7 with 200 feature maps each. 4.3. Results

4.2. Baseline methods 4.3.1. Qualitative evaluations

In Table 2, we show the example sentences in each target class
We benchmark the following baseline methods for sentence- of Stanford sentiment treebank. It is observed that many non-
level sentiment classification, some of them have been previously target sentences are small imperative sentences, have direct sub-
used in Kim (2014): jective expressions (DSEs) which consist of explicit mentions of
private states or speech events expressing private states (Irsoy &
• MNB: Multinomial naive Bayes with uni-bigrams. Cardie, 2014b), e.g., funny and enjoyable in s1, dark and disturb-
• NBSVM: SVM variant using naive Bayes log-count ratios as fea- ing in s2. For some non-target sentences, it is difficult to detect
ture values proposed by Wang and Manning (2012). its sentiment without context, e.g., it is unclear whether the word
• Tree-CRF: Dependency tree based method for sentiment classi- shower in s3 conveys positive or negative sentiment. Non-target
fication using CRF with hidden variables proposed by Nakagawa sentences tend to be short comparing with two other types of
et al. (2010). sentences. Many one-target sentences are simple sentences, which
• RAE: Semi-supervised recursive autoencoders with pre-trained contain basic constituent elements forming a sentence. The subject
word vectors from Wikipedia proposed by Socher et al. (2011). is mostly the opinionated target in a one-target sentence, e.g., the
• MV-RNN: Recursive neural network using a vector and a ma- act in s4, Mr. Ratliff in s5 and C.H.O. in s6. Almost all the multi-
trix on every node in a parse tree for semantic compositionality target sentences are compound/complex/compound-complex sen-
proposed by Socher, Huval, Manning, and Ng (2012). tences, which have two or more clauses, and are very complex
in expressions. Many of them have coordinating or subordinating
5
https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp= conjunctions, which make it difficult to identify the sentiment of a
sharing. whole sentence, e.g., but in s7, because and and in s8, while in s9.
 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230 227

Table 3 Table 4
Experimental results of sentiment classification accuracy. The class-by-class classification results using sentence type classification as well
% is omitted. The best results are highlighted in bold face. as without using sentence type classification on the four datasets. #train and
The results of the top 10 approaches have been previ- #test are the word number of sentences in training and test dataset, respec-
ously reported by Kim (2014). The top 3 approaches are tively; lmax and lavg are max and average word length of sentences, respectively;
conventional machine learning approaches with hand- AccCNN is the experimental result that we do sentiment classification directly on
crafted features. Sentic patterns is rule based approach. the four datasets using 1d-CNN (non-static) without sentence type classification,
Other 11 approaches, including our approach, are deep and statistic the accuracy on each target class the same with the target class
neural network (DNN) approaches, which can automati- recognized by BiLSTM-CRF. Accour is the experimental result of our approach
cally extract features from input data for classifier train- on each target class, which using both sentence type classification and 1d-CNN
ing without feature engineering. (non-static). Δ is the relative improvement ratio calculates. In the AccCNN , Accour
and Δ columns, % is omitted for conciseness.
Model MR SST-1 SST-2 CR
#train #test lmax lavg AccCNN Accour Δ
MNB 79.0 – – 80.0
NBSVM 79.4 – – 81.8 MR T0 6,426 698 52 18.8 83.3 84.5 1.44
Tree-CRF 77.3 – – 81.4 T1 2,552 290 56 22.7 77.9 78.8 1.16
Sentic patterns – – 86.2 – T2+ 618 78 51 27.1 73.9 75.1 1.62
RAE 77.7 43.2 82.4 – SST-1 T0 5,436 1,367 51 17.5 49.8 50.9 2.21
MV-RNN 79.0 44.4 82.9 – T1 2,495 655 56 21.0 45.1 46.1 2.22
RNTN – 45.7 85.4 – T2+ 613 189 52 25.5 38.0 39.8 4.74
Paragraph-Vec – 48.7 87.8 – SST-2 T0 4,373 1,134 50 16.9 87.6 89.6 2.28
DCNN – 48.5 86.8 – T1 2,047 534 53 20.3 83.9 86.7 3.34
CNN-non-static 81.5 48.0 87.2 84.3 T2+ 500 153 51 24.9 82.0 84.1 2.56
CNN-multichannel 81.1 47.4 88.1 85.0 CR T0 1,982 191 75 17.4 85.5 88.4 3.39
DRNN – 49.8 86.6 – T1 1,140 152 105 19.1 80.9 83.2 2.84
Multi-task LSTM – 49.6 87.9 – T2+ 273 33 95 27.9 74.1 78.0 5.26
Tree LSTM – 50.6 86.9 –
Our approach 82.3 48.5 88.3 85.4
Table 5
Experimental results of different sequence models.
% is omitted for conciseness. The best results are
Overall, as the result of the qualitative evaluations, the diffi- highlighted in bold face.
culty degree of sentiment classification on each sentence type is Model MR SST-1 SST-2 CR
T 2+ > T 1 > T 0, i.e., multi-target sentences are most difficult, while
CRF 81.7 47.6 87.4 84.4
non-target sentences are much easier for sentiment classification.
LSTM 81.3 47.5 87.6 84.1
The experimental results listed in the next subsection validate this BiRNN 81.7 48.1 87.9 84.8
observation. BiRNN-CRF 81.8 48.3 87.9 84.9
BiLSTM 82.0 48.3 88.0 85.3
4.3.2. Overall comparison BiLSTM-CRF 82.3 48.5 88.3 85.4

Table 3 shows the results achieved on the MR, SST-1, SST-2

and CR datasets. It is observed that comparing with three hand-
crafted features based methods, although RAE and MV-RNN per- ter performance because the divide-and-conquer approach, which
form worse on MR dataset, two CNN based methods gives better first classifies sentences into different types, then optimize the
results on both MR and CR datasets. This indicates the effective- sentiment classifier separately on sentences from each type.
ness of DNN approaches. Among 11 DNN approaches, our approach
outperforms other baselines on all the datasets except SST-1, i.e., 4.3.4. Comparison with different sequence models
our approach gives relative improvements of 0.98% compared to We have also experimented with different sequence models, in-
CNN-non-static on MR dataset, 0.23% and 0.47% relative improve- cluding CRF, LSTM, BiRNN (Schuster & Paliwal, 1997), BiRNN-CRF,
ments compared to CNN-multichannel on SST-2 and CR dataset, re- BiLSTM and BiLSTM-CRF, for sentence type classification. For CRF,
spectively. Comparing with two CNN based methods, our sentence we use CRFSuite (Okazaki, 2007) with word, Part-Of-Speech tag,
type classification based approach gives superior performance on prefix, suffix and a sentiment dictionary as features. For LSTM,
all the four datasets (including SST-1 dataset). These validate the BiRNN, BiRNN-CRF and BiLSTM, we also use Google News word
influences of sentence type classification in terms of sentence-level embeddings as pre-trained word embeddings. For other parame-
sentiment analysis. ters, we use default parameter settings.
Table 5 shows the experimental results on the MR, SST-1, SST-2
4.3.3. Comparison on each target class and CR datasets. It can be observed that BiLSTM-CRF outperforms
Table 4 shows the statistics and comparison of each target class all the other approaches on all the four datasets. It is because
on the MR, SST-1, SST-2 and CR datasets. The relative improvement BiLSTM-CRF has more complicated hidden units, and offers bet-
ratio Δ calculates as follows: ter composition capability than other DNN approaches. CRF with
hand-crafted features gives comparable performance to LSTM, but
Δ = (Accour − AccCNN ) ÷ AccCNN × 100 (5)
lower performance than more complex DNN models. BiRNN and
It is obvious that the performance for every target class is im- BiLSTM gives better performance compared to LSTM because they
proved using sentence type classification. Yet, the improvement for can learn each token of the sequence based on both the past and
the multi-target sentences (T2+) is more significant than other two the future context of the token, while LSTM only use the past con-
target classes on three of the four dataset, e.g. the relative im- text of the token. Comparing BiRNN and BiLSTM with BiRNN-CRF
provement ratio of T2+ class on the SST-1 and CR datasets are and BiLSTM-CRF, respectively, it is observed that combining CRF
4.75% and 5.26%, respectively, which are about twice higher than and DNN models can improve the performance of DNN approaches.
the relative improvement ratio of T1 class. Table 4 is a clear indica-
tion that the proposed sentence type classification based sentiment 4.3.5. Evaluation on opinion target extraction with BiLSTM-CRF
classification approach is very effective for complex sentences. One unavoidable problem for every multi-step approach is the
Both the AccCNN and Accour use 1d-CNN (non-static) and pre- propagation of errors. In our approach, we use a BiLSTM-CRF/1d-
trained Google News word embedding, our approach achieves bet- CNN pipeline for sentiment analysis. It is interesting to see how
228 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230

Table 6 We have also considered using other existing opinion target de-
Experimental results of target extraction with BiLSTM-CRF on SemEval16 task 5 as-
tection systems, which are specifically trained for this task. Unfor-
pect based sentiment analysis dataset subtask 1 slot 2. Best System refers to the
participation system with best performance submitted to SemEval16 task 5. Base- tunately, it is not very easy to find an applicable one. Some opin-
line refers to baseline model provided by the organizers; C refers to the model only ion target detection systems, such as Liu et al. (2014), can also be
uses the provided training data; U refers to the model uses other resources (e.g., regard as NER models.
publicly lexica) and additional data for training; “-” refers to no submissions were
made. % is omitted for conciseness. The best results are highlighted in bold face. 4.3.6. Error analysis for sentence type classification
Models English Spanish French Russian Dutch Turkish We have also done error analysis for sentence type classifica-
Best System U 72.34 68.39 66.67 33.47 56.99 –
tion. In this section, we list some result examples from the Stan-
Best System C 66.91 68.52 65.32 30.62 51.78 – ford sentiment treebank. The __O, __B and __I concatenated after
Baseline C 44.07 51.91 45.46 49.31 50.64 41.86 each word are the label predicted by BiLSTM-CRF.
BiLSTM-CRF C 72.44 71.70 73.50 67.08 64.29 63.76
Easy example 1: Yet__O the__B act__I is__O still__O charming__O
here__O .__O.
Easy example 2: The__B-MPQA movie__I-MPQA is__O pretty__O
funny__O now__O and__O then__O without__O in__O any__O
way__O demeaning__O its__O subjects__O .__O
the first stage of opinion target extraction impacts the final sen-
Easy example 3: Chomp__O chomp__O !__O.
timent classification. Evaluation on target extraction with BiLSTM-
Difficult example 1: You__B ’ll__O probably__O love__O it__B .__O
CRF is a fundamental step for this work.
Difficult example 2: This__B is__O n’t__O a__B new__I idea__I
Lample et al. (2016) reported that BiLSTM-CRF model obtained
.__O.
state-of-the-art performance in NER tasks in four languages with-
Difficult example 3: An__O engaging__O overview__O of__O John-
out resorting to any language-specific knowledge or resources.
son__O ’s__O eccentric__O career__O .__O
Specially, in CoNLL-2002 dataset, it achieved 85.75 and 81.74 F1
score in Spanish and Dutch NER tasks, respectively; In CoNLL-2003 It is observed that sentences with basic constituent elements
dataset, it achieved 90.94 and 78.76 F1 score in English and Ger- (Easy example 1), even if a litter long in length (Easy example 2),
man NER tasks, respectively. are relatively easier for target extraction with BiLSTM-CRF. One
We have also conducted experiments with BiLSTM-CRF using reason is that in these two sentences, the targets (the art and
the SemEval-2016 task 5 aspect based sentiment analysis dataset the movie) are commonly used nouns; Another reason is that the
(Pontiki et al., 2016). There are 3 subtasks in this task, each subtask MPQA dataset, used for training BiLSTM-CRF model, is obtained
contains several slots. We have conducted experiments on subtask from news sources. News text is usually more structured than the
1 slot 2: sentence-level opinion target expression extraction, on the text from other sources, such as web reviews. Small imperative
restaurants domain. F1 score is used as metric. The experimental sentence (Easy example 3) is also relatively easier for target extrac-
results are shown in Table 6. tion, because many of them are non-target sentences.
In this table, for English, the best systems are NLANG (Toh Sentences containing pronouns, such as you and it in Difficult
& Su, 2016) (U) and UWB (Hercig, Brychcín, Svoboda, & Konkol, example 1 and this in Difficult example 2, are relatively more diffi-
2016) (C), respectively; For Spanish, GTI (Álvarez López, Juncal- cult for target extraction with BiLSTM-CRF. Moreover, complex tar-
Martínez, Fernández-Gavilanes, Costa-Montenegro, & González- get, such as overview of Johnson’s eccentric career in Difficult exam-
Castaño, 2016) achieves both the best systems U and C; For French, ple 3, is also very difficult.
they are IIT-T (Kumar, Kohail, Kumar, Ekbal, & Biemann, 2016)
Example sentence: Their computer-animated faces are very ex-
(U) and XRCE (Brun, Perez, & Roux, 2016) (C); For Russian, Danii
pressive.
achieves both the best systems U and C; For Dutch, they are IIT-T
Result of CRF: Their__O computer-animated__O faces__B are__O
(Kumar et al., 2016) (U) and TGB (Çetin, Yıldırım, Özbey, & Eryiğit,
very__O expressive__O .__O
2016) (C).
Result of BiLSTM-CRF: Their__B computer-animated__I faces__I
It is observed that BiLSTM-CRF achieves the best performance
are__O very__O expressive__O .__O
on all the dataset using different languages, and outperforms the
others by a good margin in 5 out of 6 languages. It indicates that We have also analyzed examples in which BiLSTM-CRF de-
BiLSTM-CRF is effective in opinion target expression extraction. tects opinion targets better than CRF. As shown above, CRF can
We have also evaluated the performance of BiLSTM-CRF on the only identify a partial opinion target (faces), while BiLSTM-CRF can
MPQA dataset described in Section 4.1. We randomly select 90% identify the whole opinion target more accurately (their computer-
sentences in MPQA dataset for training and the remaining 10% sen- animated faces).
tences for testing. BiLSTM-CRF achieves 20.73 F1 score on opin-
ion target extraction. This is due to the complex nature of the 5. Conclusion
data that many opinion targets are not simple named entities such
as person, organization and location in typical NER tasks. Rather, This paper has presented a novel approach to improve
the opinion targets could be events, abstract nouns or multi-word sentence-level sentiment analysis via sentence type classification.
phrases. For example, “overview of Johnson’s eccentric career” in The approach employs BiLSTM-CRF to extract target expression in
sentence “An engaging overview of Johnson ’s eccentric career.”. Tar- opinionated sentences, and classifies these sentences into three
get number classification is much easier. It achieves 65.83% accu- types according to the number of targets extracted from them.
racy, when we classify the test sentences into 3 groups by the tar- These three types of sentences are then used to train separate
get numbers extracted from them. These results show that even 1d-CNNs for sentiment classification. We have conducted extensive
though the performance of the first step of our approach is not experiments on four sentence-level sentiment analysis datasets in
very high, our pipeline approach still achieves the state-of-the-art comparison with 11 other approaches. Empirical results show that
results on most benchmarking datasets. If we can improve the per- our approach achieves state-of-the-art performance on three of the
formance of the sequence model for opinion target extraction, the four datasets. We have found that separating sentences containing
final sentiment classification performance of our approach may be different opinion targets boosts the performance of sentence-level
further improved. sentiment analysis.
 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230 229

In future work, we plan to explore other sequence learning Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural Computa-
models for target expression detection and further evaluate our ap- tion, 9(8), 1735–1780.
Hu, M., & Liu, B. (2004). Mining and summarizing customer reviews. In Proceed-
proach on other languages and other domains. ings of the tenth ACM SIGKDD international conference on knowledge discovery and
data mining (KDD) (pp. 168–177). ACM.
Huang, Z., Xu, W., & Yu, K. (2015). Bidirectional lstm-crf models for sequence tag-
Acknowledgment ging. arXiv preprint arXiv:1508.01991.
Irsoy, O., & Cardie, C. (2014a). Deep recursive neural networks for compositional-
ity in language. In Advances in neural information processing systems 27: annual
This work is supported by the National Natural Science Foun- conference on neural information processing systems (NIPS) (pp. 2096–2104).
dation of China (No. 61370165, 61632011), National 863 Program Irsoy, O., & Cardie, C. (2014b). Opinion mining with deep recurrent neural networks.
of China 2015AA015405, Shenzhen Fundamental Research Funding In Proceedings of the 2014 conference on empirical methods in natural language
processing (EMNLP) (pp. 720–728).
JCYJ20150625142543470, Shenzhen Peacock Plan Research Grant
Jia, L., Yu, C., & Meng, W. (2009). The effect of negation on sentiment analysis and
KQCX20140521144507925 and Guangdong Provincial Engineering retrieval effectiveness. In Proceedings of the 18th ACM conference on information
Technology Research Center for Data Science 2016KF09. and knowledge management (CIKM) (pp. 1827–1830). ACM.
Jiang, L., Yu, M., Zhou, M., Liu, X., & Zhao, T. (2011). Target-dependent twitter
sentiment classification. In Proceedings of the 49th annual meeting of the asso-
References ciation for computational linguistics: Human language technologies (ACL): vol. 1
(pp. 151–160). Association for Computational Linguistics.
Álvarez López, T., Juncal-Martínez, J., Fernández-Gavilanes, M., Costa-Montenegro, E., Jindal, N., & Liu, B. (2006a). Identifying comparative sentences in text documents.
& González-Castaño, F. J. (2016). GTI at SemEval-2016 task 5: Svm and crf for In Proceedings of the 29th annual international ACM SIGIR conference on research
aspect detection and unsupervised aspect-based sentiment analysis. In Proceed- and development in information retrieval (SIGIR) (pp. 244–251). ACM.
ings of the 10th international workshop on semantic evaluation (SemEval-2016) Jindal, N., & Liu, B. (2006b). Mining comparative sentences and relations.
(pp. 306–311). San Diego, California: Association for Computational Linguistics. In Proceedings, the 21st national conference on artificial intelligence and the
URL: http://www.aclweb.org/anthology/S16-1049. 18th innovative applications of artificial intelligence conference (AAAI): vol. 22
Appel, O., Chiclana, F., Carter, J., & Fujita, H. (2016). A hybrid approach to the sen- (pp. 1331–1336).
timent analysis problem at the sentence level. Knowledge-Based Systems, 108, Kalchbrenner, N., Grefenstette, E., & Blunsom, P. (2014). A convolutional neural net-
110–124. work for modelling sentences. In Proceedings of the 52nd annual meeting of the
Baccianella, S., Esuli, A., & Sebastiani, F. (2010). SentiWordNet 3.0: An enhanced association for computational linguistics (ACL) (pp. 655–665). Baltimore, Mary-
lexical resource for sentiment analysis and opinion mining. In Proceedings of land: Association for Computational Linguistics.
the international conference on language resources and evaluation (LREC): vol. 10 Kessler, W., & Kuhn, J. (2014). A corpus of comparisons in product reviews. In In pro-
(pp. 2200–2204). ceedings of the 9th language resources and evaluation conference (LREC), Reykjavik,
Bengio, Y., Ducharme, R., Vincent, P., & Jauvin, C. (2003). A neural probabilistic lan- Iceland (pp. 2242–2248). URL: http://www.lrec-conf.org/proceedings/lrec2014/
guage model. Journal of Machine Learning Research, 3, 1137–1155. pdf/1001_Paper.pdf.
Brun, C., Perez, J., & Roux, C. (2016). XRCE at SemEval-2016 task 5: Feedbacked en- Kim, Y. (2014). Convolutional neural networks for sentence classification. In Proceed-
semble modeling on syntactico-semantic knowledge for aspect based sentiment ings of the 2014 conference on empirical methods in natural language processing
analysis. In Proceedings of the 10th international workshop on semantic evaluation (EMNLP) (pp. 1746–1751).
(SemEval-2016) (pp. 277–281). San Diego, California: Association for Computa- Kumar, A., Kohail, S., Kumar, A., Ekbal, A., & Biemann, C. (2016). IIT-TUDA at
tional Linguistics. URL: http://www.aclweb.org/anthology/S16-1044. SemEval-2016 task 5: Beyond sentiment lexicon: Combining domain depen-
Carrillo-de Albornoz, J., & Plaza, L. (2013). An emotion-based model of negation, dency and distributional semantics features for aspect based sentiment anal-
intensifiers, and modality for polarity and intensity classification. Journal of the ysis. In Proceedings of the 10th international workshop on semantic evaluation
American Society for Information Science and Technology, 64(8), 1618–1633. (SemEval-2016) (pp. 1129–1135). San Diego, California: Association for Compu-
Çetin, F. S., Yıldırım, E., Özbey, C., & Eryiğit, G. (2016). TGB at SemEval-2016 task tational Linguistics. URL: http://www.aclweb.org/anthology/S16-1174.
5: Multi-lingual constraint system for aspect based sentiment analysis. In Pro- Lafferty, J., McCallum, A., & Pereira, F. C. (2001). Conditional random fields: Proba-
ceedings of the 10th international workshop on semantic evaluation (SemEval-2016) bilistic models for segmenting and labeling sequence data. In Proceedings of the
(pp. 337–341). San Diego, California: Association for Computational Linguistics. eighteenth international conference on machine learning (ICML), Williams College,
URL: http://www.aclweb.org/anthology/S16-1054. Williamstown, MA, USA, June 28, - July 1, 2001 (pp. 282–289).
Chaturvedi, I., Ong, Y.-S., Tsang, I. W., Welsch, R. E., & Cambria, E. (2016). Learning Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016). Neural
word dependencies in text by means of a deep recurrent belief network. Knowl- architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
edge-Based Systems, 108, 144–154. Le, Q., & Mikolov, T. (2014). Distributed representations of sentences and documents.
Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., & Kuksa, P. (2011). In Proceedings of the 31th international conference on machine learning (ICML)
Natural language processing (almost) from scratch. The Journal of Machine Learn- (pp. 1188–1196).
ing Research, 12, 2493–2537. Liu, B. (2012). Sentiment analysis and opinion mining. Synthesis Lectures on
Crystal, D. (2011). Dictionary of linguistics and phonetics: vol. 30. John Wiley & Sons. Human Language Technologies. Morgan & Claypool Publishers. doi:10.2200/
Dong, L., Wei, F., Tan, C., Tang, D., Zhou, M., & Xu, K. (2014). Adaptive recursive neu- S00416ED1V01Y201204HLT016.
ral network for target-dependent twitter sentiment classification. In Proceedings Liu, B. (2015). Sentiment analysis: Mining opinions, sentiments, and emotions . Cam-
of the 52nd annual meeting of the association for computational linguistics (ACL) bridge University Press.
(pp. 49–54). Baltimore, Maryland: Association for Computational Linguistics. Liu, P., Qiu, X., & Huang, X. (2016). Recurrent neural network for text classification
Elman, J. L. (1990). Finding structure in time. Cognitive Science, 14(2), 179–211. with multi-task learning. arXiv preprint arXiv:1605.05101.
Fernández-Gavilanes, M., Álvarez-López, T., Juncal-Martínez, J., Costa-Montene- Liu, K., Xu, L., & Zhao, J. (2014). Extracting opinion targets and opinion words from
gro, E., & González-Castaño, F. J. (2016). Unsupervised method for sentiment online reviews with graph co-ranking. In Proceedings of the 52nd annual meet-
analysis in online texts. Expert Systems with Applications, 58, 57–75. ing of the association for computational linguistics (ACL) (pp. 314–324). Baltimore,
Ganapathibhotla, M., & Liu, B. (2008). Mining opinions in comparative sentences. Maryland: Association for Computational Linguistics.
In Proceedings of the 22nd international conference on computational linguistics Liu, Y., Yu, X., Chen, Z., & Liu, B. (2013). Sentiment analysis of sentences with modal-
(COLING): vol. 1 (pp. 241–248). Association for Computational Linguistics. ities. In Proceedings of the 2013 international workshop on mining unstructured big
González-Ibánez, R., Muresan, S., & Wacholder, N. (2011). Identifying sarcasm in data using natural language processing (UnstructureNLP@CIKM) (pp. 39–44). ACM.
twitter: a closer look. In Proceedings of the 49th annual meeting of the asso- McCallum, A. (2003). Efficiently inducing features of conditional random fields. In
ciation for computational linguistics: Human language technologies (ACL): vol. 2 Proceedings of the 19th conference on uncertainty in artificial intelligence (UAI)
(pp. 581–586). Association for Computational Linguistics. (pp. 403–410). Morgan Kaufmann.
Graves, A., Mohamed, A.-r., & Hinton, G. (2013). Speech recognition with deep re- Mitchell, M., Aguilar, J., Wilson, T., & Durme, B. V. (2013). Open domain targeted
current neural networks. In IEEE International conference on acoustics, speech and sentiment. In Proceedings of the 2013 conference on empirical methods in natural
signal processing (ICASSP) (pp. 6645–6649). IEEE. language processing, EMNLP 2013, 18–21 October 2013, Grand Hyatt Seattle, Seat-
Hatzivassiloglou, V., & Wiebe, J. M. (20 0 0). Effects of adjective orientation and grad- tle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL
ability on sentence subjectivity. In Proceedings of the 18th conference on compu- (pp. 1643–1654).
tational linguistics (COLING): vol. 1 (pp. 299–305). Association for Computational Muhammad, A., Wiratunga, N., & Lothian, R. (2016). Contextual sentiment analysis
Linguistics. for social media genres. Knowledge-Based Systems, 108, 92–101.
Hercig, T., Brychcín, T., Svoboda, L., & Konkol, M. (2016). UWB at SemEval-2016 Nakagawa, T., Inui, K., & Kurohashi, S. (2010). Dependency tree-based sentiment
task 5: Aspect based sentiment analysis. In Proceedings of the 10th international classification using CRFs with hidden variables. In Human language technologies:
workshop on semantic evaluation (SemEval-2016) (pp. 342–349). San Diego, Cali- The 2010 annual conference of the North American chapter of the association for
fornia: Association for Computational Linguistics. URL: http://www.aclweb.org/ computational linguistics (NAACL) (pp. 786–794). Association for Computational
anthology/S16-1055. Linguistics.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdi- Narayanan, R., Liu, B., & Choudhary, A. (2009). Sentiment analysis of conditional
nov, R. R. (2012). Improving neural networks by preventing co-adaptation of sentences. In Proceedings of the 2009 conference on empirical methods in natural
feature detectors. Computing Research Repository (CoRR), abs/1207.0580. URL: language processing (EMNLP): vol. 1 (pp. 180–189). Association for Computational
http://arxiv.org/abs/1207.0580. Linguistics.
230 T. Chen et al. / Expert Systems With Applications 72 (2017) 221–230

Okazaki, N. (2007). Crfsuite: a fast implementation of conditional random fields Stoyanov, V., & Cardie, C. (2008). Topic identification for fine-grained opinion anal-
(CRFs). ysis. In Proceedings of the 22nd international conference on computational linguis-
Pang, B., & Lee, L. (2004). A sentimental education: Sentiment analysis using subjec- tics (COLING): vol. 1 (pp. 817–824). Association for Computational Linguistics.
tivity summarization based on minimum cuts. In Proceedings of the 42nd annual Tai, K. S., Socher, R., & Manning, C. D. (2015). Improved semantic representa-
meeting on association for computational linguistics (ACL) (pp. 271–278). Associa- tions from tree-structured long short-term memory networks. arXiv preprint
tion for Computational Linguistics. arXiv:1503.0 0 075.
Pang, B., & Lee, L. (2005). Seeing stars: Exploiting class relationships for sentiment Tang, D., Qin, B., Feng, X., & Liu, T. (2015a). Target-dependent sentiment classifica-
categorization with respect to rating scales. In Proceedings of the 43rd annual tion with long short term memory. arXiv preprint arXiv:1512.01100.
meeting on association for computational linguistics (ACL) (pp. 115–124). Associa- Tang, D., Qin, B., & Liu, T. (2015b). Deep learning for sentiment analysis: Successful
tion for Computational Linguistics. approaches and future challenges . Wiley Interdisciplinary Reviews: Data Mining
Pang, B., Lee, L., & Vaithyanathan, S. (2002). Thumbs up?: sentiment classification and Knowledge Discovery, 5(6), 292–303. doi:10.1002/widm.1171. URL: http://
using machine learning techniques. In Proceedings of the 2002 conference on em- dx.doi.org/10.1002/widm.1171.
pirical methods in natural language processing (EMNLP) (pp. 79–86). Tang, D., Qin, B., Wei, F., Dong, L., Liu, T., & Zhou, M. (2015c). A joint segmen-
Park, M., & Yuan, Y. (2015). Linguistic knowledge-driven approach to chinese com- tation and classification framework for sentence level sentiment classification.
parative elements extraction. In Proceedings of the 8th SIGHAN workshop on chi- IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(11), 1750–
nese language processing (SIGHAN-8) (pp. 79–85). Association for Computational 1761. doi:10.1109/TASLP.2015.2449071.
Linguistics. Toh, Z., & Su, J. (2016). NLANGP at SemEval-2016 task 5: Improving aspect based
Pontiki, M., Galanis, D., Papageorgiou, H., Androutsopoulos, I., Manandhar, S., AL-S- sentiment analysis using neural network features. In Proceedings of the 10th in-
madi, M., Al-Ayyoub, M., Zhao, Y., Qin, B., Clercq, O. D., Hoste, V., Apidianaki, M., ternational workshop on semantic evaluation (SemEval-2016) (pp. 282–288). San
Tannier, X., Loukachevitch, N., Kotelnikov, E., Bel, N., Jimnez-Zafra, S. M., & Diego, California: Association for Computational Linguistics. URL: http://www.
Eryiit, G. (2016). SemEval-2016 task 5: Aspect based sentiment analysis. In Pro- aclweb.org/anthology/S16-1045.
ceedings of the 10th international workshop on semantic evaluation (SemEval). In Tsur, O., Davidov, D., & Rappoport, A. (2010). Icwsm-a great catchy name: Semi-su-
SemEval ’16 (pp. 19–30). San Diego, California: Association for Computational pervised recognition of sarcastic sentences in online product reviews. In Pro-
Linguistics. ceedings of the 4th international conference on weblogs and social media (ICWSM)
Popescu, A., & Etzioni, O. (2005). Extracting product features and opinions from (pp. 162–169).
reviews. In Proceedings of the 2005, human language technology conference and Turney, P. D. (2002). Thumbs up or thumbs down?: semantic orientation applied to
conference on empirical methods in natural language processing (HLT/EMNLP) unsupervised classification of reviews. In Proceedings of the 40th annual meeting
(pp. 9–28). Springer. on association for computational linguistics (acl) (pp. 417–424). Association for
Poria, S., Cambria, E., & Gelbukh, A. (2016). Aspect extraction for opinion min- Computational Linguistics.
ing with a deep convolutional neural network. Knowledge-Based Systems, 108, Vo, D.-T., & Zhang, Y. (2015). Target-dependent twitter sentiment classification with
42–49. rich automatic features. In Proceedings of the twenty-fourth international joint
Poria, S., Cambria, E., Winterstein, G., & Huang, G.-B. (2014). Sentic patterns: Depen- conference on artificial intelligence (IJCAI) (pp. 1347–1353).
dency-based rules for concept-level sentiment analysis. Knowledge-Based Sys- Wang, S., & Manning, C. D. (2012). Baselines and bigrams: Simple, good sentiment
tems, 69, 45–63. and topic classification. In Proceedings of the 50th annual meeting of the associa-
Qiu, G., Liu, B., Bu, J., & Chen, C. (2011). Opinion word expansion and target tion for computational linguistics (ACL): vol. 2 (pp. 90–94). Association for Com-
extraction through double propagation. Computational Linguistics, 37(1), 9–27. putational Linguistics.
doi:10.1162/coli_a_0 0 034. Wiebe, J. M., Bruce, R. F., & O’Hara, T. P. (1999). Development and use of a gold-s-
Ravi, K., & Ravi, V. (2015). A survey on opinion mining and sentiment analysis: tandard data set for subjectivity classifications. In Proceedings of the 37th annual
tasks, approaches and applications . Knowledge-Based Systems, 89, 14–46. meeting of the association for computational linguistics (ACL) (pp. 246–253). As-
Rill, S., Reinel, D., Scheidt, J., & Zicari, R. V. (2014). Politwi: Early detection of emerg- sociation for Computational Linguistics.
ing political topics on twitter and the impact on concept-level sentiment anal- Wiebe, J., & Wilson, T. (2002). Learning to disambiguate potentially subjective
ysis. Knowledge-Based Systems, 69, 24–33. expressions. In Proceedings of the 6th conference on natural language learning
Riloff, E., Qadir, A., Surve, P., De Silva, L., Gilbert, N., & Huang, R. (2013). Sarcasm as (CoNLL) (pp. 1–7). Association for Computational Linguistics.
contrast between a positive sentiment and negative situation.. In Proceedings of Wiebe, J., Wilson, T., & Cardie, C. (2005). Annotating expressions of opinions and
the 2013 conference on empirical methods on natural language processing (EMNLP) emotions in language. Language Resources and Evaluation, 39(2–3), 165–210.
(pp. 704–714). Association for Computational Linguistics. Yang, S., & Ko, Y. (2011). Extracting comparative entities and predicates from texts
Riloff, E., & Wiebe, J. (2003). Learning extraction patterns for subjective expressions. using comparative type classification. In Proceedings of the 49th annual meet-
In Proceedings of the 2003 conference on empirical methods in natural language ing of the association for computational linguistics: Human language technologies
processing (emnlp) (pp. 105–112). Association for Computational Linguistics. (ACL): vol. 1 (pp. 1636–1644). Association for Computational Linguistics.
Schuster, M., & Paliwal, K. K. (1997). Bidirectional recurrent neural networks. IEEE Yu, H., & Hatzivassiloglou, V. (2003). Towards answering opinion questions: Sepa-
Transactions on Signal Processing, 45(11), 2673–2681. rating facts from opinions and identifying the polarity of opinion sentences. In
Socher, R., Huval, B., Manning, C. D., & Ng, A. Y. (2012). Semantic compositionality Proceedings of the 2003 conference on empirical methods in natural language pro-
through recursive matrix-vector spaces. In Proceedings of the 2012 joint confer- cessing (EMNLP) (pp. 129–136). Association for Computational Linguistics.
ence on empirical methods in natural language processing and computational nat- Zhang, L., Ferrari, S., & Enjalbert, P. (2012). Opinion analysis: the effect of negation
ural language learning (EMNLP-CoNLL) (pp. 1201–1211). ACL. on polarity and intensity. In KONVENS workhop PATHOS-1st workshop on practice
Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., & Manning, C. D. (2011). Semi-su- and theory of opinion mining and sentiment analysis (pp. 282–290).
pervised recursive autoencoders for predicting sentiment distributions. In Pro- Zhang, M., Zhang, Y., & Vo, D.-T. (2015). Neural networks for open domain targeted
ceedings of the conference on empirical methods in natural language processing sentiment. In Proceedings of the 2015 conference on empirical methods in natural
(pp. 151–161). Association for Computational Linguistics. language processing (EMNLP) (pp. 612–621).
Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., &
Potts, C. (2013). Recursive deep models for semantic compositionality over a
sentiment treebank. In Proceedings of the 2013 conference on empirical methods
in natural language processing (EMNLP) (pp. 1631–1642). Citeseer.