CLMLF:A Contrastive Learning and Multi-Layer Fusion Method for

Multimodal Sentiment Detection

Zhen Li, Bing Xu∗, Conghui Zhu, Tiejun Zhao

Harbin Institute of Technology, Harbin, China
[email protected], {hitxb,conghui,tjzhao}@hit.edu.cn

Abstract
Compared with unimodal data, multimodal
data can provide more features to help the
model analyze the sentiment of data. Previ-
arXiv:2204.05515v4 [cs.CL] 14 Jun 2022

ous research works rarely consider token-level (a) Heathrow. Fly early to- (b) Blue Jays game with the
feature fusion, and few works explore learning morrow morning. (positive) fam! Let’s go! (positive)
the common features related to sentiment in
multimodal data to help the model fuse mul-
timodal features. In this paper, we propose
a Contrastive Learning and Multi-Layer Fu-
sion (CLMLF) method for multimodal senti-
ment detection. Specifically, we first encode
text and image to obtain hidden representa- (c) Ridge Avenue is closed af- (d) Flexible spinal cord im-
tions, and then use a multi-layer fusion mod- ter a partial building collapse plants will let paralyzed peo-
and electrical fire Saturday ple walk. (neutral)
ule to align and fuse the token-level features night. (negative)
of text and image. In addition to the sen-
timent analysis task, we also designed two Figure 1: Examples of multimodal sentiment tweets
contrastive learning tasks, label based con-
trastive learning and data based contrastive
learning tasks, which will help the model
multimodal sentiment analysis has become a popu-
learn common features related to sentiment in
multimodal data. Extensive experiments con- lar research topic (Kaur and Kautish, 2019).
ducted on three publicly available multimodal As for multimodal data, the complementarity
datasets demonstrate the effectiveness of our between text and image can help the model ana-
approach for multimodal sentiment detection lyze the real sentiment of the multimodal data. As
compared with existing methods. The codes shown in Figure 1, detecting sentiment with only
are available for use at https://github.
text modality or image modality may not be cer-
com/Link-Li/CLMLF
tain of the true intention of the tweet. Such as
1 Introduction Figure 1a, if we only analyze the text modality, we
will find that this is a declarative sentence that does
With the development of social networking plat- not express sentiment. In fact, the girl’s smile in
forms which have become the main platform for the image shows that the sentiment of this tweet is
people to share their personal opinions. How to positive. At the same time, in Figure 1c, we can
extract and analyze sentiments in social media data find that the ruins in the image which deepen the
efficiently and correctly has broad applications. expression of negative sentiment in the text.
Therefore, it has attracted attention from both aca- For multimodal sentiment analysis, we focus
demic and industrial communities (Zhang et al., on text-image sentiment analysis in social media
2018a; Yue et al., 2019). At the same time, with the data. In existing works, some models try to con-
increasing use of mobile internet and smartphones, catenate different modal feature vectors to fuse the
more and more users are willing to post multimodal multimodal features, such as MultiSentiNet (Xu
data (e.g., text, image, and video) about different and Mao, 2017) and HSAN (Xu, 2017). Kumar
topics to convey their feelings and sentiments. So and Vepa (2020) proposes to use gating mechanism
∗
Corresponding author and attention to obtain deep multimodal contextual
feature vectors. Multi-view Attentional Network performance of the model.
(MVAN) is proposed by Yang et al. (2020) which In this paper, we propose a Contrastive Learning
introduces memory networks to realize the interac- and Multi-Layer Fusion (CLMLF) method for
tion between modalities. Although the above men- multimodal sentiment analysis based on text and
tioned models are relatively better than unimodal image modalities. For evaluation, CLMLF is
models, the inputs with different modalities are in verified on three multimodal sentiment datasets,
different vector spaces. Therefore, it is difficult to namely MVSA-Single, MVSA-Multiple (Niu et al.,
fuse multimodal data with a simple concatenation 2016) and HFM (Cai et al., 2019). CLMLF
strategy, so the improvement is also limited. Fur- achieves better performance compared to several
thermore, the gating mechanism and memory net- baseline models in all three datasets. Through a
work are essentially not designed for multimodal comprehensive set of ablation experiments, case
fusion. Although they can help the model analyzes study, and visualizations, we demonstrate the ad-
the sentiment in the multimodal data by storing vantages of CLMLF for multimodal fusion1 . Our
and filtering the features in the data, it is obvious main contributions are summarized as follows:
that these methods are difficult to align and fuse
• We propose a multi-layer fusion module based
the features of text and image. Since Transformers
on Transformer-Encoder that multi-headed
have achieved great success in many fields, such
self-attention can help align and fuse token-
as natural language processing and computer vi-
level features of text and image, and it can
sion (Lin et al., 2021; Khan et al., 2021), we pro-
also benefit from the depth of MLF which im-
pose Multi-Layer Fusion (MLF) module based on
proves model abstraction ability. Experiments
Transformer-Encoder. Benefiting from the multi-
show that the proposed architecture of MLF
headed self-attention in Transformer, which can
is simple but effective.
capture the internal correlation of data vectors.
Therefore, text tokens and image patches with ex- • We propose two contrastive learning tasks
plicit and implicit relationships will have higher based on label and data, which leverages sen-
attention weight allocation to each other which timent label features and data augmentation.
means the MLF module can help align and fuse Those two contrastive learning tasks can help
the token-level text and image features better. And the model learn common features related to
MLF is a multi-layer encoder, which can help im- sentiment in multimodal data, which improve
prove the abstraction ability of the model and ob- the performance of the model.
tain deep features in multimodal data.
2 Approach
Some previous work has explored the applica- 2.1 Overview
tion of contrastive learning in the multimodal field. In this section, we will introduce CLMLF. Figure 2
Huang et al. (2021) proposes the application of illustrates the overall architecture of CLMLF model
contrastive learning in multilingual text-to-video for multimodal sentiment detection that consists of
search, and Yuan et al. (2021) applies contrastive two modules: multi-layer fusion module and multi-
learning to learn visual representations that em- task learning module. Specifically, the multi-layer
braces multimodal data. However, there is little fusion module is on the right in Figure 2, it includes
work to study the application of contrastive learn- a text-image encoder, image Transformer layer, and
ing in multimodal sentiment analysis, so we pro- text-image Transformer fusion layer modules. The
pose two contrastive learning tasks, Label Based multi-task learning module is on the left in Fig-
Contrastive Learning (LBCL) and Data Based ure 2, it includes three tasks, sentiment classifi-
Contrastive Learning (DBCL), which will help the cation, label based contrastive learning and data
model learn common features related to sentiment based contrastive learning tasks.
in multimodal data. For example, as shown in Fig-
ure 1a and Figure 1b. We can find that both tweets 2.2 Multi-Layer Fusion Module
show positive sentiment. And we also can find We use Multi-Layer Fusion module to align and
there are smiling expressions in the image of the fuse the token-level features of text and image.
two tweets which is a common feature of those 1
There are also the experimental results and analysis of
tweets. If the model can learn common features CLMLF in aspect based multimodal sentiment analysis task,
related to sentiment, it will greatly improve the which can refer to Appendix B
 SC Task LBCL Task DBCL Task MLF Module
pos
Sentiment … … Data Features
Label neg
… …
Attention & FC-Layer

…
…

…
…
…

…
FC-Layer &
pos 𝑓𝑓𝐶𝐶 𝑓𝑓1 𝑓𝑓2 … 𝑓𝑓𝑆𝑆 𝑓𝑓1+𝑁𝑁 𝑓𝑓2+𝑁𝑁 … 𝑓𝑓49+𝑁𝑁
Softmax … …
pos neg pos
Text-Image Transformer Fusion Layer
𝑡𝑡𝐶𝐶 𝑡𝑡1 𝑡𝑡2 … 𝑡𝑡𝑆𝑆 𝑖𝑖1 𝑖𝑖2 … 𝑖𝑖49

Image Transformer Layer

Share
Multi-Layer Fusion Parameter Multi-Layer Fusion
BERT 𝑖𝑖1′ 𝑖𝑖2′ … ′
𝑖𝑖49

ResNet
Was fully committed Data Augmentation Today I devote myself
to my health today to my health
[CLS] Was fully … [SEP]

Figure 2: The framework of the proposed CLMLF model

As shown on the right of Figure 2. First, we

use BERT (Devlin et al., 2019) and ResNet (He
{f1 , f2 , ..., fnt +ni } = T EM (concat(T, I)) (4)
et al., 2015) to encode the text and image to ob-
F = {f1 , f2 , ..., fnt +ni } (5)
tain the hidden representation of the text T =
{tC , t1 , t2 , ..., tS }, T ∈ Rnt ×dt and the hidden rep-
0 0
resentation of the image Ic ∈ Rpi ×pi ×di , and Ic is Where T EM means the vanilla Transformer-
the image feature map output by the last layer of Encoder of multimodal data.
convolution layer of ResNet. We transform the hid- Now, we obtain the sequence features of text and
0
den representation dimension of Ic into the same image fusion, but it is obvious that the sequence
dimension as the T . And we can get the sequence features can not be used in the classification task.
0
feature representation of the image I as follows: So we use a simple attention layer to get the multi-
0 0 modal representation R.
I = f latten(Ic WI + bI ) (1)
0 0 0 0 0
Where I = {i1 , i2 , ..., ini }, I ∈ Rni ×dt , ni = q̃i = GELU (fi W1 + b1 )W2 + b2 (6)
pi × pi . And the function of f latten means flat- q̃i
qi = exp( Pnt +ni ) (7)
ten the input vector by reshaping the first two- j=1 q̃j
dimensions into a one-dimensional. nt +ni
X
After that, we will encode the image sequence R̃ = qi fi (8)
0 i=1
features I . Here we use the vanilla Transformer-
R = GELU (R̃WR + bR ) (9)
Encoder proposed by Vaswani et al. (2017). Input
0
I into the image Transformer layer which is based where GELU is the activation function. R ∈ Rd
t

on a multi-layer Transformer-Encoder to obtain the

final encoding of image sequence features I. 2.3 Sentiment Classification
0 0 0 As shown in the SC task in Figure 2, we feed the
{i1 , i2 , ..., ini } = T EI ({i1 , i2 , ..., ini }) (2)
above multimodal representation R into the fully
I = {i1 , i2 , ..., ini } (3)
connected layer and employ the softmax function
for sentiment detection. We use the cross-entropy
Where T EI means the vanilla Transformer- loss as the classification loss and it is as follows:
Encoder of image.
In order to align and fuse the features of text and Lsc = Cross-Entropy(GELU (RWsc + bsc )) (10)
images, we concatenate the features of the text T
and the image sequence features I. We use a new
multi-layer Transformer-Encoder as a text-image
fusion layer which will align and fuse multimodal 2.4 Label Based Contrastive Learning
features. Then the fusion sequence features of text In order to let the model learn the sentiment related
and image can be obtained. It is as follows: features in the multimodal data, we use label based
contrastive learning task to help the model extract Algorithm 1 LBCL Algorithm
the sentiment related features while MLF module Require: The sentiment label is L, which is a list
fuses text and image data. As shown in the LBCL of all data in the batch, assuming that the senti-
task in Figure 2, we divide the data in each batch ment is divided into three categories: positive
into positive and negative examples according to (0), neutral (1) and negative (2); The Multi-
its sentiment label. For example, in Figure 2, for a Layer Fusion Model of M LF ; the texts are T ;
negative label of multimodal data, the data in the the images are I; C denotes length of Lc ; S
batch with the same negative labels as positive ex- denotes length of L.
amples (the square of pink color), and the data with Ensure: Label contrastive learning loss Ll-cl
no negative labels are taken as negative examples 1: initialize Lc = [L − 0, L − 1, L − 2] and Lt =
(the square of gray color). list()
The specific step can refer to Algorithm 1. The 2: for i = 1; i <= C; i + + do
meanings of specific functions in the algorithm are 3: initialize L̃t = list()
as follows: einsum means Einstein summation 4: for j = 1; j <= T ; j + + do
convention, gather means gathers values along 5: if Lc [i][j] equals 0 then
with an index, and τ represents the contrastive 6: L̃t .append(j)
learning’s temperature. The algorithm consists of 7: end if
two main steps: the first step is to generate the 8: end for
unmask label Lt according to the data labels in 9: Lt .append(L̃t )
the batch; the second step is to calculate the loss 10: end for
matrix lpn , and use the unmask label Lt and the 11: R = M LF (T, I)
loss matrix lpn to get the final loss Ll-cl , which are 12: ˜
lpn = einsum(nc, ck− > nk, [R, RT ])
the water-red elements in LBCL task on the left in 13: lpn = LogSof tmax(lpn /τ ).view(−1)
Figure 2. 14: Lcl = Lt [L[1]]
15: for q = 2; q <= S, q + + do
2.5 Data Based Contrastive Learning 16: Lcl = concat(Lcl , Lt [L[q]] + q × T )
17: end for
In order to strengthen the robustness of the model
18: Llbcl = gather(lpn , index = Lcl )/T
to the data and enhance the learning ability of the
19: return Llbcl
model to the invariant features in the data. We add
a contrastive learning task based on data augmenta-
tion which is DBCL task in Figure 2. Considering
the flexible expression of text and images. It may tion method called back-translation (Sennrich et al.,
cause the model to be too sensitive to the surface 2016; Edunov et al., 2018; Xie et al., 2020), which
features of data, rather than focus on fusing the in- refers to the procedure of translating an existing
variant features in text and images, that is, effective text x in language E into another language C and
features. Sentiment related features should exist in then translating it back into E to obtain an aug-
these effective features, because the true meaning mented text x. As observed by Yu et al. (2018),
of the meaning user wants to express should not back-translation can generate diverse paraphrases
change with the changes in text and images. For while preserving the semantics of the original sen-
example, both "I had ice cream today. I was very tences. So we use back-translation to construct
happy" and "I’m very happy today because I ate positive examples of text in contrastive learning.
ice cream" express positive sentiment. The key- For image augmentation, we use a method called
word "happy" has not changed which means the RandAugment (Cubuk et al., 2020), which is in-
happy is an effective feature, but some other words spired by AutoAugment (Cubuk et al., 2018). Au-
have changed greatly. The data based contrastive toAugment uses a search method to combine all
learning can force the model learning the effective transformations to find a good augmentation strat-
features in the data, which is more conducive to the egy. In RandAugment, it does not use search, but
model to learn the features related to sentiment in instead uniformly samples from the same set of
the data. Algorithm 2 describes the process of data augmentation transformations. In other words, Ran-
based contrastive learning. dAugment is simpler and requires no labeled data
Specifically, as for text, we use a data augmenta- as there is no need to search for optimal policies.
Algorithm 2 DBCL Algorithm method. We use the Bert-base6 and ResNet-507 as
Require: The Multi-Layer Fusion Model of the text and image encoder in Multi-Layer Fusion
M LF ; the texts are T ; the images are I; BT module. The batch size is set to 32, 64 and 128
means back-translation and RA means Ran- for MVSA-Single, MVSA-Multiple and HFM. We
dAugment; S denotes of batch size. use AdamW optimizer. The  is 1e-8 and β is (0.9,
Ensure: Data contrastive learning loss Ld-cl 0.999). The learning rate is set to 2e-5. Both λlbcl
1: R = M LF (T, I) and λdbcl are set to 1.0 in Equation 11 during train-
2: Rau = M LF (BT (T ), RA(I)) ing. For the number of layers of MLF, please refer
3: lpn = einsum(nc, ck− > nk, [R, Rau T ]) to Section 4.3. And all the experiments are done
4: cl_label = arange(S) on four NVIDIA 3090 GPUs.
5: Ldbcl = Cross-Entropy(lpn /τ, cl_label)
6: return Ldbcl

3.3 Compared Methods

2.6 Model Training
The label contrastive loss or data contrastive loss
can be simply added to the total loss as a regular- We compare our model with the unimodal senti-
ization. Can be combined like follows: ment models and the multimodal baseline models.

L = Lsc + λlbcl Llbcl + λdbcl Ldbcl (11) Unimodal Baselines: For text modality,
CNN (Kim, 2014) and Bi-LSTM (Zhou et al., 2016)
where λlbcl and λdbcl are coefficients to balance are well-known models for text classification tasks.
the different training losses. TGNN (Huang et al., 2019) is a text-level graph
neural network for text classification. BERT (De-
3 Experimental Setup vlin et al., 2019) is a pre-trained model for text,
and we fine-tuned on the text only. For image
3.1 Dataset
modality, OSDA (Yang et al., 2020) is an image
We demonstrate the effectiveness of our method sentiment analysis model based on multiple views.
on three public datasets which are MVSA-Single, ResNet (He et al., 2015) is pre-trained and fine-
MVSA-Multiple2 (Niu et al., 2016) and HFM3 (Cai tuned on the image only.
et al., 2019). Both datasets collect data from Twit-
ter, each text-image pair is labeled by a single sen- Multimodal Baselines: MultiSentiNet (Xu
timent. For a fair comparison, we process the orig- and Mao, 2017) is a deep semantic network
inal two MVSA datasets in the same way used in with attention for multimodal sentiment analysis.
Xu and Mao (2017), as for HFM, we adopt the HSAN (Xu, 2017) is a hierarchical semantic atten-
same data preprocessing method as that of Cai et al. tional network based on image captions for multi-
(2019). We randomly split the MVSA datasets into modal sentiment analysis. Co-MN-Hop6 (Xu et al.,
train set, validation set, and test set by using the 2018) is a co-memory network for iteratively mod-
split ratio 8:1:1. The statistics of these datasets are eling the interactions between multiple modalities.
given in Table 2. The detailed statistics of these MGNNS (Yang et al., 2021) is a multi-channel
datasets are given in Appendix A. graph neural networks with sentiment-awareness
for image-text sentiment detection. Schifanella
3.2 Implementation Details et al. (2016) concatenates different feature vectors
of different modalities as multimodal feature rep-
For the experiments of CLMLF, we use the Py-
resentation. Concat(2) means concatenating text
torch4 and HuggingFace Transformers5 (Wolf et al.,
features and image features, while Concat(3) has
2020) as the implementation of baselines and our
one more image attribute features. MMSD (Cai
2
http://mcrlab.net/research/mvsa-sentiment-analysis-on- et al., 2019) fuses text, image, and image attributes
multi-view-social-data/ with a multimodal hierarchical fusion model. Xu
3
https://github.com/headacheboy/data-of-multimodal-
sarcasm-detection
et al. (2020) proposes the D&R Net to fuse text,
4
https://pytorch.org/ image, and image attributes by constructing the
5
https://github.com/huggingface/transformers Decomposition and Relation Network.
 MVSA-Single MVSA-Multiple HFM
Modality Model Model
Acc F1 Acc F1 Acc F1
CNN 0.6819 0.5590 0.6564 0.5766 CNN 0.8003 0.7532
BiLSTM 0.7012 0.6506 0.6790 0.6790 BiLSTM 0.8190 0.7753
Text
BERT 0.7111 0.6970 0.6759 0.6624 BERT 0.8389 0.8326
TGNN 0.7034 0.6594 0.6967 0.6180
ResNet-50 0.6467 0.6155 0.6188 0.6098 ResNet-50 0.7277 0.7138
Image
OSDA 0.6675 0.6651 0.6662 0.6623 ResNet-101 0.7248 0.7122
MultiSentiNet 0.6984 0.6984 0.6886 0.6811 Concat(2) 0.8103 0.7799
HSAN 0.6988 0.6690 0.6796 0.6776 Concat(3) 0.8174 0.7874
Multimodal Co-MN-Hop6 0.7051 0.7001 0.6892 0.6883 MMSD 0.8344 0.8018
MGNNS 0.7377 0.7270 0.7249 0.6934 D&R Net 0.8402 0.8060
CLMLF 0.7533 0.7346 0.7200 0.6983 CLMLF 0.8543 0.8487

Table 1: Experimental results of different models on MVSA-Single, MVSA-Multiple and HFM datasets

Dataset Train Val Test Total multimodal sentiment analysis task which can refer
MVSA-S 3611 450 450 4511 to Appendix B for details.
MVSA-M 13624 1700 1700 17024
HFM 19816 2410 2409 24635 4.2 Ablation

Table 2: Statistics of the three datasets We further evaluate the influence of multi-layer fu-
sion module, label based contrastive learning, and
data based contrastive learning. The evaluation re-
4 Results and Analysis sults are listed in Table 3. The Result shows that
the whole CLMLF model achieves the best perfor-
4.1 Overall Result
mance among all models. We can see multi-layer
Table 1 illustrates the performance comparison of fusion module can improve the performance, which
our CLMLF model with the baseline methods. We shows that a multi-layer fusion module can fuse the
use Weighted-F1 and ACC as the evaluation met- multimodal data. On this foundation, adding the
rics for MVSA-Single and MVSA-Multiple which label and data based contrastive learning can im-
is the same as Yang et al. (2021) and use Macro-F1 prove the model performance more, which means
and ACC as the evaluation metrics for HFM. we contrastive learning can lead the model to learn
have the following observations. First of all, our common features about sentiment and lead differ-
model is competitive with the other strong baseline ent sentiment data away from each other.
models on the three datasets. Second, the multi-
modal models perform better than the unimodal 4.3 Influence of MLF Layer
models on all three datasets. What is more, we We explored the effects of different layers of
found the sentiment analysis on the image modality Transformer-Encoder on the results. As shown in
gets the worst results, this may be that the senti- Figure 3a, fix the image transformer layer and set
mental features in the image is too sparse and noisy, the text-image transformer fusion layer from 1 to
which makes it difficult for the model to obtain ef- 6. As shown in Figure 3b, fix the text-image trans-
fective features for sentiment analysis. At last, for former fusion layer and set the image transformer
simple tasks, the performance improvement of mul- layer from 1 to 3. Finally, we selected different
timodal models is limited. For example, on HFM combinations of 3-2 (which means three layers of
dataset, the improvement of CLMLF relative to text-image transformer fusion layer and two layers
BERT is less than MVSA-Single dataset that be- of image transformer layer), 4-2, and 5-1 for the
cause HFM is a binary classification task, while three datasets. This also proves that the contribu-
MVSA-Single is a three classification task. tion of text and images in the dataset is different. It
We also try to apply CLMLF to aspect based can be seen from Table 1 that CLMLF gains more
6
https://huggingface.co/bert-base-uncased from the text than images in HFM dataset. There-
7
https://pytorch.org/vision/stable/models.html fore, in MLF module, the layers of transformer
 MVSA-Single MVSA-Multiple HFM
Model
Acc F1 Acc F1 Acc F1
BERT 0.7111 0.6970 0.6759 0.6624 0.8389 0.8326
ResNet-50 0.6467 0.6155 0.6188 0.6098 0.7277 0.7138
+MLF 0.7111 0.7101 0.7059 0.6849 0.8414 0.8355
+MLF, LBCL 0.7378 0.7291 0.7112 0.6863 0.8489 0.8446
+MLF, DBCL 0.7356 0.7276 0.7153 0.6832 0.8468 0.8422
CLMLF 0.7533 0.7346 0.7200 0.6983 0.8543 0.8487

Table 3: Ablation results of our CLMLF

related to text are more than images. Image Text CLMLF BERT
Why are you feeling
despondent? Take the Positive Neutral
0.850 0.850

0.825 0.825

0.800 0.800
quiz:
0.775 MVSA-S MVSA-S
0.775
Thx for taking me to
MVSA-M MVSA-M
HFM HFM
0.750 0.750
Positive Negative
0.725 0.725 get cheap slushies ?
0.700
0.700
0.675
1 2 3 4 5 6 1 2 3 Car rolls over to avoid
real estate sign on Negative Neutral
(a) The text-image Trans- (b) The image Transformer
Burlington Skyway.
former fusion layer layer

Figure 3: Experimental results of different layer of Figure 4: Example of misclassified by BERT and cor-
multi-layer fusion module. The solid line indicates the rectly classified by CLMLF
accuracy and the dotted line indicates the F1. The x-
axis represents the number of layers of the transformer
text and image features. In particular, for Figure 5b,
although "lady" only shows half of the face in the
4.4 Case Study figure, the model still aligns the text and the im-
To further demonstrate the effectiveness of our age very accurately. These indicate that the model
model, we give a case study. We compare the aligns the text and image features at token-level
sentiment label predicted based on CLMLF and according to our assumptions.
BERT. As shown in Figure 4, We can find that if
we only consider the sentiment of the text, it is
difficult to correctly obtain the user’s sentimental
tendency. For example, for the first data in Figure 4,
the meaning of the text is to refer to the image, and
the image expresses a positive meaning. for the (a) The fishing is a little slow (b) Kimmy, you’re one
but the flowers are vibrant blessed lady!
second data, if we only observe the text, we find and beautiful.
that it may express negative sentiments. If add the
image, we find that it is just a joke and actually
expresses positive sentiment.

4.5 Visualization
Attention Visualization: We visualize the atten- (c) Martha said for Valen- (d) It is truly a hilarious,
tion weight of the first head of the Transformer- tine’s Day she wanted a heart light-hearted read that is a
Encoder in the last layer of the Multi-Layer Fusion shaped pancake for lunch. treasure on anyone’s book-
shelf.
module. The result of the attention visualization is
shown in Figure 5. We can see that for a given key- Figure 5: Attention visualization of some multimodal
word, The model can find the target from the image sentiment data examples
very well and give it more attention weight. This
shows that the model aligns the words in the text Cluster Visualization: In order to verify that
with the patch area of the image at a token-level, our proposed contrastive learning tasks can help
which plays an important role in the model to fuse the model to learn common features related to senti-
ment in multimodal data, we conducted a visualiza- effectively model the intra-modality interactions
tion experiment on the MVSA-Single dataset. The including aspect-text and aspect-image alignments,
data feature vector of the last layer of the model is and the inter-modality interactions. MVAN (Yang
visualized by dimensionality reduction. We use the et al., 2020) applies interactive learning of text and
TSNE dimensionality reduction algorithm to obtain image features through the attention memory net-
a 2-dimensional feature vector and visualize it, as work module, and the multimodal feature fusion
shown in Figure 6, Figure 6a is the visualization of module is constructed by using a multi-layer per-
the [CLS] of the Bert-base model, and Figure 6b ceptron and a stacking-pooling module. Yang et al.
shows the visualization of the fusion result output (2021) uses multi-channel graph neural networks
from the CLMLF model. From the figure, we can with sentiment-awareness which is built based on
see that after adding contrastive learning, the dis- the global characteristics of the dataset for multi-
tance between positive sentiment and negative sen- modal sentiment analysis.
timent in the vector space is greater, and the degree
of data aggregation is more obvious. This shows 5.2 Contrastive Learning
that the model distinguishes these data in vector Self-supervised learning attracts many researchers
space according to common features existing in the for its soaring performance on representation learn-
same sentimental data. Because the number of neu- ing in the last several years (Liu et al., 2021; Jing
tral sentiment data is relatively small, among the and Tian, 2020; Jaiswal et al., 2021). Many models
visualization results of the two models, CLMLF’s based on contrastive learning have been proposed
visualization results obviously gather the neutral in both natural language processing and computer
data together, rather than scattered in the vector vision fields. ConSERT (Yan et al., 2021), Sim-
space like Bert. All these indicate that adding con- CSE (Gao et al., 2021), CLEAR(Wu et al., 2020)
trast learning can help the model to learn common proposed the application of contrastive learning in
features related to sentiment which can improve the field of natural language processing. MoCo (He
the performance of the model. et al., 2020), SimCLR (Chen et al., 2020), Sim-
Siam (Chen and He, 2021), CLIP (Radford et al.,
2021) proposed the application of contrastive learn-
ing in the field of computer vision, and they also
have achieved good results in zero-shot learning
and few-shot learning. Recently, contrastive learn-
ing has been more and more widely used in the field
of multimodality. Huang et al. (2021) uses intra-
modal, inter-modal, and cross-lingual contrastive
(a) BERT (b) Contrastive Learning learning which can significantly improves the per-
Figure 6: Cluster visualization of MVSA-Single formance of video search. Yuan et al. (2021) ex-
ploits intrinsic data properties within each modality
and semantic information from cross-modal corre-
5 Related Work lation simultaneously, hence improving the quality
5.1 Multimodal Sentiment Analysis of learned visual representations.
Compared with the above works, we focus on
In recent years, deep learning models have how to align and fuse the token-level features and
achieved promising results for multimodal senti- learn the common features related to sentiment to
ment analysis. MultiSentiNet (Xu and Mao, 2017) further improve the performance of model.
and HSAN (Xu, 2017) use LSTM and CNN to
encode texts and images to get hidden represen- 6 Conclusion and Future Work
tations, then concatenate texts and images hid-
den representations to fuse multimodal features. In this paper, we propose a contrastive learning
CoMN (Xu et al., 2018) uses a co-memory net- and multi-layer fusion method for multimodal sen-
work to iteratively model the interactions between timent detection. Compared with previous works,
visual contents and textual words for multimodal our proposed MLF module performs multimodal
sentiment analysis. Yu et al. (2019) proposes an feature fusion from the fine-grained token-level,
aspect sensitive attention and fusion network to which is more conducive to the fusion of local fea-
tures of text and image. At the same time, we Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
design learning tasks based on contrastive learning Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language under-
to help the model learn sentiment related features in
standing. In Proceedings of the 2019 Conference
the multimodal data and improve the ability of the of the North American Chapter of the Association
model to extract and fuse features of multimodal for Computational Linguistics: Human Language
data. The experimental results on public datasets Technologies, Volume 1 (Long and Short Papers),
demonstrate that our proposed model is competitive pages 4171–4186, Minneapolis, Minnesota. Associ-
ation for Computational Linguistics.
with strong baseline models. Especially through
visualization, the contrastive learning tasks and Sergey Edunov, Myle Ott, Michael Auli, and David
multi-layer fusion module we proposed can be ver- Grangier. 2018. Understanding back-translation at
ified with intuitive interpretations. In future work, scale. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing,
we will incorporate other modalities such as audio
pages 489–500, Brussels, Belgium. Association for
into the sentiment detection task. Computational Linguistics.

Acknowledgements Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.

SimCSE: Simple contrastive learning of sentence
We are grateful to the anonymous reviewers, area embeddings. In Proceedings of the 2021 Conference
chair and Program Committee for their insightful on Empirical Methods in Natural Language Process-
comments and suggestions. The work of this pa- ing, pages 6894–6910, Online and Punta Cana, Do-
minican Republic. Association for Computational
per is funded by the project of National key re- Linguistics.
search and development program of China (No.
2020YFB1406902). Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and
Ross Girshick. 2020. Momentum contrast for unsu-
pervised visual representation learning. In Proceed-
References ings of the IEEE/CVF Conference on Computer Vi-
sion and Pattern Recognition, pages 9729–9738.
Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019. Multi-
modal sarcasm detection in twitter with hierarchical Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian
fusion model. In Proceedings of the 57th Annual Sun. 2015. Deep residual learning for image recog-
Meeting of the Association for Computational Lin- nition. corr abs/1512.03385 (2015).
guistics, pages 2506–2515.
Lianzhe Huang, Dehong Ma, Sujian Li, Xiaodong
Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Zhang, and Houfeng Wang. 2019. Text level graph
Yang. 2017. Recurrent attention network on mem- neural network for text classification. In Proceed-
ory for aspect sentiment analysis. In Proceedings of ings of the 2019 Conference on Empirical Methods
the 2017 conference on empirical methods in natural in Natural Language Processing and the 9th Inter-
language processing, pages 452–461. national Joint Conference on Natural Language Pro-
cessing (EMNLP-IJCNLP), pages 3444–3450, Hong
Ting Chen, Simon Kornblith, Mohammad Norouzi, Kong, China. Association for Computational Lin-
and Geoffrey Hinton. 2020. A simple framework for guistics.
contrastive learning of visual representations. In In-
ternational conference on machine learning, pages
Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham
1597–1607. PMLR.
Neubig, Florian Metze, and Alexander Hauptmann.
Xinlei Chen and Kaiming He. 2021. Exploring simple 2021. Multilingual multimodal pre-training for zero-
siamese representation learning. In Proceedings of shot cross-lingual transfer of vision-language mod-
the IEEE/CVF Conference on Computer Vision and els. In Proceedings of the 2021 Conference of the
Pattern Recognition, pages 15750–15758. North American Chapter of the Association for Com-
putational Linguistics: Human Language Technolo-
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay gies, Online.
Vasudevan, and Quoc V Le. 2018. Autoaugment:
Learning augmentation policies from data. arXiv Ashish Jaiswal, Ashwin Ramesh Babu, Moham-
preprint arXiv:1805.09501. mad Zaki Zadeh, Debapriya Banerjee, and Fillia
Makedon. 2021. A survey on contrastive self-
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and supervised learning. Technologies, 9(1):2.
Quoc V Le. 2020. Randaugment: Practical au-
tomated data augmentation with a reduced search Longlong Jing and Yingli Tian. 2020. Self-supervised
space. In Proceedings of the IEEE/CVF Conference visual feature learning with deep neural networks: A
on Computer Vision and Pattern Recognition Work- survey. IEEE transactions on pattern analysis and
shops, pages 702–703. machine intelligence.
Ramandeep Kaur and Sandeep Kautish. 2019. Mul- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
timodal sentiment analysis: A survey and compari- Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz
son. International Journal of Service Science, Man- Kaiser, and Illia Polosukhin. 2017. Attention is all
agement, Engineering, and Technology (IJSSMET), you need. In Advances in neural information pro-
10(2):38–58. cessing systems, pages 5998–6008.
Salman Khan, Muzammal Naseer, Munawar Hayat, Yequan Wang, Minlie Huang, Xiaoyan Zhu, and
Syed Waqas Zamir, Fahad Shahbaz Khan, and Li Zhao. 2016. Attention-based lstm for aspect-
Mubarak Shah. 2021. Transformers in vision: A sur- level sentiment classification. In Proceedings of the
vey. arXiv preprint arXiv:2101.01169. 2016 conference on empirical methods in natural
language processing, pages 606–615.
Yoon Kim. 2014. Convolutional neural networks
for sentence classification. In Proceedings of the Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
2014 Conference on Empirical Methods in Natural Chaumond, Clement Delangue, Anthony Moi, Pier-
Language Processing (EMNLP), pages 1746–1751, ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-
Doha, Qatar. Association for Computational Lin- icz, Joe Davison, Sam Shleifer, Patrick von Platen,
guistics. Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,
Ayush Kumar and Jithendra Vepa. 2020. Gated mecha- Quentin Lhoest, and Alexander M. Rush. 2020.
nism for attention based multi modal sentiment anal- Transformers: State-of-the-art natural language pro-
ysis. In ICASSP 2020-2020 IEEE International Con- cessing. In Proceedings of the 2020 Conference on
ference on Acoustics, Speech and Signal Processing Empirical Methods in Natural Language Processing:
(ICASSP), pages 4477–4481. IEEE. System Demonstrations, pages 38–45, Online. Asso-
Tianyang Lin, Yuxin Wang, Xiangyang Liu, and ciation for Computational Linguistics.
Xipeng Qiu. 2021. A survey of transformers. arXiv Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian
preprint arXiv:2106.04554. Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Con-
trastive learning for sentence representation. arXiv
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian,
preprint arXiv:2012.15466.
Zhaoyu Wang, Jing Zhang, and Jie Tang. 2021.
Self-supervised learning: Generative or contrastive. Qizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong,
IEEE Transactions on Knowledge and Data Engi- and Quoc Le. 2020. Unsupervised data augmenta-
neering. tion for consistency training. In Advances in Neural
Information Processing Systems 33: Annual Con-
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang,
ference on Neural Information Processing Systems
and Heng Ji. 2018. Visual attention model for name
2020, NeurIPS 2020, December 6-12, 2020, virtual.
tagging in multimodal social media. In Proceedings
of the 56th Annual Meeting of the Association for Nan Xu. 2017. Analyzing multimodal public sentiment
Computational Linguistics (Volume 1: Long Papers), based on hierarchical semantic attentional network.
pages 1990–1999. In 2017 IEEE International Conference on Intelli-
gence and Security Informatics (ISI), pages 152–154.
Teng Niu, Shiai Zhu, Lei Pang, and Abdulmotaleb IEEE.
El Saddik. 2016. Sentiment analysis on multi-view
social data. In International Conference on Multi- Nan Xu and Wenji Mao. 2017. Multisentinet: A deep
media Modeling, pages 15–27. Springer. semantic network for multimodal sentiment analy-
sis. In Proceedings of the 2017 ACM on Conference
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya on Information and Knowledge Management, pages
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish 2399–2402.
Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al. 2021. Learning transferable visual models Nan Xu, Wenji Mao, and Guandan Chen. 2018. A co-
from natural language supervision. In International memory network for multimodal sentiment analysis.
Conference on Machine Learning, pages 8748–8763. In The 41st international ACM SIGIR conference
PMLR. on research & development in information retrieval,
pages 929–932.
Rossano Schifanella, Paloma de Juan, Joel Tetreault,
and Liangliang Cao. 2016. Detecting sarcasm in Nan Xu, Wenji Mao, and Guandan Chen. 2019. Multi-
multimodal social platforms. In Proceedings of the interactive memory network for aspect based mul-
24th ACM international conference on Multimedia, timodal sentiment analysis. In Proceedings of
pages 1136–1145. the AAAI Conference on Artificial Intelligence, vol-
ume 33, pages 371–378.
Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Improving neural machine translation mod- Nan Xu, Zhixiong Zeng, and Wenji Mao. 2020. Rea-
els with monolingual data. In Proceedings of the soning with multimodal sarcastic tweets via mod-
54th Annual Meeting of the Association for Compu- eling cross-modality contrast and semantic associ-
tational Linguistics (Volume 1: Long Papers), pages ation. In Proceedings of the 58th Annual Meet-
86–96, Berlin, Germany. Association for Computa- ing of the Association for Computational Linguistics,
tional Linguistics. pages 3777–3786.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li,
Wei Wu, and Weiran Xu. 2021. ConSERT: A con- Hongwei Hao, and Bo Xu. 2016. Attention-based
trastive framework for self-supervised sentence rep- bidirectional long short-term memory networks for
resentation transfer. In Proceedings of the 59th An- relation classification. In Proceedings of the 54th
nual Meeting of the Association for Computational Annual Meeting of the Association for Computa-
Linguistics and the 11th International Joint Confer- tional Linguistics (Volume 2: Short Papers), pages
ence on Natural Language Processing (Volume 1: 207–212, Berlin, Germany. Association for Compu-
Long Papers), pages 5065–5075, Online. Associa- tational Linguistics.
tion for Computational Linguistics.
Xiaocui Yang, Shi Feng, Daling Wang, and Yifei
Zhang. 2020. Image-text multimodal emotion clas-
sification via multi-view attentional network. IEEE
Transactions on Multimedia.
Xiaocui Yang, Shi Feng, Yifei Zhang, and Daling
Wang. 2021. Multimodal sentiment detection based
on multi-channel graph neural networks. In Pro-
ceedings of the 59th Annual Meeting of the Associa-
tion for Computational Linguistics and the 11th In-
ternational Joint Conference on Natural Language
Processing (Volume 1: Long Papers), pages 328–
339.
Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui
Zhao, Kai Chen, Mohammad Norouzi, and Quoc V.
Le. 2018. Qanet: Combining local convolution with
global self-attention for reading comprehension. In
6th International Conference on Learning Represen-
tations, ICLR 2018, Vancouver, BC, Canada, April
30 - May 3, 2018, Conference Track Proceedings.
OpenReview.net.
Jianfei Yu and Jing Jiang. 2019. Adapting bert for
target-oriented multimodal sentiment classification.
In Proceedings of the Twenty-Eighth International
Joint Conference on Artificial Intelligence, IJCAI-
19, pages 5408–5414. International Joint Confer-
ences on Artificial Intelligence Organization.
Jianfei Yu, Jing Jiang, and Rui Xia. 2019. Entity-
sensitive attention and fusion network for
entity-level multimodal sentiment classification.
IEEE/ACM Transactions on Audio, Speech, and
Language Processing, 28:429–439.
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin
Wang, Michael Maire, Ajinkya Kale, and Baldo Fai-
eta. 2021. Multimodal contrastive training for vi-
sual representation learning. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pat-
tern Recognition, pages 6995–7004.
Lin Yue, Weitong Chen, Xue Li, Wanli Zuo, and Ming-
hao Yin. 2019. A survey of sentiment analysis in
social media. Knowledge and Information Systems,
60(2):617–663.
Lei Zhang, Shuai Wang, and Bing Liu. 2018a. Deep
learning for sentiment analysis: A survey. Wiley
Interdisciplinary Reviews: Data Mining and Knowl-
edge Discovery, 8(4):e1253.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang.
2018b. Adaptive co-attention network for named en-
tity recognition in tweets. In Thirty-Second AAAI
Conference on Artificial Intelligence.
A Dataset Statistics Dataset Train Val Test Total
Multi-ZOL 22743 2843 2843 28429
The detailed statistics for the MVSA-Single, Twitter-15 3179 1122 1037 5338
MVSA-Multiple and HFM datasets are listed in Twitter-17 3562 1176 1234 5972
Table 4. We can see that HFM is a binary classifi-
cation multimodal sentiment dataset, while MVSA- Table 5: Statistics of the three datasets
Single and MVSA-Multiple are three classification
multimodal sentiment datasets.
• LSTM, a standard sentence-level LSTM
Dataset Label Train Val Test model without explicitly considering the as-
Positive 2147 268 268 pect. Therefore, this result is also the worst.
MVSA-
Neutral 376 47 47
Single • AE-LSTM (Wang et al., 2016), an attention-
Negative 1088 135 135
based LSTM for aspect-level sentiment classi-
Positive 9056 1131 1131
MVSA- fication, which uses the attention mechanism
Neutral 3528 440 440
Multiple to capture the important context information
Negative 1040 129 129
related to the aspect.
Positive 8642 959 959
HFM
Negative 11174 1451 1450 • RAM (Chen et al., 2017) is a memory based
model, which builds memory on the hidden
Table 4: Number of data for each sentiment category in
states of a Bi-LSTM and generates aspect rep-
each dataset
resentation based on a Bi-LSTM. Then pays
multiple attentions on the memory to pick up
important information to predict the final sen-
B Aspect Based Multimodal Sentiment
timent, by combining the features from differ-
B.1 Experimental Setup ent attentions non-linearly.
Because CLMLF is designed for sentence-level • MIMN (Xu et al., 2019), the multimodal ap-
multimodal sentiment analysis, we have made proach for aspect-level sentiment classifica-
some minor changes to the input of CLMLF model tion task, which adopts multi-hop memory
to adapt to aspect based multimodal sentiment network to model the interactive attention be-
analysis. We change the input form from "[CLS] tween the aspect word, the textual context,
sentence [SEP]" to "[CLS] sentence [SEP] aspect and the visual context.
[SEP]" and no change the input of image modality.
Although this change is very simple, CLMLF can • TomBERT (Yu and Jiang, 2019), a multi-
work well in aspect based multimodal sentiment modal model which borrow the idea from self-
analysis tasks and achieves good results. attention and design a target attention mech-
We use three aspect based multimodal sentiment anism to perform target-image matching to
dataset: Multi-ZOL8 (Xu et al., 2019), Twitter- derive target sensitive visual representations.
15 (Zhang et al., 2018b) and Twitter-179 (Lu et al.,
• ESAFN (Yu et al., 2019) proposes an entity-
2018). The statistics of these datasets are given in
sensitive attention and fusion network which
Table 5. Compared with the dataset of sentence-
capture the intra-modality dynamics by lever-
level multimodal sentiment analysis, each sentence
ages an effective attention mechanism to gen-
will have a corresponding aspect attribute. Espe-
erate entity-sensitive textual and visual repre-
cially for the Multi-ZOL dataset, each data contains
sentations. And uses visual attention mecha-
multiple images. And we only randomly select one
nism to learn the entity-sensitive visual rep-
image for fusion. Although some features are lost,
resentation. Moreover, ESAFN further fuses
the experimental results show that it is improved
the textual and visual representations with a
compared with the only text modality.
bilinear interaction layer.
B.2 Results
Table 6 illustrates the performance comparison
We compare our model with other baseline models: of our CLMLF mdoel with the baseline methods.
8
https://github.com/xunan0812/MIMN We use Macro-F1 and ACC as the evaluation met-
9
https://github.com/jefferyYu/TomBERT rics for all datasets. The experimental results show
 Multi-ZOL Twitter-15 Twitter-17
Modality Model
Acc F1 Acc F1 Acc F1
LSTM 0.5892 0.5729 0.6798 0.5730 0.5592 0.5169
Text AE-LSTM 0.5958 0.5895 0.7030 0.6343 0.6167 0.5797
RAM 0.6018 0.5968 0.7068 0.6305 0.6442 0.6101
BERT 0.6959 0.6868 0.7387 0.7023 0.6848 0.6553
MIMN 0.6159 0.6051 0.7184 0.6569 0.6588 0.6299
ESAFN - - 0.7338 0.6737 0.6783 0.6422
Multimodal TomBERT - - 0.7715 0.7175 0.7034 0.6803
CLMLF 0.7452 0.7075 0.7811 0.7460 0.7098 0.6913

Table 6: Experimental results of different models on aspect based datasets

Multi-ZOL Twitter-15 Twitter-17

Model
Acc F1 Acc F1 Acc F1
BERT 0.6959 0.6868 0.7387 0.7023 0.6848 0.6553
+MLF 0.7301 0.6897 0.7424 0.7017 0.6848 0.6579
+MLF, LBCL 0.7336 0.6953 0.7715 0.7311 0.6969 0.6790
+MLF, DBCL 0.7347 0.7015 0.7445 0.6964 0.6921 0.6722
CLMLF 0.7452 0.7075 0.7811 0.7460 0.7098 0.6913

Table 7: Ablation results of CLMLF

that CLMLF can still achieve good results. We

also conducted ablation experiments, as shown in
Table 7. The experiments again proved that the
multi-layer fusion module, label based contrastive
learning task and data based contrastive task we
proposed are effective.