Deep Visual-Semantic Alignments for Generating Image Descriptions

Andrej Karpathy Li Fei-Fei

Department of Computer Science, Stanford University
{karpathy,feifeili}@cs.stanford.edu

Abstract

We present a model that generates natural language de-

scriptions of images and their regions. Our approach lever-
ages datasets of images and their sentence descriptions to
learn about the inter-modal correspondences between lan-
guage and visual data. Our alignment model is based on a
novel combination of Convolutional Neural Networks over
image regions, bidirectional Recurrent Neural Networks
over sentences, and a structured objective that aligns the
two modalities through a multimodal embedding. We then
describe a Multimodal Recurrent Neural Network architec-
ture that uses the inferred alignments to learn to generate
novel descriptions of image regions. We demonstrate that
our alignment model produces state of the art results in re-
trieval experiments on Flickr8K, Flickr30K and MSCOCO
datasets. We then show that the generated descriptions sig- Figure 1. Motivation/Concept Figure: Our model treats language
nificantly outperform retrieval baselines on both full images as a rich label space and generates descriptions of image regions.
and on a new dataset of region-level annotations.
generating dense descriptions of images (Figure 1). The
1. Introduction primary challenge towards this goal is in the design of a
model that is rich enough to simultaneously reason about
A quick glance at an image is sufficient for a human to
contents of images and their representation in the domain
point out and describe an immense amount of details about
of natural language. Additionally, the model should be free
the visual scene [14]. However, this remarkable ability has
of assumptions about specific hard-coded templates, rules
proven to be an elusive task for our visual recognition mod-
or categories and instead rely on learning from the training
els. The majority of previous work in visual recognition
data. The second, practical challenge is that datasets of im-
has focused on labeling images with a fixed set of visual
age captions are available in large quantities on the internet
categories and great progress has been achieved in these en-
[21, 58, 37], but these descriptions multiplex mentions of
deavors [45, 11]. However, while closed vocabularies of vi-
several entities whose locations in the images are unknown.
sual concepts constitute a convenient modeling assumption,
they are vastly restrictive when compared to the enormous Our core insight is that we can leverage these large image-
amount of rich descriptions that a human can compose. sentence datasets by treating the sentences as weak labels,
Some pioneering approaches that address the challenge of in which contiguous segments of words correspond to some
generating image descriptions have been developed [29, particular, but unknown location in the image. Our ap-
13]. However, these models often rely on hard-coded visual proach is to infer these alignments and use them to learn
concepts and sentence templates, which imposes limits on a generative model of descriptions. Concretely, our contri-
their variety. Moreover, the focus of these works has been butions are twofold:
on reducing complex visual scenes into a single sentence,
We develop a deep neural network model that in-
which we consider to be an unnecessary restriction.
fers the latent alignment between segments of sen-
In this work, we strive to take a step towards the goal of tences and the region of the image that they describe.
 Our model associates the two modalities through a sual domain [27, 39, 60, 36]. Our approach is inspired by
common, multimodal embedding space and a struc- Frome et al. [16] who associate words and images through
tured objective. We validate the effectiveness of this a semantic embedding. More closely related is the work of
approach on image-sentence retrieval experiments in Karpathy et al. [24], who decompose images and sentences
which we surpass the state-of-the-art. into fragments and infer their inter-modal alignment using a
ranking objective. In contrast to their model which is based
We introduce a multimodal Recurrent Neural Network
on grounding dependency tree relations, our model aligns
architecture that takes an input image and generates
contiguous segments of sentences which are more mean-
its description in text. Our experiments show that the
ingful, interpretable, and not fixed in length.
generated sentences significantly outperform retrieval-
based baselines, and produce sensible qualitative pre- Neural networks in visual and language domains. Mul-
dictions. We then train the model on the inferred cor- tiple approaches have been developed for representing im-
respondences and evaluate its performance on a new ages and words in higher-level representations. On the im-
dataset of region-level annotations. age side, Convolutional Neural Networks (CNNs) [32, 28]
1 have recently emerged as a powerful class of models for
We make code, data and annotations publicly available.
image classification and object detection [45]. On the sen-
2. Related Work tence side, our work takes advantage of pretrained word
vectors [41, 22, 3] to obtain low-dimensional representa-
Dense image annotations. Our work shares the high-level tions of words. Finally, Recurrent Neural Networks have
goal of densely annotating the contents of images with been previously used in language modeling [40, 50], but we
many works before us. Barnard et al. [2] and Socher et additionally condition these models on images.
al. [48] studied the multimodal correspondence between
words and images to annotate segments of images. Sev- 3. Our Model
eral works [34, 18, 15, 33] studied the problem of holistic Overview. The ultimate goal of our model is to generate
scene understanding in which the scene type, objects and descriptions of image regions. During training, the input
their spatial support in the image is inferred. However, the to our model is a set of images and their corresponding
focus of these works is on correctly labeling scenes, objects sentence descriptions (Figure 2). We first present a model
and regions with a fixed set of categories, while our focus is that aligns sentence snippets to the visual regions that they
on richer and higher-level descriptions of regions. describe through a multimodal embedding. We then treat
Generating descriptions. The task of describing images these correspondences as training data for a second, multi-
with sentences has also been explored. A number of ap- modal Recurrent Neural Network model that learns to gen-
proaches pose the task as a retrieval problem, where the erate the snippets.
most compatible annotation in the training set is transferred 3.1. Learning to align visual and language data
to a test image [21, 49, 13, 43, 23], or where training an- Our alignment model assumes an input dataset of images
notations are broken up and stitched together [30, 35, 31]. and their sentence descriptions. Our key insight is that sen-
Several approaches generate image captions based on fixed tences written by people make frequent references to some
templates that are filled based on the content of the image particular, but unknown location in the image. For exam-
[19, 29, 13, 55, 56, 9, 1] or generative grammars [42, 57], ple, in Figure 2, the words Tabby cat is leaning refer to
but this approach limits the variety of possible outputs. the cat, the words wooden table refer to the table, etc.
Most closely related to us, Kiros et al. [26] developed a log- We would like to infer these latent correspondences, with
bilinear model that can generate full sentence descriptions the eventual goal of later learning to generate these snippets
for images, but their model uses a fixed window context from image regions. We build on the approach of Karpathy
while our Recurrent Neural Network (RNN) model condi- et al. [24], who learn to ground dependency tree relations
tions the probability distribution over the next word in a sen- to image regions with a ranking objective. Our contribu-
tence on all previously generated words. Multiple closely tion is in the use of bidirectional recurrent neural network
related preprints appeared on Arxiv during the submission to compute word representations in the sentence, dispens-
of this work, some of which also use RNNs to generate im- ing of the need to compute dependency trees and allowing
age descriptions [38, 54, 8, 25, 12, 5]. Our RNN is simpler unbounded interactions of words and their context in the
than most of these approaches but also suffers in perfor- sentence. We also substantially simplify their objective and
mance. We quantify this comparison in our experiments. show that both modifications improve ranking performance.
Grounding natural language in images. A number of ap- We first describe neural networks that map words and image
proaches have been developed for grounding text in the vi- regions into a common, multimodal embedding. Then we
1 cs.stanford.edu/people/karpathy/deepimagesent introduce our novel objective, which learns the embedding
Figure 2. Overview of our approach. A dataset of images and their sentence descriptions is the input to our model (left). Our model first
infers the correspondences (middle, Section 3.1) and then learns to generate novel descriptions (right, Section 3.2).

representations so that semantically similar concepts across N words (encoded in a 1-of-k representation) and trans-
the two modalities occupy nearby regions of the space. forms each one into an h-dimensional vector. However, the
representation of each word is enriched by a variably-sized
3.1.1 Representing images context around that word. Using the index t = 1 . . . N to
Following prior work [29, 24], we observe that sentence de- denote the position of a word in a sentence, the precise form
scriptions make frequent references to objects and their at- of the BRNN is as follows:
tributes. Thus, we follow the method of Girshick et al. [17]
to detect objects in every image with a Region Convolu- xt = Ww It (2)
tional Neural Network (RCNN). The CNN is pre-trained on et = f (We xt + be ) (3)
ImageNet [6] and finetuned on the 200 classes of the Ima-
hft= f (et + Wf hft1 + bf ) (4)
geNet Detection Challenge [45]. Following Karpathy et al.
[24], we use the top 19 detected locations in addition to the hbt= f (et + Wb hbt+1 + bb ) (5)
whole image and compute the representations based on the st = f (Wd (hft + hbt ) + bd ). (6)
pixels Ib inside each bounding box as follows:

v = Wm [CNNc (Ib )] + bm , (1) Here, It is an indicator column vector that has a single one
at the index of the t-th word in a word vocabulary. The
weights Ww specify a word embedding matrix that we ini-
where CNN(Ib ) transforms the pixels inside bounding box
tialize with 300-dimensional word2vec [41] weights and
Ib into 4096-dimensional activations of the fully connected
keep fixed due to overfitting concerns. However, in prac-
layer immediately before the classifier. The CNN parame-
tice we find little change in final performance when these
ters c contain approximately 60 million parameters. The
vectors are trained, even from random initialization. Note
matrix Wm has dimensions h 4096, where h is the size
that the BRNN consists of two independent streams of pro-
of the multimodal embedding space (h ranges from 1000-
cessing, one moving left to right (hft ) and the other right to
1600 in our experiments). Every image is thus represented
left (hbt ) (see Figure 3 for diagram). The final h-dimensional
as a set of h-dimensional vectors {vi | i = 1 . . . 20}.
representation st for the t-th word is a function of both the
word at that location and also its surrounding context in the
3.1.2 Representing sentences sentence. Technically, every st is a function of all words in
To establish the inter-modal relationships, we would like the entire sentence, but our empirical finding is that the final
to represent the words in the sentence in the same h- word representations (st ) align most strongly to the visual
dimensional embedding space that the image regions oc- concept of the word at that location (It ).
cupy. The simplest approach might be to project every in-
We learn the parameters We , Wf , Wb , Wd and the respec-
dividual word directly into this embedding. However, this
tive biases be , bf , bb , bd . A typical size of the hidden rep-
approach does not consider any ordering and word context
resentation in our experiments ranges between 300-600 di-
information in the sentence. An extension to this idea is
mensions. We set the activation function f to the rectified
to use word bigrams, or dependency tree relations as pre-
linear unit (ReLU), which computes f : x 7 max(0, x).
viously proposed [24]. However, this still imposes an ar-
bitrary maximum size of the context window and requires
the use of Dependency Tree Parsers that might be trained on 3.1.3 Alignment objective
unrelated text corpora.
We have described the transformations that map every im-
To address these concerns, we propose to use a Bidirec- age and sentence into a set of vectors in a common h-
tional Recurrent Neural Network (BRNN) [46] to compute dimensional space. Since the supervision is at the level of
the word representations. The BRNN takes a sequence of entire images and sentences, our strategy is to formulate an
image-sentence score as a function of the individual region-
word scores. Intuitively, a sentence-image pair should have
a high matching score if its words have a confident support
in the image. The model of Karpathy et a. [24] interprets the
dot product viT st between the i-th region and t-th word as a
measure of similarity and use it to define the score between
image k and sentence l as:
XX
Skl = max(0, viT st ). (7)
tgl igk

Here, gk is the set of image fragments in image k and gl

is the set of sentence fragments in sentence l. The indices
k, l range over the images and sentences in the training set.
Together with their additional Multiple Instance Learning
Figure 3. Diagram for evaluating the image-sentence score Skl .
objective, this score carries the interpretation that a sentence Object regions are embedded with a CNN (left). Words (enriched
fragment aligns to a subset of the image regions whenever by their context) are embedded in the same multimodal space with
the dot product is positive. We found that the following a BRNN (right). Pairwise similarities are computed with inner
reformulation simplifies the model and alleviates the need products (magnitudes shown in grayscale) and finally reduced to
for additional objectives and their hyperparameters: image-sentence score with Equation 8.
X
Skl = maxigk viT st . (8)
alignment to the same region. Concretely, given a sentence
tgl
with N words and an image with M bounding boxes, we
Here, every word st aligns to the single best image region. introduce the latent alignment variables aj {1 . . . M } for
As we show in the experiments, this simplified model also j = 1 . . . N and formulate an MRF in a chain structure
leads to improvements in the final ranking performance. along the sentence as follows:
Assuming that k = l denotes a corresponding image and X X
sentence pair, the final max-margin, structured loss remains: E(a) = jU (aj ) + jB (aj , aj+1 ) (10)
j=1...N j=1...N 1
Xh X
C() = max(0, Skl Skk + 1) (9) jU (aj = t) = viT st (11)
jB (aj , aj+1 ) = 1[aj = aj+1 ].
k l
| {z } (12)
rank images
X i
+ max(0, Slk Skk + 1) . Here, is a hyperparameter that controls the affinity to-
l wards longer word phrases. This parameter allows us to
interpolate between single-word alignments ( = 0) and
| {z }
rank sentences
aligning the entire sentence to a single, maximally scoring
This objective encourages aligned image-sentences pairs to region when is large. We minimize the energy to find the
have a higher score than misaligned pairs, by a margin. best alignments a using dynamic programming. The output
of this process is a set of image regions annotated with seg-
3.1.4 Decoding text segment alignments to images ments of text. We now describe an approach for generating
novel phrases based on these correspondences.
Consider an image from the training set and its correspond-
ing sentence. We can interpret the quantity viT st as the un- 3.2. Multimodal Recurrent Neural Network for
normalized log probability of the t-th word describing any generating descriptions
of the bounding boxes in the image. However, since we are
In this section we assume an input set of images and their
ultimately interested in generating snippets of text instead
textual descriptions. These could be full images and their
of single words, we would like to align extended, contigu-
sentence descriptions, or regions and text snippets, as in-
ous sequences of words to a single bounding box. Note that
ferred in the previous section. The key challenge is in the
the nave solution that assigns each word independently to
design of a model that can predict a variable-sized sequence
the highest-scoring region is insufficient because it leads to
of outputs given an image. In previously developed lan-
words getting scattered inconsistently to different regions.
guage models based on Recurrent Neural Networks (RNNs)
To address this issue, we treat the true alignments as latent [40, 50, 10], this is achieved by defining a probability distri-
variables in a Markov Random Field (MRF) where the bi- bution of the next word in a sequence given the current word
nary interactions between neighboring words encourage an and context from previous time steps. We explore a simple
but effective extension that additionally conditions the gen-
erative process on the content of an input image. More for-
mally, during training our Multimodal RNN takes the image
pixels I and a sequence of input vectors (x1 , . . . , xT ). It
then computes a sequence of hidden states (h1 , . . . , ht ) and
a sequence of outputs (y1 , . . . , yt ) by iterating the following
recurrence relation for t = 1 to T :
bv = Whi [CNNc (I)] (13) Figure 4. Diagram of our multimodal Recurrent Neural Network
ht = f (Whx xt + Whh ht1 + bh + 1(t = 1)  bv ) (14) generative model. The RNN takes a word, the context from previ-
ous time steps and defines a distribution over the next word in the
yt = sof tmax(Woh ht + bo ). (15) sentence. The RNN is conditioned on the image information at the
In the equations above, Whi , Whx , Whh , Woh , xi and bh , bo first time step. START and END are special tokens.
are learnable parameters, and CNNc (I) is the last layer of
a CNN. The output vector yt holds the (unnormalized) log 4. Experiments
probabilities of words in the dictionary and one additional Datasets. We use the Flickr8K [21], Flickr30K [58] and
dimension for a special END token. Note that we provide MSCOCO [37] datasets in our experiments. These datasets
the image context vector bv to the RNN only at the first contain 8,000, 31,000 and 123,000 images respectively
iteration, which we found to work better than at each time and each is annotated with 5 sentences using Amazon
step. In practice we also found that it can help to also pass Mechanical Turk. For Flickr8K and Flickr30K, we use
both bv , (Whx xt ) through the activation function. A typical 1,000 images for validation, 1,000 for testing and the rest
size of the hidden layer of the RNN is 512 neurons. for training (consistent with [21, 24]). For MSCOCO we
use 5,000 images for both validation and testing.
RNN training. The RNN is trained to combine a word (xt ),
the previous context (ht1 ) to predict the next word (yt ). Data Preprocessing. We convert all sentences to lower-
We condition the RNNs predictions on the image informa- case, discard non-alphanumeric characters. We filter words
tion (bv ) via bias interactions on the first step. The training to those that occur at least 5 times in the training set,
proceeds as follows (refer to Figure 4): We set h0 = ~0, x1 to which results in 2538, 7414, and 8791 words for Flickr8k,
a special START vector, and the desired label y1 as the first Flickr30K, and MSCOCO datasets respectively.
word in the sequence. Analogously, we set x2 to the word 4.1. Image-Sentence Alignment Evaluation
vector of the first word and expect the network to predict
We first investigate the quality of the inferred text and image
the second word, etc. Finally, on the last step when xT rep-
alignments with ranking experiments. We consider a with-
resents the last word, the target label is set to a special END
held set of images and sentences and retrieve items in one
token. The cost function is to maximize the log probability
modality given a query from the other by sorting based on
assigned to the target labels (i.e. Softmax classifier).
the image-sentence score Skl (Section 3.1.3). We report the
RNN at test time. To predict a sentence, we compute the median rank of the closest ground truth result in the list and
image representation bv , set h0 = 0, x1 to the START vec- Recall@K, which measures the fraction of times a correct
tor and compute the distribution over the first word y1 . We item was found among the top K results. The result of these
sample a word from the distribution (or pick the argmax), experiments can be found in Table 1, and example retrievals
set its embedding vector as x2 , and repeat this process until in Figure 5. We now highlight some of the takeaways.
the END token is generated. In practice we found that beam
Our full model outperforms previous work. First, our
search (e.g. beam size 7) can improve results.
full model (Our model: BRNN) outperforms Socher et
3.3. Optimization al. [49] who trained with a similar loss but used a single
image representation and a Recursive Neural Network over
We use SGD with mini-batches of 100 image-sentence pairs
the sentence. A similar loss was adopted by Kiros et al.
and momentum of 0.9 to optimize the alignment model. We
[25], who use an LSTM [20] to encode sentences. We list
cross-validate the learning rate and the weight decay. We
their performance with a CNN that is equivalent in power
also use dropout regularization in all layers except in the
(AlexNet [28]) to the one used in this work, though simi-
recurrent layers [59] and clip gradients elementwise at 5
lar to [54] they outperform our model with a more powerful
(important). The generative RNN is more difficult to op-
CNN (VGGNet [47], GoogLeNet [51]). DeFrag are the
timize, party due to the word frequency disparity between
results reported by Karpathy et al. [24]. Since we use dif-
rare words and common words (e.g. a or the END token).
ferent word vectors, dropout for regularization and different
We achieved the best results using RMSprop [52], which is
cross-validation ranges and larger embedding sizes, we re-
an adaptive step size method that scales the update of each
implemented their loss for a fair comparison (Our imple-
weight by a running average of its gradient norm.
 Image Annotation Image Search
Model R@1 R@5 R@10 Med r R@1 R@5 R@10 Med r
Flickr30K
SDT-RNN (Socher et al. [49]) 9.6 29.8 41.1 16 8.9 29.8 41.1 16
Kiros et al. [25] 14.8 39.2 50.9 10 11.8 34.0 46.3 13
Mao et al. [38] 18.4 40.2 50.9 10 12.6 31.2 41.5 16
Donahue et al. [8] 17.5 40.3 50.8 9 - - - -
DeFrag (Karpathy et al. [24]) 14.2 37.7 51.3 10 10.2 30.8 44.2 14
Our implementation of DeFrag [24] 19.2 44.5 58.0 6.0 12.9 35.4 47.5 10.8
Our model: DepTree edges 20.0 46.6 59.4 5.4 15.0 36.5 48.2 10.4
Our model: BRNN 22.2 48.2 61.4 4.8 15.2 37.7 50.5 9.2
Vinyals et al. [54] (more powerful CNN) 23 - 63 5 17 - 57 8
MSCOCO
Our model: 1K test images 38.4 69.9 80.5 1.0 27.4 60.2 74.8 3.0
Our model: 5K test images 16.5 39.2 52.0 9.0 10.7 29.6 42.2 14.0
Table 1. Image-Sentence ranking experiment results. R@K is Recall@K (high is good). Med r is the median rank (low is good). In the
results for our models, we take the top 5 validation set models, evaluate each independently on the test set and then report the average
performance. The standard deviations on the recall values range from approximately 0.5 to 1.0.

Figure 5. Example alignments predicted by our model. For every test image above, we retrieve the most compatible test sentence and
visualize the highest-scoring region for each word (before MRF smoothing described in Section 3.1.4) and the associated scores (viT st ).
We hide the alignments of low-scoring words to reduce clutter. We assign each region an arbitrary color.

mentation of DeFrag). Compared to other work that uses Therefore, we report results on a subset of 1,000 images
AlexNets, our full model shows consistent improvement. and the full set of 5,000 test images for future comparisons.
Our simpler cost function improves performance. We Note that the 5000 images numbers are lower since Re-
strive to better understand the source of our performance. call@K is a function of test set size.
First, we removed the BRNN and used dependency tree re- Qualitative. As can be seen from example groundings in
lations exactly as described in Karpathy et al. [24] (Our Figure 5, the model discovers interpretable visual-semantic
model: DepTree edges). The only difference between this correspondences, even for small or relatively rare objects
model and Our reimplementation of DeFrag is the new, such as an accordion. These would be likely missed by
simpler cost function introduced in Section 3.1.3. We see models that only reason about full images.
that our formulation shows consistent improvements.
Learned region and word vector magnitudes. An ap-
BRNN outperforms dependency tree relations. Further- pealing feature of our model is that it learns to modulate
more, when we replace the dependency tree relations with the magnitude of the region and word embeddings. Due
the BRNN we observe additional performance improve- to their inner product interaction, we observe that repre-
ments. Since the dependency relations were shown to work sentations of visually discriminative words such as kayak-
better than single words and bigrams [24], this suggests that ing, pumpkins have embedding vectors with higher mag-
the BRNN is taking advantage of contexts longer than two nitudes, which in turn translates to a higher influence on
words. Furthermore, our method does not rely on extracting the image-sentence score. Conversely, stop words such as
a Dependency Tree and instead uses the raw words directly. now, simply, actually, but are mapped near the origin,
MSCOCO results for future comparisons. We are not which reduces their influence. See more analysis in supple-
aware of other published ranking results on MSCOCO. mentary material.
 Flickr8K Flickr30K MSCOCO 2014
Model B-1 B-2 B-3 B-4 B-1 B-2 B-3 B-4 B-1 B-2 B-3 B-4 METEOR CIDEr
Nearest Neighbor 48.0 28.1 16.6 10.0 15.7 38.3
Mao et al. [38] 58 28 23 55 24 20
Google NIC [54] 63 41 27 66.3 42.3 27.7 18.3 66.6 46.1 32.9 24.6
LRCN [8] 58.8 39.1 25.1 16.5 62.8 44.2 30.4
MS Research [12] 21.1 20.7
Chen and Zitnick [5] 14.1 12.6 19.0 20.4
Our model 57.9 38.3 24.5 16.0 57.3 36.9 24.0 15.7 62.5 45.0 32.1 23.0 19.5 66.0
Table 2. Evaluation of full image predictions on 1,000 test images. B-n is BLEU score that uses up to n-grams. High is good in all columns.
For future comparisons, our METEOR/CIDEr Flickr8K scores are 16.7/31.8 and the Flickr30K scores are 15.3/24.7.

Figure 6. Example sentences generated by the multimodal RNN for test images. We provide many more examples on our project page.

4.2. Generated Descriptions: Fulframe evaluation Here, we annotate each test image with a sentence of the
We now evaluate the ability of our RNN model to describe most similar training set image as determined by L2 norm
images and regions. We first trained our Multimodal RNN over VGGNet [47] fc7 features. Table 2 shows that the Mul-
to generate sentences on full images with the goal of veri- timodal RNN confidently outperforms this retrieval method.
fying that the model is rich enough to support the mapping Hence, even with 113,000 train set images in MSCOCO
from image data to sequences of words. For these full im- the retrieval approach is inadequate. Additionally, the RNN
age experiments we use the more powerful VGGNet image takes only a fraction of a second to evaluate per image.
features [47]. We report the BLEU [44], METEOR [7] and
Comparison to other work. Several related models have
CIDEr [53] scores computed with the coco-caption
been proposed in Arxiv preprints since the original submis-
code [4] 2 . Each method evaluates a candidate sentence
sion of this work. We also include these in Table 2 for com-
by measuring how well it matches a set of five reference
parison. Most similar to our model is Vinyals et al. [54].
sentences written by humans.
Unlike this work where the image information is commu-
Qualitative. The model generates sensible descriptions of nicated through a bias term on the first step, they incorpo-
images (see Figure 6), although we consider the last two rate it as a first word, they use a more powerful but more
images failure cases. The first prediction man in black complex sequence learner (LSTM [20]), a different CNN
shirt is playing a guitar does not appear in the training set. (GoogLeNet [51]), and report results of a model ensemble.
However, there are 20 occurrences of man in black shirt Donahue et al. [8] use a 2-layer factored LSTM (similar
and 60 occurrences of is paying guitar, which the model in structure to the RNN in Mao et al. [38]). Both models
may have composed to describe the first image. In general, appear to work worse than ours, but this is likely in large
we find that a relatively large portion of generated sentences part due to their use of the less powerful AlexNet [28] fea-
(60% with beam size 7) can be found in the training data. tures. Compared to these approaches, our model prioritizes
This fraction decreases with lower beam size; For instance, simplicity and speed at a slight cost in performance.
with beam size 1 this falls to 25%, but the performance also
deteriorates (e.g. from 0.66 to 0.61 CIDEr). 4.3. Generated Descriptions: Region evaluation
Multimodal RNN outperforms retrieval baseline. Our We now train the Multimodal RNN on the correspondences
first comparison is to a nearest neighbor retrieval baseline. between image regions and snippets of text, as inferred by
the alignment model. To support the evaluation, we used
2 https://github.com/tylin/coco-caption Amazon Mechanical Turk (AMT) to collect a new dataset
Figure 7. Example region predictions. We use our region-level multimodal RNN to generate text (shown on the right of each image) for
some of the bounding boxes in each image. The lines are grounded to centers of bounding boxes and the colors are chosen arbitrarily.

of region-level annotations that we only use at test time. The Model B-1 B-2 B-3 B-4
labeling interface displayed a single image and asked anno- Human agreement 61.5 45.2 30.1 22.0
tators (we used nine per image) to draw five bounding boxes Nearest Neighbor 22.9 10.5 0.0 0.0
and annotate each with text. In total, we collected 9,000 text RNN: Fullframe model 14.2 6.0 2.2 0.0
RNN: Region level model 35.2 23.0 16.1 14.8
snippets for 200 images in our MSCOCO test split (i.e. 45
snippets per image). The snippets have an average length of Table 3. BLEU score evaluation of image region annotations.
2.3 words. Example annotations include sports car, el- 4.4. Limitations
derly couple sitting, construction site, three dogs on
Although our results are encouraging, the Multimodal RNN
leashes, chocolate cake. We noticed that asking anno-
model is subject to multiple limitations. First, the model can
tators for grounded text snippets induces language statistics
only generate a description of one input array of pixels at a
different from those in full image captions. Our region an-
fixed resolution. A more sensible approach might be to use
notations are more comprehensive and feature elements of
multiple saccades around the image to identify all entities,
scenes that would rarely be considered salient enough to be
their mutual interactions and wider context before generat-
included in a single sentence sentence about the full image,
ing a description. Additionally, the RNN receives the image
such as heating vent, belt buckle, and chimney.
information only through additive bias interactions, which
Qualitative. We show example region model predictions are known to be less expressive than more complicated mul-
in Figure 7. To reiterate the difficulty of the task, consider tiplicative interactions [50, 20]. Lastly, our approach con-
for example the phrase table with wine glasses that is sists of two separate models. Going directly from an image-
generated on the image on the right in Figure 7. This phrase sentence dataset to region-level annotations as part of a sin-
only occurs in the training set 30 times. Each time it may gle model trained end-to-end remains an open problem.
have a different appearance and each time it may occupy a
few (or none) of our object bounding boxes. To generate 5. Conclusions
this string for the region, the model had to first correctly We introduced a model that generates natural language de-
learn to ground the string and then also learn to generate it. scriptions of image regions based on weak labels in form of
a dataset of images and sentences, and with very few hard-
Region model outperforms full frame model and rank- coded assumptions. Our approach features a novel ranking
ing baseline. Similar to the full image description task, we model that aligned parts of visual and language modalities
evaluate this data as a prediction task from a 2D array of through a common, multimodal embedding. We showed
pixels (one image region) to a sequence of words and record that this model provides state of the art performance on
the BLEU score. The ranking baseline retrieves training image-sentence ranking experiments. Second, we described
sentence substrings most compatible with each region as a Multimodal Recurrent Neural Network architecture that
judged by the BRNN model. Table 3 shows that the region generates descriptions of visual data. We evaluated its per-
RNN model produces descriptions most consistent with our formance on both fullframe and region-level experiments
collected data. Note that the fullframe model was trained and showed that in both cases the Multimodal RNN outper-
only on full images, so feeding it smaller image regions forms retrieval baselines.
deteriorates its performance. However, its sentences are
also longer than the region model sentences, which likely Acknowledgements.
negatively impacts the BLEU score. The sentence length We thank Justin Johnson and Jon Krause for helpful com-
is non-trivial to control for with an RNN, but we note that ments and discussions. We gratefully acknowledge the sup-
the region model also outperforms the fullframe model on port of NVIDIA Corporation with the donation of the GPUs
all other metrics: CIDEr 61.6/20.3, METEOR 15.8/13.3, used for this research. This research is partially supported
ROUGE 35.1/21.0 for region/fullframe respectively. by an ONR MURI grant, and NSF ISS-1115313.
References [18] S. Gould, R. Fulton, and D. Koller. Decomposing a scene
into geometric and semantically consistent regions. In Com-
[1] A. Barbu, A. Bridge, Z. Burchill, D. Coroian, S. Dickin- puter Vision, 2009 IEEE 12th International Conference on,
son, S. Fidler, A. Michaux, S. Mussman, S. Narayanaswamy, pages 18. IEEE, 2009.
D. Salvi, et al. Video in sentences out. arXiv preprint [19] A. Gupta and P. Mannem. From image annotation to im-
arXiv:1204.2742, 2012. age description. In Neural information processing. Springer,
[2] K. Barnard, P. Duygulu, D. Forsyth, N. De Freitas, D. M. 2012.
Blei, and M. I. Jordan. Matching words and pictures. JMLR,
[20] S. Hochreiter and J. Schmidhuber. Long short-term memory.
2003.
Neural computation, 9(8):17351780, 1997.
[3] Y. Bengio, H. Schwenk, J.-S. Senecal, F. Morin, and J.-L.
[21] M. Hodosh, P. Young, and J. Hockenmaier. Framing image
Gauvain. Neural probabilistic language models. In Innova-
description as a ranking task: data, models and evaluation
tions in Machine Learning. Springer, 2006.
metrics. Journal of Artificial Intelligence Research, 2013.
[4] X. Chen, H. Fang, T.-Y. Lin, R. Vedantam, S. Gupta, P. Dol-
lar, and C. L. Zitnick. Microsoft coco captions: Data collec- [22] R. JeffreyPennington and C. Manning. Glove: Global vec-
tion and evaluation server. arXiv preprint arXiv:1504.00325, tors for word representation. 2014.
2015. [23] Y. Jia, M. Salzmann, and T. Darrell. Learning cross-modality
[5] X. Chen and C. L. Zitnick. Learning a recurrent vi- similarity for multinomial data. In ICCV, 2011.
sual representation for image caption generation. CoRR, [24] A. Karpathy, A. Joulin, and L. Fei-Fei. Deep fragment em-
abs/1411.5654, 2014. beddings for bidirectional image sentence mapping. arXiv
[6] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei- preprint arXiv:1406.5679, 2014.
Fei. Imagenet: A large-scale hierarchical image database. In [25] R. Kiros, R. Salakhutdinov, and R. S. Zemel. Unifying
CVPR, 2009. visual-semantic embeddings with multimodal neural lan-
[7] M. Denkowski and A. Lavie. Meteor universal: Language guage models. arXiv preprint arXiv:1411.2539, 2014.
specific translation evaluation for any target language. In [26] R. Kiros, R. S. Zemel, and R. Salakhutdinov. Multimodal
Proceedings of the EACL 2014 Workshop on Statistical Ma- neural language models. ICML, 2014.
chine Translation, 2014. [27] C. Kong, D. Lin, M. Bansal, R. Urtasun, and S. Fidler. What
[8] J. Donahue, L. A. Hendricks, S. Guadarrama, M. Rohrbach, are you talking about? text-to-image coreference. In CVPR,
S. Venugopalan, K. Saenko, and T. Darrell. Long-term recur- 2014.
rent convolutional networks for visual recognition and de- [28] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
scription. arXiv preprint arXiv:1411.4389, 2014. classification with deep convolutional neural networks. In
[9] D. Elliott and F. Keller. Image description using visual de- NIPS, 2012.
pendency representations. In EMNLP, pages 12921302, [29] G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg,
2013. and T. L. Berg. Baby talk: Understanding and generating
[10] J. L. Elman. Finding structure in time. Cognitive science, simple image descriptions. In CVPR, 2011.
14(2):179211, 1990.
[30] P. Kuznetsova, V. Ordonez, A. C. Berg, T. L. Berg, and
[11] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and
Y. Choi. Collective generation of natural image descriptions.
A. Zisserman. The pascal visual object classes (voc) chal-
In ACL, 2012.
lenge. International Journal of Computer Vision, 88(2):303
[31] P. Kuznetsova, V. Ordonez, T. L. Berg, U. C. Hill, and
338, June 2010.
Y. Choi. Treetalk: Composition and compression of trees
[12] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng,
for image descriptions. Transactions of the Association for
P. Dollar, J. Gao, X. He, M. Mitchell, J. Platt, et al.
Computational Linguistics, 2(10):351362, 2014.
From captions to visual concepts and back. arXiv preprint
arXiv:1411.4952, 2014. [32] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-
[13] A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, based learning applied to document recognition. Proceed-
C. Rashtchian, J. Hockenmaier, and D. Forsyth. Every pic- ings of the IEEE, 86(11):22782324, 1998.
ture tells a story: Generating sentences from images. In [33] L.-J. Li and L. Fei-Fei. What, where and who? classifying
ECCV. 2010. events by scene and object recognition. In ICCV, 2007.
[14] L. Fei-Fei, A. Iyer, C. Koch, and P. Perona. What do we [34] L.-J. Li, R. Socher, and L. Fei-Fei. Towards total scene un-
perceive in a glance of a real-world scene? Journal of vision, derstanding: Classification, annotation and segmentation in
7(1):10, 2007. an automatic framework. In Computer Vision and Pattern
[15] S. Fidler, A. Sharma, and R. Urtasun. A sentence is worth a Recognition, 2009. CVPR 2009. IEEE Conference on, pages
thousand pixels. In CVPR, 2013. 20362043. IEEE, 2009.
[16] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, [35] S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y. Choi. Com-
T. Mikolov, et al. Devise: A deep visual-semantic embed- posing simple image descriptions using web-scale n-grams.
ding model. In NIPS, 2013. In CoNLL, 2011.
[17] R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich fea- [36] D. Lin, S. Fidler, C. Kong, and R. Urtasun. Visual semantic
ture hierarchies for accurate object detection and semantic search: Retrieving videos via complex textual queries. 2014.
segmentation. In CVPR, 2014.
[37] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ra- [54] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show
manan, P. Dollar, and C. L. Zitnick. Microsoft coco: Com- and tell: A neural image caption generator. arXiv preprint
mon objects in context. arXiv preprint arXiv:1405.0312, arXiv:1411.4555, 2014.
2014. [55] Y. Yang, C. L. Teo, H. Daume III, and Y. Aloimonos.
[38] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Explain Corpus-guided sentence generation of natural images. In
images with multimodal recurrent neural networks. arXiv EMNLP, 2011.
preprint arXiv:1410.1090, 2014. [56] B. Z. Yao, X. Yang, L. Lin, M. W. Lee, and S.-C. Zhu. I2t:
[39] C. Matuszek*, N. FitzGerald*, L. Zettlemoyer, L. Bo, and Image parsing to text description. Proceedings of the IEEE,
D. Fox. A Joint Model of Language and Perception for 98(8):14851508, 2010.
Grounded Attribute Learning. In Proc. of the 2012 Interna- [57] M. Yatskar, L. Vanderwende, and L. Zettlemoyer. See no
tional Conference on Machine Learning, Edinburgh, Scot- evil, say no evil: Description generation from densely la-
land, June 2012. beled images. Lexical and Computational Semantics, 2014.
[40] T. Mikolov, M. Karafiat, L. Burget, J. Cernocky, and S. Khu-
danpur. Recurrent neural network based language model. In [58] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From im-
INTERSPEECH, 2010. age descriptions to visual denotations: New similarity met-
[41] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and rics for semantic inference over event descriptions. TACL,
J. Dean. Distributed representations of words and phrases 2014.
and their compositionality. In NIPS, 2013. [59] W. Zaremba, I. Sutskever, and O. Vinyals. Recurrent neu-
[42] M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, ral network regularization. arXiv preprint arXiv:1409.2329,
A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daume, 2014.
III. Midge: Generating image descriptions from computer [60] C. L. Zitnick, D. Parikh, and L. Vanderwende. Learning the
vision detections. In EACL, 2012. visual interpretation of sentences. ICCV, 2013.
[43] V. Ordonez, G. Kulkarni, and T. L. Berg. Im2text: Describ-
ing images using 1 million captioned photographs. In NIPS,
2011.
[44] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a
method for automatic evaluation of machine translation. In
Proceedings of the 40th annual meeting on association for
computational linguistics, pages 311318. Association for
Computational Linguistics, 2002.
[45] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh,
S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein,
A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recog-
nition challenge, 2014.
[46] M. Schuster and K. K. Paliwal. Bidirectional recurrent neural
networks. Signal Processing, IEEE Transactions on, 1997.
[47] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. arXiv preprint
arXiv:1409.1556, 2014.
[48] R. Socher and L. Fei-Fei. Connecting modalities: Semi-
supervised segmentation and annotation of images using un-
aligned text corpora. In CVPR, 2010.
[49] R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y.
Ng. Grounded compositional semantics for finding and de-
scribing images with sentences. TACL, 2014.
[50] I. Sutskever, J. Martens, and G. E. Hinton. Generating text
with recurrent neural networks. In ICML, 2011.
[51] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabi-
novich. Going deeper with convolutions. arXiv preprint
arXiv:1409.4842, 2014.
[52] T. Tieleman and G. E. Hinton. Lecture 6.5-rmsprop: Divide
the gradient by a running average of its recent magnitude.,
2012.
[53] R. Vedantam, C. L. Zitnick, and D. Parikh. Cider:
Consensus-based image description evaluation. CoRR,
abs/1411.5726, 2014.
6. Supplementary Material

6.1. Magnitude modulation

An appealing feature of our alignment model is that it learns
to modulate the importance of words and regions by scaling
the magnitude of their corresponding embedding vectors.
Magnitude Word Magnitude Word
To see this, recall that we compute the image-sentence sim- 0.42 now 2.61 kayaking
ilarity between image k and sentence l as follows: 0.42 simply 2.59 trampoline
X 0.43 actually 2.59 pumpkins
Skl = maxigk viT st . (16) 0.44 but 2.58 windsurfing
tgl 0.44 neither 2.56 wakeboard
Disciminative words. As a result of this formulation, 0.45 then 2.54 acrobatics
we observe that representations of visually discriminative 0.45 still 2.54 sousaphone
0.46 obviously 2.54 skydivers
words such as kayaking, pumpkins tend to have higher
0.47 that 2.52 wakeboarders
magnitude in the embedding space, which translates to a
0.47 which 2.52 skateboard
higher influence on the final image-sentence scores due to 0.47 felt 2.51 snowboarder
the inner product. Conversely, the model learns to map stop 0.47 not 2.51 wakeboarder
words such as now, simply, actually, but near the ori- 0.47 might 2.50 skydiving
gin, which reduces their influence. Table 4 show the top 0.47 because 2.50 guitar
40 words with highest and lowest magnitudes kst k. 0.48 appeared 2.50 snowboard
0.48 therefore 2.48 kitchen
Disciminative regions. Similarly, image regions that con- 0.48 been 2.48 paraglider
tain discriminative entities are assigned vectors of higher 0.48 if 2.48 ollie
magnitudes by our model. This can be be interpreted as a 0.48 also 2.47 firetruck
measure of visual saliency, since these regions would pro- 0.48 only 2.47 gymnastics
duced large scores if their textual description was present in 0.48 so 2.46 waterfalls
a corresponding sentence. We show the regions with high 0.49 would 2.46 motorboat
magnitudes in Figure 8. Notice the common occurrence of 0.49 yet 2.46 fryer
often described regions such as balls, bikes, helmets. 0.50 be 2.46 skateboarding
0.50 had 2.46 dulcimer
0.50 revealed 2.46 waterfall
0.50 never 2.46 backflips
0.50 very 2.46 unicyclist
0.50 without 2.45 kayak
0.51 they 2.43 costumes
0.51 either 2.43 wakeboarding
0.51 could 2.43 trike
0.51 feel 2.42 dancers
0.51 otherwise 2.42 cupcakes
0.51 when 2.42 tuba
0.51 already 2.42 skijoring
0.51 being 2.41 firewood
0.51 else 2.41 elevators
0.52 just 2.40 cranes
0.52 ones 2.40 bassoon
Table 4. This table shows the top magnitudes of vectors (kst k) for
words in Flickr30K. Since the magnitude of individual words in
our model is also a function of their surrounding context in the
sentence, we report the average magnitude.

Figure 8. Flickr30K test set regions with high vector magnitude.

6.2. Alignment model Plug and play use of Natural Language Processing
toolkits. Before adopting the BRNN approach, we also
Learned appearance of text snippets. We can query our tried to use Natural Language Processing toolkits to process
alignment model with a piece of text and retrieve individual the input sentences into graphs of noun phrases and their bi-
image regions that have the highest score with that snip- nary relations. For instance, in the sentence a brown dog is
pet. We show examples of such queries in Figure 9 and chasing a young child, the toolkit would infer that there are
Figure 10. Notice that the model is sensitive to compound two noun phrases (a brown dog, young child), joined
words and modifiers. For example, red bus and yel- by a binary relationship of chasing. We then developed
low bus give very different results. Similarly, bird flying a CRF that inferred the grounding of these noun phrases to
in the sky and bird on a tree branch give different re- the detection bounding boxes in the image with a unary ap-
sults. Additionally, it can be seen that the quality of the pearance model and a spatial binary model. However, this
results deteriorates for less frequently occurring concepts, endeavor proved fruitless. First, performing CRF-like infer-
such as roof or straw hat. However, we emphasize that ence during the forward pass of a Neural Network proved
the model learned these visual appearances of text snippets to be extremely slow. Second, we found that there is sur-
from raw data of full images and sentences, without any ex- prisingly little information in the relative spatial positions
plicit correspondences. between bounding boxes. For instance, almost any two
bounding boxes in the image could correspond to the ac-
Additional alignment visualizations. See additional ex-
tion of chasing due to huge amount of possibly camera
amples of inferred alignments between image regions and
views of a scene. Hence, we were unable to extract enough
words in Figure 11. Note that one limitation of our model is
signal from the binary relations in the coordinate system
that it does not explicitly handle or support counting. For in-
of the image and suspect that more complex 3-dimensional
stance, the last example we show contains the phrase three
reasoning may be required. Lastly, we found that NLP tools
people. These words should align to the three people in
(when used out of the box) introduce a large amount of mis-
the image, but our model puts the bounding box around two
takes in the extracted parse trees, dependency trees and parts
of the people. In doing so, the model may be taking advan-
of speech tags. We tried to fix these with complex rules and
tage of the BRNN structure to modify the people vector
exceptions, but ultimately decided to abandon the idea. We
to preferentially align to regions that contain multiple peo-
believe that part of the problem is that these tools are usually
ple. However, this is still unsatisfying because such spuri-
trained on different text corpora (e.g. news articles), so im-
ous detections only exist as a result of an error in the RCNN
age captions are outside of their domain of competence. In
inference process, which presumably failed to localize the
our experience, adopting the BRNN model instead of this
individual people.
approach provided immediate performance improvements
Web demo. We have published a web demo that displays and produced significant reductions in code complexity.
our alignments for all images in the test set 3 .
Additional Flickr8K experiments. We omitted ranking
experiment results from our paper due to space constraints, 6.3. Additional examples: Image annotation
but these can be found in Table 5
Counting. We experimented with losses that perform prob-
Additional examples of generated captions on the full im-
abilistic inference in the forward pass that explicitly tried
age level can be found in Figure 12 (and our website). The
to localize exactly three distinct people in the image. How-
model often gets the right gist of the scene, but sometimes
ever, this worked poorly because while the RCNN is good
guesses specific fine-grained words incorrectly. We expect
at finding people, it is not very good at localizing them. For
that reasoning not only the global level of the image but also
instance, a single person can easily yield multiple detections
on the level of objects will significantly improve these re-
(the head, the torso, or the full body, for example). We were
sults. We find the last example (woman in bikini is jumping
not able to come up with a simple approach to collapsing
over hurdle) to be especially illuminating. This sentence
these into a single detection (non-maxim suppression by it-
does not occur in the training data. Our general qualitative
self was not sufficient in our experiments). Note that this
impression of the model is that it learns certain templates,
ambiguity is partly an artifact of the training data. For ex-
e.g. <noun>in <noun>is <verb>in <noun>, and then
ample, torsos of people can often be labeled alone if the
fills these in based on textures in the image. In this partic-
body is occluded. We are therefore lead to believe that this
ular case, the volleyball net has the visual appearance of a
additional modeling step is highly non-trivial and a worthy
hurdle, which may have caused the model to insert it as a
subject of future work.
noun (along with the woman) into one of its learned sen-
3 http://cs.stanford.edu/people/karpathy/deepimagesent/rankingdemo/ tence templates.
6.4. Additional examples: Region annotation

Additional examples of region annotations can be found

in Figure 13. Note that we annotate regions based on the
content of each image region alone, which can cause erro-
neous predictions when not enough context is available in
the bounding box (e.g. a generated description that says
container detected on the back of a dogs head in the im-
age on the right, in the second row). We found that one ef-
fective way of using the contextual information and improv-
ing the predictions is to concatenate the fullframe feature
CNN vector to the vector of the region of interest, giving
8192-dimensional input vector the to RNN. However, we
chose to omit these experiments in our paper to preserve the
simplicity of the mode, and because we believe that cleaner
and more principled approaches to this challenge can be de-
veloped.

6.5. Training the Multimodal RNN

There are a few tricks needed to get the Multimodal RNN to

train efficiently. We found that clipping the gradients (we
only experimented with simple per-element clipping) at an
appropriate value consistently gave better results and helped
on the validation data. As mentioned in our paper, we exper-
imented with SGD, SGD+Momentum, Adadelta, Adagrad,
but found RMSProp to give best results. However, some
SGD checkpoints usually also converged to nearby valida-
tion performance vicinity. Moreover, the distribution of the
words in English language are highly non-uniform. There-
fore, the model spends the first few iterations mostly learn-
ing the biases for the Softmax classifier such that it is pre-
dicting every word at random with the appropriate dataset
frequency. We found that we could obtain faster conver-
gence early in the training (and nicer loss curves) by explic-
itly initializing the biases of all words in the dictionary (in
the Softmax classifier) to log probability of their occurrence
in the training data. Therefore, with small weights and bi-
ases set appropriately the model right away predicts word
at random according to their chance distribution. After sub-
mission of our original paper we performed additional ex-
periments with comparing an RNN to an LSTM and found
that LSTMs consistently produced better results, but took
longer to train. Lastly, we initially used word2vec vectors
as our word representations xi , but found that it was suffi-
cient to train these vectors from random initialization with-
out changes in the final performance. Moreover, we found
that the word2vec vectors have some unappealing properties
when used in multimodal language-visual tasks. For exam-
ple, all colors (e.g. red, blue, green) are clustered nearby
in the word2vec representation because they are relatively
interchangeable in most language contexts. However, their
visual instantiations are very different.
chocolate cake

glass of wine

red bus

yellow bus

closeup of zebra

sprinkled donut

wooden chair

wooden office desk

shiny laptop

Figure 9. Examples of highest scoring regions for queried snippets of text, on 5,000 images of our MSCOCO test set.
bird flying in the sky

bird on a tree branch

bird sitting on roof

closeup of fruit

bowl of fruit

man riding a horse

straw hat

Figure 10. Examples of highest scoring regions for queried snippets of text, on 5,000 images of our MSCOCO test set.

Image Annotation Image Search

Model R@1 R@5 R@10 Med r R@1 R@5 R@10 Med r
Flickr8K
DeViSE (Frome et al. [16]) 4.5 18.1 29.2 26 6.7 21.9 32.7 25
SDT-RNN (Socher et al. [49]) 9.6 29.8 41.1 16 8.9 29.8 41.1 16
Kiros et al. [25] 13.5 36.2 45.7 13 10.4 31.0 43.7 14
Mao et al. [38] 14.5 37.2 48.5 11 11.5 31.0 42.4 15
DeFrag (Karpathy et al. [24]) 12.6 32.9 44.0 14 9.7 29.6 42.5 15
Our implementation of DeFrag [24] 13.8 35.8 48.2 10.4 9.5 28.2 40.3 15.6
Our model: DepTree edges 14.8 37.9 50.0 9.4 11.6 31.4 43.8 13.2
Our model: BRNN 16.5 40.6 54.2 7.6 11.8 32.1 44.7 12.4
Table 5. Ranking experiment results for the Flickr8K dataset.
Figure 11. Additional examples of alignments. For each query test image above we retrieve the most compatible sentence from the test set
and show the alignments.

Figure 12. Additional examples of captions on the level of full images. Green: Human ground truth. Red: Top-scoring sentence from
training set. Blue: Generated sentence.
Figure 13. Additional examples of region captions on the test set of Flickr30K.