Attention-Based Convolutional Neural Network

for Machine Comprehension

Wenpeng Yin, Sebastian Ebert, Hinrich Schütze

University of Munich, Germany
wenpeng, [email protected]
arXiv:1602.04341v1 [cs.CL] 13 Feb 2016

Abstract
Understanding open-domain text is one of the pri-
mary challenges in natural language processing
(NLP). Machine comprehension benchmarks eval-
uate the system’s ability to understand text based on
the text content only. In this work, we investigate
machine comprehension on MCTest, a question an-
swering (QA) benchmark. Prior work is mainly
based on feature engineering approaches. We come
up with a neural network framework, named hier-
archical attention-based convolutional neural net-
work (HABCNN), to address this task without any
manually designed features. Specifically, we ex-
plore HABCNN for this task by two routes, one Figure 1: One example with 2 out of 4 questions in the
is through traditional joint modeling of document, MCTest. “*” marks correct answer.
question and answer, one is through textual entail-
only one is correct. Questions in MCTest have two categories:
ment. HABCNN employs an attention mechanism
“one” and “multiple”. The label means one or multiple sen-
to detect key phrases, key sentences and key snip-
tences from the document are required to answer this ques-
pets that are relevant to answering the question. Ex-
tion. To correctly answer the first question in the example,
periments show that HABCNN outperforms prior
the two blue sentences are required; for the second question
deep learning approaches by a big margin.
instead, only the red sentence can help. The following obser-
vations hold for the whole MCTest. (i) Most of the sentences
1 Introduction in the document are irrelavent for a given question. It hints
Endowing machines with the ability to understand natu- that we need to pay attention to just some key regions. (ii) The
ral language is a long-standing goal in NLP and holds the answer candidates can be flexible text in length and abstrac-
promise of revolutionizing the way in which people interact tion level, and probably do not appear in the document. For
with machines and retrieve information. Richardson et al. example, candidate B for the second question is “outside”,
[2013] proposed the task of machine comprehension, along which is one word and does not exist in the document, while
with MCTest, a question answering dataset for evaluation. the answer candidates for the first question are longer texts
The ability of the machine to understand text is evaluated with some auxiliary words like “Because” in the text. This
by posing a series of questions, where the answer to each requires our system to handle flexible texts via extraction as
question can be found only in the associated text. Solutions well as abstraction. (iii) Some questions require multiple sen-
typically focus on some semantic interpretation of the text, tences to infer the answer, and those vital sentences mostly
possibly with some form of probabilistic or logic inference, appear close to each other (we call them snippet). Hence,
to answer the question. Despite intensive recent work [We- our system should be able to make a choice or compromise
ston et al., 2014; Weston et al., 2015; Hermann et al., 2015; between potential single-sentence clue and snippet clue.
Sachan et al., 2015], the problem is far from solved. Prior work on this task is mostly based on feature engi-
Machine comprehension is an open-domain question- neering. This work, instead, takes the lead in presenting a
answering problem which contains factoid questions, but the deep neural network based approach without any linguistic
answers can be derived by extraction or induction of key features involved.
clues. Figure 1 shows one example in MCTest. Each ex- Concretely, we propose HABCNN, a hierarchical
ample consists of one document, four associated questions; attention-based convolutional neural network, to address this
each question is followed by four answer candidates in which task in two roadmaps. In the first one, we project the docu-
ment in two different ways, one based on question-attention,
one based on answer-attention and then compare the two
projected document representations to determine whether
the answer matches the question. In the second one, every
question-answer pair is reformatted into a statement, then the
whole task is treated through textual entailment.
In both roadmaps, convolutional neural network (CNN) is
explored to model all types of text. As human beings usu-
ally do for such a QA task, our model is expected to be able
to detect the key snippets, key sentences, and key words or
Figure 2: Illustrations of HABCNN-QAP (top), HABCHH-
phrases in the document. In order to detect those informative
QP (middle) and HABCNN-TE (bottom). Q, A, S: question,
parts required by questions, we explore an attention mecha-
answer, statement; D: document
nism to model the document so that its representation con-
tains required information intensively. In practice, instead [Narasimhan and Barzilay, 2015; Sachan et al., 2015; Wang
of imitating human beings in QA task top-down, our system and McAllester, 2015; Smith et al., 2015]. In these works,
models the document bottom-up, through accumulating the a common route is first to define a regularized loss func-
most relevant information from word level to snippet level. tion based on assumed feature vectors, then the effort focuses
Our approach is novel in three aspects. (i) A document on designing effective features based on various rules. Even
is modeled by a hierarchical CNN for different granularity, though these researches are groundbreaking for this task, their
from word to sentence level, then from sentence to snippet flexibility and their capacity for generalization are limited.
level. The reason of choosing a CNN rather than other se- Deep learning based approaches appeal to increasing inter-
quence models like recurrent neural network [Mikolov et al., est in analogous tasks. Weston et al., [2014] introduce mem-
2010], long short-term memory unit (LSTM [Hochreiter and ory networks for factoid QA. Memory network framework
Schmidhuber, 1997]), gated recurrent unit (GRU [Cho et al., is extended in [Weston et al., 2015; Kumar et al., 2015] for
2014]) etc, is that we argue CNNs are more suitable to detect Facebook bAbI dataset. Peng et al. [2015]’s Neural Reasoner
the key sentences within documents and key phrases within infers over multiple supporting facts to generate an entity an-
sentences. Considering again the second question in Figure swer for a given question and it is also tested on bAbI. All of
1, the original sentence “They sat by the fire and talked about these works deal with some short texts with simple-grammar,
he insects” has more information than required, i.e, we do not aiming to generate an answer which is restricted to be one
need to know “they talked about the insects”. Sequence mod- word denoting a location, a person etc.
eling neural networks usually model the sentence meaning by Some works also tried over other kinds of QA tasks. For
accumulating the whole sequence. CNNs, with convolution- example, Iyyer et al., [2014] present QANTA, a recursive
pooling steps, are supposed to detect some prominent features neural network, to infer an entity based on its description text.
no matter where the features come from. (ii) In the example This task is basically a matching between description and en-
in Figure 1, apparently not all sentences are required given a tity, no explicit question exist. Another difference with us
question, and usually different snippets are required by dif- lies in that all the sentences in the entity description actually
ferent questions. Hence, the same document should have dif- contain partial information about the entity, hence a descrip-
ferent representations based on what the question is. To this tion is supposed to have only one representation. However
end, attentions are incorporated into the hierarchical CNN in our task, the modeling of document should be dynami-
to guide the learning of dynamic document representations cally changed according to the question analysis. Hermann
which closely match the information requirements by ques- et al., [2015] incorporate attention mechanism into LSTM for
tions. (iii) Document representations at sentence and snippet a QA task over news text. Still, their work does not handle
levels both are informative for the question, a highway net- some complex question types like “Why...”, they merely aim
work is developed to combine them, enabling our system to to find the entity from the document to fill the slot in the query
make a flexible tradeoff. so that the completed query is true based on the document.
Overall, we make three contributions. (i) We present a hi- Nevertheless, it inspires us to treat our task as a textual en-
erarchical attention-based CNN system “HABCNN”. It is, to tailment problem by first reformatting question-answer pairs
our knowledge, the first deep learning based system for this into statements.
MCTest task. (ii) Prior document modeling systems based on Some other deep learning systems are developed for an-
deep neural networks mostly generate generic representation, swer selection task [Yu et al., 2014; Yang et al., 2015;
this work is the first to incorporate attention so that docu- Severyn and Moschitti, 2015; Shen et al., 2015; Wang et
ment representation is biased towards the question require- al., 2010]. Differently, this kind of question answering task
ment. (iii) Our HABCNN systems outperform other deep does not involve document comprehension. They only try to
learning competitors by big margins. match the question and answer candidate without any back-
ground information. Instead, we treat machine comprehen-
2 Related Work sion in this work as a question-answer matching problem un-
der background guidance.
Existing systems for MCTest task are mostly based on man- Overall, for open-domain MCTest machine comprehension
ually engineered features. Representative work includes task, this work is the first to resort to deep neural networks.
Figure 3: HABCNN. Feature maps for phrase representations pi and the max pooling steps that create sentence representations
out of phrase representations are omitted for simplification. Each snippet covers three sentences in snippet-CNN. Symbols ◦
mean cosine similarity calculation.

3 Model vi−w+1 , . . . , vi using the convolution weights W ∈ Rd1 ×wd :

We investigate this task by three approaches, illustrated in pi = tanh(W · ci + b) (1)
Figure 2. (i) We can compute two different document (D) rep-
resentations in a common space, one based on question (Q) where bias b ∈ Rd1 . d1 is called “kernel size” in CNN.
attention, one based on answer (A) attention, and compare Note that the sentence-CNNs for query and all document
them. This architecture we name HABCNN-QAP. (ii) We sentences share the same weights, so that the resulting sen-
compute a representation of D based on Q attention (as be- tence representations are comparable.
fore), but now we compare it directly with a representation of Sentence-Level Representation. The sentence-CNN gen-
A. We name this architecture HABCNN-QP. (iii) We treat this erates a new feature map (omitted in Figure 3) for each input
QA task as textual entailment (TE), first reformatting Q-A sentence, one column in the feature map denotes a phrase rep-
pair into a statement (S), then matching S and D directly. This resentation (i.e., pi in Equation (1)).
architecture we name HABCNN-TE. All three approaches For the query and each sentence of D, we do element-wise
are implemented in the common framework HABCNN. 1-max-pooling (“max-pooling” for short) [Collobert and We-
ston, 2008] over phrase representations to form their repre-
3.1 HABCNN sentations at this level.
Recall that we use the abbreviations A (answer), Q (ques- We now treat D as a set of “vital” sentences and “noise”
tion), S (statement), D (document). HABCNN performs rep- sentences. We propose attention-pooling to learn the
resentation learning for triple (Q, A, D) in HABCNN-QP and sentence-level representation of D as follows: first identify
HABCNN-QAP, for tuple (S, D) in HABCNN-TE. For con- vital sentences by computing attention for each D’s sentence
venience, we use “query” to refer to Q, A, or S uniformly. as the cosine similarity between the its representation and the
HABCNN, depicted in Figure 3, has the following phases. query representation, then select the k highest-attention sen-
Input Layer. The input is (query,D). Query is two indi- tences to do max-pooling over them. Taking Figure 3 as an
vidual sentences (for Q, A) or one single sentence (for S), example, based on the output of sentence-CNN layer, k = 2
D is a sequence of sentences. Words are initialized by d- important sentences with blue color are combined by max-
dimensional pre-trained word embeddings. As a result, each pooling as the sentence-level representation vs of D; the other
sentence is represented as a feature map with dimensionality – white-color – sentence representations are neglected as they
of d × s (s is sentence length). In Figure 3, each sentence have low attentions. (If k = all, attention-pooling returns to
in the input layer is depicted by a rectangle with multiple the common max-pooling in [Collobert and Weston, 2008].)
columns. When the query is (Q,A), this step will be repeated, once for
Sentence-CNN. Sentence-CNN is used for sentence rep- Q, once for A, to compute representations of D at the sentence
resentation learning from word level. Given a sentence of level that are biased with respect to Q and A, respectively.
length s with a word sequence: v1 , v2 , . . . , vs , let vector Snippet-CNN. As the example in Figure 1 shows, to an-
ci ∈ Rwd be the concatenated embeddings of w words swer the first question “why did Grandpa answer the door?”,
vi−w+1 , . . . , vi where w is the filter width, d is the dimen- it does not suffice to compare this question only to the sen-
sionality of word representations and 0 < i < s + w. Em- tence “Grandpa answered the door with a smile and wel-
beddings for words vi , i < 1 and i > s, are zero padding. comed Jimmy inside”; instead, the snippet “Finally, Jimmy
We then generate the representation pi ∈ Rd1 for the phrase arrived at Grandpa’s house and knocked. Grandpa answered
the door with a smile and welcomed Jimmy inside” should with respect to A. ri also has two versions, one for Q: riq ,
be used to compare. To this end, it is necessary to stack one for A: ria .
another CNN layer, snippet-CNN, to learn representations of HABCNN-QP and HABCNN-QAP make different use of
snippets, i.e., units of one or more sentences. Thus, the ba- voq . HABCNN-QP compares voq with answer representation
sic units input to snippet-CNN (resp. sentence-CNN) are sen- ria . HABCNN-QAP compares voq with voa . HABCNN-
tences (resp. words) and the output is representations of snip- QAP projects D twice, once based on attention from Q, once
pets (resp. sentences). based on attention from A and compares the two projected
Concretely, snippet-CNN puts all sentence representations representations, shown in Figure 2 (top). HABCNN-QP only
in column sequence as a feature map and conducts another utilizes the Q-based projection of D and then compares the
convolution operation over it. With filter width w, this projected document with the answer representation, shown in
step generates representation of snippet with w consecu- Figure 2 (middle).
tive sentences. Similarly, we use the same CNN to learn
higher-abstraction query representations (just treating query 3.3 HABCNN-TE
as a document with only one sentence, so that the higher- HABCNN-TE treats machine comprehension as textual en-
abstraction query representation is in the same space with tailment. We use the statements that are provided as part of
corresponding snippet representations). MCTest. Each statement corresponds to a question-answer
Snippet-Level Representation. For the output of snippet- pair; e.g., the Q/A pair “Why did Grandpa answer the door?”
CNN, each representation is more abstract and denotes big- / “Because he saw the insects” (Figure 1) is reformatted into
ger granularity. We apply the same attention-pooling process the statement “Grandpa answered the door because he saw the
to snippet level representations: attention values are com- insects”. The question answering task is then cast as: “does
puted as cosine similarities between query and snippets and the document entail the statement?”
the snippets with the k largest attentions are retained. Max- For HABCNN-TE, shown in Figure 2 (bottom), the input
pooling over the k selected snippet representations then cre- for Figure 3 is the pair (S,D). HABCNN-TE tries to match
ates the snippet-level representation vt of D. Two selected the S’s representation ri with the D’s representation vo .
snippets are shown as red in Figure 3.
Overall Representation. Based on convolution layers at
two different granularity, we have derived query-biased rep- 4 Experiments
resentations of D at sentence level (i.e., vs ) as well as snippet 4.1 Dataset
level (i.e., vt ). In order to create a flexible choice for open
MCTest1 has two subsets. MCTest-160 is a set of 160 items,
Q/A, we develop a highway network [Srivastava et al., 2015]
each consisting of a document, four questions followed by
to combine the two levels of representations as an overall rep-
one correct anwer and three incorrect answers (split into 70
resentation vo of D:
train, 30 dev and 60 test) and MCTest-500 a set of 500 items
(split into 300 train, 50 dev and 150 test).
vo = (1 − h)  vs + h  vt (2)
where highway network weights h are learned by 4.2 Training Setup and Tricks
Our training objective is to minimize the following ranking
h = σ(Wh vs + b) (3)
loss function:
where Wh ∈ Rd1 ×d1 . With the same highway network, we
L(d, a+ , a− ) = max(0, α + S(d, a− ) − S(d, a+ )) (4)
can generate the overall query representation, ri in Figure 3,
by combining the two representations of the query at sentence where S(·, ·) is a matching score between two representation
and snippet levels. vectors. Cosine similarity is used throughout. α is a constant.
For this common ranking loss, we also have two styles to
3.2 HABCNN-QP & HABCNN-QAP utilize the data in view of each positive answer is accompa-
HABCNN-QP/QAP computes the representation of D as a nied with three negative answers. One is treating (d, a+ , a− 1,
projection of D, either based on attention from Q or based on a− −
2 , a3 ) as a training example, then our loss function can
attention from A. We hope that these two projections of the have three “max()” terms, each for a positive-negative pair;
document are close for a correct A and less close for an incor- the other one is treating (d, a+ , a−
i ) as an individual training
rect A. As we said in related work, machine comprehension example. In practice, we find the second way works better.
can be viewed as an answer selection task using the docu- We conjecture that the second way has more training exam-
ment D as critical background information. Here, HABCNN- ples, and positive answers are repeatedly used to balance the
QP/QAP do not compare Q and A directly, but they use Q and amounts of positive and negative answers.
A to filter the document differently, extracting what is critical Multitask learning: Question typing is commonly used
for the Q/A match by attention-pooling. Then they match the and proved to be very helpful in QA tasks [Sachan et al.,
two document representations in the new space. 2015]. Inspired, we stack a logistic regression layer over
For ease of exposition, we have used the symbol vo so far, question representation riq , with the purpose that this sub-
but in HABCNN-QP/QAP we compute two different docu- task can favor the parameter tuning of the whole system, and
ment representations: voq , for which attention is computed
1
with respect to Q; and voa for which attention is computed http://requery.microsoft.com/mct
 k lr d1 bs w L2 α 4.4 HABCNN Variants
[1,3] 0.05 [90, 90] 1 [2,2] 0.0065 0.2 In addition to the main architectures described above, we also
Table 1: Hyperparameters. k: top-k in attention-pooling for explore two variants of ABCHNN, inspired by [Peng et al.,
both CNN layers; lr: learning rate; d1 : kernel size in CNN 2015] and [Hermann et al., 2015], respectively.
layers; bs: mini-batch size; w: filter width; L2 : L2 normal- Variant-I: As RNNs are widely recognized as a competi-
ization; α: constant in loss function. tor of CNNs in sentence modeling, similar with [Peng et al.,
2015], we replace the sentence-CNN in Figure 3 by a GRU
finally the question is better recognized and is able to find the while keeping other parts unchanged.
answer more accurately. Variant-II: How to model attention at the granularity of
words was shown in [Hermann et al., 2015]; see their paper
To be specific, we classify questions into 12 classes: for details. We develop their attention idea and model at-
“how”, “how much”, “how many”, “what”, “who”, “where”, tention at the granularity of sentence and snippet. Our atten-
“which”, “when”, “whose”, “why”, “will” and “other”. The tion gives different weights to sentences/snippets (not words),
question label is created by querying for the label keyword in then computes the document representation as a weighted av-
the question. If more than one keyword appears in a question, erage of all sentence/snippet representations.
we adopt the one appearing earlier and the more specific one
(e.g., “how much”, not “how”). In case there is no match, the 4.5 Results
class “other” is assigned. Table 2 lists the performance of baselines, HABCNN-
We train with AdaGrad [Duchi et al., 2011] and use 50- TE variants, HABCNN systems in the first, second and
dimensional GloVe [Pennington et al., 2014] to initialize last block, respectively (we only report variants for top-
word representations,2 kept fixed during training. Table 1 performing HABCNN-TE). Consistently, our HABCNN sys-
gives hyperparameter values, tuned on dev. tems outperform all baselines, especially surpass the two
We consider two evaluation metrics: accuracy (proportion competitive deep learning based systems AR and NR. The
of questions correctly answered) and NDCG4 [Järvelin and margin between our best-performing ABHCNN-TE and NR
Kekäläinen, 2002]. Unlike accuracy which evaluates if the is 15.6/16.5 (accuracy/NDCG) on MCTest-150 and 7.3/4.6 on
question is correctly answered or not, NDCG4 , being a mea- MCTest-500. This demonstrates the promise of our architec-
sure of ranking quality, evaluates the position of the correct ture in this task.
answer in our predicted ranking. As said before, both AR and NR systems aim to generate
answers in entity form. Their designs might not suit this ma-
4.3 Baseline Systems chine comprehension task, in which the answers are openly-
formed based on summarizing or abstracting the clues. To be
This work focuses on the comparison with systems about dis- more specific, AR models D always at word level, attentions
tributed representation learning and deep learning: are also paid to corresponding word representations, which
Addition. Directly compare question and answers without is applicable for entity-style answers, but is less suitable for
considering the D. Sentence representations are computed by comprehension at sentence level or even snippet level. NR
element-wise addition over word representations. contrarily models D in sentence level always, neglecting the
Addition-proj. First compute sentence representations for discovering of key phrases which however compose most of
Q, A and all D sentences as the same way as Addition, then answers. In addition, the attention of AR system and the
match the two sentences in D which have highest similarity question-fact interaction in NR system both bring large num-
with Q and A respectively. bers of parameters, this potentially constrains their power in
NR. The Neural Reasoner [Peng et al., 2015] has an encod- a dataset of limited size.
ing layer, multiple reasoning layers and a final answer layer. For Variant-I and Variant-II (second block of Table 2),
The input for the encoding layer is a question and the sen- we can see that both modifications do harm to the original
tences of the document (called facts); each sentence is en- HABCNN-TE performance. The first variant, i.e, replacing
coded by a GRU into a vector. In each reasoning layer, NR the sentence-CNN in Figure 3 as GRU module is not help-
lets the question representation interact with each fact repre- ful for this task. We suspect that this lies in the fundamen-
sentation as reasoning process. Finally, all temporary reason- tal function of CNN and GRU. The CNN models a sentence
ing clues are pooled as answer representation. without caring about the global word order information, and
max-pooling is supposed to extract the features of key phrases
AR. The Attentive Reader [Hermann et al., 2015] is im-
in the sentence no matter where the phrases are located. This
plemented by modeling the whole D as a word sequence –
property should be useful for answer detection, as answers
without specific sentence / snippet representations – using an
are usually formed by discovering some key phrases, not all
LSTM. Attention mechanism is implemented at word repre-
words in a sentence should be considered. However, a GRU
sentation level.
models a sentence by reading the words sequentially, the im-
Overall, baselines Addition and Addition-proj do not in- portance of phrases is less determined by the question re-
volve complex composition and inference. NR and AR rep- quirement. The second variant, using a more complicated
resent the top-performing deep neural networks in QA tasks. attention scheme to model biased D representations than sim-
ple cosine similarity based attention used in our model, is less
2
http://nlp.stanford.edu/projects/glove/ effective to detect truly informative sentences or snippet. We
 MCTest-150 MCTest-500
method acc NDCG4 acc NDCG4
one mul all one mul all one mul all one mul all
Addition
Baselines 39.3 32.4 35.7 60.4 50.3 54.6 35.7 30.2 32.9 56.6 55.2 55.8
Addition-proj 42.1 38.7 40.3 65.3 61.3 63.2 39.4 36.7 38.0 63.3 60.1 61.7
AR 48.1 44.7 46.3 70.5 68.9 69.6 44.4 39.5 41.9 66.7 64.2 65.4
NR 48.4 46.8 47.6 70.7 68.2 69.7 45.7 45.6 45.6 71.9 69.5 70.6
Variant-I 50.4 47.7 49.0 73.9 71.2 72.5 45.4 42.1 43.7 68.8 64.5 66.6
Variant-II 53.6 51.0 52.2 74.6 70.1 72.3 47.1 46.0 46.5 70.2 63.8 66.9
HABCNN-QP 57.9 53.7 55.7 80.4 80.0 80.2 53.7 46.7 50.1 75.4 72.7 74.0
HABCNN-QAP 59.0 57.9 58.4 81.5 79.9 80.6 54.0 47.2 50.6 75.7 72.6 74.1
HABCNN-TE 63.3 62.9 63.1 86.6 85.9 86.2 54.2 51.7 52.9 76.1 74.4 75.2
Table 2: Experimental results for one-sentence (one), multiple-sentence (mul) and all cases.

Figure 4: Attention visualization for statement “Grandpa answered the door because Jimmy knocked” in the example Figure 1.
Too long sentences are truncated with “. . .”. Left is attention weights for each single sentence after the sentence-CNN, right is
attention weights for each snippet (two consecutive sentences as filter width w = 2 in Table 1) after the snippet-CNN.

doubt such kind of attention scheme when used in sentence way network to combine both representations as an overall
sequences of large size. In training, the attention weights after D representation. This visualization hints that our architec-
softmax normalization have actually small difference across ture provides a good way for a question to compromise key
sentences, this means the system can not distinguish key sen- information from different granularity.
tences from noise sentences effectively. Our cosine similarity We also do some preliminary error analysis. One big obsta-
based attention-pooling, though pretty simple, is able to fil- cle for our systems is the “how many” questions. For exam-
ter noise sentences more effectively, as we only pick top-k ple, for question “how many rooms did I say I checked?” and
pivotal sentences to form D representation finally. This trick the answer candidates are four digits “5,4,3,2” which never
makes the system simple while effective. appear in the D, but require the counting of some locations.
However, these digital answers can not be modeled well by
4.6 Case Study and Error Analysis distributed representations so far. In addition, digital answers
In Figure 4, we visualize the attention distribution at sentence also appear for “what” questions, like “what time did...”. An-
level as well as snippet level for the statement “ Grandpa an- other big limitation lies in “why” questions. This question
swered the door because Jimmy knocked” whose correspond- type requires complex inference and long-distance dependen-
ing question requires multiple sentences to answer. From its cies. We observed that all deep lerning systems, including the
left part, we can see that “Grandpa answered the door with a two baselines, suffered somewhat from it.
smile and welcomed Jimmy inside” has the highest attention
weight. This meets the intuition that this sentence has seman- 5 Conclusion
tic overlap with the statement. And yet this sentence does not
contain the answer. Look further the right part, in which the This work takes the lead in presenting a CNN based neu-
CNN layer over sentence-level representations is supposed to ral network system for open-domain machine comprehen-
extract high-level features of snippets. In this level, the high- sion task. Our systems tried to solve this task in a docu-
est attention weight is cast to the best snippet “Finally, Jimmy ment projection way as well as a textual entailment way. The
arrived...knocked. Grandpa answered the door...”. And the latter one demonstrates slightly better performance. Over-
neighboring snippets also get relatively higher attentions than all, our architecture, modeling dynamic document representa-
other regions. Recall that our system chooses the one sen- tion by attention scheme from sentence level to snippet level,
tence with top attention at left part and choose top-3 snip- shows promising results in this task. In the future, more fine-
pets at right part (referring to k value in Table 1) to form grained representation learning approaches are expected to
D representations at different granularity, then uses a high- model complex answer types and question types.
References of text. In Proceedings of EMNLP, volume 1, pages
[Cho et al., 2014] Kyunghyun Cho, Bart Van Merriënboer, 193–203, 2013.
Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, [Sachan et al., 2015] Mrinmaya Sachan, Avinava Dubey,
Holger Schwenk, and Yoshua Bengio. Learning phrase Eric P Xing, and Matthew Richardson. Learning answer-
representations using rnn encoder-decoder for statistical entailing structures for machine comprehension. In Pro-
machine translation. In Proceedings of EMNLP, pages ceedings of ACL, pages 239–249, 2015.
1724–1734, 2014. [Severyn and Moschitti, 2015] Aliaksei Severyn and
[Collobert and Weston, 2008] Ronan Collobert and Jason Alessandro Moschitti. Learning to rank short text pairs
Weston. A unified architecture for natural language pro- with convolutional deep neural networks. In Proceedings
cessing: Deep neural networks with multitask learning. In of SIGIR, pages 373–382, 2015.
Proceedings of ICML, pages 160–167, 2008. [Shen et al., 2015] Yikang Shen, Wenge Rong, Zhiwei Sun,
[Duchi et al., 2011] John Duchi, Elad Hazan, and Yoram Yuanxin Ouyang, and Zhang Xiong. Question/answer
Singer. Adaptive subgradient methods for online learn- matching for cqa system via combining lexical and se-
ing and stochastic optimization. The Journal of Machine quential information. In Proceedings of AAAI, pages 275–
Learning Research, 12:2121–2159, 2011. 281, 2015.
[Hermann et al., 2015] Karl Moritz Hermann, Tomas Ko- [Smith et al., 2015] Ellery Smith, Nicola Greco, Matko
cisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Bosnjak, and Andreas Vlachos. A strong lexical matching
Mustafa Suleyman, and Phil Blunsom. Teaching machines method for the machine comprehension test. In Proceed-
to read and comprehend. In Proceedings of NIPS, pages ings of EMNLP, pages 1693–1698, 2015.
1684–1692, 2015. [Srivastava et al., 2015] Rupesh K Srivastava, Klaus Greff,
[Hochreiter and Schmidhuber, 1997] Sepp Hochreiter and and Jürgen Schmidhuber. Training very deep networks.
Jürgen Schmidhuber. Long short-term memory. Neural In NIPS, pages 2368–2376, 2015.
computation, 9(8):1735–1780, 1997. [Wang and McAllester, 2015] Hai Wang and Mohit Bansal
[Iyyer et al., 2014] Mohit Iyyer, Jordan Boyd-Graber, Kevin Gimpel David McAllester. Machine comprehen-
Leonardo Claudino, Richard Socher, and Hal Daumé III. sion with syntax, frames, and semantics. In Proceedings
A neural network for factoid question answering over of ACL, Volume 2: Short Papers, pages 700–706, 2015.
paragraphs. In Proceedings of EMNLP, pages 633–644, [Wang et al., 2010] Baoxun Wang, Xiaolong Wang,
2014. Chengjie Sun, Bingquan Liu, and Lin Sun. Modeling
[Järvelin and Kekäläinen, 2002] Kalervo Järvelin and Jaana semantic relevance for question-answer pairs in web social
Kekäläinen. Cumulated gain-based evaluation of ir communities. In Proceedings of ACL, pages 1230–1238,
techniques. ACM Transactions on Information Systems 2010.
(TOIS), 20(4):422–446, 2002. [Weston et al., 2014] Jason Weston, Sumit Chopra, and An-
[Kumar et al., 2015] Ankit Kumar, Ozan Irsoy, Jonathan Su, toine Bordes. Memory networks. In Proceedings of ICLR,
James Bradbury, Robert English, Brian Pierce, Peter On- 2014.
druska, Ishaan Gulrajani, and Richard Socher. Ask me [Weston et al., 2015] Jason Weston, Antoine Bordes, Sumit
anything: Dynamic memory networks for natural language Chopra, and Tomas Mikolov. Towards ai-complete ques-
processing. arXiv preprint arXiv:1506.07285, 2015. tion answering: a set of prerequisite toy tasks. arXiv
[Mikolov et al., 2010] Tomas Mikolov, Martin Karafiát, preprint arXiv:1502.05698, 2015.
Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. Re- [Yang et al., 2015] Yi Yang, Wen-tau Yih, and Christopher
current neural network based language model. In Proceed- Meek. Wikiqa: A challenge dataset for open-domain ques-
ings of INTERSPEECH, pages 1045–1048, 2010. tion answering. In Proceedings of EMNLP, pages 2013–
[Narasimhan and Barzilay, 2015] Karthik Narasimhan and 2018, 2015.
Regina Barzilay. Machine comprehension with discourse [Yu et al., 2014] Lei Yu, Karl Moritz Hermann, Phil Blun-
relations. In Proceedings of ACL, pages 1253–1262, 2015. som, and Stephen Pulman. Deep learning for answer sen-
[Peng et al., 2015] Baolin Peng, Zhengdong Lu, Hang Li, tence selection. In ICLR workshop, 2014.
and Kam-Fai Wong. Towards neural network-based rea-
soning. CoRR, abs/1508.05508, 2015.
[Pennington et al., 2014] Jeffrey Pennington, Richard
Socher, and Christopher D Manning. Glove: Global
vectors for word representation. Proceedings of EMNLP,
12:1532–1543, 2014.
[Richardson et al., 2013] Matthew Richardson, Christo-
pher JC Burges, and Erin Renshaw. Mctest: A challenge
dataset for the open-domain machine comprehension