A Survey on Neural Network Language Models

Kun Jing∗ and Jungang Xu∗

School of Computer Science and Technology, University of Chinese Academy of Sciences, Beijing
[email protected], [email protected]
arXiv:1906.03591v2 [cs.CL] 13 Jun 2019

Abstract which are estimated by maximum likelihood estimation.

Perplexity (PPL) [Jelinek et al., 1977], an information-
As the core component of Natural Language Pro- theoretic metric that measures the quality of a probabilistic
cessing (NLP) system, Language Model (LM) can model, is a way to evaluate LMs. Lower PPL indicates a bet-
provide word representation and probability indi- ter model. Given a corpus containing N words and a language
cation of word sequences. Neural Network Lan- model LM , the PPL of LM is
guage Models (NNLMs) overcome the curse of di- PN
1
mensionality and improve the performance of tra- 2− N t=1 log2 LM (wt |w1 ···wt−1 )
. (3)
ditional LMs. A survey on NNLMs is performed in
this paper. The structure of classic NNLMs is de- It is noted that PPL is related to corpus. Two or more LMs
scribed firstly, and then some major improvements can be compared on the same corpus in terms of PPL.
are introduced and analyzed. We summarize and However, n-gram LMs have a significant drawback. The
compare corpora and toolkits of NNLMs. Further- model would assign probabilities of 0 to the n-grams that do
more, some research directions of NNLMs are dis- not appear in the training corpus, which does not match the
cussed. actual situation. Smoothing techniques can solve this prob-
lem. Its main idea is robbing the rich for the poor, i.e., reduc-
ing the probability of events that appear in the training corpus
1 Introduction and assigning the probability to events that do not appear.
LM is the basis of various NLP tasks. For example, in Ma- Although n-gram LMs with smoothing techniques work
chine Translation tasks, LM is used to evaluate probabilities out, there still are other problems. A fundamental problem is
of outputs of the system to improve fluency of the translation the curse of dimensionality, which limits modeling on larger
in the target language. In Speech Recognition tasks, LM and corpora for a universal language model. It is particularly ob-
acoustic model are combined to predict the next word. vious in the case when one wants to model the joint distri-
Early NLP systems were based primarily on manually writ- bution in discrete space. For example, if one wants to model
ten rules, which is time-consuming and laborious and cannot an n-gram LM with a vocabulary of size 10, 000, there are
cover a variety of linguistic phenomena. In the 1980s, sta- potentially 10000n − 1 free parameters.
tistical LMs were proposed, which assigns probabilities to a In order to solve this problem, Neural Network (NN) is in-
sequence s of N words, i.e., troduced for language modeling in continuous space. NNs
P (s) =P (w1 w2 · · · wN ) including Feedforward Neural Network (FFNN), Recurrent
Neural Network (RNN), can automatically learn features and
=P (w1 )P (w2 |w1 ) · · · P (wN |w1 w2 · · · wN −1 ), (1)
continuous representation. Therefore, NNs are expected to be
where wi denotes i-th word in the sequence s. The probabil- applied to LMs, even other NLP tasks, to cover the discrete-
ity of a word sequence can be broken into the product of the ness, combination, and sparsity of natural language.
conditional probability of the next word given its predeces- The first FFNN Language Model (FFNNLM) presented by
sors that are generally called a history of context or context. [Bengio et al., 2003] fights the curse of dimensionality by
Considering that it is difficult to learn the extremely many learning a distributed representation for words, which rep-
parameters of the above model, an approximate method is resents a word as a low dimensional vector, called embed-
necessary. N-gram model is an approximation method, which ding. FFNNLM performs better than n-gram LM. Then, RNN
was the most widely used and the state-of-the-art model be- Language Model (RNNLM) [Mikolov et al., 2010] also was
fore NNLMs. A (k+1)-gram model is derived from the k- proposed. Since then, the NNLM has gradually become the
order Markov assumption. This assumption illustrates that mainstream LM and has rapidly developed. Long Short-term
the current state depends only on the previous k states, i.e., Memory RNN Language Model (LSTM-RNNLM) [Sunder-
P (wt |w1 · · · wt−1 ) ≈ P (wt |wt−k · · · wt−1 ), (2) meyer et al., 2012] was proposed for the difficulty of learning
long-term dependence. Various improvements were proposed
∗
K. Jing and J. Xu are corresponding authors. for reducing the cost of training and evaluation and PPL such
 P(wt |context) x(t)

softmax

U w(t)
tanh
W
U s(t) V
y(t)
H

C(wt−n+1 ) … C(wt−2 ) C(wt−1 )

delayed
s(t-1)
wt−n+1 wt−2 wt−1

Figure 1: The FFNNLM proposed by [Bengio et al., 2003]. In this

model, in order to predict the conditional probability of the word wt ,
its previous n − 1 words are projected by the shared projection ma- Figure 2: The RNNLM proposed by [Mikolov et al., 2010; Mikolov
trix C ∈ R|V |×m into a continuous feature vector space according et al., 2011]. The RNN has an internal state that changes with the
to their index in the vocabulary, where |V | is the size of the vocabu- input on each time step, taking into account all previous contexts.
lary and m is the dimension of the feature vectors, i.e., the word wi The state st can be derived from the input word vector wt and the
is projected as the distributed feature vector C(wi ) ∈ Rm . Each row state st−1 .
of the projection matrix C is a feature vector of a word in the vocab-
ulary. The input x of the FFNN is a concatenation of feature vectors
of n − 1 words. This model is followed by Softmax output layer where H, U , and W are the weight matrixes that is for the
to guarantee all the conditional probabilities of words positive and connections between the layers; d and b are the biases of the
summing to one. The learning algorithm is the Stochastic Gradient hidden layer and the output layer.
Descent (SGD) method using the backpropagation (BP) algorithm. FFNNLM implements modeling on continuous space by
learning a distributed representation for each word. The word
as hierarchical Softmax, caching, and so on. Recently, atten- representation is a by-product of LMs, which is used to im-
tion mechanisms have been introduced to improve NNLMs, prove other NLP tasks. Based on FFNNLM, two word repre-
which achieved significant performance improvements. sentation models, CBOW and Skip-gram , were proposed by
In this paper, we concentrate on reviewing the methods and [Mikolov et al., 2013]. FFNNLM overcomes the curse of di-
trends of NNLM. Classic NNLMs are described in Section 2. mensions by converting words into low-dimensional vectors.
Different types of improvements are introduced and analyzed FFNNLM leads the trend of NNLM research.
separately in Section 3. Corpora and toolkits are described in However, it still has a few drawbacks. The context size
Section 4 and Section 5. Finally, conclusions are given, and specified before training is limited, which is quite different
new research directions of NNLMs are discussed. from the fact that people can use lots of context informa-
tion to make predictions. Words in a sequence are time-
2 Classic Neural Network Language Models related. FFNNLM does not use timing information for mod-
eling. Moreover, fully connected NN needs to learn many
2.1 FFNN Language Models trainable parameters, even though these parameters are less
[Xu and Rudnicky, 2000] tried to introduce NNs into LMs. than n-gram LM, which still is expensive and inefficient.
Although their model performs better than the baseline n-
gram LM, their model with poor generalization ability cannot 2.2 RNN Language Models
capture context-dependent features due to no hidden layer. [Bengio et al., 2003] proposed the idea of using RNN for
According to Formula 1, the goal of LMs is equiv- LMs. They claimed that introducing more structure and pa-
alent to an evaluation of the conditional probability rameter sharing into NNs could capture longer contextual in-
P (wk |w1 · · · wk−1 ). But the FFNNs cannot directly process formation.
variable-length data and effectively represent the historical The first RNNLM was proposed by [Mikolov et al., 2010;
context. Therefore, for sequence modeling tasks like LMs, Mikolov et al., 2011a]. As shown in Figure 2, at time step t,
FFNNs have to use fixed-length inputs. Inspired by the n- the RNNLM can be described as:
gram LMs (see Formula 2), FFNNLMs consider the previous
n − 1 words as the context for predicting the next word. xt = [wtT ; sT T
t−1 ] ,
[Bengio et al., 2003] proposed the architecture of the orig- st = f (U xt + b),
inal FFNNLM, as shown in Figure 1. This FFNNLM can be yt = g(V st + d), (5)
expressed as:
where U , W , V are weight matrixes; b, d are the biases of the
y = b + W x + U tanh(d + Hx), (4) state layer and the output layer respectively; in [Mikolov et
al., 2010; Mikolov et al., 2011a], f is the sigmoid function, 3 Improved Techniques
and g is the Softmax function. RNNLMs could be trained
3.1 Techniques for Reducing Perplexity
by the backpropagation through time (BPTT) or the trun-
cated BPTT algorithm. According to their experiments, the New structures and more effective information are introduced
RNNLM is significantly superior to FFNNLMs and n-gram into classic NNLMs, especially LSTM-RNNLM, for reduc-
LMs in terms of PPL. ing PPL. Inspired by linguistics and how humans process
Compared with FFNNLM, RNNLM has made a break- natural language, some novel, effective methods, including
through. RNNs have an inherent advantage in the processing character-aware models, factored models, bidirectional mod-
of sequence data as variable-length inputs could be received. els, caching, attention, etc., are proposed.
When the network input window is shifted, duplicate calcu- Character-Aware Models
lations are avoided as the internal state. And the changes of In natural language, some words in the similar form often
the internal state by the input at t step reveal timing informa- have the same or similar meaning. For example, man in
tion. Parameter sharing significantly reduces the number of superman has the same meaning as the one in policeman.
parameters. [Mikolov et al., 2012] explored RNNLM and FFNNLM at the
Although RNNLMs could take advantage of all contexts character level. Character-level NNLM can be used for solv-
for prediction, it is challenging for training models to learn ing out-of-vocabulary (OOV) word problem, improving the
long-term dependencies. The reason is that the gradients of modeling of uncommon and unknown words because char-
parameters can disappear or explode during RNN training, acter features reveal structural similarities between words.
which leads to slow training or infinite parameter value. Character-level NNLM also reduces training parameters due
to the small Softmax layers with character-level outputs.
2.3 LSTM-RNN Language Models However, the experimental results showed that it was chal-
lenging to train highly-accurate character-level NNLMs, and
Long Short-Term Memory (LSTM) RNN solved this prob- its performance is usually worse than the word-level NNLMs.
lem. [Sundermeyer et al., 2012] introduced LSTM into LM This is because character-level NNLMs have to consider a
and proposed LSTM-RNNLM. Except for the memory unit longer history to predict the next word correctly.
and the part of NN, the architecture of LSTM-RNNLM is A variety of solutions that combine character- and word-
almost the same as RNNLM. Three gate structures (includ- level information, generally called character-aware LM, have
ing input, output, and forget gate) are added to the LSTM been proposed. One approach is to organize character-level
memory unit to control the flow of information. The general features word by word and then use them for word-level
architecture of LSTM-RNNLM can be formalized as: LMs. [Kim et al., 2015] proposed Convolutional Neural Net-
work (CNN) for extracting character-level feature and LSTM
it = σ(Ui xt + Wi st−1 + Vi ct−1 + bi ), for receiving these character-level features of the word in a
ft = σ(Uf xt + Wf st−1 + Vf ct−1 + bf ), time step. [Hwang and Sung, 2016] solved the problem of
gt = f (U xt + W st−1 + V ct−1 + b), character-level NNLMs using a hierarchical RNN architec-
ture consisting of multiple modules with different time scales.
ct = ft ct−1 + it gt , Another solution is to input character- and word-level features
ot = σ(Uo xt + Wo st−1 + Vo ct + bo ), into NNLM simultaneously. [Miyamoto and Cho, 2016] sug-
st = ot · f (ct ), gested interpolating word feature vectors with character fea-
yt = g(V st + M xt + d), (6) ture vectors extracted from words by BiLSTM and inputting
interpolation vectors into LSTM. [Verwimp et al., 2017] pro-
where it , ft , ot are input gate, forget gate and output gate, re- posed a character-word LSTM-RNNLM that directly con-
spectively. ct is the internal memory of unit. st is the hidden catenated character- and word-level feature vectors and in-
state unit. Ui , Uf , Uo , U , Wi , Wf , Wo , W , Vi , Vf , Vo , and V put concatenations into the network. Character-aware LM
are all weight matrixes. bi , bf , bo , b, and d are bias. f is the directly uses character-level LM as character feature extrac-
activation function and σ is the activation function for gates, tor for word-level LM. Therefore, the LM has rich character-
generally a sigmoid function. word information for prediction.
Comparing the above three classic LMs, RNNLMs (in- Factored Models
cluding LSTM-RNNLM) perform better than FFNNLM, and NNLMs define the similarity of words based on tokens. How-
LSTM-RNNLM maintains the state-of-the-art LM. The cur- ever, similarity can also be derived from word shape features
rent NNLMs are based mostly on RNN or LSTM. (affixes, uppercase, hyphens, etc.) or other annotations (such
Although LSTM-RNNLM performs well, training the as POS). Inspired by factored LMs, [Alexandrescu and Kirch-
model on a large corpus is very time-consuming because the hoff, 2006] proposed a factored NNLMs, a novel neural prob-
distributions of predicted words are explicitly normalized by abilistic LM that can learn the mapping from words and spe-
the Softmax layer, which leads to considering all words in cific features of the words to continuous spaces.
the vocabulary when computing the log-likelihood gradients. Many studies have explored the selection of factors. Dif-
Also, better performance of LM is expected. To improve ferent linguistic features are considered first. [Wu et al.,
NNLM, researchers are still exploring different techniques 2012] introduced morphological, grammatical, and seman-
that are similar to how humans process natural language. tic features to extend RNNLMs. [Adel et al., 2015] also ex-
plored the effects of other factors, such as part-of-speech tags, proposed a continuous cache model, where the change de-
Brown word clusters, open class words, and clusters of open pends on the inner product of the hidden representations.
class word embeddings. Experimental results showed that Another type of cache mechanism is that cache is used as
Brown word clusters, part-of-speech tags, and open words a speed-up technique for NNLMs. The main idea of this
are most effective for a Mandarin-English Code-Switching method is to store the outputs and states of LMs in a hash
task. Contextual information also was explored. For ex- table for future prediction given the same contextual history.
ample, [Mikolov and Zweig, 2012] used the distribution of For example, [Huang et al., 2014] proposed the use of four
topics calculated from fixed-length blocks of previous words. caches to accelerate model reasoning. The caches are respec-
[Wang and Cho, 2015] proposed a new approach to incor- tively Query to Language Model Probability Cache, History
porating corpus bag-of-words (BoW) context into language to Hidden State Vector Cache, History to Class Normalization
modeling. Besides, some methods based on text-independent Factor Cache, and History and Class Id to Sub-vocabulary
factors were proposed. [Ahn et al., 2016] proposed a Neural Normalization Factor Cache.
Knowledge Language Model that applied the notation knowl- The caching technique is proved that it can reduce compu-
edge provided by knowledge graph for RNNLMs. tational complexity and improve the learning ability of long-
The factored model allows the model to summarize word term dependence due to its caches. It can be applied flexibly
classes with same characteristics. Applying factors other than to existing models. However, if the size of the cache is lim-
word tokens to the neural network training can better learn the ited, cache-based NNLM will not perform well.
continuous representation of words, represent OOV words,
Attention
and reduce the PPL of LMs. However, the selection of dif-
ferent factors is related to different upstream NLP tasks, ap- RNNLMs predict the next word with its context. Not every
plications, etc. of LM. And there is no way to select factors word in the context is related to the next word and effective
in addition to experimenting with various factors separately. for prediction. Similar to human beings, LM with the atten-
Therefore, for a specific task, an efficient selection method of tion mechanism uses the long history efficiently by select-
factors is necessary. Meanwhile, corpora with factor labels ing useful word representations from them. [Bahdanau et al.,
have to be established. 2014] first proposed the application of the attention mecha-
nism to NLP tasks (machine translation in this paper). [Tran
Bidirectional Models et al., 2016; Mei et al., 2016] proved that the attention mech-
Traditional unidirectional NNs only predict the outputs from anism could improve the performance of RNNLMs.
past inputs. A bidirectional NN can be established, which is The attention mechanism obtains the target areas that need
conditional on future data. [Graves et al., 2013; Bahdanau to be focused on by a set of attention coefficients for each in-
et al., 2014] introduced bidirectional RNN and LSTM neural put. The attention vector zt is calculated by the representation
networks (BiRNN and BiLSTM) into speech recognition or {r0 , r1 , · · · , rt−1 } of tokens:
other NLP tasks. The BiRNNs utilize past and future con- t−1
X
texts by processing the input data in both directions. One zt = αti ri . (7)
of the most popular works of the bidirectional model is the i=0
ELMo model [Peters et al., 2018], a new deep contextualized
word representation based on BiLSTM-RNNLMs. The vec- This attention coefficient αti is normalized by the score eti
tors of the embedding layer of a pre-trained ELMo model is by the Softmax function, where
the learned representation vector of words in the vocabulary. eti = a(ht−1 , ri ), (8)
These representations are added as the embedding layer of
the existing model and significantly improve state of the art is an alignment model that evaluates how well the representa-
across six challenging NLP tasks. tion ri of a token and the hidden state ht−1 match. This atten-
Although BiLM using past and future contexts has tion vector is a good representation of the history of context
achieved improvements, it is noted that BiLM cannot be used for prediction.
directly for LM because LM is defined in the previous con- The attention mechanism has been widely used in Com-
text. BiLMs can be used for other NLP tasks, such as machine puter Vision (CV) and NLP. Many improved attention mech-
translation, speech recognition because the word sequence is anisms have been proposed, including soft/hard attention,
regarded as a simultaneous input sequence. global/local attention, key-value/key-value-predict attention,
multi-dimensional attention, directed self-attention, self-
Caching attention, multi-headed attention, and so on. These improved
Words that have appeared recently may appear again. Due methods are used in LMs and have improved the quality of
to this assumption, the cache mechanism initially is used to LMs.
optimize n-gram LM, overcoming the length limit of depen- On the basis of the above improved attention mechanisms,
dencies. It matches a new input and histories in the cache. some competitive LMs or methods of word representation are
Cache mechanism was originally proposed for reducing PPL proposed. Transformer proposed by [Vaswani et al., 2017]
of NNLMs. [Soutner et al., 2012] attempted to combine is the basis for the development of the subsequent models.
FFNNLM with the cache mechanism and proposed a struc- Transformer is a novel structure based entirely on the atten-
ture of the cache-based NNLMs, which leads to discrete prob- tion mechanism, which consists of an encoder and a decoder.
ability change. To solve this problem, [Grave et al., 2016] Since then, GPT [Radford et al., 2018] and BERT [Devlin
 p
et al., 2018] have been proposed. The main difference is uses d+1 Softmax layers with d+1 |V | classes or log2 |V | two
that GPT uses Transformer’s decoder, and BERT uses Trans- classification instead of one Softmax layers with |V | classes.
former’s encoder. BERT is an attention-based bidirectional Nevertheless, hierarchical Softmax leads NNLMs to perform
model. Unlike CBOW, Skip-gram, and ELMo, GPT and worse. Hierarchical Softmax based on Brown clustering is a
BERT represent words through the parameters of the entire special case. At the same time, the existing class-based meth-
model. They are state-of-the-art methods of word representa- ods do not consider the context. The classification is hard
tion in NLP. classification, which is a key factor of the increase of PPL.
Attention mechanism with its applications for various tasks Therefore, it is necessary to study a method based on soft
is one of the most popular research directions. Although vari- classification. It is our hope that while reducing the cost of
ous structures of attention mechanism have been proposed, as NNLM training, PPL will remain unchanged, even decrease.
the core of attention mechanism, the methods for calculating
the similarity between words have still not been improved. Sampling-based Approximations
Proposing some novel methods for vector similarity play an When the NNLMs calculate the conditional probability of the
important role in improving attention mechanism. next word, the output layer uses the Softmax function, where
the cost of calculation of the normalized denominator is ex-
3.2 Speed-up Techniques on Large Corpora tremely expensive. Therefore, one approach is to randomly or
Training the model on a corpus with a large vocabulary is heuristically select a small portion of the output and estimate
very time-consuming. The main reason is the Softmax layer the probability and the gradient from the samples.
for large vocabulary. Many approaches have been proposed Inspired by Minimizing Contrastive Divergence, [Bengio
to address the difficulty of training deep NNs with large out- and Senecal, 2003] proposed an importance sampling method
put spaces. In general, they can be divided into four cat- and an adaptive importance sampling algorithm to accelerate
egories, i.e., hierarchical Softmax, sampling-based approx- the training of NNLMs. The gradient of the log-likelihood
imations, self-normalization, and exact gradient on limited can be expressed as:
loss functions. Among them, the former two are used widely
in NNLMs. ∂logP (wt |w1t−1 ) ∂y(wt , w1t−1 )
=−
∂θ ∂θ
Hierarchical Softmax k
Some methods based on hierarchical Softmax that decompose
X ∂y(vi , w1t−1 )
+ P (vi |w1t−1 ) . (10)
target conditional probability into multiples of some condi- i=1
∂θ
tional probabilities are proposed. [Morin and Bengio, 2005]
used a hierarchical binary tree (by the similarity from Word- The weighted sum of the negative terms is obtained by im-
net) of an output layer, in which the V words in vocabulary portance sample estimates, i.e., sampling with Q instead of
are regarded as its leaves. This technique allows exponen- P . Therefore, the estimates of the normalized denominator
tially fast calculations of word probabilities and their gradi- and the log-likelihood gradient are respectively:
ents. However, it performs much worse than non-hierarchical 0
one despite using expert knowledge. [Mnih and Hinton, 1 X e−y(w ,ht )
Ẑ(ht ) = , (11)
2008] improved it by a simple feature-based algorithm for au- N Q(w0 |ht )
w0 Q(·|ht )
tomatically building word trees from data. The performance
of the above two models is mostly dependent on the tree, ∂logP (wt |w1t−1 ) ∂y(wt , w1t−1 )
E[ ]=−
which is usually heuristically constructed. ∂θ ∂θ
By relaxing the constraints of the binary tree structure, [Le ∂y(w0 ,w1t−1 ) −y(w0 ,wt−1 )
/Q(w0 |w1t−1 )
P
et al., 2013] introduced a new, general, better class-based w0 ∈Γ ∂θ e 1
+ t−1 . (12)
NNLM with a structured output layer, called Structured Out- 0
e−y(w ,w1 ) /Q(w 0 |w t−1 )
1
put Layer NNLM. Given a history h of a word wi , the condi-
tional probability can be formed as: Experimental results showed that adopting importance sam-
pling leads to ten times faster the training of NNLMs with-
D
Y out significantly increasing PPL. [Bengio and Senecal, 2008]
P (wt |h) = P (c1 (wt )|h) P (cd (wt )|h, c1 , · · · , cd−1 ). proposed an adaptive importance sampling method using an
d=2 adaptive n-gram model instead of the simple unigram model
(9) in [Bengio and Senecal, 2003]. Other improvements have
Since then, many scholars have improved this model. Hi- been proposed, such as parallel training of small models to
erarchical models based on word frequency classification estimate loss for importance sampling, multiple importance
[Mikolov et al., 2011a] and Brown clustering [Si et al., 2012] sampling and likelihood weighting scheme, two-stage sam-
were proposed. It was proved that the model with Brown pling, and so on. In addition, there are other different sam-
clustering performed better. [Zweig and Makarychev, 2013] pling methods for the training of NNLMs, including noise
proposed a speed optimal classification, i.e., a dynamic pro- comparison estimation, Locality Sensitive Hashing (LSH)
gramming algorithm that determines the classes by minimiz- techniques, BlackOut.
ing the running time of the model. These methods have significantly accelerated the training
Hierarchical Softmax significantly reduces model parame- of NNLMs by sampling, while the model evaluation remains
ters without increasing PPL. The reason is that the technique computationally challenging. At present, the computational
 Corpus Train Valid Test Vocab or an additional knowledge and accelerating the training and
evaluation by estimate conditional probability.
Brown 800,000 200,000 181,041 47,578 During the survey, some existing problems in NNLMs are
Penn Treebank 930,000 74,000 82,000 10,000 found and summarized. Therefore, we propose the future di-
WikiText-2 2000,000 220,000 250,000 33,000 rection of LMs. Firstly, the methods that reduce the cost and
the number of parameters would continue to be explored to
Table 1: Size (words) of small corpora speed up the training and evaluation without PPL increasing.
Then, a novel structure to simulate the way humans work is
complexity of the model with sampling-based approxima- expected for improving the performance of LM. For exam-
tions still is high. Sampling strategy is relatively simple. Ex- ple, building a generative model, such as GAN, for LMs may
cept for LSH techniques, other strategies are to select at ran- be a new direction. Last but not least, the current evaluation
dom or heuristically. system of LMs is not standardized. It is necessary to build
an evaluation benchmark for unifying the preprocessing and
what results should be reported in papers.
4 Corpora
As mentioned above, it is necessary to evaluate all LMs on the 7 Conclusion
same corpus, but which is impractical. This section describes The study of NNLMs has been going on for nearly two
some of the common corpora. decades. NNLMs have made timely and significant contribu-
In general, in order to reduce the cost of training and test, tions to NLP tasks. The different architectures of the classic
the feasibility of models needs to be verified on the small cor- NNLMs and their improvement are surveyed. Their related
pus first. Common small corpora include Brown, Penn Tree- corpora and toolkits that are essential for the study of NNLMs
bank, and WikiText-2 (see Table 1). are also introduced.
After the model structure is determined, it needs to be
trained and evaluated in a large corpus to prove that the model
has a reliable generalization. Common large corpora that are
References
updated over time from website, newspaper, etc. include Wall [Adel et al., 2015] H. Adel, N. T. Vu, K. Kirchhoff,
Street Journal, Wikipedia, News Commentary, News Crawl, D. Telaar, and T. Schultz. Syntactic and semantic features
Common Crawl, Associated Press (AP) News, and more. for code-switching factored language models. IEEE/ACM
However, LMs are often trained on different large corpora. Trans. Audio, Speech & Language Processing, 23(3):431–
Even on the same corpus, various preprocessing methods and 440, 2015.
different divisions of training/test set affect the results. At the [Ahn et al., 2016] S. Ahn, H. Choi, T. Pärnamaa, and Y. Ben-
same time, training time is reported in different ways or is not gio. A neural knowledge language model. CoRR,
given in some papers. The experimental results in different abs/1608.00318, 2016.
papers do not be compared fully. [Alexandrescu and Kirchhoff, 2006] A. Alexandrescu and
K. Kirchhoff. Factored neural language models. In HLT-
5 Toolkits NAACL, 2006.
Traditional LM toolkits mainly includes CMU-Cambridge [Bahdanau et al., 2014] D. Bahdanau, K. Cho, and Y. Ben-
SLM, SRILM, IRSTLM, MITLM, BerkeleyLM, which only gio. Neural machine translation by jointly learning to align
support the training and evaluation of n-gram LMs with a va- and translate. CoRR, abs/1409.0473, 2014.
riety of smoothing. With the development of Deep Learn- [Bengio and Senecal, 2003] Y. Bengio and J. Senecal. Quick
ing, many toolkits based on NNLMs are proposed. [Mikolov training of probabilistic neural nets by importance sam-
et al., 2011b] built the RNNLM toolkit, which supports the pling. In AISTATS, 2003.
training of RNNLMs to optimize speech recognition and ma-
chine translation, but it does not support parallel training al- [Bengio and Senecal, 2008] Y. Bengio and J. Senecal. Adap-
gorithms and GPU. [Schwenk, 2013] constructed the neural tive importance sampling to accelerate training of a neural
network open source tool CSLM (Continuous Space Lan- probabilistic language model. IEEE Trans. Neural Net-
guage Modeling) to support the training and evaluation of works, 19(4):713–722, 2008.
FFNNs. [Enarvi and Kurimo, 2016] proposed the scalable [Bengio et al., 2003] Y. Bengio, R. Ducharme, P. Vincent,
neural network model toolkit TheanoLM, which trains LMs and C. Janvin. A neural probabilistic language model.
to score sentences and generate text. JMLR, 3:1137–1155, 2003.
According to our survey, we found that there is no toolkit [Devlin et al., 2018] J. Devlin, M. Chang, K. Lee, and
supporting both the traditional N-gram LM and NNLM. And K. Toutanova. BERT: pre-training of deep bidirec-
they generally do not contain commonly used LM loads. tional transformers for language understanding. CoRR,
abs/1810.04805, 2018.
6 Future Directions [Enarvi and Kurimo, 2016] S. Enarvi and M. Kurimo.
Most NNLMs are based on three classic NNLMs. LSTM- Theanolm - an extensible toolkit for neural network
RNNLMs is the state-of-the-art LM. There are two directions language modeling. In Interspeech, pages 3052–3056,
of improving LMs, i.e., reducing PPL using a novel structure 2016.
[Grave et al., 2016] E. Grave, A. Joulin, and N. Usunier. Im- [Morin and Bengio, 2005] F. Morin and Y. Bengio. Hierar-
proving neural language models with a continuous cache. chical probabilistic neural network language model. In
CoRR, abs/1612.04426, 2016. Proc. of AISTATS, 2005.
[Graves et al., 2013] A. Graves, N. Jaitly, and A. Mohamed. [Peters et al., 2018] M. E. Peters, M. Neumann, M. Iyyer,
Hybrid speech recognition with deep bidirectional LSTM. M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. Deep
In IEEE Workshop on ASRU, pages 273–278, 2013. contextualized word representations. In Proc. of NAACL-
[Huang et al., 2014] Z. Huang, G. Zweig, and B. Dumoulin. HLT Volume 1 (Long Papers), pages 2227–2237, 2018.
Cache based recurrent neural network language model in- [Radford et al., 2018] A. Radford, K. Narasimhan, T. Sali-
ference for first pass speech recognition. In IEEE ICASSP, mans, and I. Sutskever. Improving Language Understand-
pages 6354–6358, 2014. ing by Generative Pre-Training. 2018.
[Hwang and Sung, 2016] K. Hwang and W. Sung. [Schwenk, 2013] H. Schwenk. CSLM - a modular open-
Character-level language modeling with hierarchical source continuous space language modeling toolkit. In
recurrent neural networks. CoRR, abs/1609.03777, 2016. INTERSPEECH, ISCA, pages 1198–1202, 2013.
[Jelinek et al., 1977] F. Jelinek, R. L. Mercer, L. R. Bahl, and [Si et al., 2012] Y. Si, Y. Guo, Y. Liu, J. Pan, and Y. Yan. Im-
J. K. Baker. Perplexity—a measure of the difficulty of pact of Word Classing on Recurrent Neural Network Lan-
speech recognition tasks. JASA, 62(S1):S63–S63, Decem- guage Model. In GCIS, pages 100–103, November 2012.
ber 1977. [Soutner et al., 2012] D. Soutner, Z. Loose, L. Müller, and
[Kim et al., 2015] Y. Kim, Y. Jernite, D. Sontag, and A. M. A. Prazák. Neural network language model with cache. In
Rush. Character-aware neural language models. CoRR, TSD, pages 528–534. 2012.
abs/1508.06615, 2015. [Sundermeyer et al., 2012] M. Sundermeyer, R. Schlüter,
[Le et al., 2013] H. S. Le, I. Oparin, A. Allauzen, J. Gauvain, and H. Ney. LSTM neural networks for language mod-
and F. Yvon. Structured output layer neural network lan- eling. In INTERSPEECH, pages 194–197, 2012.
guage models for speech recognition. IEEE Trans. Audio, [Tran et al., 2016] K. M. Tran, A. Bisazza, and C. Monz.
Speech & Language Processing, 21(1):195–204, 2013. Recurrent memory networks for language modeling. In
[Mei et al., 2016] H. Mei, M. Bansal, and M. R. Walter. NAACL HLT, pages 321–331, 2016.
Coherent dialogue with attention-based language models. [Vaswani et al., 2017] A. Vaswani, N. Shazeer, N. Parmar,
CoRR, abs/1611.06997, 2016. J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polo-
[Mikolov and Zweig, 2012] T. Mikolov and G. Zweig. Con- sukhin. Attention is all you need. In NIPS, pages 6000–
text dependent recurrent neural network language model. 6010, 2017.
In IEEE SLT, pages 234–239, 2012. [Verwimp et al., 2017] L. Verwimp, J. Pelemans, H. Van
[Mikolov et al., 2010] T. Mikolov, M. Karafiát, L. Burget, hamme, and P. Wambacq. Character-word LSTM lan-
guage models. In Proc. of EACL Volume 1: Long Papers,
J. Cernocký, and S. Khudanpur. Recurrent neural network
pages 417–427, 2017.
based language model. In INTERSPEECH, pages 1045–
1048, 2010. [Wang and Cho, 2015] T. Wang and K. Cho. Larger-context
[Mikolov et al., 2011a] T. Mikolov, S. Kombrink, L. Burget, language modelling. CoRR, abs/1511.03729, 2015.
J. Cernocký, and S. Khudanpur. Extensions of recurrent [Wu et al., 2012] Y. Wu, X. Lu, H. Yamamoto, S. Matsuda,
neural network language model. In Proc. of IEEE ICASSP, C. Hori, and H. Kashioka. Factored language model based
pages 5528–5531, 2011. on recurrent neural network. In COLING, Proc. of the
Conference: Technical Papers, pages 2835–2850, 2012.
[Mikolov et al., 2011b] T. Mikolov, S. Kombrink, A. Deo-
ras, and L. Burget. RNNLM - Recurrent Neural Network [Xu and Rudnicky, 2000] W. Xu and A. Rudnicky. Can arti-
Language Modeling Toolkit. In IEEE ASRU, page 4, 2011. ficial neural networks learn language models? In ICSLP /
INTERSPEECH, pages 202–205, 2000.
[Mikolov et al., 2012] T. Mikolov, I. Sutskever, A. Deoras,
H. Le, and S. Kombrink. Subword Language Modeling [Zweig and Makarychev, 2013] G. Zweig and
With Neural Networks. 2012. K. Makarychev. Speed regularization and optimality
in word classing. In IEEE ICASSP, pages 8237–8241,
[Mikolov et al., 2013] T. Mikolov, K. Chen, G. Corrado, and 2013.
J. Dean. Efficient estimation of word representations in
vector space. CoRR, abs/1301.3781, 2013.
[Miyamoto and Cho, 2016] Y. Miyamoto and Ky. Cho.
Gated word-character recurrent language model. In Proc.
of EMNLP, pages 1992–1997, 2016.
[Mnih and Hinton, 2008] A. Mnih and G. E. Hinton. A scal-
able hierarchical distributed language model. In Proc. of
NIPS, pages 1081–1088, 2008.