Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17)

SeqGAN: Sequence Generative

Adversarial Nets with Policy Gradient

Lantao Yu,† Weinan Zhang,†∗ Jun Wang,‡ Yong Yu†

†
Shanghai Jiao Tong University, ‡ University College London
{yulantao,wnzhang,yyu}@apex.sjtu.edu.cn, [email protected]

Abstract of sequence increases. To address this problem, (Bengio et al.

2015) proposed a training strategy called scheduled sampling
As a new way of training generative models, Generative Ad- (SS), where the generative model is partially fed with its own
versarial Net (GAN) that uses a discriminative model to guide
synthetic data as prefix (observed tokens) rather than the true
the training of the generative model has enjoyed considerable
success in generating real-valued data. However, it has limi- data when deciding the next token in the training stage. Nev-
tations when the goal is for generating sequences of discrete ertheless, (Huszár 2015) showed that SS is an inconsistent
tokens. A major reason lies in that the discrete outputs from the training strategy and fails to address the problem fundamen-
generative model make it difficult to pass the gradient update tally. Another possible solution of the training/inference dis-
from the discriminative model to the generative model. Also, crepancy problem is to build the loss function on the entire
the discriminative model can only assess a complete sequence, generated sequence instead of each transition. For instance,
while for a partially generated sequence, it is non-trivial to in the application of machine translation, a task specific se-
balance its current score and the future one once the entire quence score/loss, bilingual evaluation understudy (BLEU)
sequence has been generated. In this paper, we propose a se- (Papineni et al. 2002), can be adopted to guide the sequence
quence generation framework, called SeqGAN, to solve the
generation. However, in many other practical applications,
problems. Modeling the data generator as a stochastic policy in
reinforcement learning (RL), SeqGAN bypasses the generator such as poem generation (Zhang and Lapata 2014) and chat-
differentiation problem by directly performing gradient policy bot (Hingston 2009), a task specific loss may not be directly
update. The RL reward signal comes from the GAN discrimi- available to score a generated sequence accurately.
nator judged on a complete sequence, and is passed back to General adversarial net (GAN) proposed by (Goodfellow
the intermediate state-action steps using Monte Carlo search. and others 2014) is a promising framework for alleviating the
Extensive experiments on synthetic data and real-world tasks above problem. Specifically, in GAN a discriminative net D
demonstrate significant improvements over strong baselines. learns to distinguish whether a given data instance is real or
not, and a generative net G learns to confuse D by generating
Introduction high quality data. This approach has been successful and
been mostly applied in computer vision tasks of generating
Generating sequential synthetic data that mimics the real one samples of natural images (Denton et al. 2015).
is an important problem in unsupervised learning. Recently, Unfortunately, applying GAN to generating sequences has
recurrent neural networks (RNNs) with long short-term two problems. Firstly, GAN is designed for generating real-
memory (LSTM) cells (Hochreiter and Schmidhuber 1997) valued, continuous data but has difficulties in directly generat-
have shown excellent performance ranging from natural lan- ing sequences of discrete tokens, such as texts (Huszár 2015).
guage generation to handwriting generation (Wen et al. 2015; The reason is that in GANs, the generator starts with random
Graves 2013). The most common approach to training an sampling first and then a determistic transform, govermented
RNN is to maximize the log predictive likelihood of each by the model parameters. As such, the gradient of the loss
true token in the training sequence given the previous ob- from D w.r.t. the outputs by G is used to guide the genera-
served tokens (Salakhutdinov 2009). However, as argued in tive model G (paramters) to slightly change the generated
(Bengio et al. 2015), the maximum likelihood approaches value to make it more realistic. If the generated data is based
suffer from so-called exposure bias in the inference stage: on discrete tokens, the “slight change” guidance from the
the model generates a sequence iteratively and predicts next discriminative net makes little sense because there is prob-
token conditioned on its previously predicted ones that may ably no corresponding token for such slight change in the
be never observed in the training data. Such a discrepancy be- limited dictionary space (Goodfellow 2016). Secondly, GAN
tween training and inference can incur accumulatively along can only give the score/loss for an entire sequence when it
with the sequence and will become prominent as the length has been generated; for a partially generated sequence, it is
∗
Weinan Zhang is the corresponding author. non-trivial to balance how good as it is now and the future
Copyright  c 2017, Association for the Advancement of Artificial score as the entire sequence.
Intelligence (www.aaai.org). All rights reserved. In this paper, to address the above two issues, we fol-

2852
low (Bachman and Precup 2015; Bahdanau et al. 2016) and to adjust the output continuously, which does not work on
consider the sequence generation procedure as a sequential discrete data generation (Goodfellow 2016).
decision making process. The generative model is treated On the other hand, a lot of efforts have been made to gen-
as an agent of reinforcement learning (RL); the state is the erate structured sequences. Recurrent neural networks can be
generated tokens so far and the action is the next token to trained to produce sequences of tokens in many applications
be generated. Unlike the work in (Bahdanau et al. 2016) that such as machine translation (Sutskever, Vinyals, and Le 2014;
requires a task-specific sequence score, such as BLEU in Bahdanau, Cho, and Bengio 2014). The most popular way of
machine translation, to give the reward, we employ a discrim- training RNNs is to maximize the likelihood of each token
inator to evaluate the sequence and feedback the evaluation in the training data whereas (Bengio et al. 2015) pointed
to guide the learning of the generative model. To solve the out that the discrepancy between training and generating
problem that the gradient cannot pass back to the generative makes the maximum likelihood estimation suboptimal and
model when the output is discrete, we regard the generative proposed scheduled sampling strategy (SS). Later (Huszár
model as a stochastic parametrized policy. In our policy gra- 2015) theorized that the objective function underneath SS is
dient, we employ Monte Carlo (MC) search to approximate improper and explained the reason why GANs tend to gen-
the state-action value. We directly train the policy (genera- erate natural-looking samples in theory. Consequently, the
tive model) via policy gradient (Sutton et al. 1999), which GANs have great potential but are not practically feasible to
naturally avoids the differentiation difficulty for discrete data discrete probabilistic models currently.
in a conventional GAN. As pointed out by (Bachman and Precup 2015), the se-
Extensive experiments based on synthetic and real data quence data generation can be formulated as a sequential
are conducted to investigate the efficacy and properties of decision making process, which can be potentially be solved
the proposed SeqGAN. In our synthetic data environment, by reinforcement learning techniques. Modeling the sequence
SeqGAN significantly outperforms the maximum likelihood generator as a policy of picking the next token, policy gradi-
methods, scheduled sampling and PG-BLEU. In three real- ent methods (Sutton et al. 1999) can be adopted to optimize
world tasks, i.e. poem generation, speech language generation the generator once there is an (implicit) reward function to
and music generation, SeqGAN significantly outperforms guide the policy. For most practical sequence generation
the compared baselines in various metrics including human tasks, e.g. machine translation (Sutskever, Vinyals, and Le
expert judgement. 2014), the reward signal is meaningful only for the entire
sequence, for instance in the game of Go (Silver et al. 2016),
Related Work the reward signal is only set at the end of the game. In those
cases, state-action evaluation methods such as Monte Carlo
Deep generative models have recently drawn significant at- (tree) search have been adopted (Browne et al. 2012). By
tention, and the ability of learning over large (unlabeled) data contract, our proposed SeqGAN extends GANs with the RL-
endows them with more potential and vitality (Salakhutdinov based generator to solve the sequence generation problem,
2009; Bengio et al. 2013). (Hinton, Osindero, and Teh 2006) where a reward signal is provided by the discriminator at the
first proposed to use the contrastive divergence algorithm end of each episode via Monte Carlo approach, and the gen-
to efficiently training deep belief nets (DBN). (Bengio et al. erator picks the action and learns the policy using estimated
2013) proposed denoising autoencoder (DAE) that learns the overall rewards.
data distribution in a supervised learning fashion. Both DBN
and DAE learn a low dimensional representation (encoding) Sequence Generative Adversarial Nets
for each data instance and generate it from a decoding net-
work. Recently, variational autoencoder (VAE) that combines The sequence generation problem is denoted as follows.
deep learning with statistical inference intended to represent Given a dataset of real-world structured sequences, train a
a data instance in a latent hidden space (Kingma and Welling θ-parameterized generative model Gθ to produce a sequence
2014), while still utilizing (deep) neural networks for non- Y1:T = (y1 , . . . , yt , . . . , yT ), yt ∈ Y, where Y is the vocabu-
linear mapping. The inference is done via variational methods. lary of candidate tokens. We interpret this problem based on
All these generative models are trained by maximizing (the reinforcement learning. In timestep t, the state s is the current
lower bound of) training data likelihood, which, as mentioned produced tokens (y1 , . . . , yt−1 ) and the action a is the next
by (Goodfellow and others 2014), suffers from the difficulty token yt to select. Thus the policy model Gθ (yt |Y1:t−1 ) is
of approximating intractable probabilistic computations. stochastic, whereas the state transition is deterministic af-
a
(Goodfellow and others 2014) proposed an alternative ter an action has been chosen, i.e. δs,s  = 1 for the next

training methodology to generative models, i.e. GANs, where state s = Y1:t if the current state s = Y1:t−1 and the action
the training procedure is a minimax game between a gener- a = yt ; for other next states s , δs,s a
 = 0.

ative model and a discriminative model. This framework Additionally, we also train a φ-parameterized discrimina-
bypasses the difficulty of maximum likelihood learning and tive model Dφ (Goodfellow and others 2014) to provide a
has gained striking successes in natural image generation guidance for improving generator Gθ . Dφ (Y1:T ) is a prob-
(Denton et al. 2015). However, little progress has been made ability indicating how likely a sequence Y1:T is from real
in applying GANs to sequence discrete data generation prob- sequence data or not. As illustrated in Figure 1, the discrimi-
lems, e.g. natural language generation (Huszár 2015). This is native model Dφ is trained by providing positive examples
due to the generator network in GAN is designed to be able from the real sequence data and negative examples from the

2853
 n n
G Next
action
MC
search D where Y1:t = (y1 , . . . , yt ) and Yt+1:T is sampled based on
True data
Reward the roll-out policy Gβ and the current state. In our experi-
Real World
State
Reward ment, Gβ is set the same as the generator, but one can use
Train
D a simplified version if the speed is the priority (Silver et al.
Reward
G
Generate 2016). To reduce the variance and get more accurate assess-
Reward
ment of the action value, we run the roll-out policy starting
Policy Gradient from current state till the end of the sequence for N times to
get a batch of output samples. Thus, we have:
Figure 1: The illustration of SeqGAN. Left: D is trained over G
QDθ (s = Y1:t−1 , a = yt ) = (4)
the real data and the generated data by G. Right: G is trained 
φ
1 N n ), n Gβ
by policy gradient where the final reward signal is provided N n=1 Dφ (Y1:T Y1:T ∈ MC (Y1:t ; N ) for t<T
by D and is passed back to the intermediate action value via Dφ (Y1:t ) for t = T,
Monte Carlo search.
where, we see that when no intermediate reward, the function
is iteratively defined as the next-state value starting from state
synthetic sequences generated from the generative model Gθ . s = Y1:t and rolling out to the end.
At the same time, the generative model Gθ is updated by em- A benefit of using the discriminator Dφ as a reward func-
ploying a policy gradient and MC search on the basis of the tion is that it can be dynamically updated to further improve
the generative model iteratively. Once we have a set of more
expected end reward received from the discriminative model realistic generated sequences, we shall re-train the discrimi-
Dφ . The reward is estimated by the likelihood that it would nator model as follows:
fool the discriminative model Dφ . The specific formulation
is given in the next subsection. min −EY ∼pdata [log Dφ (Y )] − EY ∼Gθ [log(1 − Dφ (Y ))]. (5)
φ

SeqGAN via Policy Gradient

Each time when a new discriminator model has been ob-
Following (Sutton et al. 1999), when there is no interme-
diate reward, the objective of the generator model (policy) tained, we are ready to update the generator. The proposed
Gθ (yt |Y1:t−1 ) is to generate a sequence from the start state policy based method relies upon optimizing a parametrized
s0 to maximize its expected end reward: policy to directly maximize the long-term reward. Following
 (Sutton et al. 1999), the gradient of the objective function
G
J(θ) = E[RT |s0 , θ] = Gθ (y1 |s0 ) · QDθφ (s0 , y1 ), (1) J(θ) w.r.t. the generator’s parameters θ can be derived as
y1 ∈Y

  
where RT is the reward for a complete sequence. Note that ∇θ J(θ) = EY1:t−1 ∼Gθ ∇θ Gθ (yt |Y1:t−1 ) · QG
Dφ (Y1:t−1 , yt ) .
θ

the reward is from the discriminator Dφ , which we will dis- yt ∈Y

cuss later. QG Dφ (s, a) is the action-value function of a se-

θ (6)
quence, i.e. the expected accumulative reward starting from The above form is due to the deterministic state transition
state s, taking action a, and then following policy Gθ . The and zero intermediate rewards. The detailed derivation is
rational of the objective function for a sequence is that start- provided in the supplementary material1 . Using likelihood
ing from a given initial state, the goal of the generator is to ratios (Glynn 1990; Sutton et al. 1999), we build an unbiased
generate a sequence which would make the discriminator estimation for Eq. (6) (on one episode):
consider it is real.
The next question is how to estimate the action-value
1  
T
function. In this paper, we use the REINFORCE algorithm ∇θ J(θ) 
G
∇θ Gθ (yt |Y1:t−1 ) · QDθ (Y1:t−1 , yt ) (7)
(Williams 1992) and consider the estimated probability of T φ
t=1 yt ∈Y
n
being real by the discriminator Dφ (Y1:T ) as the reward. For-
1  
T
mally, we have: G
= Gθ (yt |Y1:t−1 )∇θ log Gθ (yt |Y1:t−1 ) · QDθ (Y1:t−1 , yt )
T φ
t=1 yt ∈Y
G
QDθφ (a = yT , s = Y1:T −1 ) = Dφ (Y1:T ). (2)
1 
T
G
= Eyt ∼Gθ (yt |Y1:t−1 ) [∇θ log Gθ (yt |Y1:t−1 ) · QDθ (Y1:t−1 , yt )],
However, the discriminator only provides a reward value for T φ
t=1
a finished sequence. Since we actually care about the long-
term reward, at every timestep, we should not only consider where Y1:t−1 is the observed intermediate state sampled
the fitness of previous tokens (prefix) but also the resulted from Gθ . Since the expectation E[·] can be approximated by
future outcome. This is similar to playing the games such sampling methods, we then update the generator’s parameters
as Go or Chess where players sometimes would give up the as:
immediate interests for the long-term victory (Silver et al. θ ← θ + αh ∇θ J(θ), (8)
2016). Thus, to evaluate the action-value for an intermediate +
state, we apply Monte Carlo search with a roll-out policy Gβ where αh ∈ R denotes the corresponding learning rate
to sample the unknown last T − t tokens. We represent an at h-th step. Also the advanced gradient algorithms such as
N -time Monte Carlo search as Adam and RMSprop can be adopted here.
 
1 N
Y1:T , . . . , Y1:T = MCGβ (Y1:t ; N ), (3) 1
https://arxiv.org/abs/1609.05473

2854
Algorithm 1 Sequence Generative Adversarial Nets Short-Term Memory (LSTM) cells (Hochreiter and Schmid-
Require: generator policy Gθ ; roll-out policy Gβ ; discriminator huber 1997) to implement the update function g in Eq. (9). It
Dφ ; a sequence dataset S = {X1:T } is worth noticing that most of the RNN variants, such as the
1: Initialize Gθ , Dφ with random weights θ, φ. gated recurrent unit (GRU) (Cho et al. 2014) and soft atten-
2: Pre-train Gθ using MLE on S tion mechanism (Bahdanau, Cho, and Bengio 2014), can be
3: β ← θ used as a generator in SeqGAN.
4: Generate negative samples using Gθ for training Dφ
5: Pre-train Dφ via minimizing the cross entropy
6: repeat
The Discriminative Model for Sequences
7: for g-steps do Deep discriminative models such as deep neural network
8: Generate a sequence Y1:T = (y1 , . . . , yT ) ∼ Gθ (DNN) (Veselỳ et al. 2013), convolutional neural network
9: for t in 1 : T do (CNN) (Kim 2014) and recurrent convolutional neural net-
10: Compute Q(a = yt ; s = Y1:t−1 ) by Eq. (4) work (RCNN) (Lai et al. 2015) have shown a high perfor-
11: end for mance in complicated sequence classification tasks. In this
12: Update generator parameters via policy gradient Eq. (8)
paper, we choose the CNN as our discriminator as CNN
13: end for
14: for d-steps do has recently been shown of great effectiveness in text (to-
15: Use current Gθ to generate negative examples and com- ken sequence) classification (Zhang and LeCun 2015). Most
bine with given positive examples S discriminative models can only perform classification well
16: Train discriminator Dφ for k epochs by Eq. (5) for an entire sequence rather than the unfinished one. In this
17: end for paper, we also focus on the situation where the discriminator
18: β←θ predicts the probability that a finished sequence is real.2
19: until SeqGAN converges We first represent an input sequence x1 , . . . , xT as:
E1:T = x1 ⊕ x2 ⊕ . . . ⊕ xT , (11)
k
In summary, Algorithm 1 shows full details of the pro- where xt ∈ R is the k-dimensional token embedding and
posed SeqGAN. At the beginning of the training, we use ⊕ is the concatenation operator to build the matrix E1:T ∈
the maximum likelihood estimation (MLE) to pre-train Gθ RT ×k . Then a kernel w ∈ Rl×k applies a convolutional
on training set S. We found the supervised signal from the operation to a window size of l words to produce a new
pre-trained discriminator is informative to help adjust the feature map:
generator efficiently. ci = ρ(w ⊗ Ei:i+l−1 + b), (12)
After the pre-training, the generator and discriminator are where ⊗ operator is the summation of elementwise pro-
trained alternatively. As the generator gets progressed via duction, b is a bias term and ρ is a non-linear function.
training on g-steps updates, the discriminator needs to be re- We can use various numbers of kernels with different win-
trained periodically to keeps a good pace with the generator. dow sizes to extract different features. Finally we apply
When training the discriminator, positive examples are from a max-over-time pooling operation over the feature maps
the given dataset S, whereas negative examples are generated c̃ = max {c1 , . . . , cT −l+1 }.
from our generator. In order to keep the balance, the number To enhance the performance, we also add the highway ar-
of negative examples we generate for each d-step is the same chitecture (Srivastava, Greff, and Schmidhuber 2015) based
as the positive examples. And to reduce the variability of the on the pooled feature maps. Finally, a fully connected layer
estimation, we use different sets of negative samples com- with sigmoid activation is used to output the probability that
bined with positive ones, which is similar to bootstrapping the input sequence is real. The optimization target is to min-
(Quinlan 1996). imize the cross entropy between the ground truth label and
the predicted probability as formulated in Eq. (5).
The Generative Model for Sequences Detailed implementations of the generative and discrimi-
We use recurrent neural networks (RNNs) (Hochreiter and native models are provided in the supplementary material.
Schmidhuber 1997) as the generative model. An RNN
maps the input embedding representations x1 , . . . , xT of Synthetic Data Experiments
the sequence x1 , . . . , xT into a sequence of hidden states
h1 , . . . , hT by using the update function g recursively. To test the efficacy and add our understanding of SeqGAN,
we conduct a simulated test with synthetic data3 . To simulate
ht = g(ht−1 , xt ) (9) the real-world structured sequences, we consider a language
Moreover, a softmax output layer z maps the hidden states model to capture the dependency of the tokens. We use a
into the output token distribution randomly initialized LSTM as the true model, aka, the oracle,
to generate the real data distribution p(xt |x1 , . . . , xt−1 ) for
p(yt |x1 , . . . , xt ) = z(ht ) = softmax(c + V ht ), (10) the following experiments.
where the parameters are a bias vector c and a weight ma- 2
In our work, the generated sequence has a fixed length T , but
trix V . To deal with the common vanishing and exploding note that CNN is also capable of the variable-length sequence dis-
gradient problem (Goodfellow, Bengio, and Courville 2016) crimination with the max-over-time pooling technique (Kim 2014).
3
of the backpropagation through time, we leverage the Long Experiment code: https://github.com/LantaoYu/SeqGAN

2855
Evaluation Metric
Table 1: Sequence generation performance comparison. The
The benefit of having such oracle is that firstly, it provides
the training dataset and secondly evaluates the exact perfor- p-value is between SeqGAN and the baseline from T-test.
mance of the generative models, which will not be possible Algorithm Random MLE SS PG-BLEU SeqGAN
with real data. We know that MLE is trying to minimize the NLL 10.310 9.038 8.985 8.946 8.736
cross-entropy between the true data distribution p and our p-value < 10−6 < 10−6 < 10−6 < 10−6
approximation q, i.e. −Ex∼p log q(x). However, the most ac-
curate way of evaluating generative models is that we draw
some samples from it and let human observers review them
based on their prior knowledge. We assume that the human
observer has learned an accurate model of the natural distribu-
tion phuman (x). Then in order to increase the chance of pass-
ing Turing Test, we actually need to minimize the exact op-
posite average negative log-likelihood −Ex∼q log phuman (x)
(Huszár 2015), with the role of p and q exchanged. In our
synthetic data experiments, we can consider the oracle to be
the human observer for real-world problems, thus a perfect
evaluation metric should be Figure 2: Negative log-likelihood convergence w.r.t. the train-

T  ing epochs. The vertical dashed line represents the end of
NLLoracle = −EY1:T ∼Gθ log Goracle (yt |Y1:t−1 ) , (13) pre-training for SeqGAN, SS and PG-BLEU.
t=1

where Gθ and Goracle denote our generative model and the

oracle respectively. Results
At the test stage, we use Gθ to generate 100,000 sequence The NLLoracle performance of generating sequences from the
samples and calculate NLLoracle for each sample by Goracle compared policies is provided in Table 1. Since the evaluation
and their average score. Also significance tests are performed metric is fundamentally instructive, we can see the impact
to compare the statistical properties of the generation perfor- of SeqGAN, which outperforms other baselines significantly.
mance between the baselines and SeqGAN. A significance T-test on the NLLoracle score distribution of
the generated sequences from the compared models is also
Training Setting performed, which demonstrates the significant improvement
To set up the synthetic data experiments, we first initialize of SeqGAN over all compared models.
the parameters of an LSTM network following the normal The learning curves shown in Figure 2 illustrate the superi-
distribution N (0, 1) as the oracle describing the real data dis- ority of SeqGAN explicitly. After about 150 training epochs,
tribution Goracle (xt |x1 , . . . , xt−1 ). Then we use it to generate both the maximum likelihood estimation and the schedule
10,000 sequences of length 20 as the training set S for the sampling methods converge to a relatively high NLLoracle
generative models. score, whereas SeqGAN can improve the limit of the genera-
In SeqGAN algorithm, the training set for the discriminator tor with the same structure as the baselines significantly. This
is comprised by the generated examples with the label 0 and indicates the prospect of applying adversarial training strate-
the instances from S with the label 1. For different tasks, one gies to discrete sequence generative models to breakthrough
should design specific structure for the convolutional layer the limitations of MLE. Additionally, SeqGAN outperforms
and in our synthetic data experiments, the kernel size is from PG-BLEU, which means the discriminative signal in GAN
1 to T and the number of each kernel size is between 100 to is more general and effective than a predefined score (e.g.
2004 . Dropout (Srivastava et al. 2014) and L2 regularization BLEU) to guide the generative policy to capture the underly-
are used to avoid over-fitting. ing distribution of the sequence data.
Four generative models are compared with SeqGAN. The
first model is a random token generation. The second one Discussion
is the MLE trained LSTM Gθ . The third one is scheduled
sampling (Bengio et al. 2015). The fourth one is the Policy In our synthetic data experiments, we find that the stability
Gradient with BLEU (PG-BLEU). In the scheduled sampling, of SeqGAN depends on the training strategy. More specifi-
the training process gradually changes from a fully guided cally, the g-steps, d-steps and k parameters in Algorithm 1
scheme feeding the true previous tokens into LSTM, towards have a large effect on the convergence and performance of
a less guided scheme which mostly feeds the LSTM with its SeqGAN. Figure 3 shows the effect of these parameters. In
generated tokens. A curriculum rate ω is used to control the Figure 3(a), the g-steps is much larger than the d-steps and
probability of replacing the true tokens with the generated epoch number k, which means we train the generator for
ones. To get a good and stable performance, we decrease ω by many times until we update the discriminator. This strategy
0.002 for every training epoch. In the PG-BLEU algorithm, leads to a fast convergence but as the generator improves
we use BLEU, a metric measuring the similarity between a quickly, the discriminator cannot get fully trained and thus
generated sequence and references (training data), to score will provide a misleading signal gradually. In Figure 3(b),
the finished samples from Monte Carlo search. with more discriminator training epochs, the unstable training
process is alleviated. In Figure 3(c), we train the generator
4
Implementation details are in the supplementary material. for only one epoch and then before the discriminator gets

2856
 Table 2: Chinese poem generation performance comparison.
Algorithm Human score p-value BLEU-2 p-value
MLE 0.4165 0.6670
0.0034 < 10−6
SeqGAN 0.5356 0.7389
Real data 0.6011 0.746

(a) g-steps=100, d-steps=1, (b) g-steps=30, d-steps=1, Table 3: Obama political speech generation performance.
k=10 k=30 Algorithm BLEU-3 p-value BLEU-4 p-value
MLE 0.519 −6 0.416
< 10 0.00014
SeqGAN 0.556 0.427

Table 4: Music generation performance comparison.

Algorithm BLEU-4 p-value MSE p-value
MLE 0.9210 −6 22.38
< 10 0.00034
SeqGAN 0.9406 20.62

(c) g-steps=1, d-steps=1, k=10 (d) g-steps=1, d-steps=5, k=3

twenty characters in total. To focus on a fully automatic so-
Figure 3: Negative log-likelihood convergence performance lution and stay general, we did not use any prior knowledge
of SeqGAN with different training strategies. The vertical of special structure rules in Chinese poems such as specific
dashed line represents the beginning of adversarial training. phonological rules. In the Obama political speech genera-
tion task, we use a corpus6 , which is a collection of 11,092
paragraphs from Obama’s political speeches.
fooled, we update it immediately based on the more realistic We use BLEU score as an evaluation metric to measure
negative examples. In such a case, SeqGAN learns stably. the similarity degree between the generated texts and the
The d-steps in all three training strategies described above human-created texts. BLEU is originally designed to auto-
is set to 1, which means we only generate one set of neg- matically judge the machine translation quality (Papineni et
ative examples with the same number as the given dataset, al. 2002). The key point is to compare the similarity between
and then train the discriminator on it for various k epochs. the results created by machine and the references provided
But actually we can utilize the potentially unlimited num- by human. Specifically, for poem evaluation, we set n-gram
ber of negative examples to improve the discriminator. This to be 2 (BLEU-2) since most words (dependency) in classical
trick can be considered as a type of bootstrapping, where Chinese poems consist of one or two characters (Yi, Li, and
we combine the fixed positive examples with different neg- Sun 2016) and for the similar reason, we use BLEU-3 and
ative examples to obtain multiple training sets. Figure 3(d) BLEU-4 to evaluate Obama speech generation performance.
shows this technique can improve the overall performance In our work, we use the whole test set as the references in-
with good stability, since the discriminator is shown more stead of trying to find some references for the following line
negative examples and each time the positive examples are given the previous line (He, Zhou, and Jiang 2012). The rea-
emphasized, which will lead to a more comprehensive guid- son is in generation tasks we only provide some positive
ance for training generator. This is in line with the theorem in examples and then let the model catch the patterns of them
(Goodfellow and others 2014). When analyzing the conver- and generate new ones. In addition to BLEU, we also choose
gence of generative adversarial nets, an important assumption poem generation as a case for human judgement since a poem
is that the discriminator is allowed to reach its optimum given is a creative text construction and human evaluation is ideal.
G. Only if the discriminator is capable of differentiating real Specifically, we mix the 20 real poems and 20 each gener-
data from unnatural data consistently, the supervised signal ated from SeqGAN and MLE. Then 70 experts on Chinese
from it can be meaningful and the whole adversarial training poems are invited to judge whether each of the 60 poem is
process can be stable and effective. created by human or machines. Once regarded to be real, it
gets +1 score, otherwise 0. Finally, the average score for each
Real-world Scenarios algorithm is calculated.
To complement the previous experiments, we also test Se- The experiment results are shown in Tables 2 and 3, from
qGAN on several real-world tasks, i.e. poem composition, which we can see the significant advantage of SeqGAN over
speech language generation and music generation. the MLE in text generation. Particularly, for poem composi-
tion, SeqGAN performs comparably to real human data.
Text Generation
For text generation scenarios, we apply the proposed Seq-
Music Generation
GAN to generate Chinese poems and Barack Obama political For music composition, we use Nottingham7 dataset as our
speeches. In the poem composition task, we use a corpus5 training data, which is a collection of 695 music of folk tunes
of 16,394 Chinese quatrains, each containing four lines of 6
https://github.com/samim23/obama-rnn
5 7
http://homepages.inf.ed.ac.uk/mlap/Data/EMNLP14/ http://www.iro.umontreal.ca/∼lisa/deep/data

2857
in midi file format. We study the solo track of each music. Goodfellow, I. 2016. Generative adversarial networks for text.
In our work, we use 88 numbers to represent 88 pitches, http://goo.gl/Wg9DR7.
which correspond to the 88 keys on the piano. With the pitch Graves, A. 2013. Generating sequences with recurrent neural
sampling for every 0.4s8 , we transform the midi files into networks. arXiv:1308.0850.
sequences of numbers from 1 to 88 with the length 32. He, J.; Zhou, M.; and Jiang, L. 2012. Generating chinese classical
To model the fitness of the discrete piano key patterns, poems with statistical machine translation models. In AAAI.
BLEU is used as the evaluation metric. To model the fitness Hingston, P. 2009. A turing test for computer game bots. IEEE
of the continuous pitch data patterns, the mean squared error TCIAIG 1(3):169–186.
(MSE) (Manaris et al. 2007) is used for evaluation. Hinton, G. E.; Osindero, S.; and Teh, Y.-W. 2006. A fast learning
From Table 4, we see that SeqGAN outperforms the MLE algorithm for deep belief nets. Neural computation 18(7):1527–
significantly in both metrics in the music generation task. 1554.
Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory.
Conclusion Neural computation 9(8):1735–1780.
Huszár, F. 2015. How (not) to train your generative model: Sched-
In this paper, we proposed a sequence generation method, uled sampling, likelihood, adversary? arXiv:1511.05101.
SeqGAN, to effectively train generative adversarial nets for
Kim, Y. 2014. Convolutional neural networks for sentence classifi-
structured sequences generation via policy gradient. To our cation. arXiv:1408.5882.
best knowledge, this is the first work extending GANs to
Kingma, D. P., and Welling, M. 2014. Auto-encoding variational
generate sequences of discrete tokens. In our synthetic data
bayes. ICLR.
experiments, we used an oracle evaluation mechanism to
Lai, S.; Xu, L.; Liu, K.; and Zhao, J. 2015. Recurrent convolutional
explicitly illustrate the superiority of SeqGAN over strong
neural networks for text classification. In AAAI, 2267–2273.
baselines. For three real-world scenarios, i.e., poems, speech
language and music generation, SeqGAN showed excellent Manaris, B.; Roos, P.; Machado, P.; et al. 2007. A corpus-based
hybrid approach to music analysis and composition. In NCAI,
performance on generating the creative sequences. We also volume 22, 839.
performed a set of experiments to investigate the robustness
Papineni, K.; Roukos, S.; Ward, T.; and Zhu, W.-J. 2002. Bleu: a
and stability of training SeqGAN. For future work, we plan
method for automatic evaluation of machine translation. In ACL,
to build Monte Carlo tree search and value network (Silver et 311–318.
al. 2016) to improve action decision making for large scale
Quinlan, J. R. 1996. Bagging, boosting, and c4. 5. In AAAI/IAAI,
data and in the case of longer-term planning. Vol. 1, 725–730.
Salakhutdinov, R. 2009. Learning deep generative models. Ph.D.
References Dissertation, University of Toronto.
Bachman, P., and Precup, D. 2015. Data generation as sequential Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; et al.
decision making. In NIPS, 3249–3257. 2016. Mastering the game of go with deep neural networks and tree
Bahdanau, D.; Brakel, P.; Xu, K.; et al. 2016. An actor-critic search. Nature 529(7587):484–489.
algorithm for sequence prediction. arXiv:1607.07086. Srivastava, N.; Hinton, G. E.; Krizhevsky, A.; Sutskever, I.; and
Bahdanau, D.; Cho, K.; and Bengio, Y. 2014. Neural machine trans- Salakhutdinov, R. 2014. Dropout: a simple way to prevent neural
lation by jointly learning to align and translate. arXiv:1409.0473. networks from overfitting. JMLR 15(1):1929–1958.
Bengio, Y.; Yao, L.; Alain, G.; and Vincent, P. 2013. Generalized Srivastava, R. K.; Greff, K.; and Schmidhuber, J. 2015. Highway
denoising auto-encoders as generative models. In NIPS, 899–907. networks. arXiv:1505.00387.
Bengio, S.; Vinyals, O.; Jaitly, N.; and Shazeer, N. 2015. Scheduled Sutskever, I.; Vinyals, O.; and Le, Q. V. 2014. Sequence to sequence
sampling for sequence prediction with recurrent neural networks. learning with neural networks. In NIPS, 3104–3112.
In NIPS, 1171–1179. Sutton, R. S.; McAllester, D. A.; Singh, S. P.; Mansour, Y.; et al.
Browne, C. B.; Powley, E.; Whitehouse, D.; Lucas, S. M.; et al. 1999. Policy gradient methods for reinforcement learning with
2012. A survey of monte carlo tree search methods. IEEE TCIAIG function approximation. In NIPS, 1057–1063.
4(1):1–43. Veselỳ, K.; Ghoshal, A.; Burget, L.; and Povey, D. 2013. Sequence-
discriminative training of deep neural networks. In INTERSPEECH,
Cho, K.; Van Merriënboer, B.; Gulcehre, C.; et al. 2014. Learning
2345–2349.
phrase representations using RNN encoder-decoder for statistical
machine translation. EMNLP. Wen, T.-H.; Gasic, M.; Mrksic, N.; Su, P.-H.; Vandyke, D.; and
Young, S. 2015. Semantically conditioned LSTM-based natural lan-
Denton, E. L.; Chintala, S.; Fergus, R.; et al. 2015. Deep generative
guage generation for spoken dialogue systems. arXiv:1508.01745.
image models using a laplacian pyramid of adversarial networks. In
NIPS, 1486–1494. Williams, R. J. 1992. Simple statistical gradient-following algo-
rithms for connectionist reinforcement learning. Machine learning
Glynn, P. W. 1990. Likelihood ratio gradient estimation for stochas- 8(3-4):229–256.
tic systems. Communications of the ACM 33(10):75–84.
Yi, X.; Li, R.; and Sun, M. 2016. Generating chinese classical
Goodfellow, I., et al. 2014. Generative adversarial nets. In NIPS, poems with RNN encoder-decoder. arXiv:1604.01537.
2672–2680.
Zhang, X., and Lapata, M. 2014. Chinese poetry generation with
Goodfellow, I.; Bengio, Y.; and Courville, A. 2016. Deep learning. recurrent neural networks. In EMNLP, 670–680.
2015.
Zhang, X., and LeCun, Y. 2015. Text understanding from scratch.
8
http://deeplearning.net/tutorial/rnnrbm.html arXiv:1502.01710.

2858