2019 IEEE 2nd International Conference on Computer and Communication Engineering Technology-CCET

Generating Text using Generative Adversarial Networks and Quick-Thought

Vectors

Feng Tian
David Russell and Longzhuang Li System Engineering Institute
Dept. of Computing Sciences Xi’an Jiaotong University
Texas A&M Uni.-Corpus Christi Xi’an, Shanxi, China 710049
Corpus Christi, TX, USA 78412 [email protected]
e-mail: [email protected]

Abstract—Generative Adversarial Networks (GANs) have been Skip-Thought model [12] is a method of embedding
shown to perform very well with the image generation tasks. sentences into vectors for use in NLP. Skip-Thoughts encode
Many advancements have been made with GANs over the past the whole sentences and the created vector holds the context
few years, which are making them more and more accurate in of a sentence. This means that entire sentences can be used to
their generation tasks. State-of-the-art methods of natural classify a context of the text instead of just words
language processing (NLP) involve word embeddings such as individually. The decoder for the sentence embeddings uses
global vectors for word representation (GLoVe) and word2vec. a greedy approach to build the sentence. It creates a sentence
These word embeddings help apply text data to a neural
from scratch one word at a time. This can be costly and may
network by converting the textual data to numbers that the
not always lead to a great result. Skip-Thought vectors has
networks could use. The main focus for GANs has been image
generation and in the past few years there have been research
been utilized in STGAN to generate text with good BLEU
works to apply GANs to the text generation task. This paper scores [4]. Quick-Thought vectors are the new method of
presents a Quick-Thought GAN (QTGAN) to generate sentence embedding and encode similarly to the Skip-
sentences by incorporating the Quick-Thought model. Quick- Thought model [11]. The Quick-Thought model can train
Thought vectors offer richer representations than prior faster and produce better results. When the embeddings are
unsupervised and supervised methods and enable a classifier to decoded it does not generate a sentence word by word as
distinguish context sentences from other contrastive sentences. Skip-Thought. Instead it classifies that embedding and grabs
The proposed QTGAN is trained on a portion of the the best matching pre-existing sentence from some candidate
BookCorpus dataset that has been converted to Quick- sentences. The Quick-Thought vector model is incorporated
Thought embeddings. The embeddings generated from the in the QTGAN proposed in this paper.
generator are then classified and used to pick a generated The rest of paper is organized as follows. In Section II,
sentence. BLEU scores are used to test the results of the related works in NLP and GANs are reviewed. In Section III,
training and compared to the Skip-Thought GAN. The a new Quick-Thought GAN (QTGAN) is proposed. In
increases in BLEU-3 and BLEU-4 scores were achieved with Section IV, the proposed QTGAN is trained by using the
the QTGAN. BookCorpus dataset, and the performance is compared with
the STGAN. In Section V, the future works are discussed. In
Keywords-generative adversarial network (GAN), word vector,
Section VI, we conclude the paper.
word embedding, quick thought vector, skip thought vector,
sentence embedding, natural language processing (NLP) II. RELATED WORK

I. INTRODUCTION A. Natural Language Processing

Deep learning has had many advances over the years and NLP is a field in Computer Science that deals with how
one of those is the generative adversarial network (GAN). computers process human languages. This field focuses on
GANs are a framework that use two neural networks that are research topics including speech recognition, language
going against each other. These two networks are the translation, and text generation. Speech recognition allows
generator and discriminator. The goal of the GAN is to get programs to be able to hear and understand human’s voice
the generator to fool the discriminator. This will allow the commands, languages translation targets on converting text
generator to produce samples that should be acceptable for from one language to another, and text generation leverages
use in whatever it was aimed at. In the deep learning, images knowledge in computational linguistics and artificial
and sound are transformed into high dimensional tensors that intelligence to automatically generate natural language texts.
allow them to be mathematically represented. To generate This work falls into the subfield of natural language
text, words also need to be able to be represented by generation or text generation.
embedding vectors using methods like word2vec [14] and
One important technique in NLP is word embeddings,
GLoVe [15]. This allows a neural network to be able to use
those word embeddings in tasks that relate to natural which are a type of word representation that allows words
language processing (NLP). with similar meaning to have a similar representation of a
dense vector of real numbers. Since word2vec [14] and
GloVe [15] were developed to obtain word embeddings,

978-1-7281-2871-9/19/$31.00 ©2019 IEEE 129

Authorized licensed use limited to: Amrita School of Engineering. Downloaded on February 19,2024 at 17:33:15 UTC from IEEE Xplore. Restrictions apply.
other methods, such as Skip-Thought embeddings [12] and small when it is generating bad samples and the
Quick-Thought embeddings [11], were proposed to perform discriminator is easily catching it. Then when it starts
better in practice. generating good samples and fooling the discriminator it
will have a much larger gradient signal. This means that it is
B. Generative Adversarial Networks
learning more when it is already generating good samples
Generative Adversarial Networks (GAN) were first than it was when it was generating bad samples. The
introduced by Goodfellow et al. in [10]. GANs are made up opposite scenario is desired. The generator should learn
of two neural networks. One called the generator and the more when generating bad samples than it does when
other called the discriminator. The goal of a GAN is to get generating good samples. To do this the loss function is
the generator network trained enough to be able to create changed for the generator to maximize the probability of the
good samples that appear to be authentic to the discriminator being incorrect. Equation 2 shows the change
discriminator. The generator and discriminator play what is to the generators function.
called a minimax game. The discriminator wants to
maximize the probability that it labels the data correctly
while the generator wants to minimize the probability that
the discriminator will detect the fake samples. GANs are
primarily used for computer vision tasks such as generating
a picture of a certain animal after it has been trained or even C. TextGAN
turning one animal into another like a horse to a zebra. One example would be TextGAN from [13] that uses a
GANs have excelled in the computer vision field and there GAN to generate text. The generator of the TextGAN is
has been a lot of work done in the NLP field. This paper made of a Long-Short-Term-Memory (LSTM), and the
implements a GAN to be trained on Quick-Thought discriminator is made of a Convolutional Neural Network
embeddings to generate text. (CNN). The TextGAN was trained using a random sample
1) Generator: The generator often referred to as G of of 1 million sentences from BookCorpus and 1 million
the GAN is what is being trained to create whatever the task sentences from the ArXiv dataset. The goal was for the
is. Whether that be images, music or text which is what the GAN to learn both informal and scientific writing. The
work aims at creating. The generator takes in a random result showed that a GAN can be used for text generation.
noise vector z and generates a sample from that random D. SeqGAN
noise vector. The goal is to be able to create samples that are SeqGAN proposed in [1] is another GAN that is used for
indistinguishable from real samples. In some the text generation task. It aims to solve two problems that
implementations the generator is trained slightly faster than GANs face in text generation. One being that it has
the discriminator to give the generator a slight edge in difficulties producing discrete tokens and the other problem
training. This prevents the discriminator from instantly being that the GAN scores its performance after the entire
being able to tell that the samples are fake. sequence has been generated. The model was trained using
2) Discriminator: The discriminator often referred to as Chinese poems, political speeches from the former US
D of the GAN is what is being trained to classify whether or President Obama and music samples. The SeqGAN model
not the sample it receives is real or fake. In addition to the performed well with its implementation in generating
vector it takes in an input that tells it whether or not the Chinese poems, political speeches and music.
current input is real or fake. In text generation, The
E. Skip-Thought GAN (STGAN)
discriminator takes a vector as input. The vector is either a
real piece of data or a generated piece of data. Skip-Thought vectors as proposed by [12] are a
3) Training Loss Function: The vanilla loss function for framework that uses the encoder-decoder model. The
a GAN as shown in Equation 1 is defined as the encoder takes the words of a sentence and encodes a
sentence vector instead of encoding each word embedding
discriminator maximizes the probability that a sample is
like word2vec or GLoVe. The Skip-Thought sentence
correctly classified, and the generator minimizes the
embedding is then decoded using a greedy algorithm for
probability that the discriminator will be correct [10]. This
choosing the next word in the sentence. This means that the
is applying gradient ascent on the discriminator and gradient Skip-Thought model constructs a brand-new sentence for
descent on the generator. The loss function is shown below each embedding which can be computationally costly. The
in Equation 1. STGAN is one of many GANs in [4] that make use of the
Skip-Thought Encoder-Decoder models. These models
include the vanilla GAN, a GAN with gradient policy
updates, as well as a wasserstein distance version. The
However there exists a problem with this function for the wasserstein distance is an alternative loss calculation that is
generator. The gradient signal for the generator will be aimed at improving the training. The model is trained using
sentences from the BookCorpus dataset that are encoded

130
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on February 19,2024 at 17:33:15 UTC from IEEE Xplore. Restrictions apply.
using the Skip-Thought encoder. These encodings are loss function for the GAN is the improved loss function
passed to the discriminator for training. The generator also shown above in Equation 2.
makes samples for the discriminator to evaluate. Once B. Quick-Though Encoder
training of the STGAN is complete the generator will
produce Skip-Thought sentence embeddings to be decoded The Quick-Thought model being used for the encoder is
by the Skip-Thought decoder, which will generate new the model from [11]. The model is inspired by the multi-
channel CNN model in [7]. The model is a concatenation of
sentences based on the embedding.
two bi-directional recurrent neural networks (RNNs). One of
III. PROPOSED WORK the RNNs uses a large vocabulary of around 3 million words
while the other uses a much smaller vocabulary of about 50
In this paper, the Quick-Thought GAN (QTGAN) is thousand words. The model uses GLoVe to kick start its
proposed, which is mostly close to the STGAN. The training. The model is shown to produce lower amounts of
QTGAN is trained using sentence embeddings from a redundant features than the bi-directional model discussed in
different model name Quick-Thoughts. Quick-Thoughts are [12].
the newer method of sentence encoding providing better
results on generating sentences. Once the training of the C. Quick-Thought Decoder
QTGAN has completed then it will be used for generating The Quick-Thought decoder differs from the Skip-
sentence embeddings and then decoding those embeddings Thought decoder in that it does not actually generate a new
into sentences using the Quick-Thought decoder. sentence. Instead it looks for the best sentences that fit the
The Quick-Thought GAN is implemented based on a embedding and then chooses the best out of those. So the
standard GAN model that is shown in [10]. The architecture decoder uses a set of sentence embeddings from the training
of the QTGAN can be seen in Figure 1. There are four corpus. In other words, the decoder classifies the current
major components making up the architecture, which are the embedding and then finds candidate sentences for that
Quick-Thought Encoder, Generator, Discriminator and the embedding. The candidate with the highest probability is
Quick-Thought Decoder. chosen as the new sentence (see Equation 3). For a given
sentence position in the context of s, the probability that a
A. Quick-Thought Vectors candidate sentence scand ϵ Scand is the correct sentence for that
Quick-Thought vectors as proposed by Logeswaran & position is computed in Equation 3 as follows:
Lee in [11] improve on the Skip-Thought framework. The
Quick-Thought vector framework uses the same encoder-
decoder model. The encoder takes a sentence and encodes it
into a vector just like the Skip-Thought framework. The
difference is in the decoder. The decoder of the Quick- where c is a scoring function, f and g the parametrized
Thought framework does not try to generate the words of the functions that encode an input sentence into a fixed length
sentences around the encoded sentence. Instead the decoder vector, and Scand a set of candidate sentences. This is where
will classify the sentence embedding and pick a sentence the computational cost is saved. It works directly in the
from a selection of candidate vectors to use. This saves vector space and does not generate its own sentences.
computational power as it works directly in the embedding Instead it works with pre-existing sentences from the corpus
vector space instead of using a greedy algorithm to and allows quick results of sentence embedding decoding.
generate sentences one word at a time like the Skip-Thought
model. Quick-Thought vectors will be used in this GAN. The D. Generator

Figure 1. Architecture of quick-thought GAN model

The generator of the Quick-Thought GAN is based on 100. The generator is made of four densely connected layers.
the GAN from [10]. The generator takes in as input a The first layer takes the vector of 100 as input and expands
random noise vector z. The length of the input vector z is it to 512. The second layer expands the 512 to 1024. The

131
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on February 19,2024 at 17:33:15 UTC from IEEE Xplore. Restrictions apply.
third layer will expand it to 2048. These first 3 layers use BookCorpus dataset and give you the ability to download the
LeakyReLU as their activation functions with an alpha of dataset in full. Then perform needed actions such as making
0.2. They also use batch normalization with a momentum of the dataset one sentence per line and tokenizing if necessary.
0.5. The final layer is the layer which expands the 2048 to Due to the RAM limitations in Colab using the Quick-
4800 which is our final size that matches the word Thought encoder only 64K sentences were used from the
embeddings of the Quick-Thought model. This final layer dataset. The Quick-Thought model being used is the pre-
uses a tanh activation function. These samples that the trained MC-QT model from [11].
generator creates is similar to the real embeddings from the B. Training
Quick-Thought model. The samples are then sent to the
As previously mentioned, the training was done on a
discriminator during training for evaluation. Once the
Google Colab Python 2 GPU instance using the pre-trained
training is complete then the generator is able to generate
model from [11]. The generator takes in a random noise
embeddings for use in the Quick-Thought decoding part of vector z of dimension 100, and outputs a sample vector that
the system, which produces newly generated sentences from has a dimension of 4800 since Both Skip-Thought and
the generated sentence embeddings. Quick-Thought bi-directional encoders produce embeddings
E. Discriminator with a dimension of 4800. The training uses batch sizes of
The discriminator is used for checking whether or not the 128 with 500 batches. This gives a total of 64,000 training
input it receives is real or fake. The input for the examples to use. Training was done for 5001 iterations for a
discriminator is a 4800-dimensional vector that represents a total of 10 epochs. Going above 10 epochs sometimes leads
sentence embedding. It also takes in another input that to undesirable results. After each epoch in the training
represents whether or not the provided embedding is real or session the generator produces 50 samples for evaluation. In
a fake sample from the generator. The discriminator addition to this it also saves the states of the discriminator
evaluates the samples it receives and determine the and generator models so they can be loaded later for more
authenticity of them during training of the generator. It does training. During the training the discriminator receives
this using 3 densely connected layers. The first layer shrinks samples from the real embeddings and the generated
the vector from 4800 to 1024, and the second layer shrinks embeddings along with labels to tell it if they are real or
it from 1024 to 512. The first and second layers use fake. The loss function used is Equation 2 that was shown
LeakyReLU as the activation function like the generator above.
with an alpha of 0.2. The final layer outputs a value from 0 C. Results
to 1 using a sigmoid activation function. This value The bilingual evaluation understudy BLEU scores are
represents whether it thinks the sample is real or fake and is used to compare the performance of QTGAN, STGAN and
used in the loss calculation as well. Once the training of the STGAN minibatch. The BLEU scores check to see how
generator is complete then the discriminator has served its much the candidate sentence which in this case is the
purpose and is no longer used. generated sentence resembles the sentences in the corpus.
The BLEU-2 is to check with groups of two words at a time
IV. EXPERIMENTS and see if those two words appear in the corpus in the same
The experiments in this paper were done using Google order and scores from that. The BLEU-3 and BLEU-4 are
Colab's services. A Python 2 run time was used with GPU very similar except they use groups of 3 and 4 respectively.
acceleration. This provided easy to use method of running The scores for BLEU-2, BLEU-3 and BLEU-4 are shown
code in a GPU accelerated environment. The run time below in Table I. The BLEU scores show an improvement
provides ~12GB of RAM and ~350GB of storage space. The when using the Quick-Thought model in all but BLEU-2 for
GPU provided by Google Colab is a Nvidia Tesla K80. the STGAN minibatch model. To be more specific, the
BLEU-2 score of QTGAN is 0.736, which is higher than that
A. Dataset of STGAN and lower than STGAN minibatch. For both
The dataset used in the experiments is the BookCorpus BLEU-3 and BLEU-4 scores, QTGAN achieves 0.684 and
dataset [16], which is made up of free books from 0.771, respectively, while STGAN offers 0.564 and 0.525
unpublished authors. It contains 16 different genres and and STGAN minibatch 0.607 and 0.531. The examples of
about 45 million sentences. This dataset is used in a lot of decoded sentences are shown in Table II.
text generation works for training so it will be used for
training this GAN as well. This dataset is no longer publicly TABLE I: BLUE SCORES FOR 50 SAMPLES AVERAGED
distributed so in order to obtain it you have to use a script to BookCorpus Reference
scrape the data from the smashwords website which is where Model BLEU-2 BLEU-3 BLEU-4
the books in the BookCorpus dataset come from. In order to
STGAN 0.709 0.564 0.525
do this, we used code from a GitHub repository by
Soskek 1 This repository contains a snapshot of the STGAN (minibatch) 0.745 0.607 0.531
QTGAN 0.736 0.684 0.771
1
https://github.com/soskek/bookcorpus

132
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on February 19,2024 at 17:33:15 UTC from IEEE Xplore. Restrictions apply.
 TABLE II: THE EXAMPLES OF DECODED SENTENCES shown that the QTGAN model does perform well after
Example Sentences training and have the ability to create Quick-Thought
“An all-inclusive wakeup call into collective hive greater good action.”
“He means is there anything in his life that could help us now.”
sentence embeddings. Decoding these embeddings and
"Almost out of nowhere, a figure emerged and started towards Ray." using the BLEU scoring system allows a comparison
"We had run from so much danger in that car, but now I had to keep dealing between the QTGAN and the STGAN. This comparison
with boring politics." showed that the QTGAN had better results than the STGAN.
"I wish Phoenix or even Elathan would come back and just… fix With higher scores in all tests but the BLEU-2 against the
everything.”
“Kenneth, can you help us find your brother?” STGAN that used minibatches. This shows that the QTGAN
is capable of producing good text results that are on par if
not better than the STGAN. In conclusion a GAN can be
V. FUTURE WORK trained on quick thought encodings and be able to produce
The proposed QTGAN can be expanded and changed in good results for text generation.
many ways for better results. The loss function currently REFERENCES
used is the original loss function from Equation 2. This can
[1] L. Yu, W. Zhang, J. Wang and Y. Yong, "SeqGAN: Sequence
be changed to something like the wasserstein distance Generative Adversarial Nets with Policy Gradient", in 31st AAAI
function in order to improve the loss function. The Conference on Artificial Intelligence, 2017.
wasserstein distance provides a better insight during the [2] J. Guo, S. Lu, H. Cai, W. Zhang, Y. Yu and J. Wang, “Long Text
training because it converges better which GANs tend to Generation via Adversarial Training with Leaked Information”, in
have issues doing. Another addition that could be done is a 32nd AAAI Conference on Artificial Intelligence, 2018.
[3] A. Fan, M. Lewis and Y. Dauphin, “Hierarchical Neural Story
change to the generator. Currently the generator is not Generation”, in 56th AMACL, 2018.
conditioned on any input. It just receives its random noise [4] A. Ahamad, “Generating Text through Adversarial Training using
vector and generates a sentence embedding based on that Skip-Thought Vectors”, in NAACL Student Research Workshop
random vector. Instead the conditional GAN model can be (SRW), 2019.
implemented to change this [6],[8],[9]. The conditional [5] P. Tambwekar, M. Dhuliawala, A. Mehta, L. Martin, B. Harrison and
GAN model allows the generated sample to be conditioned M. Riedl, “Controllable Neural Story Generation via Reinforcement
Learning”, arXiv preprint arXiv:1809.10736v1. 2018.
on an input. Once a sentence is encoded the sentence
[6] M Mirza and S. Osindero, "Conditional Generative Adversarial Nets",
embedding y can be concatenated to the random noise arXiv preprint arXiv:1411.1784v1. 2014.
vector z and then becomes the input to the GAN's generator. [7] Y. Kim, "Convolutional neural networks for sentence classification",
This will produce a sample that is conditioned on the input y. in Conference on Empirical Methods in Natural Language Processing,
For example, if a sentence about having a party is entered in 2014.
the conditional GAN, then the sentence that is decoded from [8] A. Dash, J. Gamboa, S Ahmed, M Liwicki and M. Afzal, “TAC-GAN
the generated embedding should have something to do with – Text Conditioned Auxiliary Classifier Generative Adversarial
Network”, arXiv preprint arXiv:1703.06412v2. 2017.
having a party. Once conditional sentences can be generated
[9] X. Deng, Y. Zhu and S. Newsam, “Using Conditional Generative
then another addition could be making sure that the future Adversarial Networks to Generate Ground-Level Views from
generated sentences are related to each other. In this way, Overhead Imagery”, arXiv preprint arXiv:1902.06923v1. 2019.
multiple sentences could be produced out of multiple [10] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,
encodings, which allow the model to write a complete short S. Ozair, A. Courville and Y. Bengio, "Generative Adversarial Nets",
in 28th Conference on Neural Information Processing Systems, 2014.
story [2],[3],[5]. More future work would be expanding the
[11] L. Logeswaran and H. Lee, "An Efficient Framework for Learning
training data to be more of the dataset. Because of the limit Sentence Representations", in ICLR, 2018.
imposed by Google Colab only a small portion of [12] R. Kiros, Y. Zhu, R. Salakhutdinov, R. Zemel, A. Torralba, R.
BookCorpus was used. It would be better to be able to use a Urtasun and S. Fidler, "Skip-Thought Vectors", in NIPS, 2015.
larger amount of BookCorpus. [13] Y. Zhang, Z. Gan and L. Carin, "Generating Text via Adversarial
Training", Workshop on Adversarial Training", in NIPS, 2016.
VI. CONCLUSION [14] T. Mikolov, K. Chen, G. Corrado and J. Dean, "Efficient Estimation
In the paper, the QTGAN is proposed and implemented of Word Representations in Vector Space", in International
Conference on Learning Representations, 2013.
by incorporating the Quick-Thought encoding and decoding
[15] J. Pennington, R. Socher and C. Manning, "GloVe: Global Vectors
models into it. The BookCorpus dataset was used as the for Word Representation", in International Conference on on
training data and encoded using the Quick-Thought encoder. Empirical Methods in Natural Language Processing, 2014.
During the training, the QTGAN used these encodings of [16] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Utrasun, A.
the sentences as input for the discriminator to help the Torrabla and S. Fidler, "Aligning Books and Movies: Towards Story-
like Visual Explanations by Watching Movies and Reading Books",
generator learn and produce better embeddings. It was in International Conference on Computer Vision, 2015.

133
Authorized licensed use limited to: Amrita School of Engineering. Downloaded on February 19,2024 at 17:33:15 UTC from IEEE Xplore. Restrictions apply.