Scribd
Upload
39 views

08 NLP With Deep Learning

This document provides a 6 step process for using deep learning to generate new text based on a corpus of existing text data: 1) Read in text data and understand the characters 2) Process the text by vectorizing it and creating an encoding dictionary 3) Create batches of text sequences for the neural network 4) Build the neural network model with embedding, GRU, and dense layers 5) Train the model on the batches 6) Generate new text by loading the trained model and running a text generation loop

Uploaded by

조동올
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
39 views

08 NLP With Deep Learning

This document provides a 6 step process for using deep learning to generate new text based on a corpus of existing text data: 1) Read in text data and understand the characters 2) Process the text by vectorizing it and creating an encoding dictionary 3) Create batches of text sequences for the neural network 4) Build the neural network model with embedding, GRU, and dense layers 5) Train the model on the batches 6) Generate new text by loading the trained model and running a text generation loop

Uploaded by

조동올
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 31

NLP with Deep

Learning
Deep Learning

● Let’s explore how to work with text data in


conjunction with deep learning!
● This is a natural extension of the time series
and recurrent neural network topics we
just discussed.
Deep Learning

● We will create a neural network that will


generate new text based on a corpus of
text data.
● Check out “The Unreasonable Effectiveness
of RNNs” by Andrej Karpathy
● So how will this work?
Deep Learning

● Given an input string sequence, predict the


sequence shifted forward 1 character.
○ [“h” , ”e” , ”l” , ”l” ]
○ [“e” , ”l” , ”l” , ”o”]
Deep Learning

● The character based RNN will actually learn


the structure of the text.
● In our example we will use the works of
William Shakespeare.
● We will see the network clearly learn play
writing structure and spacing, just from a
character level!
Deep Learning
Deep Learning

● Step 1: Read in Text Data


○ We can use basic built in python
commands to read in a corpus of text
as string data.
○ Note, you should have a large data set
for this, at least 1 million characters for
realistic results.
Deep Learning

● Step 2: Text Processing and Vectorization


○ The neural network can’t take in raw
strings, so we will encode them each
to an integer.
■ A:1
■ B:2
■ C:3
■ ? : 55
Deep Learning

● Step 3: Creating Batches


○ We’ll use Tensorflow’s dataset object
to easily create batches of text
sequences.
■ [“h”, “e” , “l” , “l” , “o” , “ “, “m”]
■ [,“e” , “l” , “l” , “o” , “ “, “m”, “y”]
Deep Learning

● Step 3: Creating Batches


○ We’ll want to use sequence lengths
that are long enough to capture
structure and previous words.
○ But not so long that the sequence is
just historical noise.
Deep Learning

● Step 4: Creating the Model


○ We’ll use 3 layers
■ Embedding
■ GRU
■ Dense
Deep Learning

● Step 4: Creating the Model


○ Embedding Layer turns positive
integers (indexes) into dense vectors
of fixed size. eg. [[4], [20]] -> [[0.25,
0.1,0.3], [0.6, -0.2,0.9]]
○ Its up to the user to choose the
number of embedding dimensions.
Deep Learning

● Step 4: Creating the Model


○ GRU
■ Gated Recurrent Unit is a special
type of recurrent neuron unit.
■ The GRU is like a long short-term
memory (LSTM) with forget gate
but has fewer parameters than
LSTM, as it lacks an output gate.
Gated Recurrent Unit (GRU)
ht

ht-1 Ct
X
+
X rt 1- zt X

σ σ tanh

xt
Deep Learning

● Step 4: Creating the Model


○ Dense Layer
■ One neuron per character.
■ Character labels will be one hot
encoded so the final dense layer
produces a probability per
character.
Deep Learning

● Step 4: Creating the Model


○ Dense Layer
■ Probability per character means we
can play around with
“temperature”:
● Choosing less probable
characters more/less often
Deep Learning

● Step 5: Training the Model


○ We’ll set up our batches and make
sure to one-hot encode our character
labels.
Deep Learning

● Step 6: Generating new text


○ We’ll save our models weights and
show you how to reload a model’s
weights with a different batch size in
order to pass in single examples.
Let’s get started!
Text Generation
With Python and Keras
Part One
Deep Learning

● Part 1: The Data


○ Import main libraries
○ Importing Text
○ Understanding Characters
Text Generation
With Python and Keras
Part Two
Deep Learning

● Part 2: Text Processing


○ Vectorize the text
○ Create encoding dictionary
Text Generation
With Python and Keras
Part Three
Deep Learning

● Step 3: Creating Batches


○ Understand text sequences
○ Use Tensorflow datasets to generate
batches
○ Shuffle batches
Text Generation
With Python and Keras
Part Four
Deep Learning

● Step 4: Creating the Model


○ Set up loss function
○ Create Model
■ Embedding
■ GRU
■ Dense
Text Generation
With Python and Keras
Part Five
Deep Learning

● Step 5: Training the Model


○ We’ll quickly show an example of how to
train the model.
○ We’ll also show you how to load our
provided saved model file.
Text Generation
With Python and Keras
Part Six
Deep Learning

● Step 6: Generating Text


○ We’ll load our model
○ Adjust batch size to 1
○ Run a loop that generates new text

You might also like