Module 8: Natural language processing Pt 1

Natural
Language
Processing Pt. 1

This course content is being actively
developed by Delta Analytics, a 501(c)3
Bay Area nonprofit that aims to
empower communities to leverage their
data for good.
Please reach out with any questions or
feedback to inquiry@deltanalytics.org.
Find out more about our mission here.
Delta Analytics builds technical capacity
around the world.

Module 8:
Natural Language
Processing

❏ What is Natural Language Processing and its applications?
❏ What is text as data?
❏ Bag-of-words, vectors, matrices
❏ Supervised learning algorithms
❏ Text classification (Naive Bayes)
❏ Next steps for NLP
Module Checklist

What is natural language
processing?

Natural language vs. artificial language
Natural language is defined as “a language that has developed naturally in use (as
contrasted with an artificial language or computer code.)”

In one area, humans still beat computers:
Natural language!
Reading, writing, communicating and otherwise
working with unstructured data.
Source: http://imgs.xkcd.com/comics/progeny.png

All of our examples in the course have thus far involved
modeling structured data.
Why is NLP so difficult?
Structured data (data frame)
No
No
Yes
Yes
Predicted
Approval for
credit card
Y Y*
12,000
60,000
11,000
200,000
Current
Annual
Income ($)
Approval for
credit card
Yes
No
No
Yes
Outstanding
Debt
200
60000
0
10000

Our algorithms know how to pull information from structured data but are
unable to interpret unstructured data. How do we pull information from
unstructured data?
NLP is difficult because text is an example of
unstructured data.
Information
Structured data
Unstructured Data
Interpretable to a human but
not a computer program.
Examples: images, text, audio
Interpretable to both a
human and computer
program.

Represent the data to retain as much of the information as possible. This
turns out to be a non-trivial task
Focus in NLP is on how to better represent language
as structured data!
Structured dataUnstructured Data
Interpretable to a human but
not a computer program.
Interpretable to both a
human and computer
program.
data representation
music images text

Note that you can still pull labels from unstructured
data!
How do we extract labels from the unstructured data?
Books Labels:
1. Author
2. Year published
3. Genre
4. Number of pages
5. Title

Why else is text difficult to
represent?
Let’s walk through an example.

Sam sends a text to Jasmine.
Is your name Wi-fi?
Because I'm really
feeling a connection
...

What is the difference between how a
human and model would explain this text?
Is your name Wi-fi?
Because I'm really
...

%%$$$asds
tasjkshdf adbha
sfsfvdf
Sam is trying to build
his relationship with
Jasmine.
Unlike all previous models we have looked at,
computers cannot understand raw text at all!
We first need to represent raw text in a way
they can use.
Human Intuition NLP

Language difficult to represent to a machine
because:
Is your name
Wi-fi?
Because I'm
really feeling a
connection
Language is ambiguous:
- Subtleties
- Cultural references
- Jokes, sarcasm, wordplay
- Phonetic similarities
between words
There are certain grammatical
rules that we can take
advantage of, but this only
gets us half the way.

The trophy doesn't fit into the brown suitcase because
it's too small. What is too small?
The suitcase or the trophy?
Let’s try a short quiz that will demonstrate linguistic
ambiguity:

The trophy doesn't fit into the brown suitcase because
it's too big. What is too big?
The suitcase or the trophy?

You can easily tell what “it” is, even if the sentences
are totally different, using your own understanding
of trophies and suitcases. A computer won’t have such
an easy time.

If NLP is so difficult, why do
we bother?

Because NLP is important!
● Most of the data in the world is unrepresented.
● Vast amount of data are stored as text (think
about websites, laws, emails, books) and this
amount will increase in the future
● Allowing machines to understand text unlocks
vast amounts of information. NLP is incredibly
powerful!

You use NLP technology
every day!

Machine translation & recommendations
1. Google translate is driven by NLP and deep learning
2. New York Times recommends
new articles based upon keyword
topic extraction
Source: New York Times - LDA recommendation, Google translate, the great AI awakening

Familiar applications of NLP
Text classification: Adding tags to academic papers

Spell check, autocomplete and Search results
When you type a query like ‘natural language processing’
into Google search it is already doing 3 NLP tasks:
1. spell and grammar check
2. autocompletion of search results
3. Information retrieval
These are all
NLP tasks!

Artificial intelligence personal assistants
X.ai relies heavily on NLP to schedule meetings:
1. X.ai extracts information from your emails in order to schedule:
a. Day of week
b. Location
c. Number of people to schedule
2. If you reply to your non-human assistant, they reply back! How? X.ai extracts
keywords from your initial email and uses the keywords to classify the text
as having a certain intent (text classification).

Apple’s Siri converts the sound of your
voice to words to action.
When converting between words to
action, Siri needs to “understand” what
you’re asking her to do. This requires
NLP!
NLP is often an intermediate step in complex processes

NLP Tasks
NLP applications are very varied! All involve
first representing text as data, and then
extracting meaning from the text.
✓ Classifiers - spam filter
✓ Information retrieval - Google Search
✓ Information extraction - AI assistants
✓ Machine translation - Google translate
✓ Question & answering - automatic response to certain emails
✓ Summarization of text - what is this article about?
✓ Sentiment analysis - is this document positive or negative?
✓ Speech processing - options on phone helplines
✓ Optical character recognition - scan cheques at ATMs
✓ Spell check, autocomplete - Google Search
NLP applications:
Model
Wow! There
is a lot I can
do here!

How do we convert text into
data?

Representing text as data is a big part
of our NLP work!
Gather text
Represent
text as data
Modeling
Phase
Validation
Phase
What makes NLP different
from the other algorithms we
have studied so far is how
difficult and computationally
intensive this step can be.

Represent text
as data
Data cleaning
n-gramstokenizing
stemming
lemmatization
Stop words
Data
representation
bag-of-words
Firstly, we clean the
input text and then
we represent it using
a model.
Let’s go through each of these
techniques
Step by step!
Additional
cleaning
Not necessarily in
this order -
depends on your
research question

Reminder example: Sam sends a text to
Jasmine.
Is your name Wi-fi?
Because I'm really
...

First, some vocabulary!
Word Document Corpus
Many words Many documents
feature observation dataset

In our text exchange between Sam and
Jasmine:
Word Text All texts
Document Corpus
feature observation dataset

Let’s start with text cleaning:

Represent text
as data
Data cleaning stemming
lemmatization
stop words
The goal of cleaning text data is to standardize
the text as much as possible, so that words with
the same intent are grouped together.
Lower case, remove special
characters, remove spaces

Original sentence:
“Hi Jasmine, would you like to go running in
the park this weekend? We could catch a
movie and dinner afterwards! ”
Words to stem:
“runs” / “running” → “run”
Reducing words into their base
root / stem form. E.g. “Exploring
-> “Explore”
Represent
text as data
Data
cleaning
Stemming

Lemmatization is similar to
stemming, but uses
morphological analysis of
words to return them to
their root.
Represent
text as data
Data
cleaning
Lemmatization
Original Stemmed Lemmatized
walked walk walk
are are be
natural natur natural
processing process process

Original sentence:
Sentence with stopwords removed:
Removing “stop words” removes
common words that don’t add
meaning
Represent
text as data
Data
cleaning
Stop words

predetermined
list of words
adapted from
corpus
Discarding words that appear too
often in your data (using frequency
counts)
Using a list of predetermined
stopwords (available through nltk
package).
Two main ways to approach removing stop
words!.
Stop words

Original sentence:
Cleaned sentence:
“hi jasmine would you like to go running in
the park this weekend we could catch a
movie and dinner afterwards”
Lowercase, remove spaces, remove
any special characters (%,?,!)
Represent
text as data
Data
cleaning
Additional
cleaning
remove special
characters
Lowercase letters

Once text is cleaned, the next
step is tokenization.

n-grams
Data
representation
Now we have clean
text, we can
represent our text as
data using a process
called tokenization.
Let’s take a look!
tokenizing unigrams
bigrams
bag-of-words

Original sentence:
Tokenized sentence:
“Hi”, “Jasmine”, “would”, “you”, “like”, “to”,
“go”, “running”, “in”, “the”, “park”, “this”,
“weekend”, “We”, “could”, “catch”, “a”,
“movie” “and”, “dinner,” “afterwards”
Breaking up the text into words,
phrases, elements (aka “tokens”)
Data
representation
tokenizing

Unigram
Bigram
n-gram “Hi Jasmine would you”, “like to
go running”, “in the park this”,
“weekend”,
You can tokenize to unigrams, bigrams,...
n-grams. Our algorithms will use unigrams.
Tokenizing
“Hi Jasmine”, “would you”, “like
to”, “go running”, “in the”, “park
this”, “weekend”,
One word
two word
n words
“Hi”, “Jasmine”, “would”, “you”,
“like”, “to”, “go”, “running”, “in”,
“the”, “park”, “this”, “weekend”,

● Unigram models ignore context of
a word, so bias is high.
● Bigram, trigram models take into
account context, but need more
data to train.
Is there a best type of tokenization? As with
many of our choices in ML, there is a tradeoff!

Think about it: with bag-of-words, a
document that reads “it’s bad, not good”
would be considered the same as a
document that reads “it’s good, not bad”.
Bag-of-words algorithms use unigram tokenization,
so their key disadvantage is that they do not take
any word’s context into account.

Let’s take a closer look at the
Bag-of-Words representation.

Binary
counts
bag-of-wordstokenizing unigrams
Bag of words is a
representation of text
that uses unigram
tokenization.
Frequency
table
After tokenization, the words
can be represented in a few
different ways.
We will introduce binary counts,
frequency table, TF-IDF.
TF-IDF

Text 1:
“weekend”, “We”, “could”, “go”, “to”, “a”,
Text 2:
“Great”, “dinner”, “Jasmine”, “Sweet”,
“dreams”
Bag of words uses unigram
tokenization across the corpus of text
messages!
Tokenizing bag-of-words
Text 1:
“Hi Jasmine, would you like to go
running in the park this weekend? We
could catch a movie and dinner
afterwards! ”
Text 2:
“Great dinner Jasmine! Sweet dreams.”
Single text Multiple texts
document corpus

The first step in bag-of-word representation is
unigram tokenization of each document. Then these
words are aggregated using different approaches.
1) unigram tokenization
of each document
Text 1:
Text 2:
“dreams”
bag-of-words
Binary
indicators
Frequency
counts
TF-IDF
TF-IDF, Binary and frequency counts are very
simple aggregation approaches.

Binary indicators are 1 or 0
dependent on whether the word is
present in the document.
text Jasmine dinner dreams weekend ...
1 1 1 0 1 ...
2 1 1 1 0 ...
Text 1:
Text 2:
“dreams”
Binary
counts
bag-of-words

Frequency counts are the number of
times each word appears in the
document
Bag-of-words
Frequency
counts
text Jasmine dinner go going ...
1 1 1 2 0 ...
3 1 0 0 2 ...
Text 1:
Text 3:
“Hi”, “Jasmine”, “how”, “is”, “your”, “week”,
“going” “I” “am” “going” “bowling” “this”
“saturday” “want” “to” “come” Problem: How important is each word?

Bag-of-words
Frequency
counts
text Jasmine dinner go going ...
1 1 1 2 0 ...
3 1 0 0 2 ...
Text 1:
Text 3:
“Hi”, “Jasmine”, “how”, “is”, “your”, “week”,
“going” “I” “am” “going” “bowling” “this”
“saturday” “want” “to” “come”
Problem: How important is each word?
Frequency tells us how important a word is to a document. But how can we judge how
important is that word in the corpus? E.g. “Jasmine” probably has a high frequency,
but this doesn’t give us that much new information - the entire corpus is about her!
Tf-idf helps determine importance relative to the corpus.

TF-IDF: Term frequency,
inverse document frequency
Bag-of-words TF-IDF
Tf-idf is a number
assigned to each word,
intended to reflect how
important that word is to
a document.
Tf-idf increases proportionally to
the number of times a word
appears in the document, but is
offset by the frequency of the
word in the corpus.
3 Step Procedure:
1) Compute term frequency
2) Compute inverse document
frequency
3) Multiply 1)*2)

TF-IDF can tell us which words provide the most information about a given
document. Let’s compare frequency and tf-idf:
Based on frequencies alone, we see that the words “Jasmine” and “Sam” are important. But
we knew that! The tf-idf table provides a little more insight: the word “running” is relatively
more important. If we relied only on frequencies, we might have missed the relative
importance of the word “running”! Let’s get into how to calculate tf-idf next.
TF-IDF vs. frequencyBag-of-words TF-IDF
Jasmine Sam running
Text 1 1 2 0
Text 2 1 1 0
Text 3 2 1 1
Jasmine Sam running
Text 1 0.2 0.1 0
Text 2 0.01 0.02 0
Text 3 0.1 0.1 0.8
Frequency Tf-idf

Computed for every word in a document. Equal to
frequency count normalized to account for length
of document.
Term
frequency
running Total words
in text
TF(T)
Text 1 1 22 0.04
TF: Term Frequency: measures how
frequently a term occurs in a document.
TF(t) = (Number of times term t appears
in a document) / (Total number of terms
in the document).
Text 1:
Term frequency table:
The term frequency of the word
running in text 1 is 0.04 (1/22).

Computed across every single
document in the corpus
Inverse
document
frequency
Inverse Document Frequency measures how
important a term is.
While computing TF, all terms are considered
equally important.
However, certain terms, such as "is", "of", and
"that", may appear a lot of times but have
little importance.
IDF(t) = log_e(Total number of
documents / Number of documents
with term t in it)
TF IDF
All terms considered
equally important
Weighs down
frequent term while
scales up infrequent
terms.
‘Is’ as important as
‘girlfriend’
“Vacation” will be
more important than
“and”

IDF(t) = log_e(Total number of documents / Number
of documents with term t in it)
Total number of
documents
running IDF(T)
2 1 0.69
Text 1:
Text 2:
“dreams”
IDF(running ) = log_e(2/1)
Inverse
document
frequency

Let’s bring it all together!Bag-of-words TF-IDF
3 step procedure:
1) Compute term
frequency
2) Compute inverse
document frequency
3) Multiply 1)*2)
word TF(T) IDF(T) TF-IDF(T)
Text 1 running 0.04 0.69 0.0276
TF-IDF tells us which words are
important and which are not,
relative to the corpus!
Thus, composition of the corpus can
also be an important decision to
make as a researcher.

Binary, frequency and TF-IDF all result in sparse matrices.
A sparse matrix is a data frame where most elements=0.
text Jasmine dinner dreams weekend
1 1 1 0 1
2 1 1 1 0
3 1 0 0 0
Text 3:
“Hey”, “Jasmine”, “how”, “is”, “your”, “week”,
“going”
The matrix is “sparse” because each
feature is a word in the corpus. For a
big corpus, this will be a large
vocabulary. Most elements in the
matrix will be 0.
In our example, our
corpus is the set of
text messages.

So, where are we?
Gather text
Represent
text as data
Modeling
Phase
Validation
Phase
We put a lot of effort into
cleaning our data and explored
one simple model (bag-of-words)
to represent our text.
Exploratory
Analysis
Now, we can finally
start doing some
exploratory analysis.

Exploratory Analysis
with Python package “nltk”

frequency
Exploratory
Analysis
conditional
frequencies
concordance
similar
Count of times each word appears in the
corpus
Count of times each word appears in the
corpus dependent on an additional criteria
A list showing all the appearances of a text
of phrase
Returns words in a range of contexts
similar to that of the input word
collocations
Sequences of words that usually appear
together

Frequency is the count of times
each word appears in the corpus
nltk.ConditionalFreqDist()
Exploratory
Analysis
frequency
Relationship start date
Frequency of the word “love” in texts between Sam and Jasmine over time

Count of times each word appears in
the corpus subject to additional
criteria
For example, we may want to look at the frequency of words by the type of news
article. One article is categorized “news” and the other “romance.”
Source: Natural Language Processing with Python
nltk.ConditionalFreqDist()
Exploratory
Analysis
conditional
frequency

“Kids” field defined as 1 for any
description that had the words
“children”, “kids”, “child”,
“daughter”, “son”, “mother”,
“father”.
Exploratory
Analysis
You have already seen an example of
conditional frequency in our KIVA data!
This is an
example
conditional
frequency of
a word
match by
gender.
conditional
frequency

All appearances of the word “woman” in the Kiva data.
Exploratory
Analysis
concordance
A list showing the context
surrounding an input word or phrase.
nltk.concordance()

Collocations takes advantage of the fact that there are pairs of words that naturally go
together.
This is often particular to the data set, for example Acre Fund is the name
of a partner and would unlikely be a collocation in a more broad corpus.
Collocations are sequences
of words that usually
appear together
Exploratory
Analysis
collocations
nltk.collocations()

The result is a list of similar words to our input word!
Source: Sample of Kiva data
Exploratory
Analysis
Similarity
“Similar” in NLTK takes words that
appear around our input word as the
context and finds other words that
have similar context.
nltk.similar()

frequency
Exploratory
Analysis
conditional
frequencies
concordance
similar
collocations
Modeling
Phase
Validation
Phase
Now we have an idea of what is in the data, let’s move
into our modeling phase.
We now have a structured
data set, so we can use
traditional machine learning
to answer a research
question.

We will discuss two algorithms:
Naive Bayes Example: text classification.
K-means
clustering
Example: topic extraction.
This is just a starting point for your
exploration of NLP. Both of these algorithms
are fairly intuitive and for some use cases
produce powerful results.

We will use our familiar framework to discuss
both naive bayes and k-means clustering:
Task
What is the problem we want our
model to solve?
Quantitative measure we use to
evaluate the model’s performance.
ML algorithms can be supervised or
unsupervised. This determines the
learning methodology.
Source: Deep Learning Book - Chapter 5: Introduction to Machine Learning
Learning
Methodology
Performance

Unsupervised Algorithms
● For every x, there is no Y.
● We do not know the correct answers.
● Instead, we try to model our understanding of the
distribution of x to draw inference about Y.
Source: Machine Learning Mastery - Supervised and Unsupervised Learning
Remember the difference between supervised and
unsupervised?
Supervised Algorithms
● Every observation has an associated label (for every x, there is a
true Y).
● Goal is to predict Y using x.
● Since we have the true Y we can tell how far Y* is from the truth.
● Majority of machine learning is supervised.
Learning
Methodology

Unsupervised
Supervised
ML
Algorithm
No Labelled Data
Labelled Data
Learning
Methodology
Today, we will look at both a supervised and unsupervised
NLP model.
K-means clustering
Naive Bayes

1) Naive Bayes:
Text classification algorithm

NLP supervised learning algorithms

Naive Bayes: model cheat sheet
● “Naive” - makes assumptions that do
not necessarily hold in real life
● Assumes independence between
features.
● Simple yet well-performing,
especially in document
classification.
● Allow every feature or word in a
document to contribute to its
classification
● Low false positive rate
Pros Cons

Can we tell whether an email is spam
or not? (this is a real email from an inbox!)
Task

“Be informed that during our investigations we found out
that there is a total amount of $4.5 Million that has been
assigned in your name as the beneficiary and these
fraudsters are busy swindling you without any hope of
receiving your fund.”
Task
What is the difference between how a
human and machine learning model would
decide whether this is spam or not?
Email text:

Each word in the email
contributes to the
probability the email is
spam, based upon historical
frequency in spam
messages.
● I don’t have a fund in
Africa.
● Would the FBI email
me?
● How would I not
know about $4.5
million dollars?
● Same goal! Both humans & ML are assessing the probability the email is true.
● Key difference: Naive Bayes use frequency of words present in the email in
historical examples of spam to assign a probability that it is spam.
Human Intuition Naive Bayes

We train our model by assessing how each word of the
email contributes to a classification of Spam or Not
Spam, over our entire labelled sample of emails.
Compute probability of
spam for each word in an
email
Word1
word2
word3...
Word1
word2
word3...
Word1
word2
word3...
{
corpus
n=100
Word1
word2
word3...
Learning
Methodology
Repeat for each email in the
corpus of labelled data
Multiply
probabilities across
all words.
If the total
probability is >0.5
then we classify as
spam.

Classification
Predicting whether an email is spam is an example of a
classification task.Task
IMPORTANT: Notice that our data frame is a bag-of-words count. We did unigram tokenization to
transform unstructured data to structured data.
Email “millions” “FBI” spam
1 1 1 1
2 0 0 0
3 1 0 0
4 0 0 1
5 1 0 1
6 1 1 1

What we want is the probability that an email is
spam, GIVEN the words that are present in the
email.
This is conditional probability.

Formalized as an equation, our conditional
probability for the sample email above is as
follows:
P(Spam | We found out that there is a total amount of
$4.5 million… )
EMAIL TEXT: “We found out that there is a total
amount of $4.5 Million that has been assigned in your
name as the beneficiary and these fraudsters are
busy swindling you without any hope of receiving
your fund...”

This equation formalizes the probability that
it will rain today, given that it is April.
For a primer on this concept, see:
https://www.khanacademy.org/math/statistics-probability/probability-library/conditional-probability-ind
ependence/v/calculating-conditional-probability
P(Rain|April)
Conditional probability is a concept that can be
seen in our daily lives. For example:
Probably varies quite a bit,
depending on where in the
world you are!

P(April|Rain) * P(Rain)
Bayes Theorem allows us to calculate conditional
probability:
For a primer on this concept, see:
https://www.khanacademy.org/math/statistics-probability/probability-library/conditional-probability-independe
nce/v/calculating-conditional-probability
P(April)
P(Rain|April)

Task Defining f(x)
How can we use words to predict spam?
The Naive Bayes algorithm will calculate the conditional
probability that an email is spam, conditioned on the text
of the email.
Using our bag-of-words representation, we evaluate the
probability of each word separately and sum them up.
P(Spam | We found out that there is a total amount of $4.5 million… )
= P(Spam|we) * P(Spam|found) * P(Spam|total) * P(Spam|amount) *
P(Spam|million) …
NOTE: Why did we leave out words like “that”, “is”, “a”, and “of”? These are examples of stopwords that would have been
removed while we were cleaning and preparing our data.

Task Defining f(x)
Each word in an email contributes to
the probability of spam.
Let’s use the Bayes theorem to calculate the probabilities
below, starting with P(Spam|million). This will tell us how
predictive the word “millions” is in determining spam.
P(Spam | We found out that there is a total amount of $4.5 million… )
= P(Spam|we) * P(Spam|found) * P(Spam|total) * P(Spam|amount) *
P(Spam|million) …
NOTE: Why did we leave out words like “that”, “is”, “a”, and “of”? These are examples of stopwords that would have been
removed while we were cleaning and preparing our data.

P(million|Spam) * P(Spam)
Let’s use the Bayes Theorem to calculate this
probability:
P(million)
P(Spam|million)
We don’t know the probability of spam given “million”, but thanks
to our labelled training data, we do know the probability of
“million” given spam!
We will calculate each of the 3 right-hand pieces, starting with
P(million|Spam)

What is the probability that
“millions” occurs in a spam email?
Probability of
the word
“millions” being
in a spam email
sum(millions=1 & spam=1)
sum(spam=1)
3
4
P(million|Spam)
P(million|Spam)
How many spam emails in our training data set
contain the word “millions”?
Email millions spam
1 1 1
2 0 0
3 1 0
4 0 1
5 1 1
6 1 1

What is the probability that an email is
spam?
Probability of
an email being
spam
sum(spam=1)
sum(spam=1 or spam=0)
4
6
P(Spam)
P(Spam)
How many emails are spam? Email millions spam
1 1 1
2 0 0
3 1 0
4 0 1
5 1 1
6 1 1

What is the probability that an email
contains the word “million”?
Probability of
an email
containing the
word “million”
sum(million=1)
sum(million=1 or million=0)
4
6
P(million)
P(million)
How many emails contain the word “million”? Email millions spam
1 1 1
2 0 0
3 1 0
4 0 1
5 1 1
6 1 1

Task Defining f(x)
Now let’s bring it all together!
3/4 * 4/6
4/6
0.75
Probability of this email
being spam, given it has
the word “millions”
0.75
P(Spam|
million)
P(million|
Spam)
P(million)
P(Spam)
*
Email millions spam
1 1 1
2 0 0
3 1 0
4 0 1
5 1 1
6 1 1

Task Prediction
Now we can predict spam!
Email millions spam predicted
1 1 1 1
2 0 0 0
3 1 0 1
4 0 1 0
5 1 1 1
6 1 1 1
Now we can predict spam,
using a single word! Our
model predicts that if the
word “millions” is present, it
is 75% spam. Since this is a
classification problem, we
round this up to 1.

Task Defining f(x)
What about the other words?
But this model is simplistic. We should use the
other words in the email too! We can repeat
the process of calculating conditional
probability for all the words in an email.
Email “millions” “FBI” “amount” ...
1 1 1 0
2 0 0 0
3 1 0 1
4 0 0 1
5 1 0 1
6 1 1 1
P(Spam | We found out that there is a
total amount of $4.5 million… )
= P(Spam|we) * P(Spam|found) *
P(Spam|total) * P(Spam|amount) *
P(Spam|million) …

Task Defining f(x)
What about multiple words?
Once we have conditional probabilities for all
the words in an email, we multiply them
together to get the probability that this email
is spam, given all the words it contains.
Email “millions” “FBI” “amount” ...
1 1 1 0
2 0 0 0
3 1 0 1
4 0 0 1
5 1 0 1
6 1 1 1
NOTE: What if a probability is 0? Then the total probability would be 0! To avoid this problem, Naive Bayes uses “Laplace
smoothing,” to ensure the probability is never 0. Read more here: https://en.wikipedia.org/wiki/Laplace_smoothing
P(Spam | We found out that there is a
total amount of $4.5 million… )
= P(Spam|we) * P(Spam|found) *
P(Spam|total) * P(Spam|amount) *
P(Spam|million) …

Now you know how the Naive Bayes
Algorithm works, let’s examine some
of its disadvantages and assumptions.

Naive Bayes “naively” assumes each word is independent to every
other word.
This is inherited from our representation of the text as a
“bag-of-words”. Remember, an email that said “it’s not good, it’s
bad” would be considered the same as “it’s not bad, it’s good” to
the Naive Bayes algorithm.
As a result, you can end up double-counting highly correlated
words, like “dollars” and “millions,” which would incorrectly push
up the error rate.
How can we quantify algorithm performance?

Evaluating algorithm
performance

Classification
Interpreting our results for a single word “millions”Performance
75%P(S|W)
Probability of email
being spam given
presence of word
“millions” is 75%.
Word1
word2
word3...
Word1
word2
word3...
Word1
word2
word3...
{
corpus
n=100
Word1
word2
word3...

What is the accuracy
of our model?
Performance
We turn to our training data to evaluate
how our model did.
Accuracy:
# labels predicted correctly
# labels predicted
Measures of
performance
Email millions spam
1 1 1
2 0 0
3 1 0
4 0 1
5 1 1
6 1 1

What is the accuracy
of our model?
Performance
Accuracy:
# labels predicted correctly
# labels predicted
Measures of
performance
4
6
66%
1 1 1 1
2 0 0 0
3 1 0 1
4 0 1 0
5 1 1 1
6 1 1 1

Recap: false positive & false negative
Spam classification task: is the email spam?
False positive:
False negative:
Email is classified as spam when
it is not.
Email is not classified as spam
when it is. In any classification model we work on,
we will evaluate performance on FP, FN
in addition to accuracy.
Performance

What is the false
negative rate?
Performance
False negative rate:
# incorrectly predicted as spam
# total predictions
Measures of
performance
1 1 1 1
2 0 0 0
3 1 0 1
4 0 1 0
5 1 1 1
6 1 1 1

False negative ratePerformance
False negative rate:
# incorrectly predicted as spam
# total predictions
Measures of
performance
1
6
16%
1 1 1 1
2 0 0 0
3 1 0 1
4 0 1 0
5 1 1 1
6 1 1 1

What is the false
positive rate?
Performance
False positive rate:
# incorrectly predicted as non-spam
# total predictions
Measures of
performance
1 1 1 1
2 0 0 0
3 1 0 1
4 0 1 0
5 1 1 1
6 1 1 1

False positive ratePerformance
False positive rate:
# incorrectly predicted as non-spam
# total predictions
Measures of
performance
1
6
16%
1 1 1 1
2 0 0 0
3 1 0 1
4 0 1 0
5 1 1 1
6 1 1 1

We’ve just automated spam-detection
(thank goodness!)
In the next module, we will look at another NLP
algorithm.

You are on fire! Go straight to the
next module here.
Need to slow down and digest? Take a
minute to write us an email about
what you thought about the course. All
feedback small or large welcome!
Email: sara@deltanalytics.org

Congrats! You finished
module 8!
Find out more about
Delta’s machine
learning for good
mission here.

Module 8: Natural language processing Pt 1

Recommended

More Related Content

What's hot (20)

Similar to Module 8: Natural language processing Pt 1 (20)

More from Sara Hooker (12)

Recently uploaded (20)

Module 8: Natural language processing Pt 1