Bayesian Learning Method

Nguyen Van Vinh

UET-Hanoi VNU
 Content
• Bayesian Learning (NB) method
• Examples for NB
• Application (Text Classification, Spam
Mail)
 Introduction
Thomas Bayes (c. 1702 – 17 April 1761)
was a British mathematician and
Presbyterian minister.(wikipedia)

Today we learn:
• Bayesian classification
– E.g. How to decide if a patient is ill or healthy,
based on
• A probabilistic model of the observed data
• Prior knowledge
 Classification problem
• Training data: examples of the form (d,h(d))
– where d are the data objects to classify (inputs)
– and h(d) are the correct class info for d, h(d){1,…K}
• Goal: given dnew, provide h(dnew)
 Why Bayesian?
• Provides practical learning algorithms
– E.g. Naïve Bayes
• Prior knowledge and observed data can be
combined
• It is a generative (model based) approach, which
offers a useful conceptual framework
– E.g. sequences could also be classified, based on
a probabilistic model specification
– Any kind of objects can be classified, based on a
probabilistic model specification
 Bayes’ Rule
Understanding Bayes' rule
P ( d | h) P ( h) d  data
p(h | d )  h  hypothesis (model)
P(d ) - rearranging
p ( h | d ) P ( d )  P ( d | h) P ( h)
P ( d , h)  P ( d , h)
the same joint probability
Who is who in Bayes’ rule on both sides

P ( h) : prior belief (probability of hypothesis h before seeing any data)

P ( d | h) : likelihood (probability of the data if the hypothesis h is true)
P(d )   P(d | h) P(h) : data evidence (marginal probability of the data)
h

P(h | d ) : posterior (probability of hypothesis h after having seen the data d )

 Does patient have cancer or not?
• A patient takes a lab test and the result comes back
positive. It is known that the test returns a correct
positive result in only 98% of the cases and a correct
negative result in only 97% of the cases. Furthermore,
only 0.008 of the entire population has this disease.

1. What is the probability that this patient has cancer?

2. What is the probability that he does not have cancer?
3. What is the diagnosis?
 hypothesis1 : ' cancer'
} hypothesis space H
hypothesis2 : ' cancer'
 data : ' '
P ( | cancer) P (cancer) .........................
1.P (cancer |  )    ..........
P() .........................
P ( | cancer)  0.98
P (cancer)  0.008
P ( )  P ( | cancer) P (cancer)  P ( | cancer) P (cancer)
 ...................................................................
P ( | cancer)  0.03
P (cancer)  ..........

2.P (cancer |  )  ...........................

3.Diagnosis ??
 Choosing Hypotheses
• Maximum Likelihood hML  arg max P(d | h)
hypothesis: hH

• Generally we want the hMAP  arg max P(h | d )

hH
most probable hypothesis
given training data.This is
the maximum a posteriori
hypothesis:
– Useful observation: it does
not depend on the
denominator P(d)
 Now we compute the diagnosis
– To find the Maximum Likelihood hypothesis, we evaluate P(d|h)
for the data d, which is the positive lab test and chose the
hypothesis (diagnosis) that maximises it:
P( | cancer)  ............
P( | cancer)  .............
 Diagnosis : hML  .................

– To find the Maximum A Posteriori (MAP) hypothesis, we

evaluate P(d|h)P(h) for the data d, which is the positive lab test
and chose the hypothesis (diagnosis) that maximises it. This is
the same as choosing the hypotheses gives the higher posterior
probability.
P( | cancer) P(cancer)  ................
P( | cancer) P(cancer)  .............
 Diagnosis : hMAP  ......................
 Bayesian decision theory
• Let x be the value predicted by the agent and x* be
the true value of X.
• The agent has a loss function, which is 0 if x = x*
and 1 otherwise
• Expected loss for predicting x:
 L( x, x*)P( x* | e)   P( x* | e)  1  P( x | e)
x* x * x
• What is the estimate of X that minimizes the
expected loss?
– The one that has the greatest posterior probability P(x|e)
– This is called the Maximum a Posteriori (MAP) decision
 MAP decision
• Value x of X that has the highest posterior
probability given the evidence E = e:
P( E  e | X  x) P( X  x)
x*  arg max x P( X  x | E  e) 
P ( E  e)
 arg max x P( E  e | X  x) P( X  x)

P ( x | e)  P (e | x ) P ( x )
posterior likelihood prior

• Maximum likelihood (ML) decision:

x*  arg max x P (e | x)
 Naïve Bayes Classifier
• What can we do if our data d has several attributes?
• Naïve Bayes assumption: Attributes that describe data instances are
conditionally independent given the classification hypothesis
P(d | h)  P(a1 ,..., aT | h)   P(at | h)
t

– it is a simplifying assumption, obviously it may be violated in reality

– in spite of that, it works well in practice
• The Bayesian classifier that uses the Naïve Bayes assumption and
computes the MAP hypothesis is called Naïve Bayes classifier
• One of the most practical learning methods
• Successful applications:
– Medical Diagnosis
– Text classification
Example. ‘Play Tennis’ data
Day Outlook Temperature Humidity Wind Play
Tennis

Day1 Sunny Hot High Weak No

Day2 Sunny Hot High Strong No
Day3 Overcast Hot High Weak Yes
Day4 Rain Mild High Weak Yes
Day5 Rain Cool Normal Weak Yes
Day6 Rain Cool Normal Strong No
Day7 Overcast Cool Normal Strong Yes
Day8 Sunny Mild High Weak No
Day9 Sunny Cool Normal Weak Yes
Day10 Rain Mild Normal Weak Yes
Day11 Sunny Mild Normal Strong Yes
Day12 Overcast Mild High Strong Yes
Day13 Overcast Hot Normal Weak Yes
Day14 Rain Mild High Strong No
 Naïve Bayes solution
Classify any new datum instance x=(a1,…aT) as:

hNaive Bayes  arg max P(h) P(x | h)  arg max P(h) P(at | h)
h h t

• To do this based on training examples, we need to estimate the

parameters from the training examples:

– For each target value (hypothesis) h

Pˆ (h) : estimate P(h)

– For each attribute value at of each datum instance

Pˆ (at | h) : estimate P(at | h)

Based on the examples in the table, classify the following datum x:
x=(Outl=Sunny, Temp=Cool, Hum=High, Wind=strong)
• That means: Play tennis or not?
hNB  arg max P(h) P(x | h)  arg max P(h) P(at | h)
h[ yes , no ] h[ yes , no ] t

 arg max P(h) P(Outlook  sunny | h) P(Temp  cool | h) P( Humidity  high | h) P(Wind  strong | h)
h[ yes , no ]

• Working:
P ( PlayTennis  yes)  9 / 14  0.64
P ( PlayTennis  no)  5 / 14  0.36
P (Wind  strong | PlayTennis  yes)  3 / 9  0.33
P (Wind  strong | PlayTennis  no)  3 / 5  0.60
etc.
P ( yes) P( sunny | yes) P(cool | yes) P(high | yes) P ( strong | yes)  0.0053
P (no) P( sunny | no) P(cool | no) P(high | no) P( strong | no)  0.0206
 answer : PlayTennis( x)  no
 Example: Training Dataset
age income studentcredit_rating
buys_compu
<=30 high no fair no
<=30 high no excellent no
Class: 31…40 high no fair yes
C1:buys_computer = ‘yes’ >40 medium no fair yes
C2:buys_computer = ‘no’ >40 low yes fair yes
>40 low yes excellent no
Data sample
31…40 low yes excellent yes
X = (age <=30,
Income = medium, <=30 medium no fair no
Student = yes <=30 low yes fair yes
Credit_rating = Fair) >40 medium yes fair yes
<=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
 Learning to classify text
• Learn from examples which articles are of
interest
• The attributes are the words
• Observe the Naïve Bayes assumption just
means that we have a random sequence
model within each class!
• NB classifiers are one of the most effective for
this task
• Resources for those interested:
– Tom Mitchell: Machine Learning (book) Chapter 6.
Results on a benchmark text corpus
Learning and inference pipeline
Learning Training
Labels
Training
Samples
Learned
Features Training
model

Learned
model
Inference

Features Prediction
Test Sample
 Features

• A measurable variable that is (rather,

should be) distinctive of something we
want to model.
• We usually choose features that are useful
to identify something, i.e., to do
classification
– Ex: Cô gái đó rất đẹp trong bữa tiệc hôm đó.
• We often need several features to
adequately model something – but not too
many! 21
 Feature vectors
• Values for several features of an
observation can be put into a single vector
# proper # 1st # commas
nouns person
pronouns
2 0 0

5 0 0

0 1 1

22
 Feature vectors
• Features should be useful in discriminating
between categories.

23
 Feature Representation

• Convert this example to a vector using bag-

of-words features

• Very large vector space (size of vocabulary), sparse features

• Requires indexing the features (mapping them to axes)
• More sophisticated feature mappings possible (m-idf), as well as
lots of other features: character n-grams, parts of speech, …
24
 Case study:
Text document classification
• MAP decision: assign a document to the class with the highest
posterior P(class | document)

• Example: spam classification

– Classify a message as spam if P(spam | message) > P(¬spam | message)
 Case study:
Text document classification
• MAP decision: assign a document to the class with the highest
posterior P(class | document)

• We have P(class | document)  P(document | class)P(class)

• To enable classification, we need to be able to estimate the

likelihoods P(document | class) for all classes and
priors P(class)
 Naïve Bayes Representation
• Goal: estimate likelihoods P(document | class)
and priors P(class)
• Likelihood: bag of words representation
– The document is a sequence of words (w1, …, wn)
– The order of the words in the document is not important
– Each word is conditionally independent of the others given
document class
 Naïve Bayes Representation
• Goal: estimate likelihoods P(document | class)
and priors P(class)
• Likelihood: bag of words representation
– The document is a sequence of words (w1, …, wn)
– The order of the words in the document is not important
– Each word is conditionally independent of the others given
document class
n
P(document | class) = P(w1, ... , wn | class) = Õ P(wi | class)
i=1
Bag of words illustration

US Presidential Speeches Tag Cloud

http://chir.ag/projects/preztags/
Bag of words illustration

US Presidential Speeches Tag Cloud

http://chir.ag/projects/preztags/
Bag of words illustration

US Presidential Speeches Tag Cloud

http://chir.ag/projects/preztags/
 Naïve Bayes Representation
• Goal: estimate likelihoods P(document | class) and
P(class)
• Likelihood: bag of words representation
– The document is a sequence of words (w1, … , wn)
– The order of the words in the document is not important
– Each word is conditionally independent of the others given
document class
n
P(document | class) = P(w1, ... , wn | class) = Õ P(wi | class)
i=1
– Thus, the problem is reduced to estimating marginal likelihoods
of individual words P(wi | class)
 Parameter estimation
• Model parameters: feature likelihoods P(word | class) and
priors P(class)
– How do we obtain the values of these parameters?

prior P(word | spam) P(word | ¬spam)

spam: 0.33
¬spam: 0.67
 Parameter estimation
• Model parameters: feature likelihoods P(word | class) and
priors P(class)
– How do we obtain the values of these parameters?
– Need training set of labeled samples from both classes

# of occurrences of this word in docs from this class

P(word | class) =
total # of words in docs from this class

– This is the maximum likelihood (ML) estimate, or estimate

that maximizes the likelihood of the training data:
D nd

 P(w
d 1 i 1
d ,i | classd ,i )
d: index of training document, i: index of a word
 Parameter estimation
• Parameter estimate:

# of occurrences of this word in docs from this class

P(word | class) =
total # of words in docs from this class

• Parameter smoothing: dealing with words that were never

seen or seen too few times
– Laplacian smoothing: pretend you have seen every vocabulary word
one more time than you actually did

# of occurrences of this word in docs from this class + 1

P(word | class) =
total # of words in docs from this class + V

(V: total number of unique words)

 Summary: Naïve Bayes for
Document Classification
• Naïve Bayes model: assign the document to the
class with the highest posterior
n
P(class | document) µ P(class)Õ P(wi | class)
i=1
• Model parameters:
Likelihood Likelihood
of class 1 of class K
prior
P(w1 | class1) P(w1 | classK)
P(class1)
P(w2 | class1) P(w2 | classK)
…
… … …
P(classK)
P(wn | class1) P(wn | classK)
 Laplace Smoothing

• Laplace’s estimate:
– Pretend you saw every r r b
outcome once more
than you actually did
 Laplace Smoothing
• Laplace’s estimate (extended):
– Pretend you saw every outcome k r r b
extra times

– What’s Laplace with k = 0?

– k is the strength of the prior

• Laplace for conditionals:

– Smooth each condition
independently:
 Summarization
• Bayes’ rule can be turned into a classifier
• Maximum A Posteriori (MAP) hypothesis estimation
incorporates prior knowledge; Max Likelihood doesn’t
• Naive Bayes Classifier is a simple but effective Bayesian
classifier for vector data (i.e. data with several attributes)
that assumes that attributes are independent given the
class.
• Bayesian classification is a generative approach to
classification
 Reference
• Slides of ML Course, University of Birmingham
• Textbook reading (contains details about using Naïve
Bayes for text classification):
Tom Mitchell, Machine Learning (book), Chapter 6.
• Software: NB for classifying text:
http://www-2.cs.cmu.edu/afs/cs/project/theo-11/www/naive-
bayes.html
https://julianstier.com/posts/2021/01/text-classification-with-naive-
bayes-in-numpy/
• Useful reading for those interested to learn more about
NB classification, beyond the scope of this module:
http://www-2.cs.cmu.edu/~tom/NewChapters.html