Bayesian Modeling

 Bayesian modeling refers to the process of building statistical models using

Bayesian methods.
 These methods are based on Bayesian probability theory, which provides a
framework for quantifying uncertainty and updating beliefs in the presence of
evidence.
 Bayesian modeling offers several advantages over traditional statistical methods,
including the ability to incorporate prior knowledge, handle uncertainty explicitly,
and provide probabilistic predictions.

key components and steps involved in Bayesian modeling:

Components of Bayesian Modeling:
1. Prior Distribution:
- A prior distribution represents our beliefs about the parameters of the model before
observing any data.
- Priors can be chosen based on existing knowledge, expert opinions, or assumed
distributions.
2.Likelihood Function:
- The likelihood function quantifies the probability of observing the data given the
parameters of the model.
- It represents the likelihood of the observed data under different parameter values.
3. Posterior Distribution:
- The posterior distribution represents our updated beliefs about the parameters of the
model after observing the data.
- It is obtained by combining the prior distribution and the likelihood function using
Bayes' theorem.
4. Model Parameters:
- Model parameters are the unknown quantities that we want to estimate from the data.
- In Bayesian modeling, parameters are treated as random variables with probability
distributions.

Steps in Bayesian Modeling:

1. Model Specification:
- Define the structure of the model, including the variables of interest and their
relationships.
 - Choose appropriate prior distributions for the model parameters.
2. Data Collection:
- Collect data relevant to the variables in the model.
3. Parameter Estimation:
- Use Bayesian inference techniques to estimate the posterior distribution of the model
parameters given the observed data.
- This typically involves sampling methods such as Markov Chain Monte Carlo
(MCMC) or variational inference.
4. Model Checking and Validation:
- Assess the fit of the model to the data and check for any violations of model
assumptions.
- Validate the model by comparing its predictions to new data or using cross-validation
techniques.
5. Inference and Prediction:
- Use the posterior distribution to make inferences about the parameters of the model
and to make predictions about new data.
- Calculate credible intervals or posterior predictive distributions to quantify uncertainty
in the predictions.

Applications of Bayesian Modeling:

 Machine Learning: Bayesian techniques are widely used in machine learning for
tasks like classification, regression, and anomaly detection.

 Signal Processing: These models are valuable for filtering noise and extracting signals
from noisy data.

 Bioinformatics: They play a role in analyzing gene expression data and

understanding biological processes.
  Finance: Bayesian models are used for risk assessment, portfolio optimization, and
fraud detection.
 Artificial intelligence: Bayesian techniques are widely used for probabilistic modeling
and uncertainty quantification.

 Environmental science: Bayesian modeling is applied for modeling complex systems

and predicting environmental outcomes.

Overall, Bayesian modeling provides a flexible and powerful framework for building
statistical models that can capture uncertainty and incorporate prior knowledge
effectively. It is widely used in both academic research and practical applications across
diverse domains.

Bayes' theorem
Bayes' theorem is a fundamental concept in probability theory and statistics, named after
the Reverend Thomas Bayes, an 18th-century British mathematician, who developed the
theorem. It provides a way to update the probability of a hypothesis in light of new
evidence. Mathematically, Bayes' theorem can be expressed as:

P(A|B) = P(B|A) × P(A) / P(B)

Where:
- P(A|B) is the probability of event A occurring given that event B has occurred. This is
called the posterior probability.
- P(B|A) is the probability of event B occurring given that event A has occurred. This is
called the likelihood.
- P(A) is the probability of event A occurring. This is called the prior probability.
- P(B) is the probability of event B occurring. This is called the marginal likelihood or
evidence.

In words, Bayes' theorem states that the probability of A given B is proportional to the
probability of B given A, multiplied by the prior probability of A, and divided by the
marginal probability of B.
Bayes' theorem is particularly useful in situations where we want to update our beliefs or
probabilities about an event or hypothesis in light of new evidence.
It provides a formal framework for combining prior knowledge or beliefs with observed
data to arrive at a more informed or updated probability distribution.
This makes it a powerful tool in various fields including statistics, machine learning, and
artificial intelligence.

Example of bayes analysis

let's consider a classic example involving medical testing. Suppose a certain disease, let's
call it Disease X, affects about 1% of the population. We have a diagnostic test for Disease
X, but it's not perfect. Specifically, the test has the following properties:
- If a person has Disease X, the test correctly identifies them as positive 95% of the time
(this is the sensitivity).
- If a person does not have Disease X, the test incorrectly identifies them as positive 2% of
the time (this is the false positive rate, or 1 - specificity).
Now, let's say a person takes the test and it comes back positive. What is the probability
that this person actually has Disease X?
Let's use Bayes' theorem to analyze this situation.
Let:
- A be the event that the person has Disease X.
- B be the event that the test comes back positive.
We want to find P(A|B) , the probability that the person has Disease X given that the test is
positive.
We know:
- P(A) , the prior probability of having Disease X, which is 0.01 (or 1%).
- P(B|A), the probability of testing positive given that the person has Disease X, which is
0.95 (or 95%).
- P(- A) , the probability of not having Disease X, which is 1 - 0.01 = 0.99 (or 99%).
- P(B|-A) , the probability of testing positive given that the person does not have Disease X,
which is 0.02 (or 2%).

So, given that the test came back positive, the probability that the person actually has
Disease X is approximately 32.4%. Even though the test is quite accurate, the relatively
low prior probability of having the disease means that a positive test result doesn't
guarantee that the person has the disease.
Bayes’ Theorem — Example Solution

Bayes’ Theorem looks simple in mathematical expressions such as;

P(A|B) = P(B|A)*P(A)/P(B)

Problem 1:

Let’s work on a simple NLP problem with Bayes Theorem. By using NLP, I can detect spam e-
mails in my inbox. Assume that the word ‘offer’ occurs in 80% of the spam messages in my
account. Also, let’s assume ‘offer’ occurs in 10% of my desired e-mails. If 30% of the received
e-mails are considered as a scam, and I will receive a new message which contains ‘offer’,
what is the probability that it is spam?

Now, I assume that I received 100 e-mails. The percentage of spam in the whole e-mail is
30%. So, I have 30 spam e-mails and 70 desired e-mails in 100 e-mails. The percentage of the
word ‘offer’ that occurs in spam e-mails is 80%. It means 80% of 30 e-mail and it makes 24.
Now, I know that 30 e-mails of 100 are spam and 24 of them contain ‘offer’ whereas 6 of
them not contains ‘offer’.

The percentage of the word ‘offer’ that occurs in the desired e-mails is 10%. It means 7 of
them (10% of 70 desired e-mails) contain the word ‘offer’ and 63 of them not.

Now, we can see this logic in a simple chart.

The question was what is the probability of spam where the mail contains the word ‘offer’:
1. We need to find the total number of mails which contains ‘offer’ ;
24 +7 = 31 mail contain the word ‘offer’
2. Find the probability of spam if the mail contains ‘offer’ ;
In 31 mails 24 contains ‘offer’ means 77.4% = 0.774 (probability)

NOTE: In this example, I choose the percentages which give integers after calculation. As a
general approach, you can think that we have 100 units at the beginning so if the results are
not an integer, it will not create a problem. Such that, we cannot say 15.3 e-mails but we can
say 15.3 units.
Solution with Bayes’ Equation:

A = Spam

B = Contains the word ‘offer’

P( contains offer|spam) = 0.8 (given in the question)

P(spam) = 0.3 (given in the question)

Now we will find the probability of e-mail with the word ‘offer’. We can compute that by
adding ‘offer’ in spam and desired e-mails. Such that;

P(contains offer) = 0.3*0.8 + 0.7*0.1 = 0.31

As it is seen in both ways the results are the same. In the first part, I solved the same
question with a simple chart and for the second part, I solved the same question with Bayes’
theorem.

Problem 2:

I want to solve one more example from a popular topic as Covid-19. As you know, Covid-19
tests are common nowadays, but some results of tests are not true. Let’s assume; a
diagnostic test has 99% accuracy and 60% of all people have Covid-19. If a patient tests
positive, what is the probability that they actually have the disease?
The total units which have positive results= 59.4 + 0.4 = 59.8

59.4 units (true positive) is 59.8 units means 99.3% = 0.993 probability

With Bayes’;

P(positive|covid19) = 0.99

P(covid19) = 0.6

P(positive) = 0.6*0.99+0.4*0.01=0.598
Again, we find the same answer with the chart. There are many examples to learn Bayes’
Theorem’s applications such as the Monty Hall problem which is a little puzzle that you have
3 doors. Behind the doors, there are 2 goats and 1 car. You are asked to select one door to
find the car. After selecting one door, the host opens one of the not-selected doors and
revealing the goat. Then, you are asked to switch the doors or stick with your first choice. By
running this process a thousand times and simulating it, you can find the probability of
winning and figure out the idea of Bayes’ theorem and Bayesian statistics in general through
the Monty Hall problem.

When we think of Bayes’ Theorem in the machine learning concept, it provides a way to
calculate the probability of the hypothesis based on conditions by using the relationship
between data and hypothesis. Also, it is the first step for understanding True Positive, False
Positive, True Negative, and False Negative concepts in data science classification problems
and Naive Bayes classifier.

Bayesian networks
Bayesian networks, also known as belief networks or probabilistic graphical models, are
powerful tools for representing and reasoning about uncertain knowledge. They provide a
formalism for capturing probabilistic relationships among a set of variables in a domain.
Here's an overview of Bayesian networks:
Components of Bayesian Networks:
1. Nodes (Vertices):
- Each node represents a random variable in the domain. These variables can be discrete,
continuous, or hybrid.
- Nodes may represent observable quantities (e.g., symptoms in a medical diagnosis) or
latent variables (e.g., underlying causes).

2. Edges (Directed Links):

- Edges between nodes indicate direct probabilistic dependencies.
- If there's a directed edge from node A to node B, it implies that the probability
distribution of B depends on the state of A.

3. Conditional Probability Tables (CPTs):

- Each node has an associated Conditional Probability Table that quantifies the probabilistic
relationship between the node and its parents.
 - The CPT specifies the probability distribution of a node given the states of its parent
nodes.

Advantages of Bayesian Networks:

1. Modularity: Bayesian networks allow for modular representation of complex systems,

breaking down dependencies into manageable components.

2. Uncertainty Handling: They provide a principled way to represent and reason under
uncertainty, crucial for many real-world applications.

3. Interpretability: The graphical structure of Bayesian networks provides a visual

representation of the dependencies among variables, making it easier for humans to
understand and interpret.

4. Efficient Inference: Various algorithms exist for efficient probabilistic inference in Bayesian
networks, allowing for queries about the probability distribution of variables given evidence.

Applications of Bayesian Networks:

1. Medical Diagnosis: Bayesian networks are widely used in medical diagnosis systems to
combine patient symptoms with medical knowledge for accurate diagnosis.
2. Risk Assessment: They are used in risk assessment and decision-making processes, such as
determining the likelihood of failure in engineering systems or assessing financial risks.

3. Natural Language Processing: Bayesian networks are used in various natural language
processing tasks, such as language modeling, part-of-speech tagging, and parsing.

4. Computer Vision: They are used for object recognition, image segmentation, and scene
understanding in computer vision applications.

Overall, Bayesian networks provide a flexible framework for modeling uncertain domains
and are widely applied across different fields for decision-making, prediction, and inference
tasks.

Inference in Bayesian networks

Inference in Bayesian networks refers to the process of reasoning about uncertain
relationships among variables represented in a graphical probabilistic model called a
Bayesian network.

Inference in Bayesian networks refers to the process of using the network's structure and
data to estimate the probability of different outcomes (states) for variables in the network.

Types of Inference:

There are two main types of inference in Bayesian networks:

 Explanatory Inference: This involves calculating the posterior probability of a

variable given the evidence about other variables. In simpler terms, it's about finding
the probability of a specific outcome (state) for a variable considering the observed
states of other variables.

 Predictive Inference: This involves calculating the probability of a variable in the

future or a variable we haven't observed yet, given the evidence about other
variables. It's essentially about predicting the likelihood of certain outcomes based
on the current state of the network.

There are several inference algorithms used to perform this task efficiently, including:
 1. Variable Elimination: This algorithm computes the marginal probability distribution
of a subset of variables by systematically eliminating variables from the network
based on evidence.

2. Belief Propagation: Also known as message passing, this algorithm propagates

beliefs (probability distributions) between nodes in the network based on local
computations.

3. Junction Tree Algorithm: This algorithm constructs a junction tree (also known as a
clique tree) from the Bayesian network and uses it to perform efficient inference.

4. Gibbs Sampling and Markov Chain Monte Carlo (MCMC) Methods: These methods
are used for approximate inference in Bayesian networks, especially when the
network structure is complex and exact inference is computationally infeasible.

Benefits of Inference with Bayesian Networks:

 Efficient Handling of Uncertainty: Bayesian networks can represent and reason

about uncertainty inherent in real-world problems.
 Incorporating Prior Knowledge: The network structure captures causal relationships,
allowing for incorporating prior knowledge into the inference process.
 Flexible for Different Inference Types: The same network can be used for both
explanatory and predictive inference based on the type of question you're asking.

Applications of Inference with Bayesian Networks:

 Medical Diagnosis: Doctors can use Bayesian networks to diagnose diseases by

considering symptoms, test results, and patient history.
 Spam Filtering: Email filters can leverage Bayesian networks to identify spam emails
based on keywords and past behaviour patterns.
 Fraud Detection: Financial institutions can use Bayesian networks to detect
fraudulent transactions by analyzing spending patterns and account information.

Challenges of Inference:

 Complexity of Algorithms: For complex networks with many variables, inference

algorithms can become computationally expensive.
 Accuracy of Priors: The accuracy of the results depends on the accuracy of the prior
probabilities assigned to variables in the network.
Overall, inference in Bayesian networks is a powerful tool for reasoning under uncertainty
and making informed decisions in various domains.