Introduction to Probability and Random Variables

1. Introduction to Probability

Probability is a mathematical framework for quantifying uncertainty. It is used in various fields

such as engineering, economics, physics, and computer science to model random phenomena.

Basic Definitions:

 Experiment: A process that produces an uncertain outcome (e.g., tossing a coin).

 Sample Space (SS): The set of all possible outcomes of an experiment (e.g., S={H,T} for
a coin toss).
 Event: A subset of the sample space (e.g., getting heads when tossing a coin).
 Probability of an Event (P(A): A measure of how likely an event is to occur, ranging
from 0 to 1.

Types of Probability:

 Classical Probability: If all outcomes are equally likely,

 Empirical Probability: Based on experiment or observation:

 Axiomatic Probability: Based on axioms by Kolmogorov:

Conditional Probability and Independence:

Bayes' Theorem:

Used for updating probabilities based on new evidence:

2. Random Variables

A random variable is a numerical value assigned to each outcome of a random experiment.

Types of Random Variables:

 Discrete Random Variable: Takes a finite or countably infinite number of values (e.g.,
number of heads in a coin toss).
 Continuous Random Variable: Takes values from a continuous range (e.g.,
temperature, height).

Probability Distributions:

 Probability Mass Function (PMF) for Discrete Variables:

 Probability Density Function (PDF) for Continuous Variables:

Cumulative Distribution Function (CDF):

Expectation and Variance:

 Expected Value (Mean):

 Variance (Measure of Spread):

Numerical Problems on Probability and Random Variables

1. Basic Probability Problems

Q1: A bag contains 5 red balls, 3 blue balls, and 2 green balls. A ball is drawn at random.
Find the probability of drawing:
a) A red ball
b) A blue ball
c) A non-green ball

Solution:

Q2: A fair die is rolled. What is the probability of getting:

a) An even number
b) A number greater than 4

Solution:

Q3: A deck of 52 playing cards is shuffled, and one card is drawn. What is the
probability of drawing:
a) A King
b) A red card (hearts or diamonds)
c) A face card (King, Queen, or Jack)?

Solution:
2. Conditional Probability

Q4: A box contains 6 white and 4 black balls. Two balls are drawn one after the other
without replacement. Find the probability that both are white.

Solution:

Q5: A factory produces 60% of its products from Machine A and 40% from Machine B.
The probability of a defective product is 3% from Machine A and 5% from Machine B. If
a product is found defective, what is the probability that it came from Machine A?

Solution:
3. Bayes' Theorem

Q6: A factory has 3 machines: A, B, and C. They produce 50%, 30%, and 20% of the
total products, respectively. The defect rates for A, B, and C are 2%, 3%, and 5%. If a
randomly selected product is defective, find the probability it was produced by Machine
A.

Solution:
4. Random Variables & Expected Value

Q7: A discrete random variable X has the following probability distribution:

Solution:
Q8: A company assigns bonuses based on employee performance. The probability
distribution of bonuses (in thousands of dollars) is given as follows:

Solution:

Continuous Probability

Q9: The probability density function (PDF) of a continuous random variable X is given
by:
 Solution:

Q10: A continuous random variable X has a probability density function (PDF) given by:

Solution:

Q11: The probability density function (PDF) of a random variable X is given as:
Solution:

Q12: The cumulative distribution function of a random variable X is given by:

Solution:

Statistical Averages, Random Processes, and Mean

Statistical Averages

Statistical averages are numerical values that represent the central or typical value of a set of
data. The three most commonly used statistical averages are:

a) Mean (Arithmetic Average)

The mean is the sum of all values divided by the number of values.

Example:
Problem 1:
The ages of 6 employees in a company are 25, 30, 35, 40, 45, 50. Find the mean age.

Solution:

b) Median

The median is the middle value when the data is arranged in ascending order. If there is an even
number of values, the median is the average of the two middle numbers.

Example:
For the data set 10, 20, 30, 40, 50, the median is 30 (middle value).
For 10, 20, 30, 40, 50, 60, the median is (30+40)/2 = 35.

Problem 2:
Find the median of the dataset 15, 22, 26, 35, 40, 55, 60.

Solution:
Since there are 7 numbers (odd count), the median is the middle value.

Median = 35

Answer: The median is 35.

c) Mode

The mode is the value that appears most frequently in a dataset.

Example:
For the data set 4, 5, 6, 6, 7, 8, 9, the mode is 6 because it appears twice.
Problem 3:
Find the mode for the dataset 2, 3, 3, 5, 5, 5, 7, 8, 8.

Solution:
The most frequently occurring value is 5 (appears 3 times).

Answer: The mode is 5.

2. Random Processes

A random process (or stochastic process) is a collection of random variables indexed by time
or space. It describes systems that evolve unpredictably over time.

Types of Random Processes:

1. Stationary Process – The statistical properties (mean, variance) do not change over time.
2. Ergodic Process – Time averages are equal to ensemble averages.
3. Markov Process – The future state depends only on the present state, not past states.
4. Gaussian Process – All random variables follow a normal (Gaussian) distribution.
5. White Noise – A process with zero mean and constant variance.

Example:
The daily stock market prices can be considered a random process because they change
unpredictably.

3. Mean of a Random Process

Example:
Problem 4:

Solution:

Problem 5

A discrete random variable X has the following probability distribution:

Solution:

Numerical on Variance of a Random Variable

Problem 6:
Solution:

Problem 7:

Solution:
Problem 8: Numerical on Mean of a Random Process

Solution:

Problem 9:
A continuous random variable X has the probability density function (PDF):

Solution:
 Covariance Function

Definition

Covariance measures the direction of the linear relationship between two random variables. It
tells us whether an increase in one variable tends to result in an increase (or decrease) in another.

The covariance between two random variables X and Y is defined as:

Expanded Formula:

Interpretation
 Correlation Function

Definition

Correlation is a normalized version of covariance that measures both the strength and
direction of the linear relationship between two variables.

Interpretation

Key Differences Between Covariance and Correlation

 Covariance Between Two Random Variables:

Problem 10:
Let X and Y be two random variables with:

Solution:

Covariance and Correlation Example

Let’s consider the following dataset:

Problem 11: Finding Covariance

Given two discrete random variables X and Y, their joint probabilities are:

Solution:
Problem 12: Finding Correlation

Using the values from Problem 11, calculate the correlation coefficient.

Solution:
 Ergodic Process in Random Processes
A random process (or stochastic process) is a collection of random variables indexed by time or
space. An Ergodic Process is a special type of stochastic (random) process where time averages
and ensemble averages are the same. This means that observing the process over a long period
gives the same result as averaging over all possible outcomes at a fixed point in time. Examples
include:

 Daily stock market prices

 Weather conditions over time
 Audio signals

Key Concepts of Ergodic Processes:

1. Stationarity:
o A process must be stationary (its statistical properties don’t change over time) to be
ergodic, but not all stationary processes are ergodic.
2. Mean Ergodicity:
o Time average of a single realization = Statistical average (ensemble average).

3. Autocorrelation Ergodicity:

 The autocorrelation calculated over time = The statistical autocorrelation.


Example of an Ergodic Process:

White Noise:

Non-Ergodic Example:

Random Walk:

Numerical Problem to Illustrate Ergodicity:

Problem:

Solution:
 Power Spectral Density (PSD)

Definition

The Power Spectral Density (PSD) describes how the power of a signal is distributed over
different frequencies. It is especially useful in analyzing random processes in the frequency
domain.

Mathematical Representation
Relationship Between PSD and Autocorrelation Function

The PSD and autocorrelation function are a Fourier Transform pair:

This means that if we know either the autocorrelation function or the PSD, we can compute the
other using the Inverse Fourier Transform.

Gaussian Process

Definition

A Gaussian Process (GP) is a stochastic process where any finite set of samples follows a
multivariate normal distribution. It is widely used in probability theory, machine learning,
and statistical signal processing.
Mathematical Definition

Mathematically, a Gaussian Process is Defined by:

2. Properties of a Gaussian Process

1. Fully Defined by Mean and Covariance

Unlike general stochastic processes, a Gaussian process is uniquely determined by its
first-order (mean) and second-order (covariance) statistics.
2. Any Linear Combination is Gaussian

3. Stationarity Conditions: A Gaussian process can be:

 Strictly stationary if all its statistical properties remain constant over time.
  Wide-sense stationary (WSS) if its mean is constant and autocorrelation depends
only on time difference.

4. White Gaussian Noise (WGN)

Types of Gaussian Processes

1. Stationary Gaussian Process: The statistical properties do not change over time.
2. Non-Stationary Gaussian Process: The mean and covariance change over time.

Example: White Gaussian Noise

Numerical Example

Problem: Compute the PSD of a Random Process

Solution:

Using the Fourier Transform:

Information Measure of Continuous Random Variables

Information theory deals with quantifying uncertainty and information in random variables.
In the case of continuous random variables, we use differential entropy instead of discrete
entropy.

(a) Differential Entropy

Key Properties of Differential Entropy

1. Not Always Non-Negative: Unlike discrete entropy, differential entropy can be negative.
2. Depends on Units: The entropy depends on the base of the logarithm (base 2 for bits, base e for
nats).
3. Maximization Property:

(b) Mutual Information for Continuous Variables

Mutual information quantifies the amount of information that one random variable provides
about another.
1. Differential Entropy Numerical

Problem 1: Differential Entropy of a Uniform Distribution

Solution:
 Jensen’s Inequality

Jensen’s Inequality is a fundamental result in convex analysis and is widely used in probability,
statistics, and information theory.

(a) Statement of Jensen’s Inequality

(b) Applications of Jensen’s Inequality in Information Theory

Jensen’s Inequality Numerical

Problem 1: Applying Jensen’s Inequality to Logarithm

Solution:
 Fano’s Inequality

Fano’s Inequality provides a lower bound on the probability of error in a classification or

estimation problem. It is particularly useful in channel coding and information theory.

(a) Statement of Fano’s Inequality

(b) Interpretation and Applications

1. Error Probability Bound in Communication

o It shows that if a receiver cannot extract enough information, then the probability of
making an error cannot be arbitrarily small.
o Used in Shannon’s Channel Capacity Theorem.
2. Machine Learning and Estimation
o Used in proving lower bounds on classification error rates.
o Helps in feature selection in machine learning by quantifying uncertainty.

Fano’s Inequality Numerical

Problem 1: Estimating Error Probability Using Fano’s Inequality

Solution: