Statistics

CVEN2002/2702

Week 4

This lecture
4. Random variables
4.1 Introduction
4.2 Random variables
4.3 Discrete Random Variables
4.4 Continuous Random Variables
4.5 Expectation of a random variable
4.6 Variance of a random variable
4.7 Jointly distributed Random Variables

Additional reading: Sections 5.4, 1.3 and 3.6 in the textbook

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

2 / 50

4. Random variables

4. Random variables

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

3 / 50

4. Random variables

4.1 Introduction

Introduction
Often, we are not interested in all of the details of an experiment but
only in the value of some numerical quantity determined by the
outcome

Example 1: tossing two dice when playing a board game

S = {(1, 1), (1, 2), . . . , (6, 5), (6, 6)}
... but often only the sum of the points matters
each possible outcome is characterised by a real number

Example 2: buying 2 electronic items

each of which may be either defective or acceptable
S = {(d, d), (d, a), (a, d), (a, a)}
... but we might only be interested in the number of acceptable items
obtained in the purchase
again, each possible outcome is characterised by a real number

It is often much more natural to directly think in terms of the numerical

quantity of interest, called a random variable
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

4 / 50

4. Random variables

4.2 Random variables

Random variable: definition

Definition
A random variable is a real-valued function defined over the sample
space:
X :SR
X ()

Usually , a random variable is denoted by an uppercase letter

Define SX the domain of variation of X , that is the set of possible
values taken by X

Example 1: tossing two dice when playing a board game

X = sum of the points,

SX = {2, 3, 4, . . . , 12}

Example 2: buying 2 electronic items

X = number of acceptable items,

SX = {0, 1, 2}

except in your textbook

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

5 / 50

4. Random variables

4.2 Random variables

Events defined by random variables

For any fixed real value x SX , assertions like X = x or X x
correspond to a set of possible outcomes
(X = x) = { S : X () = x}

(X x) = { S : X () x}
they are events !

meaningful to talk about their probability

Example 1 (ctd.) - If the dice are fair

(X = 2) = {(1, 1)}
P(X = 2) = 1/36
(X 11) = {(5, 6), (6, 5), (6, 6)} P(X 11) = 3/36 = 1/12
The usual properties of probabilities of course apply to these ones,
e.g.
P(X SX ) = 1
P((X = x1 ) (X = x2 )) = P(X = x1 ) + P(X = x2 )
(if x1 = x2 )
P(X < x) = 1 P(X x) (X < x is the complement of X x)
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

6 / 50

4. Random variables

4.2 Random variables

Notes
Note 1
It is important not to confuse:
X , the name of the random variable
X (), the numerical value taken by the random variable at some
sample point
x, a generic numerical value

Note 2
Most interesting problems can be stated, often naturally, in terms of
random variables
many inessential details about the sample space can be left
unspecified, and one can still solve the problem
often more helpful to think of random variables simply as variables
whose values are likely to lie within certain ranges of the real
number line
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

7 / 50

4. Random variables

4.2 Random variables

Cumulative distribution function

A random variable is essentially described by its cumulative
distribution function (cdf) (or just distribution)

Definition
The cdf of the random variable X is defined for any real number x, by
F (x) = P(X x)
All probability questions about X can be answered in terms of its
distribution. We will denote X F (read X follows the distribution F )
Some properties:
for any a b, P(a < X b) = F (b) F (a)
F is a nondecreasing function
limx+ F (x) = F (+) = 1
limx F (x) = F () = 0
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

8 / 50

4. Random variables

4.2 Random variables

0.8

0.8

0.8

1.0

1.0

1.0

Cumulative distribution functions

0.6
FX(x)

0.6

0.4

FX(x)
x

Continuous distribution
continuous r.v.

0.2

0.0

0.2
0.0

0.4

0.2
0.0

0.4

FX(x)

0.6

Discrete distribution
discrete r.v.

Hybrid distribution
hybrid r.v.

Note: hybrid distributions will not be introduced in this course

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

9 / 50

4. Random variables

4.3 Discrete Random Variables

Discrete random variables

Definition
A random variable is said to be discrete if it can only assume a finite
(or at most countably infinite) number of values
Suppose that those values are SX = {x1 , x2 , . . .}

Definition
The probability mass function (pmf) of a discrete random variable X is
defined for any real number x, by
p(x) = P(X = x)
pX (x) > 0 for x = x1 , x2 , . . ., and pX (x) = 0 for any other value of x
Obviously:
P(X SX ) = P((X = x1 ) (X = x2 ) . . .) =
CVEN2002/2702 (Statistics)

Dr Justin Wishart

p(x) = 1

xSX
Session 2, 2012 - Week 4

10 / 50

4. Random variables

4.3 Discrete Random Variables

0.8

1.0

probability mass function

pX(x)

0.6

Probability mass function:

0.4

spikes at x1 , x2 , . . .

height of spike at xi = p(xi )

0.2

0.0

1.0

cumulative distribution function

0.8

FX(x)

0.6

Cumulative distribution function:

"
F (x) = i:xi x p(xi )

0.4

step function

0.2

0.0

jumps at x1 , x2 , . . .
magnitude of jump at xi = p(xi )

CVEN2002/2702 (Statistics)
x

Dr Justin Wishart

Session 2, 2012 - Week 4

11 / 50

4. Random variables

4.3 Discrete Random Variables

Discrete random variables: examples

Examples of discrete random variables include:
number of scratches on a surface, number of defective parts among 1000
tested, number of transmitted bits received in error, the sum of the points
when tossing 2 dice, . . .
discrete random variables generally arise when we count things

Example: tossing 2 dice

X = sum of the points; represent p(x) and F (x)
Check that p(x) = (6 |7 x|)/36 for x SX = {2, 3, 4, . . . , 12}
cdf

1.0

0.20

pmf

0.15

0.8

F(x)

0.10

0.2

0.05

0.4

p(x)

0.6

0.0

0.00

CVEN2002/2702 (Statistics)

8
x

10

12

14

Dr Justin Wishart

8
x

10

12

14

Session 2, 2012 - Week 4

12 / 50

4. Random variables

4.3 Discrete Random Variables

Bernoulli random variable

Named after the Swiss scientist Jakob Bernoulli (1654-1705)
That is the simplest random variable
It can only assume 2 values, SX = {0, 1}
Its pmf is given by
p(1) =
p(0) = 1
for some value
It is often used to characterise the occurrence/non-occurrence of a
given event, or the presence/absence of a given feature

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

13 / 50

4. Random variables

4.4 Continuous Random Variables

0.8

0.8

0.8

1.0

1.0

1.0

Cumulative distribution functions

0.6
FX(x)

0.6

0.4

FX(x)
x

Continuous distribution
continuous r.v.

CVEN2002/2702 (Statistics)

0.2

0.0

0.2
0.0

0.4

0.2
0.0

0.4

FX(x)

0.6

Discrete distribution
discrete r.v.

Dr Justin Wishart

Hybrid distribution
hybrid r.v.

Session 2, 2012 - Week 4

14 / 50

4. Random variables

4.4 Continuous Random Variables

Continuous random variables

As opposed to a discrete r.v., a continuous random variable X is
expected to take on an uncountable number of values. SX is therefore
an uncountable set of real numbers (like an interval), and can even be
R itself. However, this is not enough as a definition

Definition
A random variable X is said to be continuous if there exists a
nonnegative function f (x) defined for all real x R such that for any
set B of real numbers,
#
P(X B) =
f (x)dx
B

Consequence: P(X = x) = 0 for any x !

the probability mass function is useless
the probability density function (pdf) f (x) will play the central role
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

15 / 50

4. Random variables

4.4 Continuous Random Variables

Continuous random variables: remark

Note 1: the fact that P(X = x) = 0 for any x should not be disturbing
coherent when dealing with measurements, e.g.
if we report a temperature of 74.8 degrees centigrade, owing to the
limits of our ability to measure (accuracy of measuring devices), we
really mean that the temperature lies close to 74.8, for instance
between 74.75 and 74.85 degrees
Note 2: when we say that there is a zero probability that a random
variable X will take on any value x, this does not mean that it is
impossible that X will take on the value x!
In the continuous case, zero probability does not imply logical
impossibility
this should not be disturbing either, as we are always interested in
probabilities connected with intervals and not with isolated points
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

16 / 50

4. Random variables

4.4 Continuous Random Variables

Probability density function: properties

F (x) = P(X x) =

f (y )dy , that is

f (x) =

dF (x)
dx

= F (x)
(wherever F is differentiable)

f (x) 0

x R
(F (x) is nondecreasing)
# b
P(a X b) =
f (x)dx = F (b) F (a)
a

f (x)dx = 1

for a small , P(x /2 X x + /2) =

x+/2
x/2

f (y )dy f (x)

SX = {x R : f (x) > 0}
Note: as P(X = x) = 0, P(X < x) = P(X x) (for a continuous r.v.)
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

17 / 50

4. Random variables

4.4 Continuous Random Variables

Continuous random variables: examples

Examples of continuous random variables include: electrical current, length,
pressure, temperature, time, voltage, weight, speed of a car, amount of
alcohol in a persons blood, efficiency of solar collector, strength of a new
alloy, . . .
continuous random variables generally arise when we measure
things

Example
Let X denote the current measured in a thin copper wire (in mA). Assume
$
that the pdf of X is
C(4x 2x 2 ) if 0 < x < 2
f (x) =
0
otherwise
What is the value of C ? Find P(X > 1.8)
%2
% +
We must have f (x) dx = 1, so C 0 (4x 2x 2 ) dx = C 83 = 1, that is
C = 3/8
% +
%2
Then, P(X > 1.8) = 1.8 f (x) dx = 3/8 1.8 (4x 2x 2 ) dx = 0.028
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

18 / 50

4. Random variables

CVEN2002/2702 (Statistics)

4.4 Continuous Random Variables

Dr Justin Wishart

Session 2, 2012 - Week 4

19 / 50

4. Random variables

4.4 Continuous Random Variables

Discrete vs. Continuous random variables

Discrete r.v.

Continuous r.v.

Domain of variation
SX = {x1 , x2 , . . .}

SX = [, ] R

Probability mass function (pmf)

p(x) = P(X = x) 0 for all x R
p(x) > 0 if and only if x SX
"
xSX p(x) = 1

useless: p(x) 0

Probability density function (pdf)

f (x) = F (x) 0 for all x R
f (x) > 0 if and only if x SX
%
f (x) = 1
xSX

does not exist

Note the similarity between the conditions for pmf and pdf
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

20 / 50

4. Random variables

4.5 Expectation of a random variable

Parameters of a distribution
Fact
Some quantities characterise a random variable more usefully
(although incompletely) than the whole cumulative distribution function
The focus is on certain general properties of the distribution of the
r.v.
The two most important such quantities are:
the expectation (or mean) and
the variance
of a random variable
Often, we talk about the expectation or the variance of a distribution,
understood as the expectation or the variance of a random variable
having that distribution
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

21 / 50

4. Random variables

4.5 Expectation of a random variable

Expectation
The expectation or the mean of a random variable X , denoted E(X )
or , is defined by

Discrete r.v.
= E(X ) =

Continuous r.v.
!

x p(x)

= E(X ) =

xSX

x f (x)dx
SX

E(X ) is a weighted average of the possible values of X , each value

being weighted by the probability that X assumes it
Note: E(X ) has the same units as X

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

22 / 50

4. Random variables

4.5 Expectation of a random variable

Expectation
Expectation = expected value, mean value, average value of X
= central value, around which X is distributed
= centre of gravity of the distribution
In the discrete case:

localisation parameter
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

23 / 50

4. Random variables

4.5 Expectation of a random variable

Expectation: examples
Example 1
What is the expectation of the outcome when a fair die is rolled?
X = outcome, SX = {1, 2, 3, 4, 5, 6} with p(x) = 1/6 for any x SX

= E(X ) = 1 1/6 + 2 1/6 + 3 1/6 + 4 1/6 + 5 1/6 + 6 1/6

= 3.5

need not be a possible outcome !

is not the most likely outcome (this is called the mode)

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

24 / 50

4. Random variables

4.5 Expectation of a random variable

Example 2
What is the expected sum when two fair dice are rolled?
X = sum of the two dice,
SX = {2, 3, . . . , 12} with
p(x) = (6 |7 x|)/36 for any x SX
= E(X ) = 2 1/36 + 3 2/36 + . . . + 12 1/36 = 7

Example 3: Bernoulli r.v. (see Slide 13)

What is the expectation of a Bernoulli r.v.?
E(X ) = 0 (1 ) + 1 =

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

25 / 50

4. Random variables

4.5 Expectation of a random variable

Expectation: examples
Example 4
Find the mean value of the copper current measurement X for Example on
Slide 18, that is, with
$ 3
2
8 (4x 2x ) if 0 < x < 2
f (x) =
0
otherwise
The density is

By symmetry, it can be directly

concluded that = 1 mA

0.4

# 2

3
=
8
=1

0.0

0.2

f(x)

0.6

It can also be easily checked that

# +
x f (x) dx
= E(X ) =

0.0

0.5

1.0

1.5

2.0

x (4x 2x 2 ) dx

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

26 / 50

4. Random variables

4.5 Expectation of a random variable

Expectation of a function of a random variable

Sometimes we are not interested in the expected value of X , but in the
expected value of a function of X , say g(X )
There is actually no need for explicitly deriving the distribution of g(X )
Indeed, it can be shown

If X is a discrete r.v.
E(g(X )) =

If X is a continuous r.v.

g(x) p(x)

E(g(X )) =

xSX

g(x) f (x) dx
SX

In particular, for 2 constants a and b:

Linear transformation
E(aX + b) = aE(X ) + b
Proof: . . .
With a = 0 E(b) = b
CVEN2002/2702 (Statistics)

(degenerate random variable)

Dr Justin Wishart

Session 2, 2012 - Week 4

27 / 50

4. Random variables

4.6 Variance of a random variable

Variance of a random variable

Definition
The variance of a random variable X , usually denoted by Var(X ) or
2 , is defined by
&
'
Var(X ) = E (X )2
Clearly, Var(X ) 0

If X is a discrete r.v.
2 = Var(X ) =

xSX

If X is a continuous r.v.

(x )2 p(x)

2 = Var(X ) =

(x)2 f (x)dx
SX

expected square of the deviation of X from its expected value

the variance quantifies the dispersion of the possible values of X
around the central value , that is, the variability of X
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

28 / 50

4. Random variables

4.6 Variance of a random variable

Variance: illustration
Two random variables X1 and X2 , with E(X1 ) = E(X2 )

0.6

0.8

1.0

pdf of X2
variability of X2

fX(x)

fX(x)

0.6

0.8

1.0

pdf of X1

0.4
0.2

0.2

0.4

variability of X1

E(X2) = 0

0.0

0.0

E(X1) = 0
1

Var(X1 ) > Var(X2 )

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

29 / 50

4. Random variables

4.6 Variance of a random variable

Variance: notes
Note 1
An alternative formula for Var(X ) is the following:
2 = Var(X ) = E(X 2 ) (E(X ))2 = E(X 2 ) 2
Proof: . . .
in practice, this is often the easiest way to compute Var(X ), using
#
!
2
2
2
E(X ) =
x p(x) or E(X ) =
x 2 f (x)dx
SX

xSX

Note 2
The variance 2 is not in the same units as X , which may make
interpretation difficult
often, we adjust for this by taking the square root of 2

(
This is called the standard deviation of X : = 2 = Var(X )
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

30 / 50

4. Random variables

4.6 Variance of a random variable

Variance: linear transformation

A useful identity is that, for any constants a and b, we have

Linear transformation
Var(aX + b) = a2 Var(X )
Proof: . . .
Take a = 1, it follows that for any b, Var(X + b) = Var(X )
variance not affected by translation
Take a = 0, if follows that for any b, Var(b) = 0
(degenerate random variable)

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

31 / 50

4. Random variables

4.6 Variance of a random variable

Variance : examples
Example 1
What is the variance of the number of points shown when a fair die is rolled?
X = outcome, SX = {1, 2, 3, 4, 5, 6} with p(x) = 1/6 for any x SX

E(X 2 ) = 12 1/6 + 22 1/6 + 32 1/6 + 42 1/6 + 52 1/6 + 62 1/6

= 91/6

We know that = 3.5 (Slide 23), so that

2 = E(X 2 ) 2 = 91/6 3.52 2.92

The standard deviation is = 2.92 1.71

Example 2
What is the variance of the sum of the points when 2 fair dice are rolled ?
(Exercise) Check that 2 5.83, 2.41
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

32 / 50

4. Random variables

4.6 Variance of a random variable

Variance: examples
Example 3
What is the variance of a Bernoulli r.v.?
E(X 2 ) = 02 (1 ) + 12 = = E(X )

Var(X ) = E(X 2 ) (E(X ))2 = 2 = (1 )

Example 4
What is the variance of the copper current measurement X for Example on
Slide 18, that is, with
$ 3
2
8 (4x 2x ) if 0 < x < 2
f (x) =
0
otherwise
We have E(X 2 ) =

% +

x 2 f (x) dx =

3
8

%2
0

x 2 (4x 2x 2 ) dx = 1.2

We know that = 1 (Slide 24), so that 2 = 1.2 12 = 0.2 mA2

0.45 mA
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

33 / 50

4. Random variables

4.6 Variance of a random variable

Standardisation
Standardisation is a very useful linear transformation
Suppose you have a random variable X with mean and variance 2 .
Then, the associated standardised random variable, often denoted Z ,
is given by
X
Z =
,

that is Z = 1 X . Hence, using the linear transformations properties,

E(Z ) =

E(X ) = = 0

2
1
Var(X
)
=
=1
2
2
a standardised random variable has always mean 0 and variance 1
Var(Z ) =

Note 1: Z is a dimensionless variable (no unit)

Note 2: a standardised value of X is sometimes called z-score
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

34 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Joint distribution function

Often, probability statements concerning two random variables, say X
and Y , defined on the same sample space are of interest:
(X (), Y ()))

these two variables are most certainly related

they should be jointly analysed, in order to understand the degree of
relationship between them
For instance, we may simultaneously measure the weight and
hardness of a rock, the pressure and temperature of a gas, thickness
and compressive strength of a piece of glass, etc.

Definition
The joint cumulative distribution function of X and Y is given by
FXY (x, y ) = P(X x, Y y )

(x, y ) R R

Note: (X x, Y y ) is the usual notation for (X x) (Y y )

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

35 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Joint distribution: discrete case

If X and Y are both discrete, the joint probability mass function is

defined by
pXY (x, y ) = P(X = x, Y = y )
The marginal pmf of X and Y can be obtained by
!
!
pX (x) =
pXY (x, y ) and pY (y ) =
pXY (x, y )
xSX

y SY

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

36 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Joint distribution: continuous case

Definition
X and Y are said to be jointly continuous if there exists a function
fXY (x, y ) : R R R+ such that for any sets A and B of real numbers
# #
P(X A, Y B) =
fXY (x, y )dy dx
A

The function fXY (x, y ) is the joint probability density of X and Y

The marginal densities follows from
#
# #
fX (x)dx = P(X A) = P(X A, Y SY ) =
A

fXY (x, y )dy dx

SY

Thus,
fX (x) =

fXY (x, y )dy

and fY (y ) =

SY

CVEN2002/2702 (Statistics)

Dr Justin Wishart

fXY (x, y )dx

SX
Session 2, 2012 - Week 4

37 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Expectation of a function of two random variables

For any function g : R R R, the expectation of g(X , Y ) is given by

! !

E(g(X , Y )) =

g(x, y )pXY (x, y )

discrete case

x SX y SY

CVEN2002/2702 (Statistics)

SX

g(x, y )fXY (x, y )dy dx

continuous case

SY

Dr Justin Wishart

Session 2, 2012 - Week 4

38 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Expectation of a function of two random variables

For instance, in the continuous case,
# #
(ax + by )fXY (x, y )dy dx
E(aX + bY ) =
SX SY
# #
# #
=
ax fXY (x, y )dy dx +
by fXY (x, y )dy dx
SX SY
SX SY
#
#
#
#
=a
x
fXY (x, y )dy dx + b
y
fXY (x, y )dx dy
SX
SY
SY
SX
#
#
yfY (y )dy
=a
xfX (x)dx + b
SX

SY

= aE(X ) + bE(Y )

Example
What is the expected sum obtained when two fair dice are rolled?
Let X be the sum and Xi the value shown on the ith die. Then, X = X1 + X2 ,
and
E(X ) = E(X1 ) + E(X2 ) = 2 3.5 = 7
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

39 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Independent random variables

Definition
The random variables X and Y are said to be independent if, for all
(x, y ) R R,
P(X x, Y y ) = P(X x) P(Y y )
In other words, X and Y are independent if all couples of events
(X x) and (Y y ) are independent
Characterisation: For any (x, y ) R R,

which reduces to
or

FXY (x, y ) = FX (x) FY (y ),

pXY (x, y ) = pX (x) pY (y )

fXY (x, y ) = fX (x) fY (y )
CVEN2002/2702 (Statistics)

in the discrete case

in the continuous case

Dr Justin Wishart

Session 2, 2012 - Week 4

40 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Independent random variables

Property
If X and Y are independent, then for any functions h and g,
E(h(X )g(Y )) = E(h(X )) E(g(Y ))
Proof (in the continuous case):

E(h(X )g(Y )) =

##

h(x)g(y )fXY (x, y )dy dx

SX SY

SX

SX

h(x)g(y )fX (x)fY (y )dy dx

#
h(x)fX (x)dx
g(y )fY (y )dy
SY

SY

= E(h(X )) E(g(Y ))
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

41 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Covariance of two random variables

Definition
The covariance of two random variables X and Y is defined by
)
*
Cov(X , Y ) = E (X E(X ))(Y E(Y ))
Properties (proofs are left as an exercise):
Cov(X , Y ) = Cov(Y , X )
Cov(X , X ) = Var(X )
Cov(X , Y ) = E(XY ) E(X )E(Y )
Cov(aX + b, cY + d) = ac Cov(X , Y )
Cov(X1 + X2 , Y1 + Y2 )
= Cov(X1 , Y1 ) + Cov(X1 , Y2 ) + Cov(X2 , Y1 ) + Cov(X2 , Y2 )
Note: unit of Cov(X , Y ) = unit of X unit of Y
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

42 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Covariance: interpretation
Suppose X and Y are two Bernoulli random variables
Then, XY is also a Bernoulli random variable which takes the value 1 if
and only if X = 1 and Y = 1. It follows:
Cov(X , Y ) = E(XY )E(X )E(Y ) = P(X = 1, Y = 1)P(X = 1)P(Y = 1)
Then,
Cov(X , Y ) > 0 P(X = 1, Y = 1) > P(X = 1)P(Y = 1)
P(X = 1, Y = 1)
> P(Y = 1)

P(X = 1)
P(Y = 1|X = 1) > P(Y = 1)
the outcome X = 1 makes it more likely that Y = 1
Y tends to increase when X does, and vice-versa
This result holds for any r.v. X and Y (not only Bernoulli r.v.)
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

43 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Covariance: interpretation
Cov(X , Y ) > 0 X and Y
tend to increase or decrease
together

Cov(X , Y ) < 0 X tends to

increase as Y decreases and
vice-versa

Cov(X , Y ) = 0 no linear
association between X and Y
(doesnt mean there is no
association!)

(X(), Y())

Fact
X and Y independent Cov(X , Y ) = 0

Cov(X , Y ) = 0 ! X and Y independent

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

44 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Covariance: examples
Example
Let the pmf of a r.v. X be pX (1) = pX (1) = 1/2 and Y = X 2 . Find Cov(X , Y )
We have Cov(X , Y ) = E(XY ) E(X )E(Y ) = E(X 3 ) E(X )E(X 2 ), but as X
only takes values 1 and 1, X 3 = X . It remains
Cov(X , Y ) = E(X )(1 E(X 2 ))
Also, E(X ) = (1) 1/2 + 1 1/2 = 0, so that
Cov(X , Y ) = 0
However, there is a direct functional dependence between X and Y

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

45 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Variance of a sum of random variables

From the properties of the covariance, it follows:
Var (aX + bY ) = Cov (aX + bY , aX + bY )
= Cov(aX , aX ) + Cov(aX , bY )
+ Cov(bY , aX ) + Cov(bY , bY )
= Var(aX ) + Var(bY ) + 2 Cov(aX , bY )
= a2 Var(X ) + b2 Var(Y ) + 2ab Cov(X , Y )
Now, if X and Y are independent random variables,
Var (aX + bY ) = a2 Var(X ) + b2 Var(Y )
For instance, if X and Y are independent,
Var(X + Y ) = Var(X ) + Var(Y )
Var(X Y ) = Var(X ) + Var(Y )
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

46 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Example
Example
We have two scales for measuring small weights in a laboratory. Assume the
true weight of an item is 2g. Both scales give readings which have mean 2g
and variance 0.05g2 . Compare using only one scale and using both scales
then averaging the two measures in terms of the accuracy.
The first measure X has E(X ) = 2 and Var(X ) = 0.05. Now, denote the
second measure Y , independent of X , with E(Y ) = 2 and Var(Y ) = 0.05.
Then, take W = X +Y
2 . We have
E(W ) =

1
1
2 2
E(X ) + E(Y ) = + = 2 (g)
2
2
2 2

and
Var(W ) =

1
1
(Var(X ) + Var(Y )) = (0.05 + 0.05) = 0.025 (g2 )
4
4

averaging 2 measures reduces the variance by 2

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

47 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Correlation
The covariance of two r.v. is important as an indicator of the
relationship between them
However, it heavily depends on units of X and Y (difficult
interpretation, not scale-invariant)
the correlation coefficient is often used instead
It is the covariance between the standardised versions of X and Y , or,
explicitly,
Cov(X , Y )
= (
Var(X ) Var(Y )
Properties:
is dimensionless (no unit)
always has a value between 1 and 1 (Cauchy-Schwarz ineq.)
positive (resp. negative) means positive (resp. negative) linear
relationship between X and Y
the closer || is to 1, the stronger is the linear relationship
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

48 / 50

4. Random variables

4.7 Jointly distributed Random Variables

Correlation examples

CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

49 / 50

4. Random variables

Objectives

Objectives
Now you should be able to:
understand the differences between discrete and continuous r.v.
for discrete r.v., determine probabilities from pmf and the reverse
for continuous r.v., determine probabilities from pdf and the reverse
for discrete r.v., determine probabilities from cdf and cdf from pmf and
the reverse
for continuous r.v., determine probabilities from cdf and cdf from pdf and
the reverse
calculate means and variances for both discrete and continuous random
variables
use joint pmf and joint pdf to calculate probabilities
calculate and interpret covariances and correlations between two
random variables
Recommended exercises Q25 p.220, Q27 p.221, Q29&30 p.221, Q69
p.57, Q40&43 p.152, Q42 p.152, Q41 p.223, Q65 p.239
CVEN2002/2702 (Statistics)

Dr Justin Wishart

Session 2, 2012 - Week 4

50 / 50