Directed Graphical Models

Michael Gutmann

Probabilistic Modelling and Reasoning (INFR11134)

School of Informatics, University of Edinburgh

Spring Semester 2020

Recap

I Statistical independence assumptions facilitate the efficient

representation of probabilistic models by limiting the number
of variables that are allowed to directly interact with each
other.
I Statistical independencies lead to a (partial) factorisation of
pdfs/pmfs
I Equivalence between factorisation and ordered Markov
property
I Visualisation of pdfs/pmfs as directed graph

Michael Gutmann Directed Graphical Models 2 / 45

Program

1. Definition of directed graphical models

2. Three canonical connections in a DAG and their properties

3. Independencies in directed graphical models

Michael Gutmann Directed Graphical Models 3 / 45

Program

1. Definition of directed graphical models

Definition via factorisation according to the graph
Definition via ordered Markov property
Derive independencies from the ordered Markov property
with different topological orderings

2. Three canonical connections in a DAG and their properties

3. Independencies in directed graphical models

Michael Gutmann Directed Graphical Models 4 / 45

Directed graphical model

I We started with a pdf/pmf, wrote it in factorised form

according to some ordering, and associated a DAG with it.
I We can also go the other way around and start with a DAG.
I Definition (via factorisation property) A directed graphical
model based on a DAG with d nodes and associated random
variables xi is the set of pdfs/pmfs that factorise as
d
Y
p(x1 , . . . , xd ) = p(xi |pai ),
i=1

where pai denotes the parents of xi in the graph.

I Remark: a pdf/pmf p(x1 , . . . , xd ) that can be written in the
above form is said to “factorise over the graph”.

Michael Gutmann Directed Graphical Models 5 / 45

Example
DAG:
a z

q h

Random variables: a, z, q, e, h

Parent sets: paa = paz = ∅, paq = {a, z}, pae = {q}, pah = {z}.

Directed graphical model: set of pdfs/pmfs p(a, z, q, e, h) that

factorise as:

p(a, z, q, e, h) = p(a)p(z)p(q|a, z)p(e|q)p(h|z)

Michael Gutmann Directed Graphical Models 6 / 45

Alternative definition of directed graphical models

I For any DAG with d nodes we can always find an ordering of

the associated random variables that is topological to the
DAG. Re-label the nodes accordingly as x1 , . . . , xd .
I Recall: in topological orderings, the parents always come
before the children.
I Hence: pai ⊆ prei whatever topological ordering we picked.
I The derived equivalence of factorisation and ordered Markov
property, with the pai as the πi , thus yields the result:
d
Y
p(x) = p(xi |pai ) ⇐⇒ xi ⊥
⊥ (prei \ pai ) | pai for all i
i=1

I A since pai ⊆ prei whatever the topological ordering, the

result holds for all topological orderings.

Michael Gutmann Directed Graphical Models 7 / 45

Alternative definition of directed graphical models

I Two consequences:
I For a given DAG, the independencies derived from the ordered
Markov property with any topological ordering imply the
independencies derived with any other topological ordering.
I The insensitivity to the particular topological ordering used
provides an alternative definition of directed graphical models.
I Definition (via ordered Markov property) A directed graphical
model based on a DAG with d nodes and associated random
variables xi is the set of pdfs/pmfs that satisfy the ordered
Markov property

xi ⊥
⊥ (prei \ pai ) | pai for all i

for an ordering x1 , . . . , xd of the xi that is topological to the

DAG.

Michael Gutmann Directed Graphical Models 8 / 45

Example
DAG:
a z

q h

Random variables: a, z, q, e, h
Ordering: (a, z, q, e, h) (meaning: x1 = a, x2 = z, x3 = q, x4 = e, x5 = h)
Predecessor sets for the ordering:
prea = ∅, prez = {a}, preq = {a, z}, pree = {a, z, q}, preh = {a, z, q, e}
Parent sets: as before
paa = paz = ∅, paq = {a, z}, pae = {q}, pah = {z}
All models in the set defined by the DAG satisfy xi ⊥
⊥ (prei \ pai ) | pai :

z⊥
⊥a e⊥
⊥ {a, z} | q h⊥
⊥ {a, q, e} | z
Michael Gutmann Directed Graphical Models 9 / 45
Example (different topological ordering)
DAG:
a z

q h

e
Ordering: (a, z, h, q, e)
Predecessor sets for the ordering:
prea = ∅, prez = {a}, preh = {a, z}, preq = {a, z, h}, pree = {a, z, h, q}
Parent sets: as before
paa = paz = ∅, pah = {z}, paq = {a, z}, pae = {q}
All models in the set defined by the DAG satisfy xi ⊥
⊥ (prei \ pai ) | pai :

z⊥
⊥a h⊥
⊥a|z q⊥
⊥ h | a, z e⊥
⊥ {a, z, h} | q
Note: the models also satisfy those obtained with the previous ordering:
z⊥
⊥a e⊥
⊥ {a, z} | q h⊥
⊥ {a, q, e} | z
Michael Gutmann Directed Graphical Models 10 / 45
Remarks
I By using different topological orderings you can generate possibly
different independence relations satisfied by the model.
(While they imply each other, deriving them from each other from the basic
definition of independence may not be straightforward.)
I Missing edges in a DAG cause the pai to be smaller than the prei ,
and thus lead to the independencies.
I The directed graphical model corresponds to a set of probability
distributions. Two views according to the two definitions: The set
includes all those distributions that you get
I by looping over all possible conditionals p(x |pa ),
i i
I by retaining, from all possible joint distributions over the x ,
i
those that satisfy the independencies given by the ordered
Markov property
I Individual pdfs/pmf in the set are typically also called a directed
graphical model (“overloading” of the name of the set and its elements).
I Other names for directed graphical models: belief network, Bayesian
network, Bayes network.
Michael Gutmann Directed Graphical Models 11 / 45
Example: Markov model

DAG:
x1 x2 x3 x4 x5

All models, i.e. pdfs/pmfs p(x), in the set factorise as

p(x) = p(x1 )p(x2 |x1 )p(x3 |x2 )p(x4 |x3 )p(x5 |x4 )
There is only one topological ordering: (x1 , x2 , . . . , x5 )
By ordered Markov property: all models in the set satisfy:
xi+1 ⊥
⊥ x1 , . . . , xi−1 | xi
(future independent of the past given the present)

Michael Gutmann Directed Graphical Models 12 / 45

Example: Probabilistic PCA, factor analysis, ICA
(PCA: principal component analysis; ICA: independent component analysis)

DAG:
x1 x2 x3

y1 y2 y3 y4 y5

Explains properties of (observed) yi through fewer (unobserved) xi .

Different further assumptions lead to different methods (more
later).
All models in the set factorise as p(x1 , x2 , x3 , y1 , . . . , y5 ) =
p(x1 )p(x2 )p(x3 )p(y1 |x1 , x2 , x3 )p(y2 |x1 , x2 , x3 ) . . . p(y5 |x1 , x2 , x3 )
With topological ordering (x1 , x2 , x3 , y1 , y2 , y3 , y4 , y5 ): All satisfy:
xi ⊥
⊥ xj y2 ⊥⊥ y1 | x1 , x2 , x3 y3 ⊥
⊥ y1 , y2 | x1 , x2 , x3
y4 ⊥
⊥ y1 , y2 , y3 | x1 , x2 , x3 y5 ⊥
⊥ y1 , y2 , y3 , y4 |x1 , x2 , x3
Michael Gutmann Directed Graphical Models 13 / 45
Program

1. Definition of directed graphical models

Definition via factorisation according to the graph
Definition via ordered Markov property
Derive independencies from the ordered Markov property
with different topological orderings

2. Three canonical connections in a DAG and their properties

3. Independencies in directed graphical models

Michael Gutmann Directed Graphical Models 14 / 45

Further independence properties?
I Parent-child links in the graph encode (conditional)
independence properties.
I Ordered Markov property yields sets of independence
assertions.
I Questions:
I For any triple of random variables (x , y , z), can we determine
whether x ⊥ ⊥ y | z holds?
I Does the graph induce or impose additional independencies on
any probability distribution that factorises over the graph?
I Important because
I it yields increased understanding of the properties of the model
I we can exploit the independencies e.g. for inference and
learning
I Approach: Investigate how probabilistic evidence that
becomes available at a node can “flow” through the DAG and
influence our belief about another node.
Michael Gutmann Directed Graphical Models 15 / 45
Program

1. Definition of directed graphical models

2. Three canonical connections in a DAG and their properties

Serial connection
Diverging connection
Converging connection
I-equivalence

3. Independencies in directed graphical models

Michael Gutmann Directed Graphical Models 16 / 45

Three canonical connections in a DAG

In a DAG, two nodes x , y can be connected via a third node z in

three ways:
1. Serial connection (chain, head-tail or tail-head)

x z y

2. Diverging connection (fork, tail-tail)

x z y

3. Converging connection (collider, head-head, v-structure)

x z y

Note: in any case, the sequence x , z, y forms a trail

Michael Gutmann Directed Graphical Models 17 / 45

Serial connection x z y

I Markov model is made up of serial connections

I Graph: x influences z, which in turn influences y but no direct
influence from x to y .
I Factorisation: p(x , z, y ) = p(x )p(z|x )p(y |z)
I Ordered Markov property: y ⊥ ⊥ x |z
If the state or value of z is known (i.e. if the random variable
z is “instantiated”), evidence about x will not change our
belief about y , and vice versa.

We say that the z node is “closed” and that the trail between
x and y is “blocked” by the instantiated z. In other words,
knowing the value of z blocks the flow of evidence between x
and y .

Michael Gutmann Directed Graphical Models 18 / 45

Serial connection x z y

I What can we say about the marginal distribution of (x , y )?

I By sum rule, joint probability distribution of (x , y ) is
Z
p(x , y ) = p(x )p(z|x )p(y |z)dz
Z
= p(x ) p(z|x )p(y |z)dz
6= p(x )p(y )

I In a serial connection, if the state of z is unknown, then

evidence or information about x will influence our belief about
y , and the other way around. Evidence can flow through z
between x and y .
I We say that the z node is “open” and the trail between x and
y is “active”.

Michael Gutmann Directed Graphical Models 19 / 45

Diverging connection x z y

I Graph for probabilistic PCA, factor analysis, ICA has such

connections (z correspond to the latents, x and y to the
observed)
I Graph: z influences both x and y . No directed connection
between x and y .
I Factorisation: p(x , y , z) = p(z)p(x |z)p(y |z)
I Ordered Markov property (with ordering z, x , y ): y ⊥
⊥x |z
If the state or value z is known, evidence about x will not
change our belief about y , and vice versa.
I As in serial connection, knowing z closes the z node, which
blocks the trail between x and y .

Michael Gutmann Directed Graphical Models 20 / 45

Diverging connection x z y

I What can we say about the marginal distribution of (x , y )?

I By sum rule, joint probability distribution of (x , y ) is
Z
p(x , y ) = p(z)p(x |z)p(y |z)dz
6= p(x )p(y )

I In a diverging connection, as in the serial connection, if the

state of z is unknown, then evidence or information about x
will influence our belief about y , and the other way around.
Evidence can flow through z between x and y .
I The z node is open and the trail between x and y is active.

Michael Gutmann Directed Graphical Models 21 / 45

Converging connection x z y

I Graph for probabilistic PCA, factor analysis, ICA has such

connections (z corresponds to an observed, x and y to two
latents)
I Graph: x and y influence z. No direction connection between
x and y .
I Factorisation: p(x , y , z) = p(x )p(y )p(z|x , y )
I Ordered Markov property: x ⊥ ⊥y
When we do not have evidence about z, evidence about x will
not change our belief about y , and vice versa.
I If no evidence about z is available, the z node is closed, which
blocks the trail between x and y .

Michael Gutmann Directed Graphical Models 22 / 45

Converging connection x z y

I This means that the marginal distribution of (x , y ) factorises:

p(x , y ) = p(x )p(y )
I Conditional distribution of (x , y ) given z?
p(x , y , z) p(x )p(y )p(z|x , y )
p(x , y |z) = =R
p(z) p(x )p(y )p(z|x , y )dx dy
6= p(x |z)p(y |z)
This means that x 6⊥ ⊥ y | z.
I If evidence or information about z is available, evidence about
x will influence the belief about y , and vice versa.
I Information about z opens the z-node, and evidence can flow
between x and y .
I Note: information about z means that z or one of its
descendents is observed (see tutorials).
(A node w is a descendant of z if there is a directed path from z to
w .)
Michael Gutmann Directed Graphical Models 23 / 45
Explaining away
Example: cpu power

pc

I One day your computer does not start and you bring it to a
repair shop. You think the issue could be the power unit or
the cpu.
I Investigating the power unit shows that it is damaged. Is the
cpu fine?
I Without further information, finding out that the power unit is
damaged typically reduces our belief that the cpu is damaged
power 6⊥
⊥ cpu | pc
I Finding out about the damage to the power unit explains
away the observed start-issues of the computer.
Michael Gutmann Directed Graphical Models 24 / 45
Summary

Connection z node p(x , y ) p(x , y |z)

y
x z
default: open x 6⊥
⊥y x⊥
⊥y |z
instantiated: closed
y
x z
default: open x 6⊥
⊥y x⊥
⊥y |z
instantiated: closed
y
x z
default: closed x⊥
⊥y x 6⊥
⊥y |z
with evidence: opens

Think of the z node as a valve or gate through which evidence

(probability mass) can flow. Depending on the type of the connection,
it’s default state is either open or closed. Instantiation/evidence acts as a
switch on the valve.
I-equivalence

I Same independence assertions for

x z y x z y x z y

I The graphs have different causal interpretations

Consider e.g. x ≡ rain; z ≡ street wet; y ≡ car accident
I This means that based on statistical dependencies
(observational data) alone, we cannot select among the
graphs and thus determine what causes what.
I The three directed graphs are said to be
independence-equivalent (I-equivalent).

Michael Gutmann Directed Graphical Models 26 / 45

Program

1. Definition of directed graphical models

2. Three canonical connections in a DAG and their properties

Serial connection
Diverging connection
Converging connection
I-equivalence

3. Independencies in directed graphical models

Michael Gutmann Directed Graphical Models 27 / 45

Program

1. Definition of directed graphical models

2. Three canonical connections in a DAG and their properties

3. Independencies in directed graphical models

D-separation
Directed local Markov property
Equivalences of the different Markov properties and the
factorisation
Markov blanket

Michael Gutmann Directed Graphical Models 28 / 45

Further independence relations

I Given the DAG below, what can we say about the

independencies for the set of probability distributions that
factorise over the graph?
I Is x1 ⊥
⊥ x2 ? x1 ⊥
⊥ x2 | x6 ? x2 ⊥
⊥ x3 | {x1 , x4 }?
I Ordered Markov properties give some independencies.
I Limitation: it only allows us to condition on parent sets.
I Directed separation (d-separation) gives further
independencies.

x1 x3

x5

x2 x4 x6

Michael Gutmann Directed Graphical Models 29 / 45

D-separation

Let X = {x1 , . . . , xn }, Y = {y1 , . . . , ym }, and Z = {z1 , . . . , zr } be

three disjoint sets of nodes in the graph. Assume all zi are
observed (instantiated).
I Two nodes xi and yj are said to be d-separated by Z if all
trails between them are blocked by Z .
I The sets X and Y are said to be d-separated by Z if every trail
from any variable in X to any variable in Y is blocked by Z .

Michael Gutmann Directed Graphical Models 30 / 45

D-separation

A trail between nodes x and y is blocked by Z if there is a node b

on the trail such that
1. either b is part of a head-tail or tail-tail connection along the
trail and b is in Z ,
x b y x b y

2. or b is part of a head-head (collider) connection along the

trail and neither b nor any of its descendants are in Z .
x b y

Michael Gutmann Directed Graphical Models 31 / 45

D-separation and conditional independence

Theorem: If X and Y are d-separated by Z

then X ⊥
⊥ Y | Z for all probability distributions that factorise over
the DAG.

For those interested: A proof can be found in Section 2.8 of Bayesian Networks
– An Introduction by Koski and Noble (not examinable)

Important because:
1. the theorem allows us to read out (conditional)
independencies from the graph
2. no restriction on the sets X , Y , Z
3. the theorem shows that independencies detected by
d-separation do always hold. They are “true positives”
(“soundness of d-separation”).

Michael Gutmann Directed Graphical Models 32 / 45

D-separation and conditional independence

Theorem: If X and Y are not d-separated by Z

then X 6⊥
⊥ Y | Z in some probability distributions that factorise
over the DAG.

For those interested: A proof sketch can be found in Section 3.3.1 of

Probabilistic Graphical Models by Koller and Friedman (not examinable).

“not d-separated” is also called “d-connected”

6⊥
⊥ means statistically dependent

Michael Gutmann Directed Graphical Models 33 / 45

D-separation and conditional independence

I It can also be that d-connected variables are independent for

some distributions.
I Example (Koller, Example 3.3): p(x , y ) with x , y ∈ {0, 1} and

p(y = 0|x = 0) = a p(y = 0|x = 1) = a

for a > 0 and some non-zero p(x = 0).

I Graph has arrow from x to y . Variables are not d-separated.
x y

I p(y = 0) = ap(x = 0) + ap(x = 1) = a,

which is p(y = 0|x ) for all x .
I p(y = 1) = (1 − a)p(x = 0) + (1 − a)p(x = 1) = 1 − a,
which is p(y = 1|x ) for all x .
I Hence: p(y |x ) = p(y ) so that x ⊥
⊥ y.

Michael Gutmann Directed Graphical Models 34 / 45

D-separation and conditional independence

I This means that d-separation does generally not reveal all

independencies in all probability distributions that factorise
over the graph.
I In other words, individual probability distributions that
factorise over the graph may have further independencies not
included in the set obtained by d-separation.
I We say that d-separation is not “complete” (“recall-rate” is
not guaranteed to be 100%).

Michael Gutmann Directed Graphical Models 35 / 45

Recipe to determine whether two nodes are d-separated

1. Determine all trails between x and y (note: direction of the

arrows does here not matter).
2. For each trail:
i Determine the default state of all nodes on the trail.
I open if part of a head-tail or a tail-tail connection
I closed if part of a head-head connection
ii Check whether the set of observed nodes Z switches the state of
the nodes on the trail.
iii The trail is blocked if it contains a closed node.
3. The nodes x and y are d-separated if all trails between them
are closed.

Michael Gutmann Directed Graphical Models 36 / 45

Example: Are x1 and x2 d-separated?

Follows from ordered Markov property, but let us answer it with d-separation.
1. Determine all trails between x1
and x2
2. For trail x1 , x4 , x2 x1 x3
i default state
ii conditioning set is empty x5
iii ⇒ Trail is blocked
For trail x1 , x3 , x5 , x4 , x2 x2 x4 x6

i default state
ii conditioning set is empty
iii ⇒ Trail is blocked x1 ⊥ ⊥ x2 for all probabil-
Trail x1 , x3 , x5 , x6 , x4 , x2 is ity distributions that factor-
blocked too (same arguments). ise over the graph.
3. ⇒ x1 and x2 are d-separated.

Michael Gutmann Directed Graphical Models 37 / 45

Example: Are x1 and x2 d-separated by x6 ?

x1 x3
1. Determine all trails between x1
and x2 x5
2. For trail x1 , x4 , x2
x2 x4 x6
i default state
ii influence of x6
iii ⇒ Trail not blocked
No need to check the other x1 ⊥⊥ x2 | x6 does generally
trails: x1 and x2 are not not hold for probability dis-
d-separated by x6 tributions that factorise over
the graph.

Michael Gutmann Directed Graphical Models 38 / 45

Example: Are x2 and x3 d-separated by x1 and x4 ?

1. Determine all trails between x2

and x3
2. For trail x3 , x1 , x4 , x2
i default state x1 x3
ii influence of {x1 , x4 }
x5
iii ⇒ Trail blocked
For trail x3 , x5 , x4 , x2 x2 x4 x6
i default state
ii influence of {x1 , x4 }
iii ⇒ Trail blocked
x ⊥ ⊥ x3 | {x1 , x4 } for all
Trail x3 , x5 , x6 , x4 , x2 is blocked 2
probability distributions that
too (same arguments).
factorise over the graph.
3. ⇒ x2 and x3 are d-separated by
x1 and x4 .

Michael Gutmann Directed Graphical Models 39 / 45

Directed local Markov property

I The independencies that you can obtain with the ordered

Markov property depend on the topological ordering chosen.
I We introduce the “directed local Markov property” that does
not depend on the ordering but only on the graph.
I We say that p(x) satisfies the directed local Markov property
with respect to a given DAG with parent sets pai if

xi ⊥
⊥ (nondesc(xi ) \ pai ) | pai

holds for all i, where nondesc(xi ) denotes the

non-descendants of xi .
I In other words, p(x) satisfying the directed local Markov
property means that

p(xi |nondesc(xi )) = p(xi |pai ) for all i

Michael Gutmann Directed Graphical Models 40 / 45

Directed local Markov property
I We now use d-separation to show an equivalence between
p(x) satisfying the ordered and the local Markov property.
I Result: If p(x) satisfies the ordered Markov property it also
satisfies the directed local Markov property and vice versa:

xi ⊥
⊥ (prei \ pai ) |pai ⇐⇒ xi ⊥
⊥ (nondesc(xi ) \ pai ) |pai

where nondesc(xi ) denotes the non-descendants of xi .

x1 x2
x5
xi ≡ x7 x4 x6
pa7 = {x4 , x5 , x6 }
x8 x7 x9
pre7 = {x1 , x2 , . . . , x6 }
nondesc(x7 ) in blue

Michael Gutmann Directed Graphical Models 41 / 45

Directed local Markov property

xi ⊥
⊥ prei \ pai |pai ⇐ xi ⊥ ⊥ nondesc(xi ) \ pai |pai follows because
{x1 , . . . , xi−1 } ⊆ nondesc(xi ) for all topological orderings
For ⇒ consider all trails from xi to {nondesc(xi ) \ pai }.

Two cases: move upwards or downwards:

(1) upward trails are blocked by the parents x1 x2
(2) downward trails must contain a head- x5
head (collider) connection because the xj ∈ x4 x6
{nondesc(xi ) \ pai } is a non-descendant.
x8 x7 x9
These paths are blocked because the collider
node or its descendants are never part of pai .

The result now follows because all paths from

xi to all elements in {nondesc(xi ) \ pai } are
blocked.

Michael Gutmann Directed Graphical Models 42 / 45

Summary of the equivalences

Given a DAG with nodes (random variables) xi and parent sets pai , we
have the following equivalences:
Qd
p(x) factorises over the DAG p(x) = i=1
p(xi |pai )
m
p(x) satisfies the ordered MP xi ⊥
⊥ prei \ pai | pai for all i
m
p(x) satisfies the directed local MP xi ⊥
⊥ nondesc(xi ) \ pai | pai for all i
m
p(x) satisfies the directed global MP independencies asserted by d-separation

(MP: Markov property)

Broadly speaking, the graph serves two related purposes:

1. it tells us how distributions factorise
2. it represents the independence assumptions made

Michael Gutmann Directed Graphical Models 43 / 45

Markov blanket
What is the minimal set of variables such that knowing their values
makes x independent from the rest?
From d-separation:
I Isolate x from its
ancestors
⇒ condition on parents
x
I Isolate x from its
descendants
⇒ condition on children
I Deal with collider
connection
In directed graphical models, the par-
⇒ condition on ents, children, and co-parents of x are
co-parents called its Markov blanket, denoted by
(other parents of the MB(x ). We have
x⊥ ⊥ {all vars \ x \ MB(x )} | MB(x ).
children of x )
Michael Gutmann Directed Graphical Models 44 / 45
Program recap

1. Definition of directed graphical models

Definition via factorisation according to the graph
Definition via ordered Markov property
Derive independencies from the ordered Markov property with different
topological orderings

2. Three canonical connections in a DAG and their properties

Serial connection
Diverging connection
Converging connection
I-equivalence

3. Independencies in directed graphical models

D-separation
Directed local Markov property
Equivalences of the different Markov properties and the factorisation
Markov blanket

Michael Gutmann Directed Graphical Models 45 / 45