Lecture 2: Making Sequences of Good Decisions Given

a Model of the World

Emma Brunskill

CS234 Reinforcement Learning

Winter 2019

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 1 / 60
 Today’s Plan

Last Time:
Introduction
Components of an agent: model, value, policy
This Time:
Making good decisions given a Markov decision process
Next Time:
Policy evaluation when don’t have a model of how the world works

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 2 / 60
 Models, Policies, Values

Model: Mathematical models of dynamics and reward

Policy: Function mapping agent’s states to actions
Value function: future rewards from being in a state and/or action
when following a particular policy

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 3 / 60
 Today: Given a model of the world

Markov Processes
Markov Reward Processes (MRPs)
Markov Decision Processes (MDPs)
Evaluation and Control in MDPs

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 4 / 60
 Full Observability: Markov Decision Process (MDP)

MDPs can model a huge number of interesting problems and settings

Bandits: single state MDP
Optimal control mostly about continuous-state MDPs
Partially observable MDPs = MDP where state is history
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 5 / 60
 Recall: Markov Property

Information state: sufficient statistic of history

State st is Markov if and only if:

p(st+1 |st , at ) = p(st+1 |ht , at )

Future is independent of past given present

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 6 / 60
 Markov Process or Markov Chain

Memoryless random process

Sequence of random states with Markov property
Definition of Markov Process
S is a (finite) set of states (s ∈ S)
P is dynamics/transition model that specifices p(st+1 = s 0 |st = s)
Note: no rewards, no actions
If finite number (N) of states, can express P as a matrix
 
P(s1 |s1 ) P(s2 |s1 ) · · · P(sN |s1 )
 P(s1 |s2 ) P(s2 |s2 ) · · · P(sN |s2 ) 
P=
 
.. .. .. .. 
 . . . . 
P(s1 |sN ) P(s2 |sN ) · · · P(sN |sN )

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 7 / 60
 Example: Mars Rover Markov Chain Transition Matrix, P

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

 
0.6 0.4 0 0 0 0 0
0.4 0.2 0.4 0 0 0 0
 
 0 0.4 0.2 0.4 0 0 0
 
0
P= 0 0.4 0.2 0.4 0 0
0 0 0 0.4 0.2 0.4 0 
 
0 0 0 0 0.4 0.2 0.4
0 0 0 0 0 0.4 0.6

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 8 / 60
 Example: Mars Rover Markov Chain Episodes

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Example: Sample episodes starting from S4

s4 , s5 , s6 , s7 , s7 , s7 , . . .
s4 , s4 , s5 , s4 , s5 , s6 , . . .
s4 , s3 , s2 , s1 , . . .

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a Model Winter
of the World
2019 9 / 60
 Markov Reward Process (MRP)

Markov Reward Process is a Markov Chain + rewards

Definition of Markov Reward Process (MRP)
S is a (finite) set of states (s ∈ S)
P is dynamics/transition model that specifices P(st+1 = s 0 |st = s)
R is a reward function R(st = s) = E[rt |st = s]
Discount factor γ ∈ [0, 1]
Note: no actions
If finite number (N) of states, can express R as a vector

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 10 / 60
 Example: Mars Rover MRP

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Reward: +1 in s1 , +10 in s7 , 0 in all other states

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 11 / 60
 Return & Value Function

Definition of Horizon
Number of time steps in each episode
Can be infinite
Otherwise called finite Markov reward process
Definition of Return, Gt (for a MRP)
Discounted sum of rewards from time step t to horizon

Gt = rt + γrt+1 + γ 2 rt+2 + γ 3 rt+3 + · · ·

Definition of State Value Function, V (s) (for a MRP)

Expected return from starting in state s

V (s) = E[Gt |st = s] = E[rt + γrt+1 + γ 2 rt+2 + γ 3 rt+3 + · · · |st = s]

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 12 / 60
 Discount Factor

Mathematically convenient (avoid infinite returns and values)

Humans often act as if there’s a discount factor < 1
γ = 0: Only care about immediate reward
γ = 1: Future reward is as beneficial as immediate reward
If episode lengths are always finite, can use γ = 1

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 13 / 60
 Example: Mars Rover MRP

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Reward: +1 in s1 , +10 in s7 , 0 in all other states

Sample returns for sample 4-step episodes, γ = 1/2
1 1 1
s4 , s5 , s6 , s7 : 0 + 2 ×0+ 4 ×0+ 8 × 10 = 1.25

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 14 / 60
 Example: Mars Rover MRP

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Reward: +1 in s1 , +10 in s7 , 0 in all other states

Sample returns for sample 4-step episodes, γ = 1/2
1 1 1
s4 , s5 , s6 , s7 : 0 + 2 ×0+ 4 ×0+ 8 × 10 = 1.25
1 1 1
s4 , s4 , s5 , s4 : 0 + 2 ×0+ 4 ×0+ 8 ×0=0
1 1 1
s4 , s3 , s2 , s1 : 0 + 2 ×0+ 4 ×0+ 8 × 1 = 0.125

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 15 / 60
 Example: Mars Rover MRP

!" !# !$ !% !& !' !(

0.4 0.4 0.4 0.4 0.4 0.4

0.4 0.4 0.4 0.4 0.4 0.4

0.6 0.2 0.2 0.2 0.2 0.2 0.6

Reward: +1 in s1 , +10 in s7 , 0 in all other states

Value function: expected return from starting in state s
V (s) = E[Gt |st = s] = E[rt + γrt+1 + γ 2 rt+2 + γ 3 rt+3 + · · · |st = s]
Sample returns for sample 4-step episodes, γ = 1/2
1 1 1
s4 , s5 , s6 , s7 : 0 + 2 ×0+ 4 ×0+ 8 × 10 = 1.25
1 1 1
s4 , s4 , s5 , s4 : 0 + 2 ×0+ 4 ×0+ 8 ×0=0
1 1 1
s4 , s3 , s2 , s1 : 0 + 2 ×0+ 4 ×0+ 8 × 1 = 0.125
V = [1.53 0.37 0.13 0.22 0.85 3.59 15.31]
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 16 / 60
 Computing the Value of a Markov Reward Process

Could estimate by simulation

Generate a large number of episodes
Average returns
Concentration inequalities bound how quickly average concentrates to
expected value
Requires no assumption of Markov structure

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 17 / 60
 Computing the Value of a Markov Reward Process

Could estimate by simulation

Markov property yields additional structure
MRP value function satisfies
X
V (s) = R(s) + γ P(s 0 |s)V (s 0 )
s 0 ∈S
|{z}
Immediate reward | {z }
Discounted sum of future rewards

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 18 / 60
 Matrix Form of Bellman Equation for MRP

For finite state MRP, we can express V (s) using a matrix equation
 
  P(s1 |s1 ) · · ·
  P(sN |s1 )  
V (s1 ) R(s1 )  P(s1 |s2 ) · · · V (s1 )
 ..   ..  P(sN |s2 )   . 

 . = . +γ   .. 

.. .. ..
 . . . 
V (sN ) R(sN ) V (sN )
P(s1 |sN ) · · · P(sN |sN )
V = R + γPV

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 19 / 60
 Analytic Solution for Value of MRP

For finite state MRP, we can express V (s) using a matrix equation
 
  P(s1 |s1 ) · · ·
  P(sN |s1 )  
V (s1 ) R(s1 )  P(s1 |s2 ) · · · V (s1 )
 ..   ..  P(sN |s2 )   . 

 . = . +γ   .. 

.. .. ..
 . . . 
V (sN ) R(sN ) V (sN )
P(s1 |sN ) · · · P(sN |sN )
V = R + γPV
V − γPV = R
(I − γP)V = R
V = (I − γP)−1 R

Solving directly requires taking a matrix inverse ∼ O(N 3 )

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 20 / 60
 Iterative Algorithm for Computing Value of a MRP

Dynamic programming
Initialize V0 (s) = 0 for all s
For k = 1 until convergence
For all s in S
X
Vk (s) = R(s) + γ P(s 0 |s)Vk−1 (s 0 )
s 0 ∈S

Computational complexity: O(|S|2 ) for each iteration (|S| = N)

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 21 / 60
 Markov Decision Process (MDP)

Markov Decision Process is Markov Reward Process + actions

Definition of MDP
S is a (finite) set of Markov states s ∈ S
A is a (finite) set of actions a ∈ A
P is dynamics/transition model for each action, that specifies
P(st+1 = s 0 |st = s, at = a)
R is a reward function1

R(st = s, at = a) = E[rt |st = s, at = a]

Discount factor γ ∈ [0, 1]

MDP is a tuple: (S, A, P, R, γ)

1
Reward is sometimes defined as a function of the current state, or as a function of
the (state, action, next state) tuple. Most frequently in this class, we will assume reward
is a function of state and action
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 22 / 60
 Example: Mars Rover MDP

!" !# !$ !% !& !' !(

   
1 0 0 0 0 0 0 0 1 0 0 0 0 0
1 0 0 0 0 0 0 0 0 1 0 0 0 0
   
0 1 0 0 0 0 0 0 0 0 1 0 0 0
0 0
   
P(s |s, a1 ) = 
0 0 1 0 0 0 0 P(s |s, a2 ) = 

0 0 0 0 1 0 0
0 0 0 1 0 0 0 0 0 0 0 0 1 0
   
0 0 0 0 1 0 0  0 0 0 0 0 0 1
0 0 0 0 0 1 0 0 0 0 0 0 0 1

2 deterministic actions
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 23 / 60
 MDP Policies

Policy specifies what action to take in each state

Can be deterministic or stochastic
For generality, consider as a conditional distribution
Given a state, specifies a distribution over actions
Policy: π(a|s) = P(at = a|st = s)

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 24 / 60
 MDP + Policy

MDP + π(a|s) = Markov Reward Process

Precisely, it is the MRP (S, R π , P π , γ), where
X
R π (s) = π(a|s)R(s, a)
a∈A
X
0
π
P (s |s) = π(a|s)P(s 0 |s, a)
a∈A

Implies we can use same techniques to evaluate the value of a policy

for a MDP as we could to compute the value of a MRP, by defining a
MRP with R π and P π

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 25 / 60
 MDP Policy Evaluation, Iterative Algorithm

Initialize V0 (s) = 0 for all s

For k = 1 until convergence
For all s in S
X
Vkπ (s) = r (s, π(s)) + γ p(s 0 |s, π(s))Vk−1
π
(s 0 )
s 0 ∈S

This is a Bellman backup for a particular policy

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 26 / 60
 Policy Evaluation: Example & Check Your Understanding

!" !# !$ !% !& !' !(

Two actions
Reward: for all actions, +1 in state s1 , +10 in state s7 , 0 otherwise
Let π(s) = a1 ∀s. γ = 0.
What is the value of this policy?
Recall iterativ e
X
Vkπ (s) = r (s, π(s)) + γ p(s 0 |s, π(s))Vk−1
π
(s 0 )
s 0 ∈S

V π = [1 0 0 0 0 0 10]
Emma Brunskill (CS234 Reinforcement Learning)
Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 27 / 60
 Practice: MDP 1 Iteration of Policy Evaluation, Mars
Rover Example
Dynamics: p(s6 |s6 , a1 ) = 0.5, p(s7 |s6 , a1 ) = 0.5, . . .
Reward: for all actions, +1 in state s1 , +10 in state s7 , 0 otherwise
Let π(s) = a1 ∀s, assume Vk =[1 0 0 0 0 0 10] and k = 1, γ = 0.5
For all s in S
X
Vkπ (s) = r (s, π(s)) + γ p(s 0 |s, π(s))Vk−1
π
(s 0 )
s 0 ∈S

Vk+1 (s6 ) = r (s6 , a1 ) + γ ∗ 0.5 ∗ Vk (s6 ) + γ ∗ 0.5 ∗ Vk (s7 )

Vk+1 (s6 ) = 0 + 0.5 ∗ 0.5 ∗ 0 + .5 ∗ 0.5 ∗ 10

Vk+1 (s6 ) = 2.5

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 28 / 60
 MDP Control

Compute the optimal policy

π ∗ (s) = arg max V π (s)

π

There exists a unique optimal value function

Optimal policy for a MDP in an infinite horizon problem is
deterministic

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 29 / 60
 Check Your Understanding

!" !# !$ !% !& !' !(

7 discrete states (location of rover)

2 actions: Left or Right
How many deterministic policies are there?
27

Is the optimal policy for a MDP always unique?

No, there may be two actions that have the same optimal value
function

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 30 / 60
 MDP Control

Compute the optimal policy

π ∗ (s) = arg max V π (s)

π

There exists a unique optimal value function

Optimal policy for a MDP in an infinite horizon problem (agent acts
forever is
Deterministic
Stationary (does not depend on time step)
Unique? Not necessarily, may have state-actions with identical optimal
values

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 31 / 60
 Policy Search

One option is searching to compute best policy

Number of deterministic policies is |A||S|
Policy iteration is generally more efficient than enumeration

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 32 / 60
 MDP Policy Iteration (PI)

Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or kπi − πi−1 k1 > 0 (L1-norm, measures if the policy
changed for any state):
V πi ← MDP V function policy evaluation of πi
πi+1 ← Policy improvement
i =i +1

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 33 / 60
 New Definition: State-Action Value Q

State-action value of a policy

X
Q π (s, a) = R(s, a) + γ P(s 0 |s, a)V π (s 0 )
s 0 ∈S

Take action a, then follow the policy π

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 34 / 60
 Policy Improvement

Compute state-action value of a policy πi

For s in S and a in A:
X
Q πi (s, a) = R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
s 0 ∈S

Compute new policy πi+1 , for all s ∈ S

πi+1 (s) = arg max Q πi (s, a) ∀s ∈ S

a

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 35 / 60
 MDP Policy Iteration (PI)

Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or kπi − πi−1 k1 > 0 (L1-norm, measures if the policy
changed for any state):
V πi ← MDP V function policy evaluation of πi
πi+1 ← Policy improvement
i =i +1

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 36 / 60
 Delving Deeper Into Policy Improvement Step

X
Q πi (s, a) = R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 37 / 60
 Delving Deeper Into Policy Improvement Step

X
Q πi (s, a) = R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
s 0 ∈S
X
max Q πi (s, a) ≥ R(s, πi (s)) + γ P(s 0 |s, πi (s))V πi (s 0 ) = V πi (s)
a
s 0 ∈S

πi+1 (s) = arg max Q πi (s, a)

a

Suppose we take πi+1 (s) for one action, then follow πi forever
Our expected sum of rewards is at least as good as if we had always
followed πi
But new proposed policy is to always follow πi+1 ...

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 38 / 60
 Monotonic Improvement in Policy

Definition
V π1 ≥ V π2 : V π1 (s) ≥ V π2 (s), ∀s ∈ S
Proposition: V πi+1 ≥ V πi with strict inequality if πi is suboptimal,
where πi+1 is the new policy we get from policy improvement on πi

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 39 / 60
 Proof: Monotonic Improvement in Policy

V πi (s) ≤ max Q πi (s, a)

a
X
= max R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
a
s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 40 / 60
 Proof: Monotonic Improvement in Policy

V πi (s) ≤ max Q πi (s, a)

a
X
= max R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
a
s 0 ∈S
X
=R(s, πi+1 (s)) + γ P(s 0 |s, πi+1 (s))V πi (s 0 ) //by the definition of πi+1
s 0 ∈S
X  
≤R(s, πi+1 (s)) + γ P(s 0 |s, πi+1 (s)) max
0
Q πi 0 0
(s , a )
a
s 0 ∈S
X
=R(s, πi+1 (s)) + γ P(s 0 |s, πi+1 (s))
s 0 ∈S
!
X
R(s 0 , πi+1 (s 0 )) + γ P(s 00 |s 0 , πi+1 (s 0 ))V πi (s 00 )
s 00 ∈S
..
.
=V πi+1 (s)

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 41 / 60
 Policy Iteration (PI): Check Your Understanding
Note: all the below is for finite state-action spaces
Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or kπi − πi−1 k1 > 0 (L1-norm, measures if the policy
changed for any state):
V πi ← MDP V function policy evaluation of πi
πi+1 ← Policy improvement
i =i +1
If policy doesn’t change, can it ever change again?
No

Is there a maximum number of iterations of policy iteration?

|A||S| since that is the maximum number of policies, and as the policy
improvement step is monotonically improving, each policy can only
appear in one round of policy iteration unless it is an optimal policy.

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 42 / 60
 Policy Iteration (PI): Check Your Understanding

Suppose for all s ∈ S, πi+1 (s) = πi (s)

Then for all s ∈ S, Q πi+1 (s, a) = Q πi (s, a)
Recall policy improvement step
X
Q πi (s, a) = R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
s 0 ∈S

πi+1 (s) = arg max Q πi (s, a)

a
πi+1
πi+2 (s) = arg max Q (s, a) = arg max Q πi (s, a)
a a

Therefore policy cannot ever change again

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 43 / 60
 MDP: Computing Optimal Policy and Optimal Value

Policy iteration computes optimal value and policy

Value iteration is another technique
Idea: Maintain optimal value of starting in a state s if have a finite
number of steps k left in the episode
Iterate to consider longer and longer episodes

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 44 / 60
 Bellman Equation and Bellman Backup Operators

Value function of a policy must satisfy the Bellman equation

X
V π (s) = R π (s) + γ P π (s 0 |s)V π (s 0 )
s 0 ∈S

Bellman backup operator

Applied to a value function
Returns a new value function
Improves the value if possible
X
BV (s) = max R(s, a) + γ p(s 0 |s, a)V (s 0 )
a
s 0 ∈S

BV yields a value function over all states s

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 45 / 60
 Value Iteration (VI)

Set k = 1
Initialize V0 (s) = 0 for all states s
Loop until [finite horizon, convergence]:
For each state s
X
Vk+1 (s) = max R(s, a) + γ P(s 0 |s, a)Vk (s 0 )
a
s 0 ∈S

View as Bellman backup on value function

Vk+1 = BVk
X
πk+1 (s) = arg max R(s, a) + γ P(s 0 |s, a)Vk (s 0 )
a
s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 46 / 60
 Policy Iteration as Bellman Operations

Bellman backup operator B π for a particular policy is defined as

X
B π V (s) = R π (s) + γ P π (s 0 |s)V (s)
s 0 ∈S

Policy evaluation amounts to computing the fixed point of B π

To do policy evaluation, repeatedly apply operator until V stops
changing
V π = BπBπ · · · BπV

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 47 / 60
 Policy Iteration as Bellman Operations

Bellman backup operator B π for a particular policy is defined as

X
B π V (s) = R π (s) + γ P π (s 0 |s)V (s)
s 0 ∈S

To do policy improvement
X
πk+1 (s) = arg max R(s, a) + γ P(s 0 |s, a)V πk (s 0 )
a
s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 48 / 60
 Going Back to Value Iteration (VI)

Set k = 1
Initialize V0 (s) = 0 for all states s
Loop until [finite horizon, convergence]:
For each state s
X
Vk+1 (s) = max R(s, a) + γ P(s 0 |s, a)Vk (s 0 )
a
s 0 ∈S

Equivalently, in Bellman backup notation

Vk+1 = BVk

To extract optimal policy if can act for k + 1 more steps,

X
π(s) = arg max R(s, a) + γ P(s 0 |s, a)Vk+1 (s 0 )
a
s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 49 / 60
 Contraction Operator

Let O be an operator,and |x| denote (any) norm of x

If |OV − OV 0 | ≤ |V − V 0 |, then O is a contraction operator

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 50 / 60
 Will Value Iteration Converge?

Yes, if discount factor γ < 1, or end up in a terminal state with

probability 1
Bellman backup is a contraction if discount factor, γ < 1
If apply it to two different value functions, distance between value
functions shrinks after applying Bellman equation to each

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 51 / 60
 Proof: Bellman Backup is a Contraction on V for γ < 1

Let kV − V 0 k = maxs |V (s) − V 0 (s)| be the infinity norm

   

0 0  0 0 0 0 
X X
kBVk − BVj k = max R(s, a) + γ P(s |s, a)V k (s ) − max  R(s, a ) + γ P(s |s, a )V j (s )
a
a 0
s 0 ∈S s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 52 / 60
 Proof: Bellman Backup is a Contraction on V for γ < 1

Let kV − V 0 k = maxs |V (s) − V 0 (s)| be the infinity norm

   

0 0  0 0 0 0 
X X
kBVk − BVj k = max R(s, a) + γ P(s |s, a)V k (s ) − max  R(s, a ) + γ P(s |s, a )V j (s )
a
a 0
s 0 ∈S s 0 ∈S

 

0 0 0 0
X X
≤ max
R(s, a) + γ P(s |s, a)Vk (s ) − R(s, a) − γ P(s |s, a)Vj (s )
a

s 0 ∈S s 0 ∈S



0 0 0
X
= max γ P(s |s, a)(Vk (s ) − Vj (s ))
a

s 0 ∈S



0
X
≤ max γ P(s |s, a)kVk − Vj k)
a

s 0 ∈S



0
X
= γkVk − Vj k max a
P(s |s, a))
s 0 ∈S

= γkVk − Vj k

Note: Even if all inequalities are equalities, this is still a contraction if γ < 1

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 53 / 60
 Check Your Understanding

Prove value iteration converges to a unique solution for discrete state

and action spaces with γ < 1
Does the initialization of values in value iteration impact anything?

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 54 / 60
 Value Iteration for Finite Horizon H

Vk = optimal value if making k more decisions

πk = optimal policy if making k more decisions
Initialize V0 (s) = 0 for all states s
For k = 1 : H
For each state s
X
Vk+1 (s) = max R(s, a) + γ P(s 0 |s, a)Vk (s 0 )
a
s 0 ∈S
X
πk+1 (s) = arg max R(s, a) + γ P(s 0 |s, a)Vk (s 0 )
a
s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 55 / 60
 Is optimal policy stationary (independent of time step) in
finite horizon tasks? ? In general, no

Set k = 1
Initialize V0 (s) = 0 for all states s
Loop until k == H:
For each state s
X
Vk+1 (s) = max R(s, a) + γ P(s 0 |s, a)Vk (s 0 )
a
s 0 ∈S
X
πk+1 (s) = arg max R(s, a) + γ P(s 0 |s, a)Vk (s 0 )
a
s 0 ∈S

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 56 / 60
 Value vs Policy Iteration

Value iteration:
Compute optimal value for horizon = k
Note this can be used to compute optimal policy if horizon = k
Increment k
Policy iteration
Compute infinite horizon value of a policy
Use to select another (better) policy
Closely related to a very popular method in RL: policy gradient

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 57 / 60
 What You Should Know

Define MP, MRP, MDP, Bellman operator, contraction, model,

Q-value, policy
Be able to implement
Value Iteration
Policy Iteration
Give pros and cons of different policy evaluation approaches
Be able to prove contraction properties
Limitations of presented approaches and Markov assumptions
Which policy evaluation methods require the Markov assumption?

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 58 / 60
 Policy Improvement

Compute state-action value of a policy πi

X
Q πi (s, a) = R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
s 0 ∈S

Note
X
max Q πi (s, a) = max R(s, a) + γ P(s 0 |s, a)V π (s 0 )
a a
s 0 ∈S
X
≥ R(s, πi (s)) + γ P(s 0 |s, πi (s))V π (s 0 )
s 0 ∈S
= V πi (s)

Define new policy, for all s ∈ S

πi+1 (s) = arg max Q πi (s, a) ∀s ∈ S

a

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 59 / 60
 Policy Iteration (PI)

Set i = 0
Initialize π0 (s) randomly for all states s
While i == 0 or kπi − πi−1 k1 > 0 (L1-norm):
Policy evaluation of πi
i =i +1
Policy improvement:
X
Q πi (s, a) = R(s, a) + γ P(s 0 |s, a)V πi (s 0 )
s 0 ∈S

πi+1 (s) = arg max Q πi (s, a)

a

Emma Brunskill (CS234 Reinforcement Learning)

Lecture 2: Making Sequences of Good Decisions Given a ModelWinter
of the2019
World 60 / 60