Macroeconomics I

Josep Pijoan-Mas

CEMFI, September-December 2024

 Part I

Dynamic Optimization in Continuous Time

Outline

1 Introduction

2 Calculus of Variations

3 Optimal Control Theory: Hamiltonians

4 Dynamic Programming in Continuous Time: HJB Equations

Outline

1 Introduction

2 Calculus of Variations

3 Optimal Control Theory: Hamiltonians

4 Dynamic Programming in Continuous Time: HJB Equations

 Introduction Calculus of Variations Optimal Control Dynamic Programming

Introduction

● In macroeconomics we often need to solve dynamic optimization problems

– Choices at time t affect the state at time t + ∆t

● Typically, we want to maximize PDV of a stream of payoffs subject to some constraints

(HHs choose consumption s.t. budget constraint; firms choose prices s.t. to demand function; etc)

● There are many possible variations of this type of problems

– The problem may involve a finite or an infinite horizon
– The future may be deterministic or stochastic (shocks)
– Time may be discrete (t ∈ Z+ ) or continuous (t ∈ R+ )

● Most of the models we will see in Macro I will be set up in continuous time
→ This
This week
week we’ll study the mathematical tools to solve these types of problems
week:

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 1/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming

The Canonical Problem (1/3)

● The canonical problem in continuous time looks like this:
T
max W (x, y) ≡ ∫ f (t, xt , yt ) dt (1a)
x,y 0
subject to ẋt = g(t, xt , yt ), (1b)
yt ∈ Y, xt ∈ X , ∀t ∈ [0, T ] (1c)
x0 = x (1d)
● Where:
1 xt ∈ X is the state variable (variable pre-determined at time t)
2 yt ∈ Y is the control variable (variable chosen at time t)
3 f ∶ R+ × X × Y → R is the instantaneous payoff function (may depend on time t)
4 g ∶ R+ × X × Y → X is the law of motion of the state variable
5 x ∈ X is the initial condition for the state (which we take as given)
6 x ∶ [0, T ] → X and y ∶ [0, T ] → Y are the time paths of the state and control variables
7 W is a functional descibing the objective function
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 2/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming

The Canonical Problem (2/3)

● Definition A pair (x, y) is called an admissible pair if (1b)-(1d) hold
Definition
Definition:
● Assumption W (x, y) < ∞ for all admissible pairs (x, y)
Assumption:
Assumption

● Then, the solution of a dynamic optimization problem consists of picking, among all
the admissible pairs, the one that maximizes the objective function

● A note on terminal conditions:

– In problem (1a)-(1d) terminal time T is given but the terminal state xT is free to choose
– In some applications, both a terminal time T and a terminal state xT = x ∈ X are given
– In others, xT is given but T is not, so the planning horizon T is chosen optimally
– Finally, it may be that both T and xT are free (but related by a constraint)

● An example: neo-classical growth model

– The state is the capital stock, xt ≡ kt , and the control is the consumption flow, yt ≡ ct
(and X ≡ R+ and Y ≡ R++ )

– The payoff fctn is the period utility w/ exponential discounting: f (t, yt , xt ) ≡ e−ρt u(ct )
– The law of motion for the state is the resource constraint: k˙t = g(t, xt , yt ) ≡ ktα − δt kt − ct
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 3/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming

The Canonical Problem (3/3)

● How to solve this problem?

● Two challenges relative to discrete-time formulation:

a) We have to choose two functions:

y ∶ [0, T ] → Y and x ∶ [0, T ] → X

(rather than two sets of points in discrete time)

b) The constraint is a differential equation:

ẋt = g(t, xt , yt )

(rather than a set of (in)equalities in discrete time)

● It may seem like continuous time is making our lives harder

● Yet, the methods are simple, easy to implement, and deliver elegant solutions

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 4/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming

Methods

1 Calculus of Variations (or “variational arguments”)

– This is the oldest approach (dating back to the 18th century)
– Very popular in physics (e.g. Lagrangian mechanics)
– It uses perturbation arguments

2 Optimal Control
– This method can do everything economists need from calculus of variations, and more
– This will be the most useful method in the deterministic models we will see in the course
– It uses Hamiltonians

3 Continuous-Time Dynamic Programming

– It extends optimal control by exploiting that the value function can be made recursive
– Dynamic programming methods are most useful for stochastic models
– It uses Hamilton-Jacobi-Bellman (HJB) equations

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 5/48

Outline

1 Introduction

2 Calculus of Variations

3 Optimal Control Theory: Hamiltonians

4 Dynamic Programming in Continuous Time: HJB Equations

 Introduction Calculus of Variations Optimal Control Dynamic Programming

Calculus of Variations
● It was developed for a simpler version of problem (1a)-(1d):
T
max W (x) ≡ ∫ f (t, xt , ẋt ) dt (2a)
x 0
subject to xt ∈ X , ∀t ∈ [0, T ] (2b)
x0 = x (2c)

– There is no concept of control variable

– Problem (2a)-(2c) is a particular case of the more general Problem (1a)-(1d)
For the case in which we can invert the law of motion ẋt = g(t, xt , yt ) into yt = g̃(t, xt , ẋt )

● Method based on variational arguments

(developed by Euler and Lagrange in the 1750s)

→ It can only characerize interior solutions

→ Assumes there exists a solution x∗ , then it characterizes the properties that are absent in its
neighboring paths x (necessary conditions for an optimum)
⇒ Provides local optima (second order conditions ensure they are global)
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 6/48
Outline

1 Introduction

2 Calculus of Variations

3 Optimal Control Theory: Hamiltonians

4 Dynamic Programming in Continuous Time: HJB Equations

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

The Variational Argument

Key idea
● Let’s use the variational argument to derive the necesary conditions for an optimum
in the broader Problem (1a)-(1d)
● Assume f and g are continuously differentiable
● Suppose there exists a continuous function y ∗ ∶ [0, T ] → Y such that
a) y ∗ lies everywhere in the interior of Y , that is, yt∗ ∈ int(Y) ∀t ∈ [0, T ]
b) The corresponding state function x∗ ∶ [0, T ] → X
ẋ∗t = g(t, x∗t , yt∗ ), ∀t ∈ [0, T ], with x∗0 = x

also satisfies x∗t ∈ int(X ) ∀t ∈ [0, T ]

c) (x∗ , y ∗ ) form an admissible pair that is a solution to problem (1a)-(1d); that is:
W (x∗ , y ∗ ) ≥ W (x, y), ∀ admissible pairs (x, y)
∗ ∗
→ It is important that (x , y ) never hits the boundaries and does not have discontinuities
(Not a strong assumption for us as none of the models in this course will have corner solutions)

idea Since y is a solution, there should be no gain from small deviations

∗
● Key
Key idea
idea:
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 7/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

The Variational Argument

Admissible deviations

● To construct a tractable small “admissible deviation”,

1 We apply a variation to the entire function:

- Let η ∶ [0, T ] → Y be an arbitrary continuous function, and ε ∈ R some real number
- A variation from the optimal plan can be constructed as: yt (ε) ≡ yt∗ + εηt
(For any given η , we can construct many variations simply by varying ε)

2 We make sure that the variation is admissible:

- For our chosen ε, there may exist a t ∈ [0, T ] such that yt (ε) ∉ Y
- However, since yt∗ ∈ int(Y) and a continuous function over a compact set [0, T ] is bounded and
uniformly continuous, there must exist a εη > 0 such that:

yt (ε) = yt∗ + εηt ∈ int(Y), ∀t ∈ [0, T ], ∀ε ∈ [−εη , εη ]

● In this case, y(ε) constitutes a admissible deviation or feasible variation of y ∗

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 8/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

The Variational Argument

Optimality condition
● Next, define x(ε) as the path of the state variable corresponding to y(ε):

ẋt (ε) = g (t, xt (ε), yt (ε)), ∀t ∈ [0, T ], with x0 (ε) = x

● Moreover, define the objective function along the admissible variation as:
T
W (ε) ≡ W (x(ε), y(ε)) = ∫ f (t, xt (ε), yt (ε)) dt (3)
0
→ We have transformed the functional W(x, y) into a function W(ε)

● Since (x∗ , y ∗ ) is optimal and (x(ε), y(ε)) is feasible for all ε ∈ [−εη , εη ], then:

W(ε) ≤ W(0), ∀ε ∈ [−εη , εη ]

● Local optimality requires that W(0) does not change when changing ε slightly:
∂
W(ε)∣ = 0, ∀η (4)
∂ε ε=0
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 9/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

The Variational Argument

Reshaping the return function

● For any continuously differentiable function λ ∶ [0, T ] → R, we can write (3) as:

T
W(ε) = ∫ [f (t, xt (ε), yt (ε)) + λt [g(t, xt (ε), yt (ε)) − ẋt (ε)]] dt (5)
0

because g(t, xt (ε), yt (ε)) − ẋt (ε) = 0 by construction.

– We have manipulated W slightly, to make it resemble a Lagrangian

– When chosen suitably, λ will be a costate variable (akin to a Lagrange multiplier)

● The goal is to obtain the conditions that ensure that the local optimality (4) holds
→ They are the necessary conditions for optimality

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 10/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Necessary Conditions for Optimality (1/3)

T
● First, consider the integral ∫0 λt ẋt (ε) dt. Integrating by parts:
T T
∫ λt ẋt (ε) dt = λT xT (ε) − λ0 x − ∫ xt (ε)λ̇t dt
0 0

● Substituting back into (5):

T
W(ε) = ∫ [f (t, xt (ε), yt (ε)) + λt g(t, xt (ε), yt (ε)) + λ̇t xt (ε)] dt − λT xT (ε) + λ0 x
0

● Second, differentiate with respect to ε (use the Leibniz’s and chain rules):
∂W(ε) T ∂ ∂ ∂xt (ε)
= ∫ [ f (t, xt (ε), yt (ε)) + λt g(t, xt (ε), yt (ε)) + λ̇t ] dt
∂ε 0 ∂x ∂x ∂ε
T ∂ ∂ ∂yt (ε)
+ ∫ [ f (t, xt (ε), yt (ε)) + λt g(t, xt (ε), yt (ε)) ] dt
0 ∂y ∂y ∂ε
´¹¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹¶
ηt
∂xT (ε)
− λT
∂ε
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 11/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Necessary Conditions for Optimality (2/3)

● Third, evaluate this derivative at ε = 0

(and noting xt (0) = x∗ ∗
t and yt (0) = yt by construction)

∂W(ε) T ∂ ∂ ∂xt (0)

∣ = [ f (t, x∗t , yt∗ ) + λt g(t, x∗t , yt∗ ) + λ̇t ] dt
∂ε ε=0 ∫0 ∂x ∂x ∂ε
T ∂ ∂
+ ∫ [ f (t, x∗t , yt∗ ) + λt g(t, x∗t , yt∗ )]ηt dt
0 ∂y ∂y
∂xT (0)
− λT (6)
∂ε

● Finally, impose the local optimality condition:

∂W(ε)
∣ = 0, ∀η
∂ε ε=0

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 12/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Necessary Conditions for Optimality (3/3)

● Since η is arbitrary, all three additive terms in (6) must be zero.
(Note that η appears explicitly in the 2nd term and implicitly in the the other two, as ∂xt (0)/∂ε depends on η )

– Setting the second term to zero requires

∂ ∂
f (t, x∗t , yt∗ ) + λt g(t, x∗t , yt∗ ) = 0 (7a)
∂y ∂y

– Setting the first term to zero requires

∂ ∂
λ̇t = − ( f (t, x∗t , yt∗ ) + λt g(t, x∗t , yt∗ )) (7b)
∂x ∂x

– And the third term requires the boundary condition

λT = 0 (7c)

(For problems with given terminal condition xT = x, λT = 0 is not needed as ∂xT (0)/∂ε = 0)

● In addition, we have the lof of motion of the state, ẋ∗t = g(t, x∗t , yt∗ )
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 13/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Summing up

● We have characterized the necessary conditions for (x∗ , y ∗ ) to be an interior

continuous solution of the canonical Problem (1a)-(1d)

● The necessary conditions require the existence of a continuously differentiable

function λ ∶ [0, T ] → R, the so-called costate function, that satisfies (7a)-(7c)

– Condition (7a) imposes that yt is chosen such that the marginal increase in the return
function equals the marginal loss due to lowering the state

– Condition (7b) provides the law of motion for the shadow value of the state variable

– Condition (7c), λT = 0 is the second boundary condition

→ It requires that, at the end of the planning horizon, there is no value in having more or less x

● What’s missing? → Need to know whether these conditions are also sufficient

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 14/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

The Hamiltonian

● The necessary and sufficient conditions for Problem (1a)-(1d) are encapsulated in
several theorems

● Let’s assume an infinite horizon, T = ∞ (as in all the models we’ll see in this course)
– This requires to impose a lower bound on the (free) terminal condition: lim bt xt ≥ x
t→∞
(for some b ∶ R+ → R with lim bt < ∞, and x < ∞ given)
t→∞
(the case bt = 1 ∀t is perfectly feasible)

● By analogy with the Lagrangian, we construct the so-called Hamiltonian:

H(t, xt , yt , λt ) ≡ f (t, xt , yt ) + λt g(t, xt , yt )

● Since f and g are continuously differentiable, H is continuously differentiable too

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 15/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Pontryagin’s Maximum Principle (1956)

Theorem 1: Pontryagin’s Maximum Principle

● Suppose problem (1a)-(1d) has an interior continuous solution (x∗ , y ∗ )

● Then, there exists a continuously differentiable function λ ∶ R+ → R such that

1 For all t ∈ R+ the following necessary conditions hold:

∂
H(t, x∗t , yt∗ , λt ) = 0 (8a)
∂y
∂
H(t, x∗t , yt∗ , λt ) = −λ̇t (8b)
∂x
∂
H(t, x∗t , yt∗ , λt ) = ẋt (8c)
∂λ
2 The Hamiltonian satisfies: H(t, x∗t , yt∗ , λt ) ≥ H(t, x∗t , yt , λt ), ∀yt ∈ Y, ∀t ∈ R+

Note: this is essentially re-stating the necessary conditions from the variational argument

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 16/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Pontryagin’s Maximum Principle (1956)

Comments
● The solution involves finding a function λ ∶ R+ → R that satisfies these conditions
– The costate λt is informative about the marginal value of relaxing the constraint at time t

● The optimality conditions state:

(8a) Hy = 0, says that there can be no change in value for a small change in the control variable
(8b) Hx = −λ̇t , says that the change in the shadow value equals the marginal value of changing
the state variable
(8c) Hλ = ẋt , simply states that the constraint ẋt = g(t, x∗t , yt∗ ) holds

● They form a system of two differential equations, we need two boundary conditions
– The initial condition x0 = x
– When a terminal condition is not given, we need a Transversality Condition (TVC)
- In many problems limt→∞ λt xt = 0 properly generalizes the case λT = 0
- But more generally, the TVC states that the Hamiltonian evaluated at (x∗ , y ∗ ) satisfies:
lim H(t, x∗t , yt∗ , λt ) = 0
t→∞

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 17/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Sufficiency Conditions
Mangasarian (1966)

Theorem 2: Mangasarian’s Sufficiency Conditions

● Assume:
– Problem (1a)-(1d) has an interior solution, (x∗t , yt∗ ) ∈ int(X × Y) ∀t
– (x∗ , y ∗ ) satisfies Pontryagin’s necessary conditions, (8a)-(8c)

● Then, (x∗ , y ∗ ) attains the global maximum if:

1 X × Y is a convex set
2 H(t, xt , yt , λt ) is jointly concave in (xt , yt ) ∈ X × Y , for all t ∈ R+

● Moreover, (x∗ , y ∗ ) is the unique solution if H(t, xt , yt , λt ) is strictly concave

in (xt , yt ) ∈ X × Y , for all t ∈ R+

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 18/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Sufficiency Conditions
Arrow (1968)
● Given the costate function λ ∶ R+ → R, define the maximized Hamiltonian as:
M(t, xt , λt ) ≡ max H(t, xt , yt , λt )
yt ∈Y

Theorem 3: Arrow’s Sufficiency Conditions

● Assume:
– Problem (1a)-(1d) has an interior solution, (x∗t , yt∗ ) ∈ int(X × Y) ∀t
– (x∗ , y ∗ ) satisfies Pontryagin’s necessary conditions, (8a)-(8c)

● Then, (x∗ , y ∗ ) attains the global maximum if:

1 X is a convex set
2 M(t, xt , λt ) is concave in xt ∈ X , for all t ∈ R+

● Moreover, (x∗ , y ∗ ) is the unique solution if M(t, xt , λt ) is strictly concave in

xt ∈ X , ∀t ∈ R+

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 19/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Sufficiency Conditions
Comments
● Arrow’s (1968) theorem is more general than Mangasarian’s (1966)
(Mangasarian ⇒ Arrow)

– This is because if a function H(t, x, y, λ) is jointly (strictly) concave in (x, y), then
M(t, x, λ) ≡ maxy H(t, x, y, λ) is (strictly) concave in x

● Which result we use in practice depends on the application:

– Because Arrow’s condition is more general, one may typically prefer it
(Mangasarian’s condition may fail while Arrow’s may still hold)

– But in typical econ problems, it is harder to verify Arrow’s conditions than Mangasarian’s
- It is easy to check for concavity in H, not so much in M

– For example, as H = f + λg :
- Mangasarian’s sufficiency follows if both f and g are concave in (xt , yt ), provided λt ≥ 0
- The condition that λt ≥ 0 everywhere, in turn, is typically guaranteed
∂ ∂
(For instance, if f (t, x∗t , yt∗ , λt ) ≥ 0 and g(t, x∗t , yt∗ , λt ) ≤ 0, then λt ≥ 0 )
∂y ∂y
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 20/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Extensions

● Multidimensional problems

– We may have X ⊆ RKx and Y ⊆ RKy

- We need Kx laws of motion for the state

– In this case, the vector of co-state variables is λ ∶ [0, T ] → RKx

– The Maximum Principle and Mangasarian and Arrow sufficiency conditions also apply
- We have Ky (8a) conditions and Kx (8b)-(8c) conditions

● Discontinuous controls
– The Maximum Principle and Mangasarian and Arrow sufficiency conditions only apply
whenever y ∗ is a continuous function of time
– Pontryagin provided generalizations where the control is allowed to have discontinuities
– Discontinuous solutions are rare cases in macroeconomics w/ RA, so we will ignore this

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 21/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Discounting

● So far, we have not talked about a key object in economics: time discounting
– In previous slides, discounting entered implicitly in the payoff function f (t, xt , yt )
– t indicates how the return function changes over time for given values of xt and yt

● This course, we will assume rational and time-consistent behavior

(more on this later)

● One type of discounting that elicits this type of behavior is exponential discounting
– This is the analogue of proportional discounting in discrete time
– We will have a “discount rate”, ρ ∈ (0, ∞), instead of a “discount factor”, β ∈ (0, 1)

● Technical note:
Technical note:
– Exponential discounting is necessary, but not sufficient, to get time consistency
– For that, we also need a time-autonomous constraint, i.e. g(xt , yt ) and not g(t, xt , yt )

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 22/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Exponential Discounting: Where Does it Come From?

● We want to value e1 at time T , with continuous discount rate ρ

● Let’s partition time [0, T ] into T /∆t equally spaced discrete subintervals of length ∆t
– For each sub-period of length ∆t, the discount factor is ρ∆t

● The value today of $1 at time T when the period length is ∆t:

v(T ∣∆t) ≡ (1 − ρ∆t)T /∆t

● Taking the “continuous-time limit”, i.e. letting ∆t → 0, we can show:

v(T ) = lim v(T ∣∆t) = lim (1 − ρ∆t)T /∆t = exp(−ρT )

∆t→0 ∆t→0

Proof (last step w/ L’Hôpital’s rule):

log(1 − ρ∆t)
lim (1 − ρ∆t)T /∆t = lim [exp (log (1 − ρ∆t)T /∆t )] = exp ( lim ) = exp(−ρT )
∆t→0 ∆t→0 ∆t→0 ∆t/T

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 23/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Exponential Discounting: The Canonical Problem

● With exponential discounting, and from a time-0 perspective, we can write the payoff
function as follows:
f (t, xt , yt ) ≡ e−ρt f (xt , yt )

● Hence, the canonical problem w/ exponential discounting and infinite horizon is:
∞
max W (x, y) ≡ ∫ e−ρt f (xt , yt ) dt (9a)
x,y 0
subject to ẋt = g(t, xt , yt ) (9b)
yt ∈ Y, xt ∈ X , ∀t ∈ R+ (9c)
x0 = x given, and lim bt xt ≥ x (9d)
t→∞

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 24/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Exponential Discounting: The Current-Value Hamiltonian

● Now, the Hamiltonian is:

H(t, xt , yt , λt ) = e−ρt f (xt , yt ) + λt g(t, xt , yt )

= e−ρt [ f (xt , yt ) + µt g(t, xt , yt ) ] where µt ≡ eρt λt
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
current-value Hamiltonian

● The term in square brackets is called the current-value Hamiltonian (denoted w/ hat):

̂ xt , yt , µt ) ≡ f (xt , yt ) + µt g(t, xt , yt )
H(t,

̂ is a time-t Hamiltonian (hence its name)

– While H was a time-0 Hamiltonian, H
– Likewise, µt is the time-t shadow value of xt
(while λt was the time-0 shadow value of xt )

● We can re-work our theorems so they can be used for the current-value Hamiltonian

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 25/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Exponential Discounting: The Maximum Principle (1/2)

Theorem 4: Pontryagin’s Maximum Principle with Exponential Discounting (1/2)

● Suppose Problem (9a)-(9d) has an interior solution, (x∗ , y ∗ )

● ̂ satisfies:
Then, there exists a continuously differentiable function µ ∶ R+ → R s.t. H

∂ ̂
H(t, x∗t , yt∗ , µt ) = 0 (10a)
∂y
∂ ̂
H(t, x∗t , yt∗ , µt ) = ρµt − µ̇t (10b)
∂x
∂ ̂
H(t, x∗t , yt∗ , µt ) = ẋt (10c)
∂µ
̂ x∗t , yt∗ , µt ) = 0
lim e−ρt H(t, (10d)
t→∞

[continues on the next slide]

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 26/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Exponential Discounting: The Maximum Principle (2/2)

Theorem 4: Pontryagin’s Maximum Principle with Exponential Discounting (2/2)

[continued]

● Moreover, suppose:
1 f (xt , yt ) is weakly monotone in (xt , yt ), and g(t, xt , yt ) is weakly monotone in
(t, xt , yt ).
2 ∃m > 0 s.t. ∣ ∂y
∂
g(t, xt , yt )∣ ≥ m, for all admissible pairs (xt , yt ).

3 ∃M < ∞ s.t. ∣ ∂y
∂
f (xt , yt )∣ ≤ M , ∀(x, y).

∗ ẋt
4 Either lim xt < ∞ or lim ( ) < ∞.
t→∞ t→∞ x∗t

● Then, the transversality condition in equation (10d) can be strengthened to:

lim e−ρt µt x∗t = 0 (10d’)
t→∞

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 27/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

Exponential Discounting: Comments

● The Maximum Principle with exponential discounting is similar to the general one

● It provides necessary conditions for optimality:

– Two differential equations, (10b)-(10c), and two boundary conditions, x0 = x and (10d’)
– The terminal condition (10d’) reads: limt→∞ e−ρt µt x∗t = 0
It states that the state must have no value (in present-discounted terms) in the limit of time
– Condition (10d’) requires some extra assumptions relative to (10d)
(but these are typically satisfied)

● Conditions (10a)-(10d’) are necessary ... but are they sufficient?

– One can apply Mangasarian’s/Arrow’s Sufficiency Conditions to exponential discounting case

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 28/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming The Variational Argument General Theorems Exponential Discounting A Recipe

A recipe
Suppose Problem (9a)-(9d) with exponential discounting and infinite time horizon has an interior
solution (x∗ , y ∗ ), and conditions (1)-(4) of Theorem 4 hold. We characterize the solution as follows:

Present-value Hamiltonian
Present-value Hamiltonian Current-value Hamiltonian
Current-value Hamiltonian

H(t, xt , yt , λt ) ≡ eρt f (xt , yt ) + λt g(t, xt , yt ) ̂ xt , yt , µt ) ≡ f (xt , yt ) + µt g(t, xt , yt )

H(t,

∂ ∂ ̂
H(t, x∗t , yt∗ , λt ) = 0 H(t, x∗t , yt∗ , µt ) = 0
∂y ∂y
∂ ∂ ̂
H(t, x∗t , yt∗ , λt ) = − λ̇t H(t, x∗t , yt∗ , µt ) = ρµt − µ̇t
∂x ∂x
∂ ∂ ̂
H(t, x∗t , yt∗ , λt ) = ẋt H(t, x∗t , yt∗ , µt ) = ẋt
∂λ ∂µ
lim λt x∗t = 0 lim e−ρt µt x∗t = 0
t→∞ t→∞

The two cases are equivalent given the definition µt ≡ eρt λt

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 29/48

Outline

1 Introduction

2 Calculus of Variations

3 Optimal Control Theory: Hamiltonians

4 Dynamic Programming in Continuous Time: HJB Equations

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Why Use Dynamic Programming?

● In discrete time, dynamic programming transforms a dynamic problem of multiple

periods into a sequence of two-period problems
– The two-period problems are identical, so we only solve one of them
– This is achieved by exploiting the recursion of dynamic problems

● In continuous time, DP builds on the insights of discrete time DP

● In deterministic models, the tools provided by optimal control theory are great
(by “deterministic models” we mean models without uncertainty: no shocks)

● However, the recursive formulation is very useful when dealing with stochastic models
(by “stochastic models” we mean models with uncertainty: shocks)

– We will see some examples in the topic of Endogenous Growth

– The necessary conditions are expressed recursively in terms of the so-called
Hamilton-Jacobi-Bellman (HJB) equation

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 30/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

The Value Function

● Let us consider the canonical problem with infinite horizon,
∞
V0 (x0 ) ≡ max ∫ f (t, xt , yt ) dt (13a)
x,y 0
subject to ẋt = g(t, xt , yt ), (13b)
yt ∈ Y, xt ∈ X , ∀t ∈ R+ (13c)
x0 given and lim bt xt ≥ x (13d)
t→∞

● Here, V0 (x0 ) is the value function

→ It states the optimal value of the problem starting at time t = 0 w/ initial state x0
→ Given an admissible pair (x∗ , y ∗ ) that solves problem (13a)-(13d) we can write
∞
V0 (x0 ) = ∫ f (t, x∗t , yt∗ )dt
0

● More generally we can define Vτ (xτ ) as the optimal value at time 0 of the problem
starting at time τ with some state xτ
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 31/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Bellman’s Principle of Optimality

Theorem 5: Bellman’s Principle of Optimality

● Suppose problem (13a)-(13d) has an interior solution such that (x∗ , y ∗ )

reaches the maximum value V0 (x0 )

● Then, we have:
t
V0 (x0 ) = ∫ f (s, x∗s , ys∗ ) ds + Vt (x∗t ) , ∀t ≥ 0 (14)
0 ´¹¹ ¹ ¹ ¸¹¹ ¹ ¹ ¶
continuation value

Interpretation:

● The optimal sequence (x∗ , y ∗ ) has the property that, at any time t ≥ 0, the
subsequence (x∗s , ys∗ ∶ s ≥ t) is also the solution of the time-0 planner to the
optimization problem starting at time t with initial state x∗t

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 32/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Bellman’s Principle of Optimality: the logic

● Consider the following property of real-valued functions:

– Let f ∶ X × Y → R. Then V ∗ = max(x,y)∈X×Y f (x, y) = maxx∈X {maxy∈Y f (x, y)}
– That is, you can always find the maximum of a multivariate function in a 2-stage
maximization process

● Now, take Problem (13a)-(13d) and split the objective function (13a) into two stages
⎧ ⎫
⎪ t
⎪ ∞ ⎪
⎪
V0 (x0 ) = max ⎨ ∫ f (s, xs , ys ) ds + max ∫ f (s, xs , ys ) ds ⎬
(xs ,ys ∶s∈[0,t)) ⎪
⎪ 0 (xs ,ys ∶s≥t) t ⎪
⎪
⎩ ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶ ⎭
Vt (xt )

where Vt (xt ) is the value today of the continuation problem at time t with state xt resulting from the
optimal choices (x∗ ∗
s , ys ∶ s ∈ [0, t]) and the law of motion (13b)

⇒ We have made the problem recursive, i.e. “break it down” into smaller subproblems

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 33/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Hamilton-Jacobi-Bellman (HJB) Equation

● Differentiating equation (14) with respect to time t (Leibniz rule):

∂Vt (x∗t ) ∂Vt (x∗t )

f (t, x∗t , yt∗ ) + + g(t, x∗t , yt∗ ) = 0 ∀t ≥ 0 (15)
∂t ∂xt ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
ẋ∗t

● This is the Hamilton-Jacobi-Bellman equation characterizing the optimal pair (x∗ , y ∗ )

● The HJB is the time derivative of the recursive representation of the value function
→ The principle of optimality states that the choice of the optimal pair (x∗ , y ∗ ) is independent
from the time t at which we partition the problem into two.
⇒ Hence, the optimal pair (x∗ , y ∗ ) must satisfy that when increasing t, the gain in terms of the
return function f (t, x∗t , yt∗ ) must equal the loss in terms of the continuation value, which is
∂Vt (x∗
t)
given by the direct effect of the passage of time ∂t
plus the effect due to the change in
∂Vt (x∗
t)
the state, ∂xt
ẋt

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 34/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

The Stationary Problem

● Let us focus on the type of problem that will be most common in this course:
1 Exponential discounting, i.e. f (t, xt , yt ) = e−ρt f (xt , yt )
2 Time-autonomous constraint, i.e. ẋt = g(xt , yt ) instead of ẋt = g(t, xt , yt )

→ This problem generates a solution (x∗ , y ∗ ) that is time-consistent

● So we now focus on the following problem:

∞
V0 (x0 ) ≡ max ∫ e−ρt f (xt , yt ) dt
x,y 0
subject to ẋt = g(xt , yt ),
yt ∈ Y, xt ∈ X , ∀t ∈ R+
x0 given and lim bt xt ≥ x
t→∞

Most (if not all) of the problems we see in this course will be of this type

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 35/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Time consistency

Definition: Time Consistency

A solution (x∗s , ys∗ ∶ s ≥ t) of a dynamic optimization problem starting at time t is

time consistent if ∀t′ > t the sub-sequence (x∗s , ys∗ ∶ s ≥ t′ ) ⊂ (x∗s , ys∗ ∶ s ≥ t) is
the solution at time t′ of the sub-problem starting at time t′ with state x∗t′

● That is, the planner does not gain anything from revising the plan at any future date
(What was optimal at time t′ from the perspective of time t is still optimal when arriving at time t′ )

● This allows us to solve the problem just once, at time t = 0

● Time-consistent problems are much more straightforward to work with and satisfy all
the standard axioms of rational decision-making.

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 36/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

The Stationary HJB Equation: derivation

● According to the principle of optimality we can write the value function as,
t
V0 (x0 ) = ∫ e−ρs f (x∗s , ys∗ ) ds + Vt (x∗t ), ∀t ≥ 0
0
where note that
∞ ∞ ∞
Vt (x∗t ) = ∫ e−ρs f (x∗s , ys∗ ) ds = e−ρt ∫ e−ρ(s−t) f (x∗s , ys∗ ) ds = e−ρt ∫ e−ρs f (x∗t+s , yt+s
∗
)ds
t t 0
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶
V0 (x∗
t)

● Hence, stationarity ⇒ Vt (x) = e−ρt V0 (x)

● Hence, defining v (x) ≡ V0 (x) we can rewrite,

t
v(x0 ) = ∫ e−ρs f (x∗s , ys∗ ) ds + e−ρt v(x∗t ), ∀t ≥ 0
0

● And the HJB equation (15) becomes

f (x∗t , yt∗ ) = ρv(x∗t ) − vx (x∗t )ẋ∗t (17)

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 37/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

The Stationary HJB Equation: an interpretation

● The stationary HJB is more compactly written as,

ρv(x∗t ) = f (x∗t , yt∗ ) + v̇(x∗t ) (18)

´¹¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¶ ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶ ²
flow value instantaneous return change in value due
to change in the state

● This equation is often interpreted as a “no-arbitrage asset value equation”:

– Suppose people trade assets with stock market value v(x) at interest rate ρ
– The return of the asset comes from two sources:
1 Dividends, i.e. the flow payoff f (x, y)
2 Capital gains or losses (appreciation or depreciation of the asset), i.e. v̇(x)

– Equation (18) can be rewritten as

f (x, y) + v̇(x)
ρ=
v(x)
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
asset return

● Equation (18), or some form of it, is ubiquitous in (continuous-time) macro and finance
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 38/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Connecting the HJB Equation and the Hamiltonian

● Let’s write the HJB as,

ρv(xt ) = max { f (xt , yt ) + vx (xt )g (xt , yt ) }

yt
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
̂ t ,yt ,vx (xt ))
H(x

Note that the object to be maximized is the current value Hamiltonian when µt = vx (xt )

● The FOC that obtains is: fy (xt , yt ) + vx (xt )gy (xt , yt ) = 0

∂ ̂
which corresponds to the first condition of the Maximum Principle: ∂y H(xt , yt , vx (xt )) = 0

● And the Envelope Condition characterizes vx (xt )

ρvx (xt ) = fx (xt , yt ) + vxx (xt )g (xt , yt ) + vx (xt )gx (xt , yt )
which corresponds to the second condtion of the Maximum Principle. To see this, rearrange:

ρvx (xt ) − vxx (xt )ẋt = fx (xt , yt ) + vx (xt )gx (xt , yt )

´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶ ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
ρµt −µ̇t when µt = vx (xt ) ∂ ̂ t ,yt ,vx (xt ))
H(x
∂xt

● Hence, both formulations are identical when µt = vx (xt )

→ This gives a nice interpretation to the co-state variable µt .
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 39/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Stochastic Dynamic Optimization

● Often in economics we have to solve stochastic dynamic optimization problems

– By this we mean non-deterministic problems, where there is randomness (“shocks”)

– These are problems that the standard methods of optimal control cannot deal with
- That is, you won’t be able to use the “recipe” from Pontryagin’s Maximum Principle
(See slide 29)

– We will see some such examples in Topic IV of the course (endogenous growth)
(With uncertainty coming from the assumption that R&D does not always succeed)

● Fortunately, dynamic programming is well-equipped to tackle these types of problems.

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 40/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Stochastic Dynamic Optimization

There are two popular ways to introduce uncertainty in continuous-time problems:

1 Poisson point process:

– What is it? Some discontinuous event strikes at certain points in time
– Very easy to deal with, the math remains very tractable
– Popular in models of endogenous growth (Topic IV) or SaM frictions (Macro II)

2 Wiener process / Brownian motion:

– What is it? The continuous-time counterpart of a random walk
– Less straightforward than Poisson process, but still easy to deal with thanks to Itô’s Lemma
– Common approach to model things like business cycles or asset price movements

(Wiener processes are NOT part of the material for this course)

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 41/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Poisson Point Process

Counting process

● A Poisson process is a specific type of counting process

Def. A counting process is a stochastic process (Nt ∶ t ∈ R+ ) that records the number of events
(or “arrivals”) that occurred within some interval (0, t], and that satisfies the following
properties:
1 Nt ∈ Z+ ≡ N ∪ {0}, i.e. Nt must be a non-negative integer number
2 ∀t ≥ s, Nt ≥ Ns , i.e. events do not “disappear”
3 ∀t > s, Nt − Ns denotes the number of arrivals within the interval (s, t]

– Examples: Nt may record the number of

- buses that pass through a given bus stop
- job offers received by an unemployed worker
- innovations by a firm doing R&D

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 42/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Poisson Point Process

Definition
A counting process Nt ∶ t ∈ R+ is Poisson process with “arrival rate” α > 0 if:
1 N0 = 0 (i.e. at the beginning of time, no events have yet been recorded)

2 Nt has independent and stationary disjoint increments, that is:

– Independent increments:
The random variables (Nt1 − Nt0 ), (Nt2 − Nt1 ), . . . , (Ntn − Ntn−1 ) are independent
– Stationary increments:
The distribution of (Nt+s − Nt ) is a function of s but not of t

3 The number of events m ∈ Z+ within a time interval ∆ follows a Poisson distribution:

(α∆)m e−α∆
Pr[Nt+∆ − Nt = m] = , for any ∆ ≥ 0
m!
which has mean E [Nt+∆ − Nt ] = α∆
(Hence, the average number of events increases with the length ∆ of the time interval and with the parameter α defining
the rate of arrival of events)

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 43/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Poisson Point Process

Probabilities and Rates
● Probabilities. Let ∆ be some arbitrary small date. Then,
– Pr[N∆ = 0] = e−α∆ = 1 − α∆ + o(∆) ← Prob. of zero events within t ∈ [0, ∆]
−α∆
– Pr[N∆ = 1] = α∆e = α∆ + o(∆) ← Prob. of one event within t ∈ [0, ∆]
– Pr[N∆ ≥ 2] = ... = o(∆) ← Prob. of two or more events within t ∈ [0, ∆]
o(∆)
(where o(∆) contains the hihger order terms ∆2 , ∆3 , ..., and hence satisfies lim∆→0 ∆
= 0)

● Rates. We can also express these probabilities in rates per unit of time
– Pr[N∆ = 0]/∆ = o(∆)/∆ + 1/∆ − α
– Pr[N∆ = 1]/∆ = o(∆)/∆ + α
– Pr[N∆ ≥ 2]/∆ = o(∆)/∆

● Hence, taking the continuous time limit (∆ → 0) we see that

a) At any instant of time we can have either 0 or 1 event, but no more
Pr[N∆ =1]
b) α is the “Poisson arrival rate” at which events occur (lim∆→0 ∆
= α)
c) α is NOT a probability, indeed it can be that α > 1
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 44/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Poisson Point Process

Some final comments

1 The process is memoryless:

– The occurrence of one event does not affect the probability that a second event can occur
(Earthquakes are not Poisson, as one big earthquake typically generates several replicas afterward)

– The length of time without events does not predict the occurrence of the next one
(So buses arriving at a bus stop are probably not Poisson)

2 The distribution for the length of time periods between arrivals is exponential:
– Let {tn }∞
n=1 be the sequence of interarrival times.
(times elapsed between consecutive arrivals, i.e., points in time at which a worker gets a job offer)

– Then, {tn }∞
n=1 are identically distributed exponential random variables with mean 1/α:

−αt
Pr[tn ≤ t∣{tj }j=1 ] = 1 − e ∀n ∈ Z+ , ∀t > tn−1
n−1
,

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 45/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Dynamic Programming with a Poisson Point Process

Set up

● Consider the following standard non-stochastic consumption/savings problem:

∞
max ∫ e−ρs u(cs )ds, s.t. ȧs = w + ras − cs
(c ,a )
s s 0

with w and a0 given, plus some transversality condition on as

– In this simple problem, cs (consumption) is the control and as (assets) is the state

● Using the maximum principle we could write,

t
v (a0 ) ≡ max {∫ e−ρs u(cs ) ds + e−ρt v (at )}
(cs ∶s∈[0,t)) 0

for some small t > 0

● Now, we add a stochastic shock

– Let’s assume that the consumer may go bankrupt at Poisson arrival rate δ > 0
– When this happens, the consumer’s wealth drops to zero

● Let’s derive the HJB equation of this stochastic problem following a different procedure
J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 46/48
 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Dynamic Programming with a Poisson Point Process

The Bellman equation

● Let’s start by writing,

⎧
⎪
⎪ t −ρs
v(a0 ) = max ⎨ ∫ e u(cs ) ds + e−ρt [ e−δt v(at ) + e−δt δt v(0)
(cs ∶s∈[0,t)) ⎪
⎪ ± ´¹¹ ¹ ¸¹¹ ¹¶
⎩
0
Prob. of not Prob. of going
going bankrupt bankrupt only once
⎫
⎪
⎪
+ Pr[Nt ≥ 2] v(0) ]⎬
´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹¶ ⎪
⎪
Prob. of going
⎭
bankrupt twice or more

● Next, compute a Taylor expansion with respect to t around t = 0

– The first term becomes: ∫0 e−ρs u(cs ) ds = u(c0 )∆ + o(∆)

t

– The second: e−(ρ+δ)t v(at ) = v(a0 ) + va (a0 )ȧ0 ∆ − (ρ + δ)v(a0 )∆ + o(∆)

– The third: e−(ρ+δ)t δt v(0) = δv(0)∆
– And the fourth: Pr[Nt ≥ 2]v(0) = o(∆)

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 47/48

 Introduction Calculus of Variations Optimal Control Dynamic Programming Deterministic Case Stochastic Case

Dynamic Programming with a Poisson Point Process

The HJB equation

● So, we can write

v(a0 ) = max {u(c0 )∆ + v(a0 ) + va (a0 )ȧ0 ∆ − (ρ + δ)v(a0 )∆ + δv(0)∆ + o(∆)}
c0

● Divide both sides by ∆, take the limit as ∆ → 0, and write it all at time t:

∂v(at )
ρv(at ) = max {u(ct ) + ȧt + δ (v(0) − v(at )) }
ct >0 ∂at ´¹¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¸¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¹ ¶
change in value due
to bankruptcy shock

● A particular form of this HJB equation is when the problem is such that v(0) = 0:

∂v(at )
(ρ + δ)v(at ) = max {u(ct ) + ȧt }
ct >0 ∂at
(the effective discount is given by the time discount ρ plus the probability disocunt δ , as at the stochastic rate δ the value
function goes to zero)

J. Pijoan-Mas Macroeconomics I (CEMFI 2024-2025) 48/48