 Home Course Homework Lectures

Notes
02: Serial Code 03: SciML Intro 04: How Loops Work

05: Basics of Parallelism 06: Flavors of Parallelism 07: ODEs

08: Forward AD 09: Stiff ODEs 10: Reverse AD 11: δP 12: MPI

13: GPUs 14: PDEs 15: Physics Informed Learning

16: Probabilistic Programming 17: Global Sensitivity Analysis

18: Profiling & Optimization 19: Uncertainty Programming

Ordinary Differential
Equations, Applications
and Discretizations
Chris Rackauckas

October 1st, 2020

Youtube Video Link Part 1

Youtube Video Link Part 2

Now that we have a sense of parallelism, let's return back to

our thread on scientific machine learning to start
constructing parallel algorithms for integration of scientific
models. We previously introduced discrete dynamical
systems and their asymptotic behavior. However, many
physical systems are not discrete and are in fact continuous.
In this discussion we will understand how to numerically
compute ordinary differential equations by transforming
them into discrete dynamical systems, and use this to come
up with simulation techniques for physical systems.

What is an Ordinary Differential

Equation?

An ordinary differential equation is an equation defined by a

relationship on the derivative. In its general form we have
that
′
u = f (u, p, t)

describes the evolution of some variable u(t) which we

would like to solve for. In its simplest sense, the solution to
the ordinary differential equation is just the integral, since by
taking the integral of both sides and applying the
Fundamental Theorem of Calculus we have that
tf

u = ∫ f (u, p, t)dt
t0

The difficulty of this equation is that the variable u(t) is

unknown and dependent on t , meaning that the integral
cannot readily be solved by simple calculus. In fact, in almost
all cases there exists no analytical solution for u which is
readily available. However, we can understand the behavior
by looking at some simple cases.

Solving Ordinary Differential

Equations in Julia

To solve an ordinary differential equation in Julia, one can

use the DifferentialEquations.jl package to define the
differential equation you'd like to solve. Let's say we want to
solve the Lorenz equations:

dx
= σ(y − x) (1)
dt

dy
= x(ρ − z) − y (2)
dt
dz
= xy − βz (3)
dt

which was the system used in our investigation of discrete

dynamics. The first thing we need to do is give it this
differential equation. We can either write it in an in-place
form f(du,u,p,t) or an out-of-place form f(u,p,t) . Let's
write it in the in-place form:

function lorenz(du,u,p,t)
du[1] = p[1]*(u[2]-u[1])
du[2] = u[1]*(p[2]-u[3]) - u[2]
du[3] = u[1]*u[2] - p[3]*u[3]
end

lorenz (generic function with 3 methods)

Question: How could I maybe speed this up a little?

Next we give an initial condition. Here, this is a vector of

equations, so our initial condition has to be a vector. Let's
choose the following initial condition:

u0 = [1.0,0.0,0.0]

3-element Vector{Float64}:
1.0
0.0
0.0

Notice that I made sure to use Float64 values in the initial

condition. The Julia library's functions are generic and
internally use the corresponding types that you give it.
Integer types do not bode well for continuous problems.

Next, we have to tell it the timespan to solve on. Here, let's

some from time 0 to 100. This means that we would use:

tspan = (0.0,100.0)

(0.0, 100.0)

Now we need to define our parameters. We will use the

same ones as from our discrete dynamical system
investigation.

p = (10.0,28.0,8/3)

(10.0, 28.0, 2.6666666666666665)

These describe an ODEProblem . Let's bring in

DifferentialEquations.jl and define the ODE:

using DifferentialEquations
prob = ODEProblem(lorenz,u0,tspan,p)

ODEProblem with uType Vector{Float64} and tType Float64. In-place: true

timespan: (0.0, 100.0)
u0: 3-element Vector{Float64}:
1.0
 1.0
0.0
0.0

Now we can solve it by calling solve :

sol = solve(prob)

retcode: Success
Interpolation: specialized 4th order "free" interpolation, specialized 2nd
order "free" stiffness-aware interpolation
t: 1263-element Vector{Float64}:
0.0
3.5678604836301404e-5
0.0003924646531993154
0.0032624077544510573
0.009058075635317072
0.01695646895607931
0.02768995855685593
0.04185635042021763
0.06024041165841079
0.08368541255159562
⋮
99.30760258626904
99.39665422328268
99.49536147459878
99.58822928767293
99.68983993598462
99.77864535713971
99.85744078539504
99.93773320913628
100.0
u: 1263-element Vector{Vector{Float64}}:
[1.0, 0.0, 0.0]
[0.9996434557625105, 0.0009988049817849058, 1.781434788799208e-8]
[0.9961045497425811, 0.010965399721242457, 2.146955365838907e-6]
[0.9693591634199452, 0.08977060667778931, 0.0001438018342266937]
[0.9242043615038835, 0.24228912482984957, 0.0010461623302512404]
[0.8800455868998046, 0.43873645009348244, 0.0034242593451028745]
[0.8483309847495312, 0.6915629321083602, 0.008487624590227805]
[0.8495036669651213, 1.0145426355349096, 0.01821208962127994]
[0.9139069574560097, 1.4425599806525806, 0.03669382197085303]
[1.088863826836895, 2.052326595543049, 0.0740257368585531]
⋮
[4.669609096878053, 3.061564434452441, 25.1424735017959]
[4.188801916573263, 4.617474401440693, 21.09864175382292]
[5.559603854699961, 7.905631612648314, 18.79323210016923]
[8.556629716266505, 12.533041060088328, 20.6623639692711]
[12.280585075547771, 14.505154761545633, 29.332088452699942]
[11.736883151600804, 8.279294641640229, 34.68007510231878]
[8.10973327066804, 3.2495066495235854, 31.97052076740117]
[4.958629886040755, 2.194919965065022, 26.948439650907677]
[3.8020065515435855, 2.787021797920187, 23.420567509786622]

To see what the solution looks like, we can call plot :

using Plots
plot(sol)
We can also plot phase space diagrams by telling it which
vars to compare on which axis. Let's plot this in the (x,y,z)

plane:

plot(sol,vars=(1,2,3))

Note that the sentinal to time is 0 , so we can also do (t,y,z)

with:

plot(sol,vars=(0,2,3))

The equation is continuous and therefore the solution is

continuous. We can see this by checking how it is at any
random time value:

sol(0.5)

3-element Vector{Float64}:
6.503654868503322
-8.508354689912013
38.09199724760152
which gives the current evolution at that time point.

Differential Equations from Scientific

Contexts

N-Body Problems and Astronomy

There are many different contexts in which differential
equations show up. In fact, it's not a stretch to say that the
laws in all fields of science are encoded in differential
equations. The starting point for physics is Newton's laws of
gravity, which define an N-body ordinary differential
equation system by describing the force between two
particles as:
m1 m2
F = G
2
r

where r is the Euclidean distance between the two particles.

2

From here, we use the fact that

F = ma

to receive differential equations in terms of the accelerations

of each particle. The differential equation is a system, where
we know the change in position is due to the current
velocity:

′
x = v

and the change in velocity is the acceleration:

mi
′
v = F /m = G
2
r
i

where i runs over the other particles. Thus we have a vector

of position derivatives and a vector of velocity derivatives
that evolve over time to give the evolving positions and
velocity.

An example of this is the Pleiades problem, which is an

approximation to a 7-star chaotic system. It can be written as:

using OrdinaryDiffEq

function pleiades(du,u,p,t)
@inbounds begin
x = view(u,1:7) # x
y = view(u,8:14) # y
v = view(u,15:21) # x′
w = view(u,22:28) # y′
 w = view(u,22:28) # y′
du[1:7] .= v
du[8:14].= w
for i in 15:28
du[i] = zero(u[1])
end
for i=1:7,j=1:7
if i != j
r = ((x[i]-x[j])^2 + (y[i] - y[j])^2)^(3/2)
du[14+i] += j*(x[j] - x[i])/r
du[21+i] += j*(y[j] - y[i])/r
end
end
end
end
tspan = (0.0,3.0)
prob = ODEProblem(pleiades,[3.0,3.0,-1.0,-3.0,2.0,-2.0,

ODEProblem with uType Vector{Float64} and tType Float64. In-place: true

timespan: (0.0, 3.0)
u0: 28-element Vector{Float64}:
3.0
3.0
-1.0
-3.0
2.0
-2.0
2.0
3.0
-3.0
2.0
⋮
1.75
-1.5
0.0
0.0
0.0
-1.25
1.0
0.0
0.0

where we assume m = i. When we solve this equation we

i

receive the following:

sol = solve(prob,Vern8(),abstol=1e-10,reltol=1e-10)
plot(sol)

tspan = (0.0,200.0)
tspan = (0.0,200.0)
prob = ODEProblem(pleiades,[3.0,3.0,-1.0,-3.0,2.0,-2.0,
sol = solve(prob,Vern8(),abstol=1e-10,reltol=1e-10)
plot(sol,vars=((1:7),(8:14)))

Population Ecology: Lotka-Volterra

Population ecology's starting point is the Lotka-Volterra
equations which describes the interactions between a
predator and a prey. In this case, the prey grows at an
exponential rate but has a term that reduces its population by
being eaten by the predator. The predator's growth is
dependent on the available food (the amount of prey) and has
a decay rate due to old age. This model is then written as
follows:

function lotka(du,u,p,t)
du[1] = p[1]*u[1] - p[2]*u[1]*u[2]
du[2] = -p[3]*u[2] + p[4]*u[1]*u[2]
end

p = [1.5,1.0,3.0,1.0]
prob = ODEProblem(lotka,[1.0,1.0],(0.0,10.0),p)
sol = solve(prob)
plot(sol)

Biochemistry: Robertson Equations

Biochemical equations commonly display large separation of
timescales which lead to a stiffness phenomena that will be
investigated later. The classic "hard" equations for ODE
integration thus tend to come from biology (not physics!)
due to this property. One of the standard models is the
due to this property. One of the standard models is the
Robertson model, which can be described as:

using Sundials, ParameterizedFunctions

function rober(du,u,p,t)
y₁,y₂,y₃ = u
k₁,k₂,k₃ = p
du[1] = -k₁*y₁+k₃*y₂*y₃
du[2] = k₁*y₁-k₂*y₂^2-k₃*y₂*y₃
du[3] = k₂*y₂^2
end
prob = ODEProblem(rober,[1.0,0.0,0.0],(0.0,1e5),(0.04,3e7
sol = solve(prob,Rosenbrock23())
plot(sol)

plot(sol, xscale=:log10, tspan=(1e-6, 1e5), layout=(3,1

Chemical Physics: Pollution Models

Chemical reactions in physical models are also described as
differential equation systems. The following is a classic model
of dynamics between different species of pollutants:

k1=.35e0
k2=.266e2
k3=.123e5
k4=.86e-3
k5=.82e-3
k6=.15e5
k7=.13e-3
k8=.24e5
k9=.165e5
k10=.9e4
k11=.22e-1
k12=.12e5
k13=.188e1
k13=.188e1
k14=.163e5
k15=.48e7
k16=.35e-3
k17=.175e-1
k18=.1e9
k19=.444e12
k20=.124e4
k21=.21e1
k22=.578e1
k23=.474e-1
k24=.178e4
k25=.312e1
p = (k1,k2,k3,k4,k5,k6,k7,k8,k9,k10,k11,k12,k13,k14,k15
function f(dy,y,p,t)
k1,k2,k3,k4,k5,k6,k7,k8,k9,k10,k11,k12,k13,k14,k15,k16
r1 = k1 *y[1]
r2 = k2 *y[2]*y[4]
r3 = k3 *y[5]*y[2]
r4 = k4 *y[7]
r5 = k5 *y[7]
r6 = k6 *y[7]*y[6]
r7 = k7 *y[9]
r8 = k8 *y[9]*y[6]
r9 = k9 *y[11]*y[2]
r10 = k10*y[11]*y[1]
r11 = k11*y[13]
r12 = k12*y[10]*y[2]
r13 = k13*y[14]
r14 = k14*y[1]*y[6]
r15 = k15*y[3]
r16 = k16*y[4]
r17 = k17*y[4]
r18 = k18*y[16]
r19 = k19*y[16]
r20 = k20*y[17]*y[6]
r21 = k21*y[19]
r22 = k22*y[19]
r23 = k23*y[1]*y[4]
r24 = k24*y[19]*y[1]
r25 = k25*y[20]

dy[1] = -r1-r10-r14-r23-r24+
r2+r3+r9+r11+r12+r22+r25
dy[2] = -r2-r3-r9-r12+r1+r21
dy[3] = -r15+r1+r17+r19+r22
dy[4] = -r2-r16-r17-r23+r15
dy[5] = -r3+r4+r4+r6+r7+r13+r20
dy[6] = -r6-r8-r14-r20+r3+r18+r18
dy[7] = -r4-r5-r6+r13
dy[8] = r4+r5+r6+r7
dy[9] = -r7-r8
dy[10] = -r12+r7+r9
dy[11] = -r9-r10+r8+r11
dy[12] = r9
dy[13] = -r11+r10
dy[14] = -r13+r12
dy[15] = r14
dy[16] = -r18-r19+r16
dy[17] = -r20
dy[18] = r20
dy[19] = -r21-r22-r24+r23+r25
dy[20] = -r25+r24
 dy[20] = -r25+r24
end

f (generic function with 2 methods)

u0 = zeros(20)
u0[2] = 0.2
u0[4] = 0.04
u0[7] = 0.1
u0[8] = 0.3
u0[9] = 0.01
u0[17] = 0.007
prob = ODEProblem(f,u0,(0.0,60.0),p)
sol = solve(prob,Rodas5())

retcode: Success
Interpolation: specialized 4rd order "free" stiffness-aware interpolation
t: 29-element Vector{Float64}:
0.0
0.0013845590497824308
0.003242540880561935
0.007901605227525086
0.016011572765091416
0.02740615678429701
0.044612092501151696
0.07720629370280543
0.11607398786651008
0.1763063703057767
⋮
3.676321507769747
5.344945477147836
7.985769209063678
11.981251310096694
17.452504634746386
24.882247321193052
34.66462280306778
47.27763232418448
60.0
u: 29-element Vector{Vector{Float64}}:
[0.0, 0.2, 0.0, 0.04, 0.0, 0.0, 0.1, 0.3, 0.01, 0.0, 0.0, 0.0, 0.0, 0.0, 0
.0, 0.0, 0.007, 0.0, 0.0, 0.0]
[0.0002935083676916062, 0.19970649101355778, 1.6938359129632626e-10, 0.039
706733663924174, 1.1424841091201038e-7, 1.0937647553427759e-7, 0.0999996667
6440009, 0.3000003350396963, 0.00999998209858389, 5.6037724774041996e-9, 6.
047938337401518e-9, 1.004757080331196e-8, 5.9812281205517996e-12, 6.2395533
94674922e-9, 2.3022538431539757e-10, 3.1293305822799984e-17, 0.006999999417
662247, 5.823377531818463e-10, 3.824053893000698e-10, 6.930819262664891e-14
]
[0.0006836584262738197, 0.19931633629695783, 1.9606016564783863e-10, 0.039
317452226523025, 2.0821314359971537e-7, 2.5242637698192583e-7, 0.0999988408
7661927, 0.30000116344748634, 0.009999897466494299, 2.0605145128366293e-8,
1.6844788957556812e-8, 8.136618249763386e-8, 1.0724538885842349e-10, 6.4867
5094400525e-8, 3.1010416395449565e-9, 3.0986508172589845e-17, 0.00699999644
4140231, 3.555859768109287e-9, 2.0643860873886213e-9, 2.0476275988812484e-1
2]
[0.0016428179821615404, 0.19835713123489696, 2.624592441462544e-10, 0.0383
6231097512178, 3.5152731099463506e-7, 4.6645725233828785e-7, 0.099995449213
65162, 0.3000045632051474, 0.00999947365101553, 4.4964384052574966e-8, 3.33
5028459989385e-8, 4.812858451973162e-7, 1.4409606354799333e-9, 4.4444645014
965825e-7, 3.7457517454187436e-8, 3.0233751050330316e-17, 0.006999981334742
691, 1.8665257309560828e-8, 1.1746742713316084e-8, 6.886039278040633e-11]
[0.003246284952583631, 0.19675344580878915, 3.734915472124081e-10, 0.03676
 [0.003246284952583631, 0.19675344580878915, 3.734915472124081e-10, 0.03676
859821471882, 4.320335375820004e-7, 5.784969040588645e-7, 0.099987536469499
57, 0.30001250073250896, 0.00999841279050207, 5.8477926969890516e-8, 4.2283
73016580745e-8, 1.5155870313913993e-6, 8.524980254717523e-9, 1.461534608031
6128e-6, 2.1422146419315973e-7, 2.897772883427686e-17, 0.006999943344407836
, 5.6655592163595934e-8, 4.43684966594383e-8, 1.0618431054301687e-9]
[0.005361393004057219, 0.19463777403944538, 5.199898853042825e-10, 0.03466
7923081446166, 4.3013501105002817e-7, 5.629734298792673e-7, 0.0999758096289
3295, 0.30002429018799176, 0.00999682035482297, 5.77764454181714e-8, 4.1511
45797901367e-8, 3.0752433164909642e-6, 2.7267223985258345e-8, 2.98889630342
90727e-6, 6.762049531575715e-7, 2.7322164104134524e-17, 0.00699988627401965
7, 1.1372598034289961e-7, 1.1359725262068185e-7, 7.943533863137097e-9]
[0.008274964935351174, 0.19172294848166438, 7.218269559835734e-10, 0.03177
4179928789587, 3.982608445392987e-7, 5.00523146749144e-7, 0.099959353944089
18, 0.30004089901876024, 0.00999460673549988, 5.1781453866305105e-8, 3.7149
92485079143e-8, 5.2294190200072585e-6, 6.871429675796404e-8, 5.040637233611
9715e-6, 1.6909614724980285e-6, 2.504157391398737e-17, 0.006999806991160331
, 1.9300883966852227e-7, 2.3808903519156553e-7, 4.4409090022701e-8]
[0.013002670675854915, 0.1869919454695578, 1.049323517342023e-9, 0.0270783
34676230022, 3.596573982134105e-7, 4.230620972419732e-7, 0.0999319171933284
9, 0.3000687867927592, 0.009990985412449237, 4.426632357885494e-8, 3.171747
018445745e-8, 8.707148275200576e-6, 1.7539882287368083e-7, 8.15954182882010
5e-6, 4.282354997367439e-6, 2.134072762166872e-17, 0.006999677462783783, 3.
2253721621608143e-7, 4.2925313031200287e-7, 2.4842381839580386e-7]
[0.01754950653069443, 0.18244010960656093, 1.3641806124928421e-9, 0.022563
57055688371, 3.3611627982754385e-7, 3.7150093428726456e-7, 0.09990309462743
716, 0.30009836544883595, 0.009987255299681627, 3.931687421453388e-8, 2.813
0812802630264e-8, 1.2231150049080625e-5, 3.3462364587005033e-7, 1.103334852
2905705e-5, 8.113769678679295e-6, 1.7782593323554984e-17, 0.006999544244933
8795, 4.5575506612068657e-7, 5.171121404557456e-7, 7.09178639855165e-7]
[0.02286066099315797, 0.17711993361265663, 1.7318456679719284e-9, 0.017295
319170346234, 3.1879075517672363e-7, 3.280175981175673e-7, 0.09986309177576
28, 0.30013989594565554, 0.009982163705371506, 3.519652711590193e-8, 2.5136
768154051253e-8, 1.6956872533522306e-5, 6.253075543101422e-7, 1.43919101330
8265e-5, 1.5036820413025804e-5, 1.3630627583154936e-17, 0.00699936266302395
2, 6.373369760492062e-7, 5.039634032907158e-7, 1.6196514074344935e-6]
⋮
[0.03892999723063904, 0.16044023705466662, 2.8511492881988138e-9, 0.003179
5600755990337, 3.097289790606945e-7, 2.6025228296017144e-7, 0.0980811460422
5782, 0.30211293753489554, 0.009756427876936317, 2.8857054777614526e-8, 2.0
5076507452386e-8, 0.0002145197125808461, 2.4313110493778758e-5, 2.984486304
939218e-5, 0.0005806906609807025, 2.5058455898662905e-18, 0.006991260059066
824, 8.739940933174326e-6, 5.64479550247737e-7, 1.2098731834913306e-5]
[0.039731796287462894, 0.15934875829742126, 2.910031107146617e-9, 0.003267
8397244506954, 3.0340454593255866e-7, 2.533682719229086e-7, 0.0972715952068
385, 0.3030166682186243, 0.009654512408596433, 2.8039000372424056e-8, 1.991
4463596397305e-8, 0.0003034866969101812, 3.515693408225822e-5, 2.8843324342
223694e-5, 0.0008554037161571975, 2.5754197332981358e-18, 0.006987546308720
155, 1.2453691279844257e-5, 6.384025603676959e-7, 1.4123181158100541e-5]
[0.04094318678623549, 0.15768509187834306, 2.998971487551489e-9, 0.0034038
15782394805, 2.93988212875527e-7, 2.432060017297553e-7, 0.09603188713491922
, 0.3043989910057561, 0.009500566541513962, 2.6846369481449033e-8, 1.904976
991469038e-8, 0.0004379930888116998, 5.130948406305054e-5, 2.73117734497972
47e-5, 0.0012863759435122516, 2.682583931182485e-18, 0.006981869477596393,
1.8130522403605964e-5, 7.249055971399561e-7, 1.6655501124600894e-5]
[0.0426545609986256, 0.15530170721543912, 3.1246155060271694e-9, 0.0036015
75222497331, 2.8144779831986e-7, 2.2972054943080024e-7, 0.09424347009700186
, 0.30638970302322827, 0.009282897844674818, 2.528730342602189e-8, 1.791968
496025925e-8, 0.0006285309299580917, 7.35640288181715e-5, 2.531107615267494
8e-5, 0.0019296590487066839, 2.8384402789327358e-18, 0.006973700749524549,
2.629925047545015e-5, 8.253072229358661e-7, 1.9841700593870354e-5]
[0.04479393166011826, 0.15226289579317082, 3.281711134773705e-9, 0.0038589
221832575444, 2.6689443964468237e-7, 2.141467586267528e-7, 0.09194251736341
221832575444, 2.6689443964468237e-7, 2.141467586267528e-7, 0.09194251736341
381, 0.3089455007693318, 0.00901015441479903, 2.3520931242225642e-8, 1.6639
7886985959e-8, 0.0008681120853301324, 0.0001002235878291863, 2.305697615943
2428e-5, 0.002794230305325378, 3.041258194415907e-18, 0.006963219964939303,
3.6780035060698035e-5, 9.439336216189186e-7, 2.3887359967470623e-5]
[0.047387957127462514, 0.14848196819379966, 3.472264655035832e-9, 0.004187
711120538979, 2.506414859123144e-7, 1.9688141701254014e-7, 0.08904810542258
146, 0.3121528528950735, 0.008678047248147585, 2.1605580609099232e-8, 1.525
2579424714227e-8, 0.0011615458041670701, 0.00013035939391932975, 2.06304833
0498099e-5, 0.003939836311621373, 3.3003803021585434e-18, 0.006950068615422
143, 4.993138457785821e-5, 1.0960311780641968e-6, 2.9391471009553375e-5]
[0.050368820662391366, 0.14399340437115485, 3.6913545076957706e-9, 0.00459
1322479293647, 2.334751006767882e-7, 1.7884243621649027e-7, 0.0855674941007
5945, 0.31600005201609194, 0.008293774589266535, 1.9649652481043943e-8, 1.3
836777792991099e-8, 0.0015041058995731527, 0.0001612874114336738, 1.8176658
967239337e-5, 0.005401518209025111, 3.6184707672883255e-18, 0.0069342796995
7088, 6.572030042912137e-5, 1.2906023386937759e-6, 3.6839371828226355e-5]
[0.05364912800384803, 0.13885256634333903, 3.932627740345218e-9, 0.0050728
52392385736, 2.160105896887418e-7, 1.6077416721927844e-7, 0.081521656918457
5, 0.3204605702073428, 0.007866251982276728, 1.7728584506478013e-8, 1.24471
10292692562e-8, 0.0018899286669815415, 0.0001897490305316394, 1.57995587965
91382e-5, 0.007213605828677921, 3.997969685509972e-18, 0.006915930516551773
, 8.406948344822907e-5, 1.5343173858417779e-6, 4.6708238108952274e-5]
[0.05646254720110732, 0.13424842010171067, 4.139733781936828e-9, 0.0055231
39725450954, 2.01898086132199e-7, 1.4645450805851654e-7, 0.0778424951212836
6, 0.3245075304667331, 0.007494014171885846, 1.622296660466669e-8, 1.135866
3790538066e-8, 0.0022305051646553087, 0.00020871630726229673, 1.39693487378
14003e-5, 0.008964884997558712, 4.352845989434378e-18, 0.006899219722669948
, 0.00010078027733005333, 1.7721521050343056e-6, 5.6829620128181005e-5]

plot(sol)

plot(sol, xscale=:log10, tspan=(1e-6, 60), layout=(3,1))

Geometric Properties
Linear Ordinary Differential Equations
The simplest ordinary differential equation is the scalar linear
ODE, which is given in the form

′
u = αu

We can solve this by noticing that (e ) = αe satisfies the

αt ′ αt

differential equation and thus the general solution is:

αt
u(t) = u(0)e

From the analytical solution we have that:

If Re(α) > 0 then u(t) → ∞ as t → ∞

If Re(α) < 0 then u(t) → 0 as t → ∞
If Re(α) = 0 then u(t) has a constant or periodic
solution.

This theory can then be extended to multivariable systems in

the same way as the discrete dynamics case. Let u be a vector
and have

′
u = Au

be a linear ordinary differential equation. Assuming A is

diagonalizable, we diagonalize A = P DP to get
−1

′
Pu = DP u

and change coordinates z = P u so that we have

′
z = Dz

which decouples the equation into a system of linear ordinary

differential equations which we solve individually. Thus we
see that, similarly to the discrete dynamical system, we have
that:

If all of the eigenvalues negative, then u(t) → 0 as

t → ∞

If any eigenvalue is positive, then u(t) → ∞ as t → ∞

Nonlinear Ordinary Differential Equations

As with discrete dynamical systems, the geometric properties
extend locally to the linearization of the continuous
dynamical system as defined by:
 ′
df
u = u
du

where is the Jacobian of the system. This is a consequence

df

du

of the Hartman-Grubman Theorem.

Numerically Solving Ordinary

Differential Equations

Euler's Method
To numerically solve an ordinary differential equation, one
turns the continuous equation into a discrete equation by
discretizing it. The simplest discretization is the Euler method.
The Euler method can be thought of as a simple
approximation replacing dt with a small non-infinitesimal Δt
. Thus we can approximate

′
du Δu
f (u, p, t) = u = ≈
dt Δt

and now since Δu = u n+1 − un we have that

Δtf (u, p, t) = un+1 − un

We need to make a choice as to where we evaluate f at. The

simplest approximation is to evaluate it at t with u where n n

we already have the data, and thus we re-arrange to get

un+1 = un + Δtf (u, p, t)

This is the Euler method.

We can interpret it more rigorously by looking at the Taylor

series expansion. First write out the Taylor series for the
ODE's solution in the near future:
2
′
Δt ′′
u(t + Δt) = u(t) + Δtu (t) + u (t) + …
2

Recall that u = f (u, p, t) by the definition of the ODE

′

system, and thus we have that

2
u(t + Δt) = u(t) + Δtf (u, p, t) + O(Δt )

This is a first order approximation because the error in our

step can be expressed as an error in the derivative, i.e.

u(t + Δt) − u(t)

= f (u, p, t) + O(Δt)
Δt
Higher Order Methods
We can use this analysis to extend our methods to higher
order approximation by simply matching the Taylor series to
a higher order. Intuitively, when we developed the Euler
method we had to make a choice:

un+1 = un + Δtf (u, p, t)

where do we evaluate f ? One may think that the best

derivative approximation my come from the middle of the
interval, in which case we might want to evaluate it at t + Δt

. To do so, we can use the Euler method to approximate the

value at t + Δt

2
and then use that value to approximate the
derivative at t + . This looks like:
Δt

k1 = f (un , p, t)

Δt Δt
k2 = f (un + k1 , p, t + )
2 2

un+1 = un + Δtk2

which we can also write as:

Δt Δt
un+1 = un + Δtf (un + fn , p, t + )
2 2

where f = f (u , p, t) . If we do the two-dimensional Taylor

n n

expansion we get:
2
Δt
un+1 = un + Δtfn + (ft + fu f )(un , p, t)
2
3
Δt
2
+ (ftt + 2ftu f + fuu f )(un , p, t)
6

which when we compare against the true Taylor series:

2 3
Δt Δt 2 2
u(t + Δt) = un + Δtf (un , p, t) + (ft + fu f )(un , p, t) + (ftt + 2ftu + fuu f + ft fu + fu f )(un , p, t)
2 6

and thus we see that

3
u(t + Δt) − un = O(Δt )

Runge-Kutta Methods
More generally, Runge-Kutta methods are of the form:

k1 = f (un , p, t)

k2 = f (un + Δt(a21 k1 ), p, t + Δtc2 )

k3 = f (un + Δt(a31 k1 + a32 k2 ), p, t + Δtc3 )

un+1 = un + Δt(b1 k1 + … + bs ks )
where s is the number of stages. These can be expressed as a
tableau:

The order of the Runge-Kutta method is simply the number

of terms in the Taylor series that ends up being matched by
the resulting expansion. For example, for the 4th order you
can expand out and see that the following equations need to
be satisfied:

The classic Runge-Kutta method is also known as RK4 and

is the following 4th order method:

k1 = f (un , p, t)

Δt Δt
k2 = f (un + k1 , p, t + )
2 2

Δt Δt
k3 = f (un + k2 , p, t + )
2 2

k4 = f (un + Δtk3 , p, t + Δt)

Δt
un+1 = un + (k1 + 2k2 + 2k3 + k4 )
6

While it's widely known and simple to remember, it's not

necessarily good. The way to judge a Runge-Kutta method is
by looking at the size of the coefficient of the next term in
the Taylor series: if it's large then the true error can be larger,
even if it matches another one asymptotically.

What Makes a Good Method?

Leading Truncation Coefficients

For given orders of explicit Runge-Kutta methods, lower
bounds for the number of f evaluations (stages) required to
receive a given order are known:

While unintuitive, using the method is not necessarily the

one that reduces the coefficient the most. The reason is
because what is attempted in ODE solving is precisely the
opposite of the analysis. In the ODE analysis, we're looking
at behavior as Δt → 0 . However, when efficiently solving
ODEs, we want to use the largest Δt which satisfies error
tolerances.

The most widely used method is the Dormand-Prince 5th

order Runge-Kutta method, whose tableau is represented as:

Notice that this method takes 7 calls to f for 5th order. The
key to this method is that it has optimized leading truncation
error coefficients, under some extra assumptions which allow
for the analysis to be simplified.

Looking at the Effects of RK Method Choices and Code

Optimizations
Optimizations
Pulling from the SciML Benchmarks, we can see the general
effect of these different properties on a given set of Runge-
Kutta methods:

Here, the order of the method is given in the name. We can

see one immediate factor is that, as the requested error in the
calculation decreases, the higher order methods become more
efficient. This is because to decrease error, you decrease Δt ,
and thus the exponent difference with respect to Δt has
more of a chance to pay off for the extra calls to f .
Additionally, we can see that order is not the only
determining factor for efficiency: the Vern8 method seems to
have a clear approximate 2.5x performance advantage over
the whole span of the benchmark compared to the DP8
method, even though both are 8th order methods. This is
because of the leading truncation terms: with a small enough
Δt , the more optimized method (Vern8) will generally have

low error in a step for the same Δt because the coefficients in

the expansion are generally smaller.

This is a factor which is generally ignored in high level

discussions of numerical differential equations, but can lead
to orders of magnitude differences! This is highlighted in the
following plot:

Here we see ODEInterface.jl's ODEInterfaceDiffEq.jl

wrapper into the SciML common interface for the standard
dopri method from Fortran, and ODE.jl, the original ODE
dopri method from Fortran, and ODE.jl, the original ODE
solvers in Julia, have a performance disadvantage compared
to the DifferentialEquations.jl methods due in part to some
of the coding performance pieces that we discussed in the
first few lectures.

Specifically, a large part of this can be attributed to inlining

of the higher order functions, i.e. ODEs are defined by a user
function and then have to be called from the solver. If the
solver code is compiled as a shared library ahead of time, like
is commonly done in C++ or Fortran, then there can be a
function call overhead that is eliminated by JIT compilation
optimizing across the function call barriers (known as
interprocedural optimization). This is one way which a JIT
system can outperform an AOT (ahead of time) compiled
system in real-world code (for completeness, two other ways
are by doing full function specialization, which is something
that is not generally possible in AOT languages given that
you cannot know all types ahead of time for a fully generic
function, and calling C itself, i.e. c-ffi (foreign function
interface), can be optimized using the runtime information of
the JIT compiler to outperform C!).

The other performance difference being shown here is due to

optimization of the method. While a slightly different order,
we can see a clear difference in the performance of RK4 vs
the coefficient optimized methods. It's about the same order
of magnitude as "highly optimized code differences",
showing that both the Runge-Kutta coefficients and the code
implementation can have a significant impact on
performance.

Taking a look at what happens when interpreted languages

get involved highlights some of the code challenges in this
domain. Let's take a look at for example the results when
simulating 3 ODE systems with the various RK methods:

We see that using interpreted languages introduces around a

50x-100x performance penalty. If you recall in your previous
lecture, the discrete dynamical system that was being
simulated was the 3-dimensional Lorenz equation discretized
by Euler's method, meaning that the performance of that
implementation is a good proxy for understanding the
performance differences in this graph. Recall that in previous
lectures we saw an approximately 5x performance advantage
when specializing on the system function and size and around
10x by reducing allocations: these features account for the
performance differences noticed between library
implementations, which are then compounded by the use of
different RK methods (note that R uses "call by copy" which
even further increases the memory usages and makes
standard usage of the language incompatible with mutating
function calls!).

Stability of a Method
Simply having an order on the truncation error does not
imply convergence of the method. The disconnect is that the
errors at a given time point may not dissipate. What also
needs to be checked is the asymptotic behavior of a
disturbance. To see this, one can utilize the linear test
problem:

′
u = αu

and ask the question, does the discrete dynamical system

defined by the discretized ODE end up going to zero? You
would hope that the discretized dynamical system and the
continuous dynamical system have the same properties in this
simple case, and this is known as linear stability analysis of
the method.

As an example, take a look at the Euler method. Recall that

the Euler method was given by:

un+1 = un + Δtf (un , p, t)

When we plug in the linear test equation, we get that

un+1 = un + Δtαun

If we let z = Δtα , then we get the following:

un+1 = un + zun = (1 + z)un

which is stable when z is in the shifted unit circle. This

means that, as a necessary condition, the step size Δt needs to
be small enough that satisfies this condition, placing a
be small enough that z satisfies this condition, placing a
stepsize limit on the method.

If Δt is ever too large, it will cause the equation to overshoot

zero, which then causes oscillations that spiral out to infinity.

Thus the stability condition places a hard constraint on the

allowed Δt which will result in a realistic simulation.

For reference, the stability regions of the 2nd and 4th order
Runge-Kutta methods that we discussed are as follows:
Interpretation of the Linear Stability Condition
To interpret the linear stability condition, recall that the
linearization of a system interprets the dynamics as locally
being due to the Jacobian of the system. Thus

′
u = f (u, p, t)

is locally equivalent to

′
df
u = u
du

You can understand the local behavior through diagonalizing

this matrix. Therefore, the scalar for the linear stability
analysis is performing an analysis on the eigenvalues of the
Jacobian. The method will be stable if the largest eigenvalues
of df/du are all within the stability limit. This means that
stability effects are different throughout the solution of a
nonlinear equation and are generally understood locally
(though different more comprehensive stability conditions
exist!).

Implicit Methods
If instead of the Euler method we defined f to be evaluated
at the future point, we would receive a method like:

un+1 = un + Δtf (un+1 , p, t + Δt)

in which case, for the stability calculation we would have

that

un+1 = un + Δtαun

or

(1 − z)un+1 = un

which means that

1
un+1 = un
1−z

which is stable for all Re(z) < 0 a property which is known

as A-stability. It is also stable as z → ∞ , a property known as
L-stability. This means that for equations with very ill-
L-stability. This means that for equations with very ill-
conditioned Jacobians, this method is still able to be use
reasonably large stepsizes and can thus be efficient.

Stiffness and Timescale Separation

From this we see that there is a maximal stepsize whenever
the eigenvalues of the Jacobian are sufficiently large. It turns
out that's not an issue if the phenomena we see are fast, since
then the total integration time tends to be small. However, if
we have some equations with both fast modes and slow
modes, like the Robertson equation, then it is very difficult
because in order to resolve the slow dynamics over a long
timespan, one needs to ensure that the fast dynamics do not
diverge. This is a property known as stiffness. Stiffness can
thus be approximated in some sense by the condition number
of the Jacobian. The condition number of a matrix is its
maximal eigenvalue divided by its minimal eigenvalue and
gives a rough measure of the local timescale separations. If
this value is large and one wants to resolve the slow
dynamics, then explicit integrators, like the explicit Runge-
Kutta methods described before, have issues with stability. In
this case implicit integrators (or other forms of stabilized
stepping) are required in order to efficiently reach the end
time step.

Exploiting Continuity
So far, we have looked at ordinary differential equations as a
Δt → 0 formulation of a discrete dynamical system.

However, continuous dynamics and discrete dynamics have

very different characteristics which can be utilized in order to
arrive at simpler models and faster computations.

Geometric Properties: No Jumping and the Poincaré–

Bendixson theorem
In terms of geometric properties, continuity places a large
constraint on the possible dynamics. This is because of the
physical constraint on "jumping", i.e. flows of differential
equations cannot jump over each other. If you are ever at
some point in phase space and f is not explicitly time-
dependent, then the direction of u is uniquely determined
′

(given reasonable assumptions on f ), meaning that flow lines

(solutions to the differential equation) can never cross.

A result from this is the Poincaré–Bendixson theorem,

which states that, with any arbitrary (but nice) two
dimensional continuous system, you can only have 3
behaviors:

Steady state behavior

Divergence
Periodic orbits

A simple proof by picture shows this.

Published from discretizing_odes.jmd using Weave.jl v0.10.9 on

2023-07-15.