April 2nd, 2020

Mathematics of
Neural ODEs
Vikram Voleti
PhD student - Mila, University of Montreal Prof. Christopher Pal
Visiting Researcher - University of Guelph Prof. Graham Taylor

voletiv.github.io @ (virtual) University of Guelph

 Contents

1. Ordinary Differential Equations (ODEs)

○ Initial Value Problems
○ Numerical integration methods
○ Fundamental theorem of ODEs

2. Neural ODEs
○ Adjoint method
○ Applications

3. Recent research

Vikram Voleti Mathematics of Neural ODEs 2 / 48

 1. Ordinary Differential Equations (ODEs)
○ Initial Value Problems
○ Numerical Integration methods
○ Fundamental theorem of ODEs

2. Neural ODEs

3. Recent research

Vikram Voleti Mathematics of Neural ODEs 3 / 48

 Ordinary Differential Equations (ODEs)
1st order Ordinary Differential Equation:

Vikram Voleti Mathematics of Neural ODEs 4 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Many physical processes follow this template!

Vikram Voleti Mathematics of Neural ODEs 5 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Vikram Voleti Mathematics of Neural ODEs 6 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Vikram Voleti Mathematics of Neural ODEs 6 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

What if this cannot be

analytically integrated?

Vikram Voleti Mathematics of Neural ODEs 7 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

What if this cannot be

analytically integrated?

Vikram Voleti Mathematics of Neural ODEs 7 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Approximations to
i.e. Numerical Integration :
● Euler method
● Runge-Kutta methods
● ...

Vikram Voleti Mathematics of Neural ODEs 8 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

1st-order Runge-Kutta / Euler’s method:

Step size
Update using derivative

https://guide.freecodecamp.org/mathematics/differential-equations/eulers-method/

Vikram Voleti Mathematics of Neural ODEs 9 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

1st-order Runge-Kutta / Euler’s method:

Vikram Voleti Mathematics of Neural ODEs 10 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Step size matters!

Solution:

1st-order Runge-Kutta / Euler’s method:

https://lpsa.swarthmore.edu/NumInt/NumIntFirst.html

Vikram Voleti Mathematics of Neural ODEs 11 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

1st-order Runge-Kutta / Euler’s method:

Vikram Voleti Mathematics of Neural ODEs 12 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

2nd-order Runge-Kutta method:

Vikram Voleti Mathematics of Neural ODEs 13 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

4th-order Runge-Kutta method:

Default ODE solver used in MATLAB:

https://blogs.mathworks.com/loren/2015/09/23/o
de-solver-selection-in-matlab/

Vikram Voleti Mathematics of Neural ODEs 14 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Many other ODE solvers to choose from!

Vikram Voleti Mathematics of Neural ODEs 15 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Many other ODE solvers to choose from!

Considerations to choose an ODE solver: https://blogs.mathworks.com/loren/2015/09/23/

ode-solver-selection-in-matlab/
● Stiff v/s Non-stiff ODE
https://math.temple.edu/~queisser/assets/files/T
● # of calculations per iteration alk3.pdf

● Implicit v/s Explicit solver

● Single-step size v/s Multi-step size (adaptive)

Vikram Voleti Mathematics of Neural ODEs 15 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Final time
Initial time
Initial value
Differential
Any ODE solver of our choice!

Vikram Voleti Mathematics of Neural ODEs 16 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Vikram Voleti Mathematics of Neural ODEs 17 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Fundamental Theorem of ODEs

Then, the solution to the initial value problem is unique!

http://faculty.bard.edu/belk/math213/InitialValueProblems.pdf

Vikram Voleti Mathematics of Neural ODEs 18 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Fundamental Theorem of ODEs

1. The solution curves for this differential equation completely fill the plane, and
2. Solution curves for different solutions do not intersect.

http://faculty.bard.edu/belk/math213/InitialValueProblems.pdf

Vikram Voleti Mathematics of Neural ODEs 18 / 48

 Ordinary Differential Equations (ODEs)
Initial value problem:

Solution:

Fundamental Theorem of ODEs

1. The solution curves for this differential equation completely fill the plane, and
2. Solution curves for different solutions do not intersect.

http://faculty.bard.edu/belk/math213/InitialValueProblems.pdf https://openreview.net/pdf?id=B1e9Y2NYvS

Vikram Voleti Mathematics of Neural ODEs 18 / 48

 1. Ordinary Differential Equations (ODEs)
○ Initial Value Problems
○ Numerical Integration methods
○ Fundamental theorem of ODEs

2. Neural ODEs (Chen et al., 2018)

○ Adjoint method
○ Applications

3. Recent research

Vikram Voleti Mathematics of Neural ODEs 19 / 48

 Neural ODEs (Chen et al., 2018)
Initial value problem:

Solution:

is a neural network!

Paradigm shift: whereas earlier f was pre-defined/hand-designed according to the domain,

here we would like to estimate an f that suits our objective.

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 20 / 48

 Neural ODEs (Chen et al., 2018)

ODEs
Residual
networks

Vector Euler discretization

notation

Skip connection

https://arxiv.org/pdf/1806.07366.pdf https://arxiv.org/pdf/1512.03385.pdf

Vikram Voleti Mathematics of Neural ODEs 21 / 48

 Neural ODEs (Chen et al., 2018)

ODEs
Residual
networks

Euler discretization

Skip connection
Forward propagation:

Stacked ResBlocks

https://arxiv.org/pdf/1806.07366.pdf https://arxiv.org/pdf/1512.03385.pdf

Vikram Voleti Mathematics of Neural ODEs 21 / 48

 Neural ODEs (Chen et al., 2018)

ODEs
Residual
networks

Euler discretization

Skip connection
Forward propagation:

Stacked ResBlocks
How to compute this?

Update to reduce . Update to reduce .

https://arxiv.org/pdf/1806.07366.pdf https://arxiv.org/pdf/1512.03385.pdf

Vikram Voleti Mathematics of Neural ODEs 21 / 48

 Neural ODEs (Chen et al., 2018)

ODEs
Back-propagate through the
ODE Solver!

Euler discretization

Forward propagation:

Update to reduce .

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 22 / 48

 Neural ODEs (Chen et al., 2018)

ODEs
Back-propagate through the
ODE Solver!

Euler discretization High memory cost -

need to save all activations of all

iterations of ODESolve.

Forward propagation:

Can we do better?
Yes.
Update to reduce .

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 22 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 23 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

We need a(t)

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 23 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 24 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

Can be computed using autodiff

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 24 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

Can be computed using autodiff

We can use a(t1) as initial value,

and integrate backwards from t1 to t to get a(t).

We’ll use t0 as a proxy for t

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 24 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

Initial value

Backward integration from t1 to t0

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 25 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

Initial value

Vector-Jacobian Product
(can be efficiently evaluated by autodiff)
https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 25 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

But we need x(t)

(and we don’t want to have saved x(t)

in memory from forward-prop)

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 25 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:
Initial value

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 26 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

Back-propagation:

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 27 / 48

 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

Back-propagation:

https://arxiv.org/pdf/1806.07366.pdf Initial value is 0

Vikram Voleti Mathematics of Neural ODEs 27 / 48
 Neural ODEs (Chen et al., 2018)

Adjoint method (Pontryagin et al., 1962)

Forward propagation:

Back-propagation:

Combine the 3 ODE Solves into 1!

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 28 / 48

 Neural ODEs (Chen et al., 2018)

Forward propagation:

Back-propagation:

Update to reduce .

https://arxiv.org/pdf/1806.07366.pdf

Vikram Voleti Mathematics of Neural ODEs 29 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Neural ODE

Vikram Voleti Mathematics of Neural ODEs 30 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Neural ODE

Neural ODEs describe a

homeomorphism (flow).
● They preserve dimensionality.
● They form non-intersecting
trajectories.

https://openreview.net/pdf?id=B1e9Y2NYvS

Vikram Voleti Mathematics of Neural ODEs 30 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Neural ODE

Neural ODE

Neural ODEs are reversible models!

Just integrate forward/backward in time.

Vikram Voleti Mathematics of Neural ODEs 31 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Vikram Voleti Mathematics of Neural ODEs 32 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

ODE-Net:

L
i
n
e
a
Neural ODE r

~ Replacement for ResNets

Vikram Voleti Mathematics of Neural ODEs 33 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Target distribution Noise distribution

Neural ODE
(such as
real image manifold)

Sample from Sample from

target distribution noise distribution
(such as an image) (such as Gaussian)

Vikram Voleti Mathematics of Neural ODEs 23 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Target distribution Noise distribution

Neural ODE

Likelihood estimation
using Change of Variables formula

Train f to maximize the likelihood of the samples from target distribution log p(x_1),
by computing x(t_0) using the Neural ODE with x(t_1) as the initial value, and the Change of Variables formula.

Vikram Voleti Mathematics of Neural ODEs 35 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Target distribution Noise distribution

Neural ODE

Likelihood estimation
using Change of Variables formula

Generate samples
Sample from the noise distribution, transform it into a sample from the target distribution
using the trained Neural ODE.

Vikram Voleti Mathematics of Neural ODEs 25 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Encode the
observed data

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Encode into a latent distribution (such

as Gaussian)

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Sample from the

latent distribution

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Use the sample as

initial value to a Neural ODE

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Solve the Neural ODE for

latent points in later time steps

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models

Decode the
latent points

Compute loss

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 Neural ODEs (Chen et al., 2018)
https://arxiv.org/pdf/1806.07366.pdf

Applications
Supervised Learning Continuous Normalizing Flows Generative Latent Models
Solve the Neural ODE for
latent points in future time steps

Decode the
latent points

Future generation!

Vikram Voleti Mathematics of Neural ODEs 37 / 48

 1. Ordinary Differential Equations (ODEs)
○ Initial Value Problems
○ Numerical Integration methods
○ Fundamental theorem of ODEs

2. Neural ODEs (Chen et al., 2018)

○ Adjoint method
○ Applications

3. Recent research

Vikram Voleti Mathematics of Neural ODEs 38 / 48

 Recent research
FFJORD: Free-form Continuous Dynamics For Scalable Reversible
Generative Models (Grathwohl et al., ICLR 2019)

● Essentially a better Continuous Normalizing Flow.

● Makes a better estimate for the log determinant term.

● “We demonstrate our approach on high-dimensional

density estimation, image generation, and variational
inference, achieving the state-of-the-art among exact
likelihood methods with efficient sampling.”

https://arxiv.org/pdf/1810.01367.pdf

Vikram Voleti Mathematics of Neural ODEs 39 / 48

 Recent research
Augmented Neural ODEs (Dupont et al., NeurIPS 2019)

● Shows that Neural ODEs cannot model non-homeomorphisms (non-flows)

● Augments the state with additional dimensions to cover non-homeomorphisms

● Performs ablation study on toy examples and image classification

https://arxiv.org/pdf/1904.01681.pdf

Vikram Voleti Mathematics of Neural ODEs 40 / 48

 Recent research
ANODEV2: A Coupled Neural ODE Evolution Framework
(Zhang et al., NeurIPS 2019)

● Network weights are also a function of time

● Separate “weight network” generates the weights of the function network

at a given time

https://arxiv.org/pdf/1906.04596.pdf

Vikram Voleti Mathematics of Neural ODEs 41 / 48

 Recent research
Latent ODEs for Irregularly-Sampled Time Series
(Rubanova et al., NeurIPS 2019)

● Improves the generative latent variable framework for irregularly-sampled time series

● Essentially uses an ODE in the encoder where samples are missing

● Shows results on toy data, MuJoCo, PhysioNet

https://arxiv.org/pdf/1907.03907.pdf

Vikram Voleti Mathematics of Neural ODEs 42 / 48

 Recent research
Simple Video Generation using Neural ODEs
(David Kanaa*, Vikram Voleti*, Samira Kahou, Christopher Pal; NeurIPS 2019 Workshop)

● Video generation as a generative latent

variable model using Neural ODEs

https://sites.google.com/view/neurips2019lire/accepted-papers?authuser=0

Vikram Voleti Mathematics of Neural ODEs 43 / 48

 Recent research
ODE2VAE: Deep generative second order ODEs with Bayesian neural
networks (Yildiz et al., NeurIPS 2019)

● Uses 2nd-order Neural ODE

● Uses a Bayesian Neural Network

● Showed results modelling video generation

as a generative latent variable model using
(2nd-order Bayesian) Neural ODE

https://papers.nips.cc/paper/9497-ode2vae-deep-generative-second-order-odes-with-bayesian-neural-networks.pdf

Vikram Voleti Mathematics of Neural ODEs 44 / 48

 Recent research
On Robustness of Neural Ordinary Differential Equations
(Yan et al., ICLR 2020)

● Ablation study on adversarial attacks on

ODE-Nets

● Introduces new regularization term to

improve robustness

https://arxiv.org/pdf/1910.05513.pdf, https://openreview.net/pdf?id=B1e9Y2NYvS

Vikram Voleti Mathematics of Neural ODEs 45 / 48

 Recent research
How to Train Your Neural ODE (Finlay et al., 2020)

● Makes a link between the flow in Neural

ODEs and optimal transport

● Introduces two new regularization terms to

constrain flows to straight lines

● Speeds up training of Neural ODEs

https://arxiv.org/pdf/2002.02798.pdf

Vikram Voleti Mathematics of Neural ODEs 46 / 48

 Additional References
● http://faculty.bard.edu/belk/math213/InitialValueProblems.pdf

● https://math.temple.edu/~queisser/assets/files/Talk3.pdf

● Textbook : https://users.math.msu.edu/users/gnagy/teaching/ode.pdf

● https://lpsa.swarthmore.edu/NumInt/NumIntFirst.html

● http://homepages.cae.wisc.edu/~blanchar/eps/ivp/ivp

● Excellent blog post on ODE solvers: https://blogs.mathworks.com/loren/2015/09/23/ode-solver-selection-in-matlab/

● Autodiff tutorial:
http://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/readings/L06%20Automatic%20Differentiation.pdf

● Course on Neural Networks & Deep Learning by Roger Grosse & Jimmy Ba, University of Toronto -
http://www.cs.toronto.edu/~rgrosse/courses/csc421_2019/

Vikram Voleti Mathematics of Neural ODEs 47 / 48

 Thank you!

voletiv.github.io