Introduction to

Reservoir Computing

Nov. 22, 2013

Helmut Hauser
Artificial Intelligence Laboratory

university of Zurich
Overview
What are we talking about?

input Computation output

...

...
What are we talking about?

input Computation output

...

...
input Mapping output
...

...
What are we talking about?

Computation
...

...
Mapping
...

...
What are we talking about?

Computation
...

...
Mapping
...

...
What are we talking about?

Computation
...

...
Mapping
...

...
What are we talking about?

Computation
...

...
emulation

Mapping
...

...
What are we talking about?

Computation
...

...
emulation

Mapping
...

...
• We want to learn this mapping with machine learning techniques
What are we talking about?

Computation
...

...
emulation

Mapping
...

...
• We want to learn this mapping with machine learning techniques
• Different levels of complexity for the mapping are possible
 Simplest Mapping:
Look-up table

input output
...

...
 Simplest Mapping:
Look-up table

input output
...

...
Pro: Simple, fast, non-smooth mappings are possible
 Simplest Mapping:
Look-up table

input output
...

...
Pro: Simple, fast, non-smooth mappings are possible
Con: You have to know every single point of mapping
(no generalization properties), no memory
Artificial Neural Networks

input output
...

...
...
Artificial Neural Networks
hidden layer

input output
...

...
...
Artificial Neural Networks
hidden layer

input output
...

...
...

Input node
Artificial Neural Networks
hidden layer

input output
...

...
...

Input node
Node: summation of input plus nonlinearity
Artificial Neural Networks
hidden layer

input output
...

...
...

Input node
Node: summation of input plus nonlinearity
Weight of “importance”
Artificial Neural Networks
hidden layer

input output
...

...
...

Input node
Node: summation of input plus nonlinearity
Weight of “importance”
Output node (linear or nonlinear)
Artificial Neural Networks
hidden layer

input output
...

...
...
Artificial Neural Networks
hidden layer

input output
...

...
...

• Universal function approximator (Hornik et al. 1989)

Artificial Neural Networks
hidden layer

input output
...

...
...

• Universal function approximator (Hornik et al. 1989)

• Various learning algorithms available (back-propagation)
Artificial Neural Networks
hidden layer

input output
...

...
...

• Universal function approximator (Hornik et al. 1989)

• Various learning algorithms available (back-propagation)
• Guaranteed convergence
Artificial Neural Networks
hidden layer

input output
...

...
...
Artificial Neural Networks
hidden layer

input output
...

...
...

• Feedforward structure
Artificial Neural Networks
hidden layer

input output
...

...
...

• Feedforward structure
• Static (no memory) - mapping point to point
Artificial Neural Networks
hidden layer

input output
...

...
...

• Feedforward structure
• Static (no memory) - mapping point to point
• Nonlinear with generalization property
Artificial Neural Networks
hidden layer

input output
...

...
...

• Feedforward structure
• Static (no memory) - mapping point to point
• Nonlinear with generalization property
Memory is important in a lot
of cases

time
Memory is important in a lot
of cases

discrete case

input
time
Memory is important in a lot
of cases

X
output
discrete case

...

input
time
Memory is important in a lot
of cases

X
output
discrete case

...

input
time
history
Memory is important in a lot
of cases

X
output
discrete case

...

input
time
history

memory is needed
Memory is important in a lot
of cases
discrete case

...
delay line

time
history

memory is needed
Memory is important in a lot
of cases
discrete case

...
delay line

time
history

memory is needed
Memory is important in a lot
of cases
Z 0

continuous case 1
output

input
time
history

memory is needed
Memory is important in a lot
of cases
Memory is important in a lot
of cases

• Emulate dynamical systems (controller)

Memory is important in a lot
of cases

• Emulate dynamical systems (controller)

• Audio/Speech processing
Memory is important in a lot
of cases

• Emulate dynamical systems (controller)

• Audio/Speech processing
• Time Series (financial data, weather, etc.)
Memory is important in a lot
of cases

• Emulate dynamical systems (controller)

• Audio/Speech processing
• Time Series (financial data, weather, etc.)
• Robot control
Memory is important in a lot
of cases

• Emulate dynamical systems (controller)

• Audio/Speech processing
• Time Series (financial data, weather, etc.)
• Robot control
• Understanding interaction (context)
Memory is important in a lot
of cases

• Emulate dynamical systems (controller)

• Audio/Speech processing
• Time Series (financial data, weather, etc.)
• Robot control
• Understanding interaction (context)
• many other ...
Recurrent Artificial Neural
Networks
Recurrent Artificial Neural
Networks
Feedback
Recurrent Artificial Neural
Networks
Feedback
Recurrent Artificial Neural
Networks
Feedback

• Integration of over time (discrete/continuous)

Recurrent Artificial Neural
Networks
Feedback

• Integration of over time (discrete/continuous)

• We get a dynamical system
Recurrent Artificial Neural
Networks
Feedback

• Integration of over time (discrete/continuous)

• We get a dynamical system
• What structure should be used?
 Different RNNs

• Elman networks

http://en.wikipedia.org/wiki/File:Elman_srnn.png
 Different RNNs

• Elman networks

• Hopfield

http://commons.wikimedia.org/wiki/File:Hopfield-net.png
 Different RNNs

• Elman networks

• Hopfield

• Anything is possible...

http://edizquierdo.wordpress.com/2008/10/02/bacterial-chemotaxis/
 Different RNNs

• Elman networks

• Hopfield

• Anything is possible...

• Reflecting the problem

http://edizquierdo.wordpress.com/2008/10/02/bacterial-chemotaxis/
learning Rules for RNN
Learning is not trivial anymore !
learning Rules for RNN
Learning is not trivial anymore !

• Back Propagation Trough Time (BPTT)

learning Rules for RNN
Learning is not trivial anymore !

• Back Propagation Trough Time (BPTT)

• Real Time Recurrent Learning (RTRL)
learning Rules for RNN
Learning is not trivial anymore !

• Back Propagation Trough Time (BPTT)

• Real Time Recurrent Learning (RTRL)
• Global optimization approach like GA
and SA
learning Rules for RNN
Learning is not trivial anymore !

• Back Propagation Trough Time (BPTT)

• Real Time Recurrent Learning (RTRL)
• Global optimization approach like GA
and SA
learning Rules for RNN
Learning is not trivial anymore !

• Back Propagation Trough Time (BPTT)

• Real Time Recurrent Learning (RTRL)
• Global optimization approach like GA
and SA

• No guaranteed convergence to global minimum

learning Rules for RNN
Learning is not trivial anymore !

• Back Propagation Trough Time (BPTT)

• Real Time Recurrent Learning (RTRL)
• Global optimization approach like GA
and SA

• No guaranteed convergence to global minimum

• Slow learning
learning Rules for RNN
Learning is not trivial anymore !

• Back Propagation Trough Time (BPTT)

• Real Time Recurrent Learning (RTRL)
• Global optimization approach like GA
and SA

• No guaranteed convergence to global minimum

• Slow learning
• Can get stuck in local minima
Solution Reservoir
Computing Approach
Solution Reservoir
Computing Approach

• Randomly initialized recurrent network

Solution Reservoir
Computing Approach

• Randomly initialized recurrent network

• Random dynamic parameters and connections
Solution Reservoir
Computing Approach

• Randomly initialized recurrent network

• Random dynamic parameters and connections
• Complex, high-dimensional, nonlinear dynamic
system
Solution Reservoir
Computing Approach
reservoir

• Randomly initialized recurrent network

• Random dynamic parameters and connections
• Complex, high-dimensional, nonlinear dynamic
system
Solution Reservoir
Computing Approach
reservoir

fixed during
learning

• Randomly initialized recurrent network

• Random dynamic parameters and connections
• Complex, high-dimensional, nonlinear dynamic
system
 Solution Reservoir
Computing Approach
reservoir s1

input s2

sN
 Solution Reservoir
Computing Approach
reservoir s1

input s2

sN

• Input excites the dynamical system

 Solution Reservoir
Computing Approach
reservoir s1

input s2

sN

• Input excites the dynamical system

• Get complex, high-dimensional, nonlinear response
 Solution Reservoir
Computing Approach
reservoir s1

input s2

sN

• Input excites the dynamical system

• Get complex, high-dimensional, nonlinear response
• Has to be exponentially stable (fading memory)
 Solution Reservoir
Computing Approach
reservoir
s1
X
input output

...
sN
 Solution Reservoir
Computing Approach
N
X
reservoir u(t) = b0 + wi · si (t)
i=1
s1
X
input output

...
sN
 Solution Reservoir
Computing Approach
N
X
reservoir u(t) = b0 + wi · si (t)
i=1
s1
X
input output

...
sN

• Linear combination of the (partial) state of the system

• Weights wi can be learned with learning regression
• Only the weights wi are adapted during learning
 Solution Reservoir
Computing Approach
N
X
reservoir u(t) = b0 + wi · si (t)
i=1
s1
X
input output

...
sN

• Linear combination of the (partial) state of the system

• Weights wi can be learned with learning regression
• Only the weights wi are adapted during learning
Learning Setup

... X

...
 Learning Setup
one or
multiple

... X

...
Learning Setup

... X

...
Learning Setup

... X

...
randomly initialized
Learning Setup
X
u1
X
... u2

...

...
X
uK
randomly initialized
Learning Setup
X
u1
X
... u2

...

...
X
uK
randomly initialized

multitasking
Learning Setup
X
u1
X
... u2

...

...
X
uK
randomly initialized
Learning Setup

...
Learning Setup

...
mapping that we want
to emulate
(Black Box)
Learning Setup

...
mapping that we want
to emulate
(Black Box)
Learning Setup

...
target output

mapping that we want

to emulate
(Black Box)
Learning Setup

...
target output

emulation

mapping that we want

to emulate
(Black Box)
Learning Setup
supervised learning setup

...
target output

emulation

mapping that we want

to emulate
(Black Box)
Learning Setup
supervised learning setup
 Learning Setup
supervised learning setup

run system with

input
 Learning Setup
supervised learning setup collect data

s1

s2

run system with

...
input
sN
 Learning Setup
supervised learning setup collect data

s1

s2

run system with

...
input
sN

matrix
N number of signals

number of time steps

 Learning Setup
supervised learning setup collect data

s1

s2

run system with

...
input
sN

matrix
N number of signals

number of time steps

Learning Setup
supervised learning setup

...
target output
 Learning Setup
supervised learning setup

...
target output

matrix

target output
 Learning Setup
supervised learning setup

...
target output

matrix
linear optimal
regression weights
wi
target output
 Learning Setup
supervised learning setup

...
target output

matrix
linear optimal
regression weights
wi
target output
minimize the quadratic error
Learning Setup for
Feedback
supervised learning setup

...
target output

computation we want
to emulate
(Black Box)
Learning Setup for
Feedback
supervised learning setup

...
target output

unstable

computation we want
to emulate
(Black Box)
Learning Setup for
Feedback collect all data points
over time

...
target output
teacher forcing
Learning Setup for
Feedback collect all data points
noise
over time

...
target output
noise is crucial for
robustness!
 Learning Setup for
Feedback
noise

...
target output
matrix
linear optimal
regression weights
wi
target output
minimize the quadratic error
Learning Setup for
Feedback

...
Learning Setup for
Feedback

...
close loop,
let system run freely
Theories of Reservoir
Computing
 Theories of Reservoir
Computing

X
feedforward

...
 Theories of Reservoir
Computing

X
feedforward

...
X
feedback

...
 Theories of Reservoir
Computing

X
feedforward

...
two different theories

X
feedback

...
Mathematical Formulation
X
reservoir

...
Mathematical Formulation
X
reservoir

...
Mathematical Formulation
X
reservoir mathematical operator

...
Mathematical Formulation
X
reservoir mathematical operator

...

input mapping output

functions function
Mathematical Formulation
X
reservoir mathematical operator

...

input mapping output

functions function

input mathematical output

...

...
streams operators streams
Mathematical Formulation
X
reservoir mathematical operator

...

input mapping output

functions function

input mathematical output

...

...
streams operators streams

encode our computations

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Hauser, Helmut, Auke J. Ijspeert, Rudolf M. Füchslin, Rolf Pfeifer, and Wolfgang Maass. "Towards a
theoretical foundation for morphological computation with compliant bodies" Biological Cybernetics
105, no. 5-6 (2011): 355-370.
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

Hauser, Helmut, Auke J. Ijspeert, Rudolf M. Füchslin, Rolf Pfeifer, and Wolfgang Maass. "Towards a
theoretical foundation for morphological computation with compliant bodies" Biological Cybernetics
105, no. 5-6 (2011): 355-370.
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

temporal nonlinear
integration
• Encoding combination
our computational task
• Nonlinear, dynamic
B1 operator
B2 f
• Any exp. stable nonlinear dynamic system with one point
...

...

of equilibrium
Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

temporal nonlinear
integration
• Encoding combination
our computational task
• Nonlinear, dynamic
B1 operator
B2 f
• Any exp. stable nonlinear dynamic system with one point
...

...

of equilibrium
Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

temporal nonlinear nonlinear with

integration
• Encoding combinationmemory
our computational task
• Nonlinear, dynamic
B1 operator
B2 f
• Any exp. stable nonlinear dynamic system with one point
...

...

of equilibrium
Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

temporal nonlinear nonlinear with

integration
• Encoding combinationmemory
our computational task
• Nonlinear, dynamic
B1 operator
B2 f
• Any exp. stable nonlinear dynamic system with one point
...

...

of equilibrium
Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

temporal nonlinear nonlinear with

integration
• Encoding combinationmemory
our computational task
• Nonlinear, dynamic
B1 operator
B2 f
• Any exp. stable nonlinear dynamic system with one point
...

...

of equilibrium
Bk
nonlinear controller
linear, dynamic nonlinear, static
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

memory and nonlinearity

Hauser, Helmut, Auke J. Ijspeert, Rudolf M. Füchslin, Rolf Pfeifer, and Wolfgang Maass. "Towards a
theoretical foundation for morphological computation with compliant bodies" Biological Cybernetics
105, no. 5-6 (2011): 355-370.
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

memory and nonlinearity

memory
1

Hauser, Helmut, Auke J. Ijspeert, Rudolf M. Füchslin, Rolf Pfeifer, and Wolfgang Maass. "Towards a
theoretical foundation for morphological computation with compliant bodies" Biological Cybernetics
105, no. 5-6 (2011): 355-370.
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory

memory and nonlinearity

memory nonlinearity
1 2

Hauser, Helmut, Auke J. Ijspeert, Rudolf M. Füchslin, Rolf Pfeifer, and Wolfgang Maass. "Towards a
theoretical foundation for morphological computation with compliant bodies" Biological Cybernetics
105, no. 5-6 (2011): 355-370.
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory can

be uniformly approximated by computational devices, which
consist of two simple stages:

1 2
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory can

be uniformly approximated by computational devices, which
consist of two simple stages:

temporal
integration

1 2

linear, dynamic
First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory can

be uniformly approximated by computational devices, which
consist of two simple stages:

temporal nonlinear
integration combination

1 2

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operators with fading memory can

be uniformly approximated by computational devices, which
consist of two simple stages:

temporal nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Stage 1
• Arbitrary time invariant operators with fading memory can
be uniformly approximated
Has to integrate by computational devices, which
information
consist of time
over two (fading
simple memory)
stages:

Has totemporal
separate signals nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Stage 1
• Arbitrary time invariant operators with fading memory can
be uniformly approximated
Has to integrate by computational devices, which
information
consist of time
over two (fading
simple memory)
stages:

Has totemporal
separate signals nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Stage 1
• Arbitrary time invariant operators with fading memory can
be uniformly approximated
Has to integrate by computational devices, which
information
consist of time
over two (fading
simple memory)
stages:

Has totemporal
separate signals nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Stage 1
• Arbitrary time invariant operators with fading memory can
be uniformly approximated
Has to integrate by computational devices, which
information
consist of time
over two (fading
simple memory)
stages:

Has totemporal
separate signals nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Stage 1
• Arbitrary time invariant operators with fading memory can
be uniformly approximated
Has to integrate by computational devices, which
information
consist of time
over two (fading
simple memory)
stages: B1

B2

Has totemporal
separate signals nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Stage 1
• Arbitrary time invariant operators with fading memory can
be uniformly approximated
Has to integrate by computational devices, which
information
consist of time
over two (fading
simple memory)
stages: B1

B2

Has totemporal
separate signals nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

Stage 1
• Arbitrary time invariant operators with fading memory can
be uniformly approximated
Has to integrate by computational devices, which
information
consist of time
over two (fading
simple memory)
stages: B1

B2

Has totemporal
separate signals nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

• Based on a result by [Boyd and Chua 1985]

• Arbitrary time invariant operatorsStage

with2fading memory can
be uniformly approximated by computational devices, which
Has to combine integrated
consist of two simple stages:
information nonlinearily
temporal nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

nonlinear, dynamic
operator

temporal nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

nonlinear, dynamic
operator

temporal nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

nonlinear, dynamic
operator

temporal nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

First Theoretical Model

nonlinear, dynamic
operator

emulation

temporal nonlinear
integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

 First Theoretical Model

nonlinear, dynamic
operator

linear dynamic emulation

systems
ẋ = A(x) + bu temporal nonlinear
y = C(x) integration combination

B1
B2 f
...

...
...

Bk

linear, dynamic nonlinear, static

 First Theoretical Model

nonlinear, dynamic
operator

linear dynamic emulation

systems
ẋ = A(x) + bu temporal nonlinear
y = C(x) integration combination

ẋ = A1 (x) + b1 u
ẋ = A2 (x) + b2 u f
...

...
...

ẋ = Ak (x) + bk u

linear, dynamic nonlinear, static

First Theoretical Model

nonlinear, dynamic
operator

emulation

temporal nonlinear
integration combination

ẋ = A1 (x) + b1 u
ẋ = A2 (x) + b2 u f
...

...
...

ẋ = Ak (x) + bk u

linear, dynamic nonlinear, static

First Theoretical Model

nonlinear, dynamic
operator

emulation

temporal nonlinear
integration combination

ẋ = A1 (x) + b1 u
ẋ = A2 (x) + b2 u f
...

...

ANN
...

ẋ = Ak (x) + bk u

linear, dynamic nonlinear, static

NOT A RESERVOIR
SETUP YET
temporal nonlinear
integration combination
ẋ = A1 (x) + b1 u
ẋ = A2 (x) + b2 u f

...
ANN
...
...

ẋ = Ak (x) + bk u
 NOT A RESERVOIR
SETUP YET
temporal nonlinear
integration combination
ẋ = A1 (x) + b1 u
ẋ = A2 (x) + b2 u f

...
ANN
...
...

ẋ = Ak (x) + bk u

• Nonlinear, static mapping could also be done by ”kernel”

 NOT A RESERVOIR
SETUP YET
temporal nonlinear
integration combination
ẋ = A1 (x) + b1 u
ẋ = A2 (x) + b2 u f

...
ANN
...
...

ẋ = Ak (x) + bk u

• Nonlinear, static mapping could also be done by ”kernel”

• Idea: complex, nonlinear high-dimensional dynamic system could
be seen as an implementation of such a finite kernel
 NOT A RESERVOIR
SETUP YET
temporal nonlinear
integration combination
ẋ = A1 (x) + b1 u
ẋ = A2 (x) + b2 u f

...
ANN
...
...

ẋ = Ak (x) + bk u

• Nonlinear, static mapping could also be done by ”kernel”

• Idea: complex, nonlinear high-dimensional dynamic system could
be seen as an implementation of such a finite kernel
• Combining both stages - we get a reservoir
 Reservoir Setup
temporal nonlinear
integration + combination

X
...

...
• Nonlinear, static mapping could also be done by ”kernel”
• Idea: complex, nonlinear high-dimensional dynamic system could
be seen as an implementation of such a finite kernel
• Combining both stages - we get a reservoir
Reservoir Setup
temporal nonlinear
integration + combination

X
...

...
 Reservoir Setup
temporal nonlinear
integration + combination

X
...

...
nonlinearities, memory
are part of the reservoir
 Reservoir Setup
temporal nonlinear
integration + combination

X
...

...
nonlinearities, memory simple, static, linear
are part of the reservoir readout - linear regression
 Reservoir Setup
nonlinear, dynamic
operator

X
...

...
nonlinearities, memory simple, static, linear
are part of the reservoir readout - linear regression
 Reservoir Setup
nonlinear, dynamic
operator

X
...

...
nonlinearities, memory simple, static, linear
are part of the reservoir readout - linear regression
Desired Properties
temporal nonlinear
integration + combination

X
...

...
Desired Properties
temporal nonlinear
integration + combination
kernel property

X
...

...
 Desired Properties
temporal nonlinear
integration + combination
kernel property

X
...

...
The reservoir should be:
 Desired Properties
temporal nonlinear
integration + combination
kernel property

X
...

...
The reservoir should be:

• Nonlinear
 Desired Properties
temporal nonlinear
integration + combination
kernel property

X
...

...
The reservoir should be:

• Nonlinear
• High-dimensional
 Desired Properties
temporal nonlinear
integration + combination
kernel property

X
...

...
The reservoir should be:

• Nonlinear
• High-dimensional
• Dynamic (fading memory)
 Desired Properties
temporal nonlinear
integration + combination
kernel property

X
...

...
The reservoir should be: ...but otherwise
completely generic
• Nonlinear
• High-dimensional
• Dynamic (fading memory)
 Desired Properties
temporal nonlinear
integration + combination
kernel property

X
...

...
The reservoir should be: ...but otherwise
completely generic
• Nonlinear
• High-dimensional
• Dynamic (fading memory) different flavors of RC
Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
Neuron (model)

Synaptic connection

Spike trains
Connection based on
biological data
Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
Neuron (model)
Liquid State
Synaptic connection
Machine
Spike trains
Maass et al. 2002
Connection based on
biological data
Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
Simple diff. equation (nonlinearity)
weighted connection

Full connectivity
Pure machine learning
technique
Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
Simple diff. equation (nonlinearity)
weighted connection Echo State
Network
Full connectivity
Pure machine learning Jaeger 2002
technique
Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
bucket of water
 Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
bucket of water

• Water is mechanically perturbed (with motors)

 Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
bucket of water

• Water is mechanically perturbed (with motors)

• Complex response of the surface
 Different Flavors of RC
temporal nonlinear
integration + combination

X
...

...
bucket of water

• Water is mechanically perturbed (with motors)

• Complex response of the surface
• Readout is digitized picture frame + processing (vision)

Fernando and Sojakka, 2003

Different Flavors of RC
temporal nonlinear
integration + combination

Nonlinear optical X
...

effects of laser

...
Different Flavors of RC
temporal nonlinear
integration + combination

Nonlinear optical X
...

effects of laser

...
• Nonlinear optical effects are exploited
Different Flavors of RC
temporal nonlinear
integration + combination

Nonlinear optical X
...

effects of laser

...
• Nonlinear optical effects are exploited
• Calculating at the speed of light
Different Flavors of RC
temporal nonlinear
integration + combination

Nonlinear optical X
...

effects of laser

...
• Nonlinear optical effects are exploited
• Calculating at the speed of light
• Generic computational device
Different Flavors of RC
temporal nonlinear
integration + combination

Nonlinear optical X
...

effects of laser

...
• Nonlinear optical effects are exploited
Photonic RC
• Calculating at the speed of light
• Generic computational device
Massar, Dambri, Schrauwen
Different Flavors of RC
e.g., physical body of a robot

temporal nonlinear
integration + combination

X
...

...
Different Flavors of RC
e.g., physical body of a robot

temporal nonlinear
integration + combination

X
...

...
masses (scaling)
nonlinear springs

only local connectivity

Morphological computation setup
Different Flavors of RC
e.g., physical body of a robot

temporal nonlinear
integration + combination

X
...

...
masses (scaling)
nonlinear springs physical reservoir
computing
only local connectivity
Morphological computation setup Hauser et al. 2012
Different Flavors of RC
e.g., physical body of a robot

temporal nonlinear
integration + combination

X
...

...
masses (scaling)
nonlinear springs physical reservoir
computing
only local connectivity
Morphological computation setup Hauser et al. 2012
Remarkable Conclusion
nonlinear, dynamic
operator
temporal nonlinear
integration combination

X
...

...
Remarkable Conclusion
nonlinear, dynamic
operator
temporal nonlinear
integration combination

X
...

...
Outsourcing big part of the
computation to the
morphology
Remarkable Conclusion
nonlinear, dynamic
operator
temporal nonlinear
integration combination

X
...

...
Outsourcing big part of the resulting task is
computation to the easier:
morphology linear regression
Remarkable Conclusion
nonlinear, dynamic
operator
temporal nonlinear
integration combination

X
...

...
Outsourcing big part of the resulting task is
computation to the easier:
morphology linear regression

Concept of Morphological Computation

Remarkable Conclusion

temporal nonlinear
integration combination

X
...

...
Concept of Morphological Computation
Remarkable Conclusion
randomly
initialized! temporal nonlinear
integration combination

X
...

...
Concept of Morphological Computation
Remarkable Conclusion
randomly
initialized! temporal nonlinear
integration combination

l
na
sio
en
im X
...

-d

...
gh
hi

Concept of Morphological Computation

Remarkable Conclusion
randomly
initialized! temporal nonlinear
integration combination

l
na
sio
en

r
ea
im X

in
...

-d

...
nl
gh

no
hi

Concept of Morphological Computation

Remarkable Conclusion
randomly
initialized! temporal nonlinear
integration combination

l
na
sio
en

nt
ea
im X

ia
in
...

pl
-d

...
nl

m
gh

no

co
hi

Concept of Morphological Computation

 Remarkable Conclusion

high-dimensional

nonlinear

compliant

Concept of Morphological Computation

 Remarkable Conclusion

I don’t like that!

high-dimensional

nonlinear

compliant

Concept of Morphological Computation

 Remarkable Conclusion

I don’t like that!

high-dimensional

nonlinear

compliant

Actually, I do like
that!

Concept of Morphological Computation

 Limitation

temporal nonlinear
integration + combination

X
...

...
 Limitation

temporal nonlinear
integration + combination

X
...

...
• The theoretical model is limited to time-invariant operators with
fading memory
 Limitation

temporal nonlinear
integration + combination

X
...

...
• The theoretical model is limited to time-invariant operators with
fading memory
• Persistent memory is of interest too, or limit cycles
 Limitation

temporal nonlinear
integration + combination

X
...

...
• The theoretical model is limited to time-invariant operators with
fading memory
• Persistent memory is of interest too, or limit cycles
• Another theory is needed!
 Limitation

temporal nonlinear
integration + combination

X
...

...
• The theoretical model is limited to time-invariant operators with
fading memory
• Persistent memory is of interest too, or limit cycles
• Another theory is needed!
• based on feedback linearization from control theory
Second Theoretical Model
Second Theoretical Model
• Based on the concept of “feedback linearization” from control theory
• Maass et al. 2007 applied to generic neural networks, Hauser et al. to
physical bodies
 Second Theoretical Model
• Based on the concept of “feedback linearization” from control theory
• Maass et al. 2007 applied to generic neural networks, Hauser et al. to
physical bodies

• Fixed dynamical system

v(t)
x0 (t) = f (x(t)) + g(x(t)) · v(t)
• feedback linearizable
• e.g., nonlinear mass-
spring system
 Second Theoretical Model
• Based on the concept of “feedback linearization” from control theory
• Maass et al. 2007 applied to generic neural networks, Hauser et al. to
physical bodies

• Fixed dynamical system

v(t)
x0 (t) = f (x(t)) + g(x(t)) · v(t)
• feedback linearizable
• e.g., nonlinear mass-
spring system

u(t) z(t) • Desired computation

z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)
) + u(t) encoded as a nonlinear
dynamical system
 Second Theoretical Model
• Based on the concept of “feedback linearization” from control theory
• Maass et al. 2007 applied to generic neural networks, Hauser et al. to
physical bodies

• Fixed dynamical system

v(t)
x0 (t) = f (x(t)) + g(x(t)) · v(t)
• feedback linearizable
• e.g., nonlinear mass-
spring system

u(t) z(t)
z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)
) + u(t)
 Second Theoretical Model
• Based on the concept of “feedback linearization” from control theory
• Maass et al. 2007 applied to generic neural networks, Hauser et al. to
physical bodies

• Fixed dynamical system

v(t)
x0 (t) = f (x(t)) + g(x(t)) · v(t)
• feedback linearizable
• e.g., nonlinear mass-
spring system

u(t) z(t)
z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)
) + u(t)

Desired computation: very powerful description!

 Second Theoretical Model
z(t)
v(t) h(x(t))
0
x (t) = f (x(t)) + g(x(t)) · v(t)

K(x(t), u(t))
u(t)

z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t)
 Second Theoretical Model
z(t)
v(t) h(x(t))
0
x (t) = f (x(t)) + g(x(t)) · v(t)

K(x(t), u(t))
u(t)

z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t)
• Static functions h and K are static and nonlinear → define behaviour
 Second Theoretical Model
z(t)
v(t) h(x(t))
0
x (t) = f (x(t)) + g(x(t)) · v(t)

K(x(t), u(t))
u(t)

z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t)
• Static functions h and K are static and nonlinear → define behaviour
• For a given G and known f and b → h and K can be calculated
 Second Theoretical Model
z(t)
v(t) h(x(t))
0
x (t) = f (x(t)) + g(x(t)) · v(t)

K(x(t), u(t))
u(t)

z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t)
• Static functions h and K are static and nonlinear → define behaviour
• For a given G and known f and b → h and K can be calculated
• Idea: nonlinearities could be ”provided” by a reservoir
 Second Theoretical Model
z(t)
v(t) h(x(t))
0
x (t) = f (x(t)) + g(x(t)) · v(t)

K(x(t), u(t))
u(t)

z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t)
• Static functions h and K are static and nonlinear → define behaviour
• For a given G and known f and b → h and K can be calculated
• Idea: nonlinearities could be ”provided” by a reservoir
• Therefore, linear feedback and linear readouts might be sufficient
 Second Theoretical Model
z(t)
v(t) h(x(t))
0
x (t) = f (x(t)) + g(x(t)) · v(t)

K(x(t), u(t))
u(t)

z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t)
• Static functions h and K are static and nonlinear → define behaviour
• For a given G and known f and b → h and K can be calculated
• Idea: nonlinearities could be ”provided” by a reservoir
• Therefore, linear feedback and linear readouts might be sufficient
• Feedback could be even fixed and directly feed the output back
Second Theoretical Model
u(t)
X
z(t)

...
Second Theoretical Model
u(t)
X
z(t)

...
u(t) z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)
) + u(t) z(t)
Second Theoretical Model
u(t)
X
z(t)

...
u(t) z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)
) + u(t) z(t)

encodes our computation

Second Theoretical Model
u(t)
X
z(t)

...
emulation

u(t) z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t) z(t)

encodes our computation

Second Theoretical Model
u(t)
X
z(t)

...
emulation

u(t) z(t)(n) = G(z(t), z(t)0 , . . . , z(t)(n 1)

) + u(t) z(t)

encodes our computation

Applications for RC
 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

• Speech recognition (Verstraeten et al. 2005 and 2006, Jaeger et al.
2007)
 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

• Speech recognition (Verstraeten et al. 2005 and 2006, Jaeger et al.
2007)
• International financial time series forecasting
 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

• Speech recognition (Verstraeten et al. 2005 and 2006, Jaeger et al.
2007)
• International financial time series forecasting
• Nonlinear audio processing (Holzmann and Hauser 2010)
 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

• Speech recognition (Verstraeten et al. 2005 and 2006, Jaeger et al.
2007)
• International financial time series forecasting
• Nonlinear audio processing (Holzmann and Hauser 2010)
• Epileptic seizure detection
 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

• Speech recognition (Verstraeten et al. 2005 and 2006, Jaeger et al.
2007)
• International financial time series forecasting
• Nonlinear audio processing (Holzmann and Hauser 2010)
• Epileptic seizure detection
• Robot localization (Antonelo and Schrauwen 2011)
 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

• Speech recognition (Verstraeten et al. 2005 and 2006, Jaeger et al.
2007)
• International financial time series forecasting
• Nonlinear audio processing (Holzmann and Hauser 2010)
• Epileptic seizure detection
• Robot localization (Antonelo and Schrauwen 2011)
• Processing Grammatical Structure (ORGANIC project)
 Applications for RC

• Chaotic time series prediction (Jager and Haass 2004)

• Speech recognition (Verstraeten et al. 2005 and 2006, Jaeger et al.
2007)
• International financial time series forecasting
• Nonlinear audio processing (Holzmann and Hauser 2010)
• Epileptic seizure detection
• Robot localization (Antonelo and Schrauwen 2011)
• Processing Grammatical Structure (ORGANIC project)
• Morphological Computation (robot bodies as a reservoir)
The body as a Reservoir

X
...

...
...
The body as a Reservoir

X
...

...
...

Limit cycle for locomotion

 Results
Generating Stable Nonlinear Limit Cycles
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

X
x1
no input!
...

X
x2
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

X
x1
no input!
...

X
x2
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

1 1

0.5 0.5

x2
0 0

x −0.5
−0.5 1
x
2
−1 −1
0 5 10 15 −0.5 0 0.5
time [s] x1
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

#masses = 10
#springs = 22
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

#masses = 10 1
#springs = 22

0.5
x2
0

−0.5

−1
−1 −0.5 0 0.5 1
x1
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

#masses = 10 1
#springs = 22

0.5
x2
0

−0.5

−1
−1 −0.5 0 0.5 1
x1
 Results
Generating Stable Nonlinear Limit Cycles

x01 = x1 + x2 x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )
noise
#masses = 10 1
#springs = 22

0.5
x2
0

−0.5

−1
−1 −0.5 0 0.5 1
x1
 Results
Testing Robustness of Learned Nonlinear Limit Cycles
 Results
Testing Robustness of Learned Nonlinear Limit Cycles

Applied constant disturbance forces at random points

 Results
Testing Robustness of Learned Nonlinear Limit Cycles

Applied constant disturbance forces at random points

−2 x1
x
2
−4

−6
0 10 20 30 40 50
time [s]
 Results
Testing Robustness of Learned Nonlinear Limit Cycles

Applied constant disturbance forces at random points

1
0
0

x1 −1
−2
x

x2
2
−2

−4 −3
−4
−6 −5
0 10 20 30 40 50 −4 −2 0
time [s] x1
 Results
Testing Robustness of Learned Nonlinear Limit Cycles

Applied constant disturbance forces at random points

noise
1
0
0

x1 −1
−2
x

x2
2
−2

−4 −3
−4
−6 −5
0 10 20 30 40 50 −4 −2 0
time [s] x1
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

= 0.2
1
=1
=5
x2

−1
−1 0 1
x1
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

= 0.2 x1
1 1
=1 x2
=5
x2

0 0

−1 −1

−1 0 1 0 5 10 15
x1 time [s]
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

= 0.2 x1
1 1
=1 x2
=5
x2

0 0

−1 −1

−1 0 1 0 5 10 15
x1 time [s]
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

= 0.2 x1
1 1
=1 x2
=5
x2

0 0

−1 −1

−1 0 1 0 5 10 15
x1 time [s]
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

x02 = 2x1 + x2 x2 (x21 + x22 )

X
x1
"
...

X
x2
 Results
Input Dependent Limit Cycle

x01 = x1 + x2 "x1 (x21 + x22 )

0
x 2 =
one 2x1 + x2
set of x2 (x21 + x22 )
weights for
all three "
X
x1
"
...

X
x2
 Results
Input Dependent Limit Cycle

2 2
x1
"=5
1 x2 1

x2
0 0
from 0 to 20 s
−1 −1 from 20 to 40 s
from 40 to 60 s
from 60 to 80 s
from 80 to 100 s
−2 −2
0 20 40 60 −2 0 2
time [s] x1

= 0.2
1
=1
=5
x2

−1
−1 0 1
x
1
 Results
Input Dependent Limit Cycle

2 2
x1
"=1
1 x2 1

x2
0 0
from 0 to 20 s
−1 −1 from 20 to 40 s
from 40 to 60 s
from 60 to 80 s
from 80 to 100 s
−2 −2
0 20 40 60 −2 0 2
time [s] x
1

= 0.2
1
=1
=5
x2

−1
−1 0 1
x
1
 Results
Input Dependent Limit Cycle

2 2
x1
" = 0.2
1 x2 1

x2
0 0
from 0 to 20 s
−1 −1 from 20 to 40 s
from 40 to 60 s
from 60 to 80 s
from 80 to 100 s
−2 −2
0 20 40 60 −2 0 2
time [s] x
1

= 0.2
1
=1
=5
x2

−1
−1 0 1
x
1
Discussion

...
Discussion

...
constant
Discussion

squeezing

...
constant
Discussion

squeezing

...
constant
Discussion

squeezing

...
constant
Discussion
squeezing

...
constant
 Discussion
squeezing

...
constant

System is able to sense through its morphology!

 Discussion

...
constant

System is able to sense through its morphology!

 Discussion

...
constant

System is able to sense through its morphology!

 Discussion
heavy weight

...
constant

System is able to sense through its morphology!

Application in Soft Robotics
Application in Soft Robotics
Kohei
Nakajima

Tao Li
Application in Soft Robotics
Kohei
Nakajima

Tao Li

soft (passive)
silicone structure
Application in Soft Robotics
Kohei
Nakajima

Tao Li

water
tank soft (passive)
silicone structure
Application in Soft Robotics
Kohei
Nakajima

Tao Li

water
tank soft (passive)
silicone structure

10 bending
sensors
Application in Soft Robotics
Kohei
Nakajima

rotational
motor Tao Li

water
tank soft (passive)
silicone structure

10 bending
sensors
Application in Soft Robotics
X

...
Application in Soft Robotics
X

...
Application in Soft Robotics
X

...
X
...
Application in Soft Robotics
X

...
motor signal

X
...
Application in Soft Robotics
X

...
motor signal

X • exploitation of the body

...
Application in Soft Robotics
X

...
motor signal

X • exploitation of the body

...

• nonlinear and memory

Application in Soft Robotics
X

...
motor signal

X • exploitation of the body

...

• nonlinear and memory

• noise comes from the sensors
Application in Soft Robotics
X

...
motor signal

X • exploitation of the body

...

• nonlinear and memory

• noise comes from the sensors
• sensors, water and even motor is
part of the reservoir
Application in Soft Robotics
X

...
motor signal

X • exploitation of the body

...

• nonlinear and memory

• noise comes from the sensors
• sensors, water and even motor is
part of the reservoir
• able to produce robust limit cycle
Application in Soft Robotics
Summary

...
Summary

...
 Summary

...
Supervised Machine Learning technique
 Summary
Reservoir: Randomly
initialized complex,
nonlinear, dynamic
system

...
Supervised Machine Learning technique
 Summary
Reservoir: Randomly output weights learned
initialized complex, with linear regression
nonlinear, dynamic
system

...
Supervised Machine Learning technique
Thank you very much
for your Attention!