Download the Full Ebook and Access More Features - ebooknice.

com

(Ebook) Deep Learning in Multi-step Prediction of

Chaotic Dynamics: From Deterministic Models to
Real-World Systems by Matteo Sangiorgio ISBN
9783030944810, 3030944816
https://ebooknice.com/product/deep-learning-in-multi-step-
prediction-of-chaotic-dynamics-from-deterministic-models-to-
real-world-systems-44838084

OR CLICK HERE

DOWLOAD EBOOK

Download more ebook instantly today at https://ebooknice.com

 Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...

Start reading on any device today!

(Ebook) Introduction to Deep Learning Using R: A Step-by-

Step Guide to Learning and Implementing Deep Learning
Models Using R by Taweh Beysolow II ISBN 9781484227336,
1484227336
https://ebooknice.com/product/introduction-to-deep-learning-using-r-a-
step-by-step-guide-to-learning-and-implementing-deep-learning-models-
using-r-55460716
ebooknice.com

(Ebook) Multi-faceted Deep Learning: Models and Data by

Jenny Benois-Pineau, Akka Zemmari ISBN 9783030744779,
3030744779
https://ebooknice.com/product/multi-faceted-deep-learning-models-and-
data-35166776

ebooknice.com

(Ebook) Chaotic Transitions in Deterministic and

Stochastic Dynamical Systems : Applications of Melnikov
Processes in Engineering, Physics, and Neuroscience by
Emil Simiu ISBN 9781400832507, 1400832500
https://ebooknice.com/product/chaotic-transitions-in-deterministic-
and-stochastic-dynamical-systems-applications-of-melnikov-processes-
in-engineering-physics-and-neuroscience-51307068
ebooknice.com

(Ebook) Chaotic Dynamics in Planetary Systems by Sylvio

Ferraz-Mello ISBN 9783031458156, 303145815X

https://ebooknice.com/product/chaotic-dynamics-in-planetary-
systems-54704748

ebooknice.com
(Ebook) Chaotic Dynamics in Planetary Systems by Sylvio
Ferraz-Mello ISBN 9783031458163, 3031458168

https://ebooknice.com/product/chaotic-dynamics-in-planetary-
systems-54822428

ebooknice.com

(Ebook) Chaotic dynamics of nonlinear systems / by S. Neil

Rasband ISBN 9780486795997, 0486795993

https://ebooknice.com/product/chaotic-dynamics-of-nonlinear-
systems-11099820

ebooknice.com

(Ebook) Python Deep Learning: Understand how deep neural

networks work and apply them to real-world tasks by
Vasilev, Ivan ISBN 9781837638505, 1837638500
https://ebooknice.com/product/python-deep-learning-understand-how-
deep-neural-networks-work-and-apply-them-to-real-world-tasks-56065528

ebooknice.com

(Ebook) Handbook of research on multi-agent systems:

semantics and dynamics of organizational models by
Virginia Dignum ISBN 9781605662565, 9781605662572,
1605662569, 1605662577
https://ebooknice.com/product/handbook-of-research-on-multi-agent-
systems-semantics-and-dynamics-of-organizational-models-1186686

ebooknice.com

(Ebook) Deep Learning for Natural Language Processing:

Develop Deep Learning Models for Natural Language in
Python by Jason Brownlee
https://ebooknice.com/product/deep-learning-for-natural-language-
processing-develop-deep-learning-models-for-natural-language-in-
python-11575184
ebooknice.com
SPRINGER BRIEFS IN APPLIED SCIENCES AND
TECHNOLOGY · POLIMI SPRINGER BRIEFS

Matteo Sangiorgio
Fabio Dercole
Giorgio Guariso

Deep Learning
in Multi-step
Prediction
of Chaotic Dynamics
From Deterministic
Models to Real-World
Systems
SpringerBriefs in Applied Sciences and Technology

PoliMI SpringerBriefs

Editorial Board
Barbara Pernici, Politecnico di Milano, Milano, Italy
Stefano Della Torre, Politecnico di Milano, Milano, Italy
Bianca M. Colosimo, Politecnico di Milano, Milano, Italy
Tiziano Faravelli, Politecnico di Milano, Milano, Italy
Roberto Paolucci, Politecnico di Milano, Milano, Italy
Silvia Piardi, Politecnico di Milano, Milano, Italy
More information about this subseries at https://link.springer.com/bookseries/11159
http://www.polimi.it
Matteo Sangiorgio · Fabio Dercole ·
Giorgio Guariso

Deep Learning
in Multi-step Prediction
of Chaotic Dynamics
From Deterministic Models
to Real-World Systems
Matteo Sangiorgio Fabio Dercole
Dipartimento di Elettronica, Dipartimento di Elettronica,
Informazione e Bioingegneria (DEIB) Informazione e Bioingegneria (DEIB)
Politecnico di Milano Politecnico di Milano
Milan, Italy Milan, Italy

Giorgio Guariso
Dipartimento di Elettronica,
Informazione e Bioingegneria (DEIB)
Politecnico di Milano
Milan, Italy

ISSN 2191-530X ISSN 2191-5318 (electronic)

SpringerBriefs in Applied Sciences and Technology
ISSN 2282-2577 ISSN 2282-2585 (electronic)
PoliMI SpringerBriefs
ISBN 978-3-030-94481-0 ISBN 978-3-030-94482-7 (eBook)
https://doi.org/10.1007/978-3-030-94482-7

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021

This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse
of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and
transmission or information storage and retrieval, electronic adaptation, computer software, or by similar
or dissimilar methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
Non ti sia grave il fermarti alcuna volta a
vedere nelle macchie de’ muri, o nella cenere
del fuoco, o nuvoli o fanghi, od altri simili
luoghi, ne’ quali, se ben saranno da te
considerati, tu troverai invenzioni
mirabilissime… perchè nelle cose confuse
l’ingegno si desta.
(It would not be too much of an effort to
pause sometimes to look into these stains on
walls, the ashes from the fire, the clouds, the
mud, or other similar places. If these are well
contemplated, you will find fantastic
inventions… These will do you well because
they will awaken genius.)
Leonardo da Vinci, Trattato della pittura, ca.
1540.
Preface

In the present data-rich era, we know that time series of many variables can hardly
be interpreted as regular movements plus some stochastic noise. For half a century,
we have also known that even apparently simple sets of nonlinear equations can
produce extremely complex movements that remain within a limited portion of the
variables space without being periodic. Such movements have been named “chaotic”
(“deterministic chaos” when the equations include no stochasticity).
Immediately after they were discovered, Lorenz and other researchers were trou-
bled by the problem of predictability. How far into the future can we reliably forecast
the output of such systems? For many years, the answer to such a question remained
limited to very few steps. Today, however, powerful computer tools are available
and have been successfully used to accomplish complex tasks. Can we extend our
predictive ability using such tools? How far? Can we predict not just a single value,
but also an entire sequence of outputs?
This book tries to answer these questions by using deep artificial neural networks
as the forecasting tools and analyzing the performances of different architectures of
such networks. In particular, we compare the classical feed-forward (FF) architecture
with the more recent long short-term memory (LSTM) structure. For the latter, we
explore the possibility of using or not the traditional training approach known as
“teacher forcing”.
Before presenting these methods in detail, we revise the basic elements and tools
of chaos theory and nonlinear time-series analysis in Chap. 2. We take a practical
approach, looking at how chaoticity can be quantified by simulating mathematical
models, as well as from real-world measurements.
Chapter 3 presents the cases on which we test our deep neural predictors. We
consider four well-known dynamical systems (in discrete time) showing a chaotic
attractor in their variables space: the logistic and the Hénon maps, which are the
prototypes of chaos in non-reversible and reversible systems, respectively, and two
generalized Hénon maps, to include cases of low- and high-dimensional hyperchaos.
These systems easily produce arbitrarily long synthetic datasets, in the form of deter-
ministic chaotic time series, to which we also add a synthetic noise to better mimic
real situations. We finally consider two real-world time series: solar irradiance and

vii
viii Preface

ozone concentration, measured at two stations in Northern Italy. These time-series are
shown to contain a chaotic movement by means of the tools of nonlinear time-series
analysis.
Chapter 4 illustrates the structures of the neural networks that we use and intro-
duces their performance metrics. It deals, in particular, with the issue of multi-
step forecasting that is commonly performed by recursively nesting one-step-ahead
predictions (recursive predictor). An alternative explored here consists of training
the model to directly compute multiple outputs (multi-output predictor), each repre-
senting the prediction at a specific time step in the future. Multi-output prediction
can be implemented in static networks (using FF nets), as well as in a dynamic way
(adopting recurrent architectures, such as the LSTM nets). The standard training
procedures for both architectures are revised, with particular attention to the issue
related to the “teacher forcing” training approach for LSTM networks.
Chapter 5 illustrates the key results of the study on both the synthetic and the
real-world time series and explores the effect of different sources of noise. In partic-
ular, we consider a stochastic environment that mimics the observation noise, and
the presence of non-stationary dynamics that can be seen as a structural distur-
bance. Additionally, in the context of chaotic systems’ forecasting, we introduce the
concept of the generalization capability of the neural predictors in terms of “domain
adaptation”.
Chapter 6 discusses some remarks on the choice of the experimental settings that
are adopted in this work. It also presents additional aspects of the neural predic-
tors, analyzing their training method, their neural architecture, and their long-term
performances.
The main result we can draw from our journey in the vast area of deep neural
network architectures is that the LSTM networks—which are dynamical systems
themselves—trained without teacher forcing, are the best approach to predict
complex oscillatory time series. They systematically outperform the competitors
and also prove to be able to adapt to other domains with similar features without
a relevant decrease of accuracy. Overall, they represent a significant improvement
of forecasting capabilities of chaotic time series and can be used as components
of advanced control techniques, such as model predictive control, to optimize the
management of complex real-world systems. These conclusions and some consid-
erations about the expected future research in the field are summarized in the last
chapter.
The book will be useful for researchers and Ph.D. students in the area of neural
networks and chaotic systems as well as to practitioners that are interested in applying
modern deep learning techniques to the forecasting of complex real-world time series,
particularly those related to environmental systems. The book will also be of interest
for all the scholars that would like to bridge the gap between the classical theory of
nonlinear systems and recent development in machine learning techniques.
The book largely draws from the Ph.D. thesis in Information Technology by
Matteo Sangiorgio; supervisor: Giorgio Guariso; tutor: Fabio Dercole, Politecnico
di Milano, 2021. The main results presented in the text appeared in the following
publications:
Preface ix

• M. Sangiorgio and F. Dercole. “Robustness of LSTM neural networks for multi-

step forecasting of chaotic time series”. Chaos, Solitons & Fractals 139, 2020.
• G. Guariso, G. Nunnari and M. Sangiorgio. “Multi-step solar irradiance fore-
casting and domain adaptation of deep neural networks”. In: Energies 13.15,
2020.
• F. Dercole, M. Sangiorgio and Y. Schmirander. “An empirical assessment of the
universality of ANNs to predict oscillatory time series”. IFAC-PapersOnLine 53,
proceedings of IFAC World Congress, Berlin, Germany, July 2020.
• M. Sangiorgio, F. Dercole and G. Guariso. “Sensitivity of chaotic dynamics
prediction to observation noise”. IFAC-PapersOnLine 54, proceedings of IFAC
CHAOS Conference, Catania, Italy, September 2021.
• M. Sangiorgio, F. Dercole and G. Guariso. “Forecasting of noisy chaotic systems
with deep neural networks”. Chaos, Solitions and Fractals 153, 2021.

Milan, Italy Matteo Sangiorgio

September 2021 Fabio Dercole
Giorgio Guariso
Contents

1 Introduction to Chaotic Dynamics’ Forecasting . . . . . . . . . . . . . . . . . . . 1

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 Basic Concepts of Chaos Theory and Nonlinear Time-Series
Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Dynamical Systems and Their Attractors . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Lyapunov Exponents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 Average Exponents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.2 Local Exponents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3 Chaotic Systems, Predictability, and Fractal Geometry . . . . . . . . . . . 19
2.3.1 Chaotic Attractors and Lyapunov Time Scale . . . . . . . . . . . . . 19
2.3.2 Correlation and Lyapunov Dimensions . . . . . . . . . . . . . . . . . . 21
2.4 Attractor Reconstruction from Data . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.4.1 Delay-Coordinate Embedding . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.4.2 Estimation of the Largest Lyapunov Exponent . . . . . . . . . . . . 27
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3 Artificial and Real-World Chaotic Oscillators . . . . . . . . . . . . . . . . . . . . . 31
3.1 Artificial Chaotic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.1.1 Logistic Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.1.2 Hénon Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1.3 Generalized Hénon Map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.1.4 Time-Varying Logistic Map . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.2 Real-World Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2.1 Solar Irradiance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.2.2 Ozone Concentration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4 Neural Approaches for Time Series Forecasting . . . . . . . . . . . . . . . . . . . 43
4.1 Neural Approaches for Time Series Prediction . . . . . . . . . . . . . . . . . . 44
4.1.1 FF-Recursive Predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
4.1.2 FF-Multi-Output Predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

xi
xii Contents

4.1.3 LSTM Predictor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

4.2 Performance Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.3 Training Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
5 Neural Predictors’ Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.1 Deterministic Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
5.1.1 Performance Distribution over the System’s Attractor . . . . . 64
5.1.2 Sensitivity to the Embedding Dimension . . . . . . . . . . . . . . . . 65
5.2 Stochastic Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
5.3 Non-Stationary System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
5.4 Real-World Study Cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4.1 Solar Irradiance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
5.4.2 Ozone Concentration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
6 Neural Predictors’ Sensitivity and Robustness . . . . . . . . . . . . . . . . . . . . 85
6.1 Simplicity and Robustness of the Experimental Setting . . . . . . . . . . . 85
6.2 Predictors’ Long-Term Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
6.3 Remarks on the Training Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
6.3.1 Backpropagation and Backpropagation Through Time . . . . . 90
6.3.2 Training with and Without Teacher Forcing . . . . . . . . . . . . . . 93
6.4 Advanced Feed-Forward Architectures . . . . . . . . . . . . . . . . . . . . . . . . 94
6.5 Chaotic Dynamics in Recurrent Networks . . . . . . . . . . . . . . . . . . . . . . 95
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7 Concluding Remarks on Chaotic Dynamics’ Forecasting . . . . . . . . . . . 97
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
Chapter 1
Introduction to Chaotic Dynamics’
Forecasting

Abstract Chaotic dynamics are the paradigm of complex and unpredictable evolu-
tion due to their built-in feature of amplifying arbitrarily small perturbations. The
forecasting of these dynamics has attracted the attention of many scientists since
the discovery of chaos by Lorenz in the 1960s. In the last decades, machine learn-
ing techniques have shown a greater predictive accuracy than traditional tools from
nonlinear time-series analysis. In particular, artificial neural networks have become
the state of the art in chaotic time series forecasting. However, how to select their
structure and the training algorithm is still an open issue in the scientific commu-
nity, especially when considering a multi-step forecasting horizon. We implement
feed-forward and recurrent architectures, considering different training methods and
forecasting strategies. The predictors are evaluated on a wide range of problems,
from low-dimensional deterministic cases to real-world time series.

Since Edward Lorenz’s discovery of deterministic chaos in the sixties [63], many
attempts to forecast the future evolution of chaotic systems and to discover how
far they can be predicted, have been carried out adopting a wide range of models.
Some early attempts were performed in the 80s and 90s [7, 18, 31, 50, 75, 82–
84, 91, 97], but the topic has been debated more and more in recent years (see
Fig. 1.1) due to the development of lots of machine learning techniques in the field
of time-series analysis and prediction. Attempts include widely used models such
as the support vector machines [41, 73, 114] and the polynomial functions [92,
93], but the most widely used architectures are artificial neural networks (ANNs).
The main distinction within ANNs is between feed-forward (FF) structures—static
maps which approximate the relationship between inputs and outputs—and recurrent
neural nets (RNNs)—dynamical models whose outputs also depend on an internal
state characterizing each neuron.
Examples of the first category are the traditional multi-layer perceptrons [4, 9,
24, 25, 28, 42, 52, 56, 61, 103, 110], radial basis function networks [26, 38, 57, 71,
95, 96, 104], fuzzy neural networks [22, 33, 53, 69, 108, 117], deep belief neural
nets [54], extreme learning machines [30, 72], and convolutional neural networks
[8, 81]. RNNs are particularly suited to be used as predictors of nonlinear time series

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 1

M. Sangiorgio et al., Deep Learning in Multi-step Prediction of Chaotic Dynamics,
PoliMI SpringerBriefs,
https://doi.org/10.1007/978-3-030-94482-7_1
2 1 Introduction to Chaotic Dynamics’ Forecasting

350

300
Web of Science records per year

250

200

150

100

50

0
1990 1995 2000 2005 2010 2015 2020
Year

Fig. 1.1 Number of records by publication year (1986–2020) from a Web of Science search for
the topic “chaos prediction”

[15, 16, 19, 20, 43, 61, 66, 67, 113, 116], though their training is computation-
ally heavy because of the back propagation through time of the internal memory
[36]. Recently, two particular RNN classes, reservoir computers and long short-term
memory (LSTM) networks, have been demonstrated to be extremely efficient in the
prediction of chaotic dynamics [3, 10, 13, 14, 29, 40, 44, 49, 51, 64, 65, 68, 74,
79, 80, 87, 98–102, 111, 115, 118]. Reservoir computers make use of many recur-
rent neurons with random connections and training is only performed on the output
layer. Conversely, recurrent neurons are trained in LSTM nets. These nets avoid the
typical issue of vanishing or exploding optimization gradients by modulating the
information flow through so-called gates, whose behaviors are learned as well.
The comparison between feed-forward and recurrent nets is not trivial. Despite
the initial optimism, RNNs turned out to achieve similar performances to tradi-
tional FF architectures on many forecasting tasks [34, 67]. Recurrent neurons have
been demonstrated to be efficient when they are used as basic blocks to build up
sequence-to-sequence architectures. This kind of structure represents the state-of-
the-art approach in many natural language processing tasks (e.g., machine transla-
tion, question answering, automatic summarization, text-to-speech, . . . ), though they
have recently lost ground to advanced FF architectures. Other works in the field of
chaotic dynamics’ forecasting focused on wavelet networks [17], swarm-optimized
neural nets [62], Chebyshev [2], and residual [67] nets. Some authors developed
hybrid forecasting schemes that integrate machine learning tools and knowledge-
based models [27, 55, 58, 78, 98] or combine multiple predictive techniques [23,
47, 76, 90].
Forecasting chaotic dynamics one or few time steps ahead is usually an easy
task as demonstrated by the high performances obtained in many systems, in both
continuous and discrete time [1, 4, 5, 59, 68, 69, 103, 107, 109, 116]. The situation
1 Introduction to Chaotic Dynamics’ Forecasting 3

changes dramatically when considering a mid-long horizon, because of the expansion

of small errors intrinsically due to the chaotic nature of the actual data. Even when
the data are generated by a known chaotic system, small initialization or numerical
errors are amplified up to predict a distant point in the system’s attractor.
The forecasting of a time series over a multi-step horizon is unequivocally recog-
nized as an open issue in the field of time series forecasting and a challenging testing
ground for machine learning techniques [46, 70].
The multi-step ahead forecasting of a chaotic dynamic is commonly done by
recursively performing one-step ahead predictions [2, 6, 32, 60, 89, 105]. An alter-
native consists of training the model to directly compute multiple outputs [96], each
of which represents the prediction at a certain time step, or even identifying a specific
model for each future step [62].
In this book, we compare three state-of-the-art architectures for the multi-step
prediction of chaotic time series: FF-recursive, FF-multi-output, and LSTM nets, the
latter trained according to two different methods: the so-called teacher forcing (TF),
so far traditionally used, and the variant without teacher forcing (no-TF). We analyze
the strengths and weaknesses of these four predictors, from both a qualitative and
quantitative perspective.
Because they are optimized to predict only the next time step, the FF-recursive
net is trained to replicate the dynamical systems generating the data, i.e., the map
from previous data to the next one. Its performance, however, degrades on mid-long
horizons because of the system’s chaoticity. The FF-multi-output net is trained to
predict the whole forecasting horizon, but it acts as if the outputs were different
variables, rather than the same variable sampled at subsequent steps. For this reason,
it struggles to replicate the dynamical behavior of the chaotic maps.
To overcome the critical aspects of FF nets, we adopt models which are able to deal
with temporal dynamics, such as the LSTM nets. These architectures can potentially
couple the advantages of the FF-recursive and multi-output nets. However, they
are usually trained with the so-called TF algorithm, in which the target values are
used as an input for each time step ahead, rather than the outputs predicted for the
previous steps. Essentially, the net learns to predict only one step, because even
when predicting further, it receives the actual data up to the step before the one to be
predicted. In other words, the TF prevents small errors in the approximated map from
being corrected, since they are not propagated in time. In inference mode, multi-step
predictions necessarily require the previous prediction to be fed back, so that the
network behaviors in training and prediction do not coincide (this issue is known as
exposure bias in the machine learning literature [45, 85]). TF is critical, especially
when dealing with chaotic dynamics for their well-known sensitivity to initialization
and prediction errors.
To solve this issue, we propose abandoning the TF method, feeding back the
network predictions even during training. We therefore compare four neural archi-
tectures: FF-recursive, FF-multi-output, LSTM-TF, and LSTM-no-TF.
The articles in the field of chaotic dynamics’ forecasting, except for a few recent
exceptions [12, 21, 77, 88, 94], usually study noise-free stationary time series. We
analyze how the predictors’ forecasting skills are affected by different typologies of
4 1 Introduction to Chaotic Dynamics’ Forecasting

noise. To quantify the effect of the observation noise, we artificially add a stochas-
tic disturbance with different magnitudes to the deterministic time series generated
by the traditional chaotic systems. We also assess the sensitivity to the structural
noise, obtained by introducing a slow-varying dynamic for the parameter defining
the growth rate in the traditional logistic map. The resulting non-stationary process
has concurrent slow and fast dynamics that represent a challenging forecasting task.
These numerical experiments lay somewhat in between the deterministic systems,
mainly theoretical, and the practical applications. Finally, we apply the proposed
methodologies to two real-world time series that exhibit a chaotic behavior: solar
irradiance and ozone concentration.
The results obtained show that, in general, LSTM nets trained without TF are the
highest performing in predicting chaotic dynamics. The improvement with respect
to the three competitors is not uniform and strongly task-dependent. LSTM-no-TF
are also shown to be more robust when a redundant (or lacking) number of time lags
are included in the input—a feature that represents a remarkable advantage given
the well-known problem of estimating the actual embedding dimension from a time
series [11].
Another remarkable feature of the forecasting models is their generalization capa-
bility, often mentioned as “domain adaptation” (a sub-field of the so-called “transfer
learning”) in the neural networks literature [35]. It indicates the possibility of storing
the knowledge gained while solving a given task and applying it to different, though
similar, datasets [112]. Transfer learning became a hot topic in machine learning in
the last decade, but it only received attention from researchers in the field of nonlinear
science and chaotic systems in recent times [37, 39, 48, 86, 106]. To test this feature,
the neural networks developed to forecast the solar irradiance in a specific location
(source domain) have been used, without retraining, on other sites (target domains)
with quite different geographical conditions. The neural networks developed in our
study have proved able to forecast solar radiation in other stations with a minimal
loss of precision.
The rest of the book is structured as follows. Chapter 2 introduces some basic
concepts related to chaos theory. Chapter 3 presents the chaotic systems and time
series used to test the predictors. Chapter 4 describes different ways to use feed-
forward and recurrent neural networks in a multi-step ahead prediction task. The
other sections of this chapter focus on some technical issues on how to set up a
supervised learning task starting from a time series and on the performance metrics
and the training procedure. Chapter 5 reports the forecasting performance obtained
in both the artificial systems and the real-world time series considered. In Chap. 6,
we discuss some technical aspects of the training procedure and of the behavior of
the different neural predictors. We also present an overview of other alternative state-
of-the-art architectures which can be adopted for numerical time series forecasting.
Chapter 7 provides the concluding remarks of the book and presents some possible
future research directions.
References 5

References

1. Abdulkadir, S. J., Alhussian, H., & Alzahrani, A. I. (2018). Analysis of recurrent neural net-
works for henon simulated time-series forecasting. Journal of Telecommunication, Electronic
and Computer Engineering (JTEC), 10.1-8, 155–159.
2. Akritas, P., Antoniou, I., & Ivanov, V. V. (2000). Identification and prediction of discrete
chaotic maps applying a Chebyshev neural network. Chaos, Solitons and Fractals, 11.1-3,
337–344.
3. Antonik, P., et al. (2018). Using a reservoir computer to learn chaotic attractors, with appli-
cations to chaos synchronization and cryptography. Physical Review E, 98.1, 012215.
4. Atsalakis, G., Skiadas, C., & Nezis, D. (2008). Forecasting Chaotic time series by a Neural
Network. In Proceedings of the 8th International Conference on Applied Stochastic Models
and Data Analysis, Vilnius, Lithuania. (Vol. 30, p. 7782).
5. Atsalakis, G., & Tsakalaki, K. (2012). Simulating annealing and neural networks for chaotic
time series forecasting. Chaotic Model. Simul., 1, 81–90.
6. Bakker, R., et al. (2000). Learning chaotic attractors by neural networks. Neural Computation,
12.10, 2355–2383.
7. Bollt, E. M. (2000). Model selection, confidence and scaling in predicting chaotic time-series.
International Journal of Bifurcation and Chaos, 10.06, 1407–1422.
8. Bompas, S., Georgeot, B., & Guéry-Odelin, D. (2020). Accuracy of neural networks for the
simulation of chaotic dynamics: Precision of training data versus precision of the algorithm.
arXiv:2008.04222.
9. Bonnet, D., Labouisse, V., & Grumbach, A. (1997). δ-NARMA neural networks: A new
approach to signal prediction. IEEE Transactions on Signal Processing, 45.11, 2799–2810.
10. Borra, F., Vulpiani, A., & Cencini, M. (2020). Effective models and predictability of chaotic
multiscale systems via machine learning. Physical Review E, 102.5, 052203.
11. Bradley, E., & Kantz, H. (2015). Nonlinear time-series analysis revisited. Chaos: An Inter-
disciplinary Journal of Nonlinear Science, 25.9, 097610.
12. Brajard, J. et al. (2020). Combining data assimilation and machine learning to emulate a
dynamical model from sparse and noisy observations: A case study with the Lorenz 96 model.
Journal of Computational Science, 44, 101171.
13. Butcher, J. B., et al. (2013). Reservoir computing and extreme learning machines for non-
linear time-series data analysis. Neural Networks, 38, 76–89.
14. Canaday, D., Griffith, A., & Gauthier, D. J. (2018). Rapid time series prediction with a
hardware-based reservoir computer. Chaos: An Interdisciplinary Journal of Nonlinear Sci-
ence, 28.12, 123119.
15. Cannas, B., & Cincotti, S. (2002). Neural reconstruction of Lorenz attractors by an observable.
Chaos, Solitons and Fractals, 14.1, 81–86.
16. Cannas, B., et al. (2001). Learning of Chua’s circuit attractors by locally recurrent neural
networks. Chaos, Solitons and Fractals, 12.11, 2109–2115.
17. Cao, L., et al. (1995). Predicting chaotic time series with wavelet networks. Physica D:
Nonlinear Phenomena, 85.1-2, 225–238.
18. Casdagli, M. (1989). Nonlinear prediction of chaotic time series. Physica D: Nonlinear Phe-
nomena, 35.3, 335–356.
19. Cechin, A. L., Pechmann, D. R., & de Oliveira, L. P. (2008). Optimizing Markovian modeling
of chaotic systems with recurrent neural networks. Chaos, Solitons and Fractals, 37.5, pp.
1317–1327.
20. Chandra, R., & Zhang, M. (2012). Cooperative coevolution of Elman recurrent neural net-
works for chaotic time series prediction. Neurocomputing, 86, 116–123.
21. Chen, P., et al. (2020). Autoreservoir computing for multistep ahead prediction based on the
spatiotemporal information transformation. Nature Communications, 11.1, 1–15.
22. Chen, Z. (2010). A chaotic time series prediction method based on fuzzy neural network and
its application. In International Workshop on Chaos-Fractal Theories and Applications (pp.
355–359). IEEE.
6 1 Introduction to Chaotic Dynamics’ Forecasting

23. Cheng, W., et al. (2021). High-efficiency chaotic time series prediction based on time convo-
lution neural network. Chaos, Solitons and Fractals, 152, 111304.
24. Covas, E., & Benetos, E. (2019). Optimal neural network feature selection for spatial-temporal
forecasting. Chaos: An Interdisciplinary Journal of Nonlinear Science, 29.6, 063111.
25. Dercole, F., Sangiorgio, M., & Schmirander, Y. (2020). An empirical assessment of the uni-
versality of ANNs to predict oscillatory time series. IFAC-PapersOnLine, 53.2, 1255–1260.
26. Ding, H.-L., et al. (2009). Prediction of chaotic time series using L-GEM based RBFNN.
In 2009 International Conference on Machine Learning and Cybernetics (Vol. 2, pp. 1172–
1177). IEEE.
27. Doan, N. A. K., Polifke, W., & Magri, L. (2019). Physics-informed echo state networks
for chaotic systems forecasting. In International Conference on Computational Science (pp.
192–198). Springer.
28. Dudul, S. V. (2005). Prediction of a Lorenz chaotic attractor using two-layer perceptron neural
network. Applied Soft Computing, 5.4, pp. 333–355.
29. Fan, H., et al. (2020). Long-term prediction of chaotic systems with machine learning. Physical
Review Research, 2.1, 012080.
30. Faqih, A., Kamanditya, B., & Kusumoputro, B. (2018). Multi-step ahead prediction of
Lorenz’s Chaotic system using SOM ELM- RBFNN. In 2018 International Conference on
Computer, Information and Telecommunication Systems (CITS) (pp. 1–5). IEEE.
31. Farmer, J. D., & Sidorowich, J. J. (1987). Predicting chaotic time series. Physical Review
Letters, 59.8, 845.
32. M Galván, I., & Isasi, P. (2001). Multi-step learning rule for recurrent neural models: An
application to time series forecasting. Neural Processing Letters, 13.2, 115–133.
33. Gao, Y., & Joo Er, M. (2005). NARMAX time series model prediction: Feedforward and
recurrent fuzzy neural network approaches. Fuzzy Sets and Systems, 150.2, 331–350.
34. Gers, F. A., Eck, D., & Schmidhuber, J. (2002). Applying LSTM to time series predictable
through time-window approaches. In Neural Nets WIRN Vietri-01 (pp. 193–200). Springer.
35. Glorot, X., Bordes, A., & Bengio, Y. (2011). Domain adaptation for large-scale sentiment
classification: A deep learning approach. Proceedings of the 28 th International Conference
on Machine Learning, Bellevue, WA, USA.
36. Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep Learning. MIT Press.
37. Guariso, G., Nunnari, G., & Sangiorgio, M. (2020). Multi-step solar irradiance forecasting
and domain adaptation of deep neural networks. Energies, 13.15,3987.
38. Guerra, F. A., & dos Coelho, L. S. (2008). Multi-step ahead nonlinear identification of Lorenz’s
chaotic system using radial basis neural network with learning by clustering and particle swarm
optimization. Chaos, Solitons and Fractals, 35.5, 967–979.
39. Guo, Y., et al. (2020). Transfer learning of chaotic systems. arXiv:2011.09970.
40. Haluszczynski, A., & Räth, C. (2019). Good and bad predictions: Assessing and improving the
replication of chaotic attractors by means of reservoir computing. Chaos: An Interdisciplinary
Journal of Nonlinear Science, 29.10, 103143.
41. Han, L., Ding, L., & Qi, L. (2005). Chaotic Time series nonlinear prediction based on support
vector machines. Systems Engineering - Theory and Practice, 9.
42. Han, M., & Wang, Y. (2009). Analysis and modeling of multivariate chaotic time series based
on neural network. Expert Systems with Applications, 36.2, 1280–1290.
43. Han, M. et al. (2004). Prediction of chaotic time series based on the recurrent predictor neural
network. IEEE Transactions on Signal Processing, 52.12, 3409–3416.
44. Hassanzadeh, P., et al. (2019). Data-driven prediction of a multi-scale Lorenz 96 chaotic
system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-
LSTM. In Bulletin of the American Physical Society, C17-009.
45. He, T., et al. (2019). Quantifying exposure bias for neural language generation.
arXiv:1905.10617
46. Hussein, S., Chandra, R., & Sharma, A. (2016). Multi-step- ahead chaotic time series pre-
diction using coevolutionary recurrent neural networks. In IEEE Congress on Evolutionary
Computation (CEC) (pp. 3084–3091). IEEE.
References 7

47. Inoue, H., Fukunaga, Y., & Narihisa, H. (2001). Efficient hybrid neural network for chaotic
time series prediction. In International Conference on Artificial Neural Networks (pp. 712–
718). Springer.
48. Inubushi, M., & Goto, S. (2020). Transfer learning for nonlinear dynamics and its application
to fluid turbulence. Physical Review E, 102.4, 043301.
49. Jiang, J., & Lai, Y.-C. (2019). Model-free prediction of spatiotemporal dynamical systems
with recurrent neural networks: Role of network spectral radius. Physical Review Research,
1.3, 033056.
50. Jones, R. D., et al. (1990). Function approximation and time series prediction with neural
networks. In 1990 IJCNN International Joint Conference on Neural Networks (pp. 649–665).
IEEE.
51. Jüngling, T. (2019). Reconstruction of complex dynamical systems from time series using
reservoir computing. In IEEE International Symposium on Circuits and Systems (ISCAS) (pp.
1–5). IEEE.
52. Karunasinghe, D. S. K., & Liong, S.-Y. (2006). Chaotic time series prediction with a global
model: Artificial neural network. Journal of Hydrology, 323.1-4, 92–105.
53. Kuremoto, T., et al. (2003). Predicting chaotic time series by reinforcement learning. In
Proceedings of the 2nd International Conferences on Computational Intelligence, Robotics
and Autonomous Systems (CIRAS 2003).
54. Kuremoto, T. (2014). Forecast chaotic time series data by DBNs. In 7th International Congress
on Image and Signal Processing (pp. 1130–1135). IEEE.
55. Lei, Y., Hu, J., & Ding, J. (2020). A hybrid model based on deep LSTM for predicting
high-dimensional chaotic systems. arXiv:2002.00799.
56. Lellep, M., et al. (2020). Using machine learning to predict extreme events in the Hénon map.
In Chaos: An Interdisciplinary Journal of Nonlinear Science, 30.1, 013113.
57. Leung, H., Lo, T., & Wang, S. (2001). Prediction of noisy chaotic time series using an optimal
radial basis function neural network. IEEE Transactions on Neural Networks, 12.5, 1163–
1172.
58. Levine, M. E., & Stuart, A. M. (2021). A framework for machine learning of model error in
dynamical systems. arXiv:2107.06658.
59. Li, Q., & Lin, R.-C. (2016). A new approach for chaotic time series prediction using recurrent
neural network. Mathematical Problems in Engineering, 3542898.
60. Lim, T. P., & Puthusserypady, S. (2006). Error criteria for cross validation in the context
of chaotic time series prediction. Chaos: An Interdisciplinary Journal of Nonlinear Science,
16.1, 013106.
61. Lin, T.-N., et al. (1997). A delay damage model selection algorithm for NARX neural net-
works. IEEE Transactions on Signal Processing, 45.11, 2719–2730.
62. López-Caraballo, C. H., et al. (2016). Mackey-Glass noisy chaotic time series prediction
by a swarm-optimized neural network. Journal of Physics: Conference Series, 720, 1. IOP
Publishing.
63. Lorenz, E. N. (1963). Deterministic nonperiodic flow. Journal of the Atmospheric Sciences,
20.2, 130–141.
64. Lu, Z., Hunt, B. R., & Ott, E. (2018). Attractor reconstruction by machine learning. Chaos:
An Interdisciplinary Journal of Nonlinear Science, 28.6, 061104.
65. Lu, Z., et al. (2017). Reservoir observers: Model-free inference of unmeasured variables in
chaotic systems. Chaos: An Interdisciplinary Journal of Nonlinear Science, 27.4, 041102.
66. Ma, Q.-L. (2007). Chaotic time series prediction based on evolving recurrent neural networks.
In International Conference on Machine Learning and Cybernetics (Vol. 6, pp. 3496–3500).
IEEE.
67. Maathuis, H. et al. (2017). Predicting chaotic time series using machine learning techniques.
In Preproceedings of the 29th Benelux Conference on Artificial Intelligence (BNAIC 2017)
(pp. 326–340).
68. Madondo, M., & Gibbons, T. (2018). Learning and modeling chaos using lstm recurrent neural
networks. MICS 2018 Proceedings Paper 26.
8 1 Introduction to Chaotic Dynamics’ Forecasting

69. Maguire, L. P., et al. (1998). Predicting a chaotic time series using a fuzzy neural network.
Information Sciences, 112.1-4, 125–136.
70. Mariet, Z., & Kuznetsov, V. (2019). Foundations of sequence-to-sequence modeling for time
series. In The 22nd International Conference on Artificial Intelligence and Statistics, 408–417.
71. Masnadi-Shirazi, M., & Subramaniam, S. (2020). Attractor Ranked Radial Basis function
network: A nonparametric forecasting Approach for chaotic Dynamic Systems. Scientific
Reports, 10.1, 1–10.
72. Xin-Ying, W. Min, H. (2012). Multivariate chaotic time series prediction based on extreme
learning machine. Acta Physica Sinica, 8.
73. Mukherjee, S., Osuna, E., & Girosi, F. (1997). Nonlinear prediction of chaotic time series
using support vector machines. In Neural Networks for Signal Processing VII. Proceedings
of the 1997 IEEE Signal Processing Society Workshop (pp. 511–520). IEEE.
74. Nakai, K., & Saiki, Y. (2019). Machine-learning construction of a model for a macroscopic
fluid variable using the delay-coordinate of a scalar observable. arXiv:1903.05770.
75. Navone, H. D., & Ceccatto, H. A. (1995). Learning chaotic dynamics by neural networks.
Chaos, Solitons and Fractals, 6, 383–387.
76. Okuno, S., Aihara, K., & Hirata, Y. (2019). Combining multiple forecasts for multivariate
time series via state-dependent weighting. Chaos: An Interdisciplinary Journal of Nonlinear
Science, 29.3, 033128.
77. Patel, D., et al. (2021). Using machine learning to predict statistical properties of non-
stationary dynamical processes: System climate, regime transitions, and the effect of stochas-
ticity. Chaos: An Interdisciplinary Journal of Nonlinear Science, 31.3, 033149.
78. Pathak, J., et al. (2018). Hybrid forecasting of chaotic processes: Using machine learning in
conjunction with a knowledge-based model. Chaos: An Interdisciplinary Journal of Nonlinear
Science, 28.4, 041101.
79. Pathak, J., et al. (2018). Model-free prediction of large spatiotemporally chaotic systems from
data: A reservoir computing approach. Physical Review Letters, 120.2, 024102.
80. Pathak, J., et al. (2017). Using machine learning to replicate chaotic attractors and calculate
Lyapunov exponents from data. Chaos: An Interdisciplinary Journal of Nonlinear Science,
27.12, 121102.
81. Penkovsky, B., et al. (2019). Coupled nonlinear delay systems as deep convolutional neural
networks. Physical Review Letters, 123.5, 054101.
82. Principe, J. C., & Kuo, J.-M. (1995). Dynamic modelling of chaotic time series with neural
networks. Proceedings of the 7th International Conference on Neural Information Processing
Systems, 311–318.
83. Principe, J. C., Rathie, A., Kuo, J.-M. (1992). Prediction of chaotic time series with neural
networks and the issue of dynamic modeling. International Journal of Bifurcation and Chaos,
2.04, 989–996.
84. Principe, J. C., Wang, L., & Kuo, J.-M. (1998). Non-linear dynamic modelling with neural
networks. In Signal Analysis and Prediction (pp. 275–290). Springer.
85. Ranzato, M., et al. (2015). Sequence level training with recurrent neural networks.
arXiv:1511.06732.
86. Sangiorgio, M. (2021). Deep learning in multi-step forecasting of chaotic dynamics. Ph.D.
thesis. Department of Electronics, Information and Bioengineering, Politecnico di Milano.
87. Sangiorgio, M., & Dercole, F. (2020). Robustness of LSTM neural networks for multi-step
forecasting of chaotic time series. Chaos, Solitons and Fractals, 139, 110045.
88. Sangiorgio, M., Dercole, F., & Guariso, G. (2021). Forecasting of noisy chaotic systems with
deep neural networks. Chaos, Solitons & Fractals, 153, 111570.
89. Shi, X., et al. (2017). Chaos time-series prediction based on an improved recursive Levenberg-
Marquardt algorithm. Chaos, Solitons and Fractals, 100, 57–61.
90. Shi, Z., & Han, M. (2007). Support vector echo-state machine for chaotic time-series predic-
tion. IEEE Transactions on Neural Networks, 18.2, 359–372.
91. Shukla, J. (1998). Predictability in the midst of chaos: A scientific basis for climate forecasting.
Science, 282.5389, 728–731.
References 9

92. Su, L., & Li, C. (2015). Local prediction of chaotic time series based on polynomial coefficient
autoregressive model. Mathematical Problems in Engineering, 901807.
93. Su, L.-Y. (2010). Prediction of multivariate chaotic time series with local polynomial fitting.
Computers and Mathematics with Applications, 59.2, 737–744.
94. Teng, Q., & Zhang, L. (2019). Data driven nonlinear dynamical systems identification using
multi-step CLDNN. AIP Advances, 9.8, p. 085311.
95. Todorov, Y., Koprinkova-Hristova, P., & Terziyska, M. (2017). Intuitionistic fuzzy radial basis
functions network for modeling of nonlinear dynamics. In 2017 21st International Conference
on Process Control (PC) (pp. 410–415). IEEE.
96. Van Truc, N., & Anh, D. T. (2018). Chaotic time series prediction using radial basis function
networks. 2018 4th International Conference on Green Technology and Sustainable Devel-
opment (GTSD) (pp. 753–758). IEEE.
97. Verdes, P. F., et al. (1998). Forecasting chaotic time series: Global versus local methods. Novel
Intelligent Automation and Control Systems, 1, 129–145.
98. Vlachas, P. R., et al. (2018). Data-driven forecasting of high-dimensional chaotic systems
with long short-term memory networks. In Proceedings of the Royal Society A: Mathematical,
Physical and Engineering Sciences (Vol. 474.2213, p. 20170844).
99. Vlachas, P. R. et al. (2020). Backpropagation algorithms and Reservoir Computing in Recur-
rent Neural Networks for the forecasting of complex spatiotemporal dynamics. Neural Net-
works, 126, 191–217.
100. Wan, Z. Y., et al. (2018). Data-assisted reduced-order modeling of extreme events in complex
dynamical systems. PLoS One, 13.5, e0197704.
101. Wang, R., Kalnay, E., & Balachandran, B. (2019). Neural machine-based forecasting of
chaotic dynamics. Nonlinear Dynamics, 98.4, 2903–2917.
102. Weng, T., et al. (2019). Synchronization of chaotic systems and their machine-learning models.
Physical Review E, 99.4, 042203.
103. Woolley, J. W., Agarwal, P. K., & Baker, J. (2010). Modeling and prediction of chaotic systems
with artificial neural networks. International Journal for Numerical Methods in Fuids, 63.8,
989–1004.
104. Wu, K. J., & Wang, T. J. (2013). Prediction of chaotic time series based on RBF neural network
optimization. Computer Engineering, 39.10, 208–216.
105. Wu, X., et al. (2014). Multi-step prediction of time series with random missing data. Applied
Mathematical Modelling, 38.14, 3512–3522.
106. Xin, B., & Peng, W. (2020). Prediction for chaotic time series-based AE-CNN and transfer
learning. Complexity, 2680480.
107. Yanan, G., Xiaoqun, C., & Kecheng, P. (2020). Chaotic system prediction using data assimi-
lation and machine learning. In E3S Web of Conferences (Vol. 185, p. 02025).
108. Yang, H. Y. et al. (2006). Fuzzy neural very-short-term load forecasting based on chaotic
dynamics reconstruction. Chaos, Solitons and Fractals, 29.2, 462–469.
109. Yang, F.-P., & Lee, S.-J. (2008). Applying soft computing for forecasting chaotic time series.
In 2008 IEEE International Conference on Granular Computing (pp. 718–723), IEEE.
110. Yeh, J.-P. (2007). Identifying chaotic systems using a fuzzy model coupled with a linear plant.
Chaos, Solitons and Fractals, 32.3, 1178–1187.
111. Yeo, K. (2019). Data-driven reconstruction of nonlinear dynamics from sparse observation.
Journal of Computational Physics, 395, 671–689.
112. Yosinski, J. et al. (2014). How transferable are features in deep neural networks? In Proceed-
ings of the 28th Conference on Neural Information Processing Systems, 27, 3320–3328.
113. Yu, R., Zheng, S., & Liu, Y. (2017). Learning chaotic dynamics using tensor recurrent neural
networks. Proceedings of the ICML. In ICML 17 Workshop on Deep Structured Prediction.
114. Yuxia, H., & Hongtao, Z. (2012). Chaos optimization method of SVM parameters selection
for chaotic time series forecasting. Physics Procedia, 25, 588–594.
115. Zhang, C., et al. (2020). Predicting phase and sensing phase coherence in chaotic systems with
machine learning. Chaos: An Interdisciplinary Journal of Nonlinear Science, 30.8, 083114.
10 1 Introduction to Chaotic Dynamics’ Forecasting

116. Zhang, J.-S., & Xiao, X.-C. (2000). Predicting chaotic time series using recurrent neural
network. Chinese Physics Letters, 17.2, 88.
117. Zhang, J., Shu-Hung Chung, H., & Lo, W.-L. (2008). Chaotic time series prediction using a
neuro-fuzzy system with time-delay coordinates. IEEE Transactions on Knowledge and Data
Engineering, 20.7, 956–964.
118. Zhu, Q., Ma, H., & Lin, W. (2019). Detecting unstable periodic orbits based only on time
series: When adaptive delayed feedback control meets reservoir computing. Chaos: An Inter-
disciplinary Journal of Nonlinear Science, 29.9, 093125.
Chapter 2
Basic Concepts of Chaos Theory and
Nonlinear Time-Series Analysis

Abstract We introduce the basic concepts and methods to formalize and analyze
deterministic chaos, with links to fractal geometry. A chaotic dynamic is produced by
several kinds of deterministic nonlinear systems. We introduce the class of discrete-
time autonomous systems so that an output time series can directly represent data
measurements in a real system. The two basic concepts defining chaos are that of
attractor—a bounded subset of the state space attracting trajectories that originate in
a larger region—and that of sensitivity to initial conditions—the exponential diver-
gence of two nearby trajectories within the attractor. The latter is what makes chaotic
dynamics unpredictable beyond a characteristic time scale. This is quantified by the
well-known Lyapunov exponents, which measure the average exponential rates of
divergence (if positive) or convergence (if negative) of a perturbation of a reference
trajectory along independent directions. When a model is not available, an attrac-
tor can be estimated in the space of delayed outputs, that is, using a finite moving
window on the data time series as state vector along the trajectory.

The concept of chaos, more precisely deterministic chaos, is delicate to define, as it

is usually linked to the concept of predictability, such as in the iconic sentence by
Edward Lorenz: “Chaos: when the present determines the future, but the approximate
present does not approximately determine the future”.
In fact, it is much easier to list the properties of a generic chaotic system rather
than giving a precise definition of chaos. Roughly speaking, a system is defined as
chaotic if:
• Its trajectories remain bounded, meaning that, in the long term, the dynamics are
attracted and remain inside a region of the phase space rather than escaping off to
infinity (boundedness).
• It is sensitive to initial conditions (sensitivity), i.e., tiny perturbations to the system
state become locally amplified in an exponential way, so that two nearby initial
conditions soon evolve into very different states, making the system unpredictable
in the mid- to long- term. In popular culture, this property is known as the butterfly
effect.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 11

M. Sangiorgio et al., Deep Learning in Multi-step Prediction of Chaotic Dynamics,
PoliMI SpringerBriefs,
https://doi.org/10.1007/978-3-030-94482-7_2
12 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

These two features produce the aperiodic alternation between expansion and con-
traction phases of the distance between the system’s trajectories [13]. These are the
two characterizing forces of chaotic dynamics which take the names of “stretching”
and “folding”
In the following section (Sect. 2.1), we introduce the class of dynamical systems
that we use as models for real systems and we formalize the first of the above key
properties, i.e., the boundedness of the system’s trajectories on a so-called attractor.
We opt for a description in discrete time only, because it is more directly applicable
to data in real systems. Analogous concepts for continuous-time models can be found
in standard textbooks, such as [1, 10]. Then, in Sect. 2.2, we formalize the second key
property of chaos, the sensitivity to initial conditions, in terms of the well-known
Lyapunov exponents. Based on the two concepts of system’s attractors and their
Lyapunov exponents, in Sect. 2.3 we finally define a chaotic attractor and we discuss
its typical fractal geometry. Indeed, chaotic attractors typically present self-similar
complex structures and have a non-integer dimension. For this reason, they are also
called strange attractors. Section 2.4 extends the two key concepts of attractor and
Lyapunov exponents to the case in which a data time series is available, instead of a
mathematical model of the system.

2.1 Dynamical Systems and Their Attractors

Let’s consider a generic discrete-time system (also called map), whose evolution in
one time step is defined by the difference equation:

x(t + 1) = f (x(t)), (2.1)

where x(t) is the n-dimensional state vector [x1 (t), x2 (t), . . . , xn (t)] , and f (·) :
Rn → Rn is the vector-valued function (or map) whose components f 1 (·), f 2 (·),
. . . , f n (·) set the right-hand sides of the state equations in (2.1).
Given an initial condition x0 at time zero, i.e., x(0) = x0 , the sequence of state
points x(t), t = 0, 1, . . . , T is the arc of the system’s trajectory, forward in time (T >
0), that originates from x0 . If function f (·) is invertible everywhere, the trajectory is
uniquely defined also backward in time and the system is said to be reversible.
Note that system (2.1) is autonomous and time-independent, i.e., there is no exter-
nal input and no time-dependence explicitly affecting function f (·). Such dependen-
cies can be taken into account by letting some of the parameters defining function
f (·) be externally driven, as done in Chap. 3.
In the long term, besides the possibility of diverging to infinity, which should
never be the case in a realistic model of a real system, the trajectories of system (2.1)
typically converge toward a set of state values, which takes the name of attractor.
Formally speaking, an attractor is a closed subset A of the state space characterized
by the following three properties:
2.1 Dynamical Systems and Their Attractors 13

• A is an invariant set under function f (·), i.e., if x(t) is a point of A, so is x(t + 1) =

f (x(t)).
• There exists an open set B, called the basin of attraction for A, which consists of
all the initial conditions x(0) that generate trajectories converging forward in time
to A.
• A is minimal, in the sense that there is no proper subset of A matching the first
two properties.
To be comprehensive, the minimality property is equivalently replaced by some
authors by
• the presence of a trajectory densely visiting A;
• or by the following transitivity property: given any two points x  and x  of A, the
evolution of any open set of initial conditions containing x  intersect any open set
containing x  .
Attractors can be:
• Fixed points, states x̄ such that f (x̄) = x̄, corresponding to stationary trajectories.
• Cycles of period T > 1, sequence of T states periodically mapped by function
f (·).
• Tori (more precisely 1-dim closed curves or d-dim tori, d > 1) densely filled by
quasi-periodic trajectories.
• Strange attractors, simply speaking attractors not belonging to the above three
categories.
Objects of the above four categories can also be present in the system’s state space,
without being attractors. They can repel nearby trajectories, being called repellers,
or attract some and repel others, being called saddles. In particular, the system can
have multiple attractors and the boundaries of their attraction basins are typically
composed of the initial conditions attracted by saddle objects, their so-called stable
manifolds.
Essentially, a strange attractor is the closure of a backbone structure of infinitely
many (but countable) saddle equilibria and cycles. Analogously to how the rational
numbers (a countable infinite set) densely fill the reals, the backbone of saddle objects
and their stable manifolds fill the attractor. Most of the initial conditions in the
attractor (technically almost all, i.e., all but a set of zero measure in the state space)
do not belong to such objects and manifolds, so that the corresponding trajectories
end up visiting (densely) the whole attractor. And so do most of the trajectories that
originate in the basin of attraction. We say that this happens “generically” or that it
is true for the “generic” initial condition or trajectory, i.e., true with probability one
if the initial condition is chosen at random.
Finally, there can also be objects that neither attract nor repel. These are said
to be stable, but not asymptotically stable. Their presence is typically due to some
simplifying modeling assumptions, such as the absence of friction in mechanics,
that introduce the conservation of some energy-like quantity along the system’s
trajectories.
14 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

A complete treatment of the stability of equilibria, cycles, tori, and strange objects
is out of the scope of this chapter and the reader is redirected to the above-mentioned
standard textbooks [1, 10]. In particular, we will only consider systems with a unique
attractor, or limit the analysis to the initial conditions in the basin of the attractor that
is relevant for the case under study. Because our aim is to design neural predictors
able to perform in the most complex conditions, we will consider the case of chaotic
attractors, which are strange attractors characterized by the property of being sensitive
to initial conditions. As already anticipated, this property is quantified by the so-
called Lyapunov exponents of the attractor, which is the topic of the next section.

2.2 Lyapunov Exponents

The formal way to quantify the extent to which a given trajectory is sensitive to the
perturbation of its initial conditions is by its so-called Lyapunov exponents (LEs).
Intuitively, in nonlinear systems, the LEs play the same role of the eigenvalues in
linear systems.
To compute the LEs associated to a trajectory x(t), t ≥ 0, taken as reference, it is
necessary to define a perturbed trajectory x(t) + δx(t) and to linearize the evolution
of the perturbation vector δx(t) along the reference trajectory x(t). The results is the
so-called variational equation:

δx(t + 1) = J (x(t))δx(t), (2.2)

where J (x) denotes the Jacobian matrix of function f (·) at x

⎡ ∂ f1 ⎤
· · · ∂∂xf1n
∂ x1
⎢ ⎥
= ⎣ ... . . . ... ⎦
df
J (x) = (2.3)
dx ∂ fn
∂ x1
· · · ∂∂ xfnn

and the product J (x(t))δx(t) is the standard row-column matrix-vector product.

Starting with an initial perturbation δx(0) of length ε, the perturbation evolves
up to first-order in accordance with the variational equation (2.2) and the perturbed
state x(t) + δx(t) lies, at time t, on the surface of an ellipsoid (see Fig. 2.1). Changing
the initial perturbation δx(0) (still of length ε) gives another point x(t) + δx(t) on
the same ellipsoid’s surface. The ellipsoid depends on the initial state x(0) and on
the final time t of the trajectory arc. The length ε of the initial perturbation only sets
the scale of the ellipsoid due to the linearity of Eq. (2.2).
Up to first-order, the final perturbation δx(t) is linearly linked to the initial one
by the fundamental solution matrix

M(t) = J (x(t − 1))J (x(t − 2)) . . . J (x(1))J (x(0)) (2.4)

2.2 Lyapunov Exponents 15

Fig. 2.1 First-order

approximated evolution of
the hypersphere of
perturbations δx(0) of length
ε. After t iterations of the
map (2.1), the hypersphere is
transformed by the
variational equation (2.2)
into an (hyper-)ellipsoid,
with symmetry axes longer
than ε along expanding
directions and shorter than ε
along contracting directions.
The concept is illustrated in
two dimensions (n = 2)

of the variational equation (2.2), i.e.,

δx(t) = M(t)δx(0). (2.5)

In other words, the i-th column of M(t) is the solution of (2.2) at time t starting
from the natural basis vector δx(0) = v (i) , with v (i) (i)
j = 0 for j  = i and vi = 1,
i = 1, . . . n.
The matrix M(t) is generically nonsingular (it is certainly so for reversible sys-
tems) and defines the ellipsoid at time t. Indeed, from

ε2 = δx(0) δx(0) = δx(t) (M(t)−1 ) M(t)−1 δx(t)

(2.6)
= δx(t) (M(t)M(t) )−1 δx(t),

it follows that the ellipsoid equation

δx(t) E(t)δx(t) = ε2 (2.7)

holds at any t ≥ 0 with

E(t) = (M(t)M(t) )−1 . (2.8)

The ellipsoid’s symmetry axes are the eigenvectors of matrix E(t). Geometri-
cally, a symmetry axis is a vector r from the ellipsoid center that is aligned with
the gradient E(t)r of the quadratic form 21 r  E(t)r . Denoting by r (i) (t) the sym-
metry axes associated to the eigenvalue √ λi (t) of E(t), the alignment translates into
E(t)r (i) (t) = λi (t)r (i) (t), where 1/ λi (t) = σi (t) are the so-called singular values
16 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

of M(t), i = 1, . . . n (the square roots of the eigenvalues of E(t)−1 = M(t)M(t) ).

Recalling that the eigenvectors of a nonsingular matrix coincide with those of the
inverse, the ellipsoid’s symmetry axes are more commonly referred to as the eigen-
vectors of M(t)M(t) associated to its eigenvalues σi2 (t).
The length ri (t) = r (i) (t) of the i-th symmetry axis is such that the axis points
to the ellipsoid surface, that means that r (i) (t) has to solve the ellipsoid equation
(2.7), i.e.,
r (i) (t) E(t)r (i) (t) = ε2 . (2.9)

Substituting r (i) (t) = (1/σi2 (t))E(t)−1r (i) (t) (the eigenvector equation) in lieu of
the second occurrence of r (i) into Eq. (2.9) yields

ri (t) = εσi (t). (2.10)

Note that the initial perturbations r (i) (0), 1, . . . , n, that are mapped by the varia-
tional equation (2.2) into the ellipsoid’s symmetry axes at time t (see Fig. 2.1), are
the eigenvectors of M(t) M(t) associated to the same singular value σi (t) of M(t).
Indeed, by left-multiplying both sides of r (i) (0) = M(t)−1r (i) (t) by (M(t)−1 ) ,
exploiting at the righthand side the eigen-property of r (i) (t) and Eq. (2.5) to get
(M(t)−1 )r (i) (0) = (1/σi2 (t))M(t)r (i) (0), and left-multiplying both sides by M(t) ,
we get
r (i) (0) = (1/σi2 (t))M(t) M(t)r (i) (0). (2.11)

2.2.1 Average Exponents

The Lyapunov exponents (LEs), or Lyapunov characteristic exponents, are, by defi-

nition, the log of the average rates of growth of the ellipsoid’s symmetry axes, i.e.,
 1/t
ri (t) 1
L i = lim log = lim log σi (t). (2.12)
t→+∞ ε t→+∞ t

Indeed, ri (t)/ε = t−1 k=0 ri (k + 1)/ri (k) is the product of all the t rates of growth of
the i-th axis along the reference trajectory and the (1/t)-power takes the geometric
average (e.g., the average between the rates 2—doubling in one unit of time—and
1/2—halving in one unit of time—is (2 × 0.5)1/2 = 1). The limits in (2.12) exist
and are finite provided the reference solution x(t) does exist. Note that, by definition,
L 1 ≥ L 2 ≥ · · · ≥ L n , though one does not know a priori the length ordering among
the ellipsoid’s symmetry axes. Actually, the ordering might change along the trajec-
tory x(t) (especially in the initial phase), so that, in principle, one first computes the
limits following the ellipsoid’s symmetry axes in arbitrary order and then sort the
results (see [15] for a review of the algorithms to numerically compute the LEs). In
2.2 Lyapunov Exponents 17

the following, we will refer to the first symmetry axis r (1) (t) as the longest, to r (2) (t)
as the second-longest, etc., i.e., r1 (t) ≥ r2 (t) ≥ · · · ≥ rn (t), for sufficiently large t.
The LEs hence measure the average exponential rate of divergence (if positive)
or convergence (if negative) of perturbed trajectories with respect to the reference
one (the one starting from x(0)). Generically, the perturbation δx(t) will have a
nonzero component along the longest symmetry axes r (1) (t), so that L 1 , the largest
Lyapunov exponent (LLE), is the dominant rate (average, exponential) of diver-
gence/convergence of the approximate perturbed trajectory (approximated by the
variational equation (2.2)). That is, the size of the perturbation grows/decays (on
average) as ε exp(L 1 t). This is approximately true also for the true perturbed trajec-
tory (the solution of the nonlinear system (2.1) from the perturbed initial condition
x(0) + δx(0)) only if the size ε of the initial perturbation is so small that the perturbed
trajectory remains close to the reference one up to time t. However, if L 1 > 0, the
perturbed trajectory sooner of later leaves the reference one, so that the linearization
taken to define the LEs no longer represents the behavior of the perturbation.
The interpretation of the remaining exponents is more delicate. L 2 is the greatest
rate chosen from all directions orthogonal to r (1) (t). Similarly, L 3 is the greatest rate
chosen from all directions orthogonal to both r (1) (t) and r (2) (t), and so on. How-
ever, only those initial perturbations with no component along r (1) (0) (the initial
perturbation mapped into r (1) (t) by the variational equation (2.2)) generate (approx-
imate) perturbations at time t orthogonal to r (1) (t). The problem is that considering
a different time t modifies the initial perturbation r (1) (0) (it is an eigenvector of the
time-dependent matrix M(t) M(t)), so that the (approximate) perturbation δx(t)
still grows/decays (on average) as ε exp(L 1 t).
The correct interpretation of the non-leading exponents is in terms of partial sums.
The sum L 1 + L 2 is the dominant rate (average, exponential) for the area of two-
dimensional sets of perturbations, as the components along r (3) (t), . . . , r (n) (t) are
dominated (for sufficiently large t) by those along r (1) (t) and r (2) (t) for all the pertur-
k
bations in the set. Similarly, i=1 L i is the dominant rate (average, exponential) for
the k-dimensional measure (or hypervolume) of k-dimensional sets of perturbations.
Finally, if the trajectory x(t) converge to an attractor A, the resulting limits are
generically independent of the initial condition x(0) in the basin of attraction of
A. This is certainly true for attracting equilibria, cycles, and tori, whereas it is only
generic (i.e., true for almost all initial conditions in the basin of attraction) for strange
attractors. In principle, starting from an initial condition on the stable manifold of
one of the saddle objects composing the attractor’s backbone, the resulting LEs are
those characterizing the reached saddle, though unavoidable numerical errors make
the trajectory miss the saddle and generically visit the whole attractor.
Exponentially attracting equilibria and cycles have negative LEs, whereas some
LEs are null if the speed of attraction is less than exponential (e.g., a time power
law); d-dimensional tori have d null exponents, because the internal perturbations are
neither expanded nor contracted (on average), the remaining ones being negative in
case of exponential attraction; strange attractors typically have at least one positive
exponent, responsible of the divergence of the unstable objects in their backbone
(this exponent is null if the speed of divergence is weaker than exponential; these
18 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

attractors are called weakly chaotic). The sum of the attractor’s exponents is in any
case negative, at least in reversible systems, because n-dimensional volumes of initial
conditions in the basin of attractions spread, forward in time, on a lower-dimensional
object. As it will be recalled in Sect. 2.3, a chaotic attractor is formally defined as an
attractor characterized by L 1 > 0.

2.2.2 Local Exponents

The local Lyapunov exponents (loc-LEs) L (loc) 1 (x(t)) ≥ L (loc)

2 (x(t)) ≥ · · · ≥ L (loc)
n
(x(t)) are exponential rates of divergence/convergence faced by the perturbation
δx(t), locally to the point x(t) of the reference trajectory, along n orthogonal
directions r (loc-i) (x(t)) such that r (loc-1) (x(t)) is the dominant one (the direction
associated with the fastest expansion if L (loc) 1 (x(t)) > 0 or slowest contraction if
(loc) (loc-2)
L 1 (x(t)) < 0), r (x(t)) is the second-dominant direction (the dominant one
among those orthogonal to r (loc-1) (x(t))), etc. Note that the loc-LEs are characteris-
tic of the point x(t) of the trajectory and do not depend on the time t at which the
trajectory visits the point.
With reference to a generic point x, the loc-LEs (and corresponding directions)
are given by the symmetry axes (lengths and directions) of the ellipsoid formed after
one time step by a sphere of perturbation around x, i.e.

ri (1)
L i(loc) (x) = log . (2.13)
ε x(0)=x

From the ellipsoid analysis developed in Subsect. 2.2.1, it follows that the symmetry
axes at time 1 are the eigenvectors of J (x(t))J (x(t)) . The loc-LEs are therefore the
log of the singular values of the system’s Jacobian at x(t).
Note that the (average) LEs are not the (arithmetic) time average of the loc-LEs
along the reference trajectory. For example, L (loc) 1 (x(t)) is, at any t, the log of the
largest local rate along r (loc-1) (x(t)). In general, however, the longest symmetry axis
r (1) (t) of the ellipsoid originated at x(0) is not aligned at time t, with r (loc-1) (x(t)), so
that only its component along r (loc-1) (x(t)) is subject to the largest rate. Consequently,
the time average of L (loc)
1 (x(t)) is generically larger than L 1 . Similarly, the symmetry
axis r (i) (t), i ≥ 2, is not aligned, at time t, with r (loc-i) (x(t)) and generically has
nonzero component along r (loc- j) (x(t)), j < i, so that there is not even a clear sign
relation between the time average of L i(loc) (x(t)) and L i .
2.3 Chaotic Systems, Predictability, and Fractal Geometry 19

2.3 Chaotic Systems, Predictability, and Fractal Geometry

2.3.1 Chaotic Attractors and Lyapunov Time Scale

Making use of the concepts defined in the previous sections, we now formally intro-
duce the concept of chaotic attractor, i.e., a closed subset of the state space (invariant
under f (·), with its own basin of attraction, and minimal) having at least one posi-
tive Lyapunov exponent (i.e., L 1 > 0). When there are at least two positive LEs, the
attractor is defined as hyperchaotic. A dynamical system with (at least) one chaotic
attractor, is said to be chaotic.
The two features, boundedness and sensitivity, presented at the beginning of this
chapter to intuitively characterize the chaotic systems, correspond to the formal
properties listed above. The sensitivity to initial conditions is guaranteed by the
presence of at least one positive LE. A largest Lyapunov exponent (LLE), L 1 , greater
than 0 is a necessary and sufficient condition to ensure the exponential divergence of
any two nearby trajectories (“stretching”). However, if the initial distance between
two points is finite (not infinitesimal), the divergence cannot continue indefinitely
because of the presence of the attractor, which is bounded by definition. This means
that the system nonlinearities will somehow take the two trajectories close again
(“folding”).
An illustrative representation of stretching and folding is provided in Fig. 2.2.
The formal definition of chaotic attractor reported at the beginning of this sub-
section allows specifying the distinction between the terms “chaotic” and “strange”,
which are almost always used interchangeably. An attractor is chaotic if its largest
characteristic exponent L 1 is positive. As we have seen, this means that there is an
exponential sensitivity to initial conditions. On the other hand, strange attractors
do exhibit a sensitive dependence on the initial condition, because of the saddle
objects composing their internal structure. However, the divergence of nearby tra-

Fig. 2.2 Evolution in time

of a generic state variable xi .
The two trajectories, starting
from nearby initial
conditions, diverge
exponentially (“stretching”).
After that, they get close
together (“folding”)
20 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

Fig. 2.3 Example of a chaotic (blue) and a random (red) time series. Despite they seem to behave
similarly (a), the blue dynamic is deterministic as demonstrated by the mapping between y(t − 1)
and y(t) (b)

jectories can be slower than exponential and therefore yield a zero limit in (2.12)
for L 1 . Hence, chaotic attractors are also strange, while strange attractors can be
non-chaotic. Strange non-chaotic attractors are also called “weakly chaotic”.
In chaotic regime, the generic system’s trajectory shows some peculiar properties:
• Despite its random appearance, a chaotic process is deterministic [2] (see Fig. 2.3),
i.e., with the exact same initial conditions, it will always evolve over time in
the same way. However, numerical issues (e.g., round-off errors) can generate
dynamics that seem to suggest the presence of some source of randomness [12,
21]. For instance, two numerical simulations of the same system, starting from the
same initial condition, performed by different hardware architectures, can result
in two distinct trajectories visiting the attractor in apparently unrelated ways.
• The trajectories never return to a state already visited (i.e., non-periodicity), but
pass arbitrarily close to it. This happens provided that one waits for a sufficient
amount of time, because the attractor contains a dense trajectory. In principle,
a chaotic system may have sequences of values that exactly repeat themselves
(periodic behavior). However, as described in Sect. 2.1, such periodic sequences
are repelling rather than attracting, meaning that if the variable is outside the
sequence, it will never enter the sequence and in fact, will diverge from it. In this
terms, for almost all initial conditions, the variable evolves chaotically with non-
periodic behavior. The non-periodicity typical of chaotic dynamics has not to be
confused with that of quasi-periodic signals. The difference between the two is
usually investigated in the frequency domain. The spectrum of chaotic signals is
typically distributed on a range of values, while the one of quasi-periodic signals
is concentrated on a finite group of (few) peculiar values.
• The trajectories display complex paths, usually building geometries with a fractal
structure: the strange attractors presented in Sect. 2.1.
In the context of time series forecasting, the presence of stretching is the criti-
cal factor, even if one has the perfect predictor available. Indeed, the exponential
sensitivity to initial condition, soon amplifies the tiniest measurement error or unde-
sired noise on the data feeding the predictor. This is why we often say that “chaotic
2.3 Chaotic Systems, Predictability, and Fractal Geometry 21

dynamics are unpredictable”. The focus of this book is actually to investigate how
far we can go.
Starting from L 1 , it is possible to compute the Lyapunov time (LT) of the system,
which represents the characteristic time scale on which a dynamical system is chaotic.
By convention, the LT is defined as the inverse of L 1 , as the time period for the distance
between nearby trajectories of the system to increase by a factor of e. The concept of
LT is crucial in forecasting tasks, because it mirrors the limits of the predictability
of a chaotic system and allows to fairly compare the predictive accuracy in different
systems [5, 14, 17, 18, 22].
In accordance with what we have done in Sect. 2.2, it is also possible to define the
local Lyapunov time (loc-LT).

2.3.2 Correlation and Lyapunov Dimensions

The geometry of chaotic attractors is usually complex and difficult to describe. It is

thus useful to define quantitative properties to characterize these geometrical objects.
Dimension is perhaps the most basic property of an attractor. However, the concept
of dimension is somewhat ambiguous, since it can have a variety of different defini-
tions [8]. In this subsection, we present two alternatives for measuring the attractor
dimensions: the correlation and Lyapunov dimensions.
The correlation dimension is a fractal dimension specifically designed for the
attractors of a dynamical system. Consider a target system’s attractor A and let
{x(t), t = 0, . . . , T } be an arc of trajectory on the attractor A. For any given radius
r > 0, define the correlation function

# pairs (x(t  ), x(t  )) : x(t  ) − x(t  ) < r, 0 ≤ t  < t  ≤ T

C(r, T ) = , (2.14)
T (T + 1)/2

where T (T + 1)/2 is the number of point pairs in the trajectory arc. The correlation
function increases from 0 to 1 as r increases from 0 to the maximum distance between
the points in the trajectory. Let d, possibly non-integer, be the attractor’s dimension to
be determined. For small r , the number of points x(t) that are r -close to a given point
x ∈ A increases linearly with the length T of the trajectory arc and proportionally
to r d . Think, for example, to the case d = 1 (a one-dim closed curve densely filled
by a quasi-periodic trajectory). The segment of the curve contained in the r -ball
centered at x has a length proportional to r d and is filled by the trajectory as T goes
to infinity. For large T and small r , the numerator of the correlation function is hence
proportional to T 2 r d (there are T points each with a number of r -close neighbors
proportional to T r d ), while the denominator has order T 2 (see (2.14)). Solving for
d gives the correlation dimension

log C(r, T )
dcorr (A) = lim lim . (2.15)
r →0 T →∞ log r
22 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

Fig. 2.4 Schematic

illustration of the estimation
procedure of the correlation
dimension. The estimate is
obtained as dcorr (A) =
 log C(r )/ log r

In practice, we work with finite r and T . Starting from an initial condition x(0) in
the basin of attraction of the target attractor, we discard the initial trajectory transient
(say, the first t0 points) and then take T + 1 points x(t0 ), x(t0 + 1), . . . , x(t0 + T ), for
sufficiently large t0 and T . We then compute the correlation function for a sequence
of small increasing values of the radius r . Plotting log C(r ) versus log r , there will be
an interval of r in which the obtained points will approximately align. The correlation
dimension dcorr (A) is given by the slope of this linear part of the graph (see, Fig. 2.4).
To well identify the linear part of the graph, it is worth to discuss how the graph gets
distorted for too small and too large values of the radius r . The behavior for small r
is due to the finite resolution with which our finite arc of trajectory fills the attractor.
When r is smaller than such a resolution, the correlation function underestimates
the density of the attractor points within distance r and scaling this underestimate
with r d necessarily results in a d larger than the actual attractor’s dimension. This is
evident in the expression (2.15) by considering smaller and smaller r at constant T :
C(r, T ) vanishes below a resolution threshold on r , so that the numerator diverges
to – infinity, while the denominator is still finite. Vice-versa, for too large values
of r , the boundedness of the attractor makes the correlation function saturate at
1 (independently on how large is T ) and the slope of the graph correspondingly
vanishes (see again Fig. 2.4).
When the full spectrum of the attractor’s LEs is available, a simple way of esti-
mating the attractor’s dimension is the well-known Kaplan-Yorke formula [9], which
approximates the concept of Lyapunov dimension. The geometric idea behind the
concept is related to the interpretation given in Sunsect. 2.2.1 of the partial sums
of the attractor’s LEs. According to that interpretation, the Lyapunov dimension
dLyap (A) is the dimension of a small set of initial conditions, in the basin of attraction,
with dLyap (A)-dimensional measure (hypervolume) that neither grows nor decays (on
average), as the corresponding trajectories converge to the attractor. Because the tra-
jectories densely visit the attractor, this occurs when the set of initial conditions has
the same dimension of the attractor.
The dimension could however be non-integer, so that the best one can do is to
k k+1
look for the index k for which i=1 L i ≥ 0 and i=1 L i < 0. The hypervolume
of a k-dimensional set of initial conditions grows (on average), while the one of a
(k + 1)-dimensional set does vanish, while converging to the attractor, so that all we
2.3 Chaotic Systems, Predictability, and Fractal Geometry 23

Fig. 2.5 Schematic

illustration of the
Kaplan-Yorke estimation of
the fractal dimension

can say is that the dimension d is in between k and k + 1. As illustrated in Fig. 2.5,
the Kaplan-Yorke formula simply considers a continuous piece-wise interpolation
of the partial sums of the attractor’s LEs. The Lyapunov dimension dLyap (A) is
therefore approximated with the value at which the piece-wise interpolation crosses
the horizontal axis, the latter representing the zero-rate of expansion/contraction.
The result is the following formula:
k
Li
dKY (A) = k + i=1
. (2.16)
|L k+1 |

To conclude this subsection on attractors dimension, we clarify the relationship

between fractals and strange attractors. So far, we described these latter as complex
geometrical objects which usually have a non-integer dimension. In fact, these two
features (being strange and having a fractal structure) are generally strongly related,
but not exactly equivalent. The attractors characterized by a non-integer dimension
are always strange. Conversely, it is possible to find some peculiar cases where a
strange attractor has integer dimension. A classical example is the logistic map with
a parameter (i.e., the growth rate) equal to 4 (the system is presented in Subsect 3.1.1).
In such a case, it can be proved that the chaotic orbit is dense in the unit interval
[0, 1], and thus its dimension is equal to 1 (see [1], page 235).

2.4 Attractor Reconstruction from Data

The concepts presented in the first part of this chapter require to know the state
space representation of the dynamical system. In practical applications, the equations
describing the evolution of the state variables (and also the whole state vector itself)
24 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

are usually unknown. Most of the time, only a time series of observed values of the
system output y(t) = g(x(t)), is available.

2.4.1 Delay-Coordinate Embedding

Starting from a sequence of data, it is possible to build a m-dimensional delayed

version of the original phase space by considering the delay coordinates Y (t) =
[y(t), y(t − 1), . . . , y(t − m + 1)] for each time t in the time series (Fig. 2.3). This
procedure is known as delay-coordinate reconstruction, or delay-coordinate embed-
ding.
The reconstruction is extremely useful because, under broad conditions, it pre-
serves the dynamical properties of the original system, including, for instance, its
asymptotic behavior and the LEs (the first k, being k the index of the Kaplan-Yorke
formula, see Subsect. 2.4.2 for further details).
More specifically, the Takens’ theorem [20] demonstrates that a smooth attractor
can be reconstructed from a univariate sequence of observations made with a generic
function g(·) if the embedding dimension, m, is large enough. Hence, the theorem
forms a bridge between the theory of nonlinear dynamical systems and the analysis
of time series.
The reconstructed attractor, which lies in the m-dimensional delay-coordinate
space, is topologically equivalent to the original attractor (in the initial n-dimensional
state space). In other words, the reconstructed dynamic, i.e., Y (t + 1) = F(Y (t)),
is equivalent to that of the original state space; more exactly, they are related by a
smooth, invertible change of coordinates (i.e., a diffeomorphism). Thanks to these
properties of the mapping between the original and the reconstructed spaces, every
state in the phase space of the system can be uniquely represented by the output data.
The reconstructing procedure, however, does not preserve the geometric shape of the
structures of the original attractor.
Under a geometrical perspective, the idea behind Takens’ theorem is to unfold
the reconstruction of the trajectories sufficiently to avoid self-crossing. Trajectories
cannot cross because a deterministic dynamical system’s evolution is fully defined
by its current state. This means that starting from a given point in phase space, the
next point is uniquely determined. As a consequence, two trajectories meeting at the
same point in the phase space cannot evolve in two different ways.
In practice, due the numerical issues and the possible presence of noise, one
would verify that, in a space with a sufficient embedding dimension, two nearby
points should be mapped by a single iteration of the function F(·) in relatively
close positions. If this is not the case, i.e., the points evolve in completely different
positions after one time step, this is probably due to the fact that we are in presence
of self-crossings caused by an insufficient embedding dimension. For large enough
m, these points will move away from each other. This situation, however, can occur
also when using an appropriate value of m, if F(·) is highly irregular and has steep
valleys and ridges. It is a fairly uncommon case because deterministic chaos is usually
2.4 Attractor Reconstruction from Data 25

characterized by simple maps, which produce unexpectedly complex dynamics when

applied iteratively, but its occurrence cannot be excluded a priori.
The practical way usually adopted to estimate the minimal embedding dimension
from data is the so-called false nearest neighbor algorithm [11]. The idea behind this
algorithm is to examine how the number of neighbors of a point in the reconstructed
space varies with increasing embedding dimension. When the embedding dimension
is too low, many of the points that are in reality far apart could seem to be neighbors
as a consequence of projecting into a space of smaller dimension. These points are
called false neighbors. In an appropriate (or higher) embedding dimension, the false
neighbors will no longer be neighbors. Conversely, two points that are near to each
other in the sufficient embedding dimension should remain close as the dimension
increases.
A visual representation of the concept of true and false neighbor is reported in
Fig. 2.6. In a 1-dimensional embedding, the three considered points—denoted with
A, B, and C—seem to be neighbors. Increasing the embedding dimension by one, we
discover that A and B are still neighbors, while B and C are actually false neighbors.
Embedding the system in three dimensions, one would see that the points A and B
remain close, although C is still far from B. If the situation described above occurs for

Fig. 2.6 1- and 2-dimensional embeddings of a noise-free dataset obtained simulating the Hénon
map (presented in Sect. 3.1), with y(t) = x1 (t). The points A and B are true neighbors, while B
and C are false neighbors
26 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

Fig. 2.7 Fraction of false

nearest neighbors computed
for an increasing embedding
dimension (m)

all the points in the dataset, we will conclude that in this specific case, a 2-dimensional
space is sufficient to unfold the attractor.
Practically speaking, two points are considered as neighbors when their Euclidean
distance is lower than a certain threshold, that has to be properly selected depend-
ing on the specific application. The algorithm checks the neighbors in increasing
embedding dimensions until it finds only a negligible number of false neighbors
Fig. 2.7. This is chosen as the lowest embedding dimension, which is presumed to
give reconstruction without self-intersections between the trajectories.
If the data are noise-free, the percentage of false neighbors will drop to zero when
the proper dimension is reached. In the case of noisy time series, we expect that a
certain (hopefully small) number of false neighbors remain also for large embedding
dimension due to the presence of noise.
Takens’ theorem also states that, generically, the m has to be only a little larger,
more precisely, the first integer larger, than twice the dimension of the attractor. To
develop a geometric intuition about this fact, we can consider the following simple
example. Two 1-dimensional curves can have intersections in either R2 , R3 or any
higher-dimensional space. The difference is that a small perturbation will remove
the intersection in R3 , while in R2 it will only move the intersection somewhere
else. We can imagine that the two curves correspond to two invariant strands of the
attractor, e.g., two arcs belonging to the reconstruction of a 1-dimensional torus with
a quasi-periodic motion on it. A slight perturbation in the system dynamics would
generically cause the self-intersection in R3 to disappear.
In other words, although it is certainly possible for two generic curves placed at
random to have intersections in Rk for k ≥ 3, we should consider them as exceptional
situations and expect them to occur with essentially zero probability. This is the reason
why we used the term “generically” hereinbefore: if m > 2d, self-crossings should
in principle be rare, but they could still occur. When this is the case, it is necessary
to further increase m, even if it is already greater than 2d.
2.4 Attractor Reconstruction from Data 27

2.4.2 Estimation of the Largest Lyapunov Exponent

As reported in Sect. 2.2, the traditional algorithm for the computation of the LEs
makes use of the Jacobian matrix and thus requires knowledge of the state equations
of the system. However, when the model of the system is not available, it is necessary
to make use of statistical methods for detecting chaos [7].
Once an appropriate embedding dimension m of the dataset has been esti-
mated, the dynamic of the system can be analyzed in the delayed phase space
Y (t) = [y(t), y(t − 1), . . . , y(t − m + 1)] which shares the same topological prop-
erties (and thus also L 1 ) of the original phase space as stated by Takens’ theorem
[20]. After that, the following procedure [23] can be performed to estimate L 1 :
1. select N p pairs of nearby points in the m-dimensional delayed space, Y (ti ) and
Y (t j );
2. for each pair, compute the Euclidean distance between the two points δ p (0) =
Y (ti ) − Y (t j ) ;
3. recompute the distance between the two points of each pair after te time steps
δ p (te ) = Y (ti + te ) − Y (t j + te ) . On average, for small values of te , we expect
that δ p (te ) evolves following the exponential law:

δ p (te ) = δ p (0) · exp(L 1 te ); (2.17)

4. calculate Q(te ), namely the average logarithm of the divergence rate as:

1
Np  δ p (te ) 
Q(te ) = log . (2.18)
Np p=1
δ p (0)

Repeating the procedure for increasing values of the expansion step te , we can plot
Q(te ) as a function of te . In a chaotic system, the initial part of this curve is charac-
terized by a linear trend (see Fig. 2.8). This is due to the fact that two points initially
close together will diverge exponentially, and that this exponential expansion pro-
duces a linear trend because of the logarithm. After that, the average divergence tends
to a constant value because the trajectories lie inside the chaotic attractor. Finally,
the L 1 can be easily computed as the slope of the function Q(te ) in the initial part.
The algorithm presented in this subsection only provides an estimation of L 1 .
Alternative procedures for computing the largest exponent are available [6, 16, 24].
In principle, one could also make use of more complex procedures which allow us to
numerically compute the positive Lyapunov spectrum, i.e., only positive exponents
[23], or the whole spectrum of LEs [3, 4, 19]. However, even in this last case, only the
estimates of the first k exponents make sense. Going further would produce arbitrary
values since the data are typically sampled when the trajectories are already in the
attractor, meaning that they are representative of what happens inside it, while the
LEs beyond the k-th (k + 1, k + 2, …) describe the transient of convergence toward
28 2 Basic Concepts of Chaos Theory and Nonlinear Time-Series Analysis

Fig. 2.8 Schematic

illustration of the estimation
procedure of the LLE from
data. The estimate is
obtained as
L 1 = Q(te )/te

the attractor. In any case, the dimension of the original state space remains unknown,
thus we do not even know how many LEs should be computed.
In this book, we limit the analysis to L 1 because the computation of the LEs
spectrum presents computational issues which strongly affect the numerical stability
of the estimation [23]. Note, however, that in the context of time series prediction, we
are interested in deriving the Lyapunov time LT, which sets the limit of predictability
of the chaotic system. To this end, it is sufficient to know L 1 , since it defines the
system’s dominant dynamics.

References

1. Alligood, K. T., Sauer, T. D., & Yorke, J. A. (1996). Chaos. Springer.

2. Boeing, G. (2016). Visual analysis of nonlinear dynamical systems: Chaos, fractals, self-
similarity and the limits of prediction. Systems, 4.4, 37.
3. Briggs, K. (1990). An improved method for estimating Liapunov exponents of chaotic time
series. Physics Letters A, 151.1-2, 27–32.
4. Brown, R., Bryant, P., & Abarbanel, H. D. (1991). Computing the Lyapunov spectrum of a
dynamical system from an observed time series. Physical Review A, 43.6, 2787.
5. Dercole, F., Sangiorgio, M., & Schmirander, Y. (2020). An empirical assessment of the univer-
sality of ANNs to predict oscillatory time series. IFAC-PapersOnLine, 53.2, 1255–1260.
6. Ellner, S., et al. (1991). Convergence rates and data requirements for Jacobian-based estimates
of Lyapunov exponents from data. Physics Letters A, 153.6-7, 357–363.
7. Ellner, S., & Turchin, P. (1995). Chaos in a noisy world: New methods and evidence from
time-series analysis. The American Naturalist, 145.3, 343–375.
8. Farmer, J. D., Ott, E., & Yorke, J. A. (1983). The dimension of chaotic attractors. Physica D:
Nonlinear Phenomena, 7.1-3, 153–180.
9. Frederickson, P., et al. (1983). The Liapunov dimension of strange attractors. Journal of Dif-
ferential Equations, 49.2, 185–207.
10. Guckenheimer, J., & Holmes, P. (2013). Nonlinear oscillations, dynamical systems, and bifur-
cations of vector fields. (Vol. 42). Springer Science and Business Media.
11. Kennel, M. B., Brown, R., & Abarbanel, H. D. (1992). Determining embedding dimension for
phase-space reconstruction using a geometrical construction. Physical Review A, 45.6, 3403.
References 29

12. Nepomuceno, E. G., et al. (2019). Soft computing simulations of chaotic systems. International
Journal of Bifurcation and Chaos, 29.08, 1950112.
13. Ott, E. (2002). Chaos in dynamical systems. Cambridge University Press.
14. Pathak, J., et al. (2018). Model-free prediction of large spatiotemporally chaotic systems from
data: A reservoir computing approach. Physical Review Letters, 120.2, 024102.
15. Ramasubramanian, K., & Sriram, M. S. (2000). A comparative study of computation of Lya-
punov spectra with different algorithms. Physica D: Nonlinear Phenomena, 139.1-2, 72–86.
16. Rosenstein, M. T., Collins, J. J., & De Luca, C. J. (1993). A practical method for calculating
largest Lyapunov exponents from small data sets. Physica D: Nonlinear Phenomena, 65.1-2,
117–134.
17. Sangiorgio, M. (2021). Deep learning in multi-step forecasting of chaotic dynamics. Ph.D.
thesis. Department of Electronics, Information and Bioengineering, Politecnico di Milano.
18. Sangiorgio, M., & Dercole, F. (2020). Robustness of LSTM neural networks for multi-step
forecasting of chaotic time series. Chaos, Solitons and Fractals, 139, 110045.
19. Sano, M., & Sawada, Y. (1985). Measurement of the Lyapunov spectrum from a chaotic time
series. Physical Review Letters, 55.10, 1082.
20. Takens, F. (1981). Detecting strange attractors in turbulence. In: Dynamical systems and tur-
bulence, Warwick 1980 (pp. 366–381). Springer.
21. Ushio, T., & Hsu, C. (1987). Chaotic rounding error in digital control systems. IEEE Transac-
tions on Circuits and Systems, 34.2, 133–139.
22. Vlachas, P. R., et al. (2020). Backpropagation algorithms and Reservoir Computing in Recurrent
Neural Networks for the forecasting of complex spatiotemporal dynamics. Neural Networks,
126, 191–217.
23. Wolf, A., et al. (1985). Determining Lyapunov exponents from a time series. Physica D:
Nonlinear Phenomena, 16.3, 285–317.
24. Wright, J. (1984). Method for calculating a Lyapunov exponent. Physical Review A, 29.5, 2924.
Chapter 3
Artificial and Real-World Chaotic
Oscillators

Abstract Four archetypal chaotic maps are used to generate the noise-free syn-
thetic datasets for the forecasting task: the logistic and the Hénon maps, which are
the prototypes of chaos in non-reversible and reversible systems, respectively, and
two generalized Hénon maps, which represent cases of low- and high-dimensional
hyperchaos. We also present a modified version of the traditional logistic map, intro-
ducing a slow periodic dynamic of the growth rate parameter, that includes ranges
for which the map is chaotic. The resulting system exhibits concurrent slow and fast
dynamics and its forecasting represents a challenging task. Lastly, we consider two
real-world time series of solar irradiance and ozone concentration, measured at two
stations in Northern Italy. These dynamics are shown to be chaotic movements by
means of the tools of nonlinear time-series analysis.

In the past, many nonlinear systems that exhibit a chaotic behavior have been dis-
covered in several fields of science, from ecology [20] to finance [17, 18], from
population dynamics [10] to atmospheric processes [13].

3.1 Artificial Chaotic Systems

In this book, we consider four discrete-time dynamical systems universally known for
being chaotic to test the predictive power of the neural predictors: the logistic map,
the Hénon map, and two generalized Hénon maps. The logistic and Hénon maps
are the classical prototypes of chaos in non-reversible and reversible discrete-time
systems, respectively, whereas the generalized Hénon map is considered to include
low- and high-dimensional hyperchaos. We limit the analysis to a single output
variable for each system so that the corresponding time series is univariate. For the
ease of the reader, the systems are presented below.

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021 31

M. Sangiorgio et al., Deep Learning in Multi-step Prediction of Chaotic Dynamics,
PoliMI SpringerBriefs,
https://doi.org/10.1007/978-3-030-94482-7_3
32 3 Artificial and Real-World Chaotic Oscillators

Fig. 3.1 Two examples of the logistic map behavior (r = 3.7) when starting from different initial
conditions selected at random

3.1.1 Logistic Map

The logistic map is a one-dimensional quadratic map used to describe biomass or

economic growth. It has only one state variable, the density of the biological or
economic resource that we take equal to the output, so that the system can be defined
by a single difference equation that directly describes the dynamics of the output,
i.e.,  
y(t + 1) = r · y(t) · 1 − y(t) , (3.1)

where r > 1 is the growth rate at low density. As is well known, the logistic map
exhibits chaotic behavior for most of the values of r in the range 3.6-4. Two examples
of how the logistic map (r = 3.7) variable evolves in time are reported in Fig. 3.1.
The equation describing the map is a simple quadratic polynomial, which assumes
the shape represented in the first panel of Fig. 3.2. Reproducing this function with
a neural net is an easy task. Things start to become more and more complex when
this simple map is iterated (other panels in Fig. 3.2). This is a general property of
chaotic systems that can immediately be seen in the logistic map. The effect of
the increasing complexity that occurs when the map is iterated is that small errors
in the input (on the horizontal axis in Fig. 3.2) give larger and larger errors in the
output (vertical axis). A mid- to long-term prediction of chaotic systems is, there-
fore, intrinsically problematic, even if one knows the true model generating the data
[2, 3, 15].
3.1 Artificial Chaotic Systems 33

Fig. 3.2 Graph of the logistic map (r = 3.7, first panel on the left) and of the 2-to-6 iterated maps.
Grey areas show how a small error at time t propagates when the map is iterated

3.1.2 Hénon Map

The Hénon map, a toy model of celestial mechanics, is a two-dimensional (n = 2)

system traditionally defined in state space [6]:

x1 (t + 1) = 1 − a · x1 (t)2 + x2 (t)
(3.2)
x2 (t + 1) = b · x1 (t),

where a and b are model parameters, usually taking values 1.4 and 0.3, respectively.
Considering the first state variable as output, y(t) = x1 (t), the system is equivalent
to the following two-dimensional (m = 2) nonlinear regression:

y(t + 1) = 1 − a · y(t)2 + b · y(t − 1). (3.3)

The Hénon attractor and the evolution in time of the output variable y(t) are repre-
sented in Fig. 3.3.

3.1.3 Generalized Hénon Map

The generalized Hénon map is an extension of the Hénon map introduced with the
purpose of generating hyperchaos (a chaotic attractor characterized by at least two
34 3 Artificial and Real-World Chaotic Oscillators

Fig. 3.3 Chaotic attractor produced by the Hénon map (a), and evolution in time of the system
output for two random initial conditions (b)

positive Lyapunov exponents, i.e., at least two directions of divergence within the
attractor) [1, 11]. The state equations of the n-dimensional case are:

x1 (t + 1) = a − xn−1 (t)2 − b · xn (t)
(3.4)
x j (t + 1) = x j−1 (t), j = 2, . . . , n.

(for n = 2, using x1 (t)/a and (−b/a)x2 (t) as new coordinates and −b as a new
parameter gives the traditional formulation (3.2)). Hyperchaotic behavior is observed
for a = 1.9 and b = 0.03. The system can be rewritten as a nonlinear regression on
the output, y(t) = x1 (t), obtaining:

y(t + 1) = a − y(t − m + 2)2 − b · y(t − m + 1). (3.5)

We consider the 3D and the 10D generalized Hénon maps. The first is characterized by
n = m = 3. The corresponding Lyapunov exponents are L 1 = 0.276, L 2 = 0.257,
and L 3 = −4.040. The attractor’s fractal dimension, computed with the Kaplan-
Yorke formula [4], is 2.13. Figure 3.4 shows the chaotic attractor and the evolution
in time of the system’s output starting from two different initial states. The second
(n = m = 10) has 9 positive Lyapunov exponents and its attractor’s fractal dimension
is equal to 9.13.

3.1.4 Time-Varying Logistic Map

As a further test case, we consider a non-stationary system with both slow and fast
dynamics. These kinds of processes characterize many natural phenomena and for
this reason attract the attention of many researchers in the field of dynamical systems
theory (see, for instance, [12, 19]).
3.1 Artificial Chaotic Systems 35

Fig. 3.4 Chaotic attractor produced by the 3D generalized Hénon map (a), and evolution in time
of the system output for different initial conditions selected at random (b)

The slow-fast system presented here is obtained introducing, in the traditional

logistic map, a slow (periodic) dynamic for the parameter r (t) [14]:
⎧  
⎨ y(t + 1) = r (t) · y(t) · 1 − y(t)
⎩r (t) = rmax + rmin + rmax − rmin · sin( t · π ),
(3.6)
2 2 5000
where rmin = 2.9 and rmax = 3.7 represent the lower and upper bound of the growth
rate. These values have been selected so that the system in turn exhibits stable,
periodic or chaotic behavior, as shown in Fig. 3.5.
Testing the neural predictors presented in Sect. 4.1 on this kind of system is
interesting because the forecasting task requires to retain information about both the
slow-varying context (long-term memory), and the fast dynamics of the logistic map.
Moreover, this task has different degrees of complexity: it is fairly simple when the
system has stable or periodic behavior, and much more complex when the dynamic
becomes chaotic.
Formally speaking, the system defined by Eqs. (3.6) meets all the conditions listed
in Chap. 2, and thus it is a chaotic system.

3.2 Real-World Time Series

3.2.1 Solar Irradiance

As anticipated at the beginning of this chapter, chaos theory does not only consider
analytical systems like those presented above; a remarkable variety of natural phe-
nomena are thought to be chaotic, from meteorology to the orbits of celestial bodies,
from chemical reactions to electronic circuits [21].
Discovering Diverse Content Through
Random Scribd Documents
population in the Italian quarter of the North End of Boston was said
to be nearly 1.40 persons per room.[201] In the Italian quarter of
Philadelphia investigators found 30 Italian families, numbering 123
persons, living in 34 rooms. In some of the Italian tenements in this
city, lamps were kept burning all day in some of the rooms, where
day could scarcely be distinguished from night.[202] The Jews at this
time were only a little less densely crowded than the Italians. In 1891
nearly one fourth of the whole number of Jews living in two of the
precincts of the North End of Boston were living with an average of
more than two persons to a room and were found to be very
uncleanly in the care of their homes. Among the Irish an average of
1.24 persons per room was found in Boston in 1891. On the whole
they kept their tenements cleaner than did the Jews or Italians.[203]
Since the beginning of the twentieth century, interest in the slum
population of our cities has centered itself about the Slavic and other
races of southeastern Europe, even more than about the Italians and
Jews. About one sixth of the entire population of Buffalo, or 80,000
individuals, is Polish. Of these, about 4000 families, representing
20,000 persons, own their homes. They are said to be thrifty, clean,
willing, and neglected. Nearly all the Poles live in small one and two
story wooden cottages. Good tenement work thirty years ago avoided
the serious structural conditions which prevail in most cities. The
principal evil now in the Polish section is room-overcrowding. The
two-story cottages hold six or more families, while the older one-
story cottage was built for four families, though the owner is likely to
occupy two of the rear apartments. There are 15,000 of these
cottages, all subject to the tenement law. A Pole was recently made
health commissioner, and gave promise of being the best incumbent
of that office that Buffalo has ever had. That there is plenty of work
for him to do may be judged from the description of some of the
conditions which prevail.
“Counting little bedrooms, living rooms, and kitchens (and they
are pretty nearly indistinguishable), Mr. Daniels tells us that half the
Polish families in Buffalo, or 40,000 people, average two occupants
to a room. There are beds under beds (trundle beds, by the way, were
once quite respectable), and mattresses piled high on one bed during
the day will cover all the floors at night. Lodgers in addition to the
family are in some sections almost the rule rather than the exception.
Under such conditions privacy of living, privacy of sleeping, privacy
of dressing, privacy of toilet, privacy for study, are all impossible,
especially in the winter season; and those who have nerves, which
are not confined to the rich in spite of an impression to the contrary,
are led near to insanity. Brothers and sisters sleep together far
beyond the age of safety. It begins so, and parents do not realize how
fast children grow, or how dangerous it all is.”[204]
Even in Buffalo, the congestion problem is not limited to the Poles.
The author just quoted describes the Italians as tending to establish
residences in old hotels, warehouses, and abandoned homesteads,
and says, “As late as 1906 we found Italians living in large rooms,
subdivided by head-high partitions of rope and calico, with a
separate family in each division.”
In Milwaukee there are three foci of the tenement evil, the Italian
quarter, the Polish quarter, and the Jewish quarter. While there are
not the large tenement houses that prevail in larger cities, there are
the same evil conditions in the small cottages of the laboring class.
The following paragraphs give a vivid picture of some of the
conditions in each of these three sections.
In the Italian district, “Entering one of these dwellings we had to
duck our heads to escape a shower bath from leaking pipes above the
door. Incidentally, we had to dodge a crowd of the canine family
which did not seem to be particularly pleased with our visit. The
rooms were dark. Something, which I supposed was food or intended
for food, was bubbling on a little stove. A friendly goat was playing
with the baby on the floor, and the pigeons cooed cheerily near by.
Through the door of the kitchen we got the odor of the stable. The
horses had the best room. In the middle room, which was absolutely
dark, on a bed of indescribable filth, lay an aged woman, groaning
with pain from what I judged to be ulcerated teeth, but which for
aught she knew might have been a more malignant disease. In this
single dwelling, which is not unlike many we saw, there lived
together in ignorant misery one man, two women, ten children, six
dogs, two goats, five pigeons, two horses, and other animal life which
escaped our hurried observation.”
“In the Ghetto, in one building, live seventy-one people,
representing seventeen families. The toilets in the yard freeze in
winter and are clogged in summer. The overcrowding here is fearful
and the filth defies description. Within the same block are crowded a
number of tenements three and four stories high with basement
dwellings. One of these is used as a Jewish synagogue. Above and
beneath and to the rear this building is crowded with tenement
dwellers. The stairways are rickety, the rooms filthy, and all are
overcrowded. The toilets for the whole population are in the cellar
adjoining some of the dwelling rooms, reached by a short stairway.
At the time of our visit the floors of this toilet, both inside and
outside, were covered with human excrement and refuse to a depth
of eight to twelve inches. Into this den of horrors all the population,
male and female, had to go.”
A typical dwelling of the Polish working people is thus described.
“There is an entrance, perhaps under the steps, which leads to the
apartments below. In this semibasement in the front lives a family.
There are perhaps two rooms, sometimes only one. In the rear of this
same basement lives another family. Above, on the first floor, lives
another family, likewise in two or three small rooms; and in the rear
is another. Thus four or more families live in one small cottage—and,
often, in true tenement style, they ‘take in’ boarders.... Here,
together, live men, women, children, dogs, pigeons, and goats in
regular tenement and slum conditions.”[205]
Such instances as these, which might be multiplied almost
indefinitely, are individual manifestations of conditions which are
represented en masse by the figures of the Immigration Commission.
It is apparent that slum conditions exist, fully developed, in other
places than the great cities, and in other types of building than the
regulation tenement. As will be seen later, they may be found in
communities which do not come under the head of cities at all. The
slum is a condition, not a place, and will crop up in the most
unexpected places, whenever vigilance is relaxed. The slum can never
be eradicated by erecting model dwellings, however well planned,
nor by any other superficial method alone. The foundation of the
slum rests in the social and economic relations of society, and can be
effectually attacked only through them.
In the foregoing quotations, frequent reference is made to the
filthy condition in which the dwellings of the foreign-born are kept.
It is the current idea among a large class of people that extreme
uncleanliness characterizes the great majority of immigrant homes.
Unfortunately there is all too large a basis of truth for this
impression. Yet there is undoubtedly much exaggeration on this
point in the popular mind. The Immigration Commission found that
out of every 100 homes investigated in its study of city conditions, 45
were kept in good condition, and 84 in either good or fair condition,
though the foreign-born were inferior in this respect to the native-
born. In many cases the filthy appearance of the streets in the
tenement districts is due to negligence on the part of city authorities,
rather than to indifference on the part of the householders. “In
frequent cases the streets are dirty, while the homes are clean.”[206]
Not only is it an error to suppose that all immigrants are filthy, but it
is also untrue that all immigrants who are filthy are so from choice.
While the standards of decency and cleanliness of many of our
immigrant races are undoubtedly much below those of the natives,
there are many alien families who would gladly live in a different
manner, did not the very conditions of their existence seem to thrust
this one upon them, or the hardship and sordidness of their daily life
quench whatever native ambition for better things they might
originally have had.
In the foregoing paragraphs mention has been made of the
boarder as a characteristic feature of life in the tenements. He is, in
fact, a characteristic feature of the family life of the newer immigrant
wherever found. Since so large a proportion of the modern
immigrants are single men, or men unaccompanied by their wives
(see p. 191), there is an enormous demand for accommodations for
male immigrants who have no homes of their own. This demand is
met in two main ways. The most natural, and perhaps the least
objectionable, of the two, where there are a certain number of
immigrant families of the specified race already in this country, is for
a family which has a small apartment to take in one or more boarders
or lodgers of their own nationality. In this way they are able to add to
their meager income, and thereby to increase the amount of their
monthly savings, or perhaps to help pay off the mortgage on the
house if they happen to be the owners. The motive is not always a
financial one, however, but occasionally the desire to furnish a home
for some newcomer from the native land, with whom they are
acquainted, or in whom they are interested for some other reason.
[207]
The second way of solving the problem is for a number of men to
band themselves together, hire an apartment of some sort, and carry
on coöperative housekeeping in one way or another. A description of
these households will be given later (p. 247).
The keeping of boarders or lodgers[208] is a very widespread
practice among our recently immigrating families.
Among the households studied by the Immigration Commission in
its investigation of cities, 13 per cent of the native-born white
households kept boarders, and 27.2 per cent of the foreign-born. The
following foreign-born nationalities had high percentages, as shown
by the figures: Russian Hebrews, 32.1 per cent; north Italians, 42.9
per cent; Slovaks, 41 per cent; Magyars, 47.3 per cent; Lithuanians,
70.3 per cent. A similar showing is made by the figures given in the
report of the Immigration Commission on Immigrants in
Manufacturing and Mining (abstract quoted). The percentage of
households keeping boarders, as shown in that report, is as follows:

PERCENTAGE OF HOUSEHOLDS KEEPING BOARDERS[209]

Nativity Per Cent

Native-born white of native father 10.0
Native-born of foreign father 10.9
Foreign-born 32.9

Race (foreign-born)—

Norwegian 3.8
Bohemian and Moravian 8.8
Croatian 59.5
South Italian 33.5
Magyar 53.6
Polish 48.4
Roumanian 77.9
Servian 92.8

209. Rept. Imm. Com., Imms. in Mfg. and Min., Abs., p. 147.
 The average number of boarders per household, based on the
number of households keeping boarders, was as follows:

AVERAGE NUMBER OF BOARDERS PER HOUSEHOLD BASED ON THE NUMBER OF

HOUSEHOLDS KEEPING BOARDERS[210]

Nativity Number
Native-born white of native father 1.68
Native-born of foreign father 1.52
Foreign-born 3.53

Race (foreign-born)—

Bulgarian 8.29
Croatian 6.39
Roumanian 12.23
Servian 7.25

210. Ibid., p. 149.

This prevalent custom of taking boarders brings numerous evils in
its train. Foremost among these is the absolute sacrifice of family life
in the households. It is difficult at best to maintain a decent degree of
privacy when the family is left to itself; the intrusion of outsiders
makes it wholly impossible. Secondly, the taking of boarders tends to
increase a congestion which is likely already to be extreme. Thirdly, it
lays additional burdens upon the already overworked housewife. Its
great advantage is, of course, the increase of the family income,
sometimes to an amount almost double that which could be obtained
without the boarders. Among the Slavs, for example, women are rare,
and are regarded as very valuable, first as wives, and second as a
means whereby a man may take boarders.[211] The arrangements
between the boarders and the housewife differ in different localities,
and under different conditions. In a Colorado mining camp $10 a
month is the customary price for a regular boarder. A very common
arrangement is for the men to buy each his own food, and pay the
woman to cook it. The sums paid range from $2 to $4 a month for
lodging, washing, and cooking.
 The life of such a housewife in a coal mining community has been
described in the following words: “The status of the immigrant
housewife from the south and east of Europe is deplorable. The
boarding system followed is one whereby a fixed sum is paid for
lodging, cooking, washing, and mending; an individual food account
being kept with each lodger. The housewife has the beds to make
each day for a dozen men, their clothing to wash and mend, their
meals to prepare. In many cases she has also to buy the food, which
necessitates many visits to the store and separate purchases for each
boarder. She has also to carry all the water used from the hydrant or
well, which may be ten or one hundred yards distant. When the men
return from work it is a part of her duties to help them in their
ablutions by scrubbing their backs. There are also numerous children
to care for and scores of other tasks demanding her attention. Under
these conditions the marked untidiness of the immigrant households
is not to be wondered at.”[212]
The second typical method of providing for the single male
immigrant, mentioned above, is coöperative housekeeping on the
part of a group of men, either with or without a female housekeeper.
This practice is very common among many of the newer races of
immigrants, as has been suggested. It is a makeshift to which the
foreigner is driven by the absence of a normal number of women of
his own race. In households of this sort are developed some of the
very worst conditions to be found among our foreign residents.
Under this system, a number of men of a certain foreign race club
together and hire an apartment, consisting of a few rooms in a
regular tenement house, or, very frequently, a large storeroom or
warehouse, which thereupon becomes their home. In order to
minimize expense, the greatest possible number of beds are provided
in each room. If the apartment consists of a storeroom, it is often
fitted up with tiers of bunks along the sides. Such a room may be
used by two sets of men, one during the day and one during the
night. If some of the men are peddlers, the peanut stands or barrows
will be kept at night in the unoccupied spaces in the room. The lack
of woman’s care in the upkeep of such apartments is very manifest.
The meals are either prepared in the apartment or secured at some
near-by restaurant, or the two methods are combined. In the absence
of all semblance of family life, every possible expedient to reduce
expense is adopted, with the unfortunate results that might be
expected. The following description of such a household will give a
concrete idea of the type:
“To-day, in a certain mining town, there are fourteen Slavs, all
unmarried, and with only themselves to support, who rent one large,
formerly abandoned, storeroom. This is taken care of by a
housekeeper, who also prepares the meals for the men. Each man
has his own tin plate, tin knife, fork, and cup; he has his own ham
and bread, and a place in which to keep them. Some things they buy
in common, the distribution being made by the housekeeper. For
beds the men sleep on bunks arranged along the walls and
resembling shelves in a grocery store. Each has his own blanket; each
carries it out-of-doors to air when he gets up in the morning, and
back again when he returns from his work at night. The monthly cost
of living to each of these men is not over four dollars. They spend but
little on clothes the year round, contenting themselves with the
cheapest kind of material, and not infrequently wearing cast-off
garments purchased of some second-hand dealer. For fuel they burn
coal from the culm-banks or wood from along the highway, which
costs them nothing but their labor in gathering it.”[213]
That housing conditions such as have been portrayed above should
prevail so generally all over the country is a serious indictment
against the social and industrial organization of the United States. It
has been intimated that these conditions are not in all cases due to
the choice of the immigrant, or to the lack of desire for better things
on his part. Whether they are not, to a large degree, actually due to
the presence of the immigrant in this country is quite another
matter, upon the decision of which must rest much of the final
judgment as to the desirability of immigration under the present
system.
Throughout the study of housing conditions among the foreign-
born, it becomes more and more evident that there is a marked
distinction not only between the homes of the native-born and the
foreign-born, but between those of the older and newer immigrants.
By whatever test the standards of each class are measured, there is
almost invariably a decided discrepancy in favor of the older races.
As regards the number of rooms per apartment, the size of
households, the number of persons per room, the number of
boarders, the care and upkeep of the apartment, the English,
Scandinavians, Germans, and Irish come much nearer to what might
be considered a reasonable American standard than do the Italians,
either north or south, the Slavs (except perhaps the Bohemians and
Moravians), the Greeks, Syrians, Bulgarians, etc. This distinction is
well brought out in mining localities, where the newer races have
displaced the older within recent years. A graphic comparison is
given by Mr. F. J. Warne in his book, The Slav Invasion and the
Mine Workers. He says that, by the time of the coming of the Slavs,
the Irish, English, Welsh, Scotch, and German mine workers had
grown accustomed to a “social life of some dignity and comfort.” The
English-speaking mine worker wanted a home and family. That
home was usually a neat, two-story frame house, with porch and
yard. Within were pictures on the walls, and carpets on the floors of
the best rooms. He wished to have no one as a permanent resident of
the house save his own family, or very near relatives. He desired his
wife to be well dressed and comfortable, and his children to have the
benefits of school. His wants were always just beyond his wages, and
always increasing.
The Slav had no wife and children, and wished none. “He was
satisfied to live in almost any kind of a place, to wear almost
anything that would clothe his nakedness, and to eat any kind of food
that would keep body and soul together.” He was content to live in a
one-room hut, built of driftwood and roofed with tin from old
powder cans. In the mining towns he drifted to the poorer and
cheaper sections to live. He did not care with whom or with how
many he lived, provided they were of his own nationality. When two
such standards are brought into competition, it is inevitable that the
higher should yield in some way or other.
This difference in standards is undoubtedly due in part to a
difference in natural instincts and aptitudes for decency and
cleanliness between the common classes of northern and southern
Europe, but probably more to the customary standards to which they
have become habituated in their native land. The effect is the same,
whatever the cause. The new immigrant desires a certain
improvement in his standard as a reward for emigration, but the new
standard need not be by any means the equivalent of that of the
immigrant races which have preceded him. As long as we continue to
draw our immigrants from more and more backward and
undeveloped nations and races we may expect to see a progressive
degradation in the customary standard of the working people.
There are many other considerations besides congestion which
determine the character of life in the slums. Many of these have
already been suggested in preceding paragraphs. Prominent among
them are ventilation, sanitary and cooking facilities, light, water
supply, healthfulness of surroundings, and play room for children.
The degree in which evils exist in these particulars, in any locality,
depends primarily upon the stringency of the local tenement and
public health laws, and the energy and faithfulness of their
enforcement. Much is being accomplished and has been
accomplished in recent years in the direction of securing better
conditions. Yet there is almost infinite room for improvement. The
futility of relying upon the individual benevolence and humanity of
builders, owners, and agents was demonstrated long ago. Here, of all
places, eternal vigilance on the part of the better classes of society is
the price of safety. Descriptions of the homes of the foreign-born are
full of accounts of dark and absolutely unventilated bedrooms,
houses unprovided with any water supply, filthy outdoor closets and
privy vaults, toilets used by ten or twelve families conjointly,
buildings covering the entire lot, dooryards flooded with stagnant
water and refuse, basements half filled with water, domestic animals
sharing the limited accommodations with the family, and a host of
other horrors. Detailed descriptions of these dwellings are
unnecessary. Any one interested may find them in abundance in the
accounts of housing conditions in the poorer sections of our cities
and towns, for, as the Immigration Commission has amply
demonstrated, the slum, wherever found, is distinctively the home of
the foreign-born.[214]
It is almost superfluous to add that there are thousands of
immigrants, even of the newer races, who live in conditions wholly
different from those we have been discussing. Individuals of every
race, in large numbers, have succeeded in raising themselves from
the lowly estate of their compatriots, and establishing homes of
culture and refinement, even of luxury. Examples of this class are
prominent, and are frequently referred to. Yet in spite of this, the
slum remains the characteristic home of the average immigrant to
this country, and as such it must be reckoned with.
The influence of the slum must of necessity be hampering and
degrading to its denizens. No poorer training school for American
citizens could be devised. Not only is the life prejudicial to health and
morals, and destructive of ambition, but it precludes practically all
incidental or unconscious contact with the uplifting influences of
American life. Almost the only actively assimilating agency with
which the slum dweller comes into immediate relationship is the
public school, and this lacks much of its value as an assimilating
force in districts which are so largely foreign that the pupils meet
few, if any, children of native-born parents. Any practical program
for solving the immigration problem must attack the slum boldly. In
the words of Mr. Frederic Almy, “You cannot make a silk purse out of
a sow’s ear, and you cannot make an American citizen out of a
tenement slum. The slum must go. If you spare the slum, you will
spoil the child.”[215]
In regard to the housing conditions of the foreign-born outside of
the larger centers of population it is more difficult to make
generalizations. Fortunately, it is also less necessary. Some of the
foremost housing evils are essentially city matters. Particularly is it
true of immigrants who have established themselves in independent
agriculture, that they have made a long step toward Americanization.
While every grade of dwelling may be found among foreign-born
agriculturists, from the wretched hovel of the Italian market
gardener to the home of the Swedish farmer of the Northwest which
ranks with the finest in the land, yet the alien who takes up his abode
in the country has, in many respects, removed himself from the
general problem of the immigrant, and his living conditions can, with
a reasonable degree of safety, be left to look after themselves. Yet it
has been abundantly proved that slum conditions can exist even in
the country, and in small towns. This is especially true in mining
camps, and in the smaller manufacturing communities. Some of the
worst conditions of the most crowded sections of the cities are
reproduced in the shacks of the miners or the dwellings of the factory
hands. Overcrowding, bad ventilation, unsanitary toilet facilities,
inadequate heating, and filth are not city monopolies. The taking of
boarders is especially common in these communities, and, in the
mining towns, brings a peculiar evil with it, in addition to all the
regular disadvantages. This lies in the necessity which every mine
worker is under of bathing every day after work. In the absence of
bathrooms, ablutions are customarily performed in a tub set in the
kitchen, and in the crowded quarters of the miner’s cabin, the
children of the household are accustomed to the sight of nudity from
their infancy up, to the serious injury of their moral sense.[216]
It is too often true that the worst conditions prevail in the company
houses. The extreme monotony of these identical rows of ugly
dwellings is in itself sufficiently depressing. But in addition, it
appears that many employers are wholly oblivious to the higher
needs of their employees, and provide the most meager shelter which
will suffice to keep body and soul together, charging therefor
exorbitant rates. To say that these men and women are treated like
beasts, is putting the case too mildly, for no well-to-do person would
house a valuable animal as some of these human workers are housed.
The shifting character of the population and the uncertain duration
of a mining camp offer a quasi justification for some of these evils.
Yet a self-respecting nation should not permit any type of industry to
persist which requires its army of workers to live as do hundreds of
thousands of these faithful toilers.[217]
In regard to the food of our immigrant population, such studies of
individual races as have been made seem to indicate that, while the
dietary of the average foreign family falls far short of what a native
American would consider a satisfactory standard and is very
deficient in variety, yet it is ordinarily sufficient in quantity and in
amount of nourishment. Of course there are countless immigrant
families of the poorer sort, just as there are of natives, who are
habitually undernourished; yet the ordinary immigrant working
family or individual appears not to suffer for lack of sustaining food.
This condition is made possible by a long habitude in European
countries to an exceedingly simple diet, and by a resulting knowledge
of cheap and nourishing foods. The food item in the budget of an
immigrant family from southern or eastern Europe is almost
incredible to an American. The average cost of food for an individual
immigrant mine worker in Pennsylvania runs from about $4 to $10
per month. Among the Italians in Boston, during the winter months,
about a dollar a week will suffice for the food of a man. The south
Italian berry pickers in New Jersey are said to be able to get along on
as little as 25 cents per week, and other races live almost as cheaply.
[218]

There appears to be a considerable difference in this respect

between the different races, even among the newer immigrants. The
lowest standard prevails among the south Italians, Greeks, Syrians,
Bravas, etc. The Slavs are inclined to spend more of their increasing
income on food; particularly is meat a more important part of their
diet. The Jews are said to rank well above the Italians in this regard.
The quality and preparation of food leaves much to be desired.
Italian children are sent to the markets of Boston to gather
vegetables which have been thrown away as unfit for use. A brief
walk through the East Side of New York, with an eye on the push
carts, will convince one of the undesirable quality of some, at least, of
the food eaten by the residents of that section. On the other hand, the
Greek laborers on the railroads of the West are said to live
remarkably well, and themselves complain of the staleness of
American food, and object to our practice of putting everything up in
“boxes.”[219] In general, the conclusion of investigators in regard to
the food of our working classes seems to be that the faults of their
dietary lie, not so much in the failure to spend an adequate amount
of money for food, as in wasteful and ill-judged purchases,
unsatisfactory preparation, and improper balance between the
essential food elements (especially lack of sufficient proteids) and too
much fat. It is not unlikely that in this particular the immigrants fare
better than the natives in the same class. It is certainly probable that,
taken on the whole, the standard of food of the immigrant families in
this country is superior to that to which they were accustomed in
their native land.
There is probably no other aspect of life in which the immigrant
shows at least a superficial Americanization more quickly than in the
matter of clothing. It is a matter where imitation is easy, and in fact
almost inevitable. Any purchases of clothing made after the
immigrant’s arrival in this country must, almost of necessity, be
American in type. And the younger generation, at least, are eager to
have their exterior appearance correspond to that of the older
residents of their adopted country,—so eager, often, as to lead them
to adopt the most extreme of the new fashions in cut and fitting,
however cheap and flimsy the materials may be. In fact, this
Americanization affects the immigrants even before they leave their
native home. Officials on Ellis Island say that it is rare nowadays to
see groups of immigrants arriving clad in their picturesque European
costumes; the prevailing garb now is of the American type. It is a
strange fact that some writers, apparently oblivious of the ease of this
transition, seem to regard American clothes as an evidence of real
assimilation.
As regards physical adequacy of clothing, the immigrant is
probably as well off on the average as his native fellow-worker. It is
not likely that any large proportion of our working classes actually
suffer physical harm from insufficient clothing, unless it be through
lack of proper protection against dampness, particularly in the
matter of shoes.[220] In respect to cleanliness, and even decency, there
is frequently room for improvement among the immigrants, just as
there is among the native-born. There is, on the other hand, a
recognized danger that the desire for a fashionable appearance,
particularly on the part of the women, may lead to an extreme
expenditure for dress, unwarranted by the family income.[221]
 CHAPTER XIII
THE STANDARD OF LIVING (continued)

The standard of living of any family or individual[222] is the

resultant of two principal factors. These are the desires and appetites
of the individual or family and the amount of income available for
the gratification of those desires and appetites. The casual observer,
in forming his estimate of the immigrant, is in danger of forgetting
the second of these factors, and of assuming that because the
immigrant is found living in a certain status, he is therefore satisfied
with that status and has no ambition to change it. It has already been
hinted, in the foregoing paragraphs, that this is not the case. A full
understanding of the limitations under which the immigrant is
placed can come only with a study of the customary wages or income
of the class to which he belongs.
 AVERAGE ANNUAL EARNINGS OF EMPLOYEES IN THE INDUSTRIES SPECIFIED[223]
Industry Per Cent of Average Annual Earnings
Employees Males, 18 or Male Heads Average
Foreign-born Over of Families Family
Income
Iron and steel 57.7 $346 $409 $568
manufacturing
Slaughtering and
meat
packing 60.7 557 578 781
Bituminous coal 61.9 443 451 577
mining
Glass manufacturing 39.3 574 596 755
Woolen and worsted
manufacturing 61.9 346 400 661
Silk goods
manufacturing
and dyeing 34.3 431 448 635
Cotton goods 68.7 [224] 470 491
manufacturing
Clothing 72.2 513 530 713
manufacturing
Boot and shoe 27.3 502 573 765
manufacturing
Furniture 59.1 575 598 769
manufacturing
Collar, cuff, and shirt
manufacturing 13.4 637 662 861
Leather tanning,
currying,
and finishing 67.0 431 511 671
Glove manufacturing 33.5 625 650 904
Oil refining 66.7 591 662 828
Sugar refining 85.3 522 549 661
Cigars and tobacco 32.6 1.92[225]
manufacturing

223. Compiled from Rept. Imm. Com., Imms. in Mfg. and Min., Abs.

224. Not given.

225. Daily wage only given.

 The matter of wages is one of the easiest aspects of the life of the
immigrant about which to secure reliable data. It lends itself readily
to exact measurement, averaging, and tabulation. It is a subject upon
which the immigrant himself can give accurate information if he is so
inclined. As a result, there is a considerable mass of data in regard to
the earnings of the foreign-born, and it is possible to make
trustworthy generalizations thereupon. The latest and most inclusive
figures on this point are those furnished by the Immigration
Commission in its various reports. Foremost among these stands the
report of Immigrants in Manufacturing and Mining, which presents
the results of a thoroughgoing investigation of twenty of the leading
industries of the country, and a less detailed study of sixteen others,
covering in all 17,141 households and 503,732 individuals. The great
majority of these are foreign-born, but there is a sufficient number of
native-born, both of native and foreign parentage, to serve the
purposes of comparison. The table on the previous page gives the
average annual earnings of employees and the average family income
in the different industries.
A noteworthy feature of the above table is the general excess of
average family earnings over the average earnings of heads of
families, showing the extent to which other members of the family
besides the head contribute to the family support.
The average weekly earnings of male employees, 18 years of age or
over, distributed according to nativity, are as follows. (The table
includes over 200,000 individuals.)

AVERAGE WEEKLY EARNINGS OF MALE EMPLOYEES, 18 YEARS OF AGE OR OVER, BY

NATIVITY[226]

Nativity Average Weekly Earnings

Native-born white of native father $14.37
Native-born of foreign father 13.89
Foreign-born 11.92

226. Rept. Imm. Com., Imms. in Mfg. and Min., Abs. p. 91.
There is a marked difference between races in this respect. The
lowest figures among the foreign-born were: Albanian, $8.07; Greek,
$8.41; Portuguese, $8.10; Syrian, $8.12; Turkish, $7.65. Some of the
foreign-born rank well above the natives, as, for instance:
Norwegian, $15.28; Scotch, $15.24; Scotch-Irish, $15.13; Swedish,
$15.36; Welsh, $22.02.
The average yearly earnings (approximate) of male employees 18
years of age or over were as follows:

AVERAGE YEARLY EARNINGS (APPROXIMATE) OF MALE EMPLOYEES, 18 YEARS OF

AGE OR OVER, BY NATIVITY[227]

Nativity Average Yearly Earnings

Native-born white of native father $666
Native-born of foreign father 566
Foreign-born 455

227. Ibid., p. 131.

In this table, the decrease of earnings of approximately $100 from
class to class is striking.
The average family income was as follows:

AVERAGE ANNUAL FAMILY INCOME, BY NATIVITY OF HEAD OF FAMILY[228]

Nativity Average Family Income

Native-born white of native father $865
Native-born of foreign father 866
Foreign-born 704

228. Ibid., p. 136.

Comparing the last two tables, and noting that while the average
yearly earnings of native-born male employees of foreign parentage
are $100 less than those of the native-born of native parentage, yet
the family income of the native-born of foreign parentage is $1 more
than that of the native-born of native parentage, the obvious
conclusion might be that the native-born of foreign parentage are
more inclined to rely upon some one besides the head of the family
for part of the income than are the native-born of native parentage.
Closer examination, however, proves that this is not the case. The
following table gives the percentages of families of different nativities
which receive the entire income from the husband.

PER CENT OF FAMILIES HAVING ENTIRE INCOME FROM THE HUSBAND, BY

NATIVITY[229]

Nativity Per Cent

Native-born white of native father 58.4
Native-born of foreign father 61.3
Foreign-born 38.0

229. Rept. Imm. Com., Imms. in Mfg. and Min., Abs., p. 139.
Thus there is a smaller proportion of families among the native-
born of foreign fathers who rely upon other members of the family
than the husband for part of the family income than of the native-
born of native father. It appears that the explanation of the
peculiarity which has been noticed must be either that only the more
prosperous of the native-born of foreign parentage are heads of
families, or that those families of this class which do receive income
from other sources than the husband receive a much greater total
amount than among the native-born of native father, so as to raise
the average. The former explanation seems the more probable, for
while 67.3 per cent of the male native-born white employees of native
fathers, 20 years of age or over, were married, only 56.5 per cent of
the native-born of foreign fathers of the same age were married.
Native-born employees of foreign parentage who are old enough to
be the heads of families are predominantly representatives of the old
immigration, and hence stand high on the wage scale. The very small
percentage of families among the foreign-born which derive their
entire income from the husband indicates the extent to which the
children of this class contribute to the family support, and also the
extent to which boarders are taken.
Figures from other sources corroborate, in general, the showing
made in the foregoing tables, with some differences in detail. The
Immigration Commission in one of its other reports, namely that on
Immigrants in Cities, gives the average approximate yearly earnings
of over 10,000 male wage workers 18 years of age or over as follows:
native-born white of native father, $595; native-born of foreign
father, $526; foreign-born, $385.[230] These figures are less,
throughout, than those presented in the foregoing tables, and seem
to indicate that the average of wages in cities is less than in the
general run of organized industries throughout the country. It is
probable that a census of city workers would include many in
insignificant industries, and in occupations which could hardly be
classed as industries, where the wage scale is low.
The earnings of agricultural laborers on the farms of western New
York range from $1.25 to $1.75 per day of ten hours. South Italian
families of four or five members, engaged in this kind of work,
average from $350 to $450 for the season, extending from April to
November. Poles, working as general farm laborers the year round,
earn from $18 to $20 per month.[231] Among the anthracite coal
miners of Pennsylvania, the average yearly wage of the contract
miners, who make up about twenty-five per cent of persons
employed about the mines, is estimated at about $600 per year,
while “adults in other classes of mine workers, who form over sixty
per cent of the labor force, do not receive an annual average wage of
$450.”[232] In the extensive array of wage figures given by Mr.
Streightoff, distinction is not made between natives and immigrants,
but the general showing harmonizes so well with what has already
been given as to obviate the necessity of going into this question in
further detail.[233] We are justified in setting down the average
earnings of wage-working adult male immigrants as from $350 to
$650 per year, and the average annual income of immigrant families
at from $500 to $900.
The figures given for individual immigrant incomes have been
confined to male workers, for the reasons that they are
representative, and are of primary importance in determining the
status of the immigrant family in this country. The wages of female
workers range on the average from 30 to 40 per cent below those of
males. Full comparisons are given in the volume of the Immigration
Commission Report on Immigrants in Manufacturing and Mining.
The next question which arises is, to what degree are these
incomes, of individuals and families, adequate to furnish proper
support to an average family of five persons? This problem involves
the determination of the minimum amount on which a family can
live in decency under existing conditions in America. Numerous
efforts have been made to solve this question. The estimate of the
Bureau of Statistics of Massachusetts is $754.[234] The Charity
Organization Society of Buffalo regards $634 a year as the “lowest
tolerable budget which will allow the bare decencies of life for a
family of five.”[235] A special committee of the New York State
Conference of Charities and Corrections in 1907 made the following
estimates as to the income necessary for a family of five persons in
New York City.
“$600–$700 is wholly inadequate to maintain a proper standard
of living, and no self-respecting family should be asked or expected
to live on such an income.”
“With an income of between $700–$800 a family can barely
support itself, provided it is subject to no extraordinary expenditures
by reason of sickness, death, or other untoward circumstances. Such
a family can live without charitable assistance through exceptional
management and in the absence of emergencies.”
“$825 is sufficient for the average family of five individuals,
comprising the father, mother, and three children under 14 years of
age to maintain a fairly proper standard of living in the Borough of
Manhattan.”
Mr. Streightoff summarizes the evidence in the following words:
“It is, then, conservative to set $650 as the extreme low limit of the
Living Wage in cities of the North, East, and West. Probably $600 is
high enough for the cities of the South. At this wage there can be no
saving, and a minimum of pleasure.”[236]
The close correspondence of these various estimates gives them a
high degree of credibility. If we fix these standards in mind, and then
look back over the wage scales given on the foregoing pages, we are
struck with the utter inadequacy of the annual incomes of the
foreign-born to meet even these minimum requirements of decency.
It is obvious that an enormous number of immigrant families, if
dependent solely on the earnings of the head of the family, would fall
far below any of these standards, and that many of them, even when
adding to their resources by the labors of wife and children, and the
contributions of boarders, cannot possibly bring the total income up
to the minimum limit. Even the average income in many occupations
is far below this minimum, and it must be considered that while an
average indicates that there are some above, there must also be many
below, the line. What must be the condition of those below! The
average family income of the foreign-born studied in the
Immigration Commission’s investigation of the manufacturing and
mining industries was $704. Mr. Frederic Almy states that 96 per
cent of the Poles under investigation in Buffalo earn less by $110
than the $634 per year which was set as the “lowest tolerable
budget.”[237]
A vast amount of information covering a number of miscellaneous
aspects of human life, which fall under the general head of the
standard of living, is furnished by the Immigration Commission, in
its report on the manufacturing and mining industries. Some of the
most important of these facts are summarized in the following tables.
First, as to the situation of young children in the homes of
immigrants.
PER CENT OF CHILDREN 6 AND UNDER 16 YEARS OF AGE[238]
Male Female
At At At At At At
Home School Work Home School Work
Native-born white of
native father 5.4 90.9 3.6 6.9 90.5 2.6
Native-born of foreign
father 10.2 83.9 5.9 12.6 83.5 3.9
Foreign-born 13.2 77.0 9.9 19.1 73.6 7.3

237. The Survey, Feb. 4, 1911, p. 767.

238. Rept. Imm. Com., Imms. in Mfg. and Min., Abs., pp. 194–195.
Among the following races the following per cent of foreign-born
male children of the specified age were at work: German, 13.9; south
Italian, 13.3; Lithuanian, 14.3; Portuguese, 15.7; Ruthenian, 14.6;
Scotch, 19.0; Syrian, 22.6.
The following table, showing the per cent of literacy of the
employees studied in these industries, is based on information for
500,329 employees, and hence has a remarkable trustworthiness:
 LITERACY OF EMPLOYEES IN MINING AND MANUFACTURING[239]
NATIVITY MALES FEMALES
Per Cent who Per Cent who
Read Read and Write Read Read and Write
Native-born white of native father 98.2 97.9 98.8 98.4
Native-born of foreign father 99.0 98.7 99.0 98.8
Foreign-born 85.6 83.6 90.8 89.2

239. Ibid., pp. 162–165.

Foreign-born male employees of the following races have the
following literacy, as shown by the per cent who can read and write:
south Italian, 67.6; Macedonian, 67.1; Portuguese, 46.1; Ruthenian,
63.6; Servian, 69.5; Turkish, 54.1.
From the foregoing table it appears that in respect to literacy the
native-born employees of foreign fathers are superior to the native-
born whites of native fathers, and that the foreign-born females are
superior to the foreign-born males.
The important matter of ability to speak English is forcibly
portrayed in the following table:
PER CENT OF FOREIGN-BORN EMPLOYEES (EXCLUSIVE OF THE ENGLISH-SPEAKING
RACES) WHO SPEAK ENGLISH[240]
Nativity Male Female Total
Total 55.6 38.6 53.2
Bulgarian 20.3 80.0 (only 5) 20.6
Danish 96.5 98.3 96.6
German 87.5 80.2 86.8
Greek 33.5 12.3 31.5
Hebrew, Russian 74.7 75.7 75.0
Herzegovinian 14.6 14.6
Italian, south 48.7 25.8 44.4
Magyar 46.4 24.0 45.2
Norwegian 96.9 91.8 96.5
Polish 43.5 15.5 39.1
Portuguese 45.2 27.0 37.8
Slovak 55.6 26.6 55.1
Slovenian 51.7 30.3 50.9
Swedish 94.7 94.2 94.7

240. Rept. Imm. Com., Imms. in Mfg. and Min., Abs., p. 198.
 It is thus apparent how large a proportion of our foreign-born
laborers have not even taken the first essential step toward
assimilation. This evil is, of course, practically overcome in the
second generation. Almost all of the native-born persons of foreign
fathers, six years of age or over, speak English, though some races
show from 6 to 8 per cent who do not.
The percentage who can speak English naturally increases with the
length of residence in the United States, until a percentage of 83.1 is
reached for all foreign-born employees who have been in the United
States ten years or more. But even in this group a very low
percentage is found among the Cuban and Spanish cigar makers, of
whom almost three fifths are unable to speak the English language.
The age of the immigrant at the time of arriving in the United
States has a great deal to do with the ability to speak English. The
percentage of those who were under fourteen when they arrived who
can speak English is nearly twice as large as that of those who were
fourteen or over. The reasons for this are the greater adaptability of
the younger immigrants, and their greater opportunities of going to
school. The relatively poor showing of the females is probably due to
their greater segregation, which prevents them from coming in touch
with Americans or older immigrants of other races.
One of the special reports of the Immigration Commission deals
with the children of immigrants in schools and brings out some very
significant facts. Practically all of the information was secured in
December, 1908. Naturally this investigation involved a study of the
children of native-born fathers also. A general investigation was
made in the public schools of thirty cities, including the first twenty
cities in point of population, as shown by the census of 1900, with
the exception of Washington, D.C., Louisville, Ky., and Jersey City,
N.J. An investigation was also carried on in regard to parochial
schools in twenty-four cities, and an investigation of the students in
seventy-seven institutions of higher learning. In addition to this
general investigation, an intensive investigation was made in twelve
cities, including seven cities not in the previous list, making a total of
thirty-seven cities in which public schools were studied. The total
number of public school pupils for whom information was secured
was 1,815,217. Thus the investigation was a very inclusive one, and
the results may be taken as representative of educational conditions
in the cities of the entire country.
Of the total number of public school children studied in the thirty-
seven cities, 766,727 were of native-born fathers, and 1,048,490 of
foreign-born fathers. The children of native-born white fathers
constituted 39.5 per cent of the total, while among the children of
foreign-born fathers there were the following percentages of the total
number: Hebrews, 17.6; Germans, 11.6; Italians (north and south),
6.4; total, native-born father, 42.2 per cent; total, foreign-born
father, 57.8 per cent.
The different cities show a marked difference in the proportion of
children who come from foreign-born fathers, as the following table
will show:

PER CENT OF PUPILS IN PUBLIC SCHOOLS OF FOREIGN-BORN FATHERS IN SPECIFIED

CITIES

City Per Cent

Chelsea 74.1
Duluth 74.1
New York 71.5
New Bedford 68.8
Chicago 67.3
Fall River 67.2
Shenandoah 67.1
New Britain 65.3
Boston 63.5
New Orleans 18.1
Kansas City 21.3
Johnstown 24.8
Cincinnati 27.1
Baltimore 28.5
St. Louis 31.9
Los Angeles 32.0
Cedar Rapids 34.2
Haverhill 39.1
“In only 7 of the 37 cities is the proportion of pupils who are children
of native-born white fathers as high as 60 per cent.” Four cities have
less than 30 per cent. The children of German foreign-born fathers
are most numerous in Milwaukee, Detroit, Buffalo, Cleveland,
Meriden, Chicago, Cincinnati, and St. Louis; those of foreign-born
Russian Hebrew fathers in Chelsea, New York, Boston, Philadelphia,
Newark, and Baltimore, those of foreign-born south Italian fathers in
Providence, Newark, New York, Yonkers, Buffalo, and Boston.
A smaller proportion of the total number of children of foreign-
born fathers are in the higher grades of the public schools than of the
children of native-born white fathers, as the following table shows:
PER CENT OF PUPILS OF SPECIFIED NATIVITY IN THE SPECIFIED GRADES
Grade Native-born White Father Foreign-born Father
Kindergarten 4.3 4.4
Primary grades 52.1 57.6
Grammar grades 34.5 33.3
High school 9.1 4.7

Total 100.0 100.0

The Slovaks, south Italians, and Magyars have the largest
percentages in the kindergartens, and the Portuguese, Lithuanians,
Slovaks, south Italians, and Polish the largest percentages in the
primary grades. In the high schools, the Canadians, other than
French, the Scotch, the native-born white, the Welsh, Germans,
Hebrews, and English stand highest. This is due to two main facts,—
the longer residence of these latter races in the United States, and
their greater desire for a high education for their children, coupled
with a greater ability to give it to them. Especially in the case of the
kindergartens are the newer immigrating races very eager to have
their young children looked after so that the mother can be free to
work, or otherwise occupy herself.
Another interesting set of figures is that referring to the amount of
retardation among the pupils of different nativities. By “retardation”
is meant that a pupil is above the “normal” age for the grade in which
he is. In this respect the children of foreign-born fathers of the newer
immigration are decidedly inferior to those of the older immigration.
The latter, in fact, are on the whole superior to the children of native-
born white fathers. Of the total number of children of foreign-born
fathers for whom this information was secured, 77.2 per cent were
born in the United States, and 28.8 per cent were born abroad. There
is a considerably larger proportion of retardation for those children
eight years of age or over who were born abroad than among those
born in the United States. The proportion retarded increases as the
age at the time of arrival in the United States advances. The
proportion of retardation is greater among those children whose
fathers cannot speak English than among those who can, and greater
among those whose fathers have not taken out naturalization papers
than among those who have.
When we turn to the institutions of higher learning, we find a
comparatively small number of foreign-born students, as might be
expected.[241] The percentages for a total of 32,887 students are as
follows:

Nativity of Student Per Cent of Total

Native-born white of native father 64.0

Native-born of foreign father 25.3
Foreign-born 10.2

The Hebrews stand foremost among the foreign-born.

Of the 221,159 pupils included in the parochial school
investigation, 36.5 per cent are children of native-born fathers (36.3
per cent of native-born white fathers), and 63.5 per cent of foreign-
born fathers. Children of foreign-born Irish fathers number 26.9 per
cent of the total number of pupils, foreign-born German fathers, 9.7
per cent, Polish, 7.1 per cent, and Italian, 7 per cent. In the twenty-
four cities in which information was secured for both public and
parochial schools, there were 1,322,053 pupils in the public schools,
and 221,159 pupils in the parochial schools. In Philadelphia nearly
one fourth of the pupils were in parochial schools.
Information was also secured for teachers in the kindergartens and
elementary grades of the public schools in thirty cities, including
49,067 individuals. Of these, 49.8 per cent were native-born of
native white fathers, and 42.8 per cent native-born of foreign fathers,
and 5.8 per cent foreign-born. Of the foreign-born, only six races
were represented by as many as one hundred teachers each, viz.
Hebrew, English, Irish, German, Canadian (other than French), and
Scotch.
 CHAPTER XIV
THE EXPLOITATION OF IMMIGRANTS. RELIGION.
BIRTHS, MARRIAGES, AND DEATHS. RECREATION

There is a group of peculiar economic institutions which have been

developed by the immigrants in this country, and which are
especially characteristic of the new immigration. This group includes
the padrone system, the contract labor system, the immigrant bank,
and two or three similar institutions, particularly the sweating
system, which is now practically dependent on immigrants.
The word “padrone” is adopted from the Italian, and signifies
master or “boss.” In its application to American conditions, it refers
to a system of practical slavery, introduced into this country by the
Italians, and subsequently utilized by a number of other
southeastern European races. When immigration from Italy began to
assume considerable proportions, there were already in the United
States a few Italians who had been here some time, and had acquired
a certain familiarity with the language and customs of the land. They
were thereby especially fitted to be of assistance to their newly
arrived fellow-countrymen, and also, unfortunately, to exploit them.
In fact, they did both of these things. By way of assistance, they put
the green immigrants in touch with employers of labor, helped them
to find lodgings, and, in brief, acted as the go-between in every case
of contact between the immigrant and the life of the people around
him. On the other hand, the padrone charged the newcomer well for
every service rendered, and in too many cases subjected him to
various forms of extortion, which his ignorance kept him from either
recognizing or preventing. As certain of the newer immigrants
became familiar with the speech and customs of their new home,
they in turn became padrones, and extended their operations over
the ever increasing numbers of new arrivals. Thus the system spread.
 There are certain businesses or occupations which are particularly
adapted to the application of this system, such as railroad labor,
peddling, boot-blacking, etc. The Italians developed it primarily in
respect to the first of these. This race has now practically abandoned
this system in this country, but it has been taken up by others, and is
at present practiced by the Bulgarians, Turks, Macedonians, Greeks,
and Mexicans, and in some cases among Austrians and Italians.[242]
A more concrete idea of the workings of this system may be gained
by an examination of its operation in a single industry, as, for
instance, the shoe-shining industry among the Greeks. This business,
in a marked degree, combines the necessary elements for the
successful application of the system,—small capital, cheap unskilled
labor, close supervision, etc.,—and this race is well adapted to apply
it to its extreme extent, partly from natural aptitude, and partly from
custom and training. For the system, in its main outlines, has long
been familiar in Greece, though some of the most unfortunate
aspects do not develop there.
The padrone is a Greek who has been in this country for some
time, and knows the ways of the land. He decides to engage in the
boot-blacking trade, and to secure his necessary helpers contracts for
a number of boys from his native land to come over and work for him
for a certain length of time, for a specified sum. The arrangement is
sometimes made with the boys, sometimes with the parents, but
almost always with the parents’ consent. When these boys arrive,
they are taken to a room or set of rooms, which the padrone has
engaged and which thenceforth are their “home.” They are at once
put to work in the shop of the boss, and kept at work continuously
thereafter, with practically no time off which they can call their own,
except the meager allowance made for sleep. The hours are long—
twelve, fourteen, or even more hours per day. The boss furnishes
board and lodging, and pays a small sum in cash, perhaps $200 per
year. The rooms are frightfully overcrowded, miserably ventilated,
and wholly unhygienic. The boys do their own cooking, usually in
relays of two, and the noon meal is eaten hurriedly in a room in the
rear of the shop. The boys are prevented from attending night school,
and are forbidden to talk to patrons. In every way the padrone tries
to discourage their acquiring knowledge of American ways, for the
system rests on ignorance. In a majority of cases the padrone takes
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebooknice.com