Download the Full Version of textbook for Fast Typing at textbookfull.

com

Analysis and Data Based Reconstruction of Complex

Nonlinear Dynamical Systems Using the Methods of
Stochastic Processes M. Reza Rahimi Tabar

https://textbookfull.com/product/analysis-and-data-based-
reconstruction-of-complex-nonlinear-dynamical-systems-using-
the-methods-of-stochastic-processes-m-reza-rahimi-tabar/

OR CLICK BUTTON

DOWNLOAD NOW

Download More textbook Instantly Today - Get Yours Now at textbookfull.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Complex Dynamical Systems in Education Concepts Methods

and Applications 1st Edition Matthijs Koopmans

https://textbookfull.com/product/complex-dynamical-systems-in-
education-concepts-methods-and-applications-1st-edition-matthijs-
koopmans/
textboxfull.com

Analysis and Control of Output Synchronization for Complex

Dynamical Networks Jin-Liang Wang

https://textbookfull.com/product/analysis-and-control-of-output-
synchronization-for-complex-dynamical-networks-jin-liang-wang/

textboxfull.com

Dynamic Modeling of Complex Industrial Processes Data

driven Methods and Application Research 1st Edition Chao
Shang (Auth.)
https://textbookfull.com/product/dynamic-modeling-of-complex-
industrial-processes-data-driven-methods-and-application-research-1st-
edition-chao-shang-auth/
textboxfull.com

Intelligent Algorithms for Analysis and Control of

Dynamical Systems Rajesh Kumar

https://textbookfull.com/product/intelligent-algorithms-for-analysis-
and-control-of-dynamical-systems-rajesh-kumar/

textboxfull.com
Biostatistics and Computer-based Analysis of Health Data
using R 1st Edition Christophe Lalanne

https://textbookfull.com/product/biostatistics-and-computer-based-
analysis-of-health-data-using-r-1st-edition-christophe-lalanne/

textboxfull.com

Data Driven Fault Detection for Industrial Processes

Canonical Correlation Analysis and Projection Based
Methods 1st Edition Zhiwen Chen (Auth.)
https://textbookfull.com/product/data-driven-fault-detection-for-
industrial-processes-canonical-correlation-analysis-and-projection-
based-methods-1st-edition-zhiwen-chen-auth/
textboxfull.com

Structural Methods in the Study of Complex Systems Elena

Zattoni

https://textbookfull.com/product/structural-methods-in-the-study-of-
complex-systems-elena-zattoni/

textboxfull.com

Analysis of Chaotic Behavior in Non-linear Dynamical

Systems Micha■ Piórek

https://textbookfull.com/product/analysis-of-chaotic-behavior-in-non-
linear-dynamical-systems-michal-piorek/

textboxfull.com

Modeling and Analysis of Stochastic Systems, Third Edition

Vidyadhar G. Kulkarni

https://textbookfull.com/product/modeling-and-analysis-of-stochastic-
systems-third-edition-vidyadhar-g-kulkarni/

textboxfull.com
Understanding Complex Systems

M. Reza Rahimi Tabar

Analysis and
Data-Based
Reconstruction of
Complex Nonlinear
Dynamical Systems
Using the Methods of Stochastic
Processes
Springer Complexity
Springer Complexity is an interdisciplinary program publishing the best research and
academic-level teaching on both fundamental and applied aspects of complex systems—cutting
across all traditional disciplines of the natural and life sciences, engineering, economics,
medicine, neuroscience, social and computer science.
Complex Systems are systems that comprise many interacting parts with the ability to
generate a new quality of macroscopic collective behavior the manifestations of which are
the spontaneous formation of distinctive temporal, spatial or functional structures. Models
of such systems can be successfully mapped onto quite diverse “real-life” situations like
the climate, the coherent emission of light from lasers, chemical reaction-diffusion systems,
biological cellular networks, the dynamics of stock markets and of the internet, earthquake
statistics and prediction, freeway traffic, the human brain, or the formation of opinions in
social systems, to name just some of the popular applications.
Although their scope and methodologies overlap somewhat, one can distinguish the
following main concepts and tools: self-organization, nonlinear dynamics, synergetics,
turbulence, dynamical systems, catastrophes, instabilities, stochastic processes, chaos, graphs
and networks, cellular automata, adaptive systems, genetic algorithms and computational
intelligence.
The three major book publication platforms of the Springer Complexity program are the
monograph series “Understanding Complex Systems” focusing on the various applications of
complexity, the “Springer Series in Synergetics”, which is devoted to the quantitative theo-
retical and methodological foundations, and the “Springer Briefs in Complexity” which are
concise and topical working reports, case studies, surveys, essays and lecture notes of rele-
vance to the field. In addition to the books in these two core series, the program also incor-
porates individual titles ranging from textbooks to major reference works.

Series Editors

Henry D. I. Abarbanel, Institute for Nonlinear Science, University of California, San Diego, La Jolla, CA, USA
Dan Braha, New England Complex Systems Institute, University of Massachusetts, Dartmouth, USA
Péter Érdi, Center for Complex Systems Studies, Kalamazoo College, USA and Hungarian Academy of
Sciences, Budapest, Hungary
Karl J. Friston, Institute of Cognitive Neuroscience, University College London, London, UK
Hermann Haken, Center of Synergetics, University of Stuttgart, Stuttgart, Germany
Viktor Jirsa, Centre National de la Recherche Scientifique (CNRS), Université de la Méditerranée, Marseille,
France
Janusz Kacprzyk, Polish Academy of Sciences, Systems Research Institute, Warsaw, Poland
Kunihiko Kaneko, Research Center for Complex Systems Biology, The University of Tokyo, Tokyo, Japan
Scott Kelso, Center for Complex Systems and Brain Sciences, Florida Atlantic University, Boca Raton, USA
Markus Kirkilionis, Mathematics Institute and Centre for Complex Systems, University of Warwick, Coventry,
UK
Jürgen Kurths, Nonlinear Dynamics Group, University of Potsdam, Potsdam, Germany
Ronaldo Menezes, Department of Computer Science, University of Exeter, UK
Andrzej Nowak, Department of Psychology, Warsaw University, Warszawa, Poland
Hassan Qudrat-Ullah, King Fahd University of Petroleum and Minerals, Dhahran, Saudi Arabia
Linda Reichl, Center for Complex Quantum Systems, University of Texas, Austin, USA
Peter Schuster, Theoretical Chemistry and Structural Biology, University of Vienna, Vienna, Austria
Frank Schweitzer, System Design, ETH Zürich, Zürich, Switzerland
Didier Sornette, Entrepreneurial Risk, ETH Zürich, Zürich, Switzerland
Stefan Thurner, Section for Science of Complex Systems, Medical University of Vienna, Vienna, Austria
Understanding Complex Systems
Founding Editor: S. Kelso

Future scientific and technological developments in many fields will necessarily

depend upon coming to grips with complex systems. Such systems are complex in
both their composition–typically many different kinds of components interacting
simultaneously and nonlinearly with each other and their environments on multiple
levels–and in the rich diversity of behavior of which they are capable.
The Springer Series in Understanding Complex Systems series (UCS) promotes
new strategies and paradigms for understanding and realizing applications of
complex systems research in a wide variety of fields and endeavors. UCS is
explicitly transdisciplinary. It has three main goals: First, to elaborate the concepts,
methods and tools of complex systems at all levels of description and in all scientific
fields, especially newly emerging areas within the life, social, behavioral, economic,
neuro- and cognitive sciences (and derivatives thereof); second, to encourage novel
applications of these ideas in various fields of engineering and computation such as
robotics, nano-technology, and informatics; third, to provide a single forum within
which commonalities and differences in the workings of complex systems may be
discerned, hence leading to deeper insight and understanding.
UCS will publish monographs, lecture notes, and selected edited contributions
aimed at communicating new findings to a large multidisciplinary audience.

More information about this series at http://www.springer.com/series/5394

M. Reza Rahimi Tabar

Analysis and Data-Based

Reconstruction of Complex
Nonlinear Dynamical
Systems
Using the Methods of Stochastic Processes

123
M. Reza Rahimi Tabar
Department of Physics
Sharif University of Technology
Tehran, Iran

ISSN 1860-0832 ISSN 1860-0840 (electronic)

Understanding Complex Systems
ISBN 978-3-030-18471-1 ISBN 978-3-030-18472-8 (eBook)
https://doi.org/10.1007/978-3-030-18472-8
© Springer Nature Switzerland AG 2019
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part
of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission
or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are exempt from
the relevant protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this
book are believed to be true and accurate at the date of publication. Neither the publisher nor the
authors or the editors give a warranty, expressed or implied, with respect to the material contained
herein or for any errors or omissions that may have been made. The publisher remains neutral with regard
to jurisdictional claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Switzerland AG
The registered company address is: Gewerbestrasse 11, 6330 Cham, Switzerland
In memory of Professor Rudolf Friedrich
(1956–2012) whose original and innovative
thinking with Professor Joachim Peinke on
the methods that we describe in this book
motivated a large body of work by others on
the subject.
Preface

The data analysis of physical observables has a long tradition in the field of
nonlinear dynamics and complex systems. Much effort has been devoted to
answering the question of how to extract a “deterministic” dynamical system from a
suitable analysis of experimental data, given that an appropriate analysis can yield
important information on dynamical properties of the system under consideration.
Fluctuations in these time series are usually considered as a purely random or
uncorrelated variable, which is additively superimposed on a trajectory generated
by a deterministic dynamical system. The problem of dynamical noise, i.e., fluc-
tuations that interfere with the dynamical evolution, has not been addressed in much
detail, although it is of utmost importance for the analysis of fluctuating time series.
This book focuses on a central question in the field of complex systems: Given a
fluctuating (in time or space), uni- or multi-variant sequentially measured set of
experimental data (even noisy data), how should one analyse non-parametrically
the data, assess underlying trends, uncover characteristics of the fluctuations
(including diffusion and jump contributions), and construct a stochastic evolution
equation? Here, the term “non-parametrically” exemplifies that all the functions and
parameters of the constructed stochastic evolution equation can be determined
directly from the measured data.
In recent years, significant progress has been made when addressing this ques-
tion to the classes of continuous stochastic processes and of processes with jump
discontinuities. These can be modeled by nonlinear generalized Langevin equations
that include additive as well as multiplicative diffusive and even jump parts. An
important building block for the analysis approach presented in this book is a
Markovian property, which can be detected in real systems above a certain time or
length scale. This scale is referred to as the Markov–Einstein scale, and has turned
out to be an important characteristic of complex time series. The Markov–Einstein
time scale is the minimum scale above which the data can be considered as a
Markov process, and one can estimate it directly from observations. The main
advantage of the analysis approach is that is completely data-based and thus allows
one to find all functions and parameters of the modeling directly from measured
data. Due to its feasibility and simplicity, it has been successfully applied to

vii
viii Preface

fluctuating time series and spatially disordered structures of complex systems

studied in scientific fields such as physics, astrophysics, meteorology, earth science,
engineering, finance, medicine, and the neurosciences, and has led to many
important results.
This book provides an overview of methods that have been developed for the
analysis of fluctuating time series and of spatially disordered structures.
The book also offers numerical and analytical approaches to the analysis of
complex time series that are most common in the physical and natural sciences. It is
self-contained and readily accessible to students, scientists, and researchers who are
familiar with traditional methods of mathematics, such as ordinary, and partial
differential equations. Codes for analysing continuous time series are available in an
R package developed under the supervision of Joachim Peinke by the research
group Turbulence, Wind energy, and Stochastics (TWiSt) at the Carl von Ossietzky
University of Oldenburg. This package allows one to extract the (stochastic) evo-
lution equation underlying a set of data or measurements.
The book is divided into three main parts: I (Chaps. 1–9), II (Chaps. 10–21), and
III (Chaps 22–23).
Chapter 1 provides an introduction and an overview of topics covered in this
book. Chapter 2 reviews essentials of stochastic processes, namely the statistical
description of stochastic processes, stationary processes, classification of stochastic
processes, the Chapman–Kolmogorov equation as a necessary condition for
Markov processes, statistical continuous processes, as well as stochastic processes
in the presence of jump discontinuities. In Chap. 3, we present details of the
Kramers–Moyal expansion, the Pawula theorem, the Fokker–Planck equation and
its short-term propagator, and derive the master equation from the Chapman–
Kolmogorov equation. In Chap. 4, we provide Lindeberg’s condition for the con-
tinuity of stochastic trajectories. It is shown that the Fokker–Planck equation
describes a continuous stochastic process. We derive the stationary solutions of the
Fokker–Planck equation and define a potential function for dynamics. In Chap. 5,
we introduce the Langevin equation and Wiener processes along with their statis-
tical properties. Chapter 6 reviews the Itô and the Stratonovich calculus. We prove
Itô’s lemma and describe the Itô calculus for multiplicative noise. In Chap. 7, we
show the equivalence between the Langevin approach and the Fokker–Planck
equation and derive equations for statistical moments of a process whose dynamics
is given by the Langevin equation. In Chap. 8, we provide examples for stochastic
calculus using the Kubo–Anderson process, the Ornstein–Uhlenbeck process, and
the Black–Scholes process (or geometric Brownian motion). Chapter 9 covers the
following topics: Langevin dynamics in higher dimension, the Fokker–Planck
equation in higher dimension, finite-time propagators of a d-dimensional Fokker–
Planck equation, as well as discrete time evolution and discrete time approximation
of stochastic evolution equations. Chapters 1–9 can be skipped by readers who are
familiar with the standard notions of stochastic processes, or they may be useful for
examples and applications.
Preface ix

In Chap. 10, we introduce the Lévy noise-driven Langevin equation and the
fractional Fokker–Planck equations, derive the short-time propagator of Lévy
noise-driven processes, and provide limit theorems for Wiener and Lévy processes.
Finally, a non-parametric determination of Lévy noise-driven Langevin dynamics
from time series will be described. In Chap. 11, we study stochastic processes with
jump discontinuities and discuss the meaning of nonvanishing higher-order
Kramers–Moyal coefficients. We address in detail the physical meaning of
non-vanishing fourth-order Kramers–Moyal coefficients, stochastic processes with
jumps, as well as stochastic properties and statistical moments of Poisson jump
processes. In Chap. 12, we introduce the jump-diffusion processes with Gaussian
and mixed-Gaussian jumps. In Chap. 13, we introduce bi-variant jump-diffusion
equations and in Chap. 14, we describe different numerical schemes for the inte-
gration of Langevin and jump-diffusion stochastic differential equations, such as the
Euler–Maruyama scheme, the Milstein scheme, and Runge-Kutta-like methods.
This chapter closes with an introduction of packages in R and Python for the
numerical integration of stochastic differential equations. In Chap. 15, we discuss
the analysis of spatially disordered structures and provide a physical picture
of the fluctuation cascade from large to small scales. Moreover, this section
introduces the multipliers in cascade processes, and we derive a scale-dependent
solution of the Fokker–Planck equation and present the Castaing equation.
An answer to the question of how to set up stochastic equations for real-world
processes is presented in Chaps. 16–21. In Chap 16, the reader is familiarized with
the methods for estimating the Kramers–Moyal coefficients, and we introduce the
Markov–Einstein time (length) scale of a data set. This chapter also contains
important technical aspects of the method for estimating drift and diffusion coef-
ficients as well as higher-order Kramers–Moyal coefficients from time series. In
Chap. 17, we explain how to derive the Kramers–Moyal coefficients from
non-stationary time series using the Nadaraya–Watson estimator and we investigate
Kramers–Moyal coefficients in the presence of microstructure (measurement) noise.
In Chap. 18, we study the influence of a finite time step on the estimation of the
Kramers–Moyal coefficients from diffusive and jumpy data. In Chap. 19, we ana-
lytically derive a criterion (as a necessary condition) that allows one to check
whether for a given, even noisy time series the underlying process has a continuous
(diffusive) or a discontinuous (jumpy) trajectory. In Chap. 20, the steps of deriving
a Langevin equation from diffusive experimental time series are given, and we
finish the chapter with an introduction of an R Package for the modeling of one- and
two-dimensional continuous stochastic processes. In addition, the steps for deriving
a jump-diffusion stochastic equation from experimental time series with jumps are
presented. Also, two other methods for a reconstruction of time series will be
reviewed shortly. In Chap. 21 we reconstruct, as examples, some stochastic
dynamical equations from various synthetic continuous time series, from time series
with jump discontinuities and from time series generated by Lévy noise-driven
Langevin dynamics.
x Preface

Chapter 22 briefly reviews applications of the presented method (Chaps. 16–21)

to the analysis of real-world time series and ends with an outlook. As an example
of the analysis methods to real-world time series, we present in Chap. 23 results
derived from analyses of electroencephalographic time series.
I would like to thank N. Abedpour, M. Anvari, A. Barhraminasab, D. Bastine,
F. Böttcher, J. Davoudi, F. Ghasemi, J. Gottschall, Z. Fakhraai, S. M. Fazeli,
J. Friedrich, T. Jadidi, G. R. Jafari, A. Hadjihosseini, N. Hamedai-Raja,
A. M. Hashtroud, H. Heibelmann, J. Heysel, M. Hölling, C. Honisch, O. Kamps,
D. Kleinhans, M. Kohandel, P. G. Lind, G. Lohmann, St. Lück, P. Manshour,
P. Milan, E. Mirzahossein, S. Moghimi, S. M. Mousavi, S. M. S. Movahed,
I. Neunaber, M. D. Niry, J. Puczylowski, N. Reinke, Ch. Renner, P. Rinn,
V. Rezvani, F. Shahbazi, A. Sheikhan, M. Siefert, S. Siegert, F. T. Shahri,
S. M. Vaez Allaei, M. Wächter, L. Zabawa, and F. Zarei for useful discussions and
whose Ph.D. theses have contributed to this book. I am also thankful to Daniel
Nickelsen and Adrian Baule for sharing their ideas presented in Sects. 22.2.3
and 22.2.5.
Special acknowledgments should be given to Uriel Frisch (Observatoire de la
Côte d’Azur, Nice), Joachim Peinke (Carl-von-Ossietzky University Oldenburg),
Muhammad Sahimi (University of Southern California), and Holger Kantz (Max–
Planck Institute for Physics of Complex Systems, Dresden). I would also like to
thank the Alexander von Humboldt Foundation for financial support and Klaus
Lehnertz (Department of Epileptology, University of Bonn) for many detailed
discussions, the kind hospitality, and proofreading of Chap. 23.
It is greatly appreciated if readers could forward any errors, misprints, or
suggested improvements to: [email protected] or [email protected].

Oldenburg, Bonn M. Reza Rahimi Tabar

2019
Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Time Series of Complex Systems . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Stochastic Continuous Time Series . . . . . . . . . . . . . . . . . . . . . 2
1.3 Time Series with Jump Discontinuity . . . . . . . . . . . . . . . . . . . . 4
1.4 Microstructural (Measurement) Noise . . . . . . . . . . . . . . . . . . . . 5
1.5 Intermittency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2 Introduction to Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . . 9
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
2.2 Statistical Description of Time Series . . . . . . . . . . . . . . . . . . . . 10
2.2.1 The Probability Density . . . . . . . . . . . . . . . . . . . . . . . 11
2.2.2 Joint and Conditional Probability Distribution
Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Classification of Stochastic Processes . . . . . . . . . . . . . . . . . . . . 12
2.3.1 Purely Random Processes . . . . . . . . . . . . . . . . . . . . . . 13
2.3.2 Markov Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.3.3 Higher Order Processes . . . . . . . . . . . . . . . . . . . . . . . 14
2.4 Stationary Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 The Chapman–Kolmogorov Equation and the Necessary
Condition for a Process to Be Markov . . . . . . . . . . . . . . . . . . . 15
2.6 Continuous Stochastic Markov Processes . . . . . . . . . . . . . . . . . 16
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
3 Kramers–Moyal Expansion and Fokker–Planck Equation ....... 19
3.1 Kramers–Moyal Expansion . . . . . . . . . . . . . . . . . . . . ....... 19
3.2 Pawula Theorem and Fokker–Planck Equation . . . . . . ....... 21
3.3 Short-Time Propagator of Fokker–Planck Equation
in One Dimension . . . . . . . . . . . . . . . . . . . . . . . . . . . ....... 22

xi
xii Contents

3.4 Master Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
4 Continuous Stochastic Processes . . . . . . . . . . . . . . . . . . . . ....... 31
4.1 Stochastic Continuity . . . . . . . . . . . . . . . . . . . . . . . . ....... 31
4.1.1 Stochastic Mean-Square Continuity . . . . . . . . ....... 31
4.1.2 Lindeberg’s Continuity Condition for Markov
Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . ....... 32
4.2 Stochastic Differentiability . . . . . . . . . . . . . . . . . . . . . ....... 33
4.2.1 Mean-Square Differentiability of Stochastic
Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . ....... 33
4.2.2 General Condition for Non-differentiability
of Stochastic Processes . . . . . . . . . . . . . . . . . ....... 34
4.3 Description of a Continuous Stochastic Process
by a Fokker–Planck Equation . . . . . . . . . . . . . . . . . . ....... 34
4.4 Stationary Solution of the Fokker–Planck Equation
and the Potential Function . . . . . . . . . . . . . . . . . . . . . ....... 35
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....... 37
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ....... 37
5 The Langevin Equation and Wiener Process . . . . . . . . . . . . . . ... 39
5.1 The Langevin Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . ... 39
5.2 The Kramers–Moyal Coefficients of Wiener Process . . . . . . ... 40
5.3 Conditional Probability Distribution Function of the Wiener
Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
5.4 Statistical Moments of the Wiener Process . . . . . . . . . . . . . . . . 41
5.5 Markov Property of the Wiener Process . . . . . . . . . . . . . . . . . . 42
5.6 Independence of Increments of the Wiener Process . . . . . . . . . . 43
5.7 The Correlation Function of the Wiener Process . . . . . . . . . . . . 43
5.8 Wiener Process Is Not Differentiable . . . . . . . . . . . . . . . . . . . . 44
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
6 Stochastic Integration, Itô and Stratonovich Calculi . . . . . . . . . . . . 49
6.1 Stochastic Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
6.2 Nonanticipating Function and Itô Lemma . . . . . . . . . . . . . . . . . 53
6.2.1 Itô or Stratonovich . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
6.3 Integration of Polynomial and Examples of Itô Calculus . . . . . . 54
6.4 Itô Calculus for Multiplicative Noise and Itô-Taylor
Expansion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...... 56
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...... 57
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ...... 60
Contents xiii

7 Equivalence of Langevin and Fokker–Planck Equations . . . . . .... 61

7.1 Probability Distribution Functions of Langevin Dynamics . .... 61
7.2 Equation for Statistical Moments Based on the Langevin
Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
7.3 Existence of Solutions to Langevin Equation . . . . . . . . . . . . . . 64
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
8 Example of Stochastic Calculus . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.1 Anderson–Kubo Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
8.2 Ornstein–Uhlenbeck Process . . . . . . . . . . . . . . . . . . . . . . . . . . 71
8.3 Black–Scholes Process, or Geometric Brownian Motion . . . . . . 73
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
9 Langevin Dynamics in Higher Dimensions . . . . . . . . . . . . . . . . . . . 79
9.1 d-Dimensional Langevin Dynamics . . . . . . . . . . . . . . . . . . . . . 79
9.2 The Fokker–Planck Equation in Higher Dimensions . . . . . . . . . 80
9.3 The Kramers–Moyal Expansion in Higher Dimensions . . . . . . . 81
9.4 Discrete Time Evolution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
9.4.1 Proper Langevin Equations: White Noise . . . . . . . . . . . 81
9.5 Discrete-Time Approximation of Stochastic Evolution
Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 82
9.6 Short-Time Propagators of d-Dimensional Fokker–Planck
Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 83
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 84
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .... 86
10 Lévy Noise-Driven Langevin Equation and Its Time Series–Based
Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
10.1 Langevin Equation with Lévy Noise . . . . . . . . . . . . . . . . . . . . 87
10.1.1 Lévy Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
10.1.2 Fractional Fokker–Planck Equations . . . . . . . . . . . . . . 91
10.2 Discrete Time Approximation of Langevin Equations:
Lévy Noise-Driven . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 91
10.3 Short-Time Propagator of the Lévy Noise-Driven Langevin
Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 92
10.4 Joint Probability Distribution and Markovian Properties . . . . .. 92
10.5 Limit Theorems, and Wiener and Lévy Processes . . . . . . . . . .. 93
10.6 Reconstruction of Lévy Noise-Driven Langevin Dynamics
by Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 95
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 97
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. 97
xiv Contents

11 Stochastic Processes with Jumps and Non-vanishing

Higher-Order Kramers–Moyal Coefficients . . . . . . . . . . . . . . . .... 99
11.1 Continuous Stochastic Processes . . . . . . . . . . . . . . . . . . . . . . . 100
11.2 Non Vanishing Higher Order Kramers–Moyal Coefficients
and the Continuity Condition . . . . . . . . . . . . . . . . . . . . . . . . . . 101
11.3 Stochastic Properties of Poisson Process . . . . . . . . . . . . . . . . . . 102
11.4 Statistical Moments of Poisson Process . . . . . . . . . . . . . . . . . . 105
11.5 The Process in Presence of Jumps, Pure Poisson Process
as an Example, Kramers–Moyal Coefficients . . . . . . . . . . . . . . 106
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
12 Jump-Diffusion Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
12.1 Kramers–Moyal Coefficients of Jump-Diffusion Processes . . . . . 111
12.2 Gaussian Distributed Jump Amplitude . . . . . . . . . . . . . . . . . . . 114
12.3 Mixed Gaussian Jumps—The Variance Gamma Model . . . . . . . 115
12.4 Jump-Drift Process, Stochastic Solution, Example . . . . . . . . . . . 117
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
13 Two-Dimensional (Bivariate) Jump-Diffusion Processes . . . . . . . . . 123
13.1 Bivariate Jump-Diffusion Processes . . . . . . . . . . . . . . . . . . . . . 123
13.2 Kramers–Moyal Coefficients for Jump-Diffusion Processes
in Two Dimensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
14 Numerical Solution of Stochastic Differential Equations:
Diffusion and Jump-Diffusion Processes . . . . . . . . . . . . . . . . . . . . . 129
14.1 Numerical Integration of Diffusion Processes . . . . . . . . . . . . . . 129
14.1.1 Euler–Maruyama Method . . . . . . . . . . . . . . . . . . . . . . 129
14.1.2 Milstein Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
14.1.3 Runge–Kutta-Like Methods . . . . . . . . . . . . . . . . . . . . 132
14.2 Numerical Integration of Jump-Diffusion Processes . . . . . . . . . . 133
14.2.1 Euler–Maruyama Method . . . . . . . . . . . . . . . . . . . . . . 133
14.2.2 Milstein Scheme . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
14.3 Stochastic Differential Equation: Packages in R and Python . . . . 135
14.3.1 An R Package (Langevin) for Numerical Solving
and Modeling of Univariate and Bivariate “Diffusion
Processes” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
14.3.2 Simulation of Diffusion Processes, R-Package
“Sim.DiffProc” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
14.3.3 Simulation of Diffusion Processes, R-Package
“DiffusionRimp” . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
Contents xv

14.3.4 Simulation of Diffusion Processes, R-Package

“yuima” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
14.3.5 Simulation of Jump-Diffusion Processes,
Python-Solver “JiTCSDE” . . . . . . . . . . . . . . . . . . . . . 136
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
15 The Friedrich–Peinke Approach to Reconstruction of Dynamical
Equation for Time Series: Complexity in View of Stochastic
Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
15.1 Stochastic Processes in (Length or Time) Scale . . . . . . . . . . . . 143
15.1.1 Increments of Stochastic Processes . . . . . . . . . . . . . . . 143
15.1.2 Fractal and Multifractal Time Series: Linear
and Nonlinear Processes . . . . . . . . . . . . . . . . . . . . . . . 144
15.2 Intermittency and Kramers–Moyal Expansion . . . . . . . . . . . . . . 146
15.2.1 Governing Equation for the Statistical Moments
in Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
15.3 Fokker–Planck Equation and (Multifractal) Scaling
Exponents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
15.4 Langevin and Jump-Diffusion Modeling in Scale . . . . . . . . . . . 149
15.5 Multipliers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
15.6 Scale Dependent Solution of Fokker–Planck Equation . . . . . . . . 151
15.7 The Castaing Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
15.8 Multiscale Correlation Functions . . . . . . . . . . . . . . . . . . . . . . . 156
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
16 How to Set Up Stochastic Equations for Real World Processes:
Markov–Einstein Time Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
16.1 From Time Series to Stochastic Dynamical Equation . . . . . . . . 165
16.2 Markov–Einstein Time (Length) Scale . . . . . . . . . . . . . . . . . . . 166
16.3 Evaluating Markovian Properties . . . . . . . . . . . . . . . . . . . . . . . 168
16.4 Methods for Estimation of Markov–Einstein Time
or Length Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
16.5 Estimation of Drift and Diffusion Coefficients from
Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
16.5.1 Estimation of the Drift Vector . . . . . . . . . . . . . . . . . . . 172
16.5.2 Estimation of the Diffusion Matrix . . . . . . . . . . . . . . . 173
16.5.3 Higher Order Kramers–Moyal Coefficients . . . . . . . . . . 173
16.5.4 Estimation of Drift and Diffusion Coefficients
from Sparcely-Sampled Time Series . . . . . . . . . . . . . . 174
16.6 Deriving an Effective Stochastic Equation . . . . . . . . . . . . . . . . 175
xvi Contents

16.7 Self-consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175

Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
17 The Kramers–Moyal Coefficients of Non-stationary Time Series
and in the Presence of Microstructure (Measurement) Noise . . . . . 181
17.1 The Kramers–Moyal Coefficients for Non-stationary Time
Series: Nadaraya-Watson Estimator . . . . . . . . . . . . . . . . . . . . . 182
17.1.1 Time Dependent Kramers–Moyal Coefficients . . . . . . . 184
17.2 The Kramers–Moyal Coefficients in the Presence
of Microstructure (Measurement) Noise . . . . . . . . . . . . . . . . . . 184
17.2.1 Real-World Data with Microstructure Noise . . . . . . . . . 186
17.2.2 Real-World Data without Microstructure Noise . . . . . . 187
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
18 Influence of Finite Time Step in Estimating of the
Kramers–Moyal Coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
18.1 Diffusion Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
18.2 The Kramers–Moyal Conditional Moments for the Langevin
Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
18.3 Conditional Moments of the Jump-Diffusion Equation
for Different Orders of the Time Interval . . . . . . . . . . . . . . . . . 197
18.4 The Kramers–Moyal Coefficients in Vanishing Time
Interval Limit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
18.4.1 “Apparent” and “True” (in Vanishing Time
Interval Limit) Drift and Diffusion Coefficients
in Diffusion Processes . . . . . . . . . . . . . . . . . . . . . . . . 201
18.4.2 The Optimization Procedure to Extract
Kramers–Moyal Coefficients in Vanishing
Time Interval Limit . . . . . . . . . . . . . . . . . . . . . . . . . . 203
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
19 Distinguishing Diffusive and Jumpy Behaviors in Real-World
Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
19.1 Distinguishing Diffusive from Jumpy Stochastic Behavior
in Complex Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
19.2 A Jump Detection Measure Q(x) . . . . . . . . . . . . . . . . . . . . . . . 208
19.3 Application to Real-World Time Series . . . . . . . . . . . . . . . . . . 209
19.3.1 Jump Discontinuity in Non-stationary Time Series . . . . 211
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
Contents xvii

20 Reconstruction Procedure for Writing Down the Langevin

and Jump-Diffusion Dynamics from Empirical Uni- and
Bivariate Time Series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
20.1 The Reconstruction Procedure, Diffusion Processes . . . . . . . . . . 215
20.1.1 One Dimensional Time Series . . . . . . . . . . . . . . . . . . . 215
20.1.2 Two Dimensional (Bivariate) Time Series . . . . . . . . . . 217
20.1.3 An R Package for Reconstruction of One- and
Two-Dimensional Stochastic Diffusion Processes:
White Noise-Driven Langevin Dynamics . . . . . . . . . . . 221
20.2 The Reconstruction Procedure for the Lévy Noise-Driven
Langevin Equation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
20.3 The Reconstruction Procedure and Jump-Diffusion
Stochastic Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
20.3.1 One-Dimensional Time Series . . . . . . . . . . . . . . . . . . . 221
20.3.2 Two-Dimensional (Bivariate) Time Series . . . . . . . . . . 223
20.4 Other Methods for Reconstruction of Time Series . . . . . . . . . . . 224
20.4.1 Multiscale Reconstruction of Time Series . . . . . . . . . . 225
20.4.2 Mapping Stochastic Processes onto Complex
Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
21 Reconstruction of Stochastic Dynamical Equations: Exemplary
Diffusion, Jump-Diffusion Processes and Lévy Noise-Driven
Langevin Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
21.1 One and Two-Dimensional Diffusion Processes . . . . . . . . . . . . 227
21.1.1 Bistable Potential . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
21.1.2 Reconstruction of Bivariate Data Sets . . . . . . . . . . . . . 228
21.2 Jump-Diffusion Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
21.2.1 Reconstruction of an Ornstein–Uhlenbeck Process
with Jumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
21.2.2 Reconstruction of a Black-Scholes Process
with Jumps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
21.3 Lévy Noise-Driven Langevin Dynamics . . . . . . . . . . . . . . . . . . 233
21.4 Phase Dynamics and Synchronization . . . . . . . . . . . . . . . . . . . 235
21.5 Estimation of Kramers–Moyal Coefficients for Time
Series with Finite Markov–Einstein Time Scale . . . . . . . . . . . . 237
21.6 Estimation of Kramers–Moyal Conditional Moments
for Diffusion Processes with Different Precision . . . . . . . . . . . . 238
Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
22 Applications and Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
22.1 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
22.2 Outlook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
xviii Contents

22.2.1 Representation of Jump-Diffusion Dynamics

in Terms of Fractional Brownian Motion
of Order k . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
22.2.2 Langevin Dynamics Driven by Fractional
Brownian Motion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
22.2.3 The Integral Fluctuation Theorem for Diffusion
Processes (Cascade Processes) . . . . . . . . . . . . . . . . . . . 250
22.2.4 Estimation of Memory Kernel from Time Series . . . . . 252
22.2.5 Anomalous Diffusion . . . . . . . . . . . . . . . . . . . . . . . . . 254
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
23 Epileptic Brain Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
23.1 Stochastic Qualifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
23.2 Detailing the Stochastic Behavior of Epileptic Brain
Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270

Appendix A: Wilcoxon Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

Appendix B: Kernel Density Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
Chapter 1
Introduction

1.1 Time Series of Complex Systems

Complex systems are composed of a large number of subsystems that may interact
with each other. The typically nonlinear and multiscale interactions often lead to
large-scale behaviors, which are not easily predicted from the knowledge of only the
behavior of individual subsystems. Such large-scale collective emergent behaviors
may be desired or undesired. Examples of undesired emergent behaviors include
short- and long-term climate changes, hurricanes, cascading failures, and epileptic
seizures. Among the desired ones are evolution, adaptation, learning, and intelli-
gence, to name just a few [1–11].
In complex systems the fluctuations stemming from the microscopic degrees of
freedom play a fundamental role in introducing temporal variations on a fast time
scale that, quite often, can be treated as short-time correlated source of fluctuations.
In such systems, self-organized behaviour arises due to the emergence of collective
variables, or order parameters that, compared to the time or length scales of the micro-
scopic subsystems, vary on slower temporal and larger spatial scales. The interaction
of the order parameters are typically nonlinear, with the microscopic degrees of free-
dom showing up in the fluctuations that participate in the order parameter dynamics
and results in complex time series; see Fig. 1.1. Thus, the analysis of the behaviour of
complex systems must be based on the assessment of the nonlinear mutual interac-
tions, as well as the determination of the characteristics and strength of the fluctuating
forces. This immediately leads to the problem of retrieving a stochastic dynamical
system from data; see Fig. 1.2 for typical stochastic time series.
Analysis of time series has a long history in the field of nonlinear dynamics
[12–16]. The problem of dynamical noise, i.e., fluctuations that interfere with the
dynamical evolution, has not been addressed in much details, although it is of utmost
importance for the analysis of strongly fluctuating time series [17, 18]. In this book,
we provide detailed description and discussion of a non-parametric method, known
as the reconstruction method, which has been developed for analyzing continuous
stochastic processes and stochastic data with jumps in time and/or length scales.
© Springer Nature Switzerland AG 2019 1
M. R. Rahimi Tabar, Analysis and Data-Based Reconstruction of Complex Nonlinear
Dynamical Systems, Understanding Complex Systems,
https://doi.org/10.1007/978-3-030-18472-8_1
2 1 Introduction

Fig. 1.1 Complex systems are composed of a large number of subsystems behaving in a “col-
lective manner”. In such systems, which are far from equilibrium, “collectivity” arises due to
self-organization. It results in the formation of temporal, spatial, spatio-temporal and functional
structures. The state of the subsystems change over time and results in stochastic dynamics (as
shown for subsystem 8). The dynamics of order parameters in complex systems are generally non-
stationary, and the subsystems interact with each other in nonlinear manner. The arrows indicate
causal directions of influence

Fig. 1.2 Segments of

intracranial
electroencephalographic
(iEEG) time series, recorded
during a seizure-free interval
from within the epileptic
focus (red) and from a
distant brain region (black).
Source from [21]

The development of such methods has been stimulated by research on turbulent

flows and neuroscience [19–21], which has demonstrated the necessity of treating
the fluctuations as dynamical variables that interfere with the deterministic dynamics.

1.2 Stochastic Continuous Time Series

Systems under the influence of random forcing, or in the presence of nonlinear

interactions with other systems, can behave in a very complex stochastic manner
[17, 18, 22, 23]. The corresponding time series of such systems generally have
1.2 Stochastic Continuous Time Series 3

continuous trajectories, or may possess jump discontinuities. To decipher the problem

of retrieving a stochastic dynamical system from data, the main assumption is that
the measured time series represents a Markov process. A stochastic process with a
degree of stochasticity may have a finite Markov-Einstein (ME) time scale tM —the
minimum time interval over which the data can be considered as a Markov process.
Therefore one should first estimate tM for the measured time series, after which
one can apply the method described in this book to reconstruct the corresponding
dynamical stochastic equation. We will introduce various methods to estimate tM in
Chap. 16.
A process x(t) has continuous trajectories if the following relations for condi-
tional averaging, known as the Kramers–Moyal (KM) conditional moments, hold
for infinitesimal dt:

(x(t + dt) − x(t))1 |x(t)=x  = a(x, t) dt

(x(t + dt) − x(t))2 |x(t)=x  = b2 (x, t) dt

(x(t + dt) − x(t))2+s |x(t)=x  = O(dt 1+χ ) (1.1)

with s > 0 and χ > 0. The KM coefficients are defined as D(j) (x) = limdt→0 j!1 dt1
K (j) (x), where the conditional moments K (j) (x) = [x(t + dt) − x(t)]j |x(t)=x , and
can be determined non-parametrically, i.e., directly form the measured time series
[17]. Throughout this book we will use two definitions for the KM coefficients,
D(j) (x) and M (j) (x), which are related to each other by, D(j) (x) = 1/j! M (j) (x).
The dynamics of continuous stochastic processes is governed by the Langevin
dynamics that has the following expression [24–26],

dx(t) = a(x, t) dt + b(x, t) d W (t), (1.2)

where {W (t), t ≥ 0} is a scalar Wiener (Brownian) motion, and the functions a(x, t)
and b2 (x, t)/2 are known as drift and diffusion coefficients, respectively. In terms of
the conditional probability distribution, a continuous Markov process x(t) satisfies
the following continuity condition, given some δ > 0 [25],

Prob[ |Δx(t)| > δ |x(t)=x ]

C = lim =0 (1.3)
dt→0 dt

where Δx(t) = x(t + dt) − x(t). This means that jumps in the process are unlikely.
This condition is called Lindeberg’s continuity condition.
Generalization of Langevin dynamics for d -dimensional time series is straight-
forward. For d -dimensional continuous time series, a necessary ingredient of the
system under consideration is that its dynamical behavior should be described by a
set of macroscopic order parameters x(t) that are governed by the nonlinear Langevin
equations [17],
4 1 Introduction

d
x = a(x) + b(x) (t) (1.4)
dt
where x(t) denotes the d -dimensional state vector, a(x) the drift vector, and the
matrix b(x) isrelated to the diffusion matrix (second-order KM matrix) according
to, Dij(2) (x) = dk=1 bik (x) bjk (x). The noise terms l (t), lumped together in the vector
(t), are fluctuating forces with Gaussian statistics and rapidly decaying temporal
correlations, such that delta-correlation in time can be assumed, i.e., (t) = 0, and,
l (t)k (t  ) = δlk δ(t − t  ).

1.3 Time Series with Jump Discontinuity

In Eq. (1.2) for smooth functions a(x, t) and b(x, t), the process x(t) is diffusion
process and has continuous trajectory. In general, non-vanishing C in Eq. (1.3) or non-
vanishing higher order KM coefficients indicate that a measured time series does not
belong to the class of continuous diffusion processes; see Chap. 12. Accordingly, an
improvement of the Langevin-type modeling, i.e. Eq. (1.2), is needed. An important
generalization of the Langevin-type modeling is to include jump processes, also
known as the jump-diffusion processes, with properties that can also be determined
from measured time series. The jump-diffusion processes are given by dynamical
stochastic equation:

dx(t) = a(x, t) dt + b(x, t) d W (t) + ξ dJ (t), (1.5)

where {W (t), t ≥ 0} is again a scalar Wiener process, a(x, t) and b(x, t) are the drift
and multiplicative diffusion functions, respectively, and J (t) is a time-homogeneous
Poisson jump process. The jump has state-dependent rate λ(x) and jump size ξ
according to some distribution with variance σξ2 . We assume that ξ has Gaussian dis-
tribution, or follows any symmetric distribution with finite moments. This represents
the minimal modeling of a measured time series that contains jumps. In general, one
may assume any distributions for the jump events. In this book, also we will describe
how one models the jumps characteristics—its rate and variance σξ2 —with mixed
Gaussian jumps, where the variance is distributed according to a Gamma distribution.
We describe in Chap. 12 how all the unknown functions and coefficients in Eq. (1.5)
are determined/computed directly from measured time series. Two typical trajec-
tories corresponding to continuous (Brownian type) process and stochastic process
with jumps are shown in Fig. 1.3. Jumps with amplitudes ξ1 and ξ2 are marked with
arrows.
In Chap. 18, we demonstrate that sampling at discrete times gives rise to nonvan-
ishing higher-order conditional moments K (j) (x) with j > 2, even if the underlying
path is continuous. In Chap. 19 we will derive analytically a criterion that allows one
to check whether for a given, even noisy time series the underlying process has a
continuous (diffusive) or jumpy trajectories.
1.4 Microstructural (Measurement) Noise 5

Fig. 1.3 Exemples of

trajectories of a continuous
process (Brownian type,
black line) and a synthetic
discontinuous stochastic
process (red line). For the
latter, jumps with amplitudes
ξ1 and ξ2 are marked with
arrows

1.4 Microstructural (Measurement) Noise

A measured time series may also contain some other noise, which is not assimilated
by the stochastic process. In this case, the time series is written as y(t) = x(t) + (t),
where x(t) denotes the pure stochastic variable, and (t) is an additional noise that
is assumed to be a short-range correlated noise and statistically independent of x(t).
In general, such noise may have its origin in intrinsic components of the complex
dynamics, or can be caused by an external disturbance, e.g., that added to the time
series by the measurement process.
In the literature, such spoiling noise is called differently, either as observational
or measurement noise, or as microstructural noise; an example is in the financial
sciences. The method described in this book is also able to separate and determine the
stochastic behavior of the pure stochastic variable x(t) and the statistical parameters
of the noise (t), such as its variance and higher order statistical moments, etc.
Therefore, we will be able to estimate precisely the “noise” contributions in a given
time series. The microstructural noise is closely linked to the so-called Epps effect
of financial data. Epps observed that there is a dramatic drop of the absolute value
of correlations among stocks, when the sampling frequency increases. In such short
time scales the noise (t) contributes much to the dynamics of y(t) and, due to its
strong fluctuations over short time scales, the predictability of y(t) is decreased. We
note that generally, an original series x(t) has slower dynamics.

1.5 Intermittency

It turns out that the non-parametric method of analysis can also be successfully
applied to not only fluctuating time series, but also to the analysis of spatio-temporal
disordered systems, such as fluid turbulence [18, 19], or characterization of rough
surfaces [27], and the porosity distributions in large-scale porous media. Such struc-
tures can be analyzed as scale-dependent stochastic processes. Experimental observ-
6 1 Introduction

Fig. 1.4 Probability distribution functions (PDF) of the increment statistics, p(Δr x, r) for wind
power fluctuations. Continuous deformation of the increments’ PDFs for time lags r = 1, 10, 1000
sec in log-linear scale are shown. For better clarity the PDFs have been shifted in the vertical
direction, and Δr x’s are measured in units of their standard deviation σr . Extreme events up to about
20σr=1 are recorded. A Gaussian PDF with unit variance is plotted for comparison. The method
described in Chap. 15 provides an evolution equation for the change of the shape or deformation of
the PDF of p(Δr x, r) with respect to scale r. Source from [28]

ables include the field increments, i.e., the difference in the field between two points
separated by a distance or lag r, Δr x(t) = x(t + r) − x(t). The change of the incre-
ments as a function of the scale r can then be viewed as a stochastic process over a
length scale. The method described in this book provides an evolution equation for
the change of shape or deformation of the probability distribution function (PDF) of
p(Δr x, r) with respect to scale r; see Fig. 1.4.
To study the fractal and multifractal behavior of given time series, the approach
can be viewed as an extension of the multifractal description of stochastic pro-
cesses [17]. The multifractal description focusses on the scaling behavior of the
moments of quantities of interest, such as velocity or temperature increments, as a
function of the scale. The complete information on an increment, Δr x(t), is con-
tained in the probability distribution function (PDF) p(Δr x, r) for a certain scale r.
For a self-similar process, it is assumed that the increments exhibit
 ∞ scaling behavior,
Δr x ≈ r ξ , which means p(Δr x, r) = p̃(Δr x/r ξ )/r ξ , where −∞ p̃(u)du = 1. With
locally-varying scaling exponent ξ the PDF is constructed as a superposition of the
probability distributions

1
p(Δr x, r) = dξ p̃(Δr x/r ξ ) f (ξ, r) , (1.6)
rξ

where the measure f (ξ, r) characterizes the distribution of the regions with different
scaling behavior. Knowledge of the deformation equation for p(Δr x, r) with respect
to the scale r enables one to study the scaling exponents of increments for given time
series; see Chap. 15.
1.5 Intermittency 7

In addition, the approach that we describe, which is based on the characterization

of fluctuating fields by a scale-dependent stochastic process, can describe the joint
statistics of the chosen stochastic variable, i.e. increments, on many different scales.
This is achieved by the knowledge of the joint PDF, p(Δr1 x, r1 ; . . . ; Δrn x, rn ). Using
such joint PDFs, the correlations between the scales are also worked out, demon-
strating how the complexity is linked between the scales. If the statistics of the scale-
dependent measure can be regarded as a Markov process evolving in r, knowledge
of two-scale conditional PDF is sufficient for a complete description of multiscale
joint PDF; see Chap. 15 for more details.
Another consequence of the method that we describe in this book is that it enables
us to understand the cascade nature of scale-dependent processes, as well as the
intermittency of the time series. In the intermittent time series the structures arise in
a time series and exhibit themselves as correlated high peaks at random times. The
intervals between them are characterized by a low intensity and a large size. Rare
peaks are the hallmarks of non-gaussian tails of the PDFs. In Fig. 1.4, as an example,
the PDF of the increment statistics, p(Δr x, r), for wind power fluctuations are shown
[28]. Continuous deformation of the increment PDFs for time lags r = 1, 10, 1000
sec in log-linear scale are shown. Here, Δr x is the wind power increments in lag r
and measured in units of their standard deviation σr . Extreme events up to about 20
σr=1 are recorded that deviates strongly from Gaussian PDF.

References

1. H. Haken, Synergetics, An Introduction (Springer, Berlin, 1983)

2. H. Haken, Advanced Synergetics (Springer, Berlin, 1987)
3. H. Haken, Information and Self-Organization: A Macroscopic Approach to Complex Systems
(Springer, Berlin, 2000)
4. H. Haken, Synergetics: Introduction and Advanced Topics (Springer, Berlin, 2004)
5. G. Nicolis, I. Prigogine, Exploring Complexity (W. H. Freeman & Co., San Francisco, 1989)
6. P. Bak, How Nature Works: The Science of Self-Organized Criticality (Oxford University Press,
Oxford, 1999)
7. L. Schimansky-Geier, T. Poeschel, Stochastic Dynamics (Springer, Berlin, 1997)
8. F. Schweitzer, Self-Organization of Complex Structures: From Individual to Collective Dynam-
ics (Gordon and Breach, London, 1997)
9. R.N. Mantegna, H.E. Stanley, An Introduction to Econophysics: Correlations and Complexity
in Finance (Cambridge University Press, New York, 2000)
10. P.L. Gentili et al., Untangling Complex Systems: A Grand Challenge for Science vol. 36, Issue
8, (Rowman & Littlefield Publishers, 2018)
11. D. Sornette, Critical Phenomena in Natural Sciences, 2nd edn. (Springer, Heidelberg, 2003)
12. P. Grassberger, T. Schreiber, C. Schaffrath, Nonlinear time sequence analysis. Int. J. Bifurc.
Chaos 1, 521 (1991)
13. J.D. Hamilton, Time Series Analysis (Princeton University Press, Princeton, 1994)
14. R. Hegger, H. Kantz, T. Schreiber, Practical implementation of nonlinear time series methods:
the TISEAN package. Chaos 9, 413 (1999)
15. H. Kantz, T. Schreiber, Nonlinear Time Series Analysis (Cambridge University Press, Cam-
bridge, 2003)
8 1 Introduction

16. J. Argyris, G. Faust, M. Haase, R. Friedrich, An Exploration of Dynamical Systems and Chaos
(Springer, New York, 2015)
17. R. Friedrich, J. Peinke, M. Sahimi, M.R. Rahimi Tabar, Phys. Rep. 506, 87 (2011)
18. J. Peinke, M.R. Rahimi Tabar, M. Wächter, Annu. Rev. Condens. Matter Phys. 10 (2019)
19. R. Friedrich, J. Peinke, Phys. Rev. Lett. 78, 863 (1997)
20. J. Davoudi, M.R. Rahimi Tabar, Phys. Rev. Lett. 82, 1680 (1999)
21. M. Anvari, K. Lehnertz, M.R. Rahimi Tabar, J. Peinke, Scientific Reports 6, 35435 (2016)
22. P.T. Clemson, A. Stefanovska, Phy. Rep. 542, 297 (2014)
23. W.-X. Wang, Y.-C. Lai, C. Grebogie, Phys. Rep. 644, 1 (2016)
24. H. Risken, The Fokker-Planck Equation (Springer, Berlin, 1989)
25. C.W. Gardiner, Handbook of Stochastic Methods (Springer, Berlin, 1983)
26. Z. Schuss, Theory and Applications of Stochastic Processes: An Analytical Approach (Springer,
Berlin, 2010)
27. G.R. Jafari, S.M. Fazeli, F. Ghasemi, S.M. Vaez Allaei, M.R. Rahimi Tabar, A. Iraji Zad, G.
Kavei, Phys. Rev. Lett. 91, 226101 (2003)
28. M. Anvari, G. Lohmann, M. Wächter, P. Milan, E. Lorenz, D. Heinemann, M.R. Rahimi Tabar,
J. Peinke, New J. Phys. 18, 063027 (2016)
Chapter 2
Introduction to Stochastic Processes

In this chapter we provide mathematical tools to study the stochastic process from
the physical point of view.

2.1 Introduction

The term stochastic process is intuitively associated with a trajectory that randomly
fluctuates and, therefore, it requires a probabilistic description.1

1 To define a stochastic process, let us at first provide the definition of probability space. A probability

space associated with a random experiment is a triple (Ω,F ,P), where, (i) Ω is a nonempty set,
whose elements are known as outcomes or states, and is called the sample state; (ii) F is a family
of subsets of Ω, which has the structure of a Borel σ-field, this means that:
(a) ∅ ∈ F
(b) If A ∈ F , then its complement ∞ A also belongs to F
c

(c) If A1 , A2 , . . . ∈ F then i=1 Ai ∈ F ,

(iii) p is a function which associated a number p(A) to each set A ∈ F with the following properties:
(a) 0 ≤ p(A) ≤ 1
(b) p(Ω) = 1
(c)If A1 , A2 , . . . are pairwise disjoints set in F (that is Ai ∩ A j = ∅, whenever i = j), then
∞ ∞
p( i=1 Ai ) = i=1 p(Ai ).
The elements of the σ-field F are called events and the mapping P is called a probability
measure.
For one flip of a coin, Ω = {H ead = H, T ail = T }. The events F along with the corresponding
F = (Ω) contains all subsets of Ω, i.e. F = {{∅}, {H }, {T }, {H, T }} and p(H ) = p(T ) = 21 . {∅}
neither heads nor tails and {H, T } to having simultaneously H and T , with probabilities 0.
Definition: Let (Ω,F ,P) be a probability space and let T be an arbitrary set (called the index
set). Any collection of random variables x = {xt : t ∈ T } defined on (Ω,F ,P) is called a stochastic
process with index set T .
If xt1 , xt2 , . . . , xtn are random variables defined on some common probability space, then xt =
(xt1 , xt2 , . . . , xtn ) defines an Rn valued random variable, also called a random vector. Stochastic
processes are also often called random processes.

© Springer Nature Switzerland AG 2019 9

M. R. Rahimi Tabar, Analysis and Data-Based Reconstruction of Complex Nonlinear
Dynamical Systems, Understanding Complex Systems,
https://doi.org/10.1007/978-3-030-18472-8_2
10 2 Introduction to Stochastic Processes

Let us start with one dimensional array of random variables {ζ1 , ζ2 , . . . , ζ N },

where N > 1. The possible values of ζn is known as its state space. The states can be
either continues or discrete, depending on the nature of ζn . The position of Brownian
particle and possible appearing of numbers in dice are examples of continues and
discrete random variables. We can also imagine a random variable ζt the varies with
the parameter t, where t is continuous parameter, in this case we deal with a stochastic
process with continuous time. In what follows we provide some statistical tools to
study the continuous-state processes, where ζn can have any real value.

2.2 Statistical Description of Time Series

We summarize the corresponding statistical description of a given time series. Such

a description is achieved by introducing suitable statistical averages. We shall denote
the averages by the brackets · · · . For stationary processes the averages can be
viewed as time averages. For nonstationary processes the averages are defined as
ensemble averages, i.e., averages over an ensemble of experimental (or numerical)
realizations of the stochastic process. Here, we provide a review of the statistical
description of stochastic processes, following the exposition of Risken [1].
Let us start with a one-dimensional time series ζn whose N realizations are:

ζ1 , . . . , ζ N . (2.1)

We can also define ensembles of identical systems and make simultaneous experi-
ments with different initial conditions.2 Although the values of ζ1 , . . . , ζ N cannot be
predicted, some averaging in the limit N → ∞ may be predictable, which then yield
the same value for identical systems. For instance, the mean value of ζn s is defined
as,
1
ζ = lim {ζ1 + ζ2 + · · · + ζ N }. (2.2)
N →∞ N

For some arbitrary function of ζ, i.e. f (ζ), its mean is defined as,

1
 f (ζ) = lim { f (ζ1 ) + f (ζ2 ) + · · · + f (ζ N )}. (2.3)
N →∞ N

2 Thisis true for an ergodic process. A stochastic process is said to be ergodic if its statistical
properties can be deduced from a single, sufficiently long, random sample of the process.
2.2 Statistical Description of Time Series 11

2.2.1 The Probability Density

Let us assume that f (ζ) to be,

f (ζ) = θ(x − ζ) (2.4)

where θ(x) is the step function, defined as,

⎧
⎨1 x >0
θ(x) = 1
x =0 (2.5)
⎩ 2
0 x < 0.

Using the Eqs. (2.3) and (2.5) we write,

 
1 1
θ(x − ζ) = lim n(x > ζ) + n(x = ζ) + 0 (2.6)
N →∞ N 2

where n(x > ζ) and n(x = ζ) are the numbers of events for ζ with x > ζ and ζ to
be equal x, respectively. In terms of probability it can be written as,

1
θ(x − ζ) = P(x > ζ) + P(x = ζ). (2.7)
2
For the processes with continuous states the probability of having an exact value
x for process ζ is P(x = ζ) = 0. Therefore, we find θ(x − ζ) = P(x > ζ). The
probability density pζ (x) of random variable ζ is defied as,

d d d
pζ (x) = P(ζ ≤ x) = θ(x − ζ) =  θ(x − ζ) = δ(x − ζ) (2.8)
dx dx dx

where δ(x − ζ) is the Dirac δ-function. Assuming the differentiability of P, we find,

d
P(ζ ≤ x + d x) − P(ζ ≤ x) = P(ζ ≤ x) d x = pζ (x) d x. (2.9)
dx

Here d P = pζ (x)d x is the probability that the stochastic process ζ belongs to the
interval x ≤ ζ ≤ x + d x.
Using the following identity for δ-function,

f (ζ) = f (x) δ(x − ζ) d x (2.10)

one can define the average of f (ζ) over the distribution pζ (x) as,
Exploring the Variety of Random
Documents with Different Content
 “Why is it that every kid who wants to run away immediately heads
west?” asked Ned of an inattentive and thoughtful Dorothy.
“Sometimes they make a break for the seacoast, but more often it is
the wild and woolly that tempts the youthful imagination. Say, Dot,”
he added, struck by a sudden thought, “why in the world didn’t we
ask that fellow how far west Joe was going?”
“Because we are a couple of idiots, I guess,” returned Dorothy.
“However, we can still ask him—by telephone.”
“How much money did the boy have?” asked Ned, with apparent
irrelevance.
“Not much,” replied Dorothy sadly. “He couldn’t have got so very
far, Ned.”
It seemed only a moment before the train slowed to a stop at North
Birchlands. Dorothy and Ned walked rapidly homewards, eager to
share this new development with the family. But when they reached
The Cedars they found so much worry and excitement rampant there
that they temporarily forgot their own adventures.
Roger was gone, had disappeared as completely, it seemed, as Joe!
Dorothy sank down in a chair and covered her eyes with her hand.
“This is too much,” she said. “I don’t believe I can stand any more.”
Then she was on her feet in an instant again, her eyes bright,
cheeks hot.
“No one has told Dad this?” she asked, and her Aunt Winnie
replied quickly and soothingly in the negative.
“We would not have told him in any case until you returned, dear,”
she said, soon adding, with attempted reassurance: “I really don’t
think this is serious.”
“Serious!” repeated Dorothy. “Not serious that little Roger is lost,
as well as Joe?” Then she asked, looking about her as though she had
missed her chum for the first time: “Where is Tavia?”
“She and Nat have not come in yet,” replied Mrs. White, the
worried lines deepening in her forehead. “I can’t imagine what can be
keeping them.”
Then Dorothy remembered. Tavia and Nat had gone out in the
Fire Bird. Even her chum had deserted her. She felt suddenly very
helpless and forlorn.
 There came the sound of an automobile on the drive without, the
sharp tooting of a motor horn—undeniably the Fire Bird.
They all dashed to the door and flung it open just as Tavia’s glad
cry rang through the darkness:
“Hello, everybody. We’ve got Roger!”
 CHAPTER XI
DOROTHY REACHES A DECISION

Tavia made a rush for Dorothy and caught her in her arms,
hugging her hard.
“Darling Doro, see what we’ve brought you,” she cried, and drew
forward into the circle of light a sheepish and very much subdued
Roger.
Dorothy sank to her knees before Roger and hugged him to her
until he grunted. This was purely physical, however, for the returned
prodigal was willing for once that his big sister should make as much
fuss over him as she wished. It was not much fun to be stuck up in a
tree far away from home and it was most awfully good to be with his
family again. Then, too, he had feared a scolding and Dorothy’s
greeting was a welcome substitute.
It was some time before they were calm enough to discuss the
details of the rescue. But when finally Nat and Tavia did describe the
small boy’s peril and rescue, Dorothy was ashamed to think how she
had misjudged her chum. She ought to have known by this time how
right Tavia’s heart was where her friends were concerned.
They had dinner then, a merry one in spite of the shadow of worry
and anxiety that still hung heavy on their minds. Despite his
famished state, Roger was so exhausted by the strenuous and
exciting events of the past few hours that he almost fell asleep in his
chair and had to be helped to bed before he had half finished his
dinner.
Dorothy, looking down at his sleeping face, so dear and innocent
on the pillow, felt her eyes smart with fresh tears. Kneeling down
beside the bed, she pressed her cheek to his soft one.
 “Don’t ever do a thing like that again,” she whispered. “What
would Doro do if anything happened to her Roger?”
One small arm twined about her neck and Roger half opened his
eyes, smiled sleepily.
“Roger—loves—Doro,” he murmured, and fell asleep.
On the way downstairs Dorothy stopped in the Major’s room to see
how he fared and found him also asleep. She would not disturb him
now till morning although she knew how eagerly he would grasp at
the one small item of news concerning Joe that she had to tell him.
If Joe were only there too, beneath the familiar roof, asleep—
Dorothy sighed, closed the door gently, and went on downstairs.
“Ned has just been telling us about Joe’s cap, Doro,” said Tavia, as
she entered the room. “Isn’t it marvelous? We have an honest-to-
goodness clue at last.”
“Although I can’t see where it leads us——”
“To the West, of course,” interrupted Tavia. “How dull you are,
Nat.”
Nat grinned good-naturedly.
“The West is a large place, young lady,” he reminded her. “And one
that it is possible for a lad to get pretty completely lost in.”
“We will find to-morrow what town or city he bought his ticket to,”
said Dorothy. “And then we can act accordingly.”
“That sounds as if the fair Dorothy were about to get busy in
earnest,” said Tavia, with a shrewd glance at her chum. “Have you
made any plans yet, Doro?”
“Nothing definite,” Dorothy confessed. “I want to talk with Dad
first.”
It was Major Dale himself who asked for Dorothy on the following
morning, and father and daughter were closeted together for the
better part of an hour.
When Dorothy at last emerged from the interview her cheeks were
flushed and her mouth determined.
Tavia, who had been eagerly awaiting an opportunity to talk to her
chum, was the first to notice this change in her.
“You look as though you were on the war path, Doro. What’s up?”
 Dorothy held a finger to her lips as Ned’s voice at the telephone
came up to them.
“He’s calling Scranting,” Dorothy explained in a whisper. “Listen!”
They listened with breathless interest to Ned’s disjointed
monologue.
“This Mr. Dougherty, Scranting station? Mr. Dougherty, Miss Dale
and I forgot to ask you a very important question last night—. Oh,
you thought of it too, did you?—Chicago! Where did the kid get all
that money?—Yes.—All right. Many thanks for the information.—Yes,
I will.—Thanks again. Good-bye!”
“Chicago!” repeated Tavia, whistling softly. “That city is a
considerable distance from this place, Doro. Why, what’s the
matter?” She broke off and stared at her chum wonderingly.
For, impossible as it seemed to her, Dorothy’s lips had curved
suddenly in such a smile as Tavia had not seen for days.
“Oh, nothing!” said this amazing Doro. “I was just thinking that
intuition is a wonderful thing sometimes!”
Even by persistent questioning Tavia was not able to discover the
reason for what she called Dorothy’s “Mona Lisa smile,” but she did
succeed in extracting other valuable information.
Dorothy was to follow the one clue they possessed, though it was a
slight one.
“But how on earth can you go out West all alone, Doro?” cried
Tavia, when her chum had announced her decision to the rather
startled and excited family group.
“I didn’t intend to,” returned Dorothy with assumed
ingenuousness. “I thought perhaps one, Tavia Travers, would like to
go with me.”
“Good gracious, I was only scared to death for fear you wouldn’t
ask me,” Tavia confessed. “When do we start, Doro?”
“Hold your horses a minute, will you?” cried Nat. “You two girls
aren’t going on a journey like that all alone—not by a long shot!”
“O-ho! The cave man speaks!” gibed Tavia. “Who says we are not,
Mr. Smarty?”
“You really ought to stay here, Nat,” Dorothy interposed swiftly.
“We need both you and Ned here on the spot, both to take care of
Dad and follow up any new clue that may turn up.”
“Well, I like that!” exclaimed Nat, chagrined. “That’s being
relegated to the rocking chair for fair.”
“But you will do that for me, won’t you, Nat?” begged Dorothy.
“Can’t you see it’s the best way?”
“Well, no, I can’t say that I can,” confessed Nat. “But if you want it
that way, Dot, I can but oblige.”
“What are you going to do after you reach Chicago?” Mrs. White
asked. “Have you thought of that?”
“I suppose we shall have to leave our future conduct to chance,”
said Tavia flippantly, and Dorothy slowly nodded acquiescence.
“We may come up against a dead wall,” Dorothy admitted. “But
there is just a chance that we may pick up a clue there that will be
useful. Anyway, Dad thinks the chance is worth taking, and I do too.”
So it was decided that the two girls were to start for Chicago the
following day, “traveling light.”
After they had gone to their rooms that night and Tavia was
brushing her hair before the mirror, Dorothy stole in to her and
whispered:
“Tavia, if I tell you a secret will you promise never to tell a soul?”
 CHAPTER XII
A GUESS

“Cross my heart and hope to die,” said Tavia. “Tell me quickly ere I
pass away with suspense.”
“Well, I have a very good suspicion which way Joe headed.
“He headed West——”
“Exactly! And straight for the ranch of one young Westerner called
Garry Knapp.”
Tavia looked at her chum hard for a moment, then waved the hair
brush aloft in a jubilant gesture.
“I do believe you have struck it, Doro!” she cried. “Of course that is
the obvious thing for him to do.”
“He always loved Garry——”
“Seems to run in the family,” interrupted Tavia.
“And he would naturally go to him for help and advice at this
time.”
“He hasn’t reached his objective yet, if Garry’s ranch is the
objective,” Tavia pointed out. “If he had, Garry would have
telegraphed.”
“I’ve thought of that, of course,” admitted Dorothy. “But then, if he
went directly he has hardly had time yet. Anyway, there is no use
guessing any longer,” and she rose abruptly from the bed and gave
Tavia a good-night hug. “To-morrow we begin to act.”
“For which, thanks be!” said Tavia fervently.
It was a very much disgruntled Nat who saw them off the following
morning. The waiting end of a game was never a pleasant one to him.
And, it meant losing Tavia for an indefinite time!
 However, Tavia managed to tear herself away finally, and after
Dorothy also had been hugged and kissed the train moved off and
the two girls sank back in their seats with a feeling of relief that at
last their adventure was in motion.
Tavia brought forth the two-pound box of candy that the boys had
bestowed upon her and her chum and began contentedly to untie the
ribbon that bound it.
“Have one, Doro?” The latter shook her head. She was too full of
anxiety for Joe and the dear ones at home to think about anything
else.
The Major had seemed very frail that morning when he had said
good-bye, but there had been an eager light in his eyes that she
understood only too well. He had been thinking that the next time he
saw his daughter, Joe might be with her.
And Joe would be with her! Dorothy’s chin went up and her eyes
gleamed in a manner curiously suggestive of the Major in the days
when the success of the Bugle meant everything to him.
“Good gracious, Doro, don’t look like that!” cried Tavia, happening
that moment to glance at her chum. “You remind me of bulldogs and
prize fighters and other pugnacious animals.”
“How extremely complimentary you are,” laughed Dorothy. “I’ll
have you know that though I can’t get over the fact that I’m an
animal, I’m not pugnacious.”
“Far be it from me to contradict a lady,” retorted Tavia. “But if you
could have seen yourself at that moment, Doro, I am sure you
wouldn’t blame me.”
“Glad I didn’t then,” replied Dorothy a trifle crossly. “It must be an
awful bore to see yourselves as others see you.”
“Well, take off your hat, anyway,” advised Tavia irrelevantly. “We
have quite a little ride before us, you know.”
“As if I hadn’t lain awake all night thinking of that!” cried Dorothy.
“And every minute of the journey will seem like an hour.”
“Now who is being uncomplimentary?” chuckled Tavia. “You must
expect to enjoy your company.”
“I don’t expect to enjoy anything again until I get news of Joe,”
answered Dorothy morosely, and Tavia sighed gustily.
 “Here’s where all my efforts at entertainment fall upon barren
ground,” she prophesied. “Like casting pearls before swine, you
know.”
“Are you, by any chance, calling me names?” asked Dorothy,
giggling in spite of herself.
“I wouldn’t do such a thing,” protested Tavia virtuously. “I was
thinking of that cute little pig I just saw beside the road. Honestly, he
was awfully cute. His tail was all curled up and he had the pinkest
nose——”
“Goodness, Tavia, if you can’t be sensible I am going out and sit on
the observation platform by myself. I don’t want to hear about pigs.”
“I don’t know but what your suggestion about the observation
platform is a good one, at that,” remarked Tavia, unmoved. “Did you
notice that perfectly stunning man who passed through our car a few
minutes ago? He looked straight at you and you looked straight
through him.”
“Was he a ghost?” giggled Dorothy.
“Far from it!” returned Tavia, with a reproving stare. “He was an
extremely substantial looking young man, and from the way he
looked at you I shouldn’t wonder but that your amazing beauty had
quite bowled him over, Doro, my dear.”
“Well, I hope he stays bowled,” returned Dorothy unfeelingly.
“Something tells me that’s where he belongs.”
“Pearls before—” began Tavia, but this time Dorothy rebelled.
“I won’t be called a pig again, Tavia Travers!”
“Such a cute little pig!”
Dorothy fumbled at the car window and looked back at Tavia
suggestively.
“Will you stop, or shall I jump?”
“Better wait till the train slows down a bit,” replied Tavia calmly.
“Going at this rate of speed, you might skin your knuckles or
something.”
Dorothy sank back in her seat with a sigh of resignation.
“I think I shall go to the observation platform, after all,” she said,
but before she could rise Tavia seized her arm and cried excitedly:
 “He is coming back!”
Dorothy shook her arm free and frowned.
“Well, what of it?”
“And he has a companion,” added Tavia. “Good gracious, if I ever
saw a desperado, Dorothy Dale, that man is it!”
Interested in spite of herself by Tavia’s description, Dorothy
turned her head and beheld two men approaching down the car aisle,
lurching as the train lurched.
One was the tall, dark, good-looking stranger who Tavia had
vulgarly declared was “bowled over” by Dorothy’s beauty. His
companion could not have been more completely his opposite. A
short, squat fellow with a flat face and sharp black eyes, he looked for
all the world like a bird of prey, ready to snatch at his victim.
Dorothy, as she shudderingly appraised the man, was glad she was
not to be his victim. The next moment she was laughing at her
melodramatic thoughts.
“Probably a traveling salesman or something equally innocuous,”
she whispered, as the two men passed close to them.
“He’s a desperado,” Tavia reiterated stubbornly. “You mark my
words—that fellow will come to no good end—”
At that moment it seemed as if they all were to come to a very bad
end indeed.
There came a deafening crash and the car in which Dorothy and
Tavia sat seemed to rear up in the middle, like a balky horse.
“Good gracious, hold on to me, Doro!” shrieked Tavia. “It’s the end
of the world!”
 CHAPTER XIII
DERAILED

There was shrieking and confusion from one end of the train to the
other as the car righted itself again. With a horrid noise of scraping
brakes the cars ahead came to a jolting standstill.
Tavia was out of her seat bent on joining the general stampede for
the door, but Dorothy held her back firmly.
“You will be hurt in that rush!” she cried. “Wait a minute; do,
Tavia.”
Tavia obeyed, and crouched down in the seat and covered her eyes
with her trembling hands.
“Oh, listen to those cries, Doro!” she wailed presently. “Somebody
must be horribly hurt.”
“Just hysterics, Miss.”
A man, one of those who had been the first to jump from the train,
returned and sank into a seat opposite the two girls. “The car ahead
of us jumped the track, and it’s a mercy the whole train wasn’t
wrecked. As it is, they ain’t nothing to worry about, except that we
may be tied up here for some considerable time.”
Tavia uncovered her eyes and looked at him. Dorothy had already
done so and had risen from her seat and started hastily for the door,
because this man who had undertaken to reassure them was none
other than the villainous looking companion of the tall dark stranger!
At her sudden motion the man put out his hand and made as
though to rise.
“Better not go out there just now, Miss,” he said, his beady black
eyes resting upon her admiringly. “The crowd is still mighty
hysterical and it’s possible you might get hurt.”
 Dorothy might have retorted that she preferred the hysterical
crowd to the doubtful pleasure of his company, but she held her
tongue.
Instead she smiled noncommittally and held out her hand to
Tavia.
“Come along, dear,” she begged. “There may be something we can
do out there.”
“I tell you there ain’t nobody hurt,” again put in the small, squat
man in a faintly irritable voice. “Better stay right here—”
But the two girls were already half way to the door, Tavia
accompanying her chum grumblingly.
“Every time anything interesting happens, Doro, you have to come
along and spoil everything.”
“If you call that fellow interesting, then I am disappointed in your
common sense,” retorted Dorothy tartly. “Sometimes, Tavia, I really
think you need a nurse.”
“Well, any time that I feel like engaging one, I’ll tell you,” drawled
Tavia, angered in her turn, and there fell an uncomfortable silence
between the girls.
Mechanically they walked through the excited crowd on the
platform to the spot where the car had jumped the track. There it
stood, its wheels on the gravel bed of the roadside, tilted crazily and
only held upright by the cars in front and at the rear of it.
“The people in this car must have been jolted up for fair. Thought
it was an earthquake or something,” murmured Tavia, interest
getting the better of her anger at Dorothy. “It’s a wonder we didn’t
have an honest-to-goodness wreck out of this.”
“It was the quick wit of the engineer who saved us, I guess,” said a
musical voice behind her, and, astonished, the two girls turned about
to find behind them the tall good-looking stranger who had caught
Tavia’s particular attention.
The eyes of the irrepressible girl sparkled as she muttered in a tone
audible only to Dorothy:
“We can’t run amiss of ’em, no matter how hard we try.”
Dorothy flushed with annoyance and pretended she had not heard
the man’s observation. Not so Tavia! If for no other reason than to
annoy her chum she determined to see the adventure through.
“We should get up a vote of thanks and send it to the engineer,”
she said in her sweetest tones. “He really was quite heroic. Fancy
saving the lives of all the people on this train.”
“Just fancy!” mimicked Dorothy bitterly, but the young man was
not to be so easily discouraged.
He immediately ranged himself beside the two girls and launched
into a boringly detailed account of the accident. In the middle of it
Dorothy excused herself and hurried back to the car.
Her cheeks were hot and she felt unreasonably angry with Tavia.
To her mind her chum had always been far too easy-going and casual
with men, and this, Dorothy thought, was going a little too far.
It was not that Tavia had responded to the stranger—that might
have been excusable under the circumstances. It was the manner of
her response.
She wondered if the offensive, squat man would still be occupying
the seat opposite her when she returned to the car. She was busy
framing a scathing speech as she ascended the car steps, but was
immensely relieved a moment later to find that there was no need of
delivering it.
The fellow had evidently been discouraged by her manner—
sufficiently, that is, to slightly dampen his enthusiasm.
Yet he still lingered uncomfortably near. Dorothy was annoyed and
more than a little alarmed to find that he occupied a seat in the same
car with her and Tavia.
On the entire trip then, they would be forced to suffer the
annoyance of his presence, to ward off his offensive attentions.
Dorothy could see that he often glanced at her over the top of the
paper he pretended to be reading and knew that it needed only a
word or a glance from her to bring him instantly to her side.
She wished more than ever that Garry were with her. He would
know how to deal with offensive strangers who took advantage of the
confusion and excitement consequent upon a train accident to
become familiar.
She thought of Tavia, still, presumably, busy fascinating the good-
looking stranger. This was always an interesting pastime with Tavia,
and it would probably be some time before she tired of it.
If she had the audacity to bring that man into their car—Dorothy
gasped for, out of the corner of her eye, she saw that was just what
Tavia was doing.
Her color high, she turned and looked steadily out of the window
as Tavia and her latest conquest approached. The latter seemed
about to take the seat his unpleasant friend had so recently vacated
but a glance at Dorothy’s averted profile warned Tavia that, for the
time, she had gone far enough.
“Thank you so much!” she said sweetly, sinking into the opposite
seat and adroitly placing a box of candy—the gift of her new friend—
upon the other half of the seat, so that there was no room left for
him. “You are in this car, too, and going through to Chicago? How
nice! Ah, yes, thank you,” as the young man handed her a magazine
that had fallen to the floor.
The latter lingered, indulging in inanities—or so Dorothy termed
them—with Tavia, but evidently interested in Dorothy’s stubbornly
averted profile.
At length, as his room was so patently desired to his company, he
reluctantly moved on, joining his unpleasant friend.
Tavia looked at Dorothy with a sparkle in her eye. Evidently she
had been enjoying herself immensely and was in a conciliatory
mood.
“Don’t be mad with me, Doro, darling,” she coaxed. “I know I’m a
perfect simpleton. But I was born that way, you know. I really can’t
help it.”
“You could help a good many things, Tavia, if you wanted to,” said
Dorothy, turning away from the window. “Sometimes I wonder how
you can be in love with Nat and still act the way you do.”
“Well, I am in love with Nat and that’s all that matters—to Nat and
me,” retorted Tavia, her voice suddenly hard and cold. “I think you
are too absurdly conventional for words, Dorothy Dale. If you insist
on being a spoil-sport, then you can be one by yourself. I don’t intend
to help you!”
And so began the quarrel—the first real one the girls had ever had,
and one that lasted all through that miserable journey to Chicago.
 Tavia, through a perverse desire to torment her chum, was almost
constantly to be seen in the company of the young man whose name,
according to him, was Stanley Blake.
Chicago came at last, and with it an immense relief to Dorothy
Dale. Her relief vanished immediately, however, when she found that
Stanley Blake had taken the place of a porter and was to carry their
bags.
“He shan’t carry mine,” she said, in a sudden fury, to Tavia. “If you
want to go on being an—an——”
“Idiot. You might as well say it,” Tavia finished for her. “You can
do as you please, Doro. If you want to make a scene over such a
foolish little thing—— Come on, be a sport,” she added, suddenly
conciliatory again. “What’s your awful objection to saving a porter’s
tip?”
Dorothy bit her lips to keep back a flood of angry words. She could
not very well make a scene by refusing the attentions of this man
when Tavia so casually accepted them. She would, she decided, put
up with Tavia’s folly once more, but, after that— She was fortified by
the knowledge that they were now at their journey’s end and so
would automatically dispense with the company of Stanley Blake and
his fox-eyed friend.
They were in their room in the Blenheim Hotel at last. Tavia and
she were alone.
“Thank goodness, we’re rid of them,” thought Dorothy, as she
removed her hat and sank wearily upon the edge of the hard, hotel
bed. “I hope I never have to see either of them again.”
But she did, and that in a way that was not only unpleasant but
exceedingly startling.
Descending with Tavia to the hotel dining room, Dorothy saw at a
table near the door the very two persons whom she had so recently
and fervently wished never to see again! Tavia had not seen them yet,
and Dorothy prayed fervently that she might not.
The head waiter coming toward them and beaming benignly
seemed like a rescuing angel to Dorothy. She must get Tavia seated
somewhere, anywhere, before she became aware of the presence of
Blake and his friend. To have again their company thrust upon her
was unthinkable.
 Even at that last moment she would have turned away, urged Tavia
to go with her to some quiet, small restaurant outside. But it was too
late. The head waiter already was guiding them toward a table.
The table was next to the one at which Blake and his friend sat, at
the side and a little to the rear of it. Dorothy gasped, would have
protested could she have done so without rousing the suspicion of
her friend.
For Tavia was still blissfully unaware of anything unusual in the
atmosphere. And the head waiter, with a beaming smile, had
motioned one of the waiters to take their order.
Well, it couldn’t be helped, thought Dorothy resignedly. If Tavia
saw them she would have to. Lucky the two men were sitting with
their backs toward the table where the chums were ensconced, and,
by skillful maneuvering on Dorothy’s part, Tavia also had her back
turned to them.
Dorothy turned sideways so that only her profile would be exposed
to view, if either of the men chanced to glance over his shoulder.
Suddenly she stiffened, for, coming to her with a startling
distinctness above the noise and chatter all about her, she heard a
familiar name.
It was a very familiar name. The two men were talking about Garry
Knapp!
“What is the matter, Doro?” asked Tavia, looking at her curiously.
“You resemble a storybook detective on the eve of a startling
discovery.”
Dorothy motioned her sharply to be still.
“They are talking of Garry,” she explained, in a tense whisper.
“Who? When? Where?” cried Tavia, screwing her head about most
absurdly in a vain effort to bring the entire dining room within her
range of vision at the same time. “What do you mean, Doro?”
Dorothy gestured toward the two men at the table next to them, at
the same moment making an imploring gesture pleading silence.
“Why, Stanley Blake and his dear little friend!” exclaimed Tavia in
a tone of pleased surprise. “Always turning up like the proverbial bad
penny, aren’t they, Doro? Do you mind if I ask them to join us?”
 She half rose from the table as if about to carry out her
preposterous threat, but Dorothy seized her fiercely by the arm and
forced her back into her seat.
“If you move or say a word, I never will speak to you again!” she
said, and at the vehemence of the usually gentle Dorothy, Tavia
looked surprised. However, she obeyed and remained curiously
quiet.
Dorothy had missed something of what the men had said. She
realized this with a sharp annoyance. But the next moment a wave of
rage and fear swept over her, blotting out every other sensation.
They were not only speaking of Garry, these two men, but they
were threatening him as well. She held her breath so that she might
not miss one word of what was to follow.
“He is a kind of simple guy, this Dimples Knapp,” the beady-eyed
man was saying with a half-satisfied smirk. “Thinks this old world is
made up of goody-goody stiffs who believe in the Golden Rule and go
to church regular twice on Sundays. A cute little lamb to fleece!”
“And a nice fat, succulent one,” added Stanley Blake, in a voice
neither of the girls recognized. It had a cold, mean quality that made
Dorothy shiver, though the dining room was hot.
She glanced at Tavia and saw the look of bewilderment and horror
on her face. Tavia had “caught on” at last. She was beginning to find
that Dorothy’s aversion to these two men had been founded on
something very much more real than a whim.
 “THEY ARE TALKING OF GARRY,” SHE
EXPLAINED, IN A TENSE WHISPER.

“Dorothy Dale to the Rescue.” Page

99

“What does it all mean, Doro?” she whispered, but once more
Dorothy held up her hand for silence.
“Wait, and perhaps we shall hear,” she said tensely.
“The fellow thinks he’s goin’ to have the best l’il wheat ranch in the
West,” went on Stanley’s companion, pushing back his plate and
lighting a cigar. “He’s got the cash to do it and—I feel forced out o’
the kindness of my heart to say it, Cal—he’s got the brains. If it
wasn’t for that trustin’ little disposition of his—” he did not finish the
sentence, but ended with a chuckle, a thin, mean alien sound in that
convivial atmosphere.
 Dorothy was the victim of a chill fear. The man was like a snake, a
mean, poisonous snake that would lie treacherously still in a crevice
of rock awaiting the moment to strike at an unsuspecting prey.
She thought of that horrible moment during her first trip to Desert
City, seemingly ages ago, when she had flung the rock that had
snuffed out the life of the rattlesnake that had threatened the life of
her chum. She had acted then swiftly, unerringly, not thinking of
herself, but of Tavia’s peril.
But this was another, a more venomous kind of reptile, and
something told her he would be infinitely harder to deal with.
Stanley Blake was speaking now, and both she and Tavia listened
breathlessly.
“You may think this fellow Dimples Knapp is easy game, Gibbons,
but I know better,” drawled the hero of Tavia’s gay moments. “He
may be as trusting as you say he is, but I tell you he’s got friends that
were not born yesterday. And they weren’t born blind, either.”
“I s’pose you mean that snoopin’ Lance Petterby an’ his gang,”
snarled the little man, and the girls started nervously. “Well, I’m
goin’ on record now to the effect that if he tries any funny business,
it’ll be the last time, that’s all. You hear me, Cal, it’ll be the last time!”
“Say, you poor little shrimp, will you cut out calling me by my first
name? This is the second time you’ve done it in the last five minutes.
Getting childish or something, aren’t you?”
The man whose name quite obviously was not Stanley Blake
glanced hastily about the room as he gave vent to these irritable
remarks, and Dorothy turned hastily aside lest he should recognize
her profile, and so put an end to his remarkable discourse.
However, though the men continued talking and, presumably, on
the same subject, it did not take Dorothy long to realize that she
would hear nothing further of importance that day.
The two men, evidently beset by an excess of caution, had lowered
their voices so that it was impossible to catch a word of their
discourse.
Although the girls strained their ears, the conversation at the next
table became only a confused mumbling and soon afterward the two
men rose and left the dining room.
 Although she had scarcely tasted her lunch, Dorothy rose too.
“Where are you going, Doro?” asked Tavia.
“To the office,” said Dorothy. “I must send a telegram to Garry at
once!”
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com