Time Series Analysis and Forecasting

STAT-684

Dr. Said Farooq Shah

Department of Statistics UoP, Peshawar

[email protected]

8 June 2020

Farooq Shah Time Series (1 of 2) 08-Jun-20 1/ 46

Content

1 Time Series concepts

Stochastic Process
Time Series
Stationary stochastic process
White Noise Process
Random Walk Process
Unit Root Stochastic Process
Lag Operator
Difference Operator
Deterministic trend and stochastic trend
Integrated Stochastic Process

Farooq Shah Time Series (1 of 2) 08-Jun-20 2/ 46

Stochastic process

A stochastic (random) process is a collection (sequence) of random

variables which are ordered (indexed) by an index variable.
The index variable is usually time, but it can be space or another
variable that can be ordered. In this lecture, we will only consider time
as the index variable, and specifically to a discrete time sequence.
In each time point of the process we have a single random variable
(for univariate analysis) denoted by Yt, at time point t. If the time
variable is continuous the process is denoted as Y (t). However most
economic data are collected at discrete points in time.
The stochastic process itself is denoted as {Y t } t∈T where T is the
index set (usually the set of integer numbers).

Farooq Shah Time Series (1 of 2) 08-Jun-20 3/ 46

Time series

A time series is a series of data points ordered by time.

Properties:
A time series is a single occurrence of a stochastic process.
A time series model describes the variable Yt in terms of
contemporaneous (and perhaps lagged) factors X t , disturbances et,
its past (lagged) variables, Yt−1, . . . , Yt−p, and functions of time.
Just as we use samples in cross-sectional data to draw inferences
about the underlying population, in time series we use realizations of
the random variables over time (a time series) to draw inferences
about the underlying stochastic process.

Farooq Shah Time Series (1 of 2) 08-Jun-20 4/ 46

Time series - Examples

Examples (for more examples see the book (p.1 to p.7):

An example of a time series is the quarterly series on GDP of a country

over a period of time (say, from 1970 to 2007), which is a single
realization of a stochastic process, {GDP t } t∈T .
In theory, the value of the GDP for a particular time point could be
any number, depending on the economic and political climate. The
observed value, say $3, 759, 997, is a particular realization of all such
possibilities.
Therefore GDP is a stochastic process and the actual values we
observed for the period 1970 to 2007 are particular realizations of
that process. These observed values are thus samples.

Farooq Shah Time Series (1 of 2) 08-Jun-20 5/ 46

Time series - Visual examples

Farooq Shah Time Series (1 of 2) 08-Jun-20 6/ 46

Stationary stochastic process

A stochastic process (or time series) is (weakly) stationary if:

1 The means of the random variables in the series are all equal.

E(Y t ) = µ < ∞ for all t.

2 The variances of the random variables in the series are all equal.

var(Y t) = σ2 < ∞ for all t.

3 The covariances between any two random variables in the series depend only
on the distance between the two time points the variables are realized, and
not on the actual time points of their realizations. (Autocovariance)

cov(Yt, Y t+k ) = γk for all t and k.

Note: γ0 = σ2.

Farooq Shah Time Series (1 of 2) 08-Jun-20 7/ 46

Stationary stochastic process

If any of the three conditions are violated we say that the stochastic
process is nonstationary.

Farooq Shah Time Series (1 of 2) 08-Jun-20 8/ 46

Stationary stochastic process - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 9/ 46

Nonstationary stochastic process - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 10/ 46

Nonstationary stochastic process - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 11/ 46

Nonstationary stochastic process - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 12/ 46

Stationary stochastic process

What does stationarity mean, and why is it important?

The mean, variance and autocovariances of a stationary time series
are time-invariant, meaning that they are the same no matter at
which time point we started to realize the time series.
A stationary time series tends to return to its common mean µ (mean
reversion) and fluctuations around the mean have a broadly constant
amplitude. The speed of mean reversion also depends on the
autocovariances, where for small covariances the mean reversion is
faster.
If a process is nonstationary, we can study the time series for the time
period under consideration. As a consequence, it is not possible to
generalize it to other time periods (especially when forecasting).

Farooq Shah Time Series (1 of 2) 08-Jun-20 13/ 46

White noise process

A particular type of stationary stochastic process often used in statistics is

the so called white noise process (less commonly know as purely random
process). A stochastic process is called white noise if:
1 The means of the random variables in the series are all zero.

E(Y t ) = 0 for all t.

2 The variances of the random variables in the series are all equal.

var(Yt) = σ2 < ∞ for all t.

3 Any two random variables in the series, realized in to different time

points, have zero covariance, i.e., uncorrelated.

cov(Yt, Y t+k ) = γ k = 0 for all t and k.

Farooq Shah Time Series (1 of 2) 08-Jun-20 14/ 46

White noise process - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 15/ 46

White noise process - Example

An example of a white noise process is iid variables from N (0, σ 2 ). Such a

process is called Gaussian white noise process.

The error term in the classical linear regression model is assumed to be a

white noise process, i.e. independently and identically distributed as a
normal distributions with zero mean and constant variance.

yt = β0 + β1x1t + ···+ βpxpt + et,

where et ∼N (0, σ 2 ).

Farooq Shah Time Series (1 of 2) 08-Jun-20 16/ 46

Random walk process without drift

Even though our interest is in stationary time series, we often encounter

nonstationary time series. A classical example of a nonstationary time
series is the random walk process.
Suppose ut is from a white noise process with mean zero and variance
σ2. Then the series {Y t } t∈T is a random walk (without drift) if

Yt = Yt−1 + et.

The above random walk is non-stationary, since the variance of Yt is

not constant. In fact, it becomes larger and larger over time.

Farooq Shah Time Series (1 of 2) 08-Jun-20 17/ 46

Random walk process without drift

Yt = Yt−1 + et.

If we suppose the initial value of the random walk is Y0 (constant),

we can write the model as,

Yt = (Yt−2 + et−1) + et = ((Yt−3 + et−2) + et−1) + et

E(Y t ) = Y0 (constant), but var(Yt) = t × σ2 (exploding over time).

Farooq Shah Time Series (1 of 2) 08-Jun-20 18/ 46

Random walk process without drift - Visual example 1

Farooq Shah Time Series (1 of 2) 08-Jun-20 19/ 46

Random walk process without drift - Visual example 2

Farooq Shah Time Series (1 of 2) 08-Jun-20 20/ 46

Random walk process with drift

Random walk process with drift

Suppose ut is from a white noise process with mean zero and variance
σ2. Then the series {Y t } t∈T is a random walk with drift if

Yt = 𝜃 0 + Yt−1 + et.

The above random walk is non-stationary, since the expected value

and variance of Yt is not constant. In fact, both becomes larger and
larger over time.

Farooq Shah Time Series (1 of 2) 08-Jun-20 21/ 46

Random walk process with drift

Random walk process with drift

Yt = 𝜃 0 + Yt−1 + et.

If the initial value of the random walk is Y0 (constant), we have

𝑡

𝑌𝑡 = t× 𝜃 0 + 𝑌0 + 𝑒𝑖
𝑖=1
Thus, E(Y t ) = t × 𝜃 0 + Y0 and var(Yt) = t × σ2, and
therefore nonstationary.

Farooq Shah Time Series (1 of 2) 08-Jun-20 22/ 46

Random walk process with drift - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 23/ 46

Unit root stochastic process

Let us write the random walk model as

Yt = ρYt−1 + et,

where −1 ≤ ρ ≤ 1 and ut is a white noise process with mean zero and

variance σ2.

This model is in fact called the first-order autoregressive (AR(1)) model

(covered in the next lecture) and is only stationary when |ρ| < 1. If ρ = 1
the model becomes a random walk with drift and we face what is called
the unit root problem, i.e., a situation of nonstationarity. The name
originates from the fact that ρ = 1.
The terms nonstationarity, random walk, unit root and stochastic
trends can be used anonymously.

Farooq Shah Time Series (1 of 2) 08-Jun-20 24/ 46

Unit root stochastic process

Yt = ρYt−1 + et,

The mean and variance of an AR(1) process are

c σ2
E(Y t ) = and var(Yt) =
1−ρ 1 − ρ2

Thus |ρ| has to be less than 1 for the process to be stationary.

Farooq Shah Time Series (1 of 2) 08-Jun-20 25/ 46

AR(1) - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 26/ 46

Unit root stochastic process

The distinction of stationary and nonstationary time series is important on

whether trends (long-run evolution) observed in time series are
deterministic or stochastic.
If a trend in a time series is a function of time we call it a
deterministic trend.
If a trend in a time series is not predictable we call it a stochastic
trend.
However, before we talk about deterministic vs stochastic trends two
operators will be introduced, namely the lag operator and the difference
operator.

Farooq Shah Time Series (1 of 2) 08-Jun-20 27/ 46

Lag operator

The lag operator on time series, denotes as L, is defined as

LYt = Yt−1,

which reduces the time index by one (lagged one unit).

Examples:
L2Yt = L(LY t ) = L(Y t−1 ) = Yt−2.
More generally: L k Y t = Yt−k and L0Yt = Yt.

Farooq Shah Time Series (1 of 2) 08-Jun-20 28/ 46

Lag operator

The AR(1) model

Yt = ρYt−1 + et,
can be written as
(1 − ρL)Yt = et.
The term unit root refers to the root of this polynomial in the lag
operator. If ρ = 1 then the root of the polynomial is L = 1, hence the
name unit root. Obtained by setting

(1 − L) = 0.

Farooq Shah Time Series (1 of 2) 08-Jun-20 29/ 46

Difference operator

The difference operator on time series, denotes as ∆, is defined as

∆Y t = Yt − Yt−1.

The above operation is called the first order difference.

Examples:
∆ 2 Y t = ∆ (∆ Y t ) = ∆(Y t − Yt−1) = ∆Y t − ∆Y t−1 = Yt − 2Yt−1 + Yt−2
∆ 0 Y t = Yt.
Note: The differenced time series ∆Y t is a time series which can be
stationary or not, just like any other time series.

Farooq Shah Time Series (1 of 2) 08-Jun-20 30/ 46

Lag operator and difference operator

We note the relationship between the lag operator and the difference
operator
∆Y t = (1 − L)Y t
and more generally
∆ k Y t = (1 − L) k Y t .

Farooq Shah Time Series (1 of 2) 08-Jun-20 31/ 46

Deterministic trend and stochastic trend

Let et be a stationary process with mean zero and variance σ2 (for example
a stationary AR(1) or white noise), and define the series {Y t } t∈T as

Yt = f (t) + et,

where f (t) is a function of time (for example a polynomial). Then the

model is called a deterministic trend model.

Farooq Shah Time Series (1 of 2) 08-Jun-20 32/ 46

Deterministic trend and stochastic trend

Yt = f (t) + ut,

A deterministic trend model is not stationary, since E(Y t ) = f (t).

var(Yt) = σ2 so the fluctuation around the trend is the same.
If the trend is removed the time series is stationary, i.e. Yt − f (t) = ut
is stationary, and {Y t } t∈T is said to be trend stationary.
If the trend is not predictable for a time point, due to accumulation of
stochastic fluctuations, it is called a stochastic trend (as in a random
walk).

Farooq Shah Time Series (1 of 2) 08-Jun-20 33/ 46

Deterministic trend vs stochastic trend

Farooq Shah Time Series (1 of 2) 08-Jun-20 34/ 46

Examples of models with trends
Suppose et is from a white noise process with mean zero and variance σ2. Below
we find common time series models.
Pure random walk
Yt = Y t−1 + et

Random walk with drift

Yt = β1 + Y t−1 + et

Deterministic trend
Yt = β1 + β2t + et

Random walk with drift and deterministic trend

Yt = β1 + β2t + Y t−1 + et

Stationary AR(1) and deterministic trend (|β3| < 1)

Yt = β1 + β2t + β3Yt−1 + et

Farooq Shah Time Series (1 of 2) 08-Jun-20 35/ 46

Examples of models with trends

Suppose et is from a white noise process with mean zero and variance σ2
and let
Yt = β1 + β2t + β3Yt−1 + et
We have the following cases:
Random walk without drift, if β1 = 0, β2 = 0 and β3 = 1.
Random walk with drift, if β1 ≠ 0, β2 = 0 and β3 = 1.
Deterministic trend, if β1 ≠ 0, β2 ≠ 0 and β3 = 0.
Random walk with drift and deterministic trend, if β1 ≠ 0, β2 ≠ 0 and
β3 = 1.
Stationary AR(1) and deterministic trend, if β1 ≠ 0, β2 ≠ 0 and
β3 < 1.

Farooq Shah Time Series (1 of 2) 08-Jun-20 36/ 46

Integrated stochastic process

If a non-stationary stochastic process has to be differenced at least d times

to become stationary, it is called an integrated process (times series) of
order d, denoted as Yt ∼ I(d).
What does this mean?
Recall: The differenced time series ∆Y t is a time series which can be
stationary or not, just like any other time series.
It means, Yt ∼ I(d) iff ∆ d Y t is stationary but ∆ d−1 Y t is
nonstationary.
For example a random walk is I(1) and if Yt itself is stationary then it is
called integrated of order zero, Yt ∼ I(0).

Farooq Shah Time Series (1 of 2) 08-Jun-20 37/ 46

Integrated stochastic process - Visual example

Farooq Shah Time Series (1 of 2) 08-Jun-20 38/ 46

Difference-Stationary Process

If a time series has a single unit root the first difference of it results in
a stationary time series (which means it is I(1)).
Generally, if a time series has a unit root of d multiplicity, its
difference of order d is a stationary process (which means it is I(d)).
Time series that after differencing become stationary (integrated time
series) are called difference-stationary time series.
Transforming non-stationary time series is important for forecasting
and for the removal of spurious regression phenomenon.

Farooq Shah Time Series (1 of 2) 08-Jun-20 39/ 46

Difference-Stationary Process - example
A typical example of difference-stationary time series is a random walk
process with or without drift (or AR(1) with unit root).

∆Y t = Yt − Yt−1 = δ + et

Farooq Shah Time Series (1 of 2) 08-Jun-20 40/ 46

Trend-Stationary Process

Assume that a time series has a deterministic trend. If the time series
is stationary after we detrend it then the time series is said to be
“stationary around the trend line (curve)” or simply stated as
“trend-stationary”.
For example, the OLS residuals of Yt = β0 + β1t +et are stationary
and detrended. The series {Y t } itself is trend-stationary.
Note that the trend can be nonlinear too.

Farooq Shah Time Series (1 of 2) 08-Jun-20 41/ 46

Trend-Stationary Process - example

A typical example of trend-stationary time series is a deterministic trend

process
Yt = α + βt + et

Farooq Shah Time Series (1 of 2) 08-Jun-20 42/ 46

Trend-Stationary Process - example

It might be the case that the difference of a time series is trend-stationary.

In such cases, after differencing we detrend the time series.

If we after differenceing have a model similar to this we should also

detrend the time series.

∆Y t = β1 + β2t + ut

Farooq Shah Time Series (1 of 2) 08-Jun-20 43/ 46

Trend-Stationary Process - example

Note: In this course we will not consider this complicated functions of time
(e.g. sin and cos). This is just an example I found that illustrates the point
well, that is, first differencing and then detredning a stime eries.
Farooq Shah Time Series (1 of 2) 08-Jun-20 44/ 46
Difference-Stationary Process vs Trend-Stationary Process

Treating a trend-stationary time series as difference-stationary, or vice

versa, results in specification error and can be serious leading to bias.

Farooq Shah Time Series (1 of 2) 08-Jun-20 45/ 46

Assess Stationarity

To this point we have seen how stationarity/nonstationarity can be

decided by purely looking at the time series. We look for trends, varying
amplitude of volatility and so on.

Stationarity can also be evaluated using

Correlograms
Hypothesis testing (more on this in the next lecture)

Farooq Shah Time Series (1 of 2) 08-Jun-20 46/ 46