Statistics 5350/7110

Forecasting

Lecture 22
Periodicity in Time Series

Professor Robert Stine

 Preliminaries
• Questions?

• Assignments
• Assignment 5 due on Thursd y

• Quick review
• SARIMA models:
Multiplic tive ARIMA models for se son l d t
• Combining regression with SARIMA
C lend r ex mples

2
a
a
a
a
a
a
a
a
a
 Today’s Topics
Text, §6.1-6.2

• Se son l models with deterministic periodicity

• We s w this in the housing ex mple

• Fitting periodic functions

• Forming predictors from sines nd cosines
• Regression estim tes

• Periodogr m
• E icient comput tion of regression with n predictors
• Fourier frequencies: Uncorrel ted expl n tory v ri bles
• F st Fourier tr nsform

3
a
ff
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Differencing and
Deterministic Trends
 Differencing a Linear Trend
• Model
• Suppose process is line r trend plus white noise
Xt = α + β t + wt
• You di erence it, s if r ndom w lk
(1 − B) Xt = β + (1 − B) wt

• ACF nd PACF of di erences

• ACF ppe rs to “cut o ”
• PACF drops o gr du lly
• Suggests n IMA(1,1) model

5
a
ff
a
a
a
ff
a
a
a
a
a
ff
a
a
ff
a
 Differencing a Linear Trend
• Model
• Line r trend plus white noise
Xt = α + β t + wt
• Di erencing yields
(1 − B) Xt = β + (1 − B) wt

• Estim tion results

• IMA(1,1) model h s θ1̂ = − 1
• Fine di gnostics, though not invertible
• Indic tes Xt is non-stoch stic trend plus white noise
—> Fit regression!

6
ff
a
a
a
a
a
a
a
 Differencing a Seasonal Trend
• Model
• Process is se son l trend plus AR(1) noise
Xt = α + St + Zt, (1 − ϕB)Zt = wt
where St is deterministic se son l p ttern
St − St−12 = 0
• You se son lly di erence it, s if stoch stic trend
(1 − B 12) (1 − ϕB) Xt = (1 − B 12) wt

• ACF nd PACF of process

• ACF drops o s in AR(1) or AR(2), se son l?
• PACF h s spike then drops o : cut o or dec y?
• Try to it n ARMA(1,1) se son lly

7
f
a
a
a
a
a
a
a
ff
a
a
ff
a
a
a
ff
a
a
a
ff
a
a
a
a
a
 Differencing a Seasonal Trend
• Model
• Se son l trend plus AR(1) noise
Xt = α + St + zt, (1 − ϕB)zt = wt
with St − St−12 = 0
• Se son lly di erenced
(1 − B 12) (1 − ϕB) Xt = (1 − B 12) wt

• Estim tion results

• Fit initi l model s SARIMA(1,0,0)(1,1,1)12
• Model h s Θ̂ 1 =−1
• Fine di gnostics, though not invertible

8
a
a
a
a
a
a
a
a
ff
a
 Back to the Housing Example
• Monthly housing st rts
• 10 ye rs, 2010-2019
• log sc le
• Trend ppe rs stoch stic, so di erence
• Stoch stic se son lity too?

• Identi ic tion from ACF/PACF

• ACF h s persistent se son l correl tion
-> se son l di erence
• PACF drops o
-> MA sort of dependence?
• St rt with “f mili r” model SARIMA(0,1,1)(0,1,1)12

9
a
a
a
a
a
a
f
a
a
a
a
ff
a
a
a
a
a
ff
a
a
a
ff
a
 Back to the Housing Example
• Estim tion results
• Consistent with deterministic se son l trend
• Residu l di gnostics look ok y

• Wh t to do?
• Softw re is ine with the itted model
(though model is not invertible)
-> Le ve it lone (forec sts work ine)
• Revise model to h ve deterministic se son l
trend (periodic) it with regression.

10
a
a
a
a
a
f
a
f
a
a
a
f
a
a
a
a
a
f
a
 Deterministic Seasonal Trend
• Regression fe tures
12
• Use monthly dummy v ri bles to cre te se son l trend for which (1 − B ) St =0
• Ret in di erencing for over ll trend.
• Over ll it nd forec sts comp r ble to SARIMA model

Close to MA1
estim te in
SARIMA model

11
a
a
a
f
ff
a
a
a
a
a
a
a
a
a
a
a
 Deterministic Seasonal Trend
• Di erent ppro ch to regression fe tures
12
• Cre te fe tures from sines nd cosines, so th t (1 − B ) St = 0
Cosine/Sine with period 12, period 6, period 4, period 3, period 2.4, nd period 2
• Use di erencing for over ll trend. Fit nd forec sts m tch the prior regression

period 12

period 6

period 4

period 3

period 2.4

period 2

12
ff
a
ff
a
a
a
a
a
a
a
a
a
Periodic Predictors
using
Sines and Cosines
 Periodic Behavior
• Widespre d phenomen re periodic
• Astronomic l d t
v ri ble st r m gnitude, qu s rs
• Clim te
nnu l temper ture, ozone levels
• Medic l d t
pulse, EKG
• Business cycles re somewh t periodic

• Questions
• Does time series show periodic v ri tion?
• At wh t frequencies?
• Should white noise h ve periodic fe tures?

14
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Sinusoidal Functions
• Sinusoid model
• De ine
Text writes ω r ther th n
Xt = R cos(2π ν t + φ) + wt ν for the frequency. I just
c nnot do th t!
• ν is the frequency (reciproc l of the period)
• R is the mplitude of the cosine w ve.
• φ is the ph se of the cosine
(does not lw ys st rt t 1 when t = 0)

• Le st squ res estim tes

• Estim te ν, R, nd φ with regression
• Key from trigonometry
cos(a + b) = cos(a)cos(b) − sin(a)sin(b)
• Hence
Xt = A cos(2πνt) + B sin(2πνt) + wt
where
A = R cos(φ), B = − R sin(φ) 15
a
f
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Random Phase Model Not covered in lecture

• Sinusoid s st tion ry process

• Tre t ph se s r ndom v v ri ble
Xt = (R cos φ) cos(2πν t) + (R sin φ) sin(2πν t)
A B
• Suppose ph se is uniformly distributed on -π to π. Then the me ns re zero:
E A = R E(cos φ) = 0, E B = R E(sin φ) = 0
• The v ri nce doesn’t depend on ν,
Var(A) = E A 2 = R 2 E(cos2 φ) = R 2 π = σ 2, Var(B) = σ 2
• And the cosine nd sine terms re uncorrel ted See Appendix C.5

Cov(A, B) = R 2 E (cos(φ) sin(φ)) = R 2 /2 sin(2φ) = 0

• Cov ri nces
• Cov ri nce depends only on the time di erence (2 π ν = ω)
Cov(Xt, Xs) = E [(A cos(ω t) + B sin(ω t)) (A cos(ω s) + B sin(ω s))] Equ tion 6.3

= σ 2 (cos(ω t)cos(ω s) + sin(ω t)sin(ω s)) = σ 2 cos(ω(t − s))

16
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
ff
a
 What’s the frequency?
• Frequency doesn’t come from regression
• Simple estim tor
• Divide the number of pe ks by the length of the time series
ν̂ = 3/36 = 1/12

• Prominent frequencies re e sy to estim te

• St nd rd error on the order of n (r ther th n √n)

• Ex mple: M gnitude of v ri ble st r

• D ily d t from Whitt ker nd Robinson (1923)
• M in frequency is 21/600 ≈ 0.035, bout 28.6 d y period
• Cle rly not single sinusoid:
Wh t other frequencies re present?
17
time series included in sts
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Estimating a Periodic Function
• Le st squ res estim tes
• Two coe icients for e ch frequency
Xt = R cos(2πνt + φ) = A cos(2πνt) + B sin(2πνt)
where
A = R cos(φ), B = − R sin(φ)
• Converting from A, B b ck to R nd φ is little h rder
- Since cos2 + sin2 = 1, R2 = A2 + B2 Trig ppendix
C.5, p 234
–t n(2 π φ) = - B/A, solved in R s φ = t n(-B/A)

• V ri ble st r ex mple
• D t isn’t single sinusoid
• Residu ls from regression of st r m gnitudes on
cos/sine with frequency 21/600 is nother sine w ve.
• Counting pe ks suggests second frequency 25/600
18
a
a
a
a
a
a
a
ff
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Estimating a Periodic Function
• Le st squ res estim tes
• Sum of sinusoids

∑
Xt = Aj cos(2π νj t) + Bj sin(2π νj t)
j
• M gnitude of e ch given by Rj2 = Aj2 + Bj2.

• V ri ble st r ex mple
• Try with frequencies set to ν1=21/600, ν2=25/600
• Still not quite right: sinusoid rem ins

• Integer periods?
• Periods of these re 600/21≈28.6 nd 600/25=24.
• Round these to integer periods

See Bloom ield, Ch pter 3 for further discussion of inding these frequencies 19
a
a
a
a
f
a
a
a
a
a
a
a
a
a
f
 Estimating a Periodic Function
• Le st squ res estim tes
• Sum of sinusoids

∑
Xt = Aj cos(2πνj t) + Bj sin(2πνj t)
j <latexit sha1_base64="3PoouSwEOXbl2t/f9uEuUeTu4KY=">AAACk3icdZHdatswFMdldx9d9pV17Go3YmFlUFBkL3VixqDLdrGbQTeatlAnQZbVVMSWhKRsDcZvtYfYK+xtpsQecws7IDj//znnh46Uqpwbi/Fvz9+5c/fe/d0HnYePHj952n22d2rkSlM2oTKX+jwlhuVcsInlNmfnSjNSpDk7S5cfN/Wz70wbLsWJXSs2LchC8EtOiXXWvPsrSdmCi5JoTdZVqWledSBMrogtv1XzYBbC/ff7tf5Q64NajWuVEKW0vIYRwsNZeDBEcdxyA4xR5HBUt6DhLWh4Axq2x9HorYNGaDRoufAQo7CGJkxkzdXn3R5GMQ7iwyGsk9GgSaLYkfA2eqCJ43n3Z5JJuiqYsDQnxlwEWNmpo1lOc1Z1kpVhitAlWbALlwpSMDMtty9ewdfOyeCl1O4IC7due6IkhTHrInWdBbFX5nZtY/6rcaFWtuxPjPuofirT/g+pl31F1EYbS0RGdIYsu646bs2/u8D/J6chCiIUfR30jsbNwrvgJXgF3oAADMER+AyOwQRQD3kn3tSb+S/8d/7Y/1S3+l4z8xzcCP/LH9c3vks=</latexit>

R̂12 = Â21 + B̂12 ⇡ 6.072 + 7.992 ⇡ 100.6

• M gnitude of e ch given by Rj2 = Aj2 + Bj2. R̂22 = Â22 + B̂22 ⇡ 1.832 + 6.842 ⇡ 50.2
• V ri ble st r ex mple
• Try with frequencies set to ν1=1/29 nd ν2=1/24. Wow!

20
a
a
a
a
a
a
a
a
a
a
Finding Periodic Components

The periodogr m
a
 Hidden Periodicity
• Does time series cont in periodic components?
• E sy to identify periodic functions in c ses like the v ri ble st r
• n tur l in context
• obvious in d t
• Wh t bout periodicities in other time series?
• st tion ry process
• se son l d t
• residu ls from models th t supposedly c pture se son lity

• Di gnostic
• L st pplic tion is n import nt di gnostic
• Do “dese son lized” d t h ve periodic components?
• Do residu ls from n SARIMA model h ve periodic components?

• Bro d ppro ch
• Compute the mount of energy ( mplitude) of ll the possible sinusoids … the periodogr m

22
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Periodogram
• Why estim te just couple of frequencies
• If n is n even number, we c n write with no error
n/2
∑
ak = (2/n) xt cos(2πtk/n)
∑
Xt = aj cos(2π t j/n) + bj sin(2π t j/n) Equ tion 6.8
∑
bk = (2/n) xt sin(2πtk/n)
j=0
• End c ses re speci l since sin(0) = sin(k π) = 0 so b0 = bn/2 = 0 (cos(0) = 1, cos(k π)=±1)
• Sc led periodogr m
P(k/n) = ak2 + bk2 = Rk2 De inition 6.3

• Fourier frequencies
• Sine/cosines with frequencies νj = j/n, for j = 0, 1, …, n/2
Supplement l
• Highest frequency is νn/2 = 1/2, .k. . the folding frequency or Nyquist frequency slides discuss
li sing
• Asides
• Line r tr nsform tion ( rot tion) of time series (X1, …, Xn) into coe icients ( 0, 1,b1, …, n/2)

• If n is highly composite (e.g. n = 2m), then c n compute in O(n log n) using FFT

23
a
a
f
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Properties of Periodogram
• Suppose time series is white noise
• Coe icients

∑ ∑
ak = (2/n) wt cos(2πtk/n) nd bk = (2/n) wt sin(2πtk/n)
• Expected v lues re e sy
E ak = E bk = 0
Appendix
• V ri nces nd cov ri nces re lso e sy – if you remember b sic trigonometry Property C.3

4 σw2 n 2
2 σw2 p ge 234

n2 ∑
Var(ak) = Var(bk) = sin (2πkt/n) = k = 1, 2, …, n/2-1
t=1
n
n/2
Cov(ak, aj) = 0 ( j ≠ k) and Cov(ak, bj) =0
• Periodogr m
2 2 2
n 2 N(0,1) + N(0,1) χ
I(k/n) = (ak + bk2) ≈ σw2 2 Proportion l to n
• De ine s = σw2 exponenti l r ndom
4 2 2 v ri ble

24
a
a
a
a
f
a
ff
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Properties of Periodogram
• Periodogr m of white noise
• Proportion l to exponenti l r.v.
n 2 χ2
I(k/n) = (ak + bk ) ∼ σw
2 2 2
4 2
• Approxim tely independent v lues s mpled from skewed distribution: Very noisy ppe r nce

Exponenti l

• Ex mple
Distribution

• n = 256
2
• White noise, σw =5

25
a
a
a
a
a
a
a
a
 Examples of Periodogram
White Noise (n=128) V ri ble St r (n=600)

(1-B) log Housing (n=120) SARIMA residu ls

26
a
a
a
a
 What’s next?
• S mpling properties of the periodogr m
• Are pe ks in white noise import nt or just s mpling v ri tion?
• Key di gnostic:
- For residu ls from model, wh t should the periodogr m show?
- For dese son lized d t from FRED, wh t should the periodogr m show?

• Spectr l density nd cov ri nces

• Periodic represent tion of ny st tion ry processes
• Smoothing the periodogr m
• Connecting time nd frequency-dom in n lyses

27
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
Supplements

Not covered, but might be useful in your ield

f
 Not covered: More Examples
• fMRI
• Sign l indic ted by squ re w ves (Ex mple 1.6) with period 64 seconds
• C ution: D t s mpled 1 point every two seconds. M x observ ble frequency is 1/2δ = 1/4 seconds
• Expect sign l response t period 32 me surements
• Sign l is evident in 4 output series, but re these signi ic nt?

29
BOLD = blood oxygen level dependent Periodogr ms shown on zoomed sc le
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Aliasing and Nyquist Frequency
• Beh vior under s mpling
• We observe discrete time series, equ lly sp ced observ tions of continuous process
• Highest frequency we c n estim te is 1/2 since we need t le st 2 observ tions within cycle
≈ Figure 6.1
• 1/2 is known s the Nyquist or folding frequency
• Sinusoids with frequencies higher th n 1/2 ppe r
t lower frequency

• Ex mple
• Continuous process h s period 4 hours, or frequency 1/4
• Process is s mpled every 2.5 hours
• Discrete time series h s cycle t lower frequency 1/20

• M th
• S mple continuous process t times t = 0, δ, 2δ, 3δ… f 0 ))
<latexit sha1_base64="evOMpHJlmreKf4waojxhjUT0uco=">AAADIXicpVJPb9MwFHfCvxH+dXDk8kTFlogtTarR0QPSBBeOQ6LbpLmqHMfprDpOZLtAFeXT8BU48gW4IW6IG1f4EDhtJgoa4sCTorz3+yc/J0kpuDZR9MVxL12+cvXaxnXvxs1bt+90Nu8e6WKuKBvRQhTqJCGaCS7ZyHAj2EmpGMkTwY6T2fOGP37NlOaFfGUWJRvnZCp5xikxFppsOiOcsCmXFVGKLOpKUVGDB4Bpof0+4JID3jF4B7IAtp5uLXHAgmXG7zeksTT4cQ+nTBgCu5BtB4AVn54Z+6aqybJ1kdWfwcoV/FeG1S8P8Q/TSj8L1qfG19geAdZcQrtvo1kbzXk2VRdGttezHXiYybS9x0mnG4XDKB4+3odV82SvbQZDiMNoWV3U1uGk8x6nBZ3nTBoqiNancVSasU0znApWe3iuWUnojEzZqW0lyZkeV8vvX8NDi6SQFco+0sASXXdUJNd6kSdWmRNzpv/kGvAXx2U5N1VvpO1v00uKpPemULNeScpm1obIlKg0NOxt7dk1z3eBvzdH/TAehIOXe92DZ+3CG+g+eoB8FKN9dIBeoEM0QtT54Hxzvjs/3HfuR/eT+3kldZ3Wcw/9Vu7XnyI15Nw=</latexit>

cos(2⇡ t f ) = cos (2⇡t (1/

= cos (2⇡(k ) (1/ f 0 ))
• Continuous process h s frequency 1/2δ < f < 1/δ = 0
cos (2⇡(k t f ))
• De ine frequency f = 1/δ – f’ where 0 < f’ < 1/2δ = cos(2⇡k) cos(2⇡t f 0 ) + sin(2⇡k) sin(2⇡t f 0 )
= cos(2⇡ t f 0 )
• Oscill tion t f li sed to oscill tion t f’ 30
a
a
a
a
f
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
 Aliasing Example
• V ri ble st r
• Brightness of v ri ble st r is continuous process
• M gnitude h s period of 0.8 d ys, frequency f = 1.25 cycles per d y

• Discrete time series

• You observe the m gnitude e ch night t 12 m
• S mpling interv l δ = 1
• Nyquist (folding) frequency is 1/2

• Ali s
• Ali sing will h ppen since 1/2 < f
• Resulting discrete time series h s sinusoid t frequency 0.25
<latexit sha1_base64="HCG2/S1c4g5IeZmBO2k4kv2O63A=">AAACmXicdVFbb9MwFHYyLiNcVsbjXo6omFoxcqm6jj4gDXiZeBqCbhNzVTmO21lNbMt2gSrKv+J38M6/wckyoAiOZOWc813kz0lVzo2N4x+ev3Xr9p272/eC+w8ePtrpPN49M3KlKZtQmUt9kRLDci7YxHKbswulGSnSnJ2ny7c1fv6ZacOl+GjXik0LshB8zimxbjXrfMcpW3BREq3Juio1zSsIADCVpjcArDjgA4sPIAkHh33Yf7XfQIBzNrctwUIvgeeQRMM+YM0XV9Z9qa5tXP2S9AYNub8x1ZoXgA0XcOPW3xwbV6r/aeau5uAAM5G1AWadbhyO42R8eATXzcth24zGLkTcVBe1dTrrfMOZpKuCCUtzYsxlEis7dW6W05xVAV4ZpghdkgW7dK0gBTPTsnn3Cp65TQZzqd0RFprtn4qSFMasi9QxC2KvzN9YvfyNcaFWtowmxv2uKJVp9EXqZaSIqmdjiciIzkLLvlaBi3mTBf7fnA3CZBSO3g+7x2/awNtoDz1FPZSgI3SMTtApmiDqDb1PHvUyf89/7Z/4766pvtdqnqCN8j/8BL7sukg=</latexit>

cos(2⇡ t 1.25) = cos (2⇡t(1 + 1/4))

= cos(2⇡t) cos(2⇡t/4) sin(2⇡t) sin(2⇡t/4)
= cos(2⇡ t/4)
• Remedy
• S mple t higher r te, or pply low-p ss ilter prior to s mpling
31
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
a
f