Instituto Superior Tecnico

Introduction to Stochastic Processes

(p) (1 p)
k
p
1p
p
1p
p
2
p
1(1p)s
Negative Binomial (r, p)
_
k1
r1
_
p
r
(1 p)
kr r
p
r(1p)
p
2
_
ps
1(1p)s
_
r
Negative Binomial

(r, p)
_
k+r1
r1
_
p
r
(1 p)
k
r(1p)
p
r(1p)
p
2
_
p
1(1p)s
_
r
Poisson () e

k
k!
e
(1s)
X
(s)
(sampling r.v.)
k p
k
E[X]
E[X
2
]
E[X]
E[X
3
]E[X]E
2
[X
2
]
E
2
[X]

Continuous Univariate Distributions
X f
X
(x) E[X] Var[X] M
X
(t) = E[e
tX
]
Uniform(a, b)
1
ba
a+b
2
(ba)
2
12
e
bt
e
at
(ba)t
Exponential () e
x 1

t
Erlang (n, ) e
x
(x)
n1
(n1)!
n

2
_

t
_
n
Gamma (, ) e
x
(x)
1
()

2
_

t
_

Beta (p, q)
(p) (q)
(p+q)
x
p1
(1 x)
q1 p
p+q
pq
(p+q)
2
(p+q+1)

Normal (,
2
)
1

2
e

(x)
2
2
2

2
e
t+
(t)
2
2

2
k
1
2
k/2
(k/2))
x
k
2
1
e

x
2
k 2k (1 2t)

k
2
Rayleigh ()
x

2
e

x
2
2
2

2
4
2

2

X
(s)
(sampling r.v.)
xf
X
(x)
E[X]
E[X
2
]
E[X]
E[X
3
]E[X]E
2
[X
2
]
E
2
[X]

X = (X
1
, X
2
, . . . , X
k
) Multinomial (n, p
1
, p
2
, . . . , p
k
)
P(X = (n
1
, n
2
, . . . , n
k
)) E[X
j
] Var[X
j
] Cov(X
j
, X
l
) G
X
(s) = E[

k
j=1
s
X
j
j
]
n!
n
1
!n
2
!...n
k
!
p
n
1
1
p
n
2
2
. . . p
n
k
k
np
j
np
j
(1 p
j
) np
j
p
l
(p
1
s
1
+p
2
s
2
+. . . +p
k
s
k
)
n
X = (X
1
, X
2
, . . . , X
k
) Normal (, )
f
X
(x
1
, x
2
, . . . , x
k
) E[X
j
] Var[X
j
] Cov(X
j
, X
l
) M
X
(t) = E[

k
j=1
e
t
j
X
j
]
1
(2)
k/2
||
1/2
exp{
(x)

1
(x)
2
}
j

2
j
=
jj

jl
e
t

+
1
2
t

t
_

_
=
( + 1)
( + 1)(( ) + 1)
, () =
_

0
e
x
x
1
dx, (n + 1) = n!, n IN
0
,
_
1
2
_
=

.
1
Relations for expected values of nonnegative random variables
E[X] =

k=0
kP(X = k) =

k=0
P(X > k)
E[X
2
] =

k=0
k
2
P(X = k) =

k=0
(2k + 1)P(X > k)
E[X] =
_

0
xf
X
(x) dx =
_

0
P(X > x) dx
E[X
n
] =
_

0
x
n
f
X
(x) dx = n
_

0
x
n1
P(X > x) dx
Computation of expected values and variances by conditioning
E[X] = E[E[X|Y ]], Var[X] = E[Var[X|Y ]] + Var[E[X|Y ]]
Probability Generating Functions
G
X
(s) = E[s
X
] =

k=0
s
k
P(X = k), |s| 1
P(X = k) =
G
(k)
X
(0)
k!
E[X(X 1) (X k + 1)] = G
(k)
X
(1

)
E[X] = G

X
(1

) Var[X] = G

X
(1

) +G

X
(1

) [G

X
(1

)]
2
Galton-Watson Branching Processes
X
n+1
=

X
n
k=1
Z
n,k
{Z
n,k
}
iid
Z
= E[Z],
n
= E[X
n
]
n
=
0

2
= Var[Z],
2
n
= Var[X
n
]
2
n
=
2n

2
0
+
n1

n
1
1

2
, = 1
G(s) = E[s
Z
], G
n
(s) = E[s
X
n
] G
n
(s) =
X
0
=1
G
n1
(G(s)) = G(G
n1
(s))
= P(extinction)
n
= P(X
n
= 0) = G
n
(0)
Important Laplace Transforms
f(t) TL(f) = f

(s) =
_

0
e
st
f(t) dt g(t) g

(s) =
_

0
e
st
g(t) dt
1 1/s af(t) +b h(t) af

(s) +b h

(s)
t
n
n!/s
n+1
df(t)
dt
sf

(s) f(0)
t
n1
e
at
(n1)!
1/(s +a)
n
e
at
f(t) f

(s +a)
sin(at) a/(s
2
+a
2
)
_
t
0
f(u)du f

(s)/s
e
at
e
bt
ba
1/[(s +a)(s +b)] t
n
f(t) (1)
n d
n
ds
n
f

(s)
2
Bernoulli process: {X
n
}

n=1
iid
Bernoulli (p), S
n
=

n
i=1
X
i
, T
k
= inf{n 0 : S
n
= k}
(S
m
|S
n
= k) Hypergeometric(n, m, k), 0 m n, 0 k n.
{T
k
T
k1
}

k=1
iid
Geometric(p).
(Simple) Random Walk: {Y
n
}

n=1
iid
Y , with P(Y = 1) = p = 1 P(Y = 1), q = 1 p
Z
n
=

n
i=1
Y
i
, M
n
= max
0kn
Z
k
,
k
= inf{n > 0 : Z
n
= k}, = p/q
P(
0
= 2n) =
(
2n
n
)
2n1
p
n
q
n
.
P(
k
= n) =
|k|
n
_
n
n+
k
2
_
p
n+
k
2
q
n
k
2
, |k| IN, n k even.
P(
0
< ) = 2 min(p, q).
P(
k
< ) = [min(1, )]
k
, k IN.
p =
1
2
=[P(
0
> 2n) = P(Z
2n
= 0) P(M
n
= k) = P(Z
n
= k) +P(Z
n
= k + 1)].
P(
A
<
B
) =
_

A+B

A+B
1
= 1
B
A+B
= 1
, A, B IN.
Standard Brownian Motion X, with T
x
= inf{t 0 : X(t) = x}, M
t
= max
0st
X(s)
P(T
x
t) = 2P(X(t) |x|).
P(X(t) = 0, t (t
1
, t
2
)) =
2

arcsin
_
t
1
t
2
.
P(M
t
x) = P(T
x
t) = 2P(X(t) x).
M
t
d
= |X(t)|, E[|X(t)|] =
_
2t

, Var(|X(t)|) =
_
1
2

_
t.
Brownian Bridge {Z(u), 0 u 1}: Cov(Z(s), Z(t)) = s(1 t), 0 s t 1.
{W(u) =
_
u
0
X(v)dv, u 0}: Cov(W(s), W(t)) = s
2
_
t
2

s
6
_
, 0 s t.
Brownian Motion with drift , with T
x
= inf{t 0 : X(t) = x}
P(T
A
< T
B
) =
e
2B
1
e
2B
e
2A
, A, B > 0.
M = sup
0t<
X(t) Exponential
_

2
_
, < 0.
Black-Scholes Formula: c = y
0
(

t +b) Ke
t
(b), with
b =
t
2
t/2 ln(K/y
0
)

t
and = +
2
/2.
3
Non-homogeneous Poisson Process
N = {N(t), t 0} NHPP({(t), t 0}); S
n
= inf{t 0 : N(t) n}, n IN
0
.
[(S
1
, S
2
, . . . , S
n
)|N(t) = n]
d
= (Y
1:n
, Y
2:n
, . . . , Y
n:n
) with F
Y
(y) =
(y)
(t)
, 0 y t.
For independent Bernoulli recording mechanism with probability p(s, t) of recording until
time t an individual event taking place at time s in the original process N, with N
r
(t)
denoting the number of events recorded until time t:
N
r
= {N
r
(t), t 0} NHPP
__

r
(t) =
_
t
0
p(s, t) d(s), t 0
__
.
Discrete Time Markov Chains
= P E
_
kT
c(X
k
)

= a

kT
P
k
c

j
=

iS

i
p
ij
E
_
kT

k
c(X
k
)

= a

kT

k
P
k
c
= 1
t
(I P +ONE)
1
E
_
kT
c(X
k1
, X
k
)

= a

kT
P
k1
[P C] 1
[P(N
iA
n)]
i

A
=

n
k=1
P
k1

A

A
(I P
A

A
)1 E
_
kT

k
c(X
k1
, X
k
)

= a

kT
(P)
k1
[(P) C] 1
[ E[N
iA
] ]
i

A
= [I P
A

A
]
1
1 c + [P C] 1

i
p
ij
=
j
p
ji
Continuous Time Markov Chains
Q = R diag(r)

P = I +
Q
q
pQ = 0
t
p

P = p
P

(t) = QP(t) = P(t) Q {N

n=0
e
qt
(qt)
n
n!

P
n

i=j
p
i
r
ij
= p
j
r
j
[P(T
iA
t)]
i

A
=

n=0
P(N
q
(t) > n)

P
n

A

A
(I

P
A

A
)1
p
j
=

j
r
j
_

iS

i
r
i
[E[T
iA
]]
i

A
=
1
q
_
E[

N
iA
]
_
i

A
=
1
q
[I

P
A

A
]
1
1
p
i
r
ij
= p
j
r
ji
E[C(t)] =

n=0
P(N
q
(t) > n)
_
1
q
a

P
n
c +a

P
n
(

P C)1
_
= a

n=0
P(N
q
(t) > n)

P
n
_
c/q + (

P C)1
_
pc +p[R C]1
Renewal Processes with RS {S
n
, n IN}, dist.f. IRT F with mean .
RTE: H(t) = D(t) +
_
t
0
H(t x)dF(x)
|D(t)| < H(t) = D(t) +
_
t
0
D(t x)dM(x).
KRT: F aperiodic, D: dierence of two monotonic functions, and
_

0
|D(t)| dt < :
lim
t+
H(t) =
_

0
D(t) dt
_
.
F
e
(x) =
_
x
0

F(u) du/, x 0.
R(t) =

N(t)
n=1
R
n
and {(S
n
S
n1
, R
n
), n IN}
iid
(X, R):
lim
t+
R(t)/t = lim
t+
E[R(t)]/t = E[R]/.
4
Birth-death Processes
p
0
=
1
1+

+
l=1

l
k=1
(
k1
/
k
)
p
j
= p
0

j
k=1
(
k1
/
k
)
Queues
L
s
E[L
s
] =
e
E[W
s
]
E[L
s
] = s
e
+ E[L
q
]
L
q
E[L
q
] =
e
E[W
q
]
W
s
E[W
s
] =
1

+ E[W
q
]
M/M/1
L
s
L
s
Geometric

(1 )
W
s
W
s
Exp((1 ))
W
q
E[W
q
] =

(1)
Var[W
q
] =
(2)
[(1)]
2
M/M/2
W
q
E[W
q
] =

2
(1
2
)
Var[W
q
] =
_

(1
2
)
_
2
[1 +(1 )]
M/M/s
L
s
p
j
= P (L
s
= j) =
_
s!
j!
(1 )(s)
js
C(s, s), j = 0, 1, . . . , s 1
(1 )
js
C(s, s), j = s, s + 1, . . .
C(s, s) = P (L
s
s) =
(s)
s
s!(1)

s1
k=0
(s)
k
k!
+
(s)
s
s!(1)
C(1, ) =
C(2, 2) =
2
2
1+
E[L
s
] = s +

1
C(s, s)
L
q
E[L
q
] =

1
C(s, s)
Var[L
q
] =

(1)
2
C(s, s) (1 +[1 C(s, s)])
W
q
P(W
q
> t) = C(s, s)e
s(1)t
E[W
q
] =
C(s,s)
s(1)
Var[W
q
] =
C(s,s)[2C(s,s)]
[s(1)]
2
5
M/M/s/s+c
L
s
p
j
= P (L
s
= j) =
_

_
(s)
j
j!

s1
k=0
(s)
k
k!
+
(s)
s
s!

c
i=0

i
, j = 0, 1, . . . , s 1
(s)
s
s!

js

s1
k=0
(s)
k
k!
+
(s)
s
s!

c
i=0

i
, j = s, s + 1, . . . , s +c

n
k=j

k
=
_

j
1
, n = , < 1

n+1
1
, n < , = 1
n + 1 j, n < , = 1
C(s, s, c) = P (L
s
s) =
(s)
s
s!

c

c
k=0

k

s1
j=0
(s)
j
j!
+
(s)
s
s!

c
k=0

k
C(1, , c) =
_

c+2
1
c+2
, = 1
c+1
c+2
, = 1
L
q
L
q
d
= [1 C(s, s, c)] O C(s, s, c) N

c
P(O = 0) = 1
E
i
Exp(s)
P(N

n
= k) =

k

n
j=0

j
, k = 0, 1, . . . , n E[N

] =

1
Var[N

] =

(1)
2
E[L
q
] =
_
_
_
_

1
1(c+1)
c
+c
c+1
1
c+1
_
C(s, s, c), = 1
c
2
C(s, s, c), = 1
W
q
W
q
d
= [1 C(s, s, c 1)] O C(s, s, c 1)

1+N

c1
i=1
E
i
E[W
q
] =
_
_
_
_
1
1
1(c+1)
c
+c
c+1
1
c
_
C(s,s,c1)
s
, = 1
c+1
2s
C(s, s, c 1), = 1
M/G/
L
s
(t) L
s
(t) Poisson
_

_
t
0
[1 G(s)] ds
_
GI/M/1
L

s
L

s
Geometric

(1 ) with =

F((1 ))
W

s
W

s
Exp((1 ))
6