optimal compression :

o
- 10
110
*
11)
p [h(Xn =
d) =
p[45(Xn =
1)) =
=
Convergence :

point wisely convergence

W
convergence with probability 1 (SLLM)

Il
~
convergence

E
in mean

convergence in probability (WL(N)

↓
converge in distribution
pointwisely convergence :
R

xw Xim

ten
=
,

convergence with prod .

1 :

convergence in mean;

PrSbXn( =
X1 = 1 ,
"wer
ESIXn-x Y
*
= o

(limit > probl -

convergence in prob,i

hig PrENn-X/123 = 1 (prob- limit

convergence in distribution :

Fx(x) =
Ex(x)
ergodic :

⑨
-
> Law of Large number
Ergodic decomposition theorem :

Any Stationary process can be split into

multiple

stationary ergodic processes ,

·
fund inequ

i
log-sum inequality :

=(9 log(1)
: =
Balog/ j

pf
: (9 : log (1) -

15 Silog
= 59 : [10g-los]
= 9) los ]
_
9 : /1
fund -S
= zaj
-
9 =
; 0
Convex:

b3)) af(x) bf(y)

f(ax
+
+

Jensen's inequality
f : O >
-
&" is convex ,
O is a convex set

S
-
Convex set

Elf(x))[ f(E(X)

not

⑳
convex set
find min If(x)) given constraints
: 9 , (x)30
Xt *
hi(x) = 0

0
Let a =

\x( *, Si(10 ,
hi(x1 =

Lagrange mul :

L(x ,
r1 =
min(f(x) + [x : 9 : (x) + [Vihi(1)
XX

[xi9 : (x) [Vh

min(f(x) + +

Q &*

Min[f(x) kky
it is proved that ((X 2) ,
is concave
H

max((X 2) holds ?
? min
when does equality
bea flas
,

X ,
W
K KY condition :

O
[V , Dh ; x

(0x9
-
Df(x ; x, r) = [x , D9 : /x) +

(x) = 0
,

X = 0

*
hi(x) = 0
,
9 ; (x) = 0
Fano's inequality :

* x/X
Perx pr(X x)
=

X+ Y - Y ,

H (x(Y) = ho(Pel + Pelog(11-11

pf
9:
:

Let E :

H(E , X(X) H(X(x)

=
+ X3
H)E(X ,

= HIElX) + HIXIE XI ,

*
HIEl) ( H(E) = H(Pe)
&
H(X/E y1
,
= P(E 1) H(X(E
= = 1 ,
Y)

+ P(E =

0)H(X/%, x
=
PeHIXIE = 1 ,
X)
>
- Pelog (11-11
③ H(X(*) DPI)
=
H(X(y)(by
~ H(X(y) = H(pe) +
Peloy (11-1) #
 Neymann-Pearson

Axn
*
=

[x)
typet : n
=

Pr[A] ,
An =
-A
I:
Prin[Axa]
*

type Pn =

Let At Axn
&n =
PrynSA" ,
Pr =
PrESAY
*

if Ans &*, then In ? Pr