Scribd
Upload
6 views

Ex 2

Uploaded by

Satyam Sinha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
6 views

Ex 2

Uploaded by

Satyam Sinha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Intro to Statistical Learning Theory

Exercise 2

1) Find the VC dimension of the following hypothesis spaces (prove your


claim):
a) 
Parity functions. X = {0, 1}n , for any S ⊂ [n] define hS (x) =
P
xi mod 2. H = {hS , S ⊂ [n]}.
i∈S
b) The set of rectangles in Rd , i.e. H = {h(c,b) (x) = 1[∀i, |xi − ci | ≤
bi ], b, c ∈ Rd }. We have seen in class d = 2.
*c) The set of circles in R2 , i.e. H = {h(c,r) (x) = 1[|x − c| ≤ r], c ∈
R2 , r > 0}
2) For X = R, define H = {hθ (x) = dsin(θx)e, θ ∈ R} where we take d−1e =
0. Prove that V C(H) = ∞. Hint: prove and use the following lemma -
if x ∈ (0, 1) has binary expansion x = 0.x1 x2 ...xm .... then for any natural
number m, dsin(2m πx)e = 1 − xm provided that for some k ≥ m we have
xk = 1.
3) Let H1 and H2 be binary hypothesis spaces over X . define di = V C(H),
d = max(d1 , d2 ) and assume d ≥ 3.
a) Prove that V C(H1 ∪ H2 ) ≤ 2d + 1
b) Show and prove an upper bound on V C(H1 · H2 ), where H1 · H2 is
the class of all function of the form h1 (x) · h2 (x) when hi ∈ Hi .
Hint: Use the bound we found on the growth function.
4) Structural Risk Minimization: If H has uniform convergence with com-
plexity M(, δ) we define the confidence (m, δ) = min{m > M(, δ)}, i.e.
>0
the best approximation error we can learn given m examples and proba-
bility δ. Prove the following theorem:
∞
P
Let pn be a sequence of positive numbers such that pn ≤ 1. Let
n=1
H = ∪∞ n=1 Hn where Hn has uniform convergence with complexity Mn (, δ)
and confidence n (m, δ). For any distribution D we have with probability
at least 1 − δ over S ∼ Dm
∀h ∈ H, LD (h) ≤ LS (h) + min n (m, pn · δ).
n:h∈H

1
can you give a specific bound when pn = 2−n and V C(Hn ) = n?

You might also like