A Brief Introduction to Hilbert Space Theory with RKHS in Mind

Aydın Aytuna

2024

A. Aytuna RKHS 2024 1 / 58

1 Inner Product Spaces
Rudiments
Distance and Continuity
2 Hilbert Spaces
Definition
The Main Theorem
Consequences of the Main Theorem
DIGRESSION: Existence of Solutions
Orthonormal Bases
Completion
3 Reproducing Kernel Hilbert Spaces
Continuous Point Evaluations
Kernels
Properties of Kernel Functions
Reproducing Kernels
Yet Another Example
4 APPENDIX
5 Acknowledgment

A. Aytuna RKHS 2024 2 / 58

These notes are written for the ”RKHS Learning Seminars” at the Institute of Applied Mathematics,
METU. It aims to introduce the audience to the fundamental ideas of elementary Hilbert space theory.
We assume the participants have good knowledge of linear algebra and advanced calculus. The
material covered is relatively standard and contains no new mathematics. The book ”A Primer on
Reproducing Kernel Hilbert Spaces.” by J.H. Manton and P. Amblard was chosen as the principal
reference for this seminar series. Hence, it shaped the structure of these notes. Additional references
used in these notes plus references that can be used for further studies are listed in bibliography.

A. Aytuna RKHS 2024 3 / 58

 Inner Product Spaces Rudiments

Let V be a vector space over C (or R). Let B : V × V → C be a function satisfying

•B (x , y ) = B (y , x ) ∀x , y ∈ V ,
•B (x + λy , ξ) = B (x , ξ) + λB (y , ξ) ∀x , y , ξ ∈ V , λ ∈ C, (*)
•B (x , x ) ≥ 0 ∀x ∈ V .
An important property of such functions (usually refered to as sesquilinear fuctions) is:

|B (x , y )| ≤ B (x , x )1/2 B (y , y )1/2 ∀x , y ∈ V . (1)

To see this let us look at:

0 ≤ B (x + ty , x + ty ) = B (x , x ) + tB (y , x ) + tB (x , y ) + |t |2 B (y , y )
= B (x , x ) + 2Re(tB (x , y )) + |t |2 B (y , y ) ∀x , y ∈ V , t ∈ C
Choosing t = sB (y , x ), s ∈ R, we have;
0 ≤ B (x , x ) + 2s | B (x , y ) |2 +s 2 | B (y , x ) |2 B (y , y ).
So the discriminant of this polynomial P (s ) must satisfy,
4B (x , y )4 − 4B (x , x )B (y , y ) | B (y , x ) |2 ≤ 0
or
| B (x , y ) |4 ≤ B (x , x )B (y , y ) | B (y , x ) |2
| B (x , y ) |2 ≤ B (x , x )B (y , y ).

A. Aytuna RKHS 2024 4 / 58

 Inner Product Spaces Rudiments

Let V be a vector space over C (or R). Let B : V × V → C be a function satisfying

•B (x , y ) = B (y , x ) ∀x , y ∈ V ,
•B (x + λy , ξ) = B (x , ξ) + λB (y , ξ) ∀x , y , ξ ∈ V , λ ∈ C, (*)
•B (x , x ) ≥ 0 ∀x ∈ V .
An important property of such functions (usually refered to as sesquilinear fuctions) is:

|B (x , y )| ≤ B (x , x )1/2 B (y , y )1/2 ∀x , y ∈ V . (1)

To see this let us look at:

0 ≤ B (x + ty , x + ty ) = B (x , x ) + tB (y , x ) + tB (x , y ) + |t |2 B (y , y )
= B (x , x ) + 2Re(tB (x , y )) + |t |2 B (y , y ) ∀x , y ∈ V , t ∈ C
Choosing t = sB (y , x ), s ∈ R, we have;
0 ≤ B (x , x ) + 2s | B (x , y ) |2 +s 2 | B (y , x ) |2 B (y , y ).
So the discriminant of this polynomial P (s ) must satisfy,
4B (x , y )4 − 4B (x , x )B (y , y ) | B (y , x ) |2 ≤ 0
or
| B (x , y ) |4 ≤ B (x , x )B (y , y ) | B (y , x ) |2
| B (x , y ) |2 ≤ B (x , x )B (y , y ).

A. Aytuna RKHS 2024 4 / 58

 Inner Product Spaces Rudiments

This simple observation is the source of many inequalities, such as

X X 1/2  X 1/2
xi yi ≤ | xi |2 | yi |2 ,
i

or its continuous analog

Z b Z b 1/2  Z b 1/2
f (t )g (t )dt ≤ | f (t ) |2 dt | g (t ) |2 dt ∀f , g ∈ C [a , b ].
a a a

Another property of such functions is:

B (x + y , x + y ) ≤ B (x , x ) + 2ReB (x , y ) + B (y , y )
≤ B (x , x ) + 2B (x , x )1/2 B (y , y )1/2 + B (y , y ) (2)
2
B (x , x )1/2 + B (y , y )1/2 .

=

Recall that
An inner product ⟨ , ⟩ on (V , C) is a function on V × V that satisfy the conditions (*) with the additional
condition:
⟨x , x ⟩ = 0 ⇐⇒ x = 0.

A. Aytuna RKHS 2024 5 / 58

 Inner Product Spaces Rudiments

This simple observation is the source of many inequalities, such as

X X 1/2  X 1/2
xi yi ≤ | xi |2 | yi |2 ,
i

or its continuous analog

Z b Z b 1/2  Z b 1/2
f (t )g (t )dt ≤ | f (t ) |2 dt | g (t ) |2 dt ∀f , g ∈ C [a , b ].
a a a

Another property of such functions is:

B (x + y , x + y ) ≤ B (x , x ) + 2ReB (x , y ) + B (y , y )
≤ B (x , x ) + 2B (x , x )1/2 B (y , y )1/2 + B (y , y ) (2)
2
B (x , x )1/2 + B (y , y )1/2 .

=

Recall that
An inner product ⟨ , ⟩ on (V , C) is a function on V × V that satisfy the conditions (*) with the additional
condition:
⟨x , x ⟩ = 0 ⇐⇒ x = 0.

A. Aytuna RKHS 2024 5 / 58

 Inner Product Spaces Distance and Continuity

An inner product gives a "norm" on V by setting:

∥ x ∥= ⟨x , x ⟩1/2 ,

which can be thought as a "distance of x to the vector 0. In fact

d (x , y ) =∥ x − y ∥

satisfies the usual conditions of a metric on V . Namely:

d (x , y ) ≤ d (x , z ) + d (z , y ),
d (x , y ) = d (y , x ),
d (x , x ) ≥ 0 and d (x , x ) = 0 ⇔ x = 0.

Hence an inner product on V induces a notion of "distance" between points in V , therefore introduce
notions like "convergence" of sequences via

xn → x ⇔ ∥ xn − x ∥→ 0

and "continuity" of functions f : V1 → V2 between inner product spaces via

f is continuous at a point x in case: xn → x ∈ V1 =⇒ f (xn ) → f (x ) ∈ V2 .

Note that for linear operators convergence can be checked by using norms.

A. Aytuna RKHS 2024 6 / 58

 Inner Product Spaces Distance and Continuity

An inner product gives a "norm" on V by setting:

∥ x ∥= ⟨x , x ⟩1/2 ,

which can be thought as a "distance of x to the vector 0. In fact

d (x , y ) =∥ x − y ∥

satisfies the usual conditions of a metric on V . Namely:

d (x , y ) ≤ d (x , z ) + d (z , y ),
d (x , y ) = d (y , x ),
d (x , x ) ≥ 0 and d (x , x ) = 0 ⇔ x = 0.

Hence an inner product on V induces a notion of "distance" between points in V , therefore introduce
notions like "convergence" of sequences via

xn → x ⇔ ∥ xn − x ∥→ 0

and "continuity" of functions f : V1 → V2 between inner product spaces via

f is continuous at a point x in case: xn → x ∈ V1 =⇒ f (xn ) → f (x ) ∈ V2 .

Note that for linear operators convergence can be checked by using norms.

A. Aytuna RKHS 2024 6 / 58

 Inner Product Spaces Distance and Continuity

Theorem
For a linear operator T : (V1 , ⟨ , ⟩1 ) → (V2 , ⟨ , ⟩2 ) the following are equivalent:
(i) T is continuous at the point 0,
(ii) T is continuous at every point of V (3)
(iii) ∃c > 0 ; ∥ T (x ) ∥2 ≤ c ∥ x ∥1 .

Proof.
To see "(i ) =⇒ (ii )", suppose T is continuous at 0. Fix x and suppose xn → x, i.e. xn − x → 0, then
T (xn ) − T (x ) = T (xn − x ) → T (0) = 0 or, T is continuous at x.

To show "(i ) =⇒ (iii )", suppose no such c exists. Then for each N ∈ N, we can find xN , 0 such that

∥ T (xN ) ∥2 ≥ N ∥ xN ∥1 .
xN
Consider the sequence {uN }N , where uN := √ . We have:
N ∥xN ∥1

xN ∥ xN ∥1 1
∥ uN ∥1 = √ = √ = √ →0 as N → ∞.
N ∥ xN ∥1 1 N ∥ xN ∥1 N
This implies T (uN ) → 0, but we also have:
∥ T (xN ) ∥2 N ∥ xN ∥1 √
∥ T (uN ) ∥2 = √ ≥ √ = N,
N ∥ xN ∥1 N ∥ xN ∥1
so the assumption leads to a contradiction. Rest of the assertion is self evident. □
A. Aytuna RKHS 2024 7 / 58
 Inner Product Spaces Distance and Continuity

Theorem
For a linear operator T : (V1 , ⟨ , ⟩1 ) → (V2 , ⟨ , ⟩2 ) the following are equivalent:
(i) T is continuous at the point 0,
(ii) T is continuous at every point of V (3)
(iii) ∃c > 0 ; ∥ T (x ) ∥2 ≤ c ∥ x ∥1 .

Proof.
To see "(i ) =⇒ (ii )", suppose T is continuous at 0. Fix x and suppose xn → x, i.e. xn − x → 0, then
T (xn ) − T (x ) = T (xn − x ) → T (0) = 0 or, T is continuous at x.

To show "(i ) =⇒ (iii )", suppose no such c exists. Then for each N ∈ N, we can find xN , 0 such that

∥ T (xN ) ∥2 ≥ N ∥ xN ∥1 .
xN
Consider the sequence {uN }N , where uN := √ . We have:
N ∥xN ∥1

xN ∥ xN ∥1 1
∥ uN ∥1 = √ = √ = √ →0 as N → ∞.
N ∥ xN ∥1 1 N ∥ xN ∥1 N
This implies T (uN ) → 0, but we also have:
∥ T (xN ) ∥2 N ∥ xN ∥1 √
∥ T (uN ) ∥2 = √ ≥ √ = N,
N ∥ xN ∥1 N ∥ xN ∥1
so the assumption leads to a contradiction. Rest of the assertion is self evident. □
A. Aytuna RKHS 2024 7 / 58
 Inner Product Spaces Distance and Continuity

Remark
In the special case of finite dimensional inner product spaces V1 and V2 , every operator T : V1 → V2 is
automatically continuous since if {e1 , · · · , eN } is an orthonormal basis in a finite dimensional inner
product space (V1 , ⟨ , ⟩), then, if xn → x, since
N
X N
X
xn = c1n e1 + · · · + cNn eN = cin ei , x = c1 e1 + · · · + cN eN = ci ei
i =1 i =1

we have:
DP E DP E
∥ xn − x ∥2 = N c n e −PN c e ,PN c n e −PN c e
i =1 i i i =1 i i i =1 i i i =1 i i
= N (c n −c )e ,PN (c n −c )e
i =1 i i i i =1 i i i
D E
= PNi=1 (cin −ci )· ei ,PNj=1 (cjn −cj )ej = PNi=1 PNj=1 (cin −ci )(cjn −cj )⟨ei ,ej ⟩
= PNi=1 |cin −ci |2 ,

since ⟨ei , ej ⟩ = 0 for i , j and ⟨ei , ei ⟩ =∥ ei ∥2 = 1. So xn − x → 0 ⇐⇒| cin − ci |→ 0, ∀i = 1, · · · , N,

⇐⇒ cin → ci ∀i = 1, · · · , N .

With respect to orthonormal basis {ei }N

i =1
of V1 and {fj }M
j =1
of V2 , T can be represented as a matrix
PM
A = {aij }, where Tei = j =1 aij fj , so
N
X  M X
X N  M X
X N
T (x ) = T xi ei = aij xi fj , T (xn − x ) = aij (xin − xi )fj .
i =1 j =1 i =1 j =1 i =1

Hence Txn → Tx as xn → x.

A. Aytuna RKHS 2024 8 / 58

 Inner Product Spaces Distance and Continuity

Remark
In the special case of finite dimensional inner product spaces V1 and V2 , every operator T : V1 → V2 is
automatically continuous since if {e1 , · · · , eN } is an orthonormal basis in a finite dimensional inner
product space (V1 , ⟨ , ⟩), then, if xn → x, since
N
X N
X
xn = c1n e1 + · · · + cNn eN = cin ei , x = c1 e1 + · · · + cN eN = ci ei
i =1 i =1

we have:
DP E DP E
∥ xn − x ∥2 = N c n e −PN c e ,PN c n e −PN c e
i =1 i i i =1 i i i =1 i i i =1 i i
= N (c n −c )e ,PN (c n −c )e
i =1 i i i i =1 i i i
D E
= PNi=1 (cin −ci )· ei ,PNj=1 (cjn −cj )ej = PNi=1 PNj=1 (cin −ci )(cjn −cj )⟨ei ,ej ⟩
= PNi=1 |cin −ci |2 ,

since ⟨ei , ej ⟩ = 0 for i , j and ⟨ei , ei ⟩ =∥ ei ∥2 = 1. So xn − x → 0 ⇐⇒| cin − ci |→ 0, ∀i = 1, · · · , N,

⇐⇒ cin → ci ∀i = 1, · · · , N .

With respect to orthonormal basis {ei }N

i =1
of V1 and {fj }M
j =1
of V2 , T can be represented as a matrix
PM
A = {aij }, where Tei = j =1 aij fj , so
N
X  M X
X N  X N
M X
T (x ) = T xi ei = aij xi fj , T (xn − x ) = aij (xin − xi )fj .
i =1 j =1 i =1 j =1 i =1

Hence Txn → Tx as xn → x.

A. Aytuna RKHS 2024 8 / 58

 Inner Product Spaces Distance and Continuity

At this point, we would like to recall Gram-Schmidt orthogonalization process:

Given n linearly independent elements v1 , · · · , vn in an inner product space (V , ⟨ , ⟩), consider the
inductively defined sequence of vectors:

k −1
⟨v2 , u1 ⟩ X ⟨vk , ui ⟩
u1 = v1 , u2 = v2 − u1 , ··· , uk = vk − ui .
⟨u1 , u1 ⟩ i =1
⟨ui , ui ⟩

A routine check will show that ui ’s for 1 ≤ k ≤ n, are orthogonal to each other, i.e, ⟨ui , uj ⟩ = 0, and the
n u o
normalized ∥ui ∥ vectors form an orthonormal basis for span{v1 , · · · , vn }.
i

Also observe that each uk ∈ span{v1 , · · · , vk }.

This algorithm works also in infinite dimensional inner product spaces if a sequence of finitely linearly
independent vectors {vi }∞
i =1
are given, and it yields a sequence orthonormal vectors {ui }∞
i =1
spanning
the vector subspace spanned by {vi }’s.

Automatic continuity of linear operators fails if the dimension is not finite.

A. Aytuna RKHS 2024 9 / 58

 Inner Product Spaces Distance and Continuity

At this point, we would like to recall Gram-Schmidt orthogonalization process:

Given n linearly independent elements v1 , · · · , vn in an inner product space (V , ⟨ , ⟩), consider the
inductively defined sequence of vectors:

k −1
⟨v2 , u1 ⟩ X ⟨vk , ui ⟩
u1 = v1 , u2 = v2 − u1 , ··· , uk = vk − ui .
⟨u1 , u1 ⟩ i =1
⟨ui , ui ⟩

A routine check will show that ui ’s for 1 ≤ k ≤ n, are orthogonal to each other, i.e, ⟨ui , uj ⟩ = 0, and the
n u o
normalized ∥ui ∥ vectors form an orthonormal basis for span{v1 , · · · , vn }.
i

Also observe that each uk ∈ span{v1 , · · · , vk }.

This algorithm works also in infinite dimensional inner product spaces if a sequence of finitely linearly
independent vectors {vi }∞
i =1
are given, and it yields a sequence orthonormal vectors {ui }∞
i =1
spanning
the vector subspace spanned by {vi }’s.

Automatic continuity of linear operators fails if the dimension is not finite.

A. Aytuna RKHS 2024 9 / 58

 Inner Product Spaces Distance and Continuity

Example 0

Let V = (C [0, 1], ⟨ , ⟩) be the vector space of real valued continuous functions on [0, 1] with the inner
product
Z 1
⟨f , g ⟩ := f (t )g (t )dt .
0

Let k0 : C [0, 1] → (R, ⟨ , ⟩) be the linear operator k0 (f ) := f (0).

Consider the sequence {fn } in C [0, 1] defined as:


 −n 3 t + n

 0 ≤ t ≤ n12 ,
fn (t ) = 
 0 1
n2
< t ≤ 1.
R1
Note ∥ fn ∥= 0
fn (t )2 dt ≤ 1, yet ∥ k0 (fn ) ∥=| fn (0) |= n, so ∃ no C > 0
such that
∥ k0 (f ) ∥≤ C ∥ f ∥ ∀f ∈ C [0, 1].
Hence k0 is not continuous.

This is one of the reasons why continuity (in general the topology) is suppressed in finite dimensional
linear algebra and why in infinite dimensional linear algebra topology plays an important role if one
wants to develop a reasonable theory.

A. Aytuna RKHS 2024 10 / 58

 Inner Product Spaces Distance and Continuity

Example 0

Let V = (C [0, 1], ⟨ , ⟩) be the vector space of real valued continuous functions on [0, 1] with the inner
product
Z 1
⟨f , g ⟩ := f (t )g (t )dt .
0

Let k0 : C [0, 1] → (R, ⟨ , ⟩) be the linear operator k0 (f ) := f (0).

Consider the sequence {fn } in C [0, 1] defined as:


 −n 3 t + n

 0 ≤ t ≤ n12 ,
fn (t ) = 
 0 1
n2
< t ≤ 1.
R1
Note ∥ fn ∥= 0
fn (t )2 dt ≤ 1, yet ∥ k0 (fn ) ∥=| fn (0) |= n, so ∃ no C > 0
such that
∥ k0 (f ) ∥≤ C ∥ f ∥ ∀f ∈ C [0, 1].
Hence k0 is not continuous.

This is one of the reasons why continuity (in general the topology) is suppressed in finite dimensional
linear algebra and why in infinite dimensional linear algebra topology plays an important role if one
wants to develop a reasonable theory.

A. Aytuna RKHS 2024 10 / 58

 Inner Product Spaces Distance and Continuity

We close this section with an elementary observation about inner product spaces that generalizes the
Parallelogram law of elementary geometry:

∥ x − y ∥2 + ∥ x + y ∥2 = ⟨x − y , x − y ⟩ + ⟨x + y , x + y ⟩
= ⟨x , x ⟩ − 2Re⟨x , y ⟩ + ⟨y , y ⟩ + ⟨x , x ⟩ + 2Re⟨x , y ⟩ + ⟨y , y ⟩
= 2 ∥ x ∥2 +2 ∥ y ∥2 ,

or

∥ x + y ∥2 + ∥ x − y ∥2 = 2 ∥ x ∥2 +2 ∥ y ∥2 .

A. Aytuna RKHS 2024 11 / 58

 Hilbert Spaces Definition

If a sequence "clusters" in an inner product space, it is desirable that it converges somewhere. In other
words if the distances between the points of the sequence gets very small as one proceeds to the tail of
the sequence (in some sense one wants to assume that there are no "holes" in the space!)

Example
In C [0, 1] of Example 0, for n ≥ 2 let:
0<t ≤ 1 − n1



 1 2
fn ( t ) =  −nt + n 1 − 1 ≤t ≤ 1

 2 2 n 2

 0 1 <t .
2
Z 1
2 4
∥ fn − fm ∥2 = | (fn − fm )(t ) |2 dt ≤ if m > n.
1−1 n
2 n

So limn,m→∞ ∥ fn − fm ∥2 = 0, and yet if ∃F ∈ C [0, 1] such that ∥ fn − F ∥2 → 0;

Z 1−1 Z 1 Z 1
2 n 2
∥ fn − F ∥2 = | F − 1 |2 + | F − fn |2 + | F |2 → 0 .
0 1−1 1
2 n 2

Then F ≡ 0 on [1/2, 1] and F ≡ 1 on any closed interval [0, α], α < 1/2. This means:

1= lim F (t ) = lim F ( t ) = 0.
t →(1/2)− t →(1/2)+

So ∃ no such F ∈ C [0, 1].

A. Aytuna RKHS 2024 12 / 58

 Hilbert Spaces Definition

If a sequence "clusters" in an inner product space, it is desirable that it converges somewhere. In other
words if the distances between the points of the sequence gets very small as one proceeds to the tail of
the sequence (in some sense one wants to assume that there are no "holes" in the space!)

Example
In C [0, 1] of Example 0, for n ≥ 2 let:
0<t ≤ 1 − n1



 1 2
fn ( t ) =  −nt + n 1 − 1 ≤t ≤ 1

 2 2 n 2

 0 1 <t .
2
Z 1
2 4
∥ fn − fm ∥2 = | (fn − fm )(t ) |2 dt ≤ if m > n.
1−1 n
2 n

So limn,m→∞ ∥ fn − fm ∥2 = 0, and yet if ∃F ∈ C [0, 1] such that ∥ fn − F ∥2 → 0;

Z 1−1 Z 1 Z 1
2 n 2
∥ fn − F ∥2 = | F − 1 |2 + | F − fn |2 + | F |2 → 0 .
0 1−1 1
2 n 2

Then F ≡ 0 on [1/2, 1] and F ≡ 1 on any closed interval [0, α], α < 1/2. This means:

1= lim F (t ) = lim F ( t ) = 0.
t →(1/2)− t →(1/2)+

So ∃ no such F ∈ C [0, 1].

A. Aytuna RKHS 2024 12 / 58

 Hilbert Spaces Definition

Let us establish some terminology:

Definition
A sequence {xn } in an inner product space is said to be Cauchy in case ∀ε > 0 ∃N such that if
n, m ≥ N then ∥ xn − xm ∥≤ ε
An inner product space (V , ⟨ , ⟩) is called complete in case every Cauchy sequence converges to
a point in V
A complete inner product space is called a Hilbert space.

Main Example
X X
ℓ2 = {(xn ) : | xn |2 < ∞} with ⟨x , y ⟩ := xn yn , for x = (xn ), y = (yn ).
n n

This makes sense by Equation (1) of Part 1. If x α = {xnα }∞

n =1
, α = 1, 2, · · · is a Cauchy sequence in ℓ2
then fix ε > 0, ∃Mε for every N
N ∞
∥ x α − x β ∥2 = | xnα − xnβ |2 + | xnα − xnβ |2 ≤ ε2 /4
X X
if α, β ≥ M .
n=1 n =N +1

α := {x α }N
Then ξN N N
n n=1 is Cauchy in C , so it converges in C .

α→∞
In particular, ∃x = (xn )∞
n =1
such that xnα −−−−→ xn ∀n and ξN
α → (x , · · · , x ) in CN .
1 N

A. Aytuna RKHS 2024 13 / 58

 Hilbert Spaces Definition

Let us establish some terminology:

Definition
A sequence {xn } in an inner product space is said to be Cauchy in case ∀ε > 0 ∃N such that if
n, m ≥ N then ∥ xn − xm ∥≤ ε
An inner product space (V , ⟨ , ⟩) is called complete in case every Cauchy sequence converges to
a point in V
A complete inner product space is called a Hilbert space.

Main Example
X X
ℓ2 = {(xn ) : | xn |2 < ∞} with ⟨x , y ⟩ := xn yn , for x = (xn ), y = (yn ).
n n

This makes sense by Equation (1) of Part 1. If x α = {xnα }∞

n =1
, α = 1, 2, · · · is a Cauchy sequence in ℓ2
then fix ε > 0, ∃Mε for every N
N ∞
∥ x α − x β ∥2 = | xnα − xnβ |2 + | xnα − xnβ |2 ≤ ε2 /4
X X
if α, β ≥ M .
n=1 n =N +1

α := {x α }N
Then ξN N N
n n=1 is Cauchy in C , so it converges in C .

α→∞
In particular, ∃x = (xn )∞
n =1
such that xnα −−−−→ xn ∀n and ξN
α → (x , · · · , x ) in CN .
1 N

A. Aytuna RKHS 2024 13 / 58

 Hilbert Spaces Definition

Main Example(Cont.)
β
Therefore for any S, and α ≥ Mε , choose β so that β ≥ Mε and | xn − xn |2 ≤ ε2 /4.
PS
n =1
S
X 1/2 S
X 1/2 S
X 1/2
| xnα − xnβ |2
β
=⇒ | xnα − xn |2 ≤ + | xn − xn |2 ≤ ε,
n =1 n=1 n =1
∞
| xnα − xn |2 ≤ ε ∀α ≥ Mε =⇒ (xn ) ∈ ℓ2 and x α → x .
X
=⇒
n=1

So ℓ2 is a Hilbert space.

Notation
For any subset S of a Hilbert space (H , ⟨ , ⟩), S ⊥ will denote the set of all elements of H that are
orthogonal to elements of S. That is:

S ⊥ := {x ∈ H : ⟨x , s ⟩ = 0 ∀s ∈ S }.

Clearly S ⊥ is a subspace, and is closed in the sense that it contains the limit points of all Cauchy
sequences in it, since xn ∈ S ⊥ , n = 1, 2, · · · , and xn → x implies that
n→∞
| ⟨x , s ⟩ |=| ⟨xn , s ⟩ − ⟨x , s ⟩ |=| ⟨xn − x , s ⟩ |≤∥ xn − x ∥ ∥ s ∥−−−−→ 0 =⇒ x ∈ S ⊥ .

Hence S ⊥ is itself a Hilbert space under ⟨ , ⟩.

A. Aytuna RKHS 2024 14 / 58

 Hilbert Spaces Definition

Main Example(Cont.)
β
Therefore for any S, and α ≥ Mε , choose β so that β ≥ Mε and | xn − xn |2 ≤ ε2 /4.
PS
n =1
S
X 1/2 S
X 1/2 S
X 1/2
| xnα − xnβ |2
β
=⇒ | xnα − xn |2 ≤ + | xn − xn |2 ≤ ε,
n =1 n=1 n =1
∞
| xnα − xn |2 ≤ ε ∀α ≥ Mε =⇒ (xn ) ∈ ℓ2 and x α → x .
X
=⇒
n=1

So ℓ2 is a Hilbert space.

Notation
For any subset S of a Hilbert space (H , ⟨ , ⟩), S ⊥ will denote the set of all elements of H that are
orthogonal to elements of S. That is:

S ⊥ := {x ∈ H : ⟨x , s ⟩ = 0 ∀s ∈ S }.

Clearly S ⊥ is a subspace, and is closed in the sense that it contains the limit points of all Cauchy
sequences in it, since xn ∈ S ⊥ , n = 1, 2, · · · , and xn → x implies that
n→∞
| ⟨x , s ⟩ |=| ⟨xn , s ⟩ − ⟨x , s ⟩ |=| ⟨xn − x , s ⟩ |≤∥ xn − x ∥ ∥ s ∥−−−−→ 0 =⇒ x ∈ S ⊥ .

Hence S ⊥ is itself a Hilbert space under ⟨ , ⟩.

A. Aytuna RKHS 2024 14 / 58

 Hilbert Spaces The Main Theorem
In practice, one needs to know the existence of a point in a given set C = 0 of a Hilbert space that is
closest to 0, and if it exists, whether this point is unique.
One can get a satisfactory answer if C is a closed convex set in H. Closed in the sense that it contains
all its limit points (i.e., xn → x , xn ∈ C =⇒ x ∈ C), and convex in the sense that ∀x , y ∈ C, the midpoint
(x + y )/2 ∈ C.

Theorem
Given a closed convex set C in a Hilbert space H, ∃! point x ∈ C that is closest to 0.

Proof.
We might as well assume 0 < C. Choose a sequence {xn } ∈ C such that ∥ xn ∥→ inf x ∈C ∥ x ∥:= d.
1 1
∥ xn − xm ∥2 =∥ xn ∥2 + ∥ xm ∥2 − ∥ xn + xm ∥2 (parallelogram law)
2 2
2
1 xn + xm

=∥ xn ∥2 + ∥ xm ∥2 − 4 (convexity)
2 2
≤∥ xn ∥2 + ∥ xm ∥2 −2d 2 → 2d 2 − 2d 2 = 0 as n, m → ∞.

So, by taking limits as n, m → ∞, limn,m→∞ ∥ xn − xm ∥= 0, i.e., {xn } is Cauchy, so it converges to an

x0 ∈ C. Clearly:
∥ x0 ∥=∥ x0 − xn + xn ∥≤∥ x0 − xn ∥ + ∥ xn ∥,
so ∥ x0 ∥= d. If ∃x0 , y0 ∈ C with ∥ x0 ∥=∥ y0 ∥= d, then again by Parallelogram law:
∥ x0 − y0 ∥2 ∥ x0 − y0 ∥2
=− + ∥ x0 ∥2 + ∥ y0 ∥2 ≤ −2d 2 + 2d 2 = 0 =⇒ x0 = y0 . □
2 2

A. Aytuna RKHS 2024 15 / 58

 Hilbert Spaces The Main Theorem
In practice, one needs to know the existence of a point in a given set C = 0 of a Hilbert space that is
closest to 0, and if it exists, whether this point is unique.
One can get a satisfactory answer if C is a closed convex set in H. Closed in the sense that it contains
all its limit points (i.e., xn → x , xn ∈ C =⇒ x ∈ C), and convex in the sense that ∀x , y ∈ C, the midpoint
(x + y )/2 ∈ C.

Theorem
Given a closed convex set C in a Hilbert space H, ∃! point x ∈ C that is closest to 0.

Proof.
We might as well assume 0 < C. Choose a sequence {xn } ∈ C such that ∥ xn ∥→ inf x ∈C ∥ x ∥:= d.
1 1
∥ xn − xm ∥2 =∥ xn ∥2 + ∥ xm ∥2 − ∥ xn + xm ∥2 (parallelogram law)
2 2
2
1 xn + xm

=∥ xn ∥2 + ∥ xm ∥2 − 4 (convexity)
2 2
≤∥ xn ∥2 + ∥ xm ∥2 −2d 2 → 2d 2 − 2d 2 = 0 as n, m → ∞.

So, by taking limits as n, m → ∞, limn,m→∞ ∥ xn − xm ∥= 0, i.e., {xn } is Cauchy, so it converges to an

x0 ∈ C. Clearly:
∥ x0 ∥=∥ x0 − xn + xn ∥≤∥ x0 − xn ∥ + ∥ xn ∥,
so ∥ x0 ∥= d. If ∃x0 , y0 ∈ C with ∥ x0 ∥=∥ y0 ∥= d, then again by Parallelogram law:
∥ x0 − y0 ∥2 ∥ x0 − y0 ∥2
=− + ∥ x0 ∥2 + ∥ y0 ∥2 ≤ −2d 2 + 2d 2 = 0 =⇒ x0 = y0 . □
2 2

A. Aytuna RKHS 2024 15 / 58

 Hilbert Spaces The Main Theorem
In practice, one needs to know the existence of a point in a given set C = 0 of a Hilbert space that is
closest to 0, and if it exists, whether this point is unique.
One can get a satisfactory answer if C is a closed convex set in H. Closed in the sense that it contains
all its limit points (i.e., xn → x , xn ∈ C =⇒ x ∈ C), and convex in the sense that ∀x , y ∈ C, the midpoint
(x + y )/2 ∈ C.

Theorem
Given a closed convex set C in a Hilbert space H, ∃! point x ∈ C that is closest to 0.

Proof.
We might as well assume 0 < C. Choose a sequence {xn } ∈ C such that ∥ xn ∥→ inf x ∈C ∥ x ∥:= d.
1 1
∥ xn − xm ∥2 =∥ xn ∥2 + ∥ xm ∥2 − ∥ xn + xm ∥2 (parallelogram law)
2 2
2
1 xn + xm

=∥ xn ∥2 + ∥ xm ∥2 − 4 (convexity)
2 2
≤∥ xn ∥2 + ∥ xm ∥2 −2d 2 → 2d 2 − 2d 2 = 0 as n, m → ∞.

So, by taking limits as n, m → ∞, limn,m→∞ ∥ xn − xm ∥= 0, i.e., {xn } is Cauchy, so it converges to an

x0 ∈ C. Clearly:
∥ x0 ∥=∥ x0 − xn + xn ∥≤∥ x0 − xn ∥ + ∥ xn ∥,
so ∥ x0 ∥= d. If ∃x0 , y0 ∈ C with ∥ x0 ∥=∥ y0 ∥= d, then again by Parallelogram law:
∥ x0 − y0 ∥2 ∥ x0 − y0 ∥2
=− + ∥ x0 ∥2 + ∥ y0 ∥2 ≤ −2d 2 + 2d 2 = 0 =⇒ x0 = y0 . □
2 2

A. Aytuna RKHS 2024 15 / 58

 Hilbert Spaces The Main Theorem

DISCUSSION
Now let us apply this result to problem of finding the nearest point to a given point x0 in a closed
subspace M of H.

By the theorem, the point we are seeking is the point x0

minus the point with smallest norm of x0 + M. Call this point P (x0 ) ∈ M.

∥ x0 − P (x0 ) ∥= inf ∥ x0 + m ∥= inf ∥ x0 − m ∥ .

m ∈M m ∈M

Set Q (x ) := x − P (x ). Let us look at the properties of P:

For m ∈ M consider:

∥ Q (x0 ) ∥2 ≤∥ Q (x0 ) + λm ∥2 =∥ Q (x0 ) ∥2 +2Reλ⟨Q (x0 ), m⟩+ | λ |2 ∥ m ∥2 .

⟨Q (x ),m⟩
If ⟨Q (x0 ), m⟩ , 0, choose λ = t |⟨Q (x0 ),m⟩| , t ∈ R to get:
0

0≤ 2t
|t | | ⟨Q (x0 ), m⟩ | + | t | ∥ m ∥2 , ∀t ∈ R \ {0}.

t
Let t → 0− , since |t | = −1, we get | ⟨Q (x0 ), m⟩ |= 0. So Q (x ) ⊥ M ∀x ∈ H. Hence:
     
M ∋ P (x + λy ) − P (x ) + λP (y ) = P (x + λy ) − (x + λy ) − P (x ) − x − λ P (y ) − y
 
= − Q (x + λy ) − Q (x ) − λQ (y ) ∈ M ⊥

A. Aytuna RKHS 2024 16 / 58

 Hilbert Spaces The Main Theorem

DISCUSSION
Now let us apply this result to problem of finding the nearest point to a given point x0 in a closed
subspace M of H.

By the theorem, the point we are seeking is the point x0

minus the point with smallest norm of x0 + M. Call this point P (x0 ) ∈ M.

∥ x0 − P (x0 ) ∥= inf ∥ x0 + m ∥= inf ∥ x0 − m ∥ .

m ∈M m ∈M

Set Q (x ) := x − P (x ). Let us look at the properties of P:

For m ∈ M consider:

∥ Q (x0 ) ∥2 ≤∥ Q (x0 ) + λm ∥2 =∥ Q (x0 ) ∥2 +2Reλ⟨Q (x0 ), m⟩+ | λ |2 ∥ m ∥2 .

⟨Q (x ),m⟩
If ⟨Q (x0 ), m⟩ , 0, choose λ = t |⟨Q (x0 ),m⟩| , t ∈ R to get:
0

0≤ 2t
|t | | ⟨Q (x0 ), m⟩ | + | t | ∥ m ∥2 , ∀t ∈ R \ {0}.

t
Let t → 0− , since |t | = −1, we get | ⟨Q (x0 ), m⟩ |= 0. So Q (x ) ⊥ M ∀x ∈ H. Hence:
     
M ∋ P (x + λy ) − P (x ) + λP (y ) = P (x + λy ) − (x + λy ) − P (x ) − x − λ P (y ) − y
 
= − Q (x + λy ) − Q (x ) − λQ (y ) ∈ M ⊥

A. Aytuna RKHS 2024 16 / 58

 Hilbert Spaces The Main Theorem

DISCUSSION (Cont.)
By the very definition
P 2 (x ) = P (P (x )) = P (x ).
∥ Q (P (x )) ∥= inf ∥ P (x ) − m ∥= 0 ⇒ QP (x ) = 0.
m ∈M

Similarly,

Q (x ) = P (Q (x )) + Q (Q (x ))
=⇒ M ∋ P (Q (x )) = Q (x ) − Q 2 (x ) ∈ M ⊥
=⇒ Q 2 = Q , PQ = 0.

In particular:

⟨P (x ), y ⟩ = ⟨P (x ), P (y )⟩ ⟨P (x ), y ⟩ = ⟨x , P (y )⟩
and
⟨x , P (y )⟩ = ⟨P (x ), P (y )⟩ ⟨Q (x ), y ⟩ = ⟨x , Q (y )⟩

For x ∈ H, x = P (x ) + Q (x ).

Then:
⟨x , x ⟩ = ⟨P (x ), P (x )⟩ + ⟨Q (x ), Q (x )⟩ ⇒∥ P (x ) ∥≤∥ x ∥, ∥ Q (x ) ∥≤∥ x ∥ .

We will summarize our findings in:

A. Aytuna RKHS 2024 17 / 58

 Hilbert Spaces The Main Theorem

DISCUSSION (Cont.)
By the very definition
P 2 (x ) = P (P (x )) = P (x ).
∥ Q (P (x )) ∥= inf ∥ P (x ) − m ∥= 0 ⇒ QP (x ) = 0.
m ∈M

Similarly,

Q (x ) = P (Q (x )) + Q (Q (x ))
=⇒ M ∋ P (Q (x )) = Q (x ) − Q 2 (x ) ∈ M ⊥
=⇒ Q 2 = Q , PQ = 0.

In particular:

⟨P (x ), y ⟩ = ⟨P (x ), P (y )⟩ ⟨P (x ), y ⟩ = ⟨x , P (y )⟩
and
⟨x , P (y )⟩ = ⟨P (x ), P (y )⟩ ⟨Q (x ), y ⟩ = ⟨x , Q (y )⟩

For x ∈ H, x = P (x ) + Q (x ).

Then:
⟨x , x ⟩ = ⟨P (x ), P (x )⟩ + ⟨Q (x ), Q (x )⟩ ⇒∥ P (x ) ∥≤∥ x ∥, ∥ Q (x ) ∥≤∥ x ∥ .

We will summarize our findings in:

A. Aytuna RKHS 2024 17 / 58

 Hilbert Spaces The Main Theorem

Theorem (Main Theorem)

Let (H , ⟨ , ⟩) be a Hilbert space and M a closed subset of H. Then:
Every element x of H decomposes uniquely as x = xM + xM ⊥ , where xM ∈ M and xM ⊥ ∈ M ⊥ . In
other words H = M ⊕ M ⊥ .
There exist continuous linear operators P : H → M, Q : H → M ⊥ with

∥ P (x ) ∥2 + ∥ Q (x ) ∥2 =∥ x ∥2

such that ∀x ∈ H,
x = P (x ) + Q (x )
is the unique decomposition of x into M and M ⊥ above, i.e., P + Q = I =Identity.
P 2 = P, PQ = 0, Q 2 = Q and

⟨P (x ), y ⟩ = ⟨x , P (y )⟩, ⟨Q (x ), y ⟩ = ⟨x , Q (y )⟩

The operator Q satisfies the quantitative expression:

∥ Q (x ) ∥= inf ∥ x − m ∥= distance of x to M .
m ∈M

Terminology
We will call the operators P and Q above as projections onto M and M ⊥ , respectively.

A. Aytuna RKHS 2024 18 / 58

 Hilbert Spaces The Main Theorem

Theorem (Main Theorem)

Let (H , ⟨ , ⟩) be a Hilbert space and M a closed subset of H. Then:
Every element x of H decomposes uniquely as x = xM + xM ⊥ , where xM ∈ M and xM ⊥ ∈ M ⊥ . In
other words H = M ⊕ M ⊥ .
There exist continuous linear operators P : H → M, Q : H → M ⊥ with

∥ P (x ) ∥2 + ∥ Q (x ) ∥2 =∥ x ∥2

such that ∀x ∈ H,
x = P (x ) + Q (x )
is the unique decomposition of x into M and M ⊥ above, i.e., P + Q = I =Identity.
P 2 = P, PQ = 0, Q 2 = Q and

⟨P (x ), y ⟩ = ⟨x , P (y )⟩, ⟨Q (x ), y ⟩ = ⟨x , Q (y )⟩

The operator Q satisfies the quantitative expression:

∥ Q (x ) ∥= inf ∥ x − m ∥= distance of x to M .
m ∈M

Terminology
We will call the operators P and Q above as projections onto M and M ⊥ , respectively.

A. Aytuna RKHS 2024 18 / 58

 Hilbert Spaces Consequences of the Main Theorem

Theorem (Riesz Representation Theorem)

Let f : H → C be a continuous linear operator. Then there exists a unique element xf of H with
f (x ) = ⟨x , xf ⟩.
Conversely, for any y ∈ H, the assignment x → ⟨x , y ⟩ defines a continuous linear operator from H into C.

Proof.
Consider a continuous linear operator f : H → C. The kernel K of f , i.e., K = {x : f (x ) = 0} is a closed
subspace of H since f is continuous. Consider a nonzero element x0 ∈ K ⊥ (if no such element exists,
we can take xf = 0). For any x ∈ H, the decomposition
   
f (x ) f (x )
x= x− f (x0 )
x0 + f (x0 )
x0

f (x )
is the unique decomposition of x into K ⊥ and K since x − x
f (x 0 ) 0
∈ K . So:

⟨x0 ,x0 ⟩
D E D E
x, f (x0 )
∥x0 ∥2
x0 = f (x )
f (x0 )
x0 , f (x0 )
∥x0 ∥2
x0 = f (x ) ff ((xx0 )) ∥x0 ∥2
= f (x ).
0

f (x0 )
It follows f (x ) = ⟨x , xf ⟩ with xf = ∥x0 ∥2 0
x .
If there exists another xf̃ with f (x ) = ⟨x , xf̃ ⟩, then ⟨x , xf − xf̃ ⟩ = 0 ∀x ∈ H. In particular,
⟨xf − xf̃ , xf − xf̃ ⟩ = 0, which implies xf = xf̃ .
The converse part of the theorem follows readily from the inequality

| ⟨x , y ⟩ |≤∥ x ∥ ∥ y ∥ ∀x , y ∈ H . □
A. Aytuna RKHS 2024 19 / 58
 Hilbert Spaces Consequences of the Main Theorem

Theorem (Riesz Representation Theorem)

Let f : H → C be a continuous linear operator. Then there exists a unique element xf of H with
f (x ) = ⟨x , xf ⟩.
Conversely, for any y ∈ H, the assignment x → ⟨x , y ⟩ defines a continuous linear operator from H into C.

Proof.
Consider a continuous linear operator f : H → C. The kernel K of f , i.e., K = {x : f (x ) = 0} is a closed
subspace of H since f is continuous. Consider a nonzero element x0 ∈ K ⊥ (if no such element exists,
we can take xf = 0). For any x ∈ H, the decomposition
   
f (x ) f (x )
x= x− f (x0 )
x0 + f (x0 )
x0

f (x )
is the unique decomposition of x into K ⊥ and K since x − x
f (x 0 ) 0
∈ K . So:

⟨x0 ,x0 ⟩
D E D E
x, f (x0 )
∥x0 ∥2
x0 = f (x )
f (x0 )
x0 , f (x0 )
∥x0 ∥2
x0 = f (x ) ff ((xx0 )) ∥x0 ∥2
= f (x ).
0

f (x0 )
It follows f (x ) = ⟨x , xf ⟩ with xf = ∥x0 ∥2 0
x .
If there exists another xf̃ with f (x ) = ⟨x , xf̃ ⟩, then ⟨x , xf − xf̃ ⟩ = 0 ∀x ∈ H. In particular,
⟨xf − xf̃ , xf − xf̃ ⟩ = 0, which implies xf = xf̃ .
The converse part of the theorem follows readily from the inequality

| ⟨x , y ⟩ |≤∥ x ∥ ∥ y ∥ ∀x , y ∈ H . □
A. Aytuna RKHS 2024 19 / 58
 Hilbert Spaces Consequences of the Main Theorem

DISCUSSION
Now, given a continuous linear operator T : H1 → H2 , define an operator T ∗ : H2 → H1 as follows:
For any y ∈ H2 , T ∗ y is the unique element of H1 such that for all x ∈ H1

⟨Tx , y ⟩ = ⟨x , T ∗ y ⟩. (3)

Such a T ∗ y exists because the assignment x → ⟨Tx , y ⟩ is a continuous linear operator from H1 into C
(since | ⟨Tx , y ⟩ |≤∥ Tx ∥ ∥ y ∥≤ c ∥ x ∥ ∥ y ∥ for some c > 0). So in view of the Riesz Representation
Theorem there exists unique T ∗ y such that the Equation (3) above holds. This assignment is certainly
linear and continuous:

⟨x , T ∗ (y1 + λy2 )⟩ = ⟨Tx , y1 + λy2 ⟩ = ⟨Tx , y1 ⟩ + λ⟨Tx , y2 ⟩

= ⟨x , T ∗ y1 ⟩ + λ⟨x , T ∗ y2 ⟩ = ⟨x , T ∗ y1 + λT ∗ y2 ⟩ ∀x ;

∥ T ∗ y ∥2 = | ⟨T ∗ y , T ∗ y ⟩ | = | ⟨TT ∗ y , y ⟩ | ≤ c ∥ T ∗ y ∥ ∥ y ∥,
∥ T ∗y ∥ ≤ c∥y∥.

Definition
T ∗ : H2 → H1 is a continuous linear operator and is called the adjoint of T .
If T = T ∗ (defined on H = H1 = H2 ), then the operator is called self-adjoint. (Note that the projections
in the Main Theorem are self adjoint.)
An operator U : H → H is called unitary in case UU ∗ = I. A unitary operator satisfies ⟨Ux , Uy ⟩ = ⟨x , y ⟩,
i.e., U preserves the inner product of elements.
A. Aytuna RKHS 2024 20 / 58
 Hilbert Spaces Consequences of the Main Theorem

DISCUSSION
Now, given a continuous linear operator T : H1 → H2 , define an operator T ∗ : H2 → H1 as follows:
For any y ∈ H2 , T ∗ y is the unique element of H1 such that for all x ∈ H1

⟨Tx , y ⟩ = ⟨x , T ∗ y ⟩. (3)

Such a T ∗ y exists because the assignment x → ⟨Tx , y ⟩ is a continuous linear operator from H1 into C
(since | ⟨Tx , y ⟩ |≤∥ Tx ∥ ∥ y ∥≤ c ∥ x ∥ ∥ y ∥ for some c > 0). So in view of the Riesz Representation
Theorem there exists unique T ∗ y such that the Equation (3) above holds. This assignment is certainly
linear and continuous:

⟨x , T ∗ (y1 + λy2 )⟩ = ⟨Tx , y1 + λy2 ⟩ = ⟨Tx , y1 ⟩ + λ⟨Tx , y2 ⟩

= ⟨x , T ∗ y1 ⟩ + λ⟨x , T ∗ y2 ⟩ = ⟨x , T ∗ y1 + λT ∗ y2 ⟩ ∀x ;

∥ T ∗ y ∥2 = | ⟨T ∗ y , T ∗ y ⟩ | = | ⟨TT ∗ y , y ⟩ | ≤ c ∥ T ∗ y ∥ ∥ y ∥,
∥ T ∗y ∥ ≤ c∥y∥.

Definition
T ∗ : H2 → H1 is a continuous linear operator and is called the adjoint of T .
If T = T ∗ (defined on H = H1 = H2 ), then the operator is called self-adjoint. (Note that the projections
in the Main Theorem are self adjoint.)
An operator U : H → H is called unitary in case UU ∗ = I. A unitary operator satisfies ⟨Ux , Uy ⟩ = ⟨x , y ⟩,
i.e., U preserves the inner product of elements.
A. Aytuna RKHS 2024 20 / 58
 Hilbert Spaces Consequences of the Main Theorem

Theorem

(0) For a subspace M ⊆ H, (M ⊥ )⊥ = M, where M is the closure of M.

For a continuous linear operator T : H1 → H2 :

(1) Ker (T ) = Range (T ∗ )⊥ (3) T ∗∗ = T

⊥
(2) Range (T ∗ ) = Ker (T )⊥ (4) Ker (T ∗ ) = Range (T )⊥ = Range (T ) .

Proof.
(0): Recall that M ⊥ = {x ∈ H : ⟨x , m⟩ = 0 ∀m ∈ M }.
Clearly M ⊆ (M ⊥ )⊥ and since (M ⊥ )⊥ is closed M ⊆ (M ⊥ )⊥ .
For the other side, first note that M ⊥ = (M )⊥ since

∀b ∈ M , ∃xn ∈ M , n = 1, 2, · · · such that xn → b .

So for a ∈ ⟨a , xn ⟩ → ⟨a , b ⟩ by continuity. Hence ⟨a , b ⟩ = 0.

M⊥ ,
Let α ∈ (M ⊥ )⊥ = (M )⊥⊥ . By the Main Theorem α = α1 + α2 , with α1 ∈ M, α2 ∈ (M )⊥ . So:

0 = ⟨α, α2 ⟩ = ⟨α1 , α2 ⟩+ ∥ α2 ∥2 =∥ α2 ∥2 ⇒ α = α1 ∈ M .

(1): ξ ∈ Ker (T ): ⟨ξ, T ∗ a ⟩ = ⟨T ξ, a ⟩ = 0 ⇒ Ker (T ) ⊆ R (T ∗ )⊥ .

ξ ∈ R (T ∗ )⊥ : ⟨T ξ, η⟩ = ⟨ξ, T ∗ η⟩ = 0 ∀n ∴ ξ ∈ Ker (T ).
(3): ⟨T ∗∗ x , y ⟩ = ⟨y , (T ∗ )∗ x ⟩ = ⟨T ∗ y , x ⟩ = ⟨y , Tx ⟩ = ⟨Tx , y ⟩ ⇒ T ∗∗ x = Tx ∀x. □

A. Aytuna RKHS 2024 21 / 58

 Hilbert Spaces Consequences of the Main Theorem

Theorem

(0) For a subspace M ⊆ H, (M ⊥ )⊥ = M, where M is the closure of M.

For a continuous linear operator T : H1 → H2 :

(1) Ker (T ) = Range (T ∗ )⊥ (3) T ∗∗ = T

⊥
(2) Range (T ∗ ) = Ker (T )⊥ (4) Ker (T ∗ ) = Range (T )⊥ = Range (T ) .

Proof.
(0): Recall that M ⊥ = {x ∈ H : ⟨x , m⟩ = 0 ∀m ∈ M }.
Clearly M ⊆ (M ⊥ )⊥ and since (M ⊥ )⊥ is closed M ⊆ (M ⊥ )⊥ .
For the other side, first note that M ⊥ = (M )⊥ since

∀b ∈ M , ∃xn ∈ M , n = 1, 2, · · · such that xn → b .

So for a ∈ ⟨a , xn ⟩ → ⟨a , b ⟩ by continuity. Hence ⟨a , b ⟩ = 0.

M⊥ ,
Let α ∈ (M ⊥ )⊥ = (M )⊥⊥ . By the Main Theorem α = α1 + α2 , with α1 ∈ M, α2 ∈ (M )⊥ . So:

0 = ⟨α, α2 ⟩ = ⟨α1 , α2 ⟩+ ∥ α2 ∥2 =∥ α2 ∥2 ⇒ α = α1 ∈ M .

(1): ξ ∈ Ker (T ): ⟨ξ, T ∗ a ⟩ = ⟨T ξ, a ⟩ = 0 ⇒ Ker (T ) ⊆ R (T ∗ )⊥ .

ξ ∈ R (T ∗ )⊥ : ⟨T ξ, η⟩ = ⟨ξ, T ∗ η⟩ = 0 ∀n ∴ ξ ∈ Ker (T ).
(3): ⟨T ∗∗ x , y ⟩ = ⟨y , (T ∗ )∗ x ⟩ = ⟨T ∗ y , x ⟩ = ⟨y , Tx ⟩ = ⟨Tx , y ⟩ ⇒ T ∗∗ x = Tx ∀x. □

A. Aytuna RKHS 2024 21 / 58

 Hilbert Spaces DIGRESSION: Existence of Solutions

We will look at the equation Tu = f for T : H1 → H2 continuous linear operator and f , a given element of
H2 .

Given T , H1 , H2 and f ∈ H2 as above, consider the condition:

∃C > 0 : | ⟨f , x ⟩ |≤ C ∥ T ∗ x ∥ ∀x ∈ H2 (**)

If the equation Tu = f has a solution, then for any x ∈ H2 :

| ⟨f , x ⟩ |=| ⟨Tu, x ⟩ |=| ⟨u, T ∗ x ⟩ |≤∥ u ∥ ∥ T ∗ x ∥ .

So (**) is satisfied with C ≥∥ u ∥.

On the other hand, if (**) is satisfied then on R (T ∗ ) define an operator S via S (T ∗ v ) := ⟨v , f ⟩. This
assignment is well-defined since if T ∗ v1 = T ∗ v2 then (**) implies

| ⟨f , v1 ⟩ − ⟨f , v2 ⟩ |≤ C ∥ T ∗ v1 − T ∗ v2 ∥= 0 ⇒ ⟨v1 , f ⟩ = ⟨v2 , f ⟩.

Moreover by (**) we have | S (T ∗ v ) |≤ C ∥ T ∗ v ∥, so S is continuous and linear on R (T ∗ ).

As a general rule, S extends to R (T ∗ ) by defining the extension for a given x, by

S (x ) := lim S (xn )
n→∞

for some sequence {xn } ∈ R (T ∗ ) that converges to x.

A. Aytuna RKHS 2024 22 / 58

 Hilbert Spaces DIGRESSION: Existence of Solutions

We will look at the equation Tu = f for T : H1 → H2 continuous linear operator and f , a given element of
H2 .

Given T , H1 , H2 and f ∈ H2 as above, consider the condition:

∃C > 0 : | ⟨f , x ⟩ |≤ C ∥ T ∗ x ∥ ∀x ∈ H2 (**)

If the equation Tu = f has a solution, then for any x ∈ H2 :

| ⟨f , x ⟩ |=| ⟨Tu, x ⟩ |=| ⟨u, T ∗ x ⟩ |≤∥ u ∥ ∥ T ∗ x ∥ .

So (**) is satisfied with C ≥∥ u ∥.

On the other hand, if (**) is satisfied then on R (T ∗ ) define an operator S via S (T ∗ v ) := ⟨v , f ⟩. This
assignment is well-defined since if T ∗ v1 = T ∗ v2 then (**) implies

| ⟨f , v1 ⟩ − ⟨f , v2 ⟩ |≤ C ∥ T ∗ v1 − T ∗ v2 ∥= 0 ⇒ ⟨v1 , f ⟩ = ⟨v2 , f ⟩.

Moreover by (**) we have | S (T ∗ v ) |≤ C ∥ T ∗ v ∥, so S is continuous and linear on R (T ∗ ).

As a general rule, S extends to R (T ∗ ) by defining the extension for a given x, by

S (x ) := lim S (xn )
n→∞

for some sequence {xn } ∈ R (T ∗ ) that converges to x.

A. Aytuna RKHS 2024 22 / 58

 Hilbert Spaces DIGRESSION: Existence of Solutions
To see that this procedure does not depend upon the sequence chosen, suppose xn′ → x is another
sequence in R (T ∗ ). Then since
∥ S (xn − xn′ ) ∥≤ C ∥ xn − xn′ ∥,
{S (xn )} and {S (xn′ )} converges to the same point in C, and since

| S (xn ) − S (xm ) |≤ C ∥ xn − xm ∥ ∀n, m,

plainly S (xn ) converges.

Moreover ∀n,
| S (xn ) |≤ C ∥ xn ∥ ⇒ ∥ Sx ∥≤ C ∥ x ∥ .
That is, S is continuous on R (T ∗ ).
Let P be the projection on R (T ∗ ) and consider S ◦ P (x ). This is a continuous linear function from H1
into C with
| S ◦ P (x ) |≤ C ∥ P (x ) ∥≤ C ∥ x ∥ .
So by Riesz Representation Theorem ∃u ∈ H2 that satisfies

S ◦ P (x ) = ⟨x , u⟩ ∀x ∈ H1 .

In particular for x = T ∗ v we have

⟨v , f ⟩ = ⟨T ∗ v , u⟩ = ⟨v , Tu⟩, or
⟨f , v ⟩ = ⟨Tu, v ⟩, ∀v ⇒ f = Tu

Moreover we have an estimate on u, namely ∥ u ∥≤ C.

A. Aytuna RKHS 2024 23 / 58
 Hilbert Spaces DIGRESSION: Existence of Solutions
To see that this procedure does not depend upon the sequence chosen, suppose xn′ → x is another
sequence in R (T ∗ ). Then since
∥ S (xn − xn′ ) ∥≤ C ∥ xn − xn′ ∥,
{S (xn )} and {S (xn′ )} converges to the same point in C, and since

| S (xn ) − S (xm ) |≤ C ∥ xn − xm ∥ ∀n, m,

plainly S (xn ) converges.

Moreover ∀n,
| S (xn ) |≤ C ∥ xn ∥ ⇒ ∥ Sx ∥≤ C ∥ x ∥ .
That is, S is continuous on R (T ∗ ).
Let P be the projection on R (T ∗ ) and consider S ◦ P (x ). This is a continuous linear function from H1
into C with
| S ◦ P (x ) |≤ C ∥ P (x ) ∥≤ C ∥ x ∥ .
So by Riesz Representation Theorem ∃u ∈ H2 that satisfies

S ◦ P (x ) = ⟨x , u⟩ ∀x ∈ H1 .

In particular for x = T ∗ v we have

⟨v , f ⟩ = ⟨T ∗ v , u⟩ = ⟨v , Tu⟩, or
⟨f , v ⟩ = ⟨Tu, v ⟩, ∀v ⇒ f = Tu

Moreover we have an estimate on u, namely ∥ u ∥≤ C.

A. Aytuna RKHS 2024 23 / 58
 Hilbert Spaces DIGRESSION: Existence of Solutions

To summarize:

For a given f ∈ H2 the equation T (u) = f has a solution if and only if (**) holds.

In practice, sometimes explicit knowledge of T ∗ allows one to get a stronger estimate:

∃C > 0 : ∥ x ∥≤ C ∥ T ∗ x ∥ ∀x ∈ R (T ). (***)

This yields for f ∈ R (T ) and x ∈ R (T ):

| ⟨f , x ⟩ |≤∥ f ∥ ∥ x ∥≤ C ∥ f ∥ ∥ T ∗ x ∥,

which implies
∀f ∈ R (T ) ∃u with Tu = f and ∥ u ∥≤ C ∥ f ∥ .

In particular, this implies that R (T ) is closed.

A. Aytuna RKHS 2024 24 / 58

 Hilbert Spaces DIGRESSION: Existence of Solutions

To summarize:

For a given f ∈ H2 the equation T (u) = f has a solution if and only if (**) holds.

In practice, sometimes explicit knowledge of T ∗ allows one to get a stronger estimate:

∃C > 0 : ∥ x ∥≤ C ∥ T ∗ x ∥ ∀x ∈ R (T ). (***)

This yields for f ∈ R (T ) and x ∈ R (T ):

| ⟨f , x ⟩ |≤∥ f ∥ ∥ x ∥≤ C ∥ f ∥ ∥ T ∗ x ∥,

which implies
∀f ∈ R (T ) ∃u with Tu = f and ∥ u ∥≤ C ∥ f ∥ .

In particular, this implies that R (T ) is closed.

A. Aytuna RKHS 2024 24 / 58

 Hilbert Spaces Orthonormal Bases

We will close this part by generalizing a special feature of the space ℓ2 to a large class of Hilbert spaces.

Recall that:
∞
X
ℓ2 := {(ζn ) ∈ CN : | ζn |2 < ∞},
n =1

with inner product

∞
X
⟨ζ, η⟩ := ζn ηn , ζ = (ζn ), η = (ηn ).
n =1

Note that ℓ2 = span{en }, the closure of all finite linear combinations of en ’s, where

en := (0, · · · , 0, 1, 0, · · · ),

since for any ζ = (ζn ) ∈ ℓ2 ,

N
X ∞
X
∥ζ− ζn en ∥= | ζn |2 → 0 as N → ∞.
n =1 n=N +1

We will call a Hilbert space H separable in case there exists a countable set of elements such that the
closure of the span of these elements is H.

A. Aytuna RKHS 2024 25 / 58

 Hilbert Spaces Orthonormal Bases

We will close this part by generalizing a special feature of the space ℓ2 to a large class of Hilbert spaces.

Recall that:
∞
X
ℓ2 := {(ζn ) ∈ CN : | ζn |2 < ∞},
n =1

with inner product

∞
X
⟨ζ, η⟩ := ζn ηn , ζ = (ζn ), η = (ηn ).
n =1

Note that ℓ2 = span{en }, the closure of all finite linear combinations of en ’s, where

en := (0, · · · , 0, 1, 0, · · · ),

since for any ζ = (ζn ) ∈ ℓ2 ,

N
X ∞
X
∥ζ− ζn en ∥= | ζn |2 → 0 as N → ∞.
n =1 n=N +1

We will call a Hilbert space H separable in case there exists a countable set of elements such that the
closure of the span of these elements is H.

A. Aytuna RKHS 2024 25 / 58

 Hilbert Spaces Orthonormal Bases

Note that in ℓ2 every element ζ can be expressed as:

∞
X
ζ= ⟨ζn , en ⟩en ,
n =1

the series converging in ℓ2 .

Now, if H is a separable Hilbert space with span{xn } = H, by Gram-Schmidt algorithm we can get
another sequence (yn ) with span{yn } = H and ⟨yn , ym ⟩ = δn,m .

For a given x ∈ H form:

N
X
xN := ⟨x , yn ⟩yn , N ∈ N.
n =1

Since ⟨x − xN , yi ⟩ = 0 for i = 1, · · · , N;
⊥
x − xN ∈ span{y1 , · · · , yN }⊥ = span{y1 , · · · , yN } .

Then x = (x − xN ) + xN is the unique decomposition of x given by the Main Theorem.

In particular:
⟨x − xN + xN , x − xN + xN ⟩ =∥ x ∥2 =∥ x − xN ∥2 + ∥ xN ∥2 .

A. Aytuna RKHS 2024 26 / 58

 Hilbert Spaces Orthonormal Bases

Note that in ℓ2 every element ζ can be expressed as:

∞
X
ζ= ⟨ζn , en ⟩en ,
n =1

the series converging in ℓ2 .

Now, if H is a separable Hilbert space with span{xn } = H, by Gram-Schmidt algorithm we can get
another sequence (yn ) with span{yn } = H and ⟨yn , ym ⟩ = δn,m .

For a given x ∈ H form:

N
X
xN := ⟨x , yn ⟩yn , N ∈ N.
n =1

Since ⟨x − xN , yi ⟩ = 0 for i = 1, · · · , N;
⊥
x − xN ∈ span{y1 , · · · , yN }⊥ = span{y1 , · · · , yN } .

Then x = (x − xN ) + xN is the unique decomposition of x given by the Main Theorem.

In particular:
⟨x − xN + xN , x − xN + xN ⟩ =∥ x ∥2 =∥ x − xN ∥2 + ∥ xN ∥2 .

A. Aytuna RKHS 2024 26 / 58

 Hilbert Spaces Orthonormal Bases

So we can draw two conclusions from this and the Main Theorem:
1) ∥ x − xN ∥2 = distance of x to span{x1 , · · · , xN } → 0 as N → ∞ by our assumption,
| ⟨x , yn ⟩ |2 ≤∥ x ∥2 , so {⟨x , yn ⟩} ∈ ℓ2 .
PN
2) ∥ xN ∥2 = n =1

It follows that every x ∈ H can be expanded in H as:

∞
X
x= ⟨x , yn ⟩yn with {⟨x , yn ⟩} ∈ ℓ2 .
n =1

Conversely, if {λn } is in ℓ2 , the series λn yn converges in H, since (if N < M)

P∞
n =1

N
X M
X 2 M
X 2 M
X
λn yn − λn yn = λ n yn = | λn |2 → 0 as N , M → ∞.
n =1 n =1 n =N +1 n =N +1

λn yn converges to a point ζ in H. In particular, for each ys ,

PN
So n =1

N
X 
λn yn , ys = λs if N ≥ s , so ⟨ζ, ys ⟩ = λs .
n =1

So ζ = λn yn and λn = ⟨ζ, yn ⟩, n = 1, · · · .
P∞
n =1

A. Aytuna RKHS 2024 27 / 58

 Hilbert Spaces Orthonormal Bases

So we can draw two conclusions from this and the Main Theorem:
1) ∥ x − xN ∥2 = distance of x to span{x1 , · · · , xN } → 0 as N → ∞ by our assumption,
| ⟨x , yn ⟩ |2 ≤∥ x ∥2 , so {⟨x , yn ⟩} ∈ ℓ2 .
PN
2) ∥ xN ∥2 = n =1

It follows that every x ∈ H can be expanded in H as:

∞
X
x= ⟨x , yn ⟩yn with {⟨x , yn ⟩} ∈ ℓ2 .
n =1

Conversely, if {λn } is in ℓ2 , the series λn yn converges in H, since (if N < M)

P∞
n =1

N
X M
X 2 M
X 2 M
X
λn yn − λn yn = λ n yn = | λn |2 → 0 as N , M → ∞.
n =1 n =1 n =N +1 n =N +1

λn yn converges to a point ζ in H. In particular, for each ys ,

PN
So n =1

N
X 
λn yn , ys = λs if N ≥ s , so ⟨ζ, ys ⟩ = λs .
n =1

So ζ = λn yn and λn = ⟨ζ, yn ⟩, n = 1, · · · .
P∞
n =1

A. Aytuna RKHS 2024 27 / 58

 Hilbert Spaces Orthonormal Bases

To summarize
For a given separable H, ∃{yn }∞
n =1
with ⟨yn , ym ⟩ = δn,m such that every x ∈ H can be expanded uniquely
in a series
∞
X
x= λn yn , {λn }n ∈ ℓ2 , λn = ⟨x , yn ⟩, n = 1, 2, · · · .
n =1

Such a sequence {yn } will be referred as an orthonormal basis.

Continuing our discussion, it follows that there is an operator T : H → ℓ2 , Tx := {⟨x , yn ⟩}n that is one to
one and onto. Moreover, for f , g ∈ H:

N
X  N
X ∞
X
⟨f , g ⟩ = lim ⟨f , yn ⟩yn , g = lim ⟨f , yn ⟩⟨g , yn ⟩ = ⟨f , yn ⟩⟨g , yn ⟩.
N →∞ N →∞
n =1 n =1 n =1

The last equality is valid since if N < M

N
X M
X M
X
⟨f , yn ⟩⟨g , yn ⟩ − ⟨f , yn ⟩⟨g , yn ⟩ ≤ ⟨f , yn ⟩⟨g , yn ⟩
n =1 n =1 n =N +1
M
 X M
2  X 1/2
≤ | ⟨f , yn ⟩ |1/2 | ⟨g , yn ⟩ |2
n=N +1 n =N +1

→ 0 as N , M → ∞.

A. Aytuna RKHS 2024 28 / 58

 Hilbert Spaces Orthonormal Bases

To summarize
For a given separable H, ∃{yn }∞
n =1
with ⟨yn , ym ⟩ = δn,m such that every x ∈ H can be expanded uniquely
in a series
∞
X
x= λn yn , {λn }n ∈ ℓ2 , λn = ⟨x , yn ⟩, n = 1, 2, · · · .
n =1

Such a sequence {yn } will be referred as an orthonormal basis.

Continuing our discussion, it follows that there is an operator T : H → ℓ2 , Tx := {⟨x , yn ⟩}n that is one to
one and onto. Moreover, for f , g ∈ H:

N
X  N
X ∞
X
⟨f , g ⟩ = lim ⟨f , yn ⟩yn , g = lim ⟨f , yn ⟩⟨g , yn ⟩ = ⟨f , yn ⟩⟨g , yn ⟩.
N →∞ N →∞
n =1 n =1 n =1

The last equality is valid since if N < M

N
X M
X M
X
⟨f , yn ⟩⟨g , yn ⟩ − ⟨f , yn ⟩⟨g , yn ⟩ ≤ ⟨f , yn ⟩⟨g , yn ⟩
n =1 n =1 n =N +1
M
 X M
2  X 1/2
≤ | ⟨f , yn ⟩ |1/2 | ⟨g , yn ⟩ |2
n =N +1 n =N +1

→ 0 as N , M → ∞.

A. Aytuna RKHS 2024 28 / 58

 Hilbert Spaces Orthonormal Bases

Therefore
∞
X 1/2
∥ T (h ) ∥= | ⟨h , yn ⟩ |2 =∥ h ∥ .
n =1

More generally;
⟨T (h ), T (g )⟩ = ⟨f , g ⟩.
That is, T is unitary isomorphism from H onto ℓ2 .

The moral of the story is

In a separable Hilbert space H, one can introduce "coordinates" in ℓ2 just like in Cn one introduces
x ↔ (x1 , · · · , xn ), xn ∈ C and work with these coordinates.

A. Aytuna RKHS 2024 29 / 58

 Hilbert Spaces Orthonormal Bases

Therefore
∞
X 1/2
∥ T (h ) ∥= | ⟨h , yn ⟩ |2 =∥ h ∥ .
n =1

More generally;
⟨T (h ), T (g )⟩ = ⟨f , g ⟩.
That is, T is unitary isomorphism from H onto ℓ2 .

The moral of the story is

In a separable Hilbert space H, one can introduce "coordinates" in ℓ2 just like in Cn one introduces
x ↔ (x1 , · · · , xn ), xn ∈ C and work with these coordinates.

A. Aytuna RKHS 2024 29 / 58

 Hilbert Spaces Completion
Let (X , ⟨ , ⟩) be an inner product space that is not complete. This inner product induces a metric on X as
we have seen earlier via,
∥ x − y ∥2 := ⟨x − y , x − y ⟩.
Let us call a function f : X → C anti-linear in case f (x + λy ) = f (x ) + λf (y ) for every x , y ∈ X and λ ∈ C.
Consider
X ∗ = {f : X → C : f is anti-linear and continuous}
= {f : X → C : f is anti-linear and ∃C > 0 s.t. | f (x ) |≤ C ∥ x ∥}.
Note that X ∗ is a subspace of the vector space of complex valued functions on X .
We can identify elements of X with a subset of X ∗ via x 7→ fx , fx (y ) := ⟨x , y ⟩. Note that this is a one to
one and linear assignment, i.e.,
fx1 = fx2 =⇒ ⟨y , x1 − x2 ⟩ = 0 ∀y =⇒ x1 = x2 ,
and
fx +ty = fx + tfy for x , y ∈ X , t ∈ C.
Moreover
sup | fx (y ) |=∥ x ∥ ∀x ∈ X .
∥y ∥≤1
Choose a Cauchy sequence {xn } in X , then since
| fxn (t ) − fxm (t ) |=| ⟨t , xn − xm ⟩ |≤∥ xn − xm ∥ ∥ t ∥ ∀t ∈ X ,
fxn (t ) converges to a point in C as n → ∞. Call this point f (t ).
The function t 7→ f (t ) is clearly anti-linear since each fxn is, n = 1, 2, · · · .
A. Aytuna RKHS 2024 30 / 58
 Hilbert Spaces Completion
Let (X , ⟨ , ⟩) be an inner product space that is not complete. This inner product induces a metric on X as
we have seen earlier via,
∥ x − y ∥2 := ⟨x − y , x − y ⟩.
Let us call a function f : X → C anti-linear in case f (x + λy ) = f (x ) + λf (y ) for every x , y ∈ X and λ ∈ C.
Consider
X ∗ = {f : X → C : f is anti-linear and continuous}
= {f : X → C : f is anti-linear and ∃C > 0 s.t. | f (x ) |≤ C ∥ x ∥}.
Note that X ∗ is a subspace of the vector space of complex valued functions on X .
We can identify elements of X with a subset of X ∗ via x 7→ fx , fx (y ) := ⟨x , y ⟩. Note that this is a one to
one and linear assignment, i.e.,
fx1 = fx2 =⇒ ⟨y , x1 − x2 ⟩ = 0 ∀y =⇒ x1 = x2 ,
and
fx +ty = fx + tfy for x , y ∈ X , t ∈ C.
Moreover
sup | fx (y ) |=∥ x ∥ ∀x ∈ X .
∥y ∥≤1
Choose a Cauchy sequence {xn } in X , then since
| fxn (t ) − fxm (t ) |=| ⟨t , xn − xm ⟩ |≤∥ xn − xm ∥ ∥ t ∥ ∀t ∈ X ,
fxn (t ) converges to a point in C as n → ∞. Call this point f (t ).
The function t 7→ f (t ) is clearly anti-linear since each fxn is, n = 1, 2, · · · .
A. Aytuna RKHS 2024 30 / 58
 Hilbert Spaces Completion

Moreover;
| f (t ) |≤ lim | fxn (t ) |≤ lim | ⟨xn , t ⟩ |≤∥ t ∥ ∥ xn ∥ .
n→∞ n→∞

Since ∀ε > 0∃N : ∥ xn − xm ∥≤ ε if n, m ≥ N,

∥ xn ∥≤∥ xn − xN ∥ + ∥ xN ∥≤ ε+ ∥ xN ∥= C .

So f ∈ X ∗.

Note that a similar argument presented above shows that X ∗ with the metric

d (f , g ) = sup | f (y ) − g (y ) |
∥y ∥≤1

is complete, that is, every Cauchy sequence in X ∗ converges.

Note that ∥ x ∥= d (0, fx ).

Now consider the closure of X in X ∗ . For an f ∈ X ∗ , define

⟨f , fx ⟩ := f (x ), x ∈ X.

Certainly, it is linear in f and anti-linear in fx ’s, that is ⟨f , cfx ⟩ = c ⟨f , fx ⟩ for c ∈ C.

For f = fy , ⟨fy , fx ⟩ = fy (x ) = ⟨y , x ⟩.

For a g ∈ X , choose sequences {xn } and {yn } such that fxn → g and fyn → g in X ∗ .

A. Aytuna RKHS 2024 31 / 58

 Hilbert Spaces Completion

Moreover;
| f (t ) |≤ lim | fxn (t ) |≤ lim | ⟨xn , t ⟩ |≤∥ t ∥ ∥ xn ∥ .
n→∞ n→∞

Since ∀ε > 0∃N : ∥ xn − xm ∥≤ ε if n, m ≥ N,

∥ xn ∥≤∥ xn − xN ∥ + ∥ xN ∥≤ ε+ ∥ xN ∥= C .

So f ∈ X ∗.

Note that a similar argument presented above shows that X ∗ with the metric

d (f , g ) = sup | f (y ) − g (y ) |
∥y ∥≤1

is complete, that is, every Cauchy sequence in X ∗ converges.

Note that ∥ x ∥= d (0, fx ).

Now consider the closure of X in X ∗ . For an f ∈ X ∗ , define

⟨f , fx ⟩ := f (x ), x ∈ X.

Certainly, it is linear in f and anti-linear in fx ’s, that is ⟨f , cfx ⟩ = c ⟨f , fx ⟩ for c ∈ C.

For f = fy , ⟨fy , fx ⟩ = fy (x ) = ⟨y , x ⟩.

For a g ∈ X , choose sequences {xn } and {yn } such that fxn → g and fyn → g in X ∗ .

A. Aytuna RKHS 2024 31 / 58

 Hilbert Spaces Completion
Then for f ∈ X there exist a C > 0 such that,
| ⟨f , fxn − fyn ⟩ | =| ⟨f , fxn ⟩ − ⟨f , fyn ⟩ |=| f (xn ) − f (yn ) |
=| f (xn − yn ) |≤ C ∥ xn − yn ∥= Cd (fxn , fyn ),
and
| ⟨f , fxn ⟩ − ⟨f , fxm ⟩ | =| ⟨f , fxn − fxm ⟩ |=| f (xn − xm ) |
≤ C ∥ xn − xm ∥≤ Cd (fxn , fxm ).
Hence we can define
⟨f , g ⟩ := lim ⟨f , fxn ⟩, for fxn → g ,
n→∞
and this definition does not depend upon the sequence fxn chosen.
Clearly this assignment is sesquilinear.
Note that ⟨f , f ⟩ = limn→∞ ⟨f , fxn ⟩ for a sequence fxn → f in X ∗ :
| fxn (xn ) − f (xn ) | ≤∥ xn ∥ d (fxn , f ) = d (0, fxn )d (fxn , f )
≤ d (fxn , f )2 + d (0, f )d (fxn , f ).
So limn→∞ | fxn (xn ) − f (xn ) |= 0. Observe:
f (xn ) = −fxn (xn ) + f (xn ) + fxn (xn ) = f (xn ) − fxn (xn ) + ⟨xn , xn ⟩.
So ⟨f , f ⟩ = limn→∞ f (xn ) ≥ 0 and is zero if ⟨xn , xn ⟩ → 0, which in view of
| f (y ) |= lim | fxn (y ) |= lim | ⟨y , xn ⟩ |≤∥ y ∥ ∥ xn ∥,
n→∞ n→∞

implies f ≡ 0.
A. Aytuna RKHS 2024 32 / 58
 Hilbert Spaces Completion
Then for f ∈ X there exist a C > 0 such that,
| ⟨f , fxn − fyn ⟩ | =| ⟨f , fxn ⟩ − ⟨f , fyn ⟩ |=| f (xn ) − f (yn ) |
=| f (xn − yn ) |≤ C ∥ xn − yn ∥= Cd (fxn , fyn ),
and
| ⟨f , fxn ⟩ − ⟨f , fxm ⟩ | =| ⟨f , fxn − fxm ⟩ |=| f (xn − xm ) |
≤ C ∥ xn − xm ∥≤ Cd (fxn , fxm ).
Hence we can define
⟨f , g ⟩ := lim ⟨f , fxn ⟩, for fxn → g ,
n→∞
and this definition does not depend upon the sequence fxn chosen.
Clearly this assignment is sesquilinear.
Note that ⟨f , f ⟩ = limn→∞ ⟨f , fxn ⟩ for a sequence fxn → f in X ∗ :
| fxn (xn ) − f (xn ) | ≤∥ xn ∥ d (fxn , f ) = d (0, fxn )d (fxn , f )
≤ d (fxn , f )2 + d (0, f )d (fxn , f ).
So limn→∞ | fxn (xn ) − f (xn ) |= 0. Observe:
f (xn ) = −fxn (xn ) + f (xn ) + fxn (xn ) = f (xn ) − fxn (xn ) + ⟨xn , xn ⟩.
So ⟨f , f ⟩ = limn→∞ f (xn ) ≥ 0 and is zero if ⟨xn , xn ⟩ → 0, which in view of
| f (y ) |= lim | fxn (y ) |= lim | ⟨y , xn ⟩ |≤∥ y ∥ ∥ xn ∥,
n→∞ n→∞

implies f ≡ 0.
A. Aytuna RKHS 2024 32 / 58
 Hilbert Spaces Completion

It follows that ⟨ , ⟩ is an inner product on X .

Moreover, if {fn } is a Cauchy sequence in X with respect to the topology coming from this inner product,
since for x ∈ X :

| fn (x ) − fm (x ) | ≤| ⟨fn − fm , fx ⟩ |≤| ⟨fn − fm , fn − fm ⟩ |1/2 | ⟨fx , fx ⟩ |1/2

≤ ⟨fn − fm , fn − fm ⟩1/2 ⟨x , x ⟩1/2 ,
d (fn , fm ) = sup | fn − fm |≤ ⟨fn − fm , fn − fm ⟩1/2 ,
∥x ∥≤1

{fn } is a Cauchy sequence in X with respect to the original topology of X .

Since X is complete fn → f in this topology.

On the other hand, for a given g ∈ X , choosing fxk → g in X ∗ we have:

| ⟨g , g ⟩ | = lim | ⟨g , fxk ⟩ |= lim | g (xk ) |

k →∞ k →∞

≤ lim d (0, g ) ∥ xk ∥= lim d (0, g ) d (0, fxk ) = d (0, g )2 .

k →∞ k →∞

So
| ⟨fn − fm , fn − fm ⟩ |≤ d (fn , f )2 .
This implies that fn → f in (X , ⟨ , ⟩).

A. Aytuna RKHS 2024 33 / 58

 Hilbert Spaces Completion

It follows that ⟨ , ⟩ is an inner product on X .

Moreover, if {fn } is a Cauchy sequence in X with respect to the topology coming from this inner product,
since for x ∈ X :

| fn (x ) − fm (x ) | ≤| ⟨fn − fm , fx ⟩ |≤| ⟨fn − fm , fn − fm ⟩ |1/2 | ⟨fx , fx ⟩ |1/2

≤ ⟨fn − fm , fn − fm ⟩1/2 ⟨x , x ⟩1/2 ,
d (fn , fm ) = sup | fn − fm |≤ ⟨fn − fm , fn − fm ⟩1/2 ,
∥x ∥≤1

{fn } is a Cauchy sequence in X with respect to the original topology of X .

Since X is complete fn → f in this topology.

On the other hand, for a given g ∈ X , choosing fxk → g in X ∗ we have:

| ⟨g , g ⟩ | = lim | ⟨g , fxk ⟩ |= lim | g (xk ) |

k →∞ k →∞

≤ lim d (0, g ) ∥ xk ∥= lim d (0, g ) d (0, fxk ) = d (0, g )2 .

k →∞ k →∞

So
| ⟨fn − fm , fn − fm ⟩ |≤ d (fn , f )2 .
This implies that fn → f in (X , ⟨ , ⟩).

A. Aytuna RKHS 2024 33 / 58

 Hilbert Spaces Completion

So (X , ⟨ , ⟩) is a Hilbert space, it contains (X , ⟨ , ⟩), ⟨ , ⟩ induces the inner product on X , moreover, the
closure of X is the full space X .
If (H , ⟨⟨ , ⟩⟩) is another Hilbert space enjoining the above mentioned properties of (X , ⟨ , ⟩), then the
identity operator on (X , ⟨ , ⟩) plainly extends to a 1-1, onto unitary operator from X to H.

The unique Hilbert space satisfying the above mentioned properties is called the completion of (X , ⟨ , ⟩).

Now, going back to our construction, if (X , ⟨ , ⟩) is a vector space of functions on a set T where point
evaluations are continuous, then for fx , set fx (t ) := x (t ) and if fxn → f in (X , ⟨ , ⟩), then we propose to
set f (t ) = limn→∞ xn (t ).

Since
| xn (t ) − xm (t ) |≤ C ∥ xn − xm ∥= Cd (fxn , fxm ),
xn (t ) is Cauchy, so it converges.

If {x̃n } is another sequence such that fx̃n converges to f , the argument above shows that
xn (t ) − x̃n (t ) → 0; that is, fxn (t ) − fx̃n (t ) → 0, ∀t ∈ T . Therefore the assignment is well defined.

But does is characterize f completely? That is, if f (t ) ≡ 0 ∀t, does this mean that f ≡ 0?

For this, we need an extra condition.

If {xn } is Cauchy in X and lim xn (t ) = 0 ∀t ∈ T , then ∥ x ∥→ 0. (*)

n→∞

Condition (*) will give us the property we need.

A. Aytuna RKHS 2024 34 / 58

 Hilbert Spaces Completion

So (X , ⟨ , ⟩) is a Hilbert space, it contains (X , ⟨ , ⟩), ⟨ , ⟩ induces the inner product on X , moreover, the
closure of X is the full space X .
If (H , ⟨⟨ , ⟩⟩) is another Hilbert space enjoining the above mentioned properties of (X , ⟨ , ⟩), then the
identity operator on (X , ⟨ , ⟩) plainly extends to a 1-1, onto unitary operator from X to H.

The unique Hilbert space satisfying the above mentioned properties is called the completion of (X , ⟨ , ⟩).

Now, going back to our construction, if (X , ⟨ , ⟩) is a vector space of functions on a set T where point
evaluations are continuous, then for fx , set fx (t ) := x (t ) and if fxn → f in (X , ⟨ , ⟩), then we propose to
set f (t ) = limn→∞ xn (t ).

Since
| xn (t ) − xm (t ) |≤ C ∥ xn − xm ∥= Cd (fxn , fxm ),
xn (t ) is Cauchy, so it converges.

If {x̃n } is another sequence such that fx̃n converges to f , the argument above shows that
xn (t ) − x̃n (t ) → 0; that is, fxn (t ) − fx̃n (t ) → 0, ∀t ∈ T . Therefore the assignment is well defined.

But does is characterize f completely? That is, if f (t ) ≡ 0 ∀t, does this mean that f ≡ 0?

For this, we need an extra condition.

If {xn } is Cauchy in X and lim xn (t ) = 0 ∀t ∈ T , then ∥ x ∥→ 0. (*)

n→∞

Condition (*) will give us the property we need.

A. Aytuna RKHS 2024 34 / 58

 Hilbert Spaces Completion

On the other hand if f is completely determined by T then for any Cauchy sequence {xn } ∈ X , xn → x in
the closure, xn (t ) → 0 ∀t =⇒ x (t ) ≡ 0 =⇒ ∥ xn ∥→ 0.

So condition (*) is what we seek.

To Summarize
Given an inner product space (X0 , ⟨ , ⟩0 ), there exists a unique Hilbert space (X , ⟨ , ⟩) containing a copy
of X0 in the sense that ∃ı : X0 ,→ X one to one linear map that satisfies
⟨ı(x ), ı(y )⟩ = ⟨x , y ⟩0 , and
ı(X0 ) = X .

If (X0 , ⟨ , ⟩0 ) is a function space on T with continuous point evaluations, there exists a Hilbert function
space (X1 , ⟨ , ⟩1 ) on T with continuous point evaluations and satisfying the above conditions if and only if

{xn }n Cauchy in X0 , xn (t ) → 0 ∀t ∈ T =⇒ ∥ xn ∥= ⟨xn , xn ⟩10/2 → 0 as n → ∞ (*)

A. Aytuna RKHS 2024 35 / 58

 Hilbert Spaces Completion

On the other hand if f is completely determined by T then for any Cauchy sequence {xn } ∈ X , xn → x in
the closure, xn (t ) → 0 ∀t =⇒ x (t ) ≡ 0 =⇒ ∥ xn ∥→ 0.

So condition (*) is what we seek.

To Summarize
Given an inner product space (X0 , ⟨ , ⟩0 ), there exists a unique Hilbert space (X , ⟨ , ⟩) containing a copy
of X0 in the sense that ∃ı : X0 ,→ X one to one linear map that satisfies
⟨ı(x ), ı(y )⟩ = ⟨x , y ⟩0 , and
ı(X0 ) = X .

If (X0 , ⟨ , ⟩0 ) is a function space on T with continuous point evaluations, there exists a Hilbert function
space (X1 , ⟨ , ⟩1 ) on T with continuous point evaluations and satisfying the above conditions if and only if

{xn }n Cauchy in X0 , xn (t ) → 0 ∀t ∈ T =⇒ ∥ xn ∥= ⟨xn , xn ⟩10/2 → 0 as n → ∞ (*)

A. Aytuna RKHS 2024 35 / 58

 Hilbert Spaces Completion

Note that
Since we were interested in the existence of completion of an inner product X , we did not care about the
identification X in X ∗ .

Actually, one can show that X is in fact X ∗ as follows:

Take σ ∈ X ∗ . Then σ is a continuous linear functional on X , hence can be extended to a continuous

linear functional on X . Using the notation of the proof, Riesz Representation Theorem applied to the
Hilbert space (X , ⟨ , ⟩) gives an element η ∈ X such that:

σ(x ) = ⟨fx , η⟩ = ⟨η, fx ⟩ = η(x ) =⇒ σ ∈ X.

Moreover, the proof also shows that the norm ∥ . ∥ on X∗ is actually a Hilbertian norm, that is, it comes
from an inner product on X ∗ .

A. Aytuna RKHS 2024 36 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

In our previous discussions we have represented, from time to time, a given Hilbert space as a space of
functions on a set T with the property that point evaluations are continuous. Namely, we have
associated elements of a given Hilbert space (H , ⟨ , ⟩) to functions on the set H via the rule
H ∋ x ↔ x̂ (h ) := ⟨h , x ⟩, i.e., as continuous linear functions from H into C in view of Riesz
Representation Theorem. Transporting the inner product to this space of functions, i.e., setting
⟨x̂ , ŷ ⟩ := ⟨x , y ⟩ ∀x , y ∈ H, one can view H as a Hilbert space of functions such that point evaluations are
continuous.

The last assertion follows immediately from:

| x̂ (t ) |=| ⟨t , x ⟩ |≤∥ x ∥ ∥ t ∥, ∀x , t ∈ H .

However, the above realization is not unique. For example, one can also view Rn , or more generally ℓ2
as a space of functions on f : N → R with ∞ n=1 | f (n ) | < ∞, with the inner product:
P 2

∞
X
⟨f , g ⟩ = f (n)g (n),
n =1

via ℓ2 ∋ {an } = x ↔ fx : fx (n) = an ∀n.

Clearly;
∞
X 1/2
| f (n) |≤ | f (k ) |2 ,
k =1

so indeed point evaluations are continuous on this function space.

A. Aytuna RKHS 2024 37 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

In our previous discussions we have represented, from time to time, a given Hilbert space as a space of
functions on a set T with the property that point evaluations are continuous. Namely, we have
associated elements of a given Hilbert space (H , ⟨ , ⟩) to functions on the set H via the rule
H ∋ x ↔ x̂ (h ) := ⟨h , x ⟩, i.e., as continuous linear functions from H into C in view of Riesz
Representation Theorem. Transporting the inner product to this space of functions, i.e., setting
⟨x̂ , ŷ ⟩ := ⟨x , y ⟩ ∀x , y ∈ H, one can view H as a Hilbert space of functions such that point evaluations are
continuous.

The last assertion follows immediately from:

| x̂ (t ) |=| ⟨t , x ⟩ |≤∥ x ∥ ∥ t ∥, ∀x , t ∈ H .

However, the above realization is not unique. For example, one can also view Rn , or more generally ℓ2
as a space of functions on f : N → R with ∞ n=1 | f (n ) | < ∞, with the inner product:
P 2

∞
X
⟨f , g ⟩ = f (n)g (n),
n =1

via ℓ2 ∋ {an } = x ↔ fx : fx (n) = an ∀n.

Clearly;
∞
X 1/2
| f (n) |≤ | f (k ) |2 ,
k =1

so indeed point evaluations are continuous on this function space.

A. Aytuna RKHS 2024 37 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations
Naturally, it is desirable to represent a given Hilbert space as a function space on a small set.

On the other hand, some important Hilbert spaces that occur in nature are given as function spaces
with continuous point evaluations.

Example 1
Let X be the vector space of all infinitely differentiable real valued functions which vanish outside of a
finite interval and let Z ∞ Z ∞
⟨f , g ⟩ := (fg )(t )dt + (f ′ g ′ )(t )dt .
−∞ −∞
Then (X , ⟨ , ⟩) becomes an inner product space.

For a point t0 ∈ R and f ∈ X , by the Fundamental Theorem of Calculus,

Z t0 Z t0
| f (t0 ) |2 = (f 2 )′ dt = 2 f ′ fdt
−∞ −∞
Z ∞ 1/2  Z ∞ 1/2
≤2 | f ′ |2 dt | f |2 dt
−∞ −∞
Z ∞ Z ∞
≤ ′ 2
| f | dt + | f |2 dt .
−∞ −∞

Note that, we have used the inequality 2AB ≤ A 2 + B 2 for positive real numbers A and B.

So, | f (t0 ) |≤ ⟨f , f ⟩1/2 , hence point evaluations are continuous on X .

A. Aytuna RKHS 2024 38 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations
Naturally, it is desirable to represent a given Hilbert space as a function space on a small set.

On the other hand, some important Hilbert spaces that occur in nature are given as function spaces
with continuous point evaluations.

Example 1
Let X be the vector space of all infinitely differentiable real valued functions which vanish outside of a
finite interval and let Z ∞ Z ∞
⟨f , g ⟩ := (fg )(t )dt + (f ′ g ′ )(t )dt .
−∞ −∞
Then (X , ⟨ , ⟩) becomes an inner product space.

For a point t0 ∈ R and f ∈ X , by the Fundamental Theorem of Calculus,

Z t0 Z t0
| f (t0 ) |2 = (f 2 )′ dt = 2 f ′ fdt
−∞ −∞
Z ∞ 1/2  Z ∞ 1/2
≤2 | f ′ |2 dt | f |2 dt
−∞ −∞
Z ∞ Z ∞
≤ ′ 2
| f | dt + | f |2 dt .
−∞ −∞

Note that, we have used the inequality 2AB ≤ A 2 + B 2 for positive real numbers A and B.

So, | f (t0 ) |≤ ⟨f , f ⟩1/2 , hence point evaluations are continuous on X .

A. Aytuna RKHS 2024 38 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 1 (Cont.)
We would like to draw attention to two points in this context:
1) The first term in the above inner product is itself an inner product on X , but the point evaluations
are not necessarily continuous on this inner product space, as we have observed in a previous
example.
2) (X , ⟨ , ⟩) is not complete. However, it can be shown that it satisfies the condition (*) for having a
completion consisting of certain continuous functions on R with continuous point evaluations.
Hence, the completion of Cc∞ (R), W (R), is a Hilbert space of functions on R with continuous point
evaluations.
R ∞ of fact, W (R) consists
As a matter R∞ of continuous functions f on R, differentiable at "most" of the points of
R and −∞ | f (t ) |2 dt < ∞, −∞ | f ′ (t ) |2 dt < ∞ with a "reasonable" interpretation of the second integral.

Example 2
Let H 2 (D) denote the vector space of all analytic functions on the unit disc D ⊆ C whose Taylor
coefficients are in ℓ2 . In other words,

H 2 (D) := {f (z ) = | cn |2 < ∞}.

P∞ n P∞
n =1 c n z on D with n =1

On H 2 (D) we put the inner product:

⟨f , g ⟩ := n=1 cn dn , ,
P∞ P∞ n P∞
f (z ) = n =1 c n z g (z ) = n =1 dn z n ∈ H 2 (D).

A. Aytuna RKHS 2024 39 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 1 (Cont.)
We would like to draw attention to two points in this context:
1) The first term in the above inner product is itself an inner product on X , but the point evaluations
are not necessarily continuous on this inner product space, as we have observed in a previous
example.
2) (X , ⟨ , ⟩) is not complete. However, it can be shown that it satisfies the condition (*) for having a
completion consisting of certain continuous functions on R with continuous point evaluations.
Hence, the completion of Cc∞ (R), W (R), is a Hilbert space of functions on R with continuous point
evaluations.
R ∞ of fact, W (R) consists
As a matter R∞ of continuous functions f on R, differentiable at "most" of the points of
R and −∞ | f (t ) |2 dt < ∞, −∞ | f ′ (t ) |2 dt < ∞ with a "reasonable" interpretation of the second integral.

Example 2
Let H 2 (D) denote the vector space of all analytic functions on the unit disc D ⊆ C whose Taylor
coefficients are in ℓ2 . In other words,

H 2 (D) := {f (z ) = | cn |2 < ∞}.

P∞ n P∞
n =1 c n z on D with n =1

On H 2 (D) we put the inner product:

⟨f , g ⟩ := n=1 cn dn , ,
P∞ P∞ n P∞
f (z ) = n =1 c n z g (z ) = n =1 dn z n ∈ H 2 (D).

A. Aytuna RKHS 2024 39 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 2 (Cont.)
Clearly ⟨ , ⟩ defines an inner product on H 2 (D) and it makes it a Hilbert space, basically because ℓ2 is
complete.

k =1 ak z is Cauchy, then xn := {ak }k =1 , n = 1, 2, · · · is a Cauchy

Note that, if a sequence {fn }n , fn = ∞
P n k n ∞

sequence in ℓ2 , so converges to some x = {ak }∞ k =1

∈ ℓ 2 .

Now, f (z ) = ∞ n
P
k =1 ak z defines a function on D since on each subdisc ∆r , where
∆r := {z :| z |< r }, r < 1:
∞ n ∞ 1/2  1/2
1
X X X
| ak | | z n |≤ | ak | r n ≤ | ak |2 ,
k =1 k =1 k =1
1− r2
PN
and this function is analytic since it is the uniform limit of analytic polynomials k =1 ak z n on each
subdisc ∆r , r < 1.

For a w ∈ D and f ∈ H 2 (D),

∞ 1/2
1
X 
| f (w ) |≤ | an | | w |n ≤ ∥f ∥.
n =1
1− | w |2

So H 2 (D) is a Hilbert space of functions on the unit disc D with continuous point evaluations.

A. Aytuna RKHS 2024 40 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 2 (Cont.)
Clearly ⟨ , ⟩ defines an inner product on H 2 (D) and it makes it a Hilbert space, basically because ℓ2 is
complete.

k =1 ak z is Cauchy, then xn := {ak }k =1 , n = 1, 2, · · · is a Cauchy

Note that, if a sequence {fn }n , fn = ∞
P n k n ∞

sequence in ℓ2 , so converges to some x = {ak }∞ k =1

∈ ℓ 2 .

Now, f (z ) = ∞ n
P
k =1 ak z defines a function on D since on each subdisc ∆r , where
∆r := {z :| z |< r }, r < 1:
∞ n ∞ 1/2  1/2
1
X X X
| ak | | z n |≤ | ak | r n ≤ | ak |2 ,
k =1 k =1 k =1
1− r2
PN
and this function is analytic since it is the uniform limit of analytic polynomials k =1 ak z n on each
subdisc ∆r , r < 1.

For a w ∈ D and f ∈ H 2 (D),

∞ 1/2
1
X 
| f (w ) |≤ | an | | w |n ≤ ∥f ∥.
n =1
1− | w |2

So H 2 (D) is a Hilbert space of functions on the unit disc D with continuous point evaluations.

A. Aytuna RKHS 2024 40 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 3
Suppose T is any set (could be a set of humans for example), and suppose we somehow fabricate a
map ϕ from T to a Hilbert space (H , ⟨ , ⟩) not necessarily a RKHS.

Elaborating on the comment above, as an example, one can use the assignment of their
weight/height/birth year to a human in the set T , so ϕ from T to R3 with the usual inner product,
becomes a function.

This map induces a RKHS of functions on T by first considering:

H := spant ∈T ϕ(t ) ⊂ H ,

and forming:
h̃ (t ) := ⟨h , ϕ(t )⟩, for h ∈ H,
with the inner product:
⟨h̃1 , h̃2 ⟩ := ⟨h1 , h2 ⟩.

That is, we think of elements of H as functions on H and restrict them to the image of ϕ. Note that
h̃ (t ) ≡ 0 implies h ≡ 0 since h ∈ H .

A. Aytuna RKHS 2024 41 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 3
Suppose T is any set (could be a set of humans for example), and suppose we somehow fabricate a
map ϕ from T to a Hilbert space (H , ⟨ , ⟩) not necessarily a RKHS.

Elaborating on the comment above, as an example, one can use the assignment of their
weight/height/birth year to a human in the set T , so ϕ from T to R3 with the usual inner product,
becomes a function.

This map induces a RKHS of functions on T by first considering:

H := spant ∈T ϕ(t ) ⊂ H ,

and forming:
h̃ (t ) := ⟨h , ϕ(t )⟩, for h ∈ H,
with the inner product:
⟨h̃1 , h̃2 ⟩ := ⟨h1 , h2 ⟩.

That is, we think of elements of H as functions on H and restrict them to the image of ϕ. Note that
h̃ (t ) ≡ 0 implies h ≡ 0 since h ∈ H .

A. Aytuna RKHS 2024 41 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 3 (Cont.)
Another way of visualizing this example is by forming the function space

F = {f : T → C | ∃h ∈ H : f (t ) = ⟨h , ϕ(t )⟩ ∀t ∈ T }

and putting on F the norm

∥ f ∥:= inf ∥ h ∥,
h ∈H ,f (t )=⟨h ,ϕ(t )⟩

where the last norm is the norm in the Hilbert space H.

Note that with the notation above,

h ∈ H ⊥ ⇐⇒ ⟨h , ϕ(t )⟩ = 0 ∀t ∈ T .

It follows that for every f ∈ F there exits a unique hf ∈ H such that f (t ) = ⟨hf , ϕ(t )⟩ ∀t ∈ T , and if
f (t ) = ⟨h , ϕ(t )⟩ ∀t ∈ T , then h = h1 + hf with h1 ∈ H ⊥ so ∥ h ∥≥∥ hf ∥.

It follows that ∥ f ∥2 = ⟨hf , hf ⟩, hence the norm on F is coming from an inner product in view of the
polarization identity and the assignment f ←→ hf is a unitary isomorphism between the Hilbert spaces
H and F .

A. Aytuna RKHS 2024 42 / 58

 Reproducing Kernel Hilbert Spaces Continuous Point Evaluations

Example 3 (Cont.)
Another way of visualizing this example is by forming the function space

F = {f : T → C | ∃h ∈ H : f (t ) = ⟨h , ϕ(t )⟩ ∀t ∈ T }

and putting on F the norm

∥ f ∥:= inf ∥ h ∥,
h ∈H ,f (t )=⟨h ,ϕ(t )⟩

where the last norm is the norm in the Hilbert space H.

Note that with the notation above,

h ∈ H ⊥ ⇐⇒ ⟨h , ϕ(t )⟩ = 0 ∀t ∈ T .

It follows that for every f ∈ F there exits a unique hf ∈ H such that f (t ) = ⟨hf , ϕ(t )⟩ ∀t ∈ T , and if
f (t ) = ⟨h , ϕ(t )⟩ ∀t ∈ T , then h = h1 + hf with h1 ∈ H ⊥ so ∥ h ∥≥∥ hf ∥.

It follows that ∥ f ∥2 = ⟨hf , hf ⟩, hence the norm on F is coming from an inner product in view of the
polarization identity and the assignment f ←→ hf is a unitary isomorphism between the Hilbert spaces
H and F .

A. Aytuna RKHS 2024 42 / 58

 Reproducing Kernel Hilbert Spaces Kernels
In the Hilbert spaces (H , ⟨ , ⟩) of functions, on a set T , for which point evaluations are continuous, like
the examples given above, H possesses a collection of distinguished elements {kt }t ∈T defined by:

⟨x , kt ⟩ := x (t ), ∀x ∈ H . (4)

Note that such vectors exist in view of Riesz Representation Theorem since point evaluations are
continuous and are uniquely determined by the given point of T .
One can think of points h ∈ H as indexed by elements of T as {x (t )}t ∈T . The importance of the
elements kt , t ∈ T is that they give "coordinates" {x (t )}t ∈T of an x ∈ H by using the inner product on H
via equality (4).
Hence, for example, if one theoretically knows that a sequence {xn } converges, the knowledge of these
distinguished vectors will allow us to compute the "coordinates" of the limit vector x via
x (t ) = lim⟨x , kt ⟩, ∀t ∈ T .

Definition
Let (H , ⟨ , ⟩) be a Hilbert space of functions on T such that the point evaluations are continuous. Let
{kt }t ∈T be the vectors of H defined as above. One calls the scalar valued function defined on T × T via:

K (t , s ) := ⟨ks , kt ⟩ = ks (t ) = kt (s ) = ⟨kt , ks ⟩,

the kernel of (H , ⟨ , ⟩).

This kernel is reproducing in the sense that it captures the "coordinates" of x ∈ H via:
x (s ) = ⟨x , ks ⟩ = ⟨x , K (s , .)⟩, ∀s ,
since the function t 7→ K (s , t ) is just an element ks in H.
A. Aytuna RKHS 2024 43 / 58
 Reproducing Kernel Hilbert Spaces Kernels
In the Hilbert spaces (H , ⟨ , ⟩) of functions, on a set T , for which point evaluations are continuous, like
the examples given above, H possesses a collection of distinguished elements {kt }t ∈T defined by:

⟨x , kt ⟩ := x (t ), ∀x ∈ H . (4)

Note that such vectors exist in view of Riesz Representation Theorem since point evaluations are
continuous and are uniquely determined by the given point of T .
One can think of points h ∈ H as indexed by elements of T as {x (t )}t ∈T . The importance of the
elements kt , t ∈ T is that they give "coordinates" {x (t )}t ∈T of an x ∈ H by using the inner product on H
via equality (4).
Hence, for example, if one theoretically knows that a sequence {xn } converges, the knowledge of these
distinguished vectors will allow us to compute the "coordinates" of the limit vector x via
x (t ) = lim⟨x , kt ⟩, ∀t ∈ T .

Definition
Let (H , ⟨ , ⟩) be a Hilbert space of functions on T such that the point evaluations are continuous. Let
{kt }t ∈T be the vectors of H defined as above. One calls the scalar valued function defined on T × T via:

K (t , s ) := ⟨ks , kt ⟩ = ks (t ) = kt (s ) = ⟨kt , ks ⟩,

the kernel of (H , ⟨ , ⟩).

This kernel is reproducing in the sense that it captures the "coordinates" of x ∈ H via:
x (s ) = ⟨x , ks ⟩ = ⟨x , K (s , .)⟩, ∀s ,
since the function t 7→ K (s , t ) is just an element ks in H.
A. Aytuna RKHS 2024 43 / 58
 Reproducing Kernel Hilbert Spaces Properties of Kernel Functions

Theorem
Let K , defined on T × T as above, be the kernel function of a Hilbert space (H , ⟨ , ⟩). We have:
1 K (t , s ) = K (s , t ), ∀s , t ∈ T ,
For (λ1 , · · · , λN ) ∈ CN ; i ,j λi λj K (ti , tj ) ≥ 0, ∀N ∈ N, (t1 , · · · , tN ) ∈ T N .
P
2

Proof.
The first property is clear.

To see the second, choose (λ1 , · · · , λN ) ∈ CN and (t1 , · · · , tN ) ∈ T N ;

N
X N
X  N
X N
X
0≤ λi ki , λi ki = λi λj ⟨ki , kj ⟩ = λi λj K (ti , tj ).
i =1 i =1 i , j =1 i ,j =1

The second condition is usually referred to as positiveness of K since it is just the condition that the
 N
matrix K (ti , tj ) , N ∈ N is a positive matrix.
i ,j =1

A. Aytuna RKHS 2024 44 / 58

 Reproducing Kernel Hilbert Spaces Properties of Kernel Functions

Theorem
Let K , defined on T × T as above, be the kernel function of a Hilbert space (H , ⟨ , ⟩). We have:
1 K (t , s ) = K (s , t ), ∀s , t ∈ T ,
For (λ1 , · · · , λN ) ∈ CN ; i ,j λi λj K (ti , tj ) ≥ 0, ∀N ∈ N, (t1 , · · · , tN ) ∈ T N .
P
2

Proof.
The first property is clear.

To see the second, choose (λ1 , · · · , λN ) ∈ CN and (t1 , · · · , tN ) ∈ T N ;

N
X N
X  N
X N
X
0≤ λi ki , λi ki = λi λj ⟨ki , kj ⟩ = λi λj K (ti , tj ).
i =1 i =1 i , j =1 i ,j =1

The second condition is usually referred to as positiveness of K since it is just the condition that the
 N
matrix K (ti , tj ) , N ∈ N is a positive matrix.
i ,j =1

A. Aytuna RKHS 2024 44 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels
It is time to give such Hilbert function spaces a name:

Definition
A Reproducing Kernel Hilbert Space (RKHS) is a Hilbert space (H , ⟨ , ⟩) of functions on a set T such
that all the point evaluations are continuous.

In the definition we have suppressed the kernel function, however we will see later that this scalar
valued function completely determines the Hilbert space. We revisit the examples given above.

Example 1*
Let
e − t +s if t > s , s , t ∈ R.
(
X(s , t ) = e −|t −s | =
e t −s if t ≤ s
Observe that
∂X(s , t ) −e − t + s if t > s , s , t ∈ R.
(
=
∂t e t −s if t < s
R∞ 2
∂X
So Xs (t ) := X(s , t ) is differentiable except at the point s and −∞ ∂t (t ) dt is finite if we interpret the
integral as:
∞
∂Xs s
∂Xs ∞
∂Xs
Z 2
Z 2
Z 2
(t ) dt = (t ) dt + (t ) dt .
−∞ ∂t −∞ ∂t s ∂t

It follows that Xs ∈ W (R) in view of our previous discussion.

A. Aytuna RKHS 2024 45 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels
It is time to give such Hilbert function spaces a name:

Definition
A Reproducing Kernel Hilbert Space (RKHS) is a Hilbert space (H , ⟨ , ⟩) of functions on a set T such
that all the point evaluations are continuous.

In the definition we have suppressed the kernel function, however we will see later that this scalar
valued function completely determines the Hilbert space. We revisit the examples given above.

Example 1*
Let
e − t +s if t > s , s , t ∈ R.
(
X(s , t ) = e −|t −s | =
e t −s if t ≤ s
Observe that
∂X(s , t ) −e − t + s if t > s , s , t ∈ R.
(
=
∂t e t −s if t < s
R∞ 2
∂X
So Xs (t ) := X(s , t ) is differentiable except at the point s and −∞ ∂t (t ) dt is finite if we interpret the
integral as:
∞
∂Xs s
∂Xs ∞
∂Xs
Z 2
Z 2
Z 2
(t ) dt = (t ) dt + (t ) dt .
−∞ ∂t −∞ ∂t s ∂t

It follows that Xs ∈ W (R) in view of our previous discussion.

A. Aytuna RKHS 2024 45 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels

Example 1* (Cont.)
For f ∈ Cc∞ (R), we compute:
Z ∞ Z s Z ∞
f (t )Xs (t )dt = f (t )e t −s dt + f (t )e −t +s dt
−∞ −∞ s
Z ∞ Z s Z ∞
f ′ (t )X′s (t )dt = f ′ (t )e t −s dt − f ′ (t )e −t +s dt
−∞ −∞ s
Z s Z ∞ 
=− f (t )e t −s dt + f (s ) − f (t )e −t +s dt − f (s )
−∞ s
Z s Z ∞
t −s −t +s
=− f (t )e dt − f (t )e dt + 2f (s ),
−∞ s

So Z ∞ Z ∞
f (t )Xs (t )dt + f ′ (t )X′s (t )dt = 2f (s ).
−∞ −∞

So ⟨f , 21 Xs ⟩ = f (s ) for f ∈ Cc∞ (R), hence for f ∈ W ; since Cc∞ (R) = W and point evaluations are
continuous on W .

Thus the kernel on W is the function:

1 1 1
K : R2 → R K (s , t ) = ⟨ Xt , Xs ⟩ = e −|t −s | .
2 2 4

A. Aytuna RKHS 2024 46 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels

Example 1* (Cont.)
For f ∈ Cc∞ (R), we compute:
Z ∞ Z s Z ∞
f (t )Xs (t )dt = f (t )e t −s dt + f (t )e −t +s dt
−∞ −∞ s
Z ∞ Z s Z ∞
f ′ (t )X′s (t )dt = f ′ (t )e t −s dt − f ′ (t )e −t +s dt
−∞ −∞ s
Z s Z ∞ 
=− f (t )e t −s dt + f (s ) − f (t )e −t +s dt − f (s )
−∞ s
Z s Z ∞
t −s −t +s
=− f (t )e dt − f (t )e dt + 2f (s ),
−∞ s

So Z ∞ Z ∞
f (t )Xs (t )dt + f ′ (t )X′s (t )dt = 2f (s ).
−∞ −∞

So ⟨f , 21 Xs ⟩ = f (s ) for f ∈ Cc∞ (R), hence for f ∈ W ; since Cc∞ (R) = W and point evaluations are
continuous on W .

Thus the kernel on W is the function:

1 1 1
K : R2 → R K (s , t ) = ⟨ Xt , Xs ⟩ = e −|t −s | .
2 2 4

A. Aytuna RKHS 2024 46 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels
Before we proceed further, a simple observation is in order.

Suppose H ⊆ CT is a RKHS with kernel K that is separable, i.e., it contains countable elements
fn , n = 1, 2, · · · such that span{fn } = H. Then any orthonormal basis of H is countable.
Suppose {en } is any such orthonormal basis for H, then ∀t ∈ T consider the expansion in H,
∞
X ∞
X N
X
kt (s ) = ⟨kt , en ⟩en (s ) = en (t )en = lim en (t )en in H .
N →∞
n =1 n =1 n =1

Since point evaluations are continuous on H,

N
X  N
X
kt (s ) = lim kt en (t )en (s ) = lim en (s )en (t )
N →∞ N →∞
n=1 n =1
∞
X
= en (s )en (t ),
n =1

where the last convergence is in C.

Hence the kernel of H can be computed as

∞
X
K (t , s ) = kt (s ) = en (s )en (t ).
n =1

Since the right hand side of the equation depends only on H, any choice of orthonormal basis can be
used to compute the kernel of H.
A. Aytuna RKHS 2024 47 / 58
 Reproducing Kernel Hilbert Spaces Reproducing Kernels
Before we proceed further, a simple observation is in order.

Suppose H ⊆ CT is a RKHS with kernel K that is separable, i.e., it contains countable elements
fn , n = 1, 2, · · · such that span{fn } = H. Then any orthonormal basis of H is countable.
Suppose {en } is any such orthonormal basis for H, then ∀t ∈ T consider the expansion in H,
∞
X ∞
X N
X
kt (s ) = ⟨kt , en ⟩en (s ) = en (t )en = lim en (t )en in H .
N →∞
n =1 n =1 n =1

Since point evaluations are continuous on H,

N
X  N
X
kt (s ) = lim kt en (t )en (s ) = lim en (s )en (t )
N →∞ N →∞
n=1 n =1
∞
X
= en (s )en (t ),
n =1

where the last convergence is in C.

Hence the kernel of H can be computed as

∞
X
K (t , s ) = kt (s ) = en (s )en (t ).
n =1

Since the right hand side of the equation depends only on H, any choice of orthonormal basis can be
used to compute the kernel of H.
A. Aytuna RKHS 2024 47 / 58
 Reproducing Kernel Hilbert Spaces Reproducing Kernels

Example 2*
Let f be an analytic function on the unit disc and consider its Taylor series expansion:
∞
X ∞
X
f= cn z n = ⟨f , z n ⟩z n .
n =0 n =0

The last expression in the right hand side comes directly from the definition of the inner product on
H 2 (D). Moreover,
N
X 2 X
⟨f , z n ⟩z n − f = | cn |2 → 0 as N → ∞.
n =1 n >N

So the series ∞ n=1 ⟨f , z ⟩z not only converges uniformly on each disc ∆r = {z :| z |< r }, but also
P n n

converges to f in H (D), and ⟨z n , z m ⟩ = δn,m , ∀n, m ∈ N.

2

It follows that {z n }∞
n =0
is an orthonormal basis in H 2 (D).

So the kernel of H 2 (D) is:

∞ ∞
X X 1
K (ζ, η) = z n (η)z n (ζ) = (ηζ)n = .
n =0 n =0 1 − ηζ

Now we wish to relate this kernel function, obtained by functional analytic considerations to a well
known formula of Complex Analysis.

A. Aytuna RKHS 2024 48 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels

Example 2*
Let f be an analytic function on the unit disc and consider its Taylor series expansion:
∞
X ∞
X
f= cn z n = ⟨f , z n ⟩z n .
n =0 n =0

The last expression in the right hand side comes directly from the definition of the inner product on
H 2 (D). Moreover,
N
X 2 X
⟨f , z n ⟩z n − f = | cn |2 → 0 as N → ∞.
n =1 n >N

So the series ∞ n=1 ⟨f , z ⟩z not only converges uniformly on each disc ∆r = {z :| z |< r }, but also
P n n

converges to f in H (D), and ⟨z n , z m ⟩ = δn,m , ∀n, m ∈ N.

2

It follows that {z n }∞
n =0
is an orthonormal basis in H 2 (D).

So the kernel of H 2 (D) is:

∞ ∞
X X 1
K (ζ, η) = z n (η)z n (ζ) = (ηζ)n = .
n =0 n =0 1 − ηζ

Now we wish to relate this kernel function, obtained by functional analytic considerations to a well
known formula of Complex Analysis.

A. Aytuna RKHS 2024 48 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels

Example 2* (Cont.)

Let f , g be analytic functions on an open disc containing D := {z :| z |≤ 1}. Say f (z ) =

P∞ n
n =0 c n z and
g (z ) = ∞ n
P
n=0 dn z . Since:
f g (e i θ ) =
P∞ i (n−m)θ
n,m=0 cn dm e

on the unit circle and since the series converge uniformly on the unit circle, we have
R 2π
f (e i θ )g (e i θ )d θ = 2π n =0 c n d n .
P∞
0

In particular, such functions are in H 2 (D) and the above expression represents the inner product of two
such functions as an integral.

Note that for any z ∈ D,

1
kz (w ) =
1 − wz
is an analytic function near the closed unit disc. Therefore for an f that is analytic near the closed unit
disc and a point z0 = re i ψ in the unit disc,
2π 2π
dθ
Z Z
1 1
f (z0 ) = ⟨f , kz0 ⟩ = f (e i θ )kz0 (e i θ )d θ = f (e i θ )
2π 0 2π 0 1 − z0 e −i θ
2π
f (e i θ ) ie i θ d θ
Z Z
1 1 f (w )
= · = dw (take w = e i θ , dw = ie i θ d θ)
2π 0 1 − z0 e −i θ ie i θ 2πi Γ w − z0
where Γ = ∂D.
This formula is the classical Cauchy Integral Formula of Complex Analysis.

A. Aytuna RKHS 2024 49 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels

Example 2* (Cont.)

Let f , g be analytic functions on an open disc containing D := {z :| z |≤ 1}. Say f (z ) =

P∞ n
n =0 c n z and
g (z ) = ∞ n
P
n=0 dn z . Since:
f g (e i θ ) =
P∞ i (n−m)θ
n,m=0 cn dm e

on the unit circle and since the series converge uniformly on the unit circle, we have
R 2π
f (e i θ )g (e i θ )d θ = 2π n =0 c n d n .
P∞
0

In particular, such functions are in H 2 (D) and the above expression represents the inner product of two
such functions as an integral.

Note that for any z ∈ D,

1
kz (w ) =
1 − wz
is an analytic function near the closed unit disc. Therefore for an f that is analytic near the closed unit
disc and a point z0 = re i ψ in the unit disc,
2π 2π
dθ
Z Z
1 1
f (z0 ) = ⟨f , kz0 ⟩ = f (e i θ )kz0 (e i θ )d θ = f (e i θ )
2π 0 2π 0 1 − z0 e −i θ
2π
f (e i θ ) ie i θ d θ
Z Z
1 1 f (w )
= · = dw (take w = e i θ , dw = ie i θ d θ)
2π 0 1 − z0 e −i θ ie i θ 2πi Γ w − z0
where Γ = ∂D.
This formula is the classical Cauchy Integral Formula of Complex Analysis.

A. Aytuna RKHS 2024 49 / 58

 Reproducing Kernel Hilbert Spaces Reproducing Kernels

Example 3*

In this example, since for h ∈ H, h̃ (t ) = ⟨h , ϕ(t )⟩ = ⟨h̃ , ϕ̃(t )⟩, the kernel function is plainly

K (s , t ) = kt (s ) = ⟨ϕ̃(t ), ϕ̃(s )⟩ = ⟨ϕ(t ), ϕ(s )⟩.

Note that the distance between the points ϕ(t ) and ϕ(s ) for t , s ∈ T can be computed by using just the
kernel function as;

d (ϕ(t ), ϕ(s )) = ⟨ϕ(t ) − ϕ(s ), ϕ(t ) − ϕ(s )⟩

= ⟨ϕ(t ), ϕ(t )⟩ − 2Re⟨ϕ(t ), ϕ(s )⟩ + ⟨ϕ(s ), ϕ(s )⟩
= K (t , t ) − 2ReK (t , s ) + K (s , s ), t, s ∈ T.

A. Aytuna RKHS 2024 50 / 58

 Reproducing Kernel Hilbert Spaces Yet Another Example
We would like to close this part with an illustration of how the abstract ideas developed in this
presentation might be useful in handling some practical problems.
Suppose you seek a function f in a RKHS H of real valued functions on a set T with smallest norm
satisfying f (ti ) = ai , i = 1, · · · , n, for some points t1 , · · · , tn ∈ T and a1 , · · · , an ∈ R (outcomes of some
experiment?).
If it is not a priori clear that such a function exits, lets say you might be content to find a function in H
that comes ”close” to taking the given values at the specified points.
To put things in mathematical perspective, define:

T : H → Rn via T (f ) := (f (t1 ), · · · , f (tn ))

and transform the problem to the question:

Question
Find a function f0 ∈ H with smallest norm that satisfies:

∥ T (f0 ) − ⃗a ∥2 = inf ∥ T (f ) − ⃗a ∥2 , ⃗a = (a1 , · · · , an ) ∈ Rn ,

f ∈H

where ∥ . ∥ is the usual norm on Rn .

Certainly, T : H → Rn is linear and continuous since point evaluations on H are continuous. T (H ) = Σ

is a subspace of Rn , in particular, it is closed in Rn . Hence there is a unique point in Σ that is closest to
the point ⃗
a (that is the element of smallest norm in the closed convex subset ⃗ a + Σ). This point is P (⃗ a ),
where P is the projection onto Σ.
A. Aytuna RKHS 2024 51 / 58
 Reproducing Kernel Hilbert Spaces Yet Another Example
We would like to close this part with an illustration of how the abstract ideas developed in this
presentation might be useful in handling some practical problems.
Suppose you seek a function f in a RKHS H of real valued functions on a set T with smallest norm
satisfying f (ti ) = ai , i = 1, · · · , n, for some points t1 , · · · , tn ∈ T and a1 , · · · , an ∈ R (outcomes of some
experiment?).
If it is not a priori clear that such a function exits, lets say you might be content to find a function in H
that comes ”close” to taking the given values at the specified points.
To put things in mathematical perspective, define:

T : H → Rn via T (f ) := (f (t1 ), · · · , f (tn ))

and transform the problem to the question:

Question
Find a function f0 ∈ H with smallest norm that satisfies:

∥ T (f0 ) − ⃗a ∥2 = inf ∥ T (f ) − ⃗a ∥2 , ⃗a = (a1 , · · · , an ) ∈ Rn ,

f ∈H

where ∥ . ∥ is the usual norm on Rn .

Certainly, T : H → Rn is linear and continuous since point evaluations on H are continuous. T (H ) = Σ

is a subspace of Rn , in particular, it is closed in Rn . Hence there is a unique point in Σ that is closest to
the point ⃗
a (that is the element of smallest norm in the closed convex subset ⃗ a + Σ). This point is P (⃗ a ),
where P is the projection onto Σ.
A. Aytuna RKHS 2024 51 / 58
 Reproducing Kernel Hilbert Spaces Yet Another Example

However, there may be many elements of H that are mapped to this point; in fact, if f is such an
element, all the others form the set f + Ker (T ). Since this set is a closed (due to T being continuous),
and also a convex set in H; it has a unique point with the least norm. Hence our problem has a unique
solution. This solution u is in (Ker (T ))⊥ by the general theory, otherwise the decomposition of u into
KerT and (KerT )⊥ produces an element of f + KerT that has norm less than u. Now:

T (f ) = (f (t1 ), · · · , f (tn )) = (⟨f , kt1 ⟩, · · · , ⟨f , ktn ⟩),

so for a ⃗ζ = (ζ1 , · · · , ζn ) and f ∈ H, we compute using the inner product in Rn ;

n
⟨⃗ζ, T (f )⟩ =
X
ζi ⟨f , kti ⟩
i =1
 X n 
= f, ζi kti ∀f ∈ H
i =1

= ⟨T ∗ (⃗ζ), f ⟩,

where the last two inner products are in H and as usual kti (·) = K (·, ti ) ∈ H.

Hence we get the formula:

n
T ∗ (⃗ζ) =
X
ζi kti
i =1

A. Aytuna RKHS 2024 52 / 58

 Reproducing Kernel Hilbert Spaces Yet Another Example

However, there may be many elements of H that are mapped to this point; in fact, if f is such an
element, all the others form the set f + Ker (T ). Since this set is a closed (due to T being continuous),
and also a convex set in H; it has a unique point with the least norm. Hence our problem has a unique
solution. This solution u is in (Ker (T ))⊥ by the general theory, otherwise the decomposition of u into
KerT and (KerT )⊥ produces an element of f + KerT that has norm less than u. Now:

T (f ) = (f (t1 ), · · · , f (tn )) = (⟨f , kt1 ⟩, · · · , ⟨f , ktn ⟩),

so for a ⃗ζ = (ζ1 , · · · , ζn ) and f ∈ H, we compute using the inner product in Rn ;

n
⟨⃗ζ, T (f )⟩ =
X
ζi ⟨f , kti ⟩
i =1
 X n 
= f, ζi kti ∀f ∈ H
i =1

= ⟨T ∗ (⃗ζ), f ⟩,

where the last two inner products are in H and as usual kti (·) = K (·, ti ) ∈ H.

Hence we get the formula:

n
T ∗ (⃗ζ) =
X
ζi kti
i =1

A. Aytuna RKHS 2024 52 / 58

 Reproducing Kernel Hilbert Spaces Yet Another Example
Since we are looking at the solution of the equation:

Tu = P (⃗
a ); u ∈ (KerT )⊥ = R (T ∗ ),

the above observation reduces our problem to finite dimensional linear algebra since R (T ∗ ) is finite
dimensional and is spanned by kt1 , · · · , ktn .

In other words, our problem reduces to finding c1 , · · · , cn of real numbers such that:
n
X  n
X n
X
P (⃗
a) = T ci kti = ci T (kti ) = ci (⟨kti , kt1 ⟩, · · · , ⟨kti , ktn ⟩) (5)
i =1 i =1 i =1
 c1 
n n
ci ⟨kti , ktn ⟩ = A  ..  ,
X X   
ci ⟨kti , kt1 ⟩, · · · ,
 
 . 
i =1 i =1 cn

with A = {K (ti , tj )}ni,j =1 ,

where K is the kernel of the Hilbert space H.

However, the right hand side of the equation involves the projection of ⃗
a onto Σ, which is not readily
computable.

To get around this, apply T ∗ to both sides of Equation (5) to get:

n
X n
X
ai kti = T ∗ (⃗
a ) = T ∗ P (⃗
a) = (A ⃗c )i kti .
i =1 i =1

A. Aytuna RKHS 2024 53 / 58

 Reproducing Kernel Hilbert Spaces Yet Another Example
Since we are looking at the solution of the equation:

Tu = P (⃗
a ); u ∈ (KerT )⊥ = R (T ∗ ),

the above observation reduces our problem to finite dimensional linear algebra since R (T ∗ ) is finite
dimensional and is spanned by kt1 , · · · , ktn .

In other words, our problem reduces to finding c1 , · · · , cn of real numbers such that:
n
X  n
X n
X
P (⃗
a) = T ci kti = ci T (kti ) = ci (⟨kti , kt1 ⟩, · · · , ⟨kti , ktn ⟩) (5)
i =1 i =1 i =1
 c1 
n n
ci ⟨kti , ktn ⟩ = A  ..  ,
X X   
ci ⟨kti , kt1 ⟩, · · · ,
 
 . 
i =1 i =1 cn

with A = {K (ti , tj )}ni,j =1 ,

where K is the kernel of the Hilbert space H.

However, the right hand side of the equation involves the projection of ⃗
a onto Σ, which is not readily
computable.

To get around this, apply T ∗ to both sides of Equation (5) to get:

n
X n
X
ai kti = T ∗ (⃗
a ) = T ∗ P (⃗
a) = (A ⃗c )i kti .
i =1 i =1

A. Aytuna RKHS 2024 53 / 58

 Reproducing Kernel Hilbert Spaces Yet Another Example

So solution to the equation

A (⃗c ) = ⃗
a
will be the solution to our problem.

In the case A = {K (ti , tj )} is invertible, one immediately computes the solution.

Note that A is invertible in case the positive function K is positive definite; that is, for every
(λ1 , · · · , λk ) ∈ Rk and x1 , · · · , xk ∈ H,
k k
λi λj K (xi , xj ) = 0 ⇐⇒ (λ1 , · · · , λk ) = ⃗0.
X X
λi λj K (xi , xj ) ≥ 0 and
i ,j =1 i ,j =1

Note that the solution to this problem in case the kernel is positive definite involves only the kernel
function of the Hilbert space.

Concluding Remarks
In the course of this presentation we have associated to a reproducing kernel Hilbert space H of
functions on T , a positive function K : T × T → R, which we called the kernel of H, and observed that
some of the problems involving H can be solved by the use of just the kernel and nothing else.

The coming lectures will make this statement more precise. You will see that a positive function on
K : T × T → C for a set K defines a reproducing kernel Hilbert space of functions on T whose kernel is
precisely K .

A. Aytuna RKHS 2024 54 / 58

 Reproducing Kernel Hilbert Spaces Yet Another Example

So solution to the equation

A (⃗c ) = ⃗
a
will be the solution to our problem.

In the case A = {K (ti , tj )} is invertible, one immediately computes the solution.

Note that A is invertible in case the positive function K is positive definite; that is, for every
(λ1 , · · · , λk ) ∈ Rk and x1 , · · · , xk ∈ H,
k k
λi λj K (xi , xj ) = 0 ⇐⇒ (λ1 , · · · , λk ) = ⃗0.
X X
λi λj K (xi , xj ) ≥ 0 and
i ,j =1 i ,j =1

Note that the solution to this problem in case the kernel is positive definite involves only the kernel
function of the Hilbert space.

Concluding Remarks
In the course of this presentation we have associated to a reproducing kernel Hilbert space H of
functions on T , a positive function K : T × T → R, which we called the kernel of H, and observed that
some of the problems involving H can be solved by the use of just the kernel and nothing else.

The coming lectures will make this statement more precise. You will see that a positive function on
K : T × T → C for a set K defines a reproducing kernel Hilbert space of functions on T whose kernel is
precisely K .

A. Aytuna RKHS 2024 54 / 58

 APPENDIX

"The Pillars of Infinite Dimensional Linear Algebra"

(In the Context of Hilbert Spaces)

(I) Uniform Boundedness Principle

Let H be a Hilbert space. Given a collection of elements {xα }α∈T in H with the following property:

∀x ∈ H ∃C = C (x ) < ∞ s.t. sup | ⟨xα , x ⟩ |≤ C ,

α∈T

then ∃C > 0 such that ∥ xα ∥≤ C ∀xα ∈ T .

(II) Closed Graph Theorem

Let T : H1 → H2 be a linear map between two Hilbert spaces. If the graph of T is closed, i.e., xn → x
and T (xn ) → y for a sequence {xn } in H1 with x ∈ H1 and y ∈ H2 implies T (x ) = y, then T is
continuous.

(III) Alaoglu Theorem

Given a bounded sequence {xn } in a Hilbert space, i.e., ∃C > 0 such that ∥ xn ∥≤ C ∀n, one can find a
subsequence {xkn } of {xn } and a point x ∈ H such that ⟨xkn , h ⟩ → ⟨x , h ⟩ for every h ∈ H.

A. Aytuna RKHS 2024 55 / 58

 APPENDIX

"The Pillars of Infinite Dimensional Linear Algebra"

(In the Context of Hilbert Spaces)

(I) Uniform Boundedness Principle

Let H be a Hilbert space. Given a collection of elements {xα }α∈T in H with the following property:

∀x ∈ H ∃C = C (x ) < ∞ s.t. sup | ⟨xα , x ⟩ |≤ C ,

α∈T

then ∃C > 0 such that ∥ xα ∥≤ C ∀xα ∈ T .

(II) Closed Graph Theorem

Let T : H1 → H2 be a linear map between two Hilbert spaces. If the graph of T is closed, i.e., xn → x
and T (xn ) → y for a sequence {xn } in H1 with x ∈ H1 and y ∈ H2 implies T (x ) = y, then T is
continuous.

(III) Alaoglu Theorem

Given a bounded sequence {xn } in a Hilbert space, i.e., ∃C > 0 such that ∥ xn ∥≤ C ∀n, one can find a
subsequence {xkn } of {xn } and a point x ∈ H such that ⟨xkn , h ⟩ → ⟨x , h ⟩ for every h ∈ H.

A. Aytuna RKHS 2024 55 / 58

 APPENDIX

"The Pillars of Infinite Dimensional Linear Algebra"

(In the Context of Hilbert Spaces)

(I) Uniform Boundedness Principle

Let H be a Hilbert space. Given a collection of elements {xα }α∈T in H with the following property:

∀x ∈ H ∃C = C (x ) < ∞ s.t. sup | ⟨xα , x ⟩ |≤ C ,

α∈T

then ∃C > 0 such that ∥ xα ∥≤ C ∀xα ∈ T .

(II) Closed Graph Theorem

Let T : H1 → H2 be a linear map between two Hilbert spaces. If the graph of T is closed, i.e., xn → x
and T (xn ) → y for a sequence {xn } in H1 with x ∈ H1 and y ∈ H2 implies T (x ) = y, then T is
continuous.

(III) Alaoglu Theorem

Given a bounded sequence {xn } in a Hilbert space, i.e., ∃C > 0 such that ∥ xn ∥≤ C ∀n, one can find a
subsequence {xkn } of {xn } and a point x ∈ H such that ⟨xkn , h ⟩ → ⟨x , h ⟩ for every h ∈ H.

A. Aytuna RKHS 2024 55 / 58

 APPENDIX

(IV) Spectral Theorem for Compact Operators

Given a linear, continuous operator T from a separable Hilbert space H into itself with the additional
properties:
(*) For every bounded sequence {xn } in H, there exists a subsequence {T (xkn )} of {T (xn )} such that
{T (xkn )} converges in H,
(**) T ∗ = T ,
then there exists a sequence λn of real numbers converging to zero and an orthonormal basis {en }∞
n =1
of H such that
∞
X
T (x ) = λn ⟨x , en ⟩en in H .
n =1

A. Aytuna RKHS 2024 56 / 58

 Acknowledgment

My sincere thanks goes to Buket Can Bahdır, for creating this beamer presentation out of my
handwritten notes. Thank you Buket.

A. Aytuna RKHS 2024 57 / 58

 For Further Reading

H. Aronszajn, Theory of Reproducing Kernels, TAMS Vol. 68, No.3, 1950, pp. 337-404.

E. Kreyszig, Introductory Functional Analysis with Applications, John Wiley & Sons, 1978.

R. Meise, D. Vogt, Introduction to Functional Analysis, Oxford Graduate Texts in Mathematics,

Band 2, 1997.
J.H. Manton, P. Amblard, A Primer on Reproducing Kernel Hilbert Spaces, Foundations and
Trends in Signal Processing Vol. XX, No XX, 2015.
D. Okutmuştur, A Survey on Hilbert Spaces and Reproducing Kernels, In Functional Calculus,
London: Intech Open, 2020, pp 61-77.
V. Paulsen, M. Raghupathi, An Introduction to the Theory of Reproducing Kernel Hilbert Spaces,
Cambridge University Press 2016.
W. Rudin, Functional Analysis, 2. Edition, Mc Graw-Hill.

A. Aytuna RKHS 2024 58 / 58