29th December 2014 M05_LEON2218_09_SE_C05 page 254

254 Chapter 5 Orthogonality

10. Let A be an 8 × 5 matrix of rank 3, and let b be a 14. Find the equation of the circle that gives the best
nonzero vector in N(AT ). least squares circle fit to the points (−1, −2),
(a) Show that the system Ax = b must be incon- (0, 2.4), (1.1, −4), and (2.4, −1.6).
sistent. 15. Suppose that in the search procedure described
(b) How many least squares solutions will the in Example 4, the search committee made the
system Ax = b have? Explain. following judgments in evaluating the teaching
11. Let P = A(ATA)−1 AT , where A is an m × n matrix credentials of the candidates:
of rank n. (i) Gauss and Taussky have equal teaching
(a) Show that P2 = P. credentials.
(b) Prove that Pk = P for k = 1, 2, . . . . (ii) O’Leary’s teaching credentials should be
(c) Show that P is symmetric. [Hint: If B is nonsin- given 1.25 times the weight of Ipsen’s cre-
gular, then (B−1 )T = (BT )−1 .] dentials and 1.75 times the weight given to
the credentials of both Gauss and Taussky.
12. Show that if
⎧ ⎫⎧ ⎫ ⎧ ⎫ (iii) Ipsen’s teaching credentials should be
⎩ A IT ⎪
⎪ ⎭⎪⎩ x̂ ⎪
⎭=⎪ ⎩b⎪⎭
O A r 0 given 1.25 times the weight given to the
credentials of both Gauss and Taussky.
then x̂ is a least squares solution of the system
(a) Use the method given in Application 4 to de-
Ax = b and r is the residual vector.
termine a weight vector for rating the teaching
13. Let A ∈ Rm×n and let x̂ be a solution of the least credentials of the candidates.
squares problem Ax = b. Show that a vector y ∈
Rn will also be a solution if and only if y = x̂ + z, (b) Use the weight vector from part (a) to obtain
for some vector z ∈ N(A). [Hint: N(ATA) = N(A).] overall ratings of the candidates.

5.4 Inner Product Spaces

Scalar products are useful not only in Rn , but in a wide variety of contexts. To
generalize this concept to other vector spaces, we introduce the following definition.

Definition and Examples

Definition An inner product on a vector space V is an operation on V that assigns, to each pair
of vectors x and y in V, a real number x, y satisfying the following conditions:
I. x, x ≥ 0 with equality if and only if x = 0.
II. x, y = y, x for all x and y in V.
III. αx + βy, z = α x, z + β y, z for all x, y, z in V and all scalars α and β.

A vector space V with an inner product is called an inner product space.

The Vector Space Rn
The standard inner product for Rn is the scalar product
x, y = xT y
Given a vector w with positive entries, we could also define an inner product on Rn by
n
x, y = xi yi wi (1)
i=1
The entries wi are referred to as weights.
29th December 2014 M05_LEON2218_09_SE_C05 page 255

5.4 Inner Product Spaces 255

The Vector Space Rm×n

Given A and B in Rm×n , we can define an inner product by
m n
A, B = aij bij (2)
i=1 j=1

We leave it to the reader to verify that (2) does indeed define an inner product on Rm×n .

The Vector Space C[a, b]

We may define an inner product on C[a, b] by
 b
 f , g = f (x)g(x) dx (3)
a

Note that
 b
f,f  = (f (x))2 dx ≥ 0
a

If f (x0 ) = 0 for some x0 in [a, b], then, since (f (x))2 is continuous, there exists a
subinterval I of [a, b] containing x0 such that (f (x))2 ≥ (f (x0 ))2 /2 for all x in I. If we
let p represent the length of I, then it follows that
 b 
( f (x0 ))2 p
f,f  = ( f (x)) dx ≥ ( f (x))2 dx ≥
2
>0
a I 2
So if  f , f  = 0, then f (x) must be identically zero on [a, b]. We leave it to the reader
to verify that (3) satisfies the other two conditions specified in the definition of an inner
product.
If w(x) is a positive continuous function on [a, b], then
 b
 f , g = f (x)g(x)w(x) dx (4)
a

also defines an inner product on C[a, b]. The function w(x) is called a weight function.
Thus it is possible to define many different inner products on C[a, b].

The Vector Space Pn

Let x1 , x2 , . . . , xn be distinct real numbers. For each pair of polynomials in Pn , define
n
p, q = p(xi )q(xi ) (5)
i=1

It is easily seen that (5) satisfies conditions (ii) and (iii) of the definition of an inner
product. To show that (i) holds, note that
n
p, p = (p(xi ))2 ≥ 0
i=1
29th December 2014 M05_LEON2218_09_SE_C05 page 256

256 Chapter 5 Orthogonality

If p, p = 0, then x1 , x2 , . . . , xn must be roots of p(x) = 0. Since p(x) is of degree less

than n, it must be the zero polynomial.
If w(x) is a positive function, then
n
p, q = p(xi )q(xi )w(xi )
i=1
also defines an inner product on Pn .

Basic Properties of Inner Product Spaces

The results presented in Section 5.1 for scalar products in Rn all generalize to inner
product spaces. In particular, if v is a vector in an inner product space V, the length, or
norm of v is given by
v = v, v
Two vectors u and v are said to be orthogonal if u, v = 0. As in Rn , a pair of
orthogonal vectors will satisfy the Pythagorean law.

Theorem 5.4.1 The Pythagorean Law

If u and v are orthogonal vectors in an inner product space V, then
u + v2 = u2 + v2

Proof
u + v2 = u + v, u + v
= u, u + 2 u, v + v, v
= u2 + v2

Interpreted in R2 , this is just the familiar Pythagorean theorem as shown in

Figure 5.4.1.

u+v
u

Figure 5.4.1.

EXAMPLE 1 Consider the vector space C[−1, 1] with inner product defined by (3). The vectors 1
and x are orthogonal, since
 1
1, x = 1 · x dx = 0
−1
To determine the lengths of these vectors, we compute
 1
1, 1 = 1 · 1 dx = 2
−1
 1
2
x, x = x2 dx =
−1 3
29th December 2014 M05_LEON2218_09_SE_C05 page 257

5.4 Inner Product Spaces 257

It follows that
√
1 = (1, 1)1/2 = 2
√
6
x = (x, x)1/2 =
3
Since 1 and x are orthogonal, they satisfy the Pythagorean law:
2 8
1 + x2 = 12 + x2 = 2 + =
3 3
The reader may verify that
 1
8
1 + x2 = 1 + x, 1 + x = (1 + x)2 dx =
−1 3

EXAMPLE 2 For the vector space C[−π, π], if we use a constant weight function w(x) = 1/π to
define an inner product
 π
1
f , g = f (x)g(x) dx (6)
π −π

then

1 π
cos x, sin x = cos x sin x dx = 0
π −π

1 π
cos x, cos x = cos x cos x dx = 1
π −π

1 π
sin x, sin x = sin x sin x dx = 1
π −π
Thus, cos x and sin x are orthogonal unit vectors with respect to this inner product. It
follows from the Pythagorean law that
√
 cos x + sin x = 2

The inner product (6) plays a key role in Fourier analysis applications involving
trigonometric approximation of functions. We will look at some of these applications
in Section 5.5.
For the vector space Rm×n the norm derived from the inner product (2) is called
the Frobenius norm and is denoted by  · F . Thus, if A ∈ Rm×n , then
⎛ ⎞1/2
m n
AF = (A, A) 1/2
=⎝ a2ij ⎠
i=1 j=1

EXAMPLE 3 If
⎧ ⎫ ⎧ ⎫
⎪
⎪ 1 1⎪⎪ ⎪
⎪ −1 1 ⎪
⎪
A=⎪
⎪
⎪ 1 2⎪⎪
⎪ and B=⎪
⎪
⎪ 3 0⎪⎪
⎪
⎩ ⎭ ⎩ ⎭
3 3 −3 4
29th December 2014 M05_LEON2218_09_SE_C05 page 258

258 Chapter 5 Orthogonality

then
A, B = 1 · −1 + 1 · 1 + 1 · 3 + 2 · 0 + 3 · −3 + 3 · 4 = 6
Hence, A is not orthogonal to B. The norms of these matrices are given by

AF = (1 + 1 + 1 + 4 + 9 + 9)1/2 = 5
BF = (1 + 1 + 9 + 0 + 9 + 16)1/2 = 6
EXAMPLE 4 In P5 , define an inner product by (5) with xi = (i − 1)/4 for i = 1, 2, . . . , 5. The length
of the function p(x) = 4x is given by
 5 1/2  5 1/2
√
4x = (4x, 4x) =
1/2
16xi2
= (i − 1)2
= 30
i=1 i=1

Definition If u and v are vectors in an inner product space V and v = 0, then the scalar
projection of u onto v is given by
u, v
α=
v
and the vector projection of u onto v is given by

1 u, v
p=α v = v (7)
v v, v

Observations
If v = 0 and p is the vector projection of u onto v, then
I. u − p and p are orthogonal.
II. u = p if and only if u is a scalar multiple of v.

Proof of Since   2
Observation I α α α
p, p = v, v = v, v = α 2
v v v
and
(u, v)2
u, p = = α2
v, v
it follows that
u − p, p = u, p − p, p = α 2 − α 2 = 0
Therefore, u − p and p are orthogonal.

Proof of If u = βv, then the vector projection of u onto v is given by

Observation II βv, v
p= v = βv = u
v, v
29th December 2014 M05_LEON2218_09_SE_C05 page 259

5.4 Inner Product Spaces 259

Conversely, if u = p, it follows from (7) that

α
u = βv where β =
v

Observations I and II are useful for establishing the following theorem:

Theorem 5.4.2 The Cauchy–Schwarz Inequality

If u and v are any two vectors in an inner product space V, then
| u, v | ≤ u v (8)
Equality holds if and only if u and v are linearly dependent.

Proof If v = 0, then
| u, v | = 0 = u v
If v = 0, then let p be the vector projection of u onto v. Since p is orthogonal to u − p,
it follows from the Pythagorean law that
p2 + u − p2 = u2
Thus
(u, v)2
= p2 = u2 − u − p2
v2
and hence
(u, v)2 = u2 v2 − u − p2 v2 ≤ u2 v2 (9)
Therefore,
| u, v | ≤ u v
Equality holds in (9) if and only if u = p. It follows from observation II that equality
will hold in (8) if and only if v = 0 or u is a multiple of v. More simply stated, equality
will hold if and only if u and v are linearly dependent.

One consequence of the Cauchy–Schwarz inequality is that if u and v are nonzero

vectors, then
u, v
−1 ≤ ≤1
uv
and hence there is a unique angle θ in [0, π] such that
u, v
cos θ = (10)
uv
Thus equation (10) can be used to define the angle θ between two nonzero vectors u
and v.

Norms
The word norm in mathematics has its own meaning that is independent of an inner
product and its use here should be justified.
29th December 2014 M05_LEON2218_09_SE_C05 page 260

260 Chapter 5 Orthogonality

v+w
w

v
Figure 5.4.2.

Definition A vector space V is said to be a normed linear space if, to each vector v ∈ V, there
is associated a real number v, called the norm of v, satisfying
I. v ≥ 0 with equality if and only if v = 0.
II. αv = |α| v for any scalar α.
III. v + w ≤ v + w for all v, w ∈ V.

The third condition is called the triangle inequality (see Figure 5.4.2).

Theorem 5.4.3 If V is an inner product space, then the equation

v = v, v for all v ∈ V

defines a norm on V.

Proof It is easily seen that conditions I and II of the definition are satisfied. We leave this for
the reader to verify and proceed to show that condition III is satisfied.

u + v2 = u + v, u + v
= u, u + 2 u, v + v, v
≤ u2 + 2u v + v2 (Cauchy–Schwarz)
= (u + v)2

Thus,
u + v ≤ u + v

It is possible to define many different norms on a given vector space. For example,
in Rn we could define
n
x1 = |xi |
i=1

for every x = (x1 , x2 , . . . , xn )T . It is easily verified that  · 1 defines a norm on

Rn . Another important norm on Rn is the uniform norm or infinity norm, which is
defined by

x∞ = max |xi |

1≤i≤n
29th December 2014 M05_LEON2218_09_SE_C05 page 261

5.4 Inner Product Spaces 261

More generally, we could define a norm on Rn by

 n 1/p
xp = |xi |p
i=1

for any real number p ≥ 1. In particular, if p = 2, then

 n 1/2
x2 = |xi |2 = x, x
i=1

The norm  · 2 is the norm on R derived from the inner product. If p = 2,  · p does
n

not correspond to any inner product. In the case of a norm that is not derived from an
inner product, the Pythagorean law will not hold. For example,
⎧ ⎫ ⎧ ⎫
x1 = ⎪
⎩1⎪ ⎭ and x2 = ⎪⎩ −4 ⎪
⎭
2 2
are orthogonal; however,

x1 2∞ + x2 2∞ = 4 + 16 = 20

while

x1 + x2 2∞ = 16

If, however,  · 2 is used, then

x1 22 + x2 22 = 5 + 20 = 25 = x1 + x2 22

EXAMPLE 5 Let x be the vector (4, −5, 3)T in R3 . Compute x1 , x2 , and x∞ .

x1 = |4| + |−5| + |3| = 12

√ √
x2 = 16 + 25 + 9 = 5 2
x∞ = max(|4|, |−5|, |3|) = 5

It is also possible to define different matrix norms for Rm×n . In Chapter 7 we will
study other types of matrix norms that are useful in determining the sensitivity of linear
systems.
In general, a norm provides a way of measuring the distance between vectors.

Definition Let x and y be vectors in a normed linear space. The distance between x and y is
defined to be the number y − x.

Many applications involve finding a unique closest vector in a subspace S to a given

vector v in a vector space V. If the norm used for V is derived from an inner product,
then the closest vector can be computed as a vector projection of v onto the subspace S.
This type of approximation problem is discussed further in the next section.
29th December 2014 M05_LEON2218_09_SE_C05 page 262

262 Chapter 5 Orthogonality

SECTION 5.4 EXERCISES

1. Let x = (−1, −1, 1, 1)T and y = (1, 1, 5, −3)T . 10. Show that the functions x and x2 are orthogonal
Show that x ⊥ y. Calculate x2 , y2 , x + y2 in P5 with inner product defined by (5), where
and verify that the Pythagorean law holds. xi = (i − 3)/2 for i = 1, . . . , 5.
2. Let x = (1, 1, 1, 1)T and y = (8, 2, 2, 0)T . 11. In P5 with inner product as in Exercise 10 and norm
defined by
(a) Determine the angle θ between x and y.  5 1/2
 2
(b) Find the vector projection p of x onto y. p = p, p = p(xi )
(c) Verify that x − p is orthogonal to p. i=1
compute
(d) Compute x − p2 , p2 , x2 and verify that (a) x (b) x2 
the Pythagorean law is satisfied.
(c) the distance between x and x2
3. Use equation (1) with weight vector w =
 1 1 1 T 12. If V is an inner product space, show that
, ,
4 2 4
to define an inner product for R3 , and
let x = (1, 1, 1)T and y = (−5, 1, 3)T . v = v, v
(a) Show that x and y are orthogonal with respect satisfies the first two conditions in the definition of
to this inner product. a norm.
13. Show that
(b) Compute the values of x and y with re-
n
spect to this inner product.
x1 = |xi |
4. Given
i=1
⎧ ⎫ ⎧ ⎫
⎪
⎪ 1 2 2⎪ ⎪ ⎪
⎪ −4 1 1⎪
⎪ defines a norm on Rn .
A=⎪ ⎪ 1 0 2⎪ ⎪ and B = ⎪ ⎪ −3 2⎪
⎪
⎪
⎩ ⎪
⎭ ⎪
⎩
3 ⎪
⎭ 14. Show that
3 1 1 1 −2 −2
x∞ = max |xi |
determine the value of each of the following. 1≤i≤n

(a) A, B (b) AF defines a norm on R . n

(c) BF (d) A + BF 15. Compute x1 , x2 , and x∞ for each of the fol-
lowing vectors in R3 .
5. Show that equation (2) defines an inner product on
Rm×n . (a) x = (−3, 4, 0)T (b) x = (−1, −1, 2)T
(c) x = (1, 1, 1)T
6. Show that the inner product defined by equation (3)
satisfies the last two conditions of the definition of 16. Let x = (5, 2, 4)T and y = (3, 3, 2)T . Compute
an inner product. x − y1 , x − y2 , and x − y∞ . Under which
norm are the two vectors closest together? Under
7. In C[0, 1], with inner product defined by (3), com- which norm are they farthest apart?
pute 17. Let x and y be vectors in an inner product space.
   
(a) ex , e−x (b) x, sin π x (c) x2 , x3 Show that if x ⊥ y then the distance between x and
8. In C[0, 1], with inner product defined by (3), con- y is
sider the vectors 1 and x.  2 1/2
x + y2
(a) Find the angle θ between 1 and x.
18. Show that if u and v are vectors in an inner product
(b) Determine the vector projection p of 1 onto x
space that satisfy the Pythagorean law
and verify that 1 − p is orthogonal to p.
u + v2 = u2 + v2
(c) Compute 1 − p, p, 1 and verify that the
then u and v must be orthogonal.
Pythagorean law holds.
19. In Rn with inner product
9. In C[−π , π ] with inner product defined by (6),
show that cos mx and sin nx are orthogonal and x, y = xT y
that both are unit vectors. Determine the distance derive a formula for the distance between two vec-
between the two vectors. tors x = (x1 , . . . , xn )T and y = (y1 , . . . , yn )T .
29th December 2014 M05_LEON2218_09_SE_C05 page 263

5.5 Orthonormal Sets 263

!b
20. Let A be a nonsingular n × n matrix and for each (b)  f  = a | f (x)| dx
vector x in Rn define (c)  f  = max | f (x)|
a≤x≤b
xA = Ax2 (11)
29. Let x ∈ Rn and show that
Show that (11) defines a norm on Rn . √
(a) x1 ≤ nx∞ (b) x2 ≤ n x∞
21. Let x ∈ Rn . Show that x∞ ≤ x2 . Give examples of vectors in Rn for which equality
22. Let x ∈ R2 . Show that x2 ≤ x1 . [Hint: Write holds in parts (a) and (b).
x in the form x1 e1 + x2 e2 and use the triangle 30. Sketch the set of points (x1 , x2 ) = xT in R2 such
inequality.] that
23. Give an example of a nonzero vector x ∈ R2 for (a) x2 = 1 (b) x1 = 1 (c) x∞ = 1
which 31. Let K be an n × n matrix of the form
⎧ ⎫
⎪ 1 −c −c · · · −c −c ⎪
x∞ = x2 = x1 ⎪
⎪ ⎪
⎪
⎪ 0 s −sc · · · −sc −sc ⎪ ⎪
⎪
⎪
⎪ ⎪
⎪
⎪
⎪ 0 0 s2
· · · −s 2
c −s 2
c ⎪
⎪
24. Show that in any vector space with a norm K=⎪ ⎪ ⎪
⎪
⎪
⎪ .. ⎪
⎪
−v = v ⎪
⎪ . ⎪
⎪
⎪
⎪ ⎪
⎪
⎪
⎪ 0 0 0 · · · s n−2
−s n−2
c ⎪
⎪
25. Show that for any u and v in a normed vector space ⎩ ⎭
0 0 0 ··· 0 s n−1

√
u + v ≥ | u − v | where c2 + s2 = 1. Show that KF = n.
32. The trace of an n × n matrix C, denoted tr(C), is the
26. Prove that, for any u and v in an inner product sum of its diagonal entries; that is,
space V,
tr(C) = c11 + c22 + · · · + cnn
u + v2 + u − v2 = 2u2 + 2v2
If A and B are m × n matrices, show that
Give a geometric interpretation of this result for the (a) A2F = tr(ATA)
vector space R2 . (b) A + B2F = A2F + 2 tr(AT B) + B2F .
27. The result of Exercise 26 is not valid for norms
other than the norm derived from the inner product. 33. Consider the vector space Rn with inner product
Give an example of this in R2 using  · 1 . x, y = xT y. Show that for any n × n matrix A,
28. Determine whether the following define norms on  
(a) Ax, y = x, AT y
C[a, b]:  T 
(a)  f  = | f (a)| + | f (b)| (b) A Ax, x = Ax2

5.5 Orthonormal Sets

In R2 , it is generally more convenient to use the standard basis {e1 , e2 } than to use
some other basis, such as {(2, 1)T , (3, 5)T }. For example, it would be easier to find
the coordinates of (x1 , x2 )T with respect to the standard basis. The elements of the
standard basis are orthogonal unit vectors. In working with an inner product space V,
it is generally desirable to have a basis of mutually orthogonal unit vectors. Such a
basis is convenient not only in finding coordinates of vectors, but also in solving least
squares problems.

 
Definition Let v1 , v2 , . . . , vn be nonzero vectors in an inner product space V. If vi , vj = 0
whenever i = j, then {v1 , v2 , . . . , vn } is said to be an orthogonal set of vectors.