Scribd
Upload
166 views

Linear Algebra & Singular Value Decomposition

AN introduction to Linear Algebra and Singular Value Decomposition (SVD) for the purpose of Machine Learning

Uploaded by

Samir Samir
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
166 views

Linear Algebra & Singular Value Decomposition

AN introduction to Linear Algebra and Singular Value Decomposition (SVD) for the purpose of Machine Learning

Uploaded by

Samir Samir
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 5

CS5040: Introduction to Machine Learning

Jul-Nov 2014

Lecture 2:B - Linear Algebra


Lecturer: Spandana Raj B

Scribe: K.R.C. Prasad

Disclaimer: These notes have not been subjected to the usual scrutiny reserved for formal publications.
They may be distributed outside this class only with the permission of the Instructor.

1.1

Eigen Values and Vectors

To understand Eigen values and vectors intuitively, consider the following discussion,
First let us think what a square matrix does to a vector. Consider a matrix A Rnn . Let us see what
the matrix A acting on a vector x does to this vector. By action, we mean multiplication i.e. we get a new
vector y=Ax.
The matrix acting on a vector x does two things to the vector x.
It scales the vector.
It rotates the vector.
However, for any matrix A, there are some favoured vectors/directions. When the matrix acts on these
favoured vectors, the action essentially results in just scaling the vector. There is no rotation. These
favoured vectors are precisely the eigen vectors and the amount by which each of these favoured vectors
stretches or compresses is the eigenvalue.
For a more formal discussion,
Definition 1.1 Consider a square matrix A Rnn . C is an eigen value of matrix A and a vector
~x C n is its eigen vector if A~x = ~x where ~x 6= ~0.

6
Example: A =
2


 
 
5
5
35
, ~v =
, A~v =
= 7~v
1
1
7

Observation: If ~x is an eigen vector of A, then


A(t~x) = t(A~x) = t(~x) = (t~x)
Hence, if ~x is an eigen vector of A, t~x is also an eigen vector of A. Therefore, we express the eigen vector in
its normalised form.

1.1.1

Computing Eigen values and vectors

For Ann , we have,


A~x = ~x
1-1

1-2

Lecture 1: Title

(A I)~x = ~0 |(A I)| = 0


This results in a polynomial in with n roots. The polynomial so obtained, is called the Characteristic
equation of matrix A. If we compute the determinant of an n n matrix, we get n! terms in the expansion.
This is tedious to calculate. Therefore, we use a faster and practically feasible method called Power Method.
Properties:
trace(A) =

n
P

i=1

|A| =

n
Q

i=1

Rank of A = Number of non-zero eigen values


If A is invertible, and i is an eigen value of A, then

1
i

is an eigen value of A1

Theorem 1.2 Consider a matrix Ann . Let v~1 , v~2 , ..., v~n be eigen vectors of A and 1 , 2 , ..., n be the eigen
values. If each of these eigen values are distinct, then v~1 , v~2 , ..., v~n are linearly independent.
Proof: Proof by contradiction
Suppose v~1 , v~2 , ..., v~n are linearly dependent. Consider minimum value of k such that,

k
X

1kn

ai v~i = ~0

i=1

= (A k I)

k
X

ai v~i = ~0

i=1

k
X

(A k I)ai v~i = ~0

i=1

k
X

ai (i v~i k v~i ) = ~0

i=1

k
X

ai (i k )~
vi = ~0

i=1

When i = k,
ai (i k )~
vi = ~0
=

k1
X

ai (i k )~
vi = ~0

i=1

This is a contradiction since we assumed that k is the smallest. This concludes the proof.

Lecture 1: Title

1.2

1-3

Diagonalisation

Definition 1.3 Suppose A, B are two square matrices of size n n. We say A, B are similar, if A =
P 1 BP for some invertible matrix P.
Definition 1.4 Suppose A is a square matrix of size n n. We say that A is diagonalizable, if there exists
an invertible matrix P such that P 1 AP is a diagonal matrix.
So, our question is which matrices are diagonalizable? Following theorem has some answer.
Definition 1.5 Suppose A is a square matrix of size n n. Then A is diagonalizable if and only if A has
n linearly independent eigenvectors.
Proof: Suppose A is diagonalizable. So, there is an invertible matrix P such that P 1 AP = D is a diagonal
matrix. Write


P = p~1

p~2

p~3

... p~n

1
and D = .
0

0 0 ...
. . ...
0 0 ...

0
.
n

where p~1 , p~2 , ..., p~n are the columns of P. We have AP = PD. So,


 
 1 0 0 ... 0
A p~1 p~2 p~3 ... p~n = p~1 p~2 p~3 ... p~n . . . ... .
0 0 0 ... n
Therefore, i = 2, . . . , n we have A~
pi = i p~i and so p~i are eigenvectors of A. Also, since P is invertible
p1 , p2 , ..., pn are linearly independent. So, A has n linearly independent eigen-vectors. To prove the converse,
assume A has n linearly independent eigen- vectors, p1 , p2 , ..., pn . Then, for i = 2, . . . , n we have,
Api = i pi for some i . Write,


P = p~1

p~2

p~3

... p~n

1
and D = .
0

0 0
. .
0 0

...
...
...

0
.
n

It follows easily that AP = PD. Since, columns of P are linearly in- dependent, it follows that P is invertible.
Therefore, P 1 AP = D is a diagonal matrix. So, the proof is complete.

Observations:
Diagonalisation matrix S is not unique
Not all matrices have S 1 AS
We can see the use of using diagnolisation while calculating the powers of matrices from the following
example.
Example: Calculate An .

1-4

Lecture 1: Title

We know,
A = SS 1
An = (SS 1 )(SS 1 )...(SS 1 ) = (Sn S 1 )

1.3

Positive Semi Definite Matrices

Definition 1.6 A symmetric matrix A Rnn is called positive semidefinite if ~xT A~x 0 for all ~x Rn ,
and is called positive definite if ~xT A~x 0 for all nonzero ~x Rn . If A is positive semidefinite, we denote
A  0.
Theorem 1.7 Matrix A is positive semi definite if and only if all eigen values of A are non-negative.
Proof: For a matrix to be positive semidefinite,
~xT A~x 0~x
But if ~x is an eigen vector of A, then
~xT (~x) 0
~xT ~x 0
Since ~xT ~x is neccessarily a positive number, in order for ~xT A~x to be greater than or equal to 0,
= 0

1.4

Singular Value Decomposition

Definition 1.8 Any real m n matrix A can be decomposed uniquely as A = U DV T .


U is m n and column orthogonal (its columns are eigen vectors of AT A)
AT A = U DV T V DU T = U D2 U T
V is n n and orthogonal (its columns are eigen vectors of AT A)
AT A = V DU T U DV T = V D2 V T
D is n n diagonal (non negative real values called singular values). D = diag(1 , 2 , ...n ) ordered so that
1 2 ... n (if is a singular value of A, its square is an eigen value of AT A)
If U = (u1 , u2 , ...un ) and V = (v1 , v2 , ...vn ), then
A=

n
X

i ui vi T

i=1

If i are singular values of matrix A, and i are eigen values of A, then,


p
i = i

Lecture 1: Title

1.5

References

Strang, Gilbert (1993), Introduction to linear algebra


Horn, Roger A.; Johnson, Charles F. (1985), Matrix analysis

1-5

You might also like