Maths for Data Science

Topic 1.2:

Introduction to

Linear Algebra
Vipin G. Dubey
Agenda for session II:
1. Linearly Independent and Dependent set of Vectors
2. Rank of a matrix and its application
3. Eigen Values and Eigen vectors
Linear Span of a set of vectors:
The Span of a set of vectors is the set of all linear combinations of the vectors in that set, that
is, for any scalars ai ;  1, 2,......, n and for set of vectors v1 , v2 ,......., vn  , Span of set of
vectors is given by
n

a v
i 1
i i

Linear Independent and Dependent set of vectors:

If for a given set of vectors v1 , v2 ,......., vn  ;

n

a v
i 1
i i  0  ai  0 ; i  1, 2,..., n

Then given set is called as Linearly Independent set of vectors. And if at-least one of the ai is
non zero, then it’s called as Linearly Dependent set of vectors.
How many solutions for system of equations Ax=b exist?(consider m
equations in n unknowns)
Possible answers are:
1. It may have only a single unique solution.
2. It may not have any solution.
3. It may have infinitely many solutions.

**It is not possible that it will have more than one and fewer than
infinitely many solutions.
Rank of a Matrix:
Rank of a matrix is the number of linearly independent rows (or columns) in that matrix. It
is denoted as rank ( A) , r ( A) or  ( A) .
Since row rank of a matrix is same as column rank of that matrix;
rank ( A)  min{m, n} ; m No of rows, n No of columns of that matrix.

To find rank of a matrix, we convert that matrix into row-echelon form and then total
number of non-zero rows in the row-echelon form gives rank of that matrix.
Application of Rank to determine number of solutions of a System of Linear Equations:

Result:
Consider the system of m linear equations in n unknowns, given as AX  b ;
A is called as coefficient matrix (of order m  n ), X is vector of unknowns ( n1 ) and matrix
[A | b] is called as augmented matrix (of order m  (n  1) ); then

1. AX  b will not have any solution (i.e. inconsistent) if and only if rank[ A]  rank[ A | b] ,
2. AX  b will have a unique solution if and only if rank[ A]  rank[ A | b]  n ,
3. AX  b will have infinitely many solutions if and only if rank[ A]  rank[ A | b]  n .
Homogeneous System:
A system of linear equations AX  b is called as homogeneous system if and only if b  0
vector. A system AX  0 will always have at least one solution, ‘trivial solution (i.e. X = 0)’.
Are there any other solutions?
Result:
1. Let Amn . A homogeneous system AX  0 will have unique solution, trivial solution X  0
iff rank[ A]  n . In all other cases, it will have infinitely many solutions. As a consequence,
whenever number of unknowns is greater than number of equations (i.e. n  m ), the
homogeneous system will have infinitely many solutions.

2. A nxn homogeneous system of linear equations has a unique solution (the trivial
solution) if and only if its determinant is non-zero. If this determinant is zero, then the
system has an infinite number of solutions.
Eigenvalues and Eigenvectors:
The eigenvectors of a square matrix are the non-zero vectors which, after being multiplied
by the matrix, remain proportional to the original vector, i.e. any vector 𝑋 that satisfies the
equation:
𝐴𝑋 = 𝜆𝑋,
Where, 𝐴 is the matrix in question, 𝑋 is the eigenvector and 𝜆 is the eigenvalue.
In order to find the eigenvectors of a matrix, we must start by finding the eigenvalues.
𝐴𝑋 − 𝜆𝑋 = 0,
Then we pull the vector 𝑥 outside of a set of brackets:
(𝐴 − 𝜆𝐼)𝑋 = 0,
The only way this can be solved is if 𝐴 − 𝜆𝐼 does not have an inverse, therefore we find
values of 𝜆 such that the determinant of 𝐴 − 𝜆𝐼 is zero:
|𝐴 − 𝜆𝐼| = 0
Once we have a set of eigenvalues, we can substitute them back into the original equation to
find the eigenvectors.
YouTube Video: https://www.youtube.com/watch?v=PFDu9oVAE-g (Eigenvectors and eigenvalues)
Example 1
Find the eigenvalues and eigenvectors of the matrix:
2 1
𝐴=[ ]
1 2
First we start by finding the eigenvalues,

|𝐴 − 𝜆𝐼| = |[2 1
]−𝜆[
1 0
]| = |
2−𝜆 1
|
1 2 0 1 1 2−𝜆
Next we derive a formula for the determinant, which must equal zero:
2−𝜆 1
| | = (2 − 𝜆)(2 − 𝜆) − 1 = 𝜆2 − 4𝜆 + 3
1 2−𝜆
𝜆2 − 4𝜆 + 3 = 0
Now, we need to find the roots of this quadratic equation in 𝜆.
𝜆2 − 4𝜆 + 3 = (𝜆 − 3)(𝜆 − 1) = 0
Therefore, the solutions to this equation are𝜆 = 3 𝑎𝑛𝑑 𝜆 = 1
These solutions are the eigenvalues of the matrix 𝐴.
Now we will solve for eigenvectors:
First we will solve for 𝜆1 = 1:
From equation 𝐴𝑋 = 𝜆𝑋,
2 1 𝑥1 𝑥1
[ ] [𝑥 ] = 1 × [𝑥 ]
1 2 2 2

2𝑥 + 𝑥2 𝑥1
[ 1 ] = [𝑥 ]
𝑥1 + 2𝑥2 2

2𝑥1 + 𝑥2 = 𝑥1 → 𝑥1 + 𝑥2 = 0
𝑥1 + 2𝑥2 = 𝑥2 → 𝑥1 + 𝑥2 = 0
These equations are not solvable!
𝑥1 = −𝑥2
1
𝑥=[ ]
−1
This is one of the eigenvectors of the matrix 𝐴.

Now will solve for 𝜆1 = 3: (From equation 𝐴𝑋 = 𝜆𝑋)

2 1 𝑥1 𝑥1
| | [𝑥 ] = 3 × [𝑥 ]
1 2 2 2
 2𝑥1 + 𝑥2 3𝑥
[ ] = [ 1]
𝑥1 + 2𝑥2 3𝑥2
and multiply out to find a set of simultaneous equations:
2𝑥1 + 𝑥2 = 3𝑥1 → −𝑥1 + 𝑥2 = 0
𝑥1 + 2𝑥2 = 3𝑥2 → 𝑥1 − 𝑥2 = 0
These equations are also not solvable!
𝑥1 = 𝑥2
1
𝑥=[ ]
1
This is one of the eigenvectors of the matrix 𝐴.
𝟏 𝟏
The full solution is: for 𝝀𝟏 = 𝟏, 𝒙 = [ ] and for 𝝀𝟏 = 𝟑, 𝒙 = [ ]
−𝟏 𝟏
Topics to be covered in next session:

Topic: Introduction to Linear Algebra

1. Dot Product of two vectors / Cosine Similarity
2. Eigen Decomposition
3. SVD
4. Quadratic Form

Reference Book:
Introduction to Linear Algebra by Gilbert Strang; Wellesley-Cambridge Press.

We will have MCQ test on this session and will have recap of the session 2.
Discussion: On assignment 1.2 (If needed).