Linear Algebra

Lecture 03

Prof. Shreedevi K. Masuti

IIT Dharwad

07 Jan 2020

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 System of Linear Equation

A linear system of m equations in the n unknowns x1 , ..., xn is given by

a11 x1 + a12 x2 + · · · + a1n xn = b1 (1)

a21 x1 + a22 x2 + · · · + a2n xn = b2 (2)
.. .. .. .. ..
. . . . .
am1 x1 + am2 x2 + · · · + amn xn = bm , (m)

where aij ∈ R for i = 1, . . . , m; j = 1, . . . , n and also bj ∈ R for

j = 1, . . . , m are given.
Goal Solve the equations (1) to (m) simultaneously

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

  t
Let A := [aij ] ∈ Mm,n , x := x1 · · · xn ∈ Mn,1 and
 t
b := b1 · · · bm ∈ Mm,1 . Using matrix multiplication, the linear
system can be written as
Ax = b (∗).
• Coefficient Matrix: the m × n matrix A
• Solution of the above linear system of equations: A column vector
x0 ∈ Mn,1 such that Ax0 = b
• Homogeneous linear system: b = 0
• inhomogeneous linear system: b 6= 0
• Note: 0 is always a solution of homogeneous linear system of
equations
• Caution: A Non-homogeneous linear system of equations may not
have a solution, may have only one solution or may have (infinitely)
many solutions.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Examples
(1) Consider the linear system x1 + x2 = 1, 2x1 + 2x2 = 4 does not have
a solution.

2x1 + 2x2 = 4

x1 + x2 = 1

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

(ii) The linear system x1 + x2 = 1, x1 − x2 = 0 has a unique solution,
namely x1 = 1/2 = x2 .

x1 − x2 = 0

x1 + x2 = 1

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

(iii) The linear system x1 + x2 = 1, 2x1 + 2x2 = 2 has (infinitely) many
solutions, namely x1 = α, x2 = 1 − α, α ∈ R.

2x1 + 2x2 = 2

x1 + x2 = 1

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Proposition
Consider the homogeneous linear system Ax = 0. If r ∈ N and x1 , . . . , xr
are solutions of such a system, then so is their linear combination
α1 x1 + · · · + αr xr for any α1 , . . . , αr ∈ R.

Proof: A(α1 x1 + · · · + αr xr ) = α1 Ax1 + · · · + αr Axr = 0

Proposition
Consider the system of equations (∗). Let S denote the set of all
solutions of the corresponding homogeneous linear system Ax = 0. If x0
is a particular solution of (∗), then the set of all solutions of (∗) is given
by {x0 + s : s ∈ S}

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Proof: First note that A(x0 + s) = Ax0 + As = b + 0 = b.
Moreover, if x is any solution of (∗), then

A(x − x0 ) = Ax − Ax0 = b − b = 0

=⇒ x − x0 ∈ S. Let x − x0 = s for some s ∈ S. Then x = x0 + s.

Reduction: Find all solutions of a homogeneous linear system Ax = 0,
and one particular solution of the corresponding non homogeneous linear
system Ax = b.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Example Suppose

x1 − x2 + x3 = b1
x2 + x3 = b2
x3 = b3

Solution:

x3 = b3
x2 = b2 − x3 = b2 − b3
x1 = b1 + x2 − x3 = b1 + b2 − 2b3

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Back Substitution

Suppose the coefficient matrix A is upper triangular and its diagonal

elements are nonzero. Then the linear system

a11 x1 + a12 x2 + · · · + · · · + · · · + · · · + a1n xn = b1 (1)

a22 x2 + a23 x3 + · · · + · · · + · · · + a2n xn = b2 (2)
.. .. .. ..
. . . .
a(n−1)(n−1) xn−1 + a(n−1)n xn = bn−1 (n − 1)
ann xn = bn (n)

can be solved by back substitution as follows.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 xn = bn /ann
xn−1 = (bn−1 − a(n−1)n xn )/a(n−1)(n−1), where xn = bn /ann
.. .. ..
. . .
x2 = (b2 − a2n xn − · · · − a23 x3 )/a22 , where xn = · · ·, x3 = · · ·,
x1 = (b1 − a1n xn − · · · − · · · − a12 x2 )/a11 , where xn = · · ·, x2 = · · ·.

Here the homogeneous system Ax = 0 has only the zero solution and the
general system Ax = b has a unique solution.
Goal: To transform any m × n matrix to an upper triangular form.
In this process, we successively attempt to eliminate the unknown x1
from the equations (m), . . . , (2), the unknown x2 from the equations
(m), . . . , (3), and so on.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Example
Consider the linear system

x1 − x2 + x3 = 0
−x1 + x2 − x3 = 0
10x2 + 25x3 = 90
20x1 + 10x2 = 80.

Eliminating x1 from the 4th, 3rd and 2nd equations,

x1 − x2 + x3 = 0
0 = 0
10x2 + 25x3 = 90
30x2 − 20x3 = 80.

Interchanging the 2nd and the 3rd equations,

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 x1 − x2 + x3 = 0
10x2 + 25x3 = 90
0 = 0
30x2 − 20x3 = 80.

Eliminating x2 from the 4th equation, and then interchanging the 3rd
and the 4th equations,

x1 − x2 + x3 = 0
10x2 + 25x3 = 90
−95x3 = −190
0 = 0.

Now back substitution gives x3 = 2, x2 = (90 − 25x3 )/10 = 4 and

 T
x1 = −x3 + x2 = 2, that is, x = 2 4 2 .

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

The above process can be carried out without writing down the entire
linear system by considering the augmented matrix

a11 a12 ··· a1n b1

 
 a12 a22 ··· a2n b2 
[A|b] :=  .. .. .. .. ..  .
 
 . . . . .
am1 am2 ··· amn bn

This m × (n + 1) matrix completely describes the linear system.

In the above example,
 
1 −1 1 0
−1 1 −1 0
[A|b] = 
0
.
10 25 90
20 10 0 80

Subtracting 20 times the first row from the 4th row, and adding the first
row to the second row, we obtain

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

  
1 −1 1 0
R4 −20R1 , R2 +R1 , 0 0 0 0
−−−−−−−−−−→  0 10
.
25 90
0 30 −20 80
Interchanging the 2nd and the 3rd rows, we obtain
 
1 −1 1 0
R2 ←→R3  0 10 25 90
−− −−−→  0 0
.
0 0
0 30 −20 80
Finally, subtracting 3 times the 2nd row from the 4th row and
interchanging the3rd and the 4th rows, wearrive at
1 −1 1 0
R4 −3R2 , R3 ←→R4 0 10 25 90 
−−−−−−−−−−−→  0 0 −95
.
−190
0 0 0 0
The upper triangular nature of the 3 × 3 matrix on the top left enables
back substitution.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Row Echelon Form

We shall now consider a general form of a matrix for which the method
of back substitution works.
Let A be an m × n matrix with real entries.
A row of A is said to be zero if all its entries are zero.
If a row is not zero, then its first nonzero entry (from the left) is called
the pivot.Thus all entries to the left of a pivot equal 0.
Suppose A has r nonzero rows and m − r zero rows.
Then 0 ≤ r ≤ m. Clearly, r = 0 ⇐⇒ A = O.
Example
 
0 1 4
If A =  0 0 0, then m = n = 3 and r = 2.
5 6 7
The pivot in the 1st row is 1 and the pivot in the 3rd row is 5.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Definition
A m × n matrix M is said to be in row echelon form (REF) if it
satisfies the following conditions:
(a) By a zero row of M we mean a row with all entries zero. Suppose M
has r nonzero rows and m − r zero rows. Then the last m − r rows of M
are the zero rows.
(b) The first nonzero entry in a nonzero row is called a pivot. For
i = 1, 2, . . . , r , suppose that the pivot in row i occurs in column ji . Then
we have j1 < j2 < · · · < jr . The columns {j1 , . . . , jr } are called the set
of pivotal columns of M. Columns {1, . . . , n} \ {j1 , . . . , jr } are the
nonpivotal or free columns.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Examples

     
0 4 1 0 1 4 0 1 4
The matrices  0 0
0,  5 6 7, 0 5 7
5 7 6 0 0 0 0 0 0
 
5 6 7
are not in REF. The matrix  0 1 4 is in a REF.
0 0 0

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Observation

Back substitution method works when a matrix A is in a REF. For

instance, consider

0 a12 a13 a14 a15 a16

   
b1
0 0 0 a24 a25 a26  b2 
A :=   and b :=  
 .
0 0 0 0 a35 a36  b3 
0 0 0 0 0 0 b4

where a12 , a24 , a35 are nonzero. They are the pivots. Here
m = 4, n = 6, r = 3, pivotal columns: 2 , 4 and 5 .
 T
Suppose there is x := x1 · · · x6 ∈ R6×1 such that Ax = b. Then
0x1 + · · · + 0x6 = b4 , that is, b4 must be equal to 0.
Next, a35 x5 + a36 x6 = b3 , that is, x5 = (b3 − a36 x6 )/a35 , where we can
assign an arbitrary value to the unknown x6 .

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Next, a24 x4 + a25 x5 + a26 x6 = b2 , that is, x4 = (b2 − a25 x5 − a26 x6 )/a24 ,
where we back substitute the values of x5 and x6 .
Finally, a12 x2 + a13 x3 + a14 x4 + a15 x5 + a16 x6 = b1 , that is,
x2 = (b1 − a13 x3 − a14 x4 − a15 x5 − a16 x6 )/a12 , where we can assign an
arbitrary value to the variable x3 , and back substitute the values of
x4 , x5 and x6 .
Also, we can assign an arbitrary value to the variable x1 .
The variables x1 , x3 and x6 to which we can assign arbitrary values
correspond to the nonpivotal columns 1 , 3 and 6 .

Definition
Suppose an m×n matrix A is in a REF, and there are r nonzero rows.
Let the r pivots be in the columns j1 , . . . , jr with j1 < · · · < jr , and let
the columns `1 , . . . , `n−r be nonpivotal. Then xj1 , . . . , xjr are called the
pivotal variables and x`1 , . . . , x`n−r are called the free variables.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Theorem Suppose an m×n matrix A is in a REF, and there are r nonzero
rows. Let the r pivots be in the columns j1 , . . . , jr with j1 < · · · < jr , and
let the columns `1 , . . . , `n−r be nonpivotal. Then

1. The linear system has a solution ⇐⇒ br +1 = · · · = bm = 0. This is

known as the consistency condition.
2. Let the consistency condition br +1 = · · · = bm = 0 be satisfied. Then
 T
we obtain a particular solution x0 := x1 · · · xn of the linear
system by letting xk := 0 if k ∈ {`1 , . . . , `n−r }, and then by
determining the pivotal variables xj1 , . . . , xjr by back substitution.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

3. We obtain n − r basic solutions of the homogeneous linear system
Ax = 0 as follows. Fix ` ∈ {`1 , . . . , `n−r }. Define
 t
s` := x1 · · · xn by xk := 1 if k = `, while xk := 0 if
k ∈ {`1 , . . . , `n−r } but k 6= `. Then determine the pivotal variables
xk1 , . . . , xkr by back substitution.
 t
4. Let s := x1 · · · xn be any solution of the homogeneous system,
that is, As = 0. Then s is a linear combination of the n − r basic
solutions s`1 , . . . , s`n−r . Thus the general solution of the
homogeneous system is given by
s = α1 s`1 + · · · + αn−r s`n−r , where α1 , . . . , αn−r ∈ R .
5. The general solution of Ax = b is given by

x = x0 + α1 s`1 + · · · + αn−r s`n−r , where α1 , . . . , αn−r ∈ R,

provided the consistency condition is satisfied.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Example

   
0 2 1 0 2 5 0
 and b := 1 .
0 0 0 3 5 0  
Let A := 
0 2
0 0 0 1 2
0 0 0 0 0 0 0

As we have seen earlier, here m = 4, n = 6, r = 3, pivotal columns: 2 ,

4 and 5 , and nonpivotal columns: 1 , 3 , 6 .
Since b4 = 0, the linear system Ax = b is consistent. or a particular
solution of Ax = b, let x1 = x3 = x6 = 0. Then
x5 + 2x6 = 2 =⇒ x5 = 2,
3x4 + 5x5 + 0x6 = 1 =⇒ x4 = −3,
2x2 + x3 + 0x4 + 2x5 + 5x6 = 0 =⇒ x2 = −2.
 T
Thus x0 := 0 −2 0 −3 2 0 is a particular solution.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Basic solutions of Ax = 0:  T
x1 = 1, x3 = x6 = 0 gives s1 := 1 0 0 0 0 0 ,
 T
x3 = 1, x1 = x6 = 0 gives s3 := 0 −1/2 1 0 0 0 ,
 T
x6 = 1, x1 = x3 = 0 gives s6 := 0 −1/2 0 10/3 −2 1 .

The general solution of Ax = b is given by

x = x0 + α1 s1 + α3 s3 + α6 s6 , that is,
x1 = α1 , x2 = −2 − (α3 + α6 )/2, x3 = α3 , x4 = −3 + 10α6 /3,
x5 = 2(1 − α6 ), x6 = α6 , where α1 , α3 , α6 ∈ R.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Conclusion
Suppose an m × n matrix A is in a REF, and let r be the number of
nonzero rows of A. If b ∈ Mm,1 , then the linear system Ax = b has
(i) no solution if one of br +1 , . . . , bm is nonzero.
(ii) a unique solution if br +1 = · · · = bm = 0 and r = n.
(iii) infinitely many solutions if br +1 = · · · = bm = 0 and r < n. In this
case, there are n − r free variables which give


n − r degrees of freedom .
Considering the case b = 0 ∈ Rm×1 and recalling that r ≤ m, we obtain
the following important results.
Proposition
Let A ∈ Mm,n be in REF with r nonzero rows. Then the linear system
Ax = 0 has only the zero solution if and only if r = n. In particular, if
m < n, then Ax = 0 has a nonzero solution.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Gauss Elimination Method

We have seen how to solve the linear system Ax = b when the matrix A
is in a row echelon form (REF).
We now explain the Gauss Elimination Method (GEM) by which we
can transform any A to a REF.
This involves the following two elementary row operations (EROs):

Type I: Interchange of two rows

Type II: Addition of a scalar multiple of a row to another row
Type III: Multiplication of a row by a nonzero scalar

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Lemma
If the augmented matrix [A|b] is transformed to a matrix [A0 |b0 ] by any
of the EROs, then Ax = b ⇐⇒ A0 x = b0 for x ∈ Mn,1 , that is, the linear
systems Ax = b and A0 x = b0 have the same solutions.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Transformation to REF

Let A ∈ Mm,n , that is, let A be an m × n matrix with entries in R. If

A = 0, the zero matrix, then it is already in REF.
Suppose A 6= 0.
(i) Let column k1 be the first nonzero column of A, and let some nonzero
entry p1 in this column occur in the jth row of A. Interchange row j and
row 1. Then A is transformed to
0 · · · 0 p1 ∗ · · · ∗
 
0 · · · 0 ∗ ∗ · · · ∗
A0 :=  . ,
 
 .. .
.. .
.. .
.. .. .. .. 
. . .
0 ··· 0 ∗ ∗ ··· ∗

where ∗ denotes a real number. Note: p1 becomes the chosen pivot in

row 1. (This choice may not be unique.)

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

(ii) Since p1 6= 0, add suitable scalar multiples of row 1 of A0 to rows 2 to
m of A0 , so that all entries in column k1 below the pivot p1 are equal to
0. Then A0 is transformed to
 
0 · · · 0 p1 ∗ · · · ∗
0 · · · 0 0 ∗ · · · ∗
 
A00 := 
 .. .. .. .. .. .. .
.. 
. . . . . . .
0 ··· 0 0 ∗ ··· ∗

(iii) Keep row 1 of A00 intact, and repeat the above process for the
remaining (m − 1) × n submatrix of A00 to obtain

··· ∗ ∗ ∗ ∗ ··· ∗
 
0 0 p1
0
 ··· 0 0 ··· 0 p2 ∗ ··· ∗ 
A000
 
0
:=  ··· 0 0 ··· 0 0 ∗ ··· ∗ ,
 .. .. .. .. .. .. .. .. .. .. 
. . . . . . . . . .
0 ··· 0 0 ··· 0 0 ∗ ··· ∗

where p2 6= 0 and occurs in column k2 of A000 , where k1 < k2 .

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03
Note: p2 becomes the chosen pivot in row 2. (Again, this choice may not
be unique.)
(iv) Keep rows 1 and 2 of A000 intact, and repeat the above process till
the remaining submatrix has no nonzero row. The resulting m × n matrix
is in REF with pivots p1 , . . . , pr in columns k1 , . . . , kr , and the last m − r
rows are zero rows, where 1 ≤ r ≤ m.

Notation
Ri ←→ Rj will denote the interchange of the ith row Ri and the jth row
Rj for 1 ≤ i, j ≤ m with i 6= j.
Ri + αRj will denote the addition of α times the jth row Rj to the ith
row Ri for 1 ≤ i, j ≤ m with i 6= j.
αRj will denote the multiplication of the jth row Rj by the nonzero scalar
α for 1 ≤ j ≤ m.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Examples
(i) Consider the linear system

3x1 + 2x2 + x3 = 3
2x1 + x2 + x3 = 0
6x1 + 2x2 + 4x3 = 6.

We can check that

 
3 2 1 3
 
3 2 1 3
 2 1 1 0 −→  0 −1/3 1/3 −2 
  
6 2 4 6 0 −2 2 0
 
3 2 1 3
−→  0 −1/3 1/3 −2  .
 

0 0 0 12
Here m = 3 = n, r = 2 and br0 +1 = b30 = 12 6= 0. Hence the given linear
system has no solution.
Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03
(ii) Consider the linear system

x1 − x2 + x3 = 0
−x1 + x2 − x3 = 0
10x2 + 25x3 = 90
20x1 + 10x2 = 80.

As we have already seen,

 
−1
 
1 −1 1 0 1 1 0
−1 1 −1 0 EROs 

0 10 25 90 

 −−−−→ .
0 10 25 90

 0 0 −95 −190
20 10 0 80 0 0 0 0

Here m = 4, n = 3, r = 3, pivotal columns: 1 , 2 , 3 .

Since br0 +1 = b40 = 0 and r = n, the linear system has a unique solution,
 T
namely x0 := 2 4 2 , which we had obtained by back substitution.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

(iii) Consider the linear system

3.0x1 + 2.0x2 + 2.0x3 − 5.0x4 = 8.0

0.6x1 + 1.5x2 + 1.5x3 − 5.4x4 = 2.7
1.2x1 − 0.3x2 − 0.3x3 + 2.4x4 = 2.1.

We can check that

 
3 2 2 −5 8
[A|b] =  0.6 1.5 1.5 −5.4 2.7 
1.2 −0.3 −0.3 2.4 2.1
 
3 2 2 −5 8
EROs  0 0
−−−−−→ 0 1.1 1.1 −4.4 1.1  =: [A |b ].
0 0 0 0 0

Here m = 3, n = 4, r = 2, pivotal columns: 1 , 2 ,

nonpivotal columns: 3 , 4 .

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

Since br0 +1 = b30 = 0, the linear system A0 x = b0 has a solution.
For a particular solution of A0 x = b0 , let x3 = x4 = 0. Then
1.1 x2 = 1.1 =⇒ x2 = 1,
3x1 + 2x2 = 8 =⇒ x1 = 2,
 T
Thus x0 := 2 1 0 0 is a particular solution.

Since r = 2 < 4 = n, the linear system has many solutions. For basic
solutions of A0 x = 00 , where 00 = 0,
 T
let x3 = 1, x4 = 0, so that s3 := 0 −1 1 0 ,
 T
and x4 = 1, x3 = 0, so that s4 := −1 4 0 1 ,

The general solution of A0 x = b0 is given by

x = x0 + α3 s3 + α4 s4 , that is,
x1 = 2 − α4 , x2 = 1 − α3 + 4α4 , x3 = α3 , x4 = α4 , where α3 , α4 are
arbitrary real numbers. These are precisely the solutions of the given
linear system Ax = b.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03

 Row Canonical Form (RCF)
As we have seen, a matrix A may not have a unique REF. However, a
special REF of A turns out to be unique.
An m × n matrix A is said to be in a row canonical form (RCF) or a
reduced row echelon form (RREF) if
(i) it is in a row echelon form (REF),
(ii) all pivots are equal to 1 and
(iii) in each pivotal column, all entries above the pivot are (also) equal to
0.
For example, the matrix
 
0 1 ∗ 0 0 ∗
0 0 0 1 0 ∗
A :=  
0 0 0 0 1 ∗
0 0 0 0 0 0

is in a RCF, where ∗ denotes any real number.

Prof. Shreedevi K. Masuti, IIT Dharwad Linear Algebra: Lecture 03