Section 1.

2 - Systems of Linear Equations

August 29, 2018

Theorem:

Given A ∈ Rn×n and b ∈ Rn , find x ∈ Rn such that:

Ax = b.

The equation Ax = b either:

1. has no solution;
2. has exactly one solution;
3. has infinitely many solutions.

In fact, Ax = b has a unique solution x if and only if the matrix A is nonsingular.

If there is a matrix B ∈ Rn×n such that

AB = BA = I,
where I is the identity matrix, then we say that B is the inverse of A, and we write

B = A −1 .

Theorem:

Let A ∈ Rn×n . Then the following are equivalent:

1. A−1 exists.
2. There is no nonzero y such that Ay = 0.
3. The columns of A are linearly independent.
4. The rows of A are linearly independent.
5. det( A) 6= 0.
6. For any b ∈ Rn , there is exactly one x ∈ Rn such that Ax = b.

If A is nonsingular, then

1
 Ax = b =⇒ A−1 Ax = A−1 b (1)
=⇒ Ix = A−1 b (2)
=⇒ x = A −1 b . (3)
(4)

Ex. 1.2.4 page 14.

If A−1 exists, then there can be no nonzero y for which Ay = 0.

The inv and eye commands

In [1]: ?inv

search: inv invoke invmod invperm invdigamma InvalidStateException pinv erfinv

Out[1]:

inv(M)

Matrix inverse. Computes matrix N such that M * N = I, where I is the identity matrix. Com-
puted by solving the left-division N = M \ I.

Example
julia> M = [2 5; 1 3]
2×2 Array{Int64,2}:
2 5
1 3

julia> N = inv(M)
2×2 Array{Float64,2}:
3.0 -5.0
-1.0 2.0

julia> M*N == N*M == eye(2)

true

In [2]: ?eye

search: eye KeyError speye eltype keytype supertype code_typed @code_typed

Out[2]:

2
eye([T::Type=Float64,] m::Integer, n::Integer)

m-by-n identity matrix. The default element type is Float64.

eye(m, n)

m-by-n identity matrix.

eye([T::Type=Float64,] n::Integer)

n-by-n identity matrix. The default element type is Float64.

eye(A)

Constructs an identity matrix of the same dimensions and type as A.

julia> A = [1 2 3; 4 5 6; 7 8 9]
3×3 Array{Int64,2}:
1 2 3
4 5 6
7 8 9

julia> eye(A)
3×3 Array{Int64,2}:
1 0 0
0 1 0
0 0 1

Note the difference from ones.

In [3]: n = 5

A = rand(n, n)

Out[3]: 5×5 Array{Float64,2}:

0.927538 0.815802 0.854757 0.514707 0.160872
0.737663 0.322881 0.15231 0.918528 0.180922
0.714783 0.236972 0.22562 0.339328 0.570358
0.407007 0.585633 0.0429109 0.362867 0.658661
0.391564 0.439885 0.863627 0.148578 0.130783

In [4]: B = inv(A)

Out[4]: 5×5 Array{Float64,2}:

1.89175 -0.839689 1.68945 -1.27695 -2.10215
1.59023 -0.676013 -1.49884 1.3193 -1.12866
-1.14936 0.388235 0.0429452 -0.30609 2.23098
-1.5633 1.93014 -1.00208 0.432039 1.44717
-1.64676 0.0312934 0.837968 0.916197 1.35989

3
In [5]: C = [1.0 1.0; 1 1.0]
inv(C)

Base.LinAlg.SingularException(2)

Stacktrace:

[1] inv! at ./linalg/lu.jl:308 [inlined]

[2] inv(::Base.LinAlg.LU{Float64,Array{Float64,2}}) at ./linalg/lu.jl:310

[3] inv(::Array{Float64,2}) at ./linalg/dense.jl:659

[4] include_string(::String, ::String) at ./loading.jl:515

In [6]: A*B

Out[6]: 5×5 Array{Float64,2}:

1.0 6.39052e-17 -1.57417e-16 5.61103e-16 2.16786e-16
-3.27663e-16 1.0 -2.56539e-16 3.77813e-16 -3.286e-17
6.17845e-17 -1.11904e-16 1.0 1.10743e-16 1.45624e-16
8.02094e-17 -8.43108e-17 -2.40405e-17 1.0 6.38323e-17
4.10153e-17 5.85185e-17 -1.02037e-16 2.8466e-16 1.0

In [7]: I = eye(n)

Out[7]: 5×5 Array{Float64,2}:

1.0 0.0 0.0 0.0 0.0
0.0 1.0 0.0 0.0 0.0
0.0 0.0 1.0 0.0 0.0
0.0 0.0 0.0 1.0 0.0
0.0 0.0 0.0 0.0 1.0

In [8]: A*B - I

Out[8]: 5×5 Array{Float64,2}:

-3.33067e-16 6.39052e-17 -1.57417e-16 5.61103e-16 2.16786e-16
-3.27663e-16 0.0 -2.56539e-16 3.77813e-16 -3.286e-17
6.17845e-17 -1.11904e-16 -1.11022e-16 1.10743e-16 1.45624e-16
8.02094e-17 -8.43108e-17 -2.40405e-17 2.22045e-16 6.38323e-17
4.10153e-17 5.85185e-17 -1.02037e-16 2.8466e-16 0.0

In [9]: norm(A*B - I)

Out[9]: 9.332729843012912e-16

4
The \ command
In Julia or MATLAB, the linear system Ax = b is solve using the "backslash" command: \.

In [10]: ?\

search: \ .\

Out[10]:

\(x, y)

Left division operator: multiplication of y by the inverse of x on the left. Gives floating-point
results for integer arguments.

julia> 3 \ 6
2.0

julia> inv(3) * 6
2.0

julia> A = [1 2; 3 4]; x = [5, 6];

julia> A \ x
2-element Array{Float64,1}:
-4.0
4.5

julia> inv(A) * x
2-element Array{Float64,1}:
-4.0
4.5

\(A, B)

Matrix division using a polyalgorithm. For input matrices A and B, the result X is such that
A*X == B when A is square. The solver that is used depends upon the structure of A. If A is upper
or lower triangular (or diagonal), no factorization of A is required and the system is solved with
either forward or backward substitution. For non-triangular square matrices, an LU factorization
is used.
For rectangular A the result is the minimum-norm least squares solution computed by a piv-
oted QR factorization of A and a rank estimate of A based on the R factor.
When A is sparse, a similar polyalgorithm is used. For indefinite matrices, the LDLt factoriza-
tion does not use pivoting during the numerical factorization and therefore the procedure can fail
even for invertible matrices.

5
Example
julia> A = [1 0; 1 -2]; B = [32; -4];

julia> X = A \ B
2-element Array{Float64,1}:
32.0
18.0

julia> A * X == B
true

In [11]: n = 5

A = rand(n, n)
x = rand(n)

Out[11]: 5-element Array{Float64,1}:

0.968922
0.210697
0.700709
0.576107
0.328604

In [12]: b = A*x

Out[12]: 5-element Array{Float64,1}:

1.10384
1.10516
1.04277
1.52277
1.79815

In [13]: A*x - b

Out[13]: 5-element Array{Float64,1}:

0.0
0.0
0.0
0.0
0.0

In [14]: det(A)

Out[14]: -0.008517080142206717

In [15]: ### Solve Ax = b ###

x = A\b

6
Out[15]: 5-element Array{Float64,1}:
0.968922
0.210697
0.700709
0.576107
0.328604

In [16]: A*x - b

Out[16]: 5-element Array{Float64,1}:

0.0
-2.22045e-16
0.0
0.0
-2.22045e-16

In [17]: ### NEVER DO THIS!!! ###

x = inv(A)*b
### NEVER DO THIS!!! ###

A*x - b

Out[17]: 5-element Array{Float64,1}:

-2.66454e-15
-2.44249e-15
-3.33067e-15
-3.55271e-15
-2.66454e-15

Speed test: inv vs \

In [18]: n = 3000

A = rand(n, n)
x = rand(n)
b = A*x;

In [19]: @time x = inv(A)*b;

@printf "|A*x - b| = %e\n" norm(A*x - b)

1.975244 seconds (101 allocations: 138.870 MiB, 4.14% gc time)

|A*x - b| = 2.463942e-08

In [20]: @time x = A\b;

@printf "|A*x - b| = %e\n" norm(A*x - b)

7
 0.572811 seconds (14 allocations: 68.711 MiB, 15.62% gc time)
|A*x - b| = 7.392862e-11

The winner: \
Never use inv to solve a linear system.

Example: Numerical Solution of Differential Equations

The boundary value problem

x 00 − 2x 0 + x = 2t, 0 < t < 1; x (0) = 2, x 0 (1) = −4,

has the unique solution

x (t) = (t − 2)et − 3tet−1 + 2t + 4.

Finite Difference Method

Divide the interval [0, 1] into m subintervals [ti−1 , ti ] where

1
h= and ti = ih, i = 0, 1, . . . , m.
m
For h small,

x ( ti + h ) − x ( ti − h ) x ( t i +1 ) − x ( t i −1 )
x 0 ( ti ) ≈ =
2h 2h
and
x (ti+1 ) − 2x (ti ) + x (ti−1 )
x 00 (ti ) ≈ .
h2
Let xi ≈ x (ti ) and substitute into the differential equation:

xi+1 − 2xi + xi−1 x i +1 − x i −1

 
− 2 + xi = 2ti , i = 1, . . . , m − 1.
h2 2h
Also, the boundary conditions give us
x m − x m −1
x0 = 2, = −4.
h
This gives us m + 1 linear equations in m + 1 unknowns:

Ax = b
where x = ( x0 , x1 , . . . , xm ) and

   
1 0 2
m2 + m −2m2 + 1 m2 − m   2/m

   
A= . .. . .. . .. b= ..
, .
   
   .
 2 2 2
m + m −2m + 1 m − m 2(m − 1)/m
−m m −4

8
 Notice that A is a tridiagonal matrix.

In [21]: m = 10

e = ones(m-1)
dm1 = [(m^2 + m)*e; -m]
d0 = [1; (-2m^2 + 1)*e; m]
dp1 = [0; (m^2 - m)*e]

#A = spdiagm((dm1, d0, dp1), (-1, 0, 1))

A = Tridiagonal(dm1, d0, dp1)

b = [2; 2*(1:m-1)/m; -4]

@time x = A\b;

0.191772 seconds (82.90 k allocations: 4.507 MiB)

In [22]: A

Out[22]: 11×11 Tridiagonal{Float64}:

1.0 0.0 · · · ... · · · ·
110.0 -199.0 90.0 · · · · · ·
· 110.0 -199.0 90.0 · · · · ·
· · 110.0 -199.0 90.0 · · · ·
· · · 110.0 -199.0 · · · ·
· · · · 110.0 ... · · · ·
· · · · · 90.0 · · ·
· · · · · -199.0 90.0 · ·
· · · · · 110.0 -199.0 90.0 ·
· · · · · · 110.0 -199.0 90.0
· · · · · ... · · -10.0 10.0

In [23]: using PyPlot

t = linspace(0, 1, m+1)

xtrue = (t - 2).*exp(t) - 3t.*exp(t - 1) + 2t + 4

plot(t, x, label=L"x")
plot(t, xtrue, label=L"x_\mathrm{true}")

xlim([0, 1])
ylim([0, 2])

legend();

@printf("maximum(abs(x - xtrue)) = %8.1e\n", maximum(abs(x - xtrue)));

9
WARNING: exp{T <: Number}(x::AbstractArray{T}) is deprecated, use exp.(x) instead.
Stacktrace:
[1] depwarn(::String, ::Symbol) at ./deprecated.jl:70
[2] exp(::StepRangeLen{Float64,Base.TwicePrecision{Float64},Base.TwicePrecision{Float64}}) at .
[3] include_string(::String, ::String) at ./loading.jl:515
[4] include_string(::Module, ::String, ::String) at /Users/yunier/.julia/v0.6/Compat/src/Compat
[5] execute_request(::ZMQ.Socket, ::IJulia.Msg) at /Users/yunier/.julia/v0.6/IJulia/src/execute
[6] (::Compat.#inner#17{Array{Any,1},IJulia.#execute_request,Tuple{ZMQ.Socket,IJulia.Msg}})() a
[7] eventloop(::ZMQ.Socket) at /Users/yunier/.julia/v0.6/IJulia/src/eventloop.jl:8
[8] (::IJulia.##14#17)() at ./task.jl:335
while loading In[23], in expression starting on line 5
WARNING:

maximum(abs(x - xtrue)) = 1.6e-01

abs{T <: Number}(x::AbstractArray{T}) is deprecated, use abs.(x) instead.

Stacktrace:
[1] depwarn(::String, ::Symbol) at ./deprecated.jl:70
[2] abs(::Array{Float64,1}) at ./deprecated.jl:57
[3] include_string(::String, ::String) at ./loading.jl:515
[4] include_string(::Module, ::String, ::String) at /Users/yunier/.julia/v0.6/Compat/src/Compat

10
 [5] execute_request(::ZMQ.Socket, ::IJulia.Msg) at /Users/yunier/.julia/v0.6/IJulia/src/execute
[6] (::Compat.#inner#17{Array{Any,1},IJulia.#execute_request,Tuple{ZMQ.Socket,IJulia.Msg}})() a
[7] eventloop(::ZMQ.Socket) at /Users/yunier/.julia/v0.6/IJulia/src/eventloop.jl:8
[8] (::IJulia.##14#17)() at ./task.jl:335
while loading In[23], in expression starting on line 15

11