Lecture 2: Basic Linear Algebra and Matrix Operations

Notations:

Scalars (Numbers): Normal font lower case letters

e.g a b X Y

R the set of all real numbers

the set of all complex numbers
a ER a is a real number
bE C b is a complex number

Unless specified a scalar is assumed real default

by

Vectors: bold lower case letters in printing (standard lower case letters in hand-writing with

vector notation).

iiaiipfor
or dropped
the ith entry of a

simplicity
where ai are scalars

Unless specified we refer vectors as column vectors

The transpose of vector a is denoted by aT and given by

ā a di an

Obviously ā is a length n row vector

Vector spaces: R c

ā ER ā is a length n column real vector

 āec ā is a length n column complex vector

Real
spaces will be assumed if not specified

Special vectors:

ēi 1
f
length-n basis vector:
the ith component

length-n zero vector:

0
111
length-n vector of all ones:
i 点 ei
1

Matrices: bold upper case letters in printing (standard upper case letters in hand-writing).

ㄨm ㄨm
An nxm matrix A ER for A Ec

called element
A caijinxm
or dropped
or A
1 or entry

an dnz anm
for simplicity
where Qij ER for CI for i 1 n j 1

Special matrices:
m I.

Identity matrix
ii
o inn
Matrix spaces: Rmxn cmxn

A E Rm A is an mxn real matrix

AE Rm A is an mxn complex matrix

Since a vector ā ER can also be viewed as an nxl matrix

i.e ā ER
R R
and c cn
the
Similarly space of all length n row vectors is R'ㄨ or4

m
Any matrix A ER for cnxmg can be written in a column vector form as
A ān ān
where ER or ai
c j 1 m are columns of
Similarly, we can also write RnXm A AE in a row column vector form as

orcnxm A
㒞式
where ācil E Rm 1or 4m i 1 n are rows of A

Matrix transpose:

Given A 1Qij nxm its transpose AT is defined by

AT Qji mxn
 Example A
A
lǎaii lǎ 品
Obviously AT T A

Hemitian (Also called Adjoint and Conjugate transpose)

A Iaijlnxm A G mxn

Example
A
1 剿
A
lāqgi
ㄨm

If A ER then At AT
ㄨm
If A E4 then A AT

Symmetric and Hermitian

A ER is symmetric if A AT
ㄨ
A E4 is Hermitian if A At

Example A 1 is symmetric
121 31
Hermitian
ft I
A is

Some basic Operations of Matrices and Vectors

Addition and subtraction

Given two matrices A Idij 了nxm and B bij了nxm their

 ㄨm
summation difference denoted by C is in R and given by
a b d 2 b12 aim bm
C A B b21 922 b22 bzm
a211 it Q2mi

an bnianztbuzii.am bnm

2) Scalar product

Xm
Let be a scalar and AER be a matrix. The scalar product of c and A, denoted
ER
by B, is in Rnxm and given by

can Caz I aim

B CA can cazm
1921 i
can canz canm

pseudo code:

for i 1 n
for j 1 m

for j lim for i l in

bij caij bij caij

end end

end end
ji loop
ij loop
B is obtained Column
by colum
B is obtained row by row

Remark: The two implementations may have di erent performances, depending on how the

matrices are stored. If the matrices is stored row-by-row, then ij-loop may be better, because it

can save communication cost of Cache and CPU.

3) Inner product:

Given two vectors ā and in

R ca
叫
their inner product is

b aibi
pseudo code:

inner_prod 0

for i 1 n

inner.prod inner_prod tai bi

end
Remark: If <a,b>=0, then we say a, b are perpendicular or orthogonal.

4) Matrix-Vector product:

• Let A=
aij mxn E RM and x= Xi nx ER
Then the matrix-vector product of A and x, denoted by b, is a vector in , given by
R

5 器i.iii.in
l Hǎiiiit i
amxitdmzxz
测
9mn加

• It is easy to see that

cā 5 a.az and b.bz bn

劚 1㔍
ā巧 Tiā
i.e., inner product is a special case of matrix-vector product.
• If we write A in row vector form

A where ā 1ai aiz ain

1籪1

then
5 A
iàiii
州 州
āǎi
āmix̅ aim
i.e. entries of 万 are inner products of row vectors āi and x̅
This gives an implementation of b=Ax

pseudo code 1:

ij-loop i l
for m

bi 0

for j l n inner product

bi bit aij bi sail x̅

end

end

• If we write A in column vector form

A ai ai āini了 where aiji ER j 1 n are columns

of A
then 万 A x̅ ān ām ām了 xiāintxzā xnān
1劐
This reveals that b is a linear combination of columns āgi of A

This gives another implementation of b=Ax:

pseudo code 2: Set vector 万 ō

ji loop for j 1 n

for i l m
add to by jail
i bit xjaij
end
end

• Comparison of two implementations

ij loop ji loop

Outer loop on rows Outer loop on columns

Inner loop on columns Inner loop an rows

Their e ciency depends on how the matrix A stored.

If A is stored column-wise, better

ji loop is

row-wise,
ij loop is better

5) Matrix-Matrix product

• Let A Qij E Rm and B bij ER

Then the product of A and B, denoted by C, is an mxp matrix, given by

C
Cij E Rm where Cij 葥aikbkj
 n

n
Pj
m m
贸
A
ithrǐǐ Bjthidumn C

lineout
it

• Obviously, inner product and matrix-vector product are special cases of matrix-matrix

product.

• We have di erent implementations

a) By the de nition above

for i li m
for j 1 P

for j l P for i li m

Ci 0 Cij 0
j
for k l in for k 1in

Cij Cij aikbkj Cij Cijtdikbkj

end end
end end

end end
ijk loop jik loop

is computed row byrow The product is computed

The product
column column
by
b) Write A in column form and B in row form, i.e.

A 1āin an
fÜl叩
ācm B
 Then

C AB ānai Ü
any 䄗
箭 āim 151k
We get another implementation

Initialize matrix C 0

for k l in

for i l m

for j 1P add cbyaikib

Cij Cijtdikbkj
end

end
end

kij loop

Remark: Any permutation of the sequence (i, j, k) is a possible implementation. So we have

3!=6 di erent implementations of C=AB.

• Computational Complexity

What is the number of scalar operations (+, -, x ) in the algorithms?

flops
1) Inner product: Given a, b. , what is the complexity for <a,b>?
ER
<a,b>= aibi
 inner_prod 0

for i 1 n

inner.prod inner_prod tai bi

end

11 11 2n O n

iigo
What is O(n)?

O(n) is a function of n that satis es 1

品 興 for some constant C.

2) Matrix-Vector product of E Rm
X
A x ER

y=Ax can be computed by

yi 哥Qijxj i 1 m

for i l m

yi 0

for j lin

yi y i t
aijxj
end
end

器 2
Number
of flops 器
信 1 1 是 2n 2mn 三 0 mn

In particular if m n then AER and x ER y Ax costs 2m 0 n

XP
3)Matrix-Matrix product AE Rmx BE R
C=AB E Rm with Cij Ědikbkj i 1 m j 1 p
 for i l m

for j 1 P

in

bfāfl
en

end

end
Number
of flops
11 哥哥 2 哥哥 zn
Ě 1哥 10 1

2np 2nmp O nmp

In particular if m n p then
ㄨ
A ER BE R and C AB costs 2m O n3

• There are other implementations of these basic operations

1) loop ordering

2) the complexity of matrix-matrix product can be reduced for m=n=p.

e.g. the fastest matrix-matrix product algorithm is O(n2.32 ), people guess that the best

possible complexity for matrix-matrix is O(n2 ).

3)The e ciency depends not only on computational complexity, but many other issues.

• All these basic matrix computation subroutines come with your computer. We don’t need to

write our own ones.

 1) Standard interfaces (www.netlib.org/blas)

BLAS (Basic Linear Algebra Subroutine)

2) Intel———MKL (Math kernel Lib)

AMD———ACML (AMD Core Math Lib)

CuBLAS

• Matrix Partitions and Block Matrices

We will often write a matrix A into a “block” form as

X
AERm
n The blocks in the same
An A A19
row should have the
A
f Azi Azz

Api Apz
A29

Apq
IN same number of rows

columns
have the same number of
They

where
Aij ER are sub_matrices of A and mit m
p m

n that tnq n

We call it a partition of A

Example: A
品 品 ER
lǎii
931 932 933

ˋ
 iq
A
9
lǎii
器
们 Ayǎi 2
931A292 㠮 启
A valid matrix partition Not a valid matrix partition

• Matrix additions, multiplications, transpose in matrix partition forms are formally the same as

those in standard forms.

Recall A 9ij ij
a) Transpose: A Aijij AT Aij i AT 1aij i
b) Addition: A Aij ij
B Bij i
A B Aij Bij ijifAijtj
wellis
defined

c) Multiplication: B
A Aij ij Bij ij
Then AB AikBkj provided AikBkj are well

defined AE
Example: We usually write

RmX A ainai aini

into column vector form.

whereaij ERmisthei th column

Let x̅ e Rn be a vector
1劐
because
xj is a scalar
Then
此
A x̅ āin an ām
1劐 哥āg xj 房xjāij

So we see A x̅ ER is a linear combination of columns

of A
Example: Consider A E Rm BE R and C A BE Rm

and partition A, B by

A an āizi aim āgi ERm are columns

of A

B of rows of B
ER are transpose
rj
虌成
4萨
5
Then āini
āinām

鬬
C AB

点 āikl 5ᵗʰ rank 1 matrix

called tensor product outer product

of āik and 5k
C is the sum of outer products

Partition A B in

A 品
fǜi
where ā ER are transpose of rows of A

B bin bizi n.ba where 5 ER are columns

of B

Then AB āi bin

f
kā 万51
bptn.bg ij
ddjikn
i ㄒ
 the ci j th entry of C is the inner product of
i throw of A and j th column
of B

More concepts of matrices:

Given a matrix A cām ām ānl e Rm

where
āijiERm are columns of A.
• The range of A is

Ran A A x̅ I x̅ ER
āgilx ER Rm
Col AI

It is a subspace of Rm
• The null space of A is

NullAl x ER I Ax̅ 0了 R
kerA
• The rank of A is

rank A dim RanAl dim ColA dim RowA

Then rank A min m n

Vector and Matrix Norms

To measure how accurate our algorithm is, we need to de ne metric, i.e., distance between

two scalars, two vectors, or two metrics.

• Metric for scalars

For two scalars x y ER their distance is lx

yl ci
• Metric for vectors

For two vectors b ER , we need to generalize the absolute function for scalars to

vectors.

—For a scalar, the absolute function satis es

1) IX1 0 x ER and Ix1 0 x 0

2)
Ixy1 1Allyl x YER
3)
Ixtyl IN 1YI x y ER
—Norm: a generation of absolute value to vectors.

De nition: A function
R 1 R , denoted by ||x|| for any x̅ER , is called a norm, if it

satis es: 1) 11711 0 x̅ ER and 11x̅11 0 不0

2) 11xx̅11 12111 11 αER x̅ ER
3)
11x̅ ill 11x̅11 17511 x̅ JER
Remark: a) Condition 1) is to make a norm a magnitude

b) Condition 2) says that the norm of a scaling of vectors is the scaling of the norm.
戏
zi Nxx̅ll 1α111 11

c) Condition 3) is also known as the triangle inequality

The length indirect path the red

iiy in
11517 from 0
of
to P is longer than
0
min sg that of the direct path the blue
d) Given two vectors x̅ JER , their distance is 11T T11
Example 1: Euclidean norm (2-norm)

11x̅112 xi x̅ER proof of Example 2

Example 2: Manhattan norm (1-norm) 11x̅ll 平1Xil 0 and

11TH Ixil x̅ ER
11 11 0 年1xil 0

lxil 0 i xi 0 7
Example 3: Max norm ( v0 -norm) x̅ Ō
11αx̅11 1Xxil 点IN Ixi
11TH o nmyn IXil x̅ER
1α1 lxil 12111x11
Example 4: p-norm (p>= 1)
11 J11 产1Xityil
Let p be a positive integer Define 平lxiltlyil 严刚 平以
声
11x̅llp 1XiP 万11 11511
11

不511
y i 111
iillx y
lb. l 3
11不511

• Metric for Matrices

Similar to vector norms, we can de ne matrix norm.

X
De nition: A function R R denoted by 11All for any AERm , is called a norm, if it
satis es: 1) IIAll 0 A ERm and IIAll 0 A 0

2) 11αAll 1α111All αER AE Rm

X
3) 11AtBll 11All IIBIl A BE Rm
Since mxn matrix can be viewed as long vectors of length man, i.e.
䨊
Rn nm lǎiiǎ iii
We can apply norms for vectors to obtain norms for matrices.

Example 1: Frobenius norm (vector 2-norm)

11AllF 1 aij AERm

Example 2: vector p-norm
X
11Allvec
y laij1 AERm

We view matrices as linear transformations:

A Rm R ts Rm
x̅ 1 A x̅

We can de ne matrix operator p-norm (or simply matrix p-norm) as

11A1h
荷肾
器
In other words, matrix p-norm of A is the largest magnifying factor of A acting from R with

p-norm to Rm with p-norm.

Theorem:
IIAllp 11Ax̅llp
朤
Example 1: 2-norm

11All2 11A万112 Imax eigenvalue of ATA

ping
Example 2: 1-norm

the maxi norm of colu

IIAN Engin llāijill
11All 11All is
不
proof We prove the theorem by showing
HAN
nyn11ill and
gn11可
11AN 11
1g 然
First HAN gnllāc 111
we prove
然
For an arbitrary x̅ satisfying 11x11 1

āmai ām 了 11 11Ěixiāugll
11AAll 11
劐
triangle inequality

剖xiācnll
Ei In111ācc 11
1M ngn11可111
11不 my11ill

ny.liājill
Hence 11Aill
my llāij 11
for any 11N 1
Taking over all x̅ satisfying 11 11 1
11A I
朤 gnllāill
Now we prove
然
11AN
gn11可11

Let j ie 11aj.lt
ǐǎynllāill
arggllāgill

my llāiill
Then llltei.tl 11āijn11

Notice that llej.tl 1 we have

11Aēj 11 inynllācjill
喌911ANI
 Example For the identity matrix I

11Ill ǐrzgynllejll 1

Example: -norm

11Allo 11ANo imgymllā ll

器
where A E Rm
ijf

In other words 11Allo is the maximum 1 norm of row vectors

For the matrix p-norm, besides 1),2) and 3) in the de nition of matrix norm, the matrix p-norm

also satis es:

4) 11ATllp 11Allp11Nlp A E Rm and x̅ ER

以
5) 11ABllp 11All 11Bllp A ER and BE R
p