Matrix

length or magnitude of Vector

⎡a⎤
Example: for vector v = b , length or magnitude of v can be calculated as
⎣ c⎦

∥v∥ = √a 2 + b 2 + c 2

unit vector
a vector with a magnitude (length) of 1
1
⎡1⎤ ⎡ √2 ⎤
Examples: [ ]or 0 or [ ]or 1
1 0
0 ⎣0⎦ 1 ⎣ ⎦
√2

unit circle
unit circle consists of all points (vectors) with length 1, meaning every point on the unit circle is
a unit vector.
in 2D plane, all points (x, y) are exactly 1 unit away from origin. So from Pyhtagoras’ Theorem

also: u = [ ] = [

orthogonal
x
y

⎢⎥
cosθ
sinθ
x2 + y2 = 1

] since ∥u∥ = √cos 2 θ + sin 2 θ = √1 = 1

Two vectors u and v are orthogonal if their dot product is zero: u. v = 0

Example:
u = [1, 2] and v = [−2, 1]
u. v = 1(−2) + 2(1) = −2 + 2 = 0

orthonormal

A set of vectors is orthonormal if:

1. they are orthogonal

2. they are unit vector (norm = 1)
Example:
 ⎡ √12 ⎤ ⎡ 1
⎤
vector v 1 = and vector v2 = are orthonormal because:
√2
⎣ 1 ⎦ ⎣− 1 ⎦
√2 √2
3. v 1 ⋅ v 2 = 0 (orthogonal)
4. ∥v 1 ∥ = ∥v 2 ∥ = 1 (unit vector)

column space of matrix

set of all linear combinations of its columns. In simple terms, it represents all possible vectors
that can be formed by combining the columns of A using scalar multiplication and addition.
Example:

⎡1 2 ⎤
B= 3 4
⎣5 6 ⎦

⎡1⎤ ⎡2⎤
Columns of B are: 3 and 4
⎣5⎦ ⎣6⎦
⎧⎡1⎤ ⎡2⎤⎫
Column Span of B will be Col(B) = span ⎨ 3 , 4 ⎬
⎩⎣ ⎦ ⎣ ⎦⎭
5 6
⎡1⎤ ⎡2 ⎤
This means that any vector in the column space can be written as: c 1 3 + c 2 4 for some
⎣5⎦ ⎣6 ⎦
scalars c 1 , c 2

linear dependent
A set of vectors is linearly dependent if at least one of the vectors in the set can be written as
a linear combination of the others. This means that the vectors do not add new independent

Example:

⎢⎪⎥
directions to the space they span.

⎡1 ⎤
⎣3 ⎦
⎡2⎤
v1 = 2 , v2 = 4
⎣6⎦

Since v 1 = 2.v 2 mean v 2 is just scaled version of v 1 . second column does not add a new

⎧⎡1⎤⎫
Col(A) = span ⎨ 2 ⎬
⎩⎣ ⎦⎭
3
⎡ 1 2⎤
dimension to the column space. So Column space of matrix A = 2 4 will be
⎣ 3 6⎦
linearly independent

dot product or scalar product

It measures how much one vector extends in the direction of another. Results in a scalar (single
number). For two vectors a, b in R n
n
a ⋅ b = a 1 b 1 + a 2 b 2 +. . . +a n b n = ∑ a i b i
i=1

⎡1⎤ ⎡4 ⎤
Example: a = 2 and b = 5
⎣3⎦ ⎣6 ⎦

a ⋅ b = 1 ⋅ 4 + 2 ⋅ 5 + 3 ⋅ 6 = 4 + 10 + 18 = 32

Result: A single scalar value 32.

gaussian elimination or row reduction

There are three types of elementary row operations:

Swapping two rows,

Multiplying a row by a nonzero number,
Adding a multiple of one row to another row.

Row-Echelon Form(REF)
is a simplified form of a matrix achieved through gaussian elimination. It makes solving systems

⎢⎥
of equations and computing rank easier. A matrix is in row-echelon form if:
5. All zero rows (if any) are at the bottom of the matrix.
6. Each leading (first nonzero) entry in a row is to the right of the leading entry in the row above.
7. All entries below a leading entry (pivot) are zero.

rank of matrix
is the dimension of the vector space generated (or spanned) by its columns. This also
represents number of linearly independent columns in the matrix. Rank can be calculate using
Gaussian Elimination and then, Count the number of nonzero rows after row reduction.

inner product

generalized dot product that works in higher dimensions and different vector spaces ⟨a, b⟩ = a T b
Example: a, b from above dot product example:
 ⎡4⎤
$⟨a, b⟩ = [1 2 3] 5 = 32
⎣6⎦

Result: A single scalar value 32.

outer product
creates a matrix that represents all possible pairwise multiplications between two vectors. This
results in a transformation that captures area, projection, or tensor relationships in space
Example: a, b from above dot product example:

⎡1⎤ ⎡1 ⋅ 4 1 ⋅ 5 1 ⋅ 6⎤ ⎡ 4 5 6 ⎤
T
outer product of a,b = ab = 2 [4 5 6] = 2 ⋅ 4 2 ⋅ 5 2 ⋅ 6 = 8 10 12
⎣3⎦ ⎣3 ⋅ 4 3 ⋅ 5 3 ⋅ 6⎦ ⎣12 15 18⎦

Result: A matrix instead of a scalar.

singular matrix

Example: A = [

1
A[ ] = [
1
2(1)
3(1)
1 2
1 2
det(A) = (1 ⋅ 2) − (2 ⋅ 1) = 0

transformation matrix

]=[ ]
2
3
⎢⎥
a square matrix with a determinant of zero
] is singular matrix because it's square matrix and

a matrix that applies a linear transformation to a vector. If A is a transformation matrix and x

is a vector, then y = Ax represents the transformation of x into another vector y
Example: Scaling Transformation A = [
2 0
0 3
] if applied to a vector [ ] it will look like

This matrix scales the x-component by 2 and the y-component by 3.

square matrix
1
1

A matrix is square if it has the same number of rows and columns (n × n).
Example:

This is a 3 × 3 square matrix

⎡1 2 3⎤
A= 4 5 6
⎣7 8 9⎦
identity matrix (I)
a diagonal matrix with all its diagonal elements equal to 1 , and zeroes everywhere else

⎡1 0 0⎤
I3 = 0 1 0
⎣0 0 1⎦

symmetric matrix
A square matrix A is symmetric if it is equal to its transpose: A = A T
Example:

⎡1 2 3⎤
A= 2 5 4
⎣3 4 6⎦

Since A T = A, so A is a symmetric metrics

diagonal matrix

A matrix is diagonal if all non-diagonal elements are zero.

Example:

transpose
⎡d 1
D= 0
⎣0

⎢⎥ 0
d2
0
0⎤
0
d3⎦

is an operation that flips the matrix over its diagonal, converting rows into columns and columns
into rows.
Example: For a matrix A of size m × n (where m is the number of rows and n is the number of
columns), the transpose of A, denoted as A T , is a matrix of size n × m, such that:

​Example:

Transpose Key Properties

(A T ) ij = A ji

A=[
1 2
4 5

T
⎡1 4⎤
A = 2 5
⎣3 6⎦
3
6
]
 8. Size: if A is m × n then A T will be n × m
9. Transpose of Transpose: (A T ) T = A
10. Transpose of Sum: (A + B) T = A T + B T
11. Transpose of Product: (AB) T = B T A T (note the reverse order)
12. Transpose of a Scalar Multiple: (cA) T = cA T
13. matrix A is symmetric if A = A T

inverse matrix

is a matrix A −1 such that when it is multiplied by the original matrix A, it yield lds the identity
matrix.

A ⋅ A −1 = A −1 ⋅ A = I

⎡1 0 . . . . 0 ⎤
0 1 .... 0
where I is n × n matrix
. . .... .
⎣0 0 . . . . 1 ⎦

Conditions for the Inverse to Exist

14. square matrix: A must be n × n.
15. Nonzero determinant: det(A) ≠ 0
16. Linearly independent rows and columns: The rows (or columns) of A must form a linearly
independent set

invertible matrix

A matrix is invertible if there exists another matrix that "undoes" its transformation. In other
words, an invertible matrix A has an inverse matrix A −1 such that:

AA −1 = A −1 A = I

where I is identity matrix

det(A) ≠ 0 (determinant is nonzero).

It has no zero eigenvalues.

⎢⎥
A square matrix A (size n × n) is invertible (also called nonsingular) if and only if:

rank(A) = n (full rank, all columns are linearly independent).

It can be row-reduced to the identity matrix using Gaussian elimination.

determinant

determinant of a matrix measures how a matrix transformation scales space. It gives a scaling
factor for areas (in 2D) and volumes (in 3D) when applying a linear transformation.
Example: A = [
a b
], det(A) = ad − bc
c d

If det(A) > 0, the transformation preserves orientation

If det(A) = 0, Matrix is invertible. matrix is singular. the parallelogram collapses into a line,
meaning the transformation squashes the space into a lower dimension (no area left → the
matrix is singular). Also means, the columns are dependent.
If det(A) < 0, the transformation flips orientation (reflection occurs).
det(A) ≠ 0 means A is invertible

Example with values: A = [ ] then: det(A) = 4 ⋅ 1 − 2 ⋅ 1 = 2. So, the transformation A

4 2
1 1
scales areas by a factor of 2.

characteristic equation of matrix

(A − λI)v = 0

There can be 2 solution of this equation:

Trivial: make v = 0
Non-trivial: (A − λI) = 0. Which mean det(A = λI) should be 0. Implies, (A − λI) is a
singular matrix.

Eigenvalues and Eigenvectors

provide a way to understand how a linear transformation behaves geometrically

Imagine a unit vector along an eigenvector direction.

The eigenvalue tells whether it stretches or shrinks when transformed by the matrix.
in case of 2D plane,
- If we transform a unit square, it generally becomes a parallelogram.
- If we transform a unit circle, it becomes an ellipse.
✔ eigenvalue = How much stretching/shrinking happens.
✔ eigenvectors = Principal directions along which the transformation occurs.
✔ Parallelogram = Deformation of a square.
✔ Ellipse = Deformation of a unit circle.
Example: A = [
4 −2
]
1 1
Find Eigenvalues: Solve det(A − λI) = 0

4−λ −2
det [ ]=0
1 1−λ

(4 − λ)(1 − λ) − (−2)(1) = 0
 λ 2 − 5λ + 6 = 0
Eigenvalues for A are λ 1 = 3 and λ 2 = 2

Find Eigenvectors:
For λ 1 = 3, Solve (A − 3I)x = 0 equals [ ] [ 1 ] = [ ] equals
4−3 −2 x 0
1 1 − 3 x2 0
] [ ] = [ ] solving this gives x 1 = 2x 2 . So if x 2 = k then x 1 = 2k so eigenvector is
1 −2 x 1 0
[
1 −2 x 2 0
[ ] which is scaled version of k [ ] thus v 1 = [ ]
2k 2 2
k 1 1
For λ 1 = 2, Solve (A − 2I)x = 0 solving gives v 2 = [ ]
1
1

eigenvectors
tells us Principal directions along which the transformation occurs.
An eigenvector of a matrix A is a special vector v that does not change direction when the
matrix acts on it i.e. linear transformation is applied. Instead, it only gets stretched or
compressed by a scalar factor λ (the eigenvalue)

Av = λv

This means that when you multiply A by v, the result is just a scaled version of v.
The direction of v remains the same, only its length changes
In above example eigenvectors will be:

2 1
Q=[ ]
1 1

eigenvalues
tells us how much the eigenvector is stretched or shrunk

If λ > 1, the vector stretches.

If 0 < λ < 1, the vector shrinks.
If λ = 1, the vector’s length remains unchanged.
If λ = 0, the transformation flattens the vector to zero.
If λ < 0, the vector flips direction.
In above example eigenvalues will be: 3, 2

eigenvalues and rank

for square matrix, rank of matrix = non-zero eigenvalues

eigendecomposed
EigenDecomposition is a method of breaking down a square matrix A into its eigenvalues and
eigenvectors. It allows us to express A in the form:

A = QΛQ −1

Q: is the matrix of eigenvectors of A.

Λ: is diagonal matrix containing the eigenvalues of A
Q −1 : is inverse matrix of Q

Diagonalization

If a matrix can be eigen-decomposed, it means that it is diagonalizable

Diagonalization is the process of converting a square matrix A into a diagonal matrix D using a
transformation matrix P , suchthat : $A = PDP^{-1}$
Condition : Amatrixisdiagonalizableifithas$n linearly independent eigenvectors.

covariance matrix
is a square matrix that captures the relationships (covariances) between multiple variables in a
dataset. It helps measure how much two or more variables change together.
For a dataset with n features (variables), the covariance matrix Σ is an n × n matrix where:

Σ ij = Cov(X i , X j )

where:

X i , X j ​are two different features (columns of data).

Cov(X i , X j ) is the covariance between them.

Properties:
The diagonal elements of the covariance matrix Σ ii are the variances of each variable.
The off-diagonal elements Σ ij represent the covariance between pairs of variables.
Always symmetric: Σ ij = Σ ji
Eigenvalues are non-negative.
Σ > 0: Variables increase together (positive correlation)
Σ < 0: One variable increases while the other decreases (negative correlation).
Σ = 0: No relationship (independent variables).

Positive Semi-Definite (PSD) matrix

is a square matrix A that satisfies the following condition for all nonzero column vectors x:

x T Ax ≥ 0

A is Positive Semi-Definite if:

 1. symmetric matrix: A = A T
2. All eigenvalues are non-negative: λ i ≥ 0 for all i.
3. For any vector x, x T Ax ≥ 0.
4. Its determinant is non-negative.

Positive Definite (PSD) matrix

is a square matrix A that satisfies the following condition for all nonzero column vectors x:

x T Ax > 0

A is Positive Semi-Definite if:

5. Symmetric: A = A T
6. All eigenvalues are non-negative: λ i > 0 for all i.
7. For any vector x, x T Ax > 0.
8. Its determinant is non-negative.

Spectral Theorem
if X is symmetric matrix, we can write:

X = QDQ T

X: a real symmetric matrix

Q: an orthogonal matrix whose columns are orthonormal eigenvectors of X so Q T Q = I
D: is a diagonal matrix whole diagonal entries are real eigenvalues of X