Working with the Basis Inverse over a Sequence of

Iterations
Robert M. Freund
February, 2004
c 2004 Massachusetts Institute of Technology.
1
_ _

1 Equations Involving the Basis Matrix
At each iteration of the simplex method, we have a basis consisting of an
index of variables:
B(1), . . . , B(m) ,
from which we form the basis matrix B by collecting the columns A
B(1)
, . . . , A
B(m)
of A into a matrix:
. B := A
B(1)
A
B(2)
. . . A
B(m1)
A
B(m)
In order to execute the simplex method at each iteration, we need to be
able to compute:
T T
x = B
1
r
1
and/or p = r
2
B
1
, (1)
for iteration-specic vectors r
1
and r
2
, which is to say that we need to solve
equation systems of the type:
T
Bx = r
1
and/or p
T
B = r
2
(2)
for x and p.
2 LU Factorization
One way to solve (2) is to factor B into the product of a lower and upper
triangular matrix L, U:
B = LU ,
and then compute x and/or p as follows. To compute x, we solve the fol-
lowing two systems by back substitution:
First solve Lv = r
1
for v
Then solve Ux = v for x.
2
To compute p, we solve the following two systems by back substitution:
T
First solve u
T
U = r
2
for u
Then solve p
T
L = u
T
for p.
It is straightforward to verify that these procedures yield x and p that
satisfy (2). If we compute according to these procedures, then:
T
Bx = LUx = Lv = r
1
and p
T
B = p
T
LU = u
T
U = r
2
.
3 Updating the Basis and its Inverse
As the simplex method moves from one iteration to the next, the basis
matrix B changes by one column. Without loss of generality, assume that
the columns of A have been re-ordered so that
B := [ A
1
| . . . | A
j1
| A
j
| A
j+1
| . . . | A
m
]
at one iteration. At the next iteration we have a new basis matrix

B of the
form:

B := [ A
1
| . . . | A
j1
| A
k
| A
j+1
| . . . | A
m
] .
Here we see that column A
j
has been replaced by column A
k
in the new
basis.
Assume that at the previous iteration we have B and we have computed
an LU factorization of B that allows us to solve equations involving B
1
. At
the current iteration, we now have

B and we would like to solve equations

involving

B
1
. Although one might think that we might have to compute

an LU factorization of B, that is not the case. Herein we describe how

the linear algebra of working with

B
1
is computed in practice. Before we
describe the method, we rst need to digress a bit to discuss rank-1 matrices
and rank-1 updates of the inverse of a matrix.
3
_ _
3.1 Rank-1 Matrices
Consider the following matrix:
_ _
2 2 0 3
_
4 4 0 6
_
_ _
W =
_ _
14 14 0 21
.
10 10 0 15
W is an example of rank-1 matrix. All rows are linearly dependent and all
columns are linearly dependent. Now dene:
_ _
1
_
2
_
T
u =
_ _
and v = ( 2 2 0 3 ) .
_
7
_
5
If we think of u and v as n 1 matrices, then notice that it makes sense
to write:
_
1
_ _
2 2 0 3
_
W = uv
T
=
_
_
_
2
7
_
_
_
( 2 2 0 3 ) =
_
_
_
4
14
4
14
0
0
6
21
_
_
_
.
5 10 10 0 15
In fact, we can write any rank-1 matrix as uv
T
for suitable vectors u and v.
3.2 Rank-1 Update of a Matrix Inverse
Suppose we have a matrix M and we have computed its inverse M
1
. Now
consider the matrix

M := M + uv
T
M
1
for some rank-1 matrix W = uv
T
. Then there is an exact formula for

based on the data M

1
, u, and v, which is called the Sherman-Morrison
formula:

T
M
1
Property. M is invertible if and only if v u = 1, in which case
M
1
uv

= I
M
1 T
M
1
. (3)
1 + v
T
M
1
u
4
_ _ _ _
_ _ _ _
_ _
_ _
_ _ _ _
Proof: Let
_ _
M
1 T
uv
Q = I M
1
.
1 + v
T
M
1
u

Then it suces to show that MQ = I, which we now compute:

uv
MQ = M + uv
T
I
M
1 T
M
1
T
M
1
1+v u
M
1

M
1
uv
T
M
1
= M + uv
T

T
M
1
1+v u
= I + uv
T
M
1

uv
T
M
1

uv
T
M
1
uv
T
M
1
T
M
1 T
M
1
1+v u 1+v u
1
= I + uv
T
M
1
1
T
M
1

v
T
M
1
u
T
M
1
1+v u 1+v u
= I
q.e.d.

3.3 Solving Equations with M using M

1
Suppose that we have a convenient way to solve equations of the form Mx =
b (for example, if we have computed an LU factorization of M ), but that
we want to solve the equation system:

Mx = b .
Using (3), we can write:

M
1 T
x = M
1
b = I
uv
M
1
b .
1 + v
T
M
1
u
Now notice in this expression that we only need to work with M
1
, which
we presume that we can do conveniently. In fact, if we let
2
x
1
= M
1
b and x = M
1
u ,
we can write the above as:

M
1 T 2 T T 1
1 2
x = M
1
b = I
uv
M
1
b = I
x v
x = x
1

v x
x .
T
x
2 T
x
2
1 + v
T
M
1
u 1 + v 1 + v
5
_ _

Therefore we have the following procedure for solving Mx = b:

1
Solve the system Mx
1
= b for x
2
Solve the system Mx
2
= u for x
v x 2
Compute x = x
1

T 1
.
1+v
T
x
2
x
3.4 Computational Eciency
The number of operations needed to form an LU factorization of an n n
3
matrix M is on the order of n . Once the factorization has been computed,
the number of operations it takes to then solve Mx = b using back substi-
tution by solving Lv = b and Ux = v is on the order of n
2
. If we solve

Mx = b by factorizing M and then doing back substitution, the number
2
of operations needed would therefore be n
3
+ n . However, if we use the
above rank-1 update method, the number of operations is n
2
operations for
each solve step and then 3n operations for the nal step, yielding a total
operation count of 2n
2
+ 3n. This is vastly superior to n
3
+ n
2
for large n.
3.5 Application to the Simplex Method
Returning to the simplex method, recall that we presume that the current
basis is:
B := [ A
1
| . . . | A
j1
| A
j
| A
j+1
| . . . | A
m
]
at one iteration, and at the next iteration we have a new basis matrix

B of
the form:

B := [ A
1
| . . . | A
j1
| A
k
| A
j+1
| . . . | A
m
] .
Now notice that we can write:

_ _
T
B = B + (A
k
A
j
) e
j
,
where e
j
is the j
th
unit vector (e
j
has a 1 in the j
th
component and a 0 in
every other component). This means that

B is a rank-1 update of B with

j
u = (A
k
A
j
) and v = e . (4)
6
_ _
_ _
_ _
_ _
_ _
_ _ _
_ _
_ _ _
If we wish to solve the equation system

Bx = r
1
, we can apply the method
of the previous section, substituting M = B, b = r
1
, u = (A
k
A
j
) and
v = e
j
. This works out to:
1
Solve the system Bx
1
= r
1
for x
2
Solve the system Bx
2
= A
k
A
j
for x
1
(e
j
)
T
x
2
Compute x = x
1

1+(e
j
)
T
x
2
x .
This is ne if we want to update the basis only once. In practice, how-
ever, we would like to systematically apply this method over a sequence of
iterations of the simplex method. Before we indicate how this can be done,
we need to do a bit more algebraic manipulation. Notice that using (3) and
(4) we can write:

uv
B
1
= I
B

T
1
B

T
1
B
1
1+v u
B
1
(A
k
A
j
)(e
j
)
T
B
1
= I
1+(e
j
)
T
B
1
(A
k
A
j
)
.
Now notice that because A
j
= Be
j
, it follows that B
1
A
j
= e
j
, and substi-
tuting this in the above yields:
B
1
I
(B
1
A
k
e
j
)(e
j
)
T
B
1

=
(e
j
)
T
B
1
A
k

= EB
1
where
e
j
_
T

B
1
A
k
e
j
E = I .
(e
j
)
T
B
1
A
k
w be the solution of the system B Furthermore, if we let w = A
k
, that is,
E as w = B
1
A
k
, then we can write

e
j
_
T

w e
j
E = I .
(e
j
)
T
w
We state this formally as:
7
_ _
_ _ _
_ _

Property A. Suppose that the basis B is obtained by replacing the j

th
column of B with the new column A
k
. Let w be the solution of the system
B w = A
k
and dene:
e
j
_
T

w e
j
E = I .
(e
j
)
T
w
Then
B
1

= EB
1
. (5)
Once we have computed w we can easily form E. And then we have

from above:

x = B
1
r
1
= EB
1
r
1
.
Using this we can construct a slightly dierent (but equivalent) method for
solving

Bx = r
1
:
Solve the system B w w = A
k
for
we
j
)(e
j
)
T
Form and save the matrix

I
(
E =
(e
j
)
T
w
1
Solve the system Bx
1
= r
1
for x

Compute x = Ex
1
.
Notice that
_ _
1 c
1
_
1 c
2
_
_ _
_ . . _
. .
_ . .
_
_ _
E =
_ _
c
j
_ _
_ _
. .
_ . . _
. .
c
m
1
where
_
w e
j
_

c = .
(e
j
)
T
w

E is an elementary matrix, which is matrix that diers from the identity

matrix in only one column or row. To construct

E we only need to solve

8
_
w = A
k
, and that the information needed to create

B E is the n-vector w
and the index j. Therefore the amount of memory needed to store

E is just
n + 1 numbers. Also the computation of

Ex
1
involves only 2n operations if
the code is written to take advantage of the very simple special structure of

E.
4 Implementation over a Sequence of Iterations
Now let us look at the third iteration. Let

B be the basis at this iteration.

We have:

B := [ A
1
| . . . | A
i1
| A
i
| A
i+1
| . . . | A
m
]
at the second iteration, and let us suppose that at the third iteration we
replace the column A
i
with the column A
l
, and so

B is of the form:

B := [ A
1
| . . . | A
i1
| A
l
| A
i+1
| . . . | A
m
] .
Then using Property A above, let

w be the solution of the system

w = A
i
. Then B

B
1

B = E

(6)
where
_ _
_ _

w e
i
e
i
_
T

_
E =
_
I

.
(e
i
)
T
w
1

B
1
B = E

= E

It then follows that

EB
1
, and so:
1
B = E

EB
1
. (7)
B by forming E and

Therefore we can easily solve equations involving

E
and working with the original LU factorization of B.
This idea can be extended over a large sequence of pivots. We start
with a basis B and we compute and store an LU factorization of B. Let
our sequence of bases be B
0
= B, B
1
, . . . , B
k
and suppose that we have
computed matrices E
1
, . . . , E
k
with the property that
(B
l
)
1
= E
l
E
l1
E
1
B
1
, l = 1, . . . , k .
9
Then to work with the next basis inverse B
k+1
we compute a new matrix
E
k+1
and we write:
(B
k+1
)
1
= E
k+1
E
k
E
1
B
1
.
This method of working with the basis inverse over a sequence of it-
erations eventually degrades due to accumulated roundo error. In most
simplex codes this method is used for K = 50 iterations in a row, and then
the next basis is completely re-factorized from scratch. Then the process
continues for another K iterations, etc.
5 Homework Exercise
1. In Section 3.2 we considered how to compute a solution x of the equa-

tion Mx = b where M = M + uv
T
and we have on hand an LU
factorization of M . Now suppose instead that we wish to compute a

solution p of the equation p

T
M = c
T
for some RHS vector c. Using
the ideas in Section 3.2, develop an ecient procedure for computing
p by working only with an LU factorization of M .
2. In Section 3.5 we considered how to compute a solution x of the equa-
tion

B diers from B by one column, and we have on Bx = r

1
where

hand an LU factorization of B. Now suppose instead that we wish

T
to compute a solution p of the equation p
T

B = r
2
for some vector
r
2
. Using the ideas in Section 3.5, develop an ecient procedure for
computing p by working only with an LU factorization of B.
10