Linear Programming

The process of minimizing a linear objective function

subject to a finite number of linear equality and
inequality constraints.
The word programming is historical and predates
computer programming.
Example applications:
airline crew scheduling
manufacturing and production planning
telecommunications network design
Few problems studied in computer science have
greater application in the real world.

CSE 421
Introduction to Algorithms
Autumn 2010

Linear Programming

Linear Programming

An Example: The Diet Problem

An Example: The Diet Problem

A student is trying to decide on lowest cost diet that

provides sufficient amount of protein, with two choices:
steak: 2 units of protein/pound, $3/pound
peanut butter: 1 unit of protein/pound, $2/pound
In proper diet, need 4 units protein/day.
Let x = # pounds peanut butter/day in the diet.
Let y = # pounds steak/day in the diet.
Goal: minimize 2x + 3y (total cost)
subject to constraints:
This is an LP- formulation
x + 2y 4
of our problem
x 0, y 0
Linear Programming

Linear Programming

Goal: minimize 2x + 3y (total cost)

subject to constraints:
x + 2y 4
x 0, y 0
This is an optimization problem.
Any solution meeting the nutritional demands is called
a feasible solution
A feasible solution of minimum cost is called the
optimal solution.

Linear Programming

Linear Program - Definition

Feasible Set

A linear program is a problem with n variables

x1,,xn, that has:
1. A linear objective function, which must be
minimized/maximized. Looks like:
max (min) c1x1+c2x2+ +cnxn
2. A set of m linear constraints. A constraint
looks like:
ai1x1 + ai2x2 + + ainxn bi (or or =)

Each linear inequality divides n-dimensional

space into two halfspaces, one where the
inequality is satisfied, and one where its not.
Feasible Set : solutions to a family of linear
inequalities.
The linear cost functions, defines a family of
parallel hyperplanes (lines in 2D, planes in
3D, etc.). Want to find one of minimum cost

must occur at corner of feasible set.

Note: the values of the coefficients ci, ai,j are

given in the problem input.
Linear Programming

Linear Programming

Visually
x= peanut butter, y = steak

Optimal vector occurs at some

corner of the feasible set!
x=0

x=0
Opt:
x=0,
y=2

feasible set

feasible set
2x+3y=0

y=0

y=0

x+2y=4
Linear Programming

Minimal price of
one protein unit
= 6/4=1.5

x+2y=4
Linear Programming

2x+3y=6

Optimal vector occurs at some

corner of the feasible set!

Standard Form of a Linear Program.

Maximize c1x1 + c2x2 ++ cnxn
subject to 1 j n aijxj bi
i=1..m
xj 0
j=1..n

x=0
An Example
with 6
constraints.
feasible set

Maximize c T x
subject to Ax b and x 0
y=0

Linear Programming

10

The Feasible Set of Standard LP

Putting LPs Into Standard Form

Intersection of a set of half-spaces, called a

polyhedron.
If its bounded and nonempty, its a polytope.

Min to Max
Change cjxj to (-cj)xj

Change = constraints to < and > constraints

Add non-negativity constraints by substituting
xi - xi for xi and adding constraints xi > 0, xi > 0.

Change > constraints to < constraints by

using negation

Linear Programming

Linear Programming

11

There are 3 cases:

feasible set is empty.
cost function is unbounded on feasible set.
cost has a maximum on feasible set.
First two cases very uncommon for real problems
in economics and engineering.
Linear Programming

12

LP formulation: another example

Solving LP
There are several polynomial-time algorithms
that solve any linear program optimally.
 The Simplex method (later) (not polynomial time)
 The Ellipsoid method (polynomial time)
 More

These algorithms can be implemented in various

ways.
There are many existing software packages for
LP.
It is convenient to use LP as a black box for
solving various optimization problems.
Linear Programming

13

LP formulation: Bobs bakery

Flour
Eggs
Sugar

Bagels Muffins
5
4
2
4
1
2

Revenue 10

Avail.
50
30
20

2 4
1 2

12
b = 10 12

Maximize 10x1+12x2
s.t. 5x1+4x2 50
2x1+4x2 30
x1+2x2 20
x1 0, x2 0

50
c = 30
20

Maximize bx
s.t. Ax c
x 0.
Linear Programming

Linear Programming

14

In class exercise: Formulate as LP

5 4
A=

Bobs bakery sells bagel and muffins.

To bake a dozen bagels Bob needs 5 cups of flour, 2
eggs, and one cup of sugar.
To bake a dozen muffins Bob needs 4 cups of flour, 4
eggs and two cups of sugar.
Bob can sell bagels in $10/dozen and muffins in
$12/dozen.
Bob has 50 cups of flour, 30 eggs and 20 cups of sugar.
How many bagels and muffins should Bob bake in order
to maximize his revenue?

15

You want to invest $1000 in 3 stocks, at most $400

per stock
price/share
dividends/year
stock A
$50
$2
stock B
$200
$5
stock C
$20
0
Stock C has probability of appreciating to $25 in
a year, and prob of staying $20.
What amount of each stock should be bought to
maximize dividends + expected appreciation over
a year?
Linear Programming

16

In class exercise: Formulate as LP

Reduction Example: Max Flow

Solution: Let xa, xb, and xc denote the amounts of A,B,C

stocks to be bought.

Max Flow is reducible to LP

Variables: f(e) - the flow on edge e.
Max eout(s) f(e) (assume s has zero in-degree)
Subject to
(Edge condition)
f(e) c(e), eE

Objective function:
Constraints:

e in(v) f(e) - e out(v) f(e) = 0 , vV-{s,t}

(Vertex condition)
f(e) 0, eE
Linear Programming

17

Example - L1 Best Fit

n

a x
i =1

ij

= bj 1 j m

Linear Programming

L1 Best Fit Reduced to LP

Example:
2x + 3y = 6
3x - 5y = 2
4x + 5y = 7

minimize

e
j=1

j
n

If m > n then overconstrained.

Find x1,,xn to minimize

subject to b j aij x i e j
i=1

a x b

a x b
j=1 i=1

ij

18

L1 norm

i=1

ij

ej

ej 0
for 1 j m

Linear Programming

19

Linear Programming

20

Primal: Maximize cTx subject to Ax b, x 0

A Central Result of LP Theory:

Duality Theorem

Dual: Minimize yTb subject to yTA cT, y 0

In the primal, c is cost function and b was in

the constraint. In the dual, reversed.
Inequality sign is changed and maximization
turns to minimization.
Primal:
Dual:

Every linear program has a dual

If the original is a maximization, the dual is a
minimization and vice versa
Solution of one leads to solution of other
Primal: Maximize cTx subject to Ax b, x 0
Dual: Minimize yTb subject to yTA cT, y 0
If one has optimal solution so does other, and their
values are the same.

Linear Programming

maximize 2x + 3y
s.t x+2y < 4,
2x + 5y < 1,
x - 3y < 2,
x 0, y 0
21

Weak Duality Theorem

j=1

i=1

2p+5q -3r > 3,

p,q,r 0

Linear Programming

c jx j
j=1

c j x j bi y i

m
aij y i x j

j=1 i=1

m n

= aij x j y i
i=1 j=1

b y
i=1

Linear Programming

22

Proof of Weak Duality

Theorem: If x is a feasible solution to

the primal problem and y is a feasible
solution to the dual problem then
n

minimize 4p +q + 2r
s.t p+2q + r > 2,

23

Linear Programming

24

Simple Example of Duality

Duality Theorem

Diet problem: minimize 2x + 3y

subject to x+2y 4,
x 0, y 0
Dual problem: maximize 4p
subject to p 2,
2p 3,
p0
Dual: the problem faced by a druggist who sells
synthetic protein, trying to compete with peanut
butter and steak

If x* is optimal for the primal and y* is

optimal for the dual, then
n

j=1

i=1

c j x *j = bi yi*

Linear Programming

25

26

Whats going on?

Simple Example
The druggist wants to maximize the price p,
subject to constraints:
synthetic protein must not cost more than protein
available in foods.
price must be non-negative or he wont sell any
revenue to druggist will be 4p

Solution: p 3/2
objective value = 4p = 6

Not coincidence that its equal the minimal cost in
original problem.

Linear Programming

Linear Programming

27

Notice: feasible sets completely different for primal

and dual, but nonetheless an important relation
between them.
Duality theorem says that in the competition between
the grocer and the druggist the result is always a tie.
Optimal solution to primal tells purchaser what to do.
Optimal solution to dual fixes the natural prices at
which economy should run.
The diet x and vitamin prices y are optimal when
grocer sells zero of any food that is priced above its vitamin
equivalent.
druggist charges 0 for any vitamin that is oversupplied in the
diet.
Linear Programming

28

Linear Programming, Mid-Summary

Duality Theorem

Of great practical importance to solve linear

programs:

Druggists max revenue = Purchasers min cost

they model important practical problems

Practical Use of Duality:

Sometimes simplex algorithm (or other
algorithms) will run faster on the dual than on
the primal.
Can be used to bound how far you are from
optimal solution.
Important implications for economists.
Linear Programming

29

solving an LP is often an important component of

solving or approximating the solution to an
integer linear programming problem.

Can be solved in poly-time, the simplex

algorithm works very well in practice.
One problem where you really do not want to
write your own code.
Linear Programming

30

Weighted Vertex Cover

LP-based approximations
We dont know any polynomial-time algorithm
for any NP-complete problem
We know how to solve LP in polynomial time
We will see that LP can be used to get
approximate solutions to some NP-complete
problems.

Linear Programming

production, approximating the solution of

inconsistent equations, manufacturing, network
design, flow control, resource allocation.

31

Input: Graph G=(V,E) with non-negative

weights w(v) on the vertices.
Goal: Find a minimum-cost set of vertices
S, such that all the edges are covered.
An edge is covered iff at least one of its
endpoints is in S.
Recall: Vertex Cover is NP-complete.
The best known approximation factor is
2- (log log |V|/2 log|V|).
Linear Programming

32

Weighted Vertex Cover

The LP Relaxation

Variables: for each vV, x(v) is v in the cover?

This is not a linear program: the constraints of type

x(v) {0,1} are not linear.

Min vV w(v)x(v)
s.t.
x(v) + x(u) 1, (u,v)E

Such problems (LP's with integrality constraints on

variables) are called integer linear programs (IP).
Solving IP's is an NP-hard problem.
However, if we replace the constraints x(v) {0,1}
by x(v) 0 and x(v) 1, we will get a linear program.

x(v) {0,1} vV

The resulting LP is called a Linear Relaxation of

IP, since we relax the integrality constraints.
Linear Programming

33

Linear Programming

LP Relaxation of Weighted Vertex

Cover

LP Relaxation of Weighted Vertex

Cover - example

Min vV w(v)x(v)
s.t.
x(v) + x(u) 1, (u,v)E

Consider the case in which all

weights are 1.
An optimal VC has cost 2 (any two
vertices)
An optimal relaxation has cost 3/2 (for
all three vertices x(v)=1/2)

x(v) 0, vV
x(v) 1, vV

Linear Programming

34

35

The LP and the IP are different

problems. Can we still learn
something about Integer VC?
Linear Programming

36

Approximation of
Vertex Cover Using LP-Rounding

Why LP Relaxation Is Useful ?

The optimal value of LP-solution provides a
bound on the optimal value of the original
optimization problem. OPT(LP) is always better
than OPT(IP) (why?)

1. Solve the LP-Relaxation.

2. Let S be the set of all the vertices v with x(v) 1/2.
Output S as the solution.

Therefore, if we find an integral solution within a

factor r of OPTLP, it is also an r-approximation of
the original problem.

The value of the solution is: vs w(v) = {v|x(v) 1/2} w(v)

2vV w(v)x(v) =2OPTLP
Since OPTLP OPTVC, the cost of the solution is
2OPTVC.

These can be done by wise rounding.

Linear Programming

Analysis: The solution is feasible: for each edge e=(u,v),

either x(v) 1/2 or x(u) 1/2

37

Linear Programming

The Simplex Method

Phase II

Start with LP into standard form.

Phase I : Assume x = 0 is a feasible solution, that is,
0 is in the feasible region. If not, then an auxiliary
simplex method is used to start find a feasible
solution (more later).
Phase II: Use the slack version of the LP to move
from corner to corner along the edges of the feasible
region. Technically, well see how to move from one
slack version of the LP to another to achieve this.
When reach a local maximum youve found the
optimum.
Linear Programming

38

39

x=0

Constraint

Current
feasible
maximum

Objective
function
Feasible
region

y=0

Linear Programming

40

10

Phase II

Phase II
Objective
function

x=0

Constraint

Objective
function

x=0

Constraint

Current
feasible
maximum
Feasible
region

Current
feasible
maximum
Feasible
region

y=0

Linear Programming

y=0

41

Phase II

Objective
function
x=0

42

Simplex Algorithm: An Example in 3D

Constraint

Current
feasible
maximum
Feasible
region
y=0

Linear Programming

Linear Programming

43

Maximize 5x + 4y + 3z
subject to 2x+ 3y + z 5
4x + y + 2z 11
3x + 4y + 2z 8
x,y,z 0.
Convert inequalities into equalities by introducing slack
variables a,b,c.
Define: a = 5-2x-3y-z

a0
b = 11-4x-y-2z

b0
c = 8-3x-4y-2z

c0
F = 5x+4y+3z, objective function
Linear Programming

44

11

Initial State of Simplex

Basic

Nonbasic

Choose a Pivot and Tighten

Objective

Objective

Basic

Nonbasic

x=y=z = 0, a = 5, b = 11, c = 8, F = 0

x=y=z = 0, a = 5, b = 11, c = 8, F = 0

a = 5-2x-3y-z
b = 11-4x-y-2z
c = 8-3x-4y-2z

a = 5-2x-3y-z
b = 11-4x-y-2z
c = 8-3x-4y-2z

F = 5x+4y+3z

F = 5x+4y+3z

x < 5/2 most stringent!

x < 11/4
x < 8/3

Pivot is x

Linear Programming

45

Remove Pivot From RHS

Nonbasic

Basic

Linear Programming

46

Choose a Pivot and Tighten

Objective

Nonbasic

Basic

Objective

a=y=z = 0, x = 5/2, b = 1, c = 1/2, F = 25/2

a=y=z = 0, x = 5/2, b = 1, c = 1/2, F = 25/2

a = 5-2x-3y-z
b = 11-4x-y-2z
c = 8-3x-4y-2z

 x = 5/2 -3/2 y -1/2 z - 1/2 a

 b = 1 + 5y
+2a
 c = 1/2 +1/2 y -1/2 z + 3/2 a

x = 5/2 -3/2 y -1/2 z - 1/2 a

b = 1 + 5y
+2a
c = 1/2 +1/2 y -1/2 z + 3/2 a

F = 5x+4y+3z

 F= 25/2-7/2 y +1/2 z -5/2 a

F= 25/2-7/2 y +1/2 z -5/2 a

z<5
z<
z < 1 most stringent!

Pivot is z

Linear Programming

47

Linear Programming

48

12

Remove Pivot from RHS

Nonbasic

Final Solution

Objective

Basic

Original Problem
Maximize 5x + 4y + 3z
subject to 2x+ 3y + z 5 tight
4x + y + 2z 11 not tight
3x + 4y + 2z 8 tight
x,y,z 0.

a=y=c = 0, x = 2, b = 1, z = 1, F = 13
x = 5/2 -3/2 y -1/2 z - 1/2 a  x = 2 - 2y - 2a + c
b = 1 + 5y
+2a  b = 1 + 5y +2a
c = 1/2 +1/2 y -1/2 z + 3/2 a  z = 1 + y + 3a - 2c

Solution: x = 2, y = 0, z = 1, F = 13

F= 25/2-7/2 y +1/2 z -5/2 a  F = 13 - 3y - a - c

No way to increase F so were done
Linear Programming

49

Finding a Feasible Solution

50

Example
Maximize 2x - y

Suppose that 0 is not feasible, then Phase 1 finds a

feasible solution in terms of the basic variables.

subject to 2x - y 2
x - 5y -4

Maximize c1x1 + c2x2 ++ cnxn

subject to 1 j n aijxj bj
i=1..m
xj 0
j=1..n

x,y 0.
x = y = 0 is not a feasible solution

Ignore the objective function

Maximize -x0
subject to 1 j n aijxj - x0 bj
xj 0

Linear Programming

i=1..m
j=0..n

Feasible solution exist only if maximum has x0 = 0

Linear Programming

51

Linear Programming

52

13

Auxiliary LP

Move to Slack Form

Assuming x=y=0, we need to make sure
a,b > 0

Maximize -z
subject to 2x - y - z 2
x - 5y - z -4
x,y,z 0.

a = 2 - 2x + y + z

z > -2

b = -4 - x + 5y + z

z > 4 most stringent!

F = -z
We do a pivot step with z.

Linear Programming

53

Remove Pivot from RHS

Nonbasic

Basic

Linear Programming

Choose a Pivot and Tighten

Objective

Nonbasic

Basic

Objective

x=y=b=0, a = 6, z = 4, F = -4

x=y=b=0, a = 6, z = 4, F = -4

a = 2 - 2x + y + z  a = 6 - x - 4y + b

a = 6 - x - 4y + b

y < 6/4

b = -4 - x + 5y + z  z = 4 + x - 5y + b

z = 4 + x - 5y + b

y < 4/5 most stringent!

 F = -4 - x + 5y - b

F = -4 - x + 5y - b

F = -z

54

Pivot is y

Linear Programming

55

Linear Programming

56

14

Remove Pivot from RHS

Nonbasic

Starting the Simplex

Objective

Basic

Nonbasic

x=z=b=0, a = 14/5, y = 4/5, F = 0

Basic

x=b=0, a = 14/5, y = 4/5, F = -4/5

a = 6 - x - 4y + b

 a = 14/5 - 9/5 x - 4/5 z - 1/5 b

a = 14/5 - 9/5 x - 1/5 b

z = 4 + x - 5y + b

 y = 4/5 + 1/5 x - 1/5 z + 1/5 b

y = 4/5 + 1/5 x + 1/5 b

F = -4 - x + 5y - b  F = 0 - z

F = 2x - y = -4/5 + 9/5 x - 1/5 b

Auxiliary LP is solved because there is no way to increase F.

Force z = 0 permanently.
Start Simplex with x = b = 0, a = 14/5, y = 4/5, and
F equal to the original F with nonbasic variables on right hand side
Linear Programming

Objective

57

Class Exercise

Original objective
function

Linear Programming

58

Choosing a Pivot

Complete the solving of this LP

Largest increase in F
Largest positive coefficient
First positive coefficient

Maximize 2x - y
subject to 2x - y 2
x - 5y -4
x,y 0.

Linear Programming

59

Linear Programming

60

15

Termination

LP Summary

Technically, simplex can loop.

Only so many choices for basic
variables. Lack of termination can be
detected.
Termination is not usually a problem

Linear Programming

LP is widely used in practice.

LP has a very nice theory.
LP relaxation and rounding used in
approximation algorithms
Simplex method is a good method but
has some pathological exponential
cases.
61

Linear Programming

62

16