University of Zagreb, Croatia

Lectures 3.a
Heuristic Optimization Methods:
Overview of exact methods
Slides prepared by Nina Skorin-Kapov

Academic year 2020/2021

Graduate Studies Programme
 Outline
University of Zagreb, Croatia

 Concept behind LP solvers (simplex)

 Why is IP harder than LP?

 Branch and bound

Zavod za telekomunikacije 2
 Some exact methods:
University of Zagreb, Croatia

 Simplex and interior point methods for LP

problems –variables are continuous
 Branch and Bound and A*: based on implicit
enumeration of all solutions;
 ‘divide and conquer’ concept

Zavod za telekomunikacije 3
 Outline
University of Zagreb, Croatia

 Some concepts behind LP solvers

 Why is IP harder than LP?

 Branch and bound

Zavod za telekomunikacije 4
 Linear programming (LP)
University of Zagreb, Croatia

Standard form of a In matrix form: min c T x 

LP problem subject to :
Ax = b
min c1 x1 + ... + cn xn  x  0
Vector of
subject to : continuous
decision variables xT = ( x1 ,..., xn ),
a11 x1 + ... + a1n xn = b1 cT = (c1 ,..., cn ),
Constant
... vectors b T = (b1 ,..., bm ),

am1 x1 + ... + amn xn = bm  a11


... a1n 

Constant A =  ... ... ... 
x1  0,..., xn  0 matrix a ... amn 
 m1

Zavod za telekomunikacije 5
 LP Example
University of Zagreb, Croatia

 A company makes 2 types of ice cream: A and

B. To make ice cream A, they need 1 bag of
sugar, one bottle of milk and 1 bag of flavoring
per kg of ice cream. To make ice cream B, they
need 2 bottles of milk and 1 bag of flavoring per
kg (it is sugar-free). The company buys 7 bags
of sugar, 10 bottles of milk and 8 bags of
flavoring daily. The profit per kg of ice cream A
is 2.25 Euros and for ice cream B is 4 Euros.
How much of each type should the company
produce daily to maximize their profit?

Zavod za telekomunikacije 6
 LP Example – LP formulation
University of Zagreb, Croatia

Objective
Max 2.25x1 + 4x2
Function
s.t. x1 < 7
Problem specific
x1 + 2x2 < 10
constraints
x1 + x 2 < 8

x1 > 0 and x2 > 0 Non-negativity

Constraints

Zavod za telekomunikacije 7
 LP Example: Graphical Solution
University of Zagreb, Croatia

 1st constraint: x1 < 7 (and non-negativity)

x2

8
7 x1 = 7
6
Feasible points
5 for first constraint
4
3
2
(7, 0)
1
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 8
 LP Example: Graphical Solution
University of Zagreb, Croatia

 2nd constraint: x1 + 2x2 < 10 (and non-negativity)

x2

8
7
(0, 5)
6
5 x1 + 2x2 = 10
4
3 Feasible points
2 for second constraint
(10, 0)
1
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 9
 LP Example: Graphical Solution
University of Zagreb, Croatia

 3rd constraint: x1 + x2 < 8 (and non-negativity)

x2
(0, 8)
8
7
6 x1 + x2 = 8
5
4
Feasible points
3
for third constraint
2
1 (8, 0)
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 10
 LP Example 1: Graphical Solution
University of Zagreb, Croatia

 Feasible region (solution space)

x2

8 x1 + x2 = 8
7
6 x1 = 7
5
4
3
Feasible x1 + 2x2 = 10
2
Region
1
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 11
 LP Example: Graphical Solution
University of Zagreb, Croatia

 Objective Function: 2.25x1 + 4x2

x2

8
7
6 (0, 4.5) Possible objective Function line
5 2.25x1 + 4x2 = 18
4
3
2 Feasible
1 Region (8, 0)
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 12
 LP Example: Graphical Solution
University of Zagreb, Croatia

 Objective Function: 2.25x1 + 4x2

x2
Example Objective
obj. func. 8 Objective Function functions
lines 2.25x1 + 4x2 = 16 lines are
7
parallel
6 Objective Function
5 2.25x1 + 4x2 = 18
4 Objective Function
3 2.25x1 + 4x2 = 20
2 Feasible
1 Region
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 13
 LP Example: Graphical Solution
University of Zagreb, Croatia

 Objective Function: 2.25x1 + 4x2

x2

8 Objective
7 function grows
6
5 Obj. function =
20
4
18
3 16
2 Feasible
1 Region
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 14
 LP Example: Graphical Solution
University of Zagreb, Croatia

 Optimal solution
x2
Maximum
8
Objective Function Line
7 2.25x1 + 4x2 = 21.5
6
5 Optimal Solution
4
(x1 = 6, x2 = 2)
3
2
Feasible
1
Region
x1
1 2 3 4 5 6 7 8 9 10

Zavod za telekomunikacije 15
 Solving linear programs
University of Zagreb, Croatia

 Efficient exact algorithms have been developed

to solve LPs
 Simplex method
 Interior point methods

 This is possible because the feasible region

(solution space) is a convex set and the
objective function is a convex function
 Thus the global optimum is necessarily a node of the
polytope representing the feasible region
 A local optimum is also a global optimum

 ‘fairly easy’; No need for heuristics

Zavod za telekomunikacije 16
 Convex Sets and Convex Hull
University of Zagreb, Croatia

 A set is convex if for

every pair of points in
the set, the line
segment connecting
them is also in the convex concave
set.

 The convex hull for a

set of points X is the
minimal convex set
containing X.

Zavod za telekomunikacije 17
 Simplex
University of Zagreb, Croatia

 Initially developed by G. Danzig in 1947

 Main idea: The search jumps from one feasible

corner point solution to the next adjacent one
until an optimal one is found
 Optimality test: if a feasible corner point solution has
no adjacent feasible corner point solution that is
better, then it is an optimal solution (convex hull;
local optimum is a global optimum)

Zavod za telekomunikacije 18
 Interior point methods
University of Zagreb, Croatia

 Initially developed: 1984, N. Karmarker, AT&T Bell Labs

 Potential to solve huge LPs that are too much for
Simplex
 Main idea:
 Iteratively checks feasible solutions, but unlike simplex it
traverses interior feasible solutions towards the optimal; always
moves to a better solution
 The search moves in the direction of the fastest improvement
(perpendicular to objective function) using the projected gradient
and step size alpha

Zavod za telekomunikacije 19
 Outline
University of Zagreb, Croatia

 Concept behind LP solvers

 Why is IP harder than LP?

 Branch and bound

Zavod za telekomunikacije 20
 Integer linear programming (ILP or IP)
University of Zagreb, Croatia

 BUT, many real-world problems have discrete

variables!!
 ILP: Discrete decision variables, linear
objective function and constraints
 Mixed ILP (MILP): discrete and continuous
decision variables
 Harder to solve than LP
 Exacttechniques: Branch and bound,
 Approximate: relaxations, decomposition
approaches, cutting plane algorithms

Zavod za telekomunikacije 21
Why is IP more difficult than LP: Example
University of Zagreb, Croatia

max 3 x1 + 2 x2
subject to : x2
3 x1 + x2  9 -x1 + x2 < 1
x1 + 3 x2  7 5
3x1 + x2 < 9
− x1 + x2  1
4
x1 , x2  0, x1 , x2 integer Max 3x1 + 2x2
3
Optimum LP (2.5,1.5)
2
x1 + 3x2 < 7
Solution space = 1
feasible solutions x1
are the integer 1 2 3 4 5 6 7
points in the
shaded region

Zavod za telekomunikacije 22
 Why is IP more difficult than LP: Example
University of Zagreb, Croatia

max 3 x1 + 2 x2 Round the LP optimum?

subject to : “upwards”: (3,2) not feasible point
3 x1 + x2  9
“downwards” : (2,1) feasible, obj. func. = 8
x1 + 3 x2  7 x2
− x1 + x2  1
x1 , x2  0, x1 , x2 integer 5

4 (3,2) not feasible

3 The LP optimum (2.5,1.5)
Is (2,1) the optimum solution? (2,1) feasible, obj.func. = 8
2
NO!!!!
Integer solutions closest to the 1
LP solution are not neccessarily x1
optimal for the IP!!! 1 2 3 4 5 6 7

Zavod za telekomunikacije 23
 Why is IP more difficult than LP: Example
University of Zagreb, Croatia

max 3 x1 + 2 x2 We solve the problem with brute-force here

subject to :
3 x1 + x2  9 NOTE:The optimal
x1 + 3 x2  7 x2
solution doesn’t have
− x1 + x2  1 to be a corner point!!!!
x1 , x2  0, x1 , x2 integer 5 That makes it harder
to solve!
4
x1 x2 obj.func.
1. 0 0 0 3 Optimum LP (2.5,1.5)
2. 1 0 3 (2,1) feasible, obj.func.=8
3. 2 0 6 2
4. 3 0 9 (3,0) feasible, obj.func.=9,
5. 0 1 2 1 OPTIMAL IP
6. 1 1 5 x1
7. 2 1 8 1 2 3 4 5 6 7
8. 1 2 7

Zavod za telekomunikacije 24
 LP-relaxation
University of Zagreb, Croatia

 LP-relaxation: relax the integral constraints

 A solution to the LP-relaxation is an upper bound
(maximization problem) or a lower bound (minimization
problem) on the solution to the IP
 The only time IP is as easy as LP is if the LP relaxation
has an integral optimal solution
 There are some special types of IP problems which guarantee
integral solution of the LP-relaxation by their structure:
transport, transshipment, max flow…
 A method to solve IP problems: LP-relaxation+rounding
but doesn’t always work well (see example)
 Obtained solution may not even be feasible
 Doesn’t guarantee optimality

Zavod za telekomunikacije 25
 Outline
University of Zagreb, Croatia

 Concept behind LP solvers

 Why is IP harder than LP?

 Branch and bound

Zavod za telekomunikacije 26
 Branch and Bound
University of Zagreb, Croatia

 Used to help solve IP and BIP problems exactly

 “divide and conquer” principle

 Main idea: divide the original problem into

subproblems and try to eliminate those
subproblems which surely do not contain the
optimal solution

 Implicit enumeration of possible solutions –

build a solution tree
Zavod za telekomunikacije 27
 Initialization
University of Zagreb, Croatia

1. Solve the LP-relaxation of the original problem

 If the LP solution is integral→ problem solved!
 OPT(LP-relaxation) >= OPT(IP) for maximization
 If not, we have an upper bound on the optimal
solution
2. Find a feasible integer solution
 We can round the LP solution, but check feasibility!!!
 Trial and error
…

Zavod za telekomunikacije 28
Example – IP formulation & LP-relaxation
University of Zagreb, Croatia

Initialization:
Max Z=2x1 + 3x2 1. Solve the LP-
s.t. 195x1 +273x2 < 1365 relaxation
4x1 + 40x2 < 140
x1 < 4
Solution to the LP-
Drop the x1 > 0 and x2 > 0 relaxation:
integrality x1, x2 integer Z=14.66
constraints (x1, x2)=(2.44, 3.26)
IP!

Zavod za telekomunikacije 29
 Root node
University of Zagreb, Croatia

Solution to the LP- Upper Bound to the IP:

relaxation: UB(IP)=OPT(LP)= 14.66
Z=14.66
(x1, x2)=(2.44, 3.26)
Since the coefficients in Z are
Branch-and-Bound integral and the variables must be
tree root node integral for the IP, an Upper
Bound is UB(IP)=14
1 If Z doesn’t
Upper bound on IP solution have integral
UB=14 coefficients, use
(x1,x2)= the OPT(LP)
value
(2.44, 3.26) Values of the variables for the
optimal LP solution
Zavod za telekomunikacije 30
 Find a feasible solution
University of Zagreb, Croatia

Initialization:
2. Find a feasible solution
 E.g., rounding:
rounding
(x1, x2)=(2.44, 3.26) (x1, x2)=(2, 3)

Z = 13
Check feasibility:
195x1 +273x2 < 1365
4x1 + 40x2 < 140 Incumbent
x1 < 4 solution
Z*=13
x1 > 0 and x2 > 0
Zavod za telekomunikacije 31
 Branching
University of Zagreb, Croatia

 Branching: choose a branching variable xi

which is non-integral in the LP solution (xi(LP))
of that node
 Ifthere are multiple non-integral variables, choose
any or make a problem specific or general rule,
e.g. the variable farthest from integrality
 Divide the problem into 2 subproblems each
with an added constraint:

xi  xi (LP ) xi  xi (LP )

Zavod za telekomunikacije 32
 Branching
University of Zagreb, Croatia

Choose 1 Original
branching UB=14 problem 1
variable (x1,x2)=
x1=2.44 (2.44,3.26)

x1  2 x1  3
Subproblem 3
Subproblem 2
2 3

Zavod za telekomunikacije 33
 Subproblems
University of Zagreb, Croatia

Subproblem 2 (SP2) Subproblem 3 (SP3)

Max Z=2x1 + 3x2 Max Z=2x1 + 3x2

s.t. 195x1 +273x2 < 1365 s.t. 195x1 +273x2 < 1365
4x1 + 40x2 < 140 4x1 + 40x2 < 140
x1 < 4 x1 < 4
x1 < 2 x1 ≥ 3
x1 > 0 and x2 > 0 x1 > 0 and x2 > 0
x1, x2 integer x1, x2 integer
New
constraints

Zavod za telekomunikacije 34
 Subproblems
University of Zagreb, Croatia
Solve LP-relaxation of
Subproblem 2:
Z=13.9
Subproblem 2 (SP2) (x1, x2)=(2, 3.3)
Max Z=2x1 + 3x2
Since the coefficients in Z are
s.t. 195x1 +273x2 < 1365 integral→ Upper bound on SP2
4x1 + 40x2 < 140 UB(SP2)=13
x1 < 4
x1 < 2
2
x1 > 0 and x2 > 0
UB=13
x1, x2 integer (x1, x2)=
(2, 3.3)

Zavod za telekomunikacije 35
 Subproblems
University of Zagreb, Croatia
Solve LP-relaxation of
Subproblem 3:
Subproblem 3 (SP3) Z=14.52
(x1, x2)=(3, 2.86)
Max Z=2x1 + 3x2

s.t. 195x1 +273x2 < 1365 Since the coefficients in Z are

integral→ Upper bound on SP3
4x1 + 40x2 < 140
UB(SP2)=14
x1 < 4
x1 ≥ 3
x1 > 0 and x2 > 0 3
x1, x2 integer UB=14
(x1, x2)=
(3, 2.86)

Zavod za telekomunikacije 36
 1st Branching Iteration
University of Zagreb, Croatia

1
UB(IP)=14 Z*=13
(x1,x2)=
(2.44,3.26)

x1  2 x1  3

2 3
UB=13 UB=14
(x1, x2)= (x1, x2)=
(2, 3.3) (3, 2.86)

Zavod za telekomunikacije 37
 Fathoming tests
University of Zagreb, Croatia

Fathoming tests are used to ‘conquer’ nodes

(subproblems)–one or more must hold
1. The LP relaxation has no solution
 The additional constraint has made it infeasible
2. The Upper bound of a node ≤ Z*
3. The value of the variables of the optimal
solution of the LP-relaxation is integral, i.e. the
optimal solution to the subproblem has been
found
1. If it is better than Z*, update Z*
2. If not, the node was conquered by test 2

Zavod za telekomunikacije 38
 Node 2 – fathoming tests
University of Zagreb, Croatia

FATHOMING TESTS
1. LP-relaxation infeasible Z*=13
2. UB(node)of a node ≤ Z*
3. OPT(node) found
2
1. LP solution feasible UB=13
2. UB(SP2)=13 ≤ Z*=13 (x1, x2)=
(2, 3.3)
Conquered by test 2 :
The best solution to subproblem
2 is 13, and we already found CONQUERED
such a solution so it’s a waste of
time to look for another
Zavod za telekomunikacije 39
 Node 3 – fathoming tests
University of Zagreb, Croatia

FATHOMING TESTS
1. LP-relaxation infeasible Z*=13

2. UB(node)of a node ≤ Z*
3. OPT(node) found
3
UB=14
1. LP solution feasible (x1, x2)=
2. UB(SP3)=14 ≤ Z*=13 (3, 2.86)
3. LP solution is not integral

Node 3 is not conquered – keep branching!

Zavod za telekomunikacije 40
 2nd Branching Iteration
University of Zagreb, Croatia

1 Z*=13
UB(IP)=14
(x1,x2)=
(2.44,3.26)
Choose
branching
2 3 variable.
UB=13 UB=14 x1 is integral,
(x1, x2)= (x1, x2)= so choose
(2, 3.3) (3, 2.86) x2=2.86

x2  2 x2  3

4 5

Zavod za telekomunikacije 41
 New Subproblems
University of Zagreb, Croatia

Subproblem 4 (SP4) Subproblem 5 (SP5)

Max Z=2x1 + 3x2 Max Z=2x1 + 3x2

s.t. 195x1 +273x2 < 1365 s.t. 195x1 +273x2 < 1365
4x1 + 40x2 < 140 4x1 + 40x2 < 140
x1 < 4 x1 < 4
x1 ≥ 3 x1 ≥ 3
x2 < 2 x2 ≥ 3
x1 > 0 and x2 > 0 x1 > 0 and x2 > 0
x1, x2 integer x1, x2 integer
Constraints New
from SP3 constraints

Zavod za telekomunikacije 42
 New Subproblems
University of Zagreb, Croatia
Solve LP-relaxation of
Subproblem 4 (SP4) Subproblem 4:
Z=14
Max Z=2x1 + 3x2 (x1, x2)=(4,2)

s.t. 195x1 +273x2 < 1365

4x1 + 40x2 < 140 Upper bound on SP4
UB(SP4)=14
x1 < 4
x1 ≥ 3
x2 < 2 4
x1 > 0 and x2 > 0 UB=14
x1, x2 integer (x1, x2)=
(4,2)

Zavod za telekomunikacije 43
 New Subproblems
University of Zagreb, Croatia

Subproblem 5 (SP5)
Solve LP-relaxation of
Max Z=2x1 + 3x2 Subproblem 5→ No solution!
s.t. 195x1 +273x2 < 1365
4x1 + 40x2 < 140
x1 < 4 5
x1 ≥ 3 No solution,
x2 ≥ 3 LP-relax
x1 > 0 and x2 > 0 infeasible
x1, x2 integer INFEASIBLE!!!

Zavod za telekomunikacije 44
 2nd Branching Iteration
University of Zagreb, Croatia

1
UB(IP)=14 Z*=13
(x1,x2)=
(2.44,3.26)

2 3
UB=13 UB=14
(x1, x2)= (x1, x2)=
(2, 3.3) (3, 2.86)

x2  2 x2  3

4 5
UB=14 No solution,
(x1, x2)= LP-relax
(4,2) infeasible

Zavod za telekomunikacije 45
 Node 4 – fathoming tests
University of Zagreb, Croatia

FATHOMING TESTS
1. LP-relaxation infeasible
Z*=13
2. UB(node)of a node ≤ Z*
3. OPT(node) found
4
UB=14
1. LP solution feasible (x1, x2)=
2. UB(SP4)=14 ≤ Z*=13 (4,2)
3. LP solution is integral
Conquered by test 3
Solution to the LP-relaxation = CONQUERED
14 is better than incumbent Z*=14
solution Z* =13 → new Z*=14
Zavod za telekomunikacije 46
 Node 5 – fathoming tests
University of Zagreb, Croatia

FATHOMING TESTS
1. LP-relaxation infeasible Z*=14

2. UB(node)of a node ≤ Z*
3. OPT(node) found
5
No solution,
1. LP solution infeasible LP-relax
infeasible

Conquered by test 1
CONQUERED

Zavod za telekomunikacije 47
 2nd Fathoming Iteration
University of Zagreb, Croatia

1
UB(IP)=14
Z*=14 When all nodes
(x1,x2)=
(2.44,3.26)
are conquered→
STOP.
Z* is the optimal
2 3
UB=13 UB=14
solution to the
(x1, x2)= (x1, x2)= original IP
(2, 3.3) (3, 2.86)

x2  2 x2  3

4 5
UB=14 No solution,
(x1, x2)= LP-relax
(4,2) infeasible

Zavod za telekomunikacije 48
 Branch and Bound algorithm
University of Zagreb, Croatia

1. Initialization
1. Solve LP-relaxation of the original problem; get UB on IP problem
2. Find any feasible itegral solution (e.g. rounding) → Z*
2. Branching iterations
1. Branching
1. Choose an unconquered node n (e.g. the last created or with max UB)
2. Choose a branching variable xi (e.g. furthest from integer at node n)
3. Make 2 new nodes (subproblems) with constraints:
xi  xi (LP ) xi  xi (LP )
2. Solve LP-relaxation of new subproblems and find their UBs
3. Perform fathoming tests to try to conquer the new nodes
3. Optimality check
1. If there are unconquered nodes, go to Step 2. Otherwise, STOP and
Z* is the optimal solution

Zavod za telekomunikacije 49
 Branch and Cut
University of Zagreb, Croatia

 Improved technique for solving pure Binary IP

problems
 Combines branch and bound with cutting
planes
 Cutting plane (CP): a CP for an IP problem
is a new constraint that reduces the feasible
region of the LP-relaxation without
eliminating any IP feasible solutions

Zavod za telekomunikacije 50