Texts in Mathematics
Uploaded by
João DuarteTexts in Mathematics
Uploaded by
João DuarteTexts in Mathematics
Volume 6
A Mathematical Primer on
Linear Optimization
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Volume 1
An Introduction to Discrete Dynamical Systems and their General Solutions
F. Oliveira-Pinto
Volume 2
Adventures in Formalism
Craig Smoryński
Volume 3
Chapters in Mathematics. From π to Pell
Craig Smoryński
Volume 4
Chapters in Probability
Craig Smoryński
Volume 5
A Treatise on the Binomial Theorem
Craig Smoryński
Volume 6
A Mathematical Primer on Linear Optimization
Diogo Gomes, Amílcar Sernadas, Cristina Sernadas, João Rasga, Paulo Mateus
Texts in Mathematics Series Editor
Dov Gabbay [email protected]
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
A Mathematical Primer on
Linear Optimization
Diogo Gomes
Amílcar Sernadas
Cristina Sernadas
João Rasga
Paulo Mateus
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
© Individual author and College Publications 2019. All rights reserved.
ISBN 978-1-84890-315-9
College Publications
Scientific Director: Dov Gabbay
Managing Director: Jane Spurr
Department of Computer Science
King’s College London, Strand, London WC2R 2LS, UK
http://www.collegepublications.co.uk
Cover designed by Laraine Welch
All rights reserved. No part of this publication may be reproduced, stored in a retrieval system
or transmitted in any form, or by any means, electronic, mechanical, photocopying, recording
or otherwise without prior permission, in writing, from the publisher.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
v
Preface
Our main objective is to provide a self-contained mathematical introduction
to linear optimization for undergraduate students of Mathematics. This book
is equally suitable for Science, Engineering, and Economics students who are
interested in gaining a deeper understanding of the mathematical aspects of the
subject. The linear optimization problem is analyzed from different perspec-
tives: topological, algebraic, geometrical, logical, and algorithmic. Neverthe-
less, no previous knowledge of these subjects is required. The essential details
are always provided in a special section at the end of each chapter. The techni-
cal material is illustrated with multiple examples, problems with fully-worked
solutions, and a range of proposed exercises.
In Chapter 1, the optimization problem is presented along with the con-
cepts of set of admissible vectors and set of optimizers. Then, we discuss the
linear case, including the canonical and the standard optimization problems.
Finally, we relate the general linear optimization problem and the canonical op-
timization problem and analyze the relationship between the canonical and the
standard optimization problems. The relevant background section of this chap-
ter includes some basic algebraic concepts, notation, and preliminary results
namely about groups, fields, and vector spaces.
Chapter 2 explores some topological techniques that provide sufficient con-
ditions for the existence of optimizers in canonical optimization problems. The
chapter starts with the definition of interior and boundary of the set of ad-
missible vectors (later on it is shown that these concepts coincide with the
topological ones) and the proof that optimizers are always on the boundary
of the set of admissible vectors when the objective map is not the zero map.
Then, we provide sufficient conditions for a canonical optimization problem to
have maximizers. In the relevant background section, we provide a modicum
of topological notions and results.
The main objective of Chapter 3 is to provide a way for deciding whether or
not an admissible vector is an optimizer, relying on Farkas’ Lemma. Namely,
we introduce the concept of line active in an admissible vector and prove the
Local Maximizer Theorem and the Maximizer Theorem using convex cones.
Moreover, a technique for deciding whether or not there are admissible vectors
is presented, using Farkas’ Lemma again.
In Chapter 4, linear algebra concepts are used for computing optimizers for
a standard optimization problem. The objective is to find the basic admissible
vectors of the problem at hand. These vectors are relevant since we prove that,
under certain conditions, there is always a basic admissible vector which is
an optimizer. Moreover, the notion of a non-degenerate standard optimization
problem is introduced and a characterization of basic vectors is provided in this
case. In the relevant background section of this chapter, we provide an overview
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
vi
of notions and results from linear algebra including dimension, span, rank,
determinant of a matrix, singular matrices, and some well-known theorems.
Chapter 5 concentrates on geometrical aspects of linear optimization, namely
it provides a geometrical characterization of the basic admissible vectors of a
standard optimization problem as vertices of an appropriate convex polyhe-
dron. In the relevant background section, we present the relevant results about
affine spaces and subspaces.
Chapter 6 presents several aspects of duality. The objective is to show that
the dual provides yet a new technique for finding optimizers of the original prob-
lem. The dual problems are found using the Lagrange Multiplier Technique.
We start by presenting two main results: the Weak and the Strong Duality
Theorems. Then, we concentrate on slacks for the linear optimization problem
and prove the Slack Complementarity Theorem. Afterward, the Equilibrium
Theorem is established as well as the Uniqueness Theorem. Finally, a Logic of
Inequalities is presented and some results are proved relating consistency and
satisfiability of a formula.
Chapter 7 provides an introduction to computational complexity to analyze
the efficiency of linear optimization algorithms. In Section 7.1, we present the
rigorous definition of the decision problems associated with linear optimization,
and in Section 7.2, we discuss the representation of vectors and matrices. In
Section 7.3, we prove that the standard decision problem is in NP. A deter-
ministic algorithm for getting an optimizer whenever there is one, based on
a brute-force method, is discussed in Section 7.4. The complexity of this al-
gorithm is shown not to be polynomial. Finally, in the relevant background
section, we introduce the central notions and techniques that are necessary for
assessing the computational efficiency of an algorithm.
Chapter 8 is targeted at the Simplex Algorithm that allows us to find an op-
timizer, whenever there is one, for the standard optimization problem satisfying
some mild conditions. In Section 8.2, we prove the soundness and completeness
of the Simplex Algorithm and point out that its complexity is not polynomial.
Finally, in Chapter 9, a description of the integer linear optimization prob-
lem is presented along with the respective relaxed problem. Then, we compare
the sets of admissible vectors and optimizers of both problems. The integral-
ity gap is introduced and discussed. Afterward, we discuss totally unimodular
problems and provide a sufficient condition for a matrix to be totally uni-
modular. Furthermore, we give the Assignment Problem as an example. The
importance of this concept is made clear by proving a sufficient condition for
the existence and characterization of the optimizers of totally unimodular prob-
lems. We conclude by presenting and illustrating an algorithm based on the
Branch and Bound Technique for solving integer optimization problems.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
vii
Acknowledgements
We would like to express our deepest gratitude to the many undergraduate
math students of Instituto Superior Técnico that attended the Introduction to
Optimization course.
Diogo Gomes was partially supported by King Abdullah University of Sci-
ence and Technology (KAUST) baseline funds and KAUST OSR-CRG2017-
3452. Amílcar Sernadas, Cristina Sernadas and João Rasga acknowledge the
National Funding from Fundação para a Ciência e a Tecnologia (FCT) under
the project UID/MAT/04561/2019 granted to Centro de Matemática, Apli-
cações Fundamentais e Investigação Operacional (CMAFcIO) of Universidade
de Lisboa. Paulo Mateus acknowledges the National Funding from FCT under
project PEst-OE/EEI/LA0008/2019 granted to Instituto de Telecomunicações.
Last but not least, we greatly acknowledge the excellent working environ-
ment provided by the Department of Mathematics of Instituto Superior Téc-
nico, Universidade de Lisboa.
Lisbon, Diogo Gomes
July 2019 Amílcar Sernadas
Cristina Sernadas
João Rasga
Paulo Mateus
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
viii
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Contents
1 Optimization Problems 1
1.1 General Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Linear Optimization . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3 Canonical and standard optimization problems . . . . . . . . . 11
1.4 Relating Problems . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.5 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 23
1.6 Relevant Background . . . . . . . . . . . . . . . . . . . . . . . . 30
2 Optimizers 39
2.1 Boundary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.2 Existence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.3 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 52
2.4 Relevant Background . . . . . . . . . . . . . . . . . . . . . . . . 54
3 Deciding Optimizers 63
3.1 Farkas’ Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.2 Using Convex Cones . . . . . . . . . . . . . . . . . . . . . . . . 74
3.3 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 80
4 Computing Optimizers 87
4.1 Basic Vectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.2 Using Basic Vectors . . . . . . . . . . . . . . . . . . . . . . . . 95
4.3 Basic Through Counting . . . . . . . . . . . . . . . . . . . . . . 99
4.4 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 102
4.5 Relevant Background . . . . . . . . . . . . . . . . . . . . . . . . 108
5 Geometric View 117
5.1 Admissibility . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
5.2 Optimizers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
5.3 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 137
ix
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
x CONTENTS
5.4 Relevant Background . . . . . . . . . . . . . . . . . . . . . . . . 142
6 Duality 147
6.1 Weak and Strong Duality . . . . . . . . . . . . . . . . . . . . . 147
6.2 Complementarity . . . . . . . . . . . . . . . . . . . . . . . . . . 158
6.3 Equilibrium . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
6.4 Logic of Inequalities . . . . . . . . . . . . . . . . . . . . . . . . 166
6.5 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 170
7 Complexity 177
7.1 Optimization Decision Problem . . . . . . . . . . . . . . . . . . 177
7.2 Representation . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
7.3 Non-Deterministic Approach . . . . . . . . . . . . . . . . . . . 184
7.4 Deterministic Approach . . . . . . . . . . . . . . . . . . . . . . 186
7.5 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 189
7.6 Relevant Background . . . . . . . . . . . . . . . . . . . . . . . . 197
8 The Simplex Algorithm 205
8.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
8.2 Soundness, Completeness and Complexity . . . . . . . . . . . . 209
8.3 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 217
9 Integer Optimization 223
9.1 Relaxed Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 224
9.2 Totally Unimodular Problems . . . . . . . . . . . . . . . . . . . 227
9.3 The Branch and Bound Technique . . . . . . . . . . . . . . . . 235
9.4 Solved Problems and Exercises . . . . . . . . . . . . . . . . . . 240
Table of Symbols 249
Subject Index 251
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 1
Optimization Problems
The objective of this chapter is to present the optimization problem and its
different formulations.
1.1 General Concepts
In this section, we introduce the general form of an optimization problem.
Furthermore, we discuss admissible tuples, maximizers and minimizers.
Definition 1.1
An (n-dimensional) constraint, with n ∈ N+ , is a triple (g, o
n, b), written
g(x1 , . . . , xn ) o
n b,
where g : Rn → R is a constraint map, on ∈ {≤, =, ≥} is a binary relation and
b ∈ R. When o n is =, we say that the constraint is an equality and when o
n is
either ≤ or ≥, we say that the constraint is an inequality.
Definition 1.2
An (n-dimensional) description, with n ∈ N+ , is a pair
ni , bi ) : 1 ≤ i ≤ m}, U )
({(gi , o
for some m ∈ N+ , where (gi , o ni , bi ) is an n-dimensional constraint for i =
1, . . . , m and U ⊆ R is a non-empty set.
When U = R in an n-dimensional description, we may simply present the
description by the set of constraints.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2 CHAPTER 1. OPTIMIZATION PROBLEMS
Example 1.1 (Rectangle Problem)
Assume that we want to consider all rectangles with a fixed area b ∈ R+ . This
universe can be expressed as a 2-dimensional description
{x1 x2 = b, x1 ≥ 0, x2 ≥ 0},
where x1 and x2 represent the dimensions (length and width) of the rectangles.
Example 1.2 (Knapsack Problem)
Assume that we have n kinds of objects that we want to accommodate in a
knapsack that has a weight limit b. These requirements can be stated as an
n-dimensional description
({a1 x1 + · · · + an xn ≤ b}, N),
where each xj represents the number of objects of kind j and aj is the weight
of each object of kind j.
Definition 1.3
We say that (d1 , . . . , dn ) ∈ U n is an admissible vector for an n-dimensional
description
n bi : 1 ≤ i ≤ m}, U )
({gi (x1 , . . . , xn ) o
when
gi (d1 , . . . , dn ) o
n bi
holds for every i = 1, . . . , m.
Notation 1.1
We denote by
XD
the set of admissible tuples or vectors for a description D.
Example 1.3 (Rectangle Problem)
Let D be the description introduced in Example 1.1. Then,
√ √
b b
b, b ∈ XD and , ∈/ XD .
7 7
Definition 1.4
An (n-dimensional) optimization problem P is a tuple
(D, f, ),
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.1. GENERAL CONCEPTS 3
where D is an n-dimensional description, f : Rn → R is the objective map and
∈ {≤, ≥}. We denote by
SP
the set of optimizers for P composed by each (s1 , . . . , sn ) ∈ XD such that
f (d1 , . . . , dn ) f (s1 , . . . , sn ),
for every (d1 , . . . , dn ) ∈ XD .
Notation 1.2
Given an optimization problem (D, f, ), we denote by
XP
the set XD . Moreover, when is ≤, P is a maximization problem and each
element of SP is a maximizer of P . Otherwise, P is a minimization problem
and each element of SP is a minimizer of P .
Example 1.4 (Rectangle Problem)
Recall the 2-dimensional description D introduced in Example 1.1. Let f :
R2 → R be such that
f (x1 , x2 ) = 2x1 + 2x2 .
Assume that we want to find a rectangle with the minimum perimeter among
the rectangles satisfying the description D. This can be expressed by the
minimization problem:
P = (D, f, ≥).
√ √
We now show that the rectangle ( b, b) is a minimizer of P . Observe that if
(d1 , d2 ) ∈ XP then d1 = db2 ; that is,
b
XP = , d : d ∈ R+ .
d
Let h : R+ → R be such that
b b
h(d) = f , d = 2 + 2d.
d d
Then, finding a minimizer of P is the same as finding a value for which h is
minimal. Taking the derivative h0 of h we have:
b
h0 (d) = −2 +2
d2
√
which has a unique zero in R+ at d = √ b. Taking into account that the second
derivative of h at d is positive then b is a minimizer of P (see [3]).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4 CHAPTER 1. OPTIMIZATION PROBLEMS
Remark 1.1
In the sequel, we may use
x 7→ f (x)
for presenting a map f .
Example 1.5 (Knapsack Problem)
Recall the n-dimensional description D introduced in Example 1.2. Assume
that each object of kind j has value cj . The knapsack problem consists in
maximizing the value of objects that can be put in the knapsack respecting the
limit weight. Hence, it can be described as the optimization problem
(D, f, ≤),
where f : (x1 , . . . , xn ) 7→ c1 x1 + · · · + cn xn . In Chapter 9, we shall return to
this problem.
Notation 1.3
In the sequel, we present the n-dimensional optimization problem
n bi : 1 ≤ i ≤ m}, U ), f, ≤)
(({gi (x1 , . . . , xn ) o
by
max f (x1 , . . . .xn )
(x1 ,...,xn )
g (x , . . . .xn ) on1 b1
1 1
..
.
gm (x1 , . . . .xn ) o
nm bm
x1 , . . . , x n ∈ U
or even
max f (x)
x
g (x) o
n1 b1
1
..
.
gm (x) o nm bm
x ∈ U n.
Similarly for ≥ using min instead of max. When U = R, we may omit the
constraint on U .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.1. GENERAL CONCEPTS 5
Example 1.6 (Knapsack Problem)
The n-dimensional optimization problem introduced in Example 1.5 can be
presented as follows:
max c x + · · · + cn xn
(x1 ,...,xn ) 1 1
a1 x1 + · · · + an xn ≤ b
x , . . . , x ∈ N.
1 n
We can present the optimization problem in a more condensed way. For
that, we need to introduce the following relations.
Definition 1.5
Let ≤ be the binary relation over Rn such that
x≤y
whenever xj ≤ yj for every j = 1, . . . , n. Similarly, for ≥ and =.
Example 1.7
As an illustration, observe that (0, 1) ≤ (2, 3). Nevertheless, ≤ is not a total
relation. For example
(−3, 5) 6≤ (1, 0) and (1, 0) 6≤ (−3, 5);
that is, (−3, 5) and (1, 0) are not comparable.
Exercise 1.1
Present x and y in Rn such that
• x 6= y;
• x 6≤ y;
• x 6≥ y.
Notation 1.4
Sometimes, it is convenient to group constraints by their associated relational
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6 CHAPTER 1. OPTIMIZATION PROBLEMS
symbols; that is, given an n-dimensional optimization problem of the form
max f (x1 , . . . , xn )
(x1 ,...,xn )
gi0 (x1 , . . . , xn ) ≤ b0i , for i = 1, . . . , m0
gi00 (x1 , . . . , xn ) ≥ b00i , for i = 1, . . . , m00
gi000 (x1 , . . . , xn ) = b000 for i = 1, . . . , m000
i ,
x1 , . . . , xn ∈ U,
where m0 , m00 , m000 ∈ N, we denote by
0
g≤ : Rn → Rm
the ≤-constraint map such that
g≤ (x1 , . . . , xn ) = (g10 (x1 , . . . , xn ), . . . , gm
0
0 (x1 , . . . , xn )).
Similarly, for g≥ and g= . Thus, the previous problem can be presented as
follows:
max f (x1 , . . . , xn )
(x1 ,...,xn )
g≤ (x1 , . . . , xn ) ≤ b0
g≥ (x1 , . . . , xn ) ≥ b00
g= (x1 , . . . , xn ) = b000
x1 , . . . , xn ∈ U,
where b0 = (b01 , . . . , b0m0 ), b00 = (b001 , . . . , b00m00 ) and b000 = (b000 000
1 , . . . , bm000 ). Similarly
for minimization problems.
1.2 Linear Optimization
There are several versions of optimization problems depending on the properties
of the objective map and of the constraint maps (see [55, 10, 29, 32, 11, 53]). In
this book, we concentrate on the linear case. The reader can recall the relevant
(linear) algebraic notions needed in Section 1.6. We consider the vector spaces
Rj over R for j ∈ N+ . Moreover, we assume that the constraint maps and the
objective map are linear maps between appropriate vector spaces as we discuss
now.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.2. LINEAR OPTIMIZATION 7
Definition 1.6
We say that a maximization problem
max f (x1 , . . . , xn )
(x1 ,...,xn )
g≤ (x1 , . . . , xn ) ≤ b0
g≥ (x1 , . . . , xn ) ≥ b00
g= (x1 , . . . , xn ) = b000
x1 , . . . , xn ∈ U
is general linear when f, g≤ , g≥ and g= are linear maps. Moreover, a general
linear problem is linear when U = R. Similarly, for minimization problems.
Example 1.8
The optimization problem
max 2x1 + x2
(x1 ,x2 )
3x1 − x2 ≤ 6
(x1 − 3x2 , x1 , x2 ) ≥ (−6, 0, 0)
is linear.
Notation 1.5
We denote by
L
the set of all linear optimization problems.
Remark 1.2
From now on, we present the linear maps in an optimization problem by the
induced matrices with respect to the standard basis (see Section 1.6). Moreover,
we assume that each line of the induced matrices is non-null.
Notation 1.6
Taking into account the previous remark, any general linear maximization prob-
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8 CHAPTER 1. OPTIMIZATION PROBLEMS
lem can be presented in matricial form either in an expanded way as:
max cx
x
A0 x ≤ b0
A00 x ≥ b00
A000 x = b000
x ∈ U n
or, in a more compact way, as
(A0 , A00 , A000 , b0 , b00 , b000 , c, ≤, U ),
where
0 00 000
• A0 = (a0ij ) ∈ Rm ×n , A00 = (a00ij ) ∈ Rm ×n
and A000 = (a000
ij ) ∈ R
m ×n
;
0 00 000
• b0 = (b0i ) ∈ Rm ×1 , b00 = (b00i ) ∈ Rm ×1
and b000 = (b000
i )∈R
m ×1
;
• c = (cj ) ∈ R1×n .
Similarly for minimization problems. When m0 = 0, then A0 is the empty
matrix and similarly for m00 and m000 . For simplification, when there is no
ambiguity, we omit empty matrices and vectors in the expanded presentation.
Furthermore, as before, when U = R, we may omit the constraint on U .
Example 1.9
The linear optimization problem presented in Example 1.8 can be described in
expanded matricial form as follows:
h i
max 2 1 x
x
h i h i
3 −1 x ≤ 6
1 −3 −6
1 0 x ≥ 0 .
0 1 0
In a more compact way, we can write
1 −3 −6
( 3 −1 , 1 0 , [] , 6 , 0 , [] , 2 1 , ≤).
0 1 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.2. LINEAR OPTIMIZATION 9
Example 1.10 (Diet Problem)
A dog food production factory wants to minimize the fat included in each meal
while complying with specific nutritional requirements establishing that each
meal should have at least a certain quantity of essential nutrients. To do so,
the nutritionist has decided that each meal should use given food components.
Each food component has a certain amount of fat and a certain quantity of
nutrients. To model this problem, assume that:
• the important nutrients range from nutrient 1 to nutrient m;
• the least recommended quantity of nutrient i in each meal is bi ;
• the food components that can be included in a meal range from compo-
nent 1 to component n;
• each food component j contains the quantity aij of nutrient i;
• each food component j contains cj unities of fat.
The goal is to minimize the amount of fat in each meal. The only way to do
so is by adjusting the amount of each food component in a meal. Thus, let
xj
be the amount of food component j in the meal. Hence, the total amount of
fat in a meal is given by:
X n
cj xj .
j=1
The goal of the problem is to minimize this quantity taking account the re-
quirements on the nutrients, which are modeled by
n
X
aij xj ≥ bi
j=1
for i = 1, . . . , m. Therefore, the linear optimization problem can be presented
as follows:
n
X
min cj xj
(x1 ,...,xn ) j=1
Pn
j=1 aij xj ≥ bi , for i = 1, . . . , m
x1 , . . . , x n ≥ 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
10 CHAPTER 1. OPTIMIZATION PROBLEMS
or, in matricial form, as follows:
h i
min c1 . . . cn x
x
a11 ... a1n b1
. . .. .. ..
. . .
.
am1 . . . amn bm
x ≥ .
1 ... 0 0
. .. .. ..
. .
. . .
0 ... 1 0
Exercise 1.2
Let A ∈ Rm×n , b ∈ Rm and f : Rm+n → R be such that
f (x1 , . . . , xn , z1 , . . . , zm ) = z1 + · · · + zm .
Suppose that we want to find the minimizers
(x1 , . . . , xn , z1 , . . . , zm ) ≥ 0
of f fulfilling A(x1 , . . . , xn ) − (z1 , . . . , zm ) ≤ b. Present a linear optimization
formulation of the problem.
Definition 1.7
A linear maximization problem
max cx
x
A0 x ≤ b0
A00 x ≥ b00
A000 x = b000
x ∈ U n
is integer linear when
0 00 000
• A0 = (a0ij ) ∈ Qm ×n , A00 = (a00ij ) ∈ Qm ×n
and A000 = (a000
ij ) ∈ Q
m ×n
;
0 00 000
• b0 = (b0i ) ∈ Qm ×1 , b00 = (b00i ) ∈ Qm ×1
and b000 = (b000
i )∈Q
m ×1
;
• c = (cj ) ∈ Q1×n ;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.3. CANONICAL AND STANDARD OPTIMIZATION PROBLEMS 11
and U = N. Similarly for minimization problems.
Observe that, in the literature, the constraint
x ∈ Nn
may be replaced by the constraints x ≥ 0 and x ∈ Zn .
Example 1.11 (Knapsack Problem)
The optimization problem introduced in Example 1.6 is integer linear.
1.3 Canonical and standard optimization prob-
lems
There are two very important classes of linear optimization problems. In the
sequel, we use I to denote the identity matrix.
Definition 1.8
A linear optimization problem is canonical if it has the form:
max
x
cx
Ax ≤ b
x ≥ 0,
or, in a compact way, (A, I, [] , b, 0, [] , c, ≤).
Notation 1.7
When a problem is known to be canonical, it can simply be presented as a
triple of the form:
(A, b, c).
Notation 1.8
Sometimes, we need to present a canonical optimization problem
max
x
cx
Ax ≤ b
x≥0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
12 CHAPTER 1. OPTIMIZATION PROBLEMS
in the following (pure) form:
max cx
x
" # " #
A b
x ≤ .
−I 0
In the sequel, we may write
A b
and
−I 0
as A and b, respectively.
Example 1.12
The following linear optimization problem
max 2x1 + x2
x
3x1 − x2 ≤ 6
−x1 + 3x2 ≤ 6
x ≥ 0;
that is, h i
max 2 1 x
x
" # " #
3 −1 6
x ≤
−1 3 6
x ≥ 0,
is canonical. Its pure form is:
h i
max 2 1 x
x
3 −1 6
−1 3 6
≤
x .
−1
0 0
0 −1 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.3. CANONICAL AND STANDARD OPTIMIZATION PROBLEMS 13
Exercise 1.3
Let P be a canonical optimization problem. Show that
SP = {s ∈ XP : cs = sup cx}.
x∈XP
Notation 1.9
We denote by
the set of all canonical linear optimization problems.
Definition 1.9
A linear n-dimensional optimization problem is standard if it has the form
min
x
cx
Ax = b
x ≥ 0;
that is, is a tuple of the form ([] , I, A, [] , 0, b, c, ≥).
Notation 1.10
When a problem is known to be standard, it can simply be presented as a triple
of the form:
(A, b, c).
Example 1.13
The following linear optimization problem
min −2x1 − x2
x
3x1 − x2 + x3 = 6
−x1 + 3x2 + x4 = 6
x ≥ 0;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
14 CHAPTER 1. OPTIMIZATION PROBLEMS
that is, h i
min −2 −1 0 0 x
x
" # " #
3 −1 1 0 6
x =
−1 3 0 1 6
x ≥ 0,
is standard.
Notation 1.11
We denote by
S
the set of all standard linear optimization problems.
One of the objectives of this book is to provide techniques for establishing
the existence and for computing optimizers of linear optimization problems.
However, it is essential to get intuition in simple cases by using elementary
geometric arguments.
x2
XP
t = 8 t = 9 t = 10
x1
Figure 1.1: Graphical presentation of problem in Example 1.12.
Example 1.14
Denote the canonical optimization problem in Example 1.12 by P . The set XP
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.4. RELATING PROBLEMS 15
of admissible vectors is depicted in Figure 1.2. We now provide an intuition
for concluding that SP = {(3, 3)}. Since the goal is to maximize f (x1 , x2 ) =
2x1 + x2 , we need to find the maximum t ∈ R such that the intersection of the
line
2x1 + x2 = t
with XP is non-empty. As can be seen in Figure 1.1, this is achieved for t = 9
at (3, 3).
1.4 Relating Problems
The objective of this section is to show how the sets of admissible vectors and
optimizers of different linear optimization problems are related. The following
notation is useful, to extract some coordinates of vectors.
Notation 1.12
Given a set Q ⊆ Rk and 1 ≤ i ≤ j ≤ k, we denote by
Q|ij
the set
{(xi , . . . , xj ) ∈ Rj−i : (x1 , . . . , xi , . . . , xj , . . . , xk ) ∈ Q}.
We omit i when i = 1 and j when j = k.
We start by showing that there is a canonical counterpart of every linear
optimization problem. Moreover, their sets of admissible vectors and optimizers
are the same modulo minor differences.
Proposition 1.1
Let LC : L → C be the map
h i
max −c c y
y
min cx
A0 −A0 b0
x
A0 x ≤ b0
−A00 A00 −b00
LC =
y ≤
A00 x ≥ b00 000 000
−A b000
A
−A000 A000 −b000
000
A x = b000
y ≥ 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
16 CHAPTER 1. OPTIMIZATION PROBLEMS
Then,
XP = XLC(P ) |n − XLC(P ) |n+1 and SP = SLC(P ) |n − SLC(P ) |n+1 ,
for each n-dimensional P ∈ L. Mutatis mutandis for linear maximization
problems.
Proof:
Assume that A0 is an m0 × n-matrix, A00 is an m00 × n-matrix and A000 is an
m000 × n-matrix.
(1) XP = XLC(P ) |n − XLC(P ) |n+1 . Given x ∈ XP , let y ∈ R2n be such that
(
(xj , 0) if xj ≥ 0
(yj , yn+j ) =
(0, −xj ) otherwise
for each j = 1, . . . , n. Observe that
yj − yn+j = xj .
Hence,
y1 yn+1
A0 −A0 y = A0 ... − ... = A0 x ≤ b0 .
yn y2n
The other restrictions for y ∈ XLC(P ) are proved in a similar way. Thus,
y1 yn+1
.. . n+1
. ∈ XLC(P ) |n and .. ∈ XLC(P ) | .
yn y2n
So, x ∈ XLC(P ) |n − XLC(P ) |n+1 , since
y1 yn+1
x = ... − ... .
yn y2n
For the other inclusion, let y ∈ XLC(P ) . Then, it is immediate that
y1 yn+1
.. ..
. − .
yn y2n
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.4. RELATING PROBLEMS 17
is in XP .
(2) SP = SLC(P ) |n − SLC(P ) |n+1 . Given s ∈ SP let r ∈ R2n be such that
(
(sj , 0) if sj ≥ 0
(rj , rn+j ) =
(0, −sj ) otherwise
for each j = 1, . . . , n. Observe that
rj − rn+j = sj .
Let y ∈ XLC(P ) . Then,
r1 rn+1
c r = −c ... + c ... = −cs
−c
rn r2n
≥
y1 yn+1
−c ... − ... = −c
c y.
yn y2n
Thus, r ∈ SLC(P ) and so s ∈ SLC(P ) |n − SLC(P ) |n+1 . The proof of the other
inclusion follows in a similar way. QED
Example 1.15
For instance,
h i
min −3 −5 x
x
" # " #
−5 2 5
LC x ≤ =
2 −1 5
h i h i
6 1 x ≥
−7
h i
max 3 5 −3 −5 y
y
−5 2 5 −2 5
=
2 −1 −2 1 y ≤ 5
−6 −1 6 1 7
y ≥ 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
18 CHAPTER 1. OPTIMIZATION PROBLEMS
Next, our aim now is to relate a standard optimization problem with a
canonical optimization problem in such a way that their sets of admissible
vectors and optimizers are related, in a natural manner.
Proposition 1.2
Let SC : S → C be the map
max −cx
x
min cx
" # " #
x A b
SC Ax = b =
x ≤
−A −b
x≥0
x ≥ 0.
Then, XP = XSC(P ) and SP = SSC(P ) , for each P ∈ S.
Proof:
Assume that A is an m × n-matrix.
(1) XP = XSC(P ) . It is enough to observe that, for every i = 1, . . . , m,
n
X
aij xj ≤ bi
Xn
j=1
aij xj = bi if and only if X n
j=1
− aij xj ≤ −bi .
j=1
(2) SP = SSC(P ) . Indeed,
cs ≤ cx, for every x ∈ XP iff −cx ≤ −cs, for every x ∈ XP
iff −cx ≤ −cs, for every x ∈ XSC(P ) .
Hence, s ∈ SP if and only if s ∈ SSC(P ) . QED
Example 1.16
For instance,
h i
min −2 −1 0 0 x
x
" # " #
3 −1 0 1 6
SC
x =
=
−1 3 1 0 6
x≥0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.4. RELATING PROBLEMS 19
h i
max 2 1 0 0 x
x
3 −1 0 1 6
= −1 3 1 0 6
x ≤
−3 −1 −6
1 0
−3 −1 −6
1 0
x ≥ 0.
We now concentrate on obtaining a standard optimization problem from a
canonical optimization problem.
Proposition 1.3
Let CS : C → S be the map
h i
min −c 0 x̄
x
max cx
x h i
CS Ax ≤ b =
A I x =b
x≥0
x ≥ 0.
Then, f : Rn → Rn+m defined by
x
f (x) =
b − Ax
when restricted to XP is a bijection between XP and XCS(P ) and when re-
stricted to SP is a bijection between SP and SCS(P ) . Hence,
XP = XCS(P ) |n and SP = SCS(P ) |n ,
for each n-dimensional P ∈ C.
Proof:
Assume that A is an m × n-matrix. It is immediate that f is injective. We
now show that f (XP ) = XCS(P ) :
(⊆) Given x ∈ XP , it is easy to check that f (x) ∈ XCS(P ) . Indeed,
A I f (x) = Ax + I(b − Ax) = b.
Furthermore, f (x) ≥ 0 since x ≥ 0 and Ax ≤ b, because x ∈ XP .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
20 CHAPTER 1. OPTIMIZATION PROBLEMS
(⊇) Given x ∈ XCS(P ) , we now show that there exists x ∈ XP such that
f (x) = x. Take x as the vector with the first n components of x and y the
vector with the remaining components. Thus,
x
x= .
y
Then, x ≥ 0 because x ≥ 0. Furthermore, from
A I x=b
it follows that Ax + Iy = b. Since y ≥ 0, it follows that Ax ≤ b. Hence,
x ∈ XP . Moreover, y = b − Ax. So,
x
x= = f (x).
b − Ax
Thus, f when restricted to XP is a bijection between XP and XCS(P ) .
(1) XP = XCS(P ) |n . Let x ∈ XP . Then, f (x) ∈ XCS(P ) . Hence, the vector
with the first n components of f (x) (that is, x) is in XCS(P ) |n . Let x0 ∈ XCS(P ) .
Then, there is x ∈ XP such that f (x) = x0 . Then, the first n components of x0
is in XP since it is x.
(2) SP = SCS(P ) |n . Let s ∈ SP . Let x0 ∈ XCS(P ) . Then,
f (s) = −cs ≤ −cf −1 (x0 ) = x0 .
−c 0 −c 0
Therefore, f (s) ∈ SCS(P ) . Hence, s ∈ SCS(P ) |n . Let s0 ∈ SCS(P ) and x ∈ XP .
Then,
cf −1 (s0 ) = − −c 0 s0 ≥ − −c 0 f (x) = cx.
So, f −1 (s0 ), that is, the vector with the first n components of s0 , is in SP . QED
Example 1.17
For instance,
h i
max 2 1 x
x
" # " #
3 −1 6
CS
x ≤
=
−1 3 6
x≥0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.4. RELATING PROBLEMS 21
h i
min −2 −1 0 0 x
x
" # " #
= 3 −1 1 0 6
x =
−1 3 0 1 6
x ≥ 0.
The following result shows that deciding whether or not the set of admissi-
ble vectors of a canonical optimization problem is non-empty, is equivalent to
deciding whether or not the set of optimizers of another canonical optimization
problem is non-empty.
Proposition 1.4
Let P be the n-dimensional canonical optimization problem
max cx
x
Ax ≤ b
x ≥ 0
and P 0 the n + m-dimensional linear optimization problem
−ux0
max
x0
h i
A −I x0 ≤ b
0
x ≥ 0,
where u ∈ Rn+m is composed by n zeros followed by m ones. Then, XP 6= ∅ if
and only if there exists s0 ∈ SP 0 such that s0j = 0 for j = n + 1, . . . , n + m.
Proof:
Observe that, for every x0 ∈ XP 0 ,
−ux0 = −x0m+1 − · · · − x0m+n ≤ 0.
(→) Assume that XP 6= ∅. Let x ∈ XP . Observe that
(x1 , . . . , xn , 0, . . . , 0) ∈ XP 0 .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
22 CHAPTER 1. OPTIMIZATION PROBLEMS
Therefore,
(x1 , . . . , xn , 0, . . . , 0) ∈ SP 0
since −u(x1 , . . . , xn , 0, . . . , 0) = 0.
(←) Suppose that (s01 , . . . , s0n , 0, . . . , 0) ∈ SP 0 . Then, A(s01 , . . . , s0n ) ≤ b and
(s01 , . . . , s0n ) ≥ 0. Therefore, (s01 , . . . , s0n ) ∈ XP . QED
Finally, we present a map for getting a standard optimization problem with
positive restriction vector from a given standard problem.
Exercise 1.4
Let SP : S → S be the map
h i
min c 0 y
y
+
b+
min cx A 0
x −A −
0 −b−
SP Ax = b =
y =
A0
1 1
x≥0
0 1 1
y ≥ 0,
where
• A+ and b+ are obtained from A and b by removing the lines i such that
bi ≤ 0, respectively;
• A− and b− are obtained from A and b by removing the lines i such that
bi ≥ 0, respectively;
• A0 is the matrix containing the lines of A that are neither in A+ nor in
A− .
Show that, for each P ∈ S,
x s
XSP (P ) = : x ∈ XP and SSP (P ) = : s ∈ SP .
1 1
Exercise 1.5
Define a map for getting a canonical optimization problem with positive re-
striction vector from a given canonical problem in such a way that the sets of
admissible vectors and optimizers are equivalent.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.5. SOLVED PROBLEMS AND EXERCISES 23
1.5 Solved Problems and Exercises
Problem 1.1 (Transportation Problem)
A company wants to minimize the weekly CO2 emission caused by the trans-
portation of a given good between its factories and its distribution centers.
Each distribution center needs at least a certain weekly amount of units of the
good, that may vary from one center to another. Weekly, each factory produces
several units of the product that may vary from one factory to another. Assume
that the weekly CO2 emission for the transportation of a unit of good between
a factory and a distribution center is known. Furthermore, the total weekly
CO2 emission is proportional to the number of units transported. Model this
problem as a linear optimization problem.
Solution:
Assume that
• the factories range from factory 1 to factory m;
• the distribution centers range from center 1 to center n;
• the weekly amount of units of the good required by distribution center j
is at least vj ;
• the weekly amount of units of the good produced by factory i is ui ;
• the weekly CO2 emission for the transportation of a unit of the good
between factory i and distribution center j is eij .
The goal is to minimize the total weekly CO2 emission produced by the trans-
portation of the good between the factories and the distribution centers. The
objective is attained by picking up the adequate number of units that should
be transported from factory i to distribution center j that we represent by
yij .
Therefore, the total weekly CO2 emission is
m X
X n
eij yij .
i=1 j=1
We want to minimize this quantity taking into account the following conditions:
Pm
• i=1 yij ≥ vj for j = 1, . . . , n; that is, the total weekly amount of units
of the good transported to the distribution center j should be at least vj ;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
24 CHAPTER 1. OPTIMIZATION PROBLEMS
Pn
• j=1 yij ≤ ui for i = 1, . . . , m; that is, the total weekly amount of units
of the good transported from factory i is at most ui .
Thus, the problem is represented as follows:
Xm X n
min eij yij
(y11 ,...,ymn )
i=1 j=1
n
X
y ≤u
ij i for i = 1, . . . , m
j=1
X m
yij ≥ vj for j = 1, . . . , n
i=1
(y11 , . . . , ymn ) ≥ 0.
To present the problem in matricial form consider a bijection:
β : {1, . . . , mn} → {1, . . . , m} × {1, . . . , n}.
Hence, the problem is:
h i
min e β(1) . . . e β(mn) x
x
a11 . . . a1 mn u1
.. .. .. .
. x ≤ .
. . .
am1 . . . am mn um
a011 . . . a01 mn
v1
. .. .. .
. . x ≥ . ,
. . .
0 0
an1 . . . an mn vn
where (
1 if β(j)1 = i
aij =
0 otherwise
and (
1 if β(j)2 = i
a0ij =
0 otherwise
and x is (yβ(1) , . . . , yβ(mn) ). /
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.5. SOLVED PROBLEMS AND EXERCISES 25
Problem 1.2
Consider the following linear optimization problem P :
max 2x1 + x2
x
−x1 − x2 ≤ −1
x1 + x2 ≤ 3
x1 − x2 ≤ 1
−x1 + x2 ≤ 1
x ≥ 0.
(1) Present the problem in matricial form.
(2) Define and provide a graphical presentation of XP . Verify whether or not
(2, 1), (0, 2) and (1, 1) are in XP .
(3) Verify whether or not (0, 2) and (1, 1) are in SP .
(4) Find the standard optimization problem corresponding to P according to
the reduction CS : C → S.
(5) Find the canonical optimization problem corresponding to CS(P ) according
to the reduction SC : S → C.
(6) Is CS : C → S the inverse of SC : S → C?
Solution:
(1) The matricial form is as follows:
h i
max 2 1 x
x
−1 −1 −1
1 1
3
x ≤
1 −1
1
−1 1 1
x ≥ 0.
(2) The set XP of admissible vectors is:
{(x1 , x2 ) ∈ R2 : −x1 −x2 ≤ −1, x1 +x2 ≤ 3, x1 −x2 ≤ 1, −x1 +x2 ≤ 1, x1 , x2 ≥ 0}.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
26 CHAPTER 1. OPTIMIZATION PROBLEMS
x2
XP
x1
Figure 1.2: Set of admissible vectors.
Vector (2, 1) is in XP since
−1 −1 −3 −1
1 1 2 3 3
1 ≤ 1
=
1 −1 1
−1 1 −1 1
and
2 0
≥ .
1 0
Vector (0, 2) is not in XP since
−1 −1 −2 −1
1 1 0
2 3
−2 ≤
= 6
1 .
1 −1 2
−1 1 2 1
Vector (1, 1) is in XP since
−1 −1 −2 −1
1 1 1 2 3
0 ≤ 1
=
1 −1 1
−1 1 0 1
and
1 0
≥ .
1 0
(3) Vector (0, 2) is not in SP since it is not in XP .
Vector (1, 1) is not in SP since (2, 1) is in XP and
2
2 1 =5
1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.5. SOLVED PROBLEMS AND EXERCISES 27
and
1
2 1 = 3.
1
So, (1, 1) is not a maximizer of P .
(4) The standard optimization problem CS(P ) corresponding to P according
to the reduction CS : C → S is as follows:
min −2x1 − x2
x
−x 1 − x2 + x3 = −1
x1 + x2 + x4 = 3
x1 − x2 + x5 = 1
−x1 + x2 + x6 = 1
x ≥ 0
or in matricial form
h i
min −2 −1 0 0 0 0 x
x
−1 −1 1 0 0 0 −1
1 1 0 1 0 0 3
x =
1 −1
0 0 1 0 1
−1
1 0 0 0 1 1
x ≥ 0.
(5) The canonical optimization problem SC(CS(P )) corresponding to CS(P )
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
28 CHAPTER 1. OPTIMIZATION PROBLEMS
according to the reduction SC : S → C is as follows:
max 2x1 + x2
x
−x1 − x2 + x3 ≤ −1
x1 + x2 + x4 ≤ 3
x1 − x2 + x5 ≤ 1
−x1 + x2 + x6 ≤ 1
+x1 + x2 − x3 ≤1
−x1 − x2 − x4 ≤ −3
−x1 + x2 − x5 ≤ −1
+x1 − x2 − x6 ≤ −1
x1 , . . . , x6 ≥ 0
or in matricial form:
h i
max 2 1 0 0 0 0 x
x
−1 −1 1 0 0 0 −1
1 1 0 1 0 0 3
1
−1 0 0 1 0
1
−1
1 0 0 0 1 1
x ≤
1 −1
1 0 0 0 1
−1 −1 0 −1 0 0 −3
−1 1 0 0 −1 0 −1
1 −1 0 0 0 −1 −1
x ≥ 0.
(6) The map CS : C → S is inverse of the map SC : S → C if and only if
SC CS id
• CS ◦ SC = idS ; that is, S → C → S is equal to S →S S;
CS SC id
• SC ◦ CS = idC ; that is, C → S → C is equal to C →C C.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.5. SOLVED PROBLEMS AND EXERCISES 29
Hence, the map CS is not the inverse of the map SC since
SC(CS(P )) 6= P,
as we have seen before. /
Exercise 1.6
Consider the following linear optimization problem P :
min −3x1 − 4x2
x
−8x1 − 3x2 ≥ −5
−6x1 + 4x2 ≥ 5
2x1 − x2 ≤ 2.
Present the problem in matricial form. Obtain the corresponding canonical
optimization problem given by the map LC : L → C. Relate their sets of
admissible vectors and optimizers.
Exercise 1.7
Consider the following canonical optimization problem P :
max 2x1 + x2
x
x1 ≤ 5
4x1 + x2 ≤ 25
x ≥ 0.
(1) Present the problem in matricial form.
(2) Define and provide a graphical presentation of XP . Verify whether or not
(5, 5) and (7, 2) are in XP .
(3) Verify whether or not (7, 2) is in SP .
(4) Find the standard optimization problem corresponding to P according to
the reduction CS : C → S. Present XCS(P ) and SCS(P ) .
(5) Find the canonical optimization problem corresponding to CS(P ) according
to the reduction SC : S → C. Present XSC(CS(P )) and SSC(CS(P )) .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
30 CHAPTER 1. OPTIMIZATION PROBLEMS
1.6 Relevant Background
We start by introducing some basic algebraic concepts and notations, namely
about groups, fields and vector spaces. The interested reader can consult [38,
15, 7, 47, 59].
Definition 1.10
A group is a pair
(G, +),
where
• G is a non-empty set;
• + : G2 → G is a map;
such that
• x + (y + z) = (x + y) + z;
• there exists 0 ∈ G such that 0 + x = x + 0 = x (identity);
• for every x ∈ G there is y ∈ G with x + y = 0.
The group is Abelian whenever x + y = y + x for every x, y ∈ G.
Example 1.18
The pair
(R, +),
where + is the sum of real numbers, is an Abelian group. On the other hand,
(N, +), where + is the sum of natural numbers, is not a group.
Example 1.19
Let (G, +) be a group. Then, the pair
(Gn , +),
where
• Gn = {(x1 , . . . , xn ) : xj ∈ G, j = 1, . . . , n};
• (x1 , . . . , xn ) + (y1 , . . . , yn ) = (x1 + y1 , . . . , xn + yn );
is a group. Observe that the identity is the tuple (0, . . . , 0).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.6. RELEVANT BACKGROUND 31
Remark 1.3
Given a group (G, +) and x ∈ G, we denote by
−x
the unique element of G such that
x + (−x) = 0.
Moreover, we may write
x−y
instead of
x + (−y)
for every x, y ∈ G.
Definition 1.11
A field is a tuple
(K, +, ×),
where
• K is a non-empty set;
• +, × : K 2 → K;
such that
• (K, +) is an Abelian group with identity 0;
• x × (y × z) = (x × y) × z;
• x × y = y × x;
• x × (y + z) = (x × y) + (x × z);
• there is 1 ∈ K such that 1 × x = x (multiplicative identity);
• 0 6= 1;
• if x 6= 0 then there is w ∈ K such that x × w = 1;
for every x, y, z ∈ K.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
32 CHAPTER 1. OPTIMIZATION PROBLEMS
Example 1.20
The tuple
(Q, +, ×),
where + and × are the usual operations over rational numbers, is a field.
Furthermore, the tuple
(R, +, ×),
where + and × are the usual operations over real numbers, is a field.
Notation 1.13
In the sequel, we may denote a field (K, +, ×) by K.
We are ready to present the essential concept of vector space.
Definition 1.12
A vector space over a field K is an Abelian group (V, +) with a map
(α, x) 7→ αx : K × V → V
satisfying the following properties:
• (αµ)x = α(µx);
• (α + µ)x = αx + µx;
• α(x + y) = αx + αy;
• 1x = x, where 1 is the multiplicative identity of K.
The elements of V and the elements of K are called vectors and scalars, re-
spectively.
Example 1.21
Let K be a field, {v} a singleton set and + : {v}2 → {v} such that v + v = v.
Then, ({v}, +) and the map (α, v) 7→ v : K × {v} → {v} is a vector space over
K.
Notation 1.14
Consider the vector space introduced in Example 1.21. We start by observing
that v is the identity of the group and so is usually denoted by 0. Furthermore,
we denote such a vector space by
{0}K
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.6. RELEVANT BACKGROUND 33
and refer to it as the zero vector space over K.
Definition 1.13
The vector space induced by a field K and n ∈ N+ , denoted by
K n,
is the vector space (K n , +) over K with the map
(α, (β1 , . . . , βn )) 7→ (α × β1 , . . . , α × βn ),
where (α1 , . . . , αn ) + (β1 , . . . , βn ) = (α1 + β1 , . . . , αn + βn ). The
(0, . . . , 0)
element of K n , denoted by 0, is called the null vector .
Example 1.22
We denote by
Rn ,
where n ∈ N+ , the vector space induced by R and n.
We now introduce some notions and results related to subspaces.
Definition 1.14
A subspace S of a vector space V over a field K is a subset of V containing 0,
closed under vector addition and multiplication by scalar.
Proposition 1.5
A subspace of a vector space V over a field K with the multiplication by scalar
and addition induced by V in S is a vector space over K.
Example 1.23
The smallest subspace of a vector space V over a field K is {0}K . That is,
{0}K ⊆ V1
for every subspace V1 of V .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
34 CHAPTER 1. OPTIMIZATION PROBLEMS
Definition 1.15
Let V be a vector space over a field K and V1 , V2 ⊆ V . Then,
V1 + V2 ,
called the sum of V1 and V2 , is the set
{v 1 + v 2 : v 1 ∈ V1 and v 2 ∈ V2 }.
Proposition 1.6
The class of all subspaces of a vector space over a field K is closed under
intersections and finite sums. On the other hand, it is not closed under unions
except in trivial case.
We now introduce the important concept of span of a set of vectors.
Definition 1.16
Let V be a vector space over a field K and U ⊆ V . Then, the set
spanV (U ),
the span of U in V , is
\
V1 .
{V1 : V1 is a subspace of V and U ⊆V1 }
The following result is a direct consequence of Proposition 1.6.
Proposition 1.7
Let V be a vector space over a field K and U ⊆ V . Then, spanV (U ) is a
subspace of V .
Example 1.24
Let V be a vector space over a field K. Note that
spanV (∅) = {0}K ,
since {0}K is a subspace of V containing ∅ and
{0}K
is the smallest subspace of V , see Example 1.23.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.6. RELEVANT BACKGROUND 35
Proposition 1.8
Let V be a vector space over a field K and V1 , V2 ⊆ V . Then,
spanV (V1 ∪ V2 ) = V1 + V2 .
The following result provides another characterization of span of a subset
in V assuming the convention that
0
X
αj uj
j=1
is the 0 vector in V .
Proposition 1.9
Let V be a vector space over a field K and U ⊆ V . Then,
X k
spanV (U ) = αj uj : k ∈ N, αj ∈ K, uj ∈ U .
j=1
Hence, the span of a set U is seen as the set of all linear combination of
vectors in U .
We now discuss finite-dimensional vector spaces, and begin by stating their
definition.
Definition 1.17
A vector space V over a field K is finite-dimensional whenever there is a finite
set U ⊆ V such that spanV (U ) = V .
Remark 1.4
From now on, when referring to vector spaces we mean finite-dimensional vector
spaces.
Definition 1.18
A finite set U is linearly independent in a vector space V over a field K whenever
U ⊆ V and for any u1 , . . . , un ∈ U and α1 , . . . , αn ∈ K, if
α1 u1 + · · · + αn un = 0
then α1 , . . . , αn = 0. Otherwise, the set is linearly dependent in V .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
36 CHAPTER 1. OPTIMIZATION PROBLEMS
Example 1.25
The sets
{(1, 0), (0, 1)} and {(2, 2), (−3, 2)}
are linearly independent in R2 . On the other hand, the set
{(1, 1), (2, 2)}
is not linearly independent in R2 . Indeed, taking α1 = 2 and α2 = −1 we get
2(1, 1) − (2, 2) = 0.
Example 1.26
The set
∅
is linearly independent in every vector space.
Proposition 1.10
Let V be a vector space over a field K and U1 ⊆ U2 ⊆ V . Then, U1 is linearly
independent in V whenever U2 is linearly independent in V .
Definition 1.19
Let V be a vector space over a field K. A finite set B is a basis of V whenever
V = spanV (B)
and B is a linearly independent set in V .
Example 1.27
The set {(1, 0), (0, 1)} is a basis of R2 .
Definition 1.20
The standard or canonical basis of a vector space K n induced by a field K is
composed by the vectors
e1 = (1, 0, . . . , 0), e2 = (0, 1, 0, . . . , 0), . . . , en = (0, . . . , 0, 1).
Example 1.28
Let K be a field. Observe that
∅ is a basis of {0}K
taking into account Example 1.24 and Example 1.26.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
1.6. RELEVANT BACKGROUND 37
An important consequence of the following result is that every vector space
has a basis.
Proposition 1.11
Let V be a vector space over a field K and U ⊆ V a finite set with spanV (U ) =
V . Then, there is a linearly independent set B ⊆ U in V such that spanV (B) =
V.
Proposition 1.12
Every vector space over a field K has a basis. Moreover, all bases have the
same number of elements.
We now define maps between vector spaces over the same field.
Definition 1.21
Let V and W be vector spaces over the same field K. We say that a map
h : V → W is linear whenever the following properties hold:
• h(x + y) = h(x) + h(y);
• h(αx) = αh(x);
for x, y ∈ V and α ∈ K.
Exercise 1.8
Show that h is linear if and only if h(αx + βy) = αh(x) + βh(y).
Example 1.29
Let W be a vector space over K. Then, h : {0}K → W such that h(0) = 0 is a
linear map.
Definition 1.22
Let V and W be vector spaces over the same field K with bases v1 , . . . , vn and
w1 , . . . , wm , respectively. The m × n-matrix, or simply the matrix , induced by
the linear map h : V → W with respect to the given bases, denoted by
M (h, (v1 , . . . , vn ), (w1 , . . . , wm )),
is an m-by-n tuple
(aij ) ∈ K m×n
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
38 CHAPTER 1. OPTIMIZATION PROBLEMS
such that
h(vj ) = a1j w1 + · · · + amj wm .
Notation 1.15
When either V or W is {0}K , the matrix induced by the linear map h : V → W
is called the empty matrix , denoted by
[ ].
Notation 1.16
When V and W are the vector spaces Rn and Rm over R, respectively, and
v1 , . . . , vn and w1 , . . . , wm are their standard or canonical bases, we may write
M (h)
for M (h, (v1 , . . . , vn ), (w1 , . . . , wm )).
Example 1.30
Let h : R2 → R3 be such that h(x1 , x2 ) = (x1 − 3x2 , x1 , x2 ). Then,
M (h)
is the matrix
1 −3
1 0 .
0 1
Remark 1.5
In the sequel, we may refer to a vector (x1 , . . . , xn ) ∈ Rn as
x1
..
. .
xn
Remark 1.6
In the sequel, we may present a matrix without referring to the underlying
linear map.
Definition 1.23
A submatrix of a matrix is obtained by deleting any collection of rows and/or
columns of the matrix.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 2
Optimizers
This chapter investigates the existence of optimizers for linear optimization
problems using topological techniques. In Section 2.1, we introduce the notion
of interior and boundary of a canonical optimization problem and show that
these concepts coincide with the topological ones with the same name. The
main result of the section states that no maximizer is an interior point when c 6=
0. In Section 2.2, we prove sufficient conditions for the existence of maximizers
after showing that the sets of admissible vectors and maximizers are closed.
We also provide in Section 2.4, a modicum of relevant topological notions and
results.
2.1 Boundary
The first insight provided by the topological analysis of the linear optimization
problem is the characterization of the subset of the set of admissible vectors
where we should look for the optimizers. We start by defining the interior and
the boundary of the set of admissible vectors. It turns out that these notions
agree with the corresponding topological notions.
Definition 2.1
The interior of the set of admissible vectors of a canonical n-dimensional op-
timization problem P
max
x
cx
Ax ≤ b
x ≥ 0,
39
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
40 CHAPTER 2. OPTIMIZERS
denoted by
XP◦ ,
is the set {x ∈ Rn : Ax < b and x > 0}. The boundary of XP , denoted by
∂XP ,
is the set XP \ XP◦ .
Example 2.1
Recall Example 1.12 and Example 1.14. Note that
• XP = {(x1 , x2 ) ∈ R2 : 3x1 − x2 ≤ 6, −x1 + 3x2 ≤ 6, x1 ≥ 0, x2 ≥ 0};
• XP◦ = {(x1 , x2 ) ∈ XP : 3x1 − x2 < 6, −x1 + 3x2 < 6, x1 > 0, x2 > 0};
• ∂XP is the set
{(x1 , x2 ) ∈ XP : 3x1 − x2 = 6}
∪{(x1 , x2 ) ∈ XP : −x1 + 3x2 = 6}
∪{(x1 , x2 ) ∈ XP : x1 = 0}
∪{(x1 , x2 ) ∈ XP : x2 = 0}.
Then, (1, 1) ∈ XP◦ and (3, 3) ∈ ∂XP . Moreover, (3, 3) ∈
/ XP◦ and (1, 1) ∈
/ ∂XP .
We are now ready to prove that the interior of the set of admissible vectors
of a linear optimization problem coincides with the topological interior (see
Section 2.4). Recall that each line of matrix A is non-null (see Remark 1.2).
The first step is to characterize, from a topological point of view, the constraint
map (as well as the objective map).
Proposition 2.1
Every linear map from Rn to Rm is continuous.
Proof:
Observe that |xj | ≤ kxk, for every j = 1, . . . , n, by Proposition 2.12. Therefore,
n
X
|xj | ≤ nkxk (†).
j=1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.1. BOUNDARY 41
Let f : Rn → Rm be a linear map and µ = max{kf (ej )k : j = 1, . . . , n} (see
Definition 1.20). Then,
n
X
kf (x)k = kf ( xj ej )k
j=1
n
X
= k xj f (ej )k by linearity of f (see Exercise 1.8)
j=1
n
X
≤ kxj f (ej )k by triangular inequality
j=1
Xn
= |xj |kf (ej )k by positive homogeneity
j=1
Xn
≤ µ |xj | by definition of µ
j=1
≤ µnkxk by (†);
that is,
kf (x)k ≤ µnkxk, (‡).
Finally, we show that f is continuous. Consider two cases:
(1) µ = 0. Then, f (x) = 0 for every x (by positive definiteness). It is immedi-
ate that f is continuous.
(2) µ 6= 0. Let {xk }k∈N be a sequence converging to x. We show that
{f (xk )}k∈N is a sequence converging to f (x). Take any δ > 0. Note that
δ
∈ R+ .
µn
Since {xk }k∈N converges to x, there exists j such that
δ
kxm − xk <
µn
for every m ≥ j. Hence, for every m ≥ j, it follows that
kf (xm ) − f (x)k = kf (xm − x)k by linearity of f
≤ µnkxm − xk by (‡)
< δ.
Thus, {f (xk )}k∈N converges to f (x). QED
Hence, we have the following corollary (recall Notation 1.8).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
42 CHAPTER 2. OPTIMIZERS
Proposition 2.2
The objective and the constraint maps of a (pure) canonical optimization prob-
lem are continuous.
Proof:
Immediate by Proposition 2.1 taking into account that, by definition, the ob-
jective and the constraint maps of a linear optimization problem are linear
maps. QED
We now show that the interior of a canonical optimization problem coincides
with its topological counterpart (see Notation 2.1).
Proposition 2.3
Consider a canonical linear optimization problem P . Then,
XP◦ = int(XP ).
Proof:
Assume that P is in pure form and that A is an m × n matrix.
(⊆) Let x ∈ XP◦ . Then,
Ax < b.
We must find ε > 0 such that
Bε (x) ⊂ XP .
Let
δ = min{(bi − (Ax)i ) : 1 ≤ i ≤ m + n}.
Note that δ > 0. Since x 7→ Ax is continuous (by Proposition 2.2), then, by
Proposition 2.23, there exists ε > 0 such that
if ky − xk < ε then kAy − Axk < δ.
In other words, let ε > 0 be such that
if y ∈ Bε (x) then kAy − Axk < δ.
We now show that Bε (x) ⊆ XP◦ . Let y ∈ Bε (x). Then, from kAy − Axk < δ,
using Proposition 2.12, it follows that
|(Ay)i − (Ax)i | < δ
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.1. BOUNDARY 43
for each i = 1, . . . , m + n. We prove that
(Ay)i < bi
for every i = 1, . . . , m + n. There are two cases to consider:
(1) (Ay)i ≤ (Ax)i . Then, we have (Ay)i < bi since (Ax)i < bi .
(2) (Ay)i > (Ax)i . Hence,
(Ay)i − (Ax)i = |(Ay)i − (Ax)i | < δ.
Therefore,
(Ay)i < δ + (Ax)i ≤ bi − (Ax)i + (Ax)i = bi .
Therefore, y ∈ XP◦ . Since XP◦ ⊂ XP , then Bε (x) ⊂ XP . Hence, x ∈ int(XP ).
(⊇) Assume that x ∈ int(XP ). Then, let ε > 0 be such that Bε (x) ⊂ XP . Take
v ∈ Rn such that
• vj 6= 0 for j = 1, . . . , n;
Pn
• j=1 aij vj 6= 0 for i = 1, . . . , m (see Remark 1.2);
• kvk < ε.
Then, x + v ∈ Bε (x) and x − v ∈ Bε (x) by the last constraint on v. So, by
hypothesis, x + v ∈ XP and x − v ∈ XP . We now prove that
x ∈ XP◦ .
Indeed:
(1) xj > 0 for j = 1, . . . , n. Observe that
(
(†) xj + vj ≥ 0
(‡) xj − vj ≥ 0.
By the first constraint on v, vj 6= 0. So, we have two cases:
(a) vj < 0. Then, by (†), xj > 0;
(b) vj > 0. Then, by (‡), xj > 0.
(2) (Ax)i < bi , for i = 1, . . . , m using the second constraint on v and reasoning
as in (1). Observe that (Ax)i + (Av)i ≤ bi and (Ax)i − (Av)i ≤ bi . QED
Exercise 2.1
Let P be a canonical optimization problem. Show that ∂XP = bn(XP ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
44 CHAPTER 2. OPTIMIZERS
x3
(0,0,3) (0,3,3)
(2,4,3)
(4,0,3)
(0,3,0)
(0,0,0)
x2
(2,4,0)
(4,0,0)
(12,4,3)
(12,4,0)
x1
Figure 2.1: Set of admissible vectors of problem in Example 2.2
Example 2.2
Consider the following linear optimization problem P
max 3x1 + 3x2 + 2x3
(x1 ,x2 ,x3 )
x2 ≤ 4
x3 ≤ 3
2x2 − x1 ≤ 6
−2x2 + x1 ≤ 4
x1 , x2 , x3 ≥ 0
with set of admissible vectors depicted in Figure 2.1. We now present examples
of vectors in the interior and in the boundary of XP . We start by showing that
(2, 2, 2) ∈ XP◦ .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.1. BOUNDARY 45
Observe that XP◦ is the set
{(x1 , x2 , x3 ) ∈ (R+ )3 : x2 < 4, x3 < 3, 2x2 − x1 < 6, −2x2 + x1 < 4}.
Hence, (2, 2, 2) ∈ XP◦ .
Another way to show that (2, 2, 2) ∈ XP◦ is to prove that
(2, 2, 2) ∈ int(XP )
and then use Proposition 2.3. Hence, we must find a real number ε > 0
such that Bε (2, 2, 2) ⊂ XP . Take ε equal to 12 . It remains to show that
B 12 (2, 2, 2) ⊆ XP . Let y ∈ B 12 (2, 2, 2). Then,
1
||(2, 2, 2) − (y1 , y2 , y3 )|| < .
2
That is,
1
(2 − y1 )2 + (2 − y2 )2 + (2 − y3 )2 < .
4
Thus, (2 − y1 )2 < 14 , (2 − y2 )2 < 1
4 and (2 − y3 )2 < 14 . Therefore,
1 1 1
|2 − y1 | < , |2 − y2 | < and |2 − y3 | < .
2 2 2
So,
3 5 3 5 3 5
< y1 < , < y2 < and < y3 < .
2 2 2 2 2 2
Capitalizing on these bounds, it is immediate that y ∈ XP . So, (2, 2, 2) ∈
int(XP ). Hence,
(2, 2, 2) ∈
/ bn(XP ).
Moreover, (2, 2, 2) ∈ XP◦ , by Proposition 2.3. Furthermore, (2, 2, 2) ∈
/ ∂XP , by
Exercise 2.1. We now prove that
(12, 4, 2) ∈ ∂XP .
Observe that ∂XP is the set
{(x1 , x2 , x3 ) ∈ XP : x2 = 4} ∪ · · · ∪ {(x1 , x2 , x3 ) ∈ XP : x3 = 0}.
So, (12, 4, 2) ∈ ∂XP since (12, 4, 2) is in
{(x1 , x2 , x3 ) ∈ XP : x2 = 4}.
Another way to show that (12, 4, 2) ∈ ∂XP is by proving that
(12, 4, 2) ∈ bn(XP )
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
46 CHAPTER 2. OPTIMIZERS
and then use Exercise 2.1. Hence, we must prove that for every real number
ε > 0 there is y ∈ Bε (12, 4, 2) ∩ XP and there is y ∈ Bε (12, 4, 2) ∩ (R3 \ XP ).
Let ε > 0 be a real number. Observe that
(12, 4, 2) ∈ Bε (12, 4, 2) ∩ XP .
On the other hand, consider the vector
ε ε ε
12 + , 4 + , 2 + .
2 2 2
Then,
ε ε ε
(12 + , 4 + , 2 + ) ∈ Bε (12, 4, 2)
2 2 2
since
ε ε ε
||(12, 4, 2) − (12 + 2ε , 4 + 2ε , 2 + 2ε )|| = ||(− , − , − )||
2 2 2
ε2 ε2 ε2 1
= ( + + )2
4 4 4
√
3
= ε
2
< ε.
On the other hand, it is immediate that
ε ε ε
12 + , 4 + , 2 + ∈
/ XP .
2 2 2
The next result is an important property of the set of optimizers of a canon-
ical optimization problem.
Proposition 2.4
Let P be a canonical optimization problem. If c 6= 0 then
SP ∩ XP◦ = ∅.
Proof:
The result is established by contradiction. Take s ∈ SP ∩ XP◦ . Then, As < b
and s > 0. Consider the family of vectors
{sε }ε∈R+ , where sε = s + εcT .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.1. BOUNDARY 47
Then,
csε > cs
for any ε ∈ R+ , because cεcT = εccT > 0, since c 6= 0.
We now show that there is ε such that sε ∈ XP . We concentrate first on
choosing a set D such that sε ≥ 0 for every ε ∈ D. Let
sj
D = {ε ∈ R+ : ε ≤ − for j = 1, . . . , n whenever cj < 0}.
cj
Consider two cases:
(1) cj ≥ 0 for every j. Then, for every j, sεj = sj + εcj ≥ 0 since sj > 0 and
cj ≥ 0. So, sε ≥ 0 for every ε ∈ R+ = D.
(2) cj < 0 for some j. Observe that
i
i sj
D = 0, min − : cj < 0 .
cj
Let k ∈ {1, . . . , n}. When ck ≥ 0, then sεk ≥ 0, similarly to (1). Otherwise,
ε sk
sk = sk + εck ≥ sk + − ck = 0.
ck
We now identify a set E such that Asε ≤ b for every ε ∈ E. Let
bi − (As)i
E = {ε ∈ R+ : ε ≤ for i = 1, . . . , m whenever (AcT )i > 0}.
(AcT )i
Consider two cases:
(1) (AcT )i ≤ 0 for every i = 1, . . . , m. Then, (As)i + ε(AcT )i < bi since
(As)i < bi and (AcT )i ≤ 0. Thus, Asε ≤ b.
(2) (AcT )i > 0 for some i. Observe that
i
i bi − (As)i T
E = 0, min : (Ac )i > 0 .
(AcT )i
Let k ∈ {1, . . . , m}. When (AcT )k ≤ 0, then (Asε )k ≤ bk , similarly to (1).
Otherwise,
bk − (As)k
(Asε )k = (As)k + ε(AcT )k ≤ (As)k + (AcT )k = bk .
(AcT )k
In any case,
D ∩ E 6= ∅.
Pick up ε ∈ D ∩ E. Then, sε is admissible and csε > cs contradicting the
hypothesis that s ∈ SP . QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
48 CHAPTER 2. OPTIMIZERS
x2
XP
x1
Figure 2.2: Boundary and interior of problem in Example 1.12
In other words, if c 6= 0 then the maximizers of the problem must lie on the
boundary of the set of admissible vectors.
Example 2.3
Consider the canonical optimization problem P in Example 1.12 and the graph-
ical representation of XP in Figure 1.1. In Figure 2.2, we use different colors to
represent the interior and the boundary of XP . Since c 6= 0, by Proposition 2.4,
any maximizer of P is in the darker part of the representation of XP .
2.2 Existence
The objective of this section is to provide sufficient conditions for the existence
of optimizers. We start by proving a result about the existence of optimizers
of a canonical optimization problem when b is 0.
Proposition 2.5
In a canonical optimization problem, if b = 0 then either 0 is a maximizer or
the objective map has no upper bound on the set of admissible vectors.
Proof:
Let P be a canonical optimization problem. Note that 0 is admissible. If 0 is
a maximizer, the thesis follows immediately. Otherwise, observe that there is
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.2. EXISTENCE 49
y ∈ XP such that cy > c0 = 0. We want to show that
{cx : x ∈ XP }
has no upper bound. Let r ∈ R. Take z as the vector
|r| + 1
y.
cy
|r|+1
Observe that z ≥ 0, since y ≥ 0 and cy > 0. Moreover,
|r| + 1 |r| + 1
Az = A y= Ay ≤ 0.
cy cy
So, z ∈ XP and cz ∈ {cx : x ∈ XP }. On the other hand,
|r| + 1 |r| + 1
cz = c y= cy = |r| + 1 > r.
cy cy
Thus, the objective map has no upper bound on XP . QED
As we discuss below, topological notions are useful in the study of the
existence of optimizers. We start by proving that the set of admissible vectors
and the set of optimizers of the canonical optimization problem are closed (see
Definition 2.9).
Proposition 2.6
The set {x ∈ Rn : x ≥ 0} is closed.
Proof:
Let {xk }k∈N be a sequence in {x ∈ Rn : x ≥ 0} converging to z ∈ Rn . Moreover,
let {ak }k∈N be a sequence in Rn such that ak = 0 for every k ∈ N. Observe
that {ak }k∈N converges to 0 ∈ Rn by Proposition 2.17. On the other hand,
xk ≥ ak for every k ∈ N. Hence, by Proposition 2.16, z ≥ 0. Thus, z ∈ {x ∈
Rn : x ≥ 0}. So, by Proposition 2.18, set {x ∈ Rn : x ≥ 0} is closed. QED
Proposition 2.7
Let A be an m × n-matrix and b ∈ Rm . Then, {x ∈ Rn : Ax ≤ b} is closed.
Proof:
Let {xk }k∈N be a sequence in {x ∈ Rn : Ax ≤ b} converging to z ∈ Rn . Then,
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
50 CHAPTER 2. OPTIMIZERS
{Axk }k∈N is a sequence converging to Az ∈ Rm since x 7→ Ax : Rn → Rm
is continuous by Proposition 2.2. Moreover, let {ak }k∈N be a sequence in
Rm such that ak = b for every k ∈ N. Observe that {ak }k∈N converges to
b ∈ Rm , by Proposition 2.17. On the other hand, Axk ≤ ak for every k ∈ N.
Therefore, by Proposition 2.16, Az ≤ b. Hence, z ∈ {x ∈ Rn : Ax ≤ b}. So, by
Proposition 2.18, set {x ∈ Rn : Ax ≤ b} is closed. QED
Proposition 2.8
The set of admissible vectors of a canonical optimization problem is closed.
Proof:
Let P be a canonical optimization problem. Observe that
XP = {x ∈ Rn : x ≥ 0} ∩ {x ∈ Rn : Ax ≤ b}.
Therefore, XP is closed by Proposition 2.19, since {x ∈ Rn : x ≥ 0} is closed
by Proposition 2.6 and {x ∈ Rn : Ax ≤ b} is closed by Proposition 2.7. QED
Proposition 2.9
The set of optimizers of a canonical optimization problem is closed.
Proof:
Let P be a canonical optimization problem. If SP = ∅ then SP is closed (see
Remark 2.2). Otherwise, let {sk }k∈N be a sequence in SP converging to s. By
Proposition 2.2, the map x 7→ cx is continuous and so the sequence {csk }k∈N
converges to cs. On the other hand, since each sk ∈ SP , then
csk = max{cx : x ∈ XP }
for every k. Hence, {csk }k∈N is a constant sequence and so, by Proposition 2.15,
{csk }k∈N converges to max{cx : x ∈ XP }. Therefore,
cs = max{cx : x ∈ XP }.
On the other hand, s ∈ XP , by Proposition 2.18, since, by Proposition 2.8, XP
is closed. Therefore, s ∈ SP . So, by Proposition 2.18, SP is closed. QED
The following result, called the Existence of Maximizer Theorem, estab-
lishes a sufficient condition for the existence of a maximizer of a canonical
optimization problem.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.2. EXISTENCE 51
Theorem 2.1
When the set of admissible vectors is non-empty and bounded, there exists a
maximizer for a canonical optimization problem.
Proof:
Let P be a canonical optimization problem. The set XP is bounded, by hypoth-
esis, and closed, by Proposition 2.8. Hence, XP is compact (see Definition 2.12).
Observe that, the objective map x 7→ cx is continuous, by Proposition 2.2.
Thus, since XP 6= ∅, by Proposition 2.28, the objective map x 7→ cx has a
maximum in XP . Therefore, P has a maximizer. QED
Example 2.4
Recall the canonical optimization problem P in Example 2.2. We now show
that XP is bounded. We must prove that there is a positive real number µ
such that
||x|| ≤ µ
for every x ∈ XP . Take µ = 13. Let x ∈ XP . Then,
x2 ≤ 4, x3 ≤ 3 and − 2x2 + x1 ≤ 4.
Hence,
x1 ≤ 4 + 2x2 ≤ 12.
Therefore,
1 1
||x|| = (x21 + x22 + x23 ) 2 ≤ (122 + 42 + 32 ) 2 .
On the other hand,
XP 6= ∅
since (2, 2, 2) ∈ XP , as shown in Example 2.2. Thus, by Theorem 2.1,
SP 6= ∅.
Moreover, no maximizer of P is in XP◦ , by Proposition 2.4 since c 6= 0 for P .
Consider the problem P 0 obtained from P by removing the restriction x3 ≤ 3.
We now show that XP 0 is not bounded. Suppose, by contradiction, that there
is a positive real number µ such that
||x|| ≤ µ
for every x ∈ XP 0 . Consider the vector (1, 1, µ). It is immediate that (1, 1, µ) ∈
XP 0 . On the other hand,
1
||(1, 1, µ)|| = (1 + 1 + µ2 ) 2 > µ,
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
52 CHAPTER 2. OPTIMIZERS
which is a contradiction. Therefore, XP 0 is not bounded. Hence, Theorem 2.1
cannot be applied to P 0 .
The next proposition uses Theorem 2.1 for standard optimization problems.
Proposition 2.10
Let P be a standard optimization problem with XP 6= ∅. Then, SP 6= ∅
provided that XSC(P ) is a bounded set.
Proof:
Note that XSC(P ) 6= 0 since XSC(P ) = XP (see Proposition 1.2). Because, by
hypothesis, XSC(P ) is a bounded set, then, by Theorem 2.1, SSC(P ) 6= ∅. Thus,
SP 6= ∅, again by Proposition 1.2. QED
2.3 Solved Problems and Exercises
Problem 2.1
Consider a canonical optimization problem P = (A, b, c) such that XP is
bounded by
• line passing through points (0, 1) and (1, 2);
• line passing through points (1, 2) and (2, 1);
• line passing through points (2, 1) and (1, 0);
• x1 -axis and x2 -axis.
(1) Choose A and b in such a way that XP contains the points (1, 0) and (0, 1).
(2) Show that XP is bounded.
(3) Choose an interior point of XP and show that it is in int(XP ).
Solution:
(1) Recall that the equation of a line has the form
x2 = dx1 + e.
So, the equation of
• the line passing through points (0, 1) and (1, 2) is −x1 + x2 = 1;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.3. SOLVED PROBLEMS AND EXERCISES 53
• the line passing through points (1, 2) and (2, 1) is x1 + x2 = 3;
• the line passing through points (2, 1) and (1, 0) is x1 − x2 = 1.
Hence, the restrictions for the set of admissible vectors contains:
• −x1 + x2 ≤ 1 since the vector (1, 0) is admissible;
• x1 + x2 ≤ 3 since the vector (1, 0) is admissible;
• x1 − x2 ≤ 1 since the vector (0, 1) is admissible.
Thus, A is the matrix
−1 1
1 1
1 −1
and b is the vector:
1
b= 3
1
(2) We must show that there is a positive real number µ such that ||x|| ≤ µ for
every x ∈ XP . Take µ = 5 and let x ∈ XP . Observe that
• x2 ≤ 3 because x1 + x2 ≤ 3 and x1 ≥ 0;
• x1 ≤ 3 because x1 − x2 ≤ 1 and x2 ≥ 0.
Therefore,
1
||x|| ≤ (32 + 32 ) 2 ≤ 5.
(3) Consider the vector (1, 1). It is immediate that this vector is in XP◦ . Then,
(1, 1) is in int(X) (see Proposition 2.3). /
Exercise 2.2
Consider the following canonical optimization problem P
max x1 + 4x2
(x1 ,x2 )
x1 ≤ 2
2x1 + x2 ≤ 7
x , x ≥ 0.
1 2
(1) Show that the vector (1, 1) is in int(XP ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
54 CHAPTER 2. OPTIMIZERS
(2) Investigate whether or not XP is bounded.
(3) What can be concluded about the emptiness of SP ?
(4) Is there an optimizer of P in XP◦ ?
(5) Consider the problem P 0 obtained from P by removing the restriction 2x1 +
x2 ≤ 7. Is XP 0 bounded? What can be said about the existence of optimizers
of P 0 .
Exercise 2.3
Consider the following standard optimization problem P
min −2x1 − 3x2
x
2x1 + x2 + x3 = 6
x ≥ 0.
Does P have a minimizer?
2.4 Relevant Background
In this section, we provide an overview of the topological notions and results
over Rn relevant for the optimization material presented in this chapter. The
interested reader can also consult [4, 48]. We start with the concept of norm.
Definition 2.2
A norm on the vector space Rn is a map
N : Rn → R+
0
such that
• N (x) = 0 if and only if x = 0;
• N (αx) = |α|N (x), for any α ∈ R;
• N (x + y) ≤ N (x) + N (y).
The first requirement is called positive definiteness, the second positive ho-
mogeneity, and the last triangular inequality. The number
N (x)
is called the norm of vector x.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.4. RELEVANT BACKGROUND 55
Definition 2.3
The Euclidean norm (on Rn ) is the map k · k : Rn → R+
0 such that
v
u n 2
uX
kxk = t xj .
j=1
The Euclidean norm provides a distance from 0 to x, for every x ∈ Rn .
Exercise 2.4
Show that k · k : Rn → R+
0 when n = 1 coincides with the absolute value map
| · | : R → R+
0 .
Definition 2.4
An inner product on the vector space Rn is a map
I : (Rn )2 → R
satisfying the following properties:
• I(x, y) = I(y, x);
• I(αx, y) = αI(x, y), for any α ∈ R;
• I(x + y, z) = I(x, z) + I(y, z);
• I(x, x) ≥ 0;
• I(x, x) = 0 if and only if x = 0.
The first requirement is called symmetry, the second and the third are called
linearity conditions, and the fourth and the fifth are called positive definiteness.
The usual inner product over Rn is as follows:
Definition 2.5
The Euclidean inner product (on Rn ) is the map · : (Rn )2 → R such that
n
X
x·y = xj yj .
j=1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
56 CHAPTER 2. OPTIMIZERS
Remark 2.1
Observe that √
kxk = x · x.
Definition 2.6
The vectors v 1 , v 2 ∈ Rn are perpendicular or orthogonal , indicated by
v1 ⊥ v2 ,
whenever v 1 · v 2 = 0.
Example 2.5
The vectors (1, 1) and (−1, 1) are orthogonal.
Proposition 2.11 (Cauchy–Schwarz Inequality)
Given x, y ∈ Rn , then
|x · y| ≤ kxkkyk.
Proof:
Consider two cases:
(1) y = 0. Immediate, by inspection.
(2) y 6= 0. Then, y · y 6= 0 by positive definiteness. Let z be the vector
x·y
x− y.
y·y
Observe that
x·y x·y x·y x·y
z·y =x·y− y · y = 0 and x = x − y+ y=z+ y.
y·y y·y y·y y·y
Therefore,
x·y x·y
x·x = (z + y) · (z + y)
y·y y·y
x·y (x · y)2
= z·z+2 z·y+ y·y
y·y (y · y)2
(x · y)2
= z·z+
y·y
(x · y)2
≥
y·y
and so the thesis follows immediately. QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.4. RELEVANT BACKGROUND 57
The next results shows that the norm of a vector is an upper bound of the
absolute value of each component of the vector.
Proposition 2.12
Let x ∈ Rn . Then,
|xj | ≤ kxk
for every j = 1, . . . , n.
Proof:
Then,
|xj | = |x · ej |
≤ kxkkej k (by Proposition 2.11)
= kxk
using the standard basis of Rn (see Definition 1.20). QED
Definition 2.7
The open ball of radius ε > 0 centered on x ∈ Rn , denoted by
Bε (x),
is the set {y ∈ Rn : kx − yk < ε}. Moreover, the closed ball of radius ε centered
on x, denoted by
Bε [x],
is the set {y ∈ Rn : kx − yk ≤ ε}.
Example 2.6
Observe that
1 1 1
, , ∈ B1 (1, 1, 1)
2 2 2
since r
1 1 1 1 1 1 3
||(1, 1, 1) − , , || = || , , || = < 1.
2 2 2 2 2 2 4
On the other hand,
(0, 0, 0) ∈
/ B 12 (2, 2, 2).
Indeed:
√ 1
||(2, 2, 2) − (0, 0, 0)|| = ||(2, 2, 2)|| = 12 > .
2
Thus, (0, 0, 0) ∈
/ B 12 (2, 2, 2).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
58 CHAPTER 2. OPTIMIZERS
x2
x1
Figure 2.3: Boundary and interior of problem in Example 2.7
Definition 2.8
Let x ∈ Rn and U ⊆ Rn . We say that x is an interior point of U if there is
ε > 0 such that:
Bε (x) ⊂ U.
Furthermore, we say that x is a boundary point of U if for every ε > 0 we have
Bε (x) ∩ U 6= ∅ and Bε (x) ∩ (Rn \ U ) 6= ∅.
Example 2.7
Let X be the set
{(x1 , x2 ) ∈ R2 : x1 + x2 ≤ 2, x1 ≥ 0, x2 ≥ 0}
depicted in Figure 2.3. Then,
1 1
, is an interior point
2 2
and
(0, 1) is a boundary point.
Notation 2.1
The set of all interior points of U is called the interior of U and is denoted by
int(U ).
Moreover, the set of all boundary points of U is called the boundary of U and
is denoted by
bn(U ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.4. RELEVANT BACKGROUND 59
Definition 2.9
We say that a set U ⊆ Rn is open if U = int(U ). The complement of an open
set is called a closed set.
Remark 2.2
Observe that ∅ is an open and a closed set.
We now define convergence in Rn .
Definition 2.10
A sequence {xk }k∈N in Rn converges to x ∈ Rn if for every real number δ > 0
there is k ∈ N such that
kxm − xk < δ, for every m > k.
Example 2.8
The sequence
1 1
− ,
k k k∈N
converges to (0, 0).
Sometimes, the following characterization of convergence is also useful. For
more details see [4].
Proposition 2.13
Let {xk }k∈N be a sequence in Rn and x ∈ Rn . Then, {xk }k∈N converges to x if
and only if, for each j = 1, . . . , n, the sequence {(xk )j }k∈N converges to xj in
R.
That is, convergence in Rn amounts to pointwise convergence in R.
Proposition 2.14
Let {uk }k∈N and {wk }k∈N be sequences in R such that there is m ∈ N with
un ≤ wn for every natural number n > m. Assume that {uk }k∈N and {wk }k∈N
converge to u and w, respectively. Then, u ≤ w.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
60 CHAPTER 2. OPTIMIZERS
Proposition 2.15
Let {uk }k∈N be a constant sequence in R. Then, {uk }k∈N converges to u0 .
Taking account Proposition 2.13, counterparts of the previous results are
stated for Rn .
Proposition 2.16
Let {xk }k∈N and {y k }k∈N be sequences in Rn such that there is m ∈ N with
xn ≤ y n for every natural number n > m. Assume that {xk }k∈N and {y k }k∈N
converge to x and y, respectively. Then, x ≤ y.
Proposition 2.17
Let {xk }k∈N be a constant sequence in Rn . Then, {xk }k∈N converges to x0 .
The next result provides a useful characterization of closed sets.
Proposition 2.18
A set U ⊆ Rn is closed if and only if, for every sequence {xk }k∈N of elements
of U , if {xk }k∈N converges to x in Rn , then x ∈ U .
Observe that for showing that x ∈ U when U is closed, it is enough to find
a sequence in U that converges to x.
The following is an immediate consequence of Proposition 2.18.
Proposition 2.19
The class of closed sets is closed under finite union and intersection.
Proposition 2.20
A closed ball is a closed set.
Definition 2.11
We say that a set U ⊆ Rn is bounded if there exists µ ∈ R+ such that kuk ≤ µ
for every u ∈ U .
Definition 2.12
A set U ⊆ Rn is compact if it is closed and bounded.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
2.4. RELEVANT BACKGROUND 61
Example 2.9
For instance, the closed unit ball in Rn centered in 0 is compact; that is, the
set
{x ∈ Rn : x21 + · · · + x2n ≤ 1}
is compact.
Before proceeding we need some additional results on compact sets.
Proposition 2.21
Every sequence taking values on a compact set has a convergent subsequence
in that set.
The following result provides a sufficient condition for a set to be compact.
Proposition 2.22
Let U ⊆ Rn be a set such that every infinite sequence in U has a subsequence
converging to an element of U . Then, U is compact.
We now introduce the notion of continuous map which will be essential for
some results in linear optimization.
Definition 2.13
A map f : Rn → Rm is continuous at x in Rn if, for every sequence {xk }k∈N in
Rn that converges to x, {f (xk )}k∈N converges to f (x). A map is continuous if
it is continuous at every x ∈ Rn .
The following results provide useful alternative characterizations of contin-
uous maps.
Proposition 2.23
A map f : Rn → Rm is continuous at x if and only if, for every δ > 0, there
exists > 0 such that if ky − xk < then kf (y) − f (x)k < δ.
Proposition 2.24
A map f : Rn → Rm is continuous if and only if the map fi : Rn → R such
that fi (x) = (f (x))i is continuous, for every i = 1, . . . , m.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
62 CHAPTER 2. OPTIMIZERS
Proposition 2.25
The map f = u 7→ ku − vk : Rn → R+ n
0 is continuous, where v ∈ R .
The following result is also useful in the sequel.
Proposition 2.26
The inverse image of a closed set under a continuous map is closed.
The following result shows that compactness is preserved by continuous
maps.
Proposition 2.27
Let U ⊆ Rn be a compact set and f : Rn → R a continuous map. Then, f (U )
is compact.
The following result provides a sufficient condition for a map to have a
maximum. First we introduce the relevant notions.
Definition 2.14
A set U ⊆ R has an upper bound r whenever u ≤ r for every u ∈ U . A
map f : Rn → R has an upper bound or is bounded from above whenever
{f (x) : x ∈ Rn } has an upper bound. Furthermore, f has a maximum in
X ⊆ Rn when there is s ∈ X such that f (x) ≤ f (s) for every x ∈ X. Similarly
for lower bound , bounded from below and minimum.
Proposition 2.28
Let X ⊆ Rn be a non-empty compact set and f : Rn → R a continuous map.
Then, f has a maximum and has a minimum in X.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 3
Deciding Optimizers
The objective of this chapter is to provide a way for deciding whether or not a
given admissible vector is an optimizer, relying on the Farkas’ Lemma. Further-
more, we also use Farkas’ Lemma to decide whether or not there are admissible
vectors for the optimization problem at hand.
3.1 Farkas’ Lemma
The purpose of this section is to prove the important result known as the
Geometric Variant of the Farkas’ Lemma as well as other variants relevant for
optimization. Before that, we need to introduce the concepts of convex set,
convex cone and primitive cone as well as some results about them.
Definition 3.1
A subset U of Rn is convex whenever U is closed under convex combinations;
that is,
αx + (1 − α)y ∈ U
whenever x, y ∈ U and α ∈ [0, 1].
Example 3.1
The vector ( 34 , 1) is a convex combination of vectors (2, 3) and (1, 0). Indeed:
4 " # " #
1 2 2 1
3 = + .
1 3 3 3 0
63
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
64 CHAPTER 3. DECIDING OPTIMIZERS
Exercise 3.1
Show that ∅ is a convex set.
Definition 3.2
The convex cone generated by {u1 , . . . , uk } ⊂ Rn ,
C({u1 , . . . , uk }),
is the set ( k )
X
αi ui : αi ∈ R+
0 ,1 ≤ i ≤ k .
i=1
Exercise 3.2
Show that C({u1 , . . . , uk }) is a convex set.
Example 3.2
Let {e1 , . . . , en } be the canonical basis of Rn (recall Definition 1.20). Then,
C({e1 , . . . , en }) = {x ∈ Rn : x ≥ 0}
as we now show.
(⊆) Let x ∈ C({e1 , . . . , en }). Then,
n
X
x= αj ej
j=1
for some α ∈ (R+ n
0 ) . Hence, xj is αj for each j = 1, . . . , n. Thus, x ≥ 0.
(⊇) Assume that x ≥ 0. Observe that
n
X
x= xj e j .
j=1
Moreover, xj ≥ 0 for every j = 1, . . . , n. Hence, x ∈ C({e1 , . . . , en }).
Definition 3.3
A subset C of Rn is a convex cone if there exists a finite subset U of Rn such
that C = C(U ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.1. FARKAS’ LEMMA 65
Example 3.3
Recall Example 3.2. Then, the set
{x ∈ Rn : x ≥ 0}
is a convex cone.
Definition 3.4
A convex cone C is primitive if there exists a set U of linearly independent
vectors such that C = C(U ).
Example 3.4
Recall Example 3.3. Then, the convex cone
{x ∈ Rn : x ≥ 0}
is primitive.
Example 3.5
Observe that ∅ is a primitive convex cone since
∅ = C(∅)
and ∅ is a linearly independent set (see Example 1.26).
We now prove several results that are needed for Farkas’ Lemma.
Proposition 3.1
Every primitive convex cone is closed.
Proof:
Let C ⊆ Rn be a primitive convex cone. Since C is a primitive cone, then
C = C(U ) for some set U = {u1 , . . . , uk } of linearly independent vectors in Rn .
Let h : Rk → Rn be the map
h(x) = x1 u1 + · · · + xk uk .
Let {e1 , . . . , ek } be the canonical basis of Rk (recall Definition 1.20). We start
by proving that
C = h(C({e1 , . . . , ek })).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
66 CHAPTER 3. DECIDING OPTIMIZERS
(⊆) Let x ∈ C. Then,
x = α1 u1 + · · · + αk uk , for some α1 , . . . , αk ∈ R+
0.
Observe that α ∈ C({e1 , . . . , ek }), by Example 3.2, where α is the vector in Rk
with components α1 , . . . , αk . Hence,
x = h(α)
and so x ∈ h(C({e1 , . . . , ek })).
(⊇) Assume that x ∈ h(C({e1 , . . . , ek })). Then, by Example 3.2,
x = h(α) for some α ≥ 0, α ∈ Rk .
Therefore, x ∈ C.
We now show that
h is an injective map.
Indeed, assume that h(x) = h(y). Then,
(x1 − y1 )u1 + · · · + (xk − yk )uk = 0.
Hence, (xj − yj ) = 0 for j = 1, . . . , k because U is a linearly independent set.
Let g : h(Rk ) → Rk be such that
g(y) is the unique x such that h(x) = y.
This map is well defined since y = h(x) for some x ∈ Rk and there is only
one such x, for each y ∈ h(Rk ). Observe that g is a bijection. Moreover,
g −1 : Rk → h(Rk ) is such that
g −1 (x) = h(x)
for every x ∈ Rk . Therefore,
C = g −1 (C({e1 , . . . , ek })).
It is immediate that g is a linear map. Thus, by Proposition 2.1, g is con-
tinuous. Recall that {x ∈ Rk : x ≥ 0} is a closed set by Proposition 2.6.
Thus, C({e1 , . . . , ek }) is closed by Proposition 3.2. Then, C is a closed set (see
Proposition 2.26). QED
The next result states that the set of primitive convex cones generates the
class of convex cones.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.1. FARKAS’ LEMMA 67
Proposition 3.2
Every convex cone is a finite union of of primitive convex cones.
Proof:
Let C be a convex cone. Then,
C = C(U )
for some finite set U = {u1 , . . . , uk } ⊂ Rn . We will show that each element x of
C(U ) belongs to a primitive convex cone generated by a subset of U . The thesis
follows from the fact that the number of primitive convex cones generated by
subsets of U is finite (indeed there are 2k subsets of U but not all of them may
be linearly independent sets).
Take x ∈ C(U ). Consider two cases:
(1) x = 0. Then, x ∈ C({u1 }). So, x belongs to a primitive convex cone
generated by a subset of U .
(2) x 6= 0. Let Ux = {ui1 , . . . , ui` } ⊆ U be such that x ∈ C(Ux ) and
|Ux | = min{|V | : V ⊆ U, x ∈ C(V )}.
V
Then,
x = α1 ui1 + · · · + α` ui` ,
where αj > 0 for each j = 1, . . . , `. Moreover, Ux 6= ∅. We now prove that
Ux is a linearly independent set.
Assume, by contradiction, that Ux is not linearly independent. Hence, there
exists β ∈ R` such that β1 ui1 + · · · + β` ui` = 0 and βi 6= 0 for some i = 1, . . . , `.
Pick j such that
αj
= min αi : βi 6= 0 ∧ 1 ≤ i ≤ ` .
βj βi
Take
αj
γ =α− β.
βj
Observe that there exists i = 1, . . . , k such that γi = 0. Indeed:
αj
γj = αj − βj = 0.
βj
We now show that
αj
γi = αi − βi ≥ 0, for every i 6= j.
βj
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
68 CHAPTER 3. DECIDING OPTIMIZERS
Consider the following cases:
(a) βi = 0. Then,
αj
αi − βi = αi ≥ 0.
βj
(b) βj , βi ∈ R− . Then,
αj αi
− ≤− .
βj βi
Hence,
αj αi
− βi ≥ − βi ,
βj βi
and so
αj αi
γi = αi − βi ≥ αi − βi = 0.
βj βi
(c) Either βj ∈ R− and βi ∈ R+ or βj ∈ R+ and βi ∈ R− . Then,
αj
αi − βi ≥ 0.
βj
(d) βj , βi ∈ R+ . Then,
αj αi
βi ≤ βi
βj βi
and so
αj αi
γi = αi − βi ≥ αi − βi = 0.
βj βi
Thus, γ ≥ 0.
Therefore,
γ1 ui1 + · · · + γ` ui` =
αj αj
(α1 − β1 )ui1 + · · · + (α` − β` )ui` =
βj βj
αj
(α1 ui1 + · · · + α` ui` ) − (β1 ui1 + · · · + β` ui` ) = x,
βj
thus contradicting the choice of Ux , because V = {uim ∈ Ux : γm 6= 0} ( Ux
and x ∈ C(V ). QED
Proposition 3.3
Every convex cone is closed.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.1. FARKAS’ LEMMA 69
Proof:
Let C be a convex cone. Then, by Proposition 3.2, C is a finite union of
primitive cones. Since, by Proposition 3.1, each primitive cone is a closed set
and any finite union of closed sets is also a closed set (see Proposition 2.19),
we conclude that C is a closed set. QED
The following result states an important fact concerning closed sets.
Proposition 3.4
Let U ⊆ Rn be a non-empty closed set and v ∈ Rn . Then, there exists at least
one element of U whose distance to v is minimal.
Proof:
Let x ∈ U and
B = {u ∈ U : ku − vk ≤ kx − vk}.
Observe that x ∈ B. We start by proving that B is a closed set. Observe that
B = Bkx−vk [v] ∩ U,
where Bkx−vk [v] = {z ∈ Rn : kz − vk ≤ kx − vk}. Since U is closed by hy-
pothesis and Bkx−vk [v] is closed (see Proposition 2.20), then B is also closed
because it is the intersection of closed sets (by Proposition 2.19). Moreover, B
is a bounded set. Indeed, let u ∈ B. Then,
kuk ≤ ku − vk + kvk triangular inequality
≤ kx − vk + kvk definition of B
Therefore, B is a compact set (recall Definition 2.12). Consider the map
f = u 7→ ku − vk : Rn → R+
0.
By Proposition 2.28, since f is continuous (see Proposition 2.25), it has a
minimum xv on the compact set B. Hence, xv is the vector of B closest to v;
that is,
kxv − vk ≤ ku − vk
for every u ∈ B. In particular, kxv − vk ≤ kx − vk since x ∈ B. Finally, let
y ∈ U \ B. Then,
ky − vk > kx − vk ≥ kxv − vk.
Thus, kxv − vk ≤ ku − vk for every u ∈ U . QED
The next result is a direct consequence of the proposition above for convex
cones.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
70 CHAPTER 3. DECIDING OPTIMIZERS
Proposition 3.5
Let U ⊆ Rn be a non-empty convex cone and v ∈ Rn . Then, there exists at
least one element of U whose distance to v is minimal.
Proof:
Let U be a convex cone. Then, by Proposition 3.3, U is a closed set. Hence,
by Proposition 3.4, we conclude the thesis. QED
The next result called the Geometric Variant of the Farkas’ Lemma, is
particularly relevant for optimization. It was stated and proved by Julius Farkas
(see [30]). Its generalization to topological vector spaces is known as the Hahn–
Banach Theorem (see [57]).
Proposition 3.6
Let U = {u1 , . . . , uk } ⊂ Rn and v ∈ Rn . Then, exactly one of the following
alternatives holds:
• v ∈ C(U );
• there exists a non-null row vector w ∈ Rn such that wui ≥ 0 for i =
1, . . . , k and wv < 0.
Proof:
Let z be an element of C(U ) such that the distance from z to v is minimal
(such a z exists by Proposition 3.5) and
w = (z − v)T .
We have two cases.
(1) w is the zero vector. Then, v ∈ C(U ).
(2) w is not the zero vector. We start by showing that
wz = 0.
Suppose, by contradiction, that wz 6= 0. Then, z 6= 0. Consider the family
{y α }α∈]0,1]
when wz > 0, otherwise, consider the family
{y α }α∈R−
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.1. FARKAS’ LEMMA 71
when wz < 0, where
y α = (1 − α)z.
Then,
y α ∈ C(U ).
On the other hand,
yα − v = (1 − α)z − v
= (1 − α)z − (z − wT )
= wT − αz.
Therefore,
2
ky α − vk = (wT − αz)T (wT − αz)
= (w − αz T )(wT − αz)
= wwT − αz T wT − αwz + α2 z T z
2 2
= kwk − 2αwz + α2 kzk .
Observe that αwz > 0. Consider the set composed by the α’s such that
2
2αwz > α2 kzk ;
that is,
2wz
α< 2.
kzk
Hence, if wz > 0 pick up
!
2wz
0 < α < min 1, 2 ,
kzk
otherwise
2wz
α< 2.
kzk
Thus, in both cases,
2 2 2
ky α − vk < kwk = kz − vk
contradicting the choice of z as the vector at the minimal distance from v.
Hence, wz = 0.
Then,
(a) wv < 0. Indeed
0 < wwT = w(z − v) = wz − wv = −wv.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
72 CHAPTER 3. DECIDING OPTIMIZERS
(b) wx ≥ 0 for every x ∈ C(U ):
If x = z, then as proved before wx = 0. Hence, it remains to prove the thesis
when x 6= z. Note that
−wx = −wx + wz
= −w(x − z)
= (v − z)T (x − z).
Therefore, we must show that (v − z)T (x − z) ≤ 0. Assume, by contradiction,
that (v − z)T (x − z) > 0. Consider the family
{y α }α∈R+ , where y α = αx + (1 − α)z.
Observe that y α ∈ C(U ) provided that α ≤ 1. On the other hand,
y α − v = αx + wT − αz = wT + α(x − z).
Therefore,
2
ky α − vk = (y α − v)T (y α − v)
= (wT + α(x − z))T (wT + α(x − z))
2 2
= kwk + 2αw(x − z) + α2 kx − zk
2 2
= kwk + 2α(z − v)T (x − z) + α2 kx − zk .
Since (z − v)T (x − z) < 0, then
2 2 2
ky α − vk < kwk = kz − vk
provided that
2
−2α(z − v)T (x − z) > α2 kx − zk .
Choose α such that
!
−2(z − v)T (x − z)
0 < α < min 1, 2 .
kx − zk
Then, y α contradicts the choice of z as minimizing distance to v.
It remains to prove that the two alternatives of the statement of the proposition
do not hold at the same time. Assume, by contradiction that both alternatives
hold. Then,
v = α1 u1 + · · · + αk uk
for some α1 , . . . , αk ∈ R+ j
0 . On the other hand, wu ≥ 0 for every j = 1, . . . , k.
Hence,
wv = α1 wu1 + · · · + αk wuk ≥ 0
contradicting that wv < 0. QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.1. FARKAS’ LEMMA 73
In the sequel, we need the following notations.
Notation 3.1
Given a matrix A, we denote by
a•j
the j-th column of A and by
ai•
the i-th row of A.
Notation 3.2
Given an m × n-matrix A, we denote by
C(A)
the convex cone in Rn generated by the set {a1• , . . . , am• } of lines of A.
We provide a version of Farkas’ Lemma for Matrices.
Proposition 3.7
Let A be an m × n-matrix and v ∈ Rn . Then, exactly one of the following
alternatives holds:
• v ∈ C(A);
• there is a non-null row vector w ∈ Rn with wAT ≥ 0T and wv < 0.
Proof:
Using Proposition 3.6 with U = {a1• , . . . , am• }, exactly one of the following
alternatives holds:
• v ∈ C(U );
• there exists a non-null row vector w ∈ Rn such that waT
i• ≥ 0 for i =
1, . . . , m and wv < 0.
The thesis follows since C(A) = C(U ) and waT
i• ≥ 0 for i = 1, . . . , m is equiva-
lent to saying that wAT ≥ 0T . QED
Exercise 3.3
Let A be an m × n-matrix and v ∈ Rm . Show that exactly one of the following
alternatives holds:
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
74 CHAPTER 3. DECIDING OPTIMIZERS
• v ∈ C(AT );
• there exists a non-null row vector w ∈ Rm such that wA ≥ 0T and wv < 0.
3.2 Using Convex Cones
The objective of this section is to show that the Farkas’ Lemma provides a
test for deciding whether or not an admissible vector is a local maximum for
the objective in a canonical optimization problem. Recall the pure form of a
canonical optimization problem introduced in Notation 1.8.
Definition 3.5
Let P be a canonical problem in pure form and x ∈ XP . The i-th line of A is
active in x if
ai• x = bi .
Notation 3.3
We denote by
Ax
the matrix containing the lines of A that are active in x, where x ∈ XP .
Example 3.6
Consider the canonical optimization problem P in Example 1.12 presented in
pure form. The 1st and the 2nd lines of A are active in (3, 3) ∈ XP . Hence,
(3,3) 3 −1
A =
−1 3
On the other hand, the 3rd and the 4th lines of A are active in (0, 0) ∈ XP .
Hence,
−1 0
A(0,0) = .
0 −1
With the example above, the reader can start to infer that for any vector in
the boundary of the set of admissible vectors there exists a line active in that
vector. This is the case as we show now.
Proposition 3.8
Consider a canonical optimization problem P in pure form and x ∈ XP . Then,
x ∈ ∂XP if and only if there exists a line of A active in x.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.2. USING CONVEX CONES 75
Proof:
We just prove one of the implications. The other implication is proved in a
similar way.
(→) Assume that x ∈ ∂XP . There are two cases.
(1) There is i ∈ {1, . . . .m} such that ai• x = bi . Then, the i-th line of A is
active in x.
(2) There is ` ∈ {m + 1, . . . , m + n} such that x`−m = 0. Thus, the `-th line of
A is active in x. QED
The aim is to provide a test for checking whether or not an admissible vector
is a maximizer. For that, we introduce the concept of local maximum for maps
from Rn into R as follows.
Definition 3.6
A vector x is a local maximum of f : Rn → R in D ⊆ Rn if there exists ε > 0
such that, for every y ∈ D,
if kx − yk ≤ ε then f (x) ≥ f (y).
The following result, called the Local Maximizer Theorem, provides a way
to decide whether or not a boundary vector is a local maximum, using convex
cones.
Theorem 3.1
Let P be a canonical optimization problem in pure form. Then, for every
x ∈ ∂XP ,
x is a local maximum of x 7→ cx in XP if and only if cT ∈ C(Ax ).
Proof:
If c = 0 then the thesis follows immediately. Otherwise, we show the equivalent
statement that exactly one of the following alternatives holds:
• cT ∈ C(Ax ) and x is a local maximum of the map x 7→ cx in XP ;
• cT 6∈ C(Ax ) and x is not a local maximum of the map x 7→ cx in XP .
By Proposition 3.8, there is a line of A active in x. By applying Proposition 3.7
to matrix Ax and vector cT , it follows that exactly one of the following alter-
natives holds:
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
76 CHAPTER 3. DECIDING OPTIMIZERS
(1) cT ∈ C(Ax );
(2) there exists a non-null row vector w ∈ Rn such that w(Ax )T ≥ 0T and
wcT < 0.
In case (1), the thesis is established by contradiction as follows. Assume that
x is not a local maximum of the map x 7→ cx in XP . That is, for every ε > 0
there exists y ∈ XP such that kx − yk ≤ ε and cx < cy. Pick ε > 0 and y in
these conditions.
Since cT ∈ C(Ax ), there exists α ≥ 0 such that (Ax )T α = cT ; that is, αT Ax = c.
Hence, from cx < cy it follows that
αT Ax x < αT Ax y.
Thus, there exists i such that αi > 0 (otherwise 0 < 0) and
(Ax x)i < (Ax y)i
as we now show. Assume, by contradiction, that for every j such that αj > 0
we have (Ax x)j ≥ (Ax y)j . Then,
X X
αT Ax x = αj (Ax x)j ≥ αj (Ax y)j = αT Ax y,
j∈{j:αj >0} j∈{j:αj >0}
contradicting αT Ax x < αT Ax y. Then, by definition of Ax , for such an i,
bi = ai• x < ai• y.
Therefore, it is not the case that Ay ≤ b. Hence, y 6∈ XP , contradicting the
choice of y.
Assume now that cT ∈ / C(Ax ). Then, (2) holds. Take ε > 0. We show that it
is possible to find y ∈ Rn such that:
(a) kx − yk ≤ ε;
(b) cx < cy;
(c) y ∈ XP .
Consider the family
{y β }β∈R+ , where y β = x − βwT .
Observe that (a) is guaranteed by considering the subfamily
ε
{y β }β∈B , where B = β ∈ R+ : β ≤ .
kwT k
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.2. USING CONVEX CONES 77
Since wcT < 0 and β > 0, from
cy β = cx − βcwT
= cx − β(wcT )T
> cx
we conclude (b). Finally, to prove (c), that is, Ay β ≤ b, it suffices to establish,
for each i-th line of A, that
ai• y β ≤ bi .
Consider two cases:
(i) The i-th line of A is active in x:
In this case, ai• x = bi . Then,
ai• y β = bi − βai• wT .
Since w(Ax )T ≥ 0T , in particular, ai• wiT ≥ 0. Hence, because β is positive,
ai• y β ≤ bi .
(ii) The i-th line of A is not active in x:
In this case, ai• x < bi . Then, we want to ensure that
ai• y β = ai• x − βai• wT ≤ bi .
If ai• wT ≥ 0 then ai• y β ≤ bi . Otherwise ai• wT < 0. In this case, consider the
subfamily
β 0 + bi − ai• x
{y }β∈Bi0 , where Bi = β ∈ R : β ≤ .
|ai• wT |
Then, it is immediate that
ai• y β ≤ bi .
By choosing y in the family
{y β }β∈B∩B 0 ,
where \
B0 = Bi0
i∈I
and I is the set
{i ∈ {1, . . . , m + n} : i-th is not active in x and ai• wT < 0}
we conclude that x is not a local maximum of x 7→ cx in XP . QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
78 CHAPTER 3. DECIDING OPTIMIZERS
Example 3.7
Recall Example 3.6. Then, (3, 3) is a local maximum of x 7→ 2x1 + x2 in XP
since (2, 1)T ∈ C(A(3,3) ), by Theorem 3.1. Indeed,
2 3 −1
= α1 + α2
1 −1 3
7 5
has a positive solution with α1 = and α2 = . On the other hand, (0, 0)
8 8
/ C(A(0,0) ), by
is not a local maximum of x 7→ 2x1 + x2 in XP since (2, 1)T ∈
Theorem 3.1. Indeed,
2 −1 0
= α1 + α2
1 0 −1
does not have a positive solution.
The following result is known as the Maximizer Theorem.
Theorem 3.2
Every local maximum of the objective map of a canonical optimization problem
in the set of admissible vectors is a maximizer and vice versa.
Proof:
Let P be a canonical optimization problem. It is immediate that a maximizer is
also a local maximum. The proof of the other implication is by contraposition.
Assume that x ∈ XP is not a maximum of x 7→ cx in XP . Then, there is
z ∈ XP such that cx < cz. We must show that x is not a local maximum
of x 7→ cx in XP . That is, we must show that, for every ε > 0, there exists
y ∈ XP such that kx − yk ≤ ε and cx < cy. Let ε > 0. Consider the family
{y β }β∈B , where y β = (1 − β)x + βz
with
ε
B = β ∈ R+ : β ≤ .
kx − zk
Each y β satisfies kx − y β k ≤ ε. Pick β ∈ B such that β < 1. Then, y β is a
convex combination of elements of XP . Moreover, y β ∈ XP as we show now.
Indeed, it is immediate that y β ≥ 0. Furthermore,
Ay β = (1 − β)Ax + βAz ≤ (1 − β)b + βb = b.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.2. USING CONVEX CONES 79
Finally,
cy β = (1 − β)cx + βcz > (1 − β)cx + βcx = cx
because cz > cx and β is positive. QED
Example 3.8
Recall Example 3.7. Then, (3, 3) is a maximizer of P and (0, 0) is not a maxi-
mizer of P , taking into account Theorem 3.2.
We now provide a characterization, based on convex cones, for the set of
admissible vectors of a standard optimization problem to be non-empty.
Proposition 3.9
Let P be a standard optimization problem. Then,
XP 6= ∅ if and only if b ∈ C(AT ).
Proof:
(→) Let A be an m × n-matrix and x ∈ XP . Then,
bi = ai1 x1 + . . . ain xn
for every i = 1, . . . , m and xj ≥ 0 for every j = 1, . . . , n. We want to find
α ∈ (R+ n
0 ) such that
b = α1 a•1 + · · · + αn a•n .
It is enough to take αj as xj for j = 1, . . . , n.
(←) This implication follows in a similar way. QED
The following results, for deciding the non-emptiness of the set of admissible
vectors, are called the Standard and the Canonical Variants of the Farkas’
Lemma, respectively.
Proposition 3.10
Let P be a standard optimization problem. Then,
XP 6= ∅ if and only if for every w 6= 0, wb ≥ 0 when wA ≥ 0.
Proof:
Observe that, by Proposition 3.9,
XP 6= ∅ iff b ∈ C(AT ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
80 CHAPTER 3. DECIDING OPTIMIZERS
By Exercise 3.3 and since the two possibilities are exclusive, we conclude that
b ∈ C(AT ) iff for every w 6= 0, wb ≥ 0 when wA ≥ 0.
Thus, the thesis follows. QED
Proposition 3.11
Let P be a canonical optimization problem. Then,
XP 6= ∅ if and only if for every w 6= 0, wb ≥ 0 when wA ≥ 0 and w ≥ 0.
Proof:
Using Proposition 1.3, we have that
XP 6= ∅ iff XCS(P ) 6= ∅.
On the other hand, Proposition 3.10 states that
XCS(P ) 6= ∅ iff for every w 6= 0, wb ≥ 0 when w A I ≥ 0.
Observe that
w A I ≥ 0 iff wA ≥ 0 and w ≥ 0
and so the thesis follows. QED
Exercise 3.4
Prove the standard variant from the canonical variant for deciding the non-
emptiness of the set of admissible vectors using the Farkas’ Lemma.
3.3 Solved Problems and Exercises
Problem 3.1
Let U ⊆ Rn be a finite set and V the class of all V ⊆ Rn closed under sum and
multiplication by non-negative scalars. Show that
\
C(U ) = V.
U ⊆V,V ∈V
Solution:
Let U = {u1 , . . . , uk }.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.3. SOLVED PROBLEMS AND EXERCISES 81
(⊇) It is enough to show that C(U ) ∈ V:
(1) C(U ) is closed under sum of vectors. Let x, y ∈ C(U ). Then,
k
X k
X
x= αi ui and y = βi ui
i=1 i=1
for some α, β ∈ (R+ n
0 ) . Therefore,
k
X
x+y = (αi + βi )ui ∈ C(U ),
i=1
since (αi + βi ) ∈ R+
0 for i = 1, . . . , k.
(2) C(U ) is closed under multiplication by non-negative scalars. Let x ∈ C(U ).
Then,
k
X
x= αi ui
i=1
for some α ∈ (R+ n
0) . Let β ∈ R+
0. Hence,
k
X
βx = (βαi )ui ∈ C(U )
i=1
since (βαi ) ∈ R+
0 for i = 1, . . . , k.
(⊆) We show that C(U ) ⊆ V for any V ∈ V such that U ⊆ V .
We start by observing that
k
[ [
C(U ) = C(U 0 ),
`=1 U 0 ⊆ U
|U 0 | = `
We prove that
C(U 0 ) ⊆ V
for every V ∈ V and U 0 such that U 0 ⊆ U ⊆ V and |U 0 | = `, by induction on `.
(Base) ` = 1. Let U 0 = {u0 } be such that U 0 ⊆ U . Then, C(U 0 ) has the form
{αu0 : α ∈ R+
0}
and so is contained in V since U ⊆ V and V is closed under multiplication by
non-negative scalars.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
82 CHAPTER 3. DECIDING OPTIMIZERS
(Step) Assume that C(U 00 ) ⊆ V for every U 00 such that U 00 ⊆ U and |U 00 | ≤ `.
Let
U 0 = {u0 1 , . . . , u0 `+1 }
be such that U 0 ⊆ U . Take x ∈ C(U 0 ). Then, there is α ∈ (R+ 0)
`+1
such that
`+1 `
!
X X
x= α i u0 i = αi u0 i + α`+1 u0 `+1 .
i=1 i=1
Then,
`
X
• αi u0 i ∈ V , using the induction hypothesis;
i=1
• α`+1 u0 `+1 ∈ V , since u0 `+1 ∈ V and V is closed under multiplication by
non-negative scalars.
Hence, x ∈ V , since V is closed under sum of vectors. /
Problem 3.2
Consider the following canonical optimization problem P :
max
x
2x1 + 3x2
x1 + x2 ≤ 3
−x1 + x2 ≤ −1
x ≥ 0.
(1) Present P in pure form.
(2) Given v ∈ R2 , discuss the different possibilities for Av .
(3) Find the lines of A active in (2, 0) and the lines of A active in (2, 1) and
conclude whether or not these vectors are maximizers.
Solution:
(1) The pure form of P is:
h i
max 2 3 x
x
1 1 3
−1 1 −1
≤
x .
−1
0 0
0 −1 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.3. SOLVED PROBLEMS AND EXERCISES 83
(2) Let x = (x1 , x2 ) ∈ XP . There are the following possibilities for Ax :
• Ax is the empty matrix. Then, there are no active lines for x and so, by
Proposition 3.8, x ∈ XP◦ ;
• Ax is the matrix 0 −1 . Then, by Proposition 3.8, x ∈ ∂XP and is
in the x1 -axis;
• Ax is the matrix 1 1 . Then, by Proposition 3.8, x ∈ ∂XP . More-
over, x is in the segment defined by (2, 1) and (3, 0);
• Ax is the matrix −1 1 . Then, by Proposition 3.8, x ∈ ∂XP . More-
over, x is in the segment defined by (1, 0) and (2, 1);
1 1
• Ax is the matrix . Then, by Proposition 3.8, x ∈ ∂XP and x
−1 1
is (2, 1);
x 1 1
• A is the matrix . Then, by Proposition 3.8, x ∈ ∂XP and x
0 −1
is (3, 0);
−1 1
• Ax is the matrix . Then, by Proposition 3.8, x ∈ ∂XP and
0 −1
x is (1, 0).
Observe that
−1 0
cannot be a line of Ax since XP does not intersect the x2 -axis.
(3) The 4-th line is the unique active in (2, 0). On the other hand, the 1st and
the 2nd lines are active in (2, 1). Therefore,
A(2,0) = 0 −1
and
(2,1) 1 1
A = .
−1 1
Using Theorem 3.2, to prove whether or not x ∈ XP is a maximizer, we must
show that x is a local maximum for the objective map in XP . Moreover, by
Theorem 3.1 it is enough to verify whether or not cT ∈ C(Ax ).
• (2, 0): in this case, there is no α ∈ R+
0 such that
2 0
=α .
3 −1
Hence, (2, 0) is not a maximizer.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
84 CHAPTER 3. DECIDING OPTIMIZERS
• (2, 1): in this case, we show that there are α1 and α2 in R+
0 such that
2 1 −1
= α1 + α2 .
3 1 1
Indeed, take
α1 = 5
2
α2 = 1
2.
Thus, (2, 1) is a maximizer. /
Problem 3.3
Let P be the following canonical optimization problem
max 2x1 + 5x2
x
− 4 x1 − x2 ≤ −2
3
2x1 + x2 ≤ 10
x ≥ 0.
Show that XP is non-empty using convex cones.
Solution:
Observe that CS(P ) is as follows:
min −2x1 − 5x2
x
− 4 x1 − x2 + x3 = −2
3
2x1 + x2 + x4 = 10
x ≥ 0.
Hence, ACS(P ) and bCS(P ) are:
" # " #
− 34 −1 1 0 −2
and ,
2 1 0 1 10
respectively. Using Proposition 3.9, for proving that XCS(P ) 6= ∅ it is enough
to show that
bCS(P ) ∈ C(ATCS(P ) ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
3.3. SOLVED PROBLEMS AND EXERCISES 85
Therefore, we must find α1 , α2 , α3 , α4 ∈ R+
0 such that
4 "
−1
# "
1
# "
0
# " #
− −2
α1 3 + α2 + α3 + α4 = .
2 1 0 1 10
That is, we must find a non-negative solution for the following system of equa-
tions:
− 4 α1 − α2 + α3 = −2
3
2α1 + α2 + α4 = 10.
For instance, take
α1 = 0, α2 = 2, α3 = 0 and α4 = 8.
Thus, by Proposition 3.9, XCS(P ) 6= ∅. Then,
XP 6= ∅
by Proposition 1.3. /
Exercise 3.5
Let A be an m × n-matrix and x ∈ Rn . Show that exactly one of the following
alternatives holds:
• x ∈ C(A);
• there exists a non-null row vector e ∈ Rn such that eAT ≥ 0T and ex < 0.
Exercise 3.6
Consider the following canonical optimization problem P :
max 2x1 + x2
x
−x1 − x2 ≤ −1
x1 + x2 ≤ 3
x1 − x2 ≤ 1
−x1 + x2 ≤ 1
−x1 ≤ 0
−x2 ≤ 0.
3 1
Decide whether or not (2, 1) and , are maximizers of P .
2 2
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
86 CHAPTER 3. DECIDING OPTIMIZERS
Exercise 3.7
Consider the following canonical optimization problem:
max 2x1 + x2
x
−x + x ≤ 1
1
2
x1 + x2 ≤ 3
x1 − x2 ≤ 1
x ≥ 0.
(1) Verify that (2, 1) is a local maximum and that (1, 2) is not a local maximum.
(2) Find an objective map for which (1, 2) is a local maximum.
(3) Is there an objective map such that all points in an edge are local maxima?
Justify your answer.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 4
Computing Optimizers
The main objective of this chapter is to obtain a constructive counterpart of
Theorem 2.1 by providing the means to calculate optimizers, using techniques
from linear algebra. The reader can find, in Section 4.5, a brief survey of the
relevant background including concepts and results. Throughout this chapter,
we focus on standard optimization problems.
4.1 Basic Vectors
The main objective of this section is to introduce the important subset of the
set of admissible vectors composed by the so called basic vectors. For that, it is
useful to consider restricted standard optimization problems. We will show that
for every optimization problem there is always an equivalent restricted standard
problem. Recall that each line of matrix A is non-null (see Remark 1.2).
Definition 4.1
A standard optimization problem
min
x
cx
Ax = b
x ≥ 0,
where A is an m × n-matrix, is restricted whenever
1. m < n;
2. rank(A) = m.
87
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
88 CHAPTER 4. COMPUTING OPTIMIZERS
We start by showing that the standard optimization problem resulting from
applying the map CS (see Proposition 1.3) to a canonical optimization problem
is restricted.
Proposition 4.1
Let P be a canonical optimization problem. Then,
CS(P )
is a restricted standard optimization problem.
Proof:
Assume that P is
max cx
x
Ax ≤ b
x ≥ 0.
Then, CS(P ) is h i
min −c 0 y
y
h i h i
A I y= b
y ≥ 0,
where I is the m × m identity matrix. Note that the dimension of the matrix
of CS(P ) is m × (n+m). Therefore, m < n + m. Moreover, the rank of the
matrix is m because it includes the identity matrix of dimension m. QED
The following result shows that every standard optimization problem has
an equivalent restricted counterpart (recall the definition of map SC in Propo-
sition 1.2).
Proposition 4.2
Let P be a standard optimization problem. Then,
CS(SC(P ))
is a restricted standard optimization problem.
Proof:
It is enough to use Proposition 4.1 over SC(P ). QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.1. BASIC VECTORS 89
Remark 4.1
From now on, we assume, without loss of generality, that any standard opti-
mization problem is restricted, except when otherwise stated.
Notation 4.1
Given an m × n-matrix, we denote by
M = {1, . . . , m} and N = {1, . . . , n}
the sets of row and column indices, respectively.
Notation 4.2
Given a matrix A and B ⊆ N , we denote by
AB
the submatrix (see Definition 1.23) of A where the columns are the ones with
indices in B.
Definition 4.2
Let A be an m × n matrix of a standard optimization problem. An (index)
basis for A is a subset B of N with cardinality m such that the set of columns
in AB is linearly independent (that is, AB is nonsingular).
Exercise 4.1
Show that
n n!
=
m m!(n − m)!
is an upper bound on the number of basis.
Example 4.1
Consider the following standard optimization problem
h i
min
x −1 −4 0 0 x
" # " #
−1 1 1 0 1
x=
2 1 0 1 6
x ≥ 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
90 CHAPTER 4. COMPUTING OPTIMIZERS
Observe that N = {1, 2, 3, 4}. The subsets of N that are candidates to basis
are the following:
{1, 2}, {1, 3}, {1, 4}, {2, 3}, {2, 4}, {3, 4}
corresponding to the matrices
" # " # " #
−1 1 −1 1 −1 0
A{1,2} = , A{1,3} = , A{1,4} = ,
2 1 2 0 2 1
" # " # " #
1 1 1 0 1 0
A{2,3} = , A{2,4} = , A{3,4} = .
1 0 1 1 0 1
It is immediate that the determinant of each matrix is non-null. Hence, each
candidate is a basis.
This terminology is justified by the following result:
Proposition 4.3
Consider a standard optimization problem. Let B ⊆ N be an index basis for
A. Then, the columns in AB constitute a basis for span(A).
Proof:
Since B is an index basis for A, then the set composed by the columns of AB
is linearly independent. Moreover, rank(A) = rank(AB ). Observe that, by
Proposition 4.24, the dimension of span(A) is rank(A). So, the set composed
by the columns of AB is a basis for span(A), by Proposition 4.12. QED
Among the admissible vectors of a standard optimization problem, some
are of particular significance.
Definition 4.3
Consider a standard optimization problem. An admissible vector x is basic
whenever there exists B ⊆ N such that:
• B is an (index) basis;
• xj = 0 for every j ∈ (N \ B).
In this case, B is admissible. Moreover, x is degenerate when the number of
zero components of x is strictly greater than |N \ B|.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.1. BASIC VECTORS 91
Example 4.2
Recall the standard optimization problem P introduced in Example 4.1. We
now investigate which bases are admissible.
(1) B = {1, 2}. Let x ∈ R4 be such that x3 = 0 and x4 = 0. Observe that
x1
" # " #
−1 1 1 0 x2
1
=
2 1 0 1 0
6
0
if and only if
8
x2 =
(
−x1 + x2
= 1 3
if and only if
2x1 + x2 = 6
x 5
1 = .
3
Hence, x ∈ XP . So, this basis is admissible with basic admissible vector
5 8
, , 0, 0 .
3 3
This vector is non-degenerate.
(2) B = {1, 3}. Let x ∈ R4 be such that x2 = 0 and x4 = 0. Observe that
x1
" # " #
−1 1 1 0 0
1
=
2 1 0 1 x3
6
0
if and only if
( (
−x1 + x3 = 1 x3 = 4
if and only if
2x1 = 6 x1 = 3.
Hence, x ∈ XP . Therefore, this basis is admissible with basic admissible vector
(3, 0, 4, 0).
This vector is non-degenerate.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
92 CHAPTER 4. COMPUTING OPTIMIZERS
(3) B = {1, 4}. Let x ∈ R4 be such that x2 = 0 and x3 = 0. Note that
x1
" # " #
−1 1 1 0 0
1
=
2 1 0 1 0
6
x4
if and only if (
−x1 = 1
2x1 + x4 = 6.
So, x ∈
/ XP . Thus, the basis B = {1, 4} is not admissible.
We omit details for the other bases. Anyhow, the admissible bases are:
{1, 2}, {1, 3}, {2, 4} and {3, 4},
with the following basic admissible vectors:
5 8
, , 0, 0 , (3, 0, 4, 0), (0, 1, 0, 5) and (0, 0, 1, 6),
3 3
respectively.
Proposition 4.4
There is at most one basic admissible vector for each basis of a standard opti-
mization problem.
Proof:
For x ∈ Rn to be admissible, it is necessary that Ax = b; that is, AB xB +
AN \B xN \B = b. Then, for B to be a basis for x, it is necessary that AB xB +
AN \B 0 = b; that is, AB xB = b. This system has exactly one solution, since
AB is nonsingular. If this solution (in Rm ) is non-negative, then adding zeros
in the adequate positions yields an admissible vector basic for B. QED
Proposition 4.5
The set of basic admissible vectors of a standard optimization problem is finite.
Proof:
The thesis follows from Exercise 4.1 and Proposition 4.4. QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.1. BASIC VECTORS 93
Notation 4.3
Given x ∈ Rn , we denote by
Px
the set {j ∈ N : xj > 0}.
The following results provide characterizations of basic admissible vectors.
Proposition 4.6
Consider a standard optimization problem with matrix A and let x be an
admissible vector. Then, x is basic if and only if the set of columns of A with
indices in Px is linearly independent.
Proof:
(→) Assume that x is basic and B ⊆ N is a basis for x. Thus, xj = 0 for each
j ∈ (N \ B). Hence, Px ⊆ B. Therefore, since the set {a•j : j ∈ B} is linearly
independent, so is the set {a•j : j ∈ Px }.
(←) Assume that the set of columns with indices in Px (that is, the set
{a•j : j ∈ Px }) is linearly independent. We consider two cases:
(1) |Px | = rank(A). Then, Px is an index basis for x. Hence, x is basic.
(2) |Px | < rank(A). By Proposition 4.11, there is a set of columns of A consti-
tuting a basis for span(A) and containing {a•j : j ∈ Px }. Let B be the set of
indices of the columns in that set. Then, B ⊃ Px and |B| = rank(A). Finally,
xj = 0 for each j ∈ (N \ B) follows from (N \ B) ⊂ (N \ Px ). QED
Proposition 4.7
Consider a standard optimization problem and let x be an admissible vector.
Then, x is basic if and only if for every admissible vectors y and z, whenever
there exists α ∈ ]0, 1[ such that x = αy + (1 − α)z then x = y = z.
Proof:
Let A be the matrix of the problem.
(→) Assume that x is basic. Let y and z be admissible vectors and α ∈ ]0, 1[
such that
x = αy + (1 − α)z.
Let B be an admissible basis for x. Hence, xj = 0 for every j ∈ N \ B. Thus,
yj = zj = 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
94 CHAPTER 4. COMPUTING OPTIMIZERS
for every j ∈ N \ B, since xj = αyj + (1 − α)zj , yj ≥ 0, zj ≥ 0, αyj ≥ 0,
(1 − α) ≥ 0 and (1 − α)zj ≥ 0. Therefore,
AB xB = AB yB = AB zB = b
since Ax = Ay = Az = b. So,
AB (xB − yB ) = AB (xB − zB ) = 0.
Because AB is nonsingular there is a unique w such that AB w = 0. Hence,
xB − yB = xB − zB
and so yB = zB . Thus, y = z and because x = αy +(1−α)z we have x = y = z.
(←) Assume that, for every admissible vectors y and z if
x = αy + (1 − α)z
for some α ∈ ]0, 1[ then x = y = z. Suppose, by contradiction, that x is
not basic. Then, by Proposition 4.6, the set of columns in APx is not linearly
independent. Then, there is
β 6= 0
in R|Px | such that APx β = 0. Let ` be the index of the component of β with
the greatest absolute value, `0 the index of the smallest component of xPx , and
let
(xPx )`0
γ= .
|β` |
Then, for every j = 1, . . . , |Px |, we have
|β` | |βj | (xPx )`0
(xPx )j ≥ (xPx )`0 = (xPx )`0 ≥ (xPx )`0 = |βj | = γ|βj |.
|β` | |β` | |β` |
Hence,
(xPx )j ≥ γβj and (xPx )j ≥ −γβj .
So,
(xPx )j − γβj ≥ 0 and (xPx )j + γβj ≥ 0.
Observe that γβ 6= 0. Let w ∈ Rn be such that
wPx = γβ and wj = 0 for every j ∈ N \ Px .
Then, w 6= 0 and
x − w ≥ 0 and x + w ≥ 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.2. USING BASIC VECTORS 95
since x ≥ 0. On the other hand,
A(x + w) = Ax + Aw = b + APx wPx = b + APx γβ = b + 0 = b
and similarly
A(x − w) = b.
Therefore, x + w and x − w are admissible vectors. Observe that
1 1
x= (x + w) + (x − w).
2 2
Thus, x = x + w = x − w and so w = 0 which is a contradiction since we proved
before that w 6= 0. QED
4.2 Using Basic Vectors
In this section, we prove sufficient conditions to find optimizers. The following
result gives a condition that guarantees that, for each admissible vector, there
is a basic admissible vector with a better or at least the same objective.
Proposition 4.8
Let P be a standard optimization problem such that the objective map is
bounded from below in XP . Then, for every z ∈ XP , there exists a basic
vector z̃ ∈ XP such that cz̃ ≤ cz.
Proof:
Take z ∈ XP . Consider the set
XPz = {x ∈ XP : cx ≤ cz}.
Observe that XPz 6= ∅. Among the elements of XPz , pick z̃ with the largest
possible number of zero components. That is, if y ∈ XPz then
|N \ Py | ≤ |N \ Pz̃ |.
Consider two cases.
(1) z̃ = 0. Then, Pz̃ = ∅. Thus, the set of columns of A with indices in Pz̃
is linearly independent since it is the emptyset (see Example 1.26). Hence, by
Proposition 4.6, z̃ is a basic vector.
(2) z̃ 6= 0. Assume, by contradiction, that the set of columns of A with indices
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
96 CHAPTER 4. COMPUTING OPTIMIZERS
in Pz̃ is linearly dependent. Then, there exists a non-zero vector v ∈ R|Pz̃ | such
that
APz̃ v = 0.
Assume that cPz̃ v < 0 (the other cases follow in a similar way). Let w ∈ Rn
be such that (
wj = vj whenever j ∈ Pz̃
0 otherwise.
That is, wPz̃ = v and, moreover, Aw = 0. Furthermore, cw < 0. Consider the
family
{y t }t∈R+
0
of vectors in Rn , where
y t = z̃ + tw
for each t ∈ R+
0 . Observe that
Ay t = b,
since Az̃ = b and Aw = 0. The proof proceeds by cases:
(a) w ≥ 0. Then, for every t ≥ 0, the vector y t = z̃ + tw is admissible, since
y t ≥ 0, because z̃ ≥ 0, as z̃ is admissible, and tw ≥ 0. On the other hand,
cy t = cz̃ + tcw
and so the value of cy t → −∞ when t → +∞ because cw < 0. Therefore, the
objective map is not bounded from below on XP , contradicting the hypothesis.
(b) There exists j ∈ Pz̃ such that wj < 0. Let
z̃`
• u = max{− : w` < 0};
w`
z̃`∗
• `∗ be such that u = − .
w`∗
(i) yku = 0 for k ∈ (N \ Pz̃ ).
It is enough to observe that yju = 0 whenever z̃j = 0 due to the definition of w.
(ii) y u has one more zero than z̃.
By construction, y`u∗ = 0. Since w`∗ < 0, then `∗ ∈ Pz̃ . So, z̃`∗ > 0.
(iii) yku ≥ 0 for k ∈ Pz̃ \ {`∗ }.
We have three cases. If wk = 0 then by definition yku = z̃k ≥ 0. Assume that
wk > 0. Then, yku ≥ 0. Assume now that wk < 0. By definition of u we have
z̃k
− ≤ u.
wk
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.2. USING BASIC VECTORS 97
Thus, yku = z̃k + uwk ≥ 0.
Hence, y u has at least one more zero than z̃ and y u ∈ XP . This fact contradicts
the choice of z̃ since
cy u = cz̃ + ucw ≤ cz̃ ≤ cz.
Hence, y u ∈ XPz .
Thus, the set of columns of APz̃ is linearly independent. Hence, by Proposi-
tion 4.6, we conclude that z̃ is basic. QED
The importance of Proposition 4.8 is clear in the next result, called Basic
Minimizer Theorem.
Theorem 4.1
Consider a standard optimization problem P . Assume that XP is non-empty
and that the objective map is bounded from below in XP . Then, there is a
basic admissible vector in SP .
Proof:
By Proposition 4.8, for each x ∈ XP there exists a basic vector x̃ ∈ XP such
that cx̃ ≤ cx. Hence,
inf{cx : x ∈ XP } = inf{cx̃ : x ∈ XP }.
Since the set of basic admissible vectors is finite, see Proposition 4.5, the set
{cx̃ : x ∈ XP } is also finite but non-empty. Hence, this set has a minimum.
Then, there is a minimizer which is basic. QED
The constructive nature of this result should be stressed: it provides an
algorithm to find a minimizer when the objective map has a lower bound in
XP and XP 6= ∅. Indeed, since the set of basic admissible vectors is finite (see
Proposition 4.5), it suffices to search for a minimizer in this set. Obviously,
this is an inefficient algorithm (see Proposition 7.10).
Example 4.3
Consider the following standard optimization problem:
min
x
x1
x2 − x3 = 0
x ≥ 0.
Observe that any vector (0, r, r) with r ≥ 0 is a minimizer but only the vector
(0, 0, 0) is a basic admissible vector.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
98 CHAPTER 4. COMPUTING OPTIMIZERS
Example 4.4
Recall the standard optimimization problem P in Example 4.2. The basic
admissible vectors are
5 8
, , 0, 0 , (3, 0, 4, 0), (0, 1, 0, 5) and (0, 0, 1, 6).
3 3
We now prove that the objective map
(x1 , x2 , x3 , x4 ) 7→ −x1 − 4x2
is bounded from below in XP . Taking into account the expression of the objec-
tive map, it is enough to provide an upper bound for the first two components
of each admissible vector. It is immediate that x1 ≤ 3 and x2 ≤ 6 because
2x1 + x2 + x4 = 6 and x1 , x2 , x4 ≥ 0. Hence, Theorem 4.1 can be used to
conclude that there is a basic admissible vector which is a minimizer. Since the
minimum value of the objective map among the basic vectors is
37
− ,
3
which is achieved for the vector
5 8
, , 0, 0 ,
3 3
this vector is a minimizer of P .
Although Theorem 4.1 was established for a standard optimization prob-
lem, it is clearly useful, by applying the relevant transformations, to canonical
optimization problems (Proposition 1.3). The corresponding result is called
Constructive Maximizer Theorem.
Theorem 4.2
Consider a canonical optimization problem P . Assume that XP is non-empty
and that the objective map is bounded from above in XP . Then, there is
s ∈ SP such that f (s) ∈ SCS(P ) is a basic admissible vector.
Proof:
Observe that CS(P ) is a standard optimization problem, by Proposition 4.1.
By Proposition 1.3, XCS(P ) 6= ∅ since XP 6= ∅. Let µ be such that
cx ≤ µ
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.3. BASIC THROUGH COUNTING 99
for every x ∈ XP . Let y ∈ XCS(P ) . Then,
" #
x
y=
b − Ax
for some x ∈ XP . Hence,
−c 0 y = −cx ≥ −µ.
Therefore, the objective map of CS(P ) is bounded from below in XCS(P ) . Let
s0 ∈ SCS(P ) be a basic admissible vector (see Theorem 4.1). Hence, s0 |n ∈ SP ,
by Proposition 1.3. QED
4.3 Basic Through Counting
We now provide a characterization of basic vectors that will allow to check
whether or not a given admissible vector is basic, by just counting the number
of non-zero components of the vector. This characterization holds when the
optimization problem is non-degenerate.
Recall by Proposition 4.16 that when Ax = b then b is a linear combination
of the columns of A.
Definition 4.4
A standard optimization problem is non-degenerate whenever b is not a linear
combination of less than m columns of A.
Example 4.5
Recall problem P in Example 1.13. We now show that P is non-degenerate.
Because of m = 2, we must check whether or not b is a combination of any of
the columns of A. Indeed, it is not the case that either
6 3
=α
6 −1
for some α ∈ R, or
6 −1
=α
6 3
for some α ∈ R, or
6 1
=α
6 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
100 CHAPTER 4. COMPUTING OPTIMIZERS
for some α ∈ R, or
6 0
=α
6 1
for some α ∈ R.
The following result is known as the Non-Degeneracy Lemma.
Proposition 4.9
Let P be a non-degenerate standard optimization problem and x ∈ XP . Then,
x is basic if and only if |Px | = m.
Proof:
(→) Assume x is basic. Then, by Definition 4.3 of basic vector, |Px | ≤ m. We
now show that |Px | ≥ m. Indeed, assume, by contradiction, that |Px | < m.
Observe that, taking into account Proposition 4.16,
Ax = APx xPx + A(N \Px ) x(N \Px ) = APx xPx = b.
Hence, X
a•j xj = b.
j∈Px
Therefore, b is a linear combination of less than m columns of A, contradicting
the non-degeneracy hypothesis.
(←) Suppose |Px | = m. Without loss of generality, assume that Px = {1, . . . , m} =
M . Then, by Proposition 4.16,
b = Ax = x1 a•1 + · · · + xm a•m .
We show that the set of columns {a•1 , . . . , a•m } is linearly independent, which
by Proposition 4.6 is equivalent to saying that x is basic. Suppose, by contra-
diction, that the set of columns {a•1 , . . . , a•m } is linearly dependent. Then,
there exists a non-zero v ∈ Rm such that
v1 a•1 + · · · + vm a•m = 0.
Observe that
(x1 − αv1 )a•1 + · · · + (xm − αvm )a•m = b,
for every α ∈ R. Take
xi
v if vi > 0
i
α=
− xi
otherwise,
vi
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.3. BASIC THROUGH COUNTING 101
where i ∈ M is such that vi 6= 0. Then, (xi − αvi ) = 0. Therefore, it follows
that b is a linear combination of less than m columns of A, contradicting the
non-degeneracy hypothesis. QED
Hence, in the case of non-degeneracy, every basic admissible vector is non-
degenerate (see Definition 4.3). Moreover, for any non-degenerate standard
optimization problem, we can find the basic admissible vectors by picking up
all combinations of n − m variables on {x1 , . . . , xn } and solve the system of
equations of the problem when those variables are 0.
Example 4.6
Consider the standard optimization problem in Example 1.13. There are
4
=6
2
possible basic admissible vectors. The basic vectors are found as follows:
• x3 = x1 = 0: the system
(
−x2 = 6
3x2 + x4 = 6
has the solution (0, −6, 0, 24) which is not a basic admissible vector since
(0, −6, 0, 24) ∈
/ XP .
• x3 = x2 = 0: the system
(
3x1 = 6
−x1 + x4 = 6
has the solution (2, 0, 0, 8) which is a basic admissible vector since (2, 0, 0, 8) ∈
XP .
• x3 = x4 = 0: the system
(
3x1 − x2 = 6
−x1 + 3x2 = 6
has the solution (3, 3, 0, 0) which is a basic admissible vector since (3, 3, 0, 0) ∈
XP .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
102 CHAPTER 4. COMPUTING OPTIMIZERS
• x4 = x1 = 0: the system
(
−x2 + x3 = 6
3x2 = 6
has the solution (0, 2, 8, 0) which is a basic admissible vector since (0, 2, 8, 0) ∈
XP .
• x4 = x2 = 0: the system
(
3x1 + x3 = 6
−x1 = 6
has the solution (−6, 0, 24, 0) which is not a basic admissible vector since
(−6, 0, 24, 0) ∈
/ XP .
• x1 = x2 = 0: the system (
x3 = 6
x4 = 6
has the solution (0, 0, 6, 6) which is a basic admissible vector since (0, 0, 6, 6) ∈
XP .
4.4 Solved Problems and Exercises
Problem 4.1
Consider a standard optimization problem. Assume that the coefficients of the
matrix are integer numbers. Show that, if x is a basic admissible vector, then
for every j ∈ N ,
|xj | ≤ m! αm−1 β,
where
• α = maxi∈M,j∈N |aij |;
• β = maxi∈M |bi |.
Solution:
Let x be a basic admissible vector. Then,
Ax = AB xB = b
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.4. SOLVED PROBLEMS AND EXERCISES 103
where B is a basis for x. Since AB is a nonsingular matrix,
xB = (AB )−1 b.
Observe that, (see Proposition 4.22)
1
(AB )−1 = (cof AB )T
det AB
where
cof AB = {(−1)i+j det (AB )i,j }i,j=1,...,m .
Let
(AB )i,j = {ek` }k,`=1,...,m−1
and P be the set of all permutations of {1, . . . , m − 1}. Then, by the Leibniz’s
Formula (see Proposition 4.19), we have:
X m−1
Y
det (AB )i,j = sgn(p) ekp(k) .
p∈P k=1
Taking into account that
α= max |aij |,
i∈M,j∈N
then
m−1
Y
|ekp(k) | ≤ αm−1 .
k=1
Hence,
| det (AB )i,j | ≤ (m − 1)! αm−1 .
Since AB has integer coefficients and is nonsingular, then | det AB | is at least
one. Thus,
1
0< ≤ 1.
| det AB |
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
104 CHAPTER 4. COMPUTING OPTIMIZERS
Finally,
m
X
|(xB )j | = | ((AB )−1 )j,i bi |
i=1
m
X
≤ |((AB )−1 )j,i | |bi |
i=1
m
X 1
= | ((cof AB )T )j,i | |bi |
i=1
det AB
m
X 1
= | (det (AB )i,j )| |bi |
i=1
det AB
m
1 X
≤ | | (m − 1)! αm−1 |bi |
det AB i=1
m
X
≤ (m − 1)! αm−1 |bi |
i=1
Xm
≤ (m − 1)! αm−1 β
i=1
≤ m! αm−1 β,
for each j = 1, . . . , |B|. /
Problem 4.2
Let P be a standard optimization problem. Show that:
1. if there is a degenerate basic vector then P is degenerate;
2. if b is a linear combination, with non-negative coefficients, of less than
m columns (hence, the problem is degenerate) and the objective map in
XP is bounded from below, then there exists a degenerate basic vector
in XP .
Solution:
(1) Let x be a degenerate basic admissible vector of P . Observe that, by
definition of degenerate vector,
|Px | < m.
Since,
APx xPx = b,
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.4. SOLVED PROBLEMS AND EXERCISES 105
then b is a linear combination of less than m columns of matrix A. Therefore,
P is degenerate.
(2) Assume that b is a linear combination of less than m columns of A with
non-negative coefficients. Let b be such that
b = αi1 a•i1 + · · · + αik a•ik
with k < m and αi` ≥ 0 for ` = 1, . . . , k. Let x ∈ Rn be such that
(
αj if j ∈ {i1 , . . . , ik }
xj =
0 otherwise.
Hence, x ∈ XP . Thus, invoking Proposition 4.8, there exists a basic admissible
vector x̃ such that cx̃ ≤ cx. From the proof of that result, it follows that x̃ is
such that
x̃ ∈ {z ∈ XP : cz ≤ cx}
and no element in the latter set has more zero components. Since x ∈ {z ∈
XP : cz ≤ cx}, the number of zeros of x can not be greater than the number
of zeros of x̃. Therefore,
|P x̃| ≤ |Px | = k < m.
Hence, x̃ is degenerate. /
Problem 4.3
Let P be the following standard optimization problem:
min 2x1 + x2
x
1 − x2 + x3 = −1
−x
x1 + x2 + x4 = 3
x1 − x2 + x5 = 1
−x1 + x2 + x6 = 1
x ≥ 0.
Show that there exists a degenerate basic vector in XP .
Solution:
Observe that
−1 −1 0 0
3 1 1 0
1 = 1
+ 2 + 2 .
0 0
1 −1 0 1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
106 CHAPTER 4. COMPUTING OPTIMIZERS
Therefore, b is a linear combination of columns with indices 1, 4 and 6, and
hence of less than 4 columns. On the other hand, 1 is a lower bound in XP
for the objective map. Thus, by Problem 4.2, we conclude that there is a
degenerate basic vector in XP . Namely
x̃ = (0, 1, 0, 2, 0, 2) ∈ XP
is basic and degenerate. /
Exercise 4.2
Consider the following canonical optimization problem P :
max 2x1 + 5x2
x
−2x1 + 2x2 ≤ 2
x1 ≤ 2
x ≥ 0.
(1) Depict the set of admissible vectors.
(2) Use the map CS to obtain a standard optimization problem.
(3) Find the bases and the basic admissible vectors of CS(P ).
(4) The set of maximizers of P is non-empty?
(5) Consider the canonical optimization problem resulting from P by removing
x1 ≤ 2. What can be said about the set of maximizers?
(6) Consider the canonical optimization problem obtained from P by changing
the objective map to (x1 , x2 ) 7→ −x1 + x2 . What can be said about the set of
maximizers?
Exercise 4.3
Consider the following standard optimization problems:
min cx min cx
x x
0 0
P1 = Ax = b P = Ax=b
x ≥ 0 x ≥ 0,
where " # " #
A b
A0 = and b0 = .
c inf{cx : x ∈ XP1 }
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.4. SOLVED PROBLEMS AND EXERCISES 107
Assume that XP1 6= ∅ and that the objective map is bounded from below on
XP1 . Show that SP1 = SP .
Exercise 4.4
Consider the following standard optimization problems
min cx
min cx x
x
Ax = b
P1 = Ax = b P2 =
x≥0
x ≥ 0
x ≤ µ,
where
• P1 is restricted;
• the components of c are in Z;
• the objective map is bounded from below on XP1 ;
• the standard optimization problem
min cx
x
0 0
P = Ax=b
x ≥ 0
with " # " #
A b
A0 = and b0 =
c inf{cx : x ∈ XP1 }
is restricted;
• µ is
(m + 1)! × max{α, max |cj |}m × max{β, inf{cx : x ∈ XP1 }},
1≤j≤n
where
– α = max1≤i≤m,1≤j≤n |aij |;
– β = maxi∈M |bi |.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
108 CHAPTER 4. COMPUTING OPTIMIZERS
(1) Show that SP1 = SP2 .
(2) Choose A, b and c so that XP1 6= XP2 .
Exercise 4.5
Show that every basic admissible vector with more than one basis is degenerate.
4.5 Relevant Background
In this section, we provide a brief survey of the relevant linear algebra con-
cepts and results needed for getting the new perspective on linear optimiza-
tion discussed in this chapter. The interested reader can find more details
in [46, 38, 7, 14, 43, 59].
The following result and Proposition 1.12 are crucial for the notion of di-
mension of a vector space to be well defined.
Proposition 4.10
Any two bases of a vector space over a field K have the same the cardinality.
Definition 4.5
The dimension of a vector space V over a field K, denoted by
dim V,
is the cardinality of any basis of the vector space.
Example 4.7
The dimension of the vector space
{0}K
is 0, by Example 1.28.
Example 4.8
The dimension of the vector space Rn is n taking account the standard basis
of Rn (Definition 1.20).
Proposition 4.11
Let V be a vector space over a field K of dimension n and {v 1 , . . . , v k } ⊆ V
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.5. RELEVANT BACKGROUND 109
be a linearly independent set of vectors in V with k < n. Then, there are
v k+1 , . . . , v n ∈ V such that {v 1 , . . . , v n } is a basis of V .
Proposition 4.12
Let V be a vector space over a field K of dimension n and V 0 ⊆ V a linearly
independent set in V such that |V 0 | = n. Then, V 0 is a basis for V .
Proposition 4.13
Let V1 band V2 be subspaces of a vector space over a field K. Then, dim V1 ≤
dim V2 .
Proposition 4.14
Let V1 and V2 be subspaces of a vector space over a field K. Then,
dim(V1 + V2 ) = dim(V1 ) + dim(V2 ) − dim(V1 ∩ V2 ).
Finally, we introduce the concept of orthogonal complement of a subspace,
to be used later on.
Definition 4.6
Let V1 be a subspace of the vector space V over a field K. The orthogonal
complement of V1 , denoted by
V1⊥ ,
is the set {v ∈ V : v · v 1 = 0, for each v 1 ∈ V1 }.
Example 4.9
Let
V1 = {(x1 , x2 ) ∈ R2 : x1 = x2 }.
Then, V1 is a subspace of R2 with dim V1 = 1. Moreover,
V1⊥ = {(x1 , x2 ) ∈ R2 : x1 = −x2 }.
Proposition 4.15
Let V1 be a subspace of the vector space V over a field K. Then,
• V1⊥ is a subspace of V ;
• dim V1⊥ = dim V − dim V1 ;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
110 CHAPTER 4. COMPUTING OPTIMIZERS
• (V1⊥ )⊥ = V1 .
Proposition 4.16
Let A be an m × n-matrix, x ∈ Rn and b ∈ Rm such that Ax = b. Then,
b = x1 a•1 + · · · + xn a•n .
Definition 4.7
An m × m-matrix A is nonsingular whenever
{a•1 , . . . , a•m }
is linearly independent in Rm .
Example 4.10
Let A be the following matrix
−1 1 1
2 −1 − 23 .
1 3 1
We start by observing that the set {a•1 , a•2 , a•3 } of columns of A is
−1
1 1
2 , −1 , − 32 .
1 3 1
We now show that this set is linearly dependent. Assume that
−1 1 1
α1 2 + α2 −1 + α3 − 23 = 0.
1 3 1
Hence,
−α1 + α2 + α3
= 0
2α1 − α2 − 32 α3 = 0
α1 + 3α2 + α3 = 0.
Thus,
α1
= 12 α3
0α3 = 0
= − 12 α3 .
α2
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.5. RELEVANT BACKGROUND 111
Thus, choosing, for instance, α3 = 2, α1 = 1 and α2 = −1 we have
−1 1 1 0
3
α1 2 + α2 −1 + α3 − 2 = 0 .
1 3 1 0
Therefore, A is singular.
Proposition 4.17
An m × m-matrix A is nonsingular if and only if there is an m × m-matrix B
such that
AB = BA = I,
where I is the identity matrix.
We now present the important notion of determinant of a square matrix.
Before that, we need some notation.
Notation 4.4
Let A be an n × n-matrix and i, j ∈ N . We denote by
Ai,j
the (n − 1) × (n − 1)-matrix obtained from A by removing line i and column j.
We define the determinant of a square matrix following the Laplace’s Ex-
pansion Formula.
Definition 4.8
Let A be an n × n-matrix. The determinant of A denoted by
det A
is
1 if n = 0
Xn
(−1)i+j aij det Ai,j otherwise
j=1
for a fixed i ∈ M .
Observe that the definition of det A is independent of i.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
112 CHAPTER 4. COMPUTING OPTIMIZERS
Proposition 4.18
Let A be an n × n-matrix. Then,
1 if n = 0
det A = X n
(−1)i+j aij det Ai,j otherwise,
i=1
for a fixed j ∈ M .
Definition 4.9
Let A be an n × n-matrix and i, j ∈ N . The (i, j)-minor of A, denoted by
A
Mi,j ,
is
det Ai,j .
Moreover, the (i, j)-cofactor of A, denoted by
A
Ci,j ,
is (−1)i+j Mi,j
A
.
Remark 4.2
Observe that, given an n × n-matrix A, we have
1 if n = 0
n
det A = X A
aij Ci,j otherwise,
j=1
for a fixed i ∈ M .
This remark explains why the Laplace’s Expansion Formula is also known
as the Cofactor Expansion. The following characterization of the determinant
is known as the Leibniz’s Formula (see [7]).
Proposition 4.19
Let A be an n × n-matrix. Then,
X n
Y
det A = sgn(p) aip(i) ,
p∈P i=1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.5. RELEVANT BACKGROUND 113
where P is the set of permutation of N and
(
1 if p is an even permutation
sgn(p) =
−1 otherwise
(an even permutation is obtained from the identity permutation as the com-
position of an even number of exchanges of two elements and similarly for odd
permutations).
Proposition 4.20
Let A and B be n × n-matrices and I the identity n × n-matrix. Then,
• det I = 1;
• det(αA) = αn det A;
• det(A × B) = (det A) × (det B);
1
• det A−1 = whenever det A 6= 0.
det A
The following result provides a characterization of a nonsingular matrix via
the determinant of A.
Proposition 4.21
An n × n-matrix A is nonsingular if and only if det A 6= 0.
Proposition 4.22
Let A be a nonsingular n × n-matrix. Then,
1
A−1 = (cof A)T ,
det A
where
A
cof A = (Ci,j )i,j=1,...,n
is the matrix of cofactors of A.
The following result is known as the Cramer’s Rule (see [14]).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
114 CHAPTER 4. COMPUTING OPTIMIZERS
Proposition 4.23
Let A be a nonsingular n × n-matrix and x, b ∈ Rn such that x = A−1 b. Then,
for each j = 1, . . . , n,
det [A]jb
xj = ,
det A
where [A]jb is the matrix obtained from A by replacing column j by b.
Definition 4.10
The rank of a matrix A, denoted by
rank(A),
is the number of elements of the largest linearly independent set of rows, which
coincides with the number of elements of the largest linearly independent set
of columns.
Observe that the rank of an m × n-matrix is at most min(m, n).
Example 4.11
Let A be the following matrix
1 0
2 3 .
1 2
Then, rank(A) is 2 since the set composed by the two columns is linearly
independent.
Recall the concept of span of a set of vectors in Definition 1.16.
Definition 4.11
Given an m × n-matrix A, the set
span(A) = spanRm ({a•j : j ∈ N })
is called the span of A.
Proposition 4.24
The span of an m × n-matrix A is a subspace of the vector space Rm and the
dimension of span(A) is rank(A).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
4.5. RELEVANT BACKGROUND 115
It is possible to establish the relationship between the rank and the number
of columns of a matrix. Before that, we need to introduce the concept of kernel
of a matrix.
Definition 4.12
The kernel of an m × n-matrix A, denoted by
ker(A),
is the set {u ∈ Rn : Au = 0}.
The following result is known as the Rank-Nullity Theorem for matrices.
Theorem 4.3
Given a matrix A, then
rank(A) + dim ker(A)
is the number of columns of A.
Example 4.12
Observe that
−2 1
ker = {(x1 , x2 ) ∈ R2 : x2 = 2x1 }.
2 −1
Hence, by Theorem 4.3,
−2 1
rank =1
2 −1
since
−2 1
dim ker =1
2 −1
and the number of columns is 2.
Proposition 4.25
Let A be an m × n-matrix. Then, {a•1 , . . . , a•n }⊥ = ker(A).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
116 CHAPTER 4. COMPUTING OPTIMIZERS
Proposition 4.26
Let V be a k-dimensional subspace of Rn and B be a basis of V ⊥ . Then,
V = ker(E),
where E is the (n − k) × n matrix whose lines are the elements of B.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 5
Geometric View
The main objective of this chapter is to give a geometrical characterization of
the basic admissible vectors as vertices of an appropriate convex polyhedron.
Capitalizing on this relationship, it is possible to compute optimizers through
geometrical techniques. In Section 5.4, the reader can find the relevant concepts
and results of affine spaces.
5.1 Admissibility
The goal of this section is to introduce several notions concernbing affine spaces
that are relevant to standard optimization problems.
We start by introducing the affine space (see Definition 5.12) useful for
optimization.
Proposition 5.1
The pair
(Rn , Θ),
where Θ is the map
(u, w) 7→ w − u,
is an affine space over Rn .
Proof:
(1) Θv : Rn → Rn is a bijection for each v ∈ Rn . Indeed:
(a) Injectivity. Let u 6= w. Then, u − v 6= w − v. Therefore, Θv (u) 6= Θv (w).
117
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
118 CHAPTER 5. GEOMETRIC VIEW
(b) Θv is an onto map. Let u ∈ V . Then, Θv (u + v) = u.
(2) Θ(u, z) = Θ(u, w) + Θ(w, z). Indeed:
Θ(u, z) = z − u = (z − w) + (w − u) = Θ(w, z) + Θ(u, w).
Hence, (Rn , Θ) is an affine space over Rn . QED
The objective now is to define the smallest affine subspace containing a set.
Definition 5.1
The smallest affine subspace of (Rn , Θ) containing W , denoted by
A(W ),
is the affine subspace
\
U .
{U :U is an affine subspace of (Rn ,Θ) and W ⊆U }
This notion is well defined since the intersection of affine subspaces is an
affine subspace (Proposition 5.20).
Example 5.1
Observe that A({u}) = {u} since {u} is an affine subspace, by Example 5.9.
Moreover, A(∅) = ∅ since ∅ is an affine subspace (see Definition 5.13).
We now provide another characterization of A(W ). With this purpose in
mind, we define affine combination in Rn .
Definition 5.2
An affine combination of points u, w ∈ Rn is a point of the form
αu + (1 − α)w,
for some α ∈ R.
Observe that every convex combination (see Definition 3.1) is an affine
combination but not the other around.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.1. ADMISSIBILITY 119
Example 5.2
The point (3, 6) is an affine combination of the points (2, 3) and (1, 0). Indeed:
" # " # " #
3 2 1
=2 − .
6 3 0
Definition 5.3
A set W ⊆ Rn is closed under affine combinations if
αu + (1 − α)w ∈ U
whenever u, w ∈ W and α ∈ R.
Exercise 5.1
Show that W ⊆ Rn is closed under affine combinations if and only if
k
X
αi wi ∈ W
i=1
Pk
for every w1 , . . . , wk ∈ W and α1 , . . . , αk ∈ R such that i=1 αi = 1.
Proposition 5.2
Any subset of Rn is closed under affine combinations if and only if it is an affine
subspace of (Rn , Θ).
Proof:
Let U ⊆ Rn .
(→) Assume that U is closed under affine combinations. There are two cases:
(1) U = ∅. Then, U is an affine subspace of (Rn , Θ).
(2) U 6= ∅. Take t ∈ U . We show that
Θt (U ) = {x − t : x ∈ U }
is a subspace of Rn .
(a) 0 ∈ Θt (U ), since 0 = t − t;
(b) If v ∈ Θt (U ), then αv ∈ Θt (U ).
Assume that v ∈ Θt (U ). By definition of Θt (U ), we have v = x − t for some
x ∈ U . Then, αv = α(x − t). Thus,
α(x − t) = α(x − t) + (1 − α)(t − t) = αx + (1 − α)t − t.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
120 CHAPTER 5. GEOMETRIC VIEW
Observe that
αx + (1 − α)t ∈ U
because U is closed under affine combinations. Hence, α(x − t) ∈ Θt (U ).
(c) If v 1 , v 2 ∈ Θt (U ) then v 1 + v 2 ∈ Θt (U ).
Assume that v 1 , v 2 ∈ Θt (U ). By definition of Θt (U ), there exist x1 , x2 ∈ U
such that v 1 = x1 − t and v 2 = x2 − t. Therefore, we must show that
v 1 + v 2 = (x1 + x2 ) − (t + t) ∈ Θt (U ).
Since
1 1 1 1 1
(x + x2 ) = x + 1− x2 ,
2 2 2
then
1 1
(x + x2 ) ∈ U
2
since U is closed under affine combinations. Hence,
1 1 1
(x + x2 ) − (t + t) ∈ Θt (U ).
2 2
Thus,
1 1 1
(x1 + x2 ) − (t + t) = 2 (x + x2 ) − (t + t) ∈ Θt (U ),
2 2
using (b).
(←) Assume that U is an affine subspace of (Rn , Θ). Then, let t ∈ U be such
that
Θt (U )
is a subspace of Rn . Let u, w ∈ U and α ∈ R. Then,
αΘt (u) + (1 − α)Θt (w) ∈ Θt (U );
that is,
αu + (1 − α)w − t ∈ Θt (U ).
Therefore, αu + (1 − α)w ∈ U . QED
Example 5.3
The set {x ∈ Rn : x ≥ 0} is not closed for affine combinations as the following
case shows. Observe that
2e1 − e2 ∈
/ {x ∈ Rn : x ≥ 0},
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.1. ADMISSIBILITY 121
where e1 and e2 are vectors in the canonical basis of Rn (see Definition 1.20).
Therefore, by Proposition 5.2, the set {x ∈ Rn : x ≥ 0} is not an affine subspace
of (Rn , Θ).
Definition 5.4
The affine hull of W ⊆ Rn , denoted by
A(W ),
is the set
( k k
)
X X
αi wi : w1 , . . . , wk ∈ W, α1 , . . . , αk ∈ R, αi = 1 .
i=1 i=1
Proposition 5.3
Let W ⊆ Rn . Then
A(W ) = A(W ).
Proof:
(⊆) Let U be an affine subspace containing W . We must show that A(W ) ⊆ U .
Observe that, by Proposition 5.2, U is closed under affine combinations. Since
U contains W then U is closed under affine combinations of elements of W .
(⊇) Observe that W ⊆ A(W ). Moreover, by Proposition 5.2, A(W ) is an affine
subspace, since A(W ) is closed under affine combinations, by Exercise 5.1.
Then, A(W ) ⊆ A(W ), by definition of A(W ). QED
The following result states that the set of solutions of a system of equations
is an affine subspace (see Definition 5.13) over (Rn , Θ).
Proposition 5.4
0
Let A0 be an m0 × n0 -matrix, b0 ∈ Rm ,
0
U = {x ∈ Rn : A0 x = b0 }
0
and t ∈ U . Then, Θt (U ) = ker(A0 ). So, U is an affine subspace of (Rn , Θ).
Proof:
Indeed:
(⊆) Assume that v ∈ Θt (U ). Then, v = x − t, where x ∈ U . Hence, A0 v =
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
122 CHAPTER 5. GEOMETRIC VIEW
A0 x − A0 t = b0 − b0 = 0. That is, v ∈ ker(A0 ).
(⊇) Let z ∈ ker(A0 ). Observe that z = (z + t) − t and A0 (z + t) = A0 z + A0 t =
0 + b0 = b0 . Thus, z + t ∈ U . Therefore, z ∈ Θt (U ). QED
To find a characterization of the affine hull and the dimension of the set of
admissible vectors, we need the following auxiliary definition and lemma.
Definition 5.5
Let P be a standard optimization problem where A is an m × n-matrix. The
matrix and the vector of implicit equality restrictions of P , denoted by
A= and b= ,
are " # " #
A b
and ,
I= 0
respectively, where I = is the submatrix of I obtained by removing each line
with index in
N > = {j ∈ N : xj > 0 for some x ∈ XP }.
Example 5.4
Consider the following standard optimization problem P :
min −2x1 − 5x2
x
x1 + x2 + x3 = 1
−x1 − x2 + x4 = −1
x ≥ 0.
Observe that
1 1
, , 0, 0 ∈ XP
2 2
and so 1, 2 ∈ {1, 2, 3, 4}> . Then, x3 + x4 = 0. Therefore, x3 = x4 = 0. Hence,
{1, 2, 3, 4}> = {1, 2}.
So,
1 1 1 0 1
−1 −1 0 1 −1
A= = and b= = .
0 0 1 0 0
0 0 0 1 0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.1. ADMISSIBILITY 123
Exercise 5.2
Let P be a standard optimization problem where A is an m × n-matrix. Show
that
XP = {x ∈ Rn : A= x = b= and xj ≥ 0 for every j ∈ N > }.
Proposition 5.5
Let P be a standard optimization problem where A is an m × n-matrix with
XP 6= ∅. Then, there is x ∈ XP such that xj > 0 for every j ∈ N > .
Proof:
Let wj ∈ XP be a vector such that wjj > 0 and αj ∈]0, 1[ for every j ∈ N >
such that X
αj = 1.
j∈N >
Let X
x= αj wj .
j∈N >
It is immediate that xj > 0 for every j ∈ N > . Moreover,
X X
A= x = αj A= wj = αj b= = b= .
j∈N > j∈N >
Therefore, by Exercise 5.2, x ∈ XP . QED
Exercise 5.3
Let W1 , W2 ⊆ Rn be such that W1 ⊆ W2 . Show that A(W1 ) ⊆ A(W2 ).
Proposition 5.6
Let P be a standard optimization problem where A is an m × n-matrix with
XP 6= ∅. Then
A(XP ) = {y ∈ Rn : A= y = b= }.
Proof:
(⊆) Observe that
A(XP ) ⊆ A({y ∈ Rn : A= y = b= }),
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
124 CHAPTER 5. GEOMETRIC VIEW
by Exercise 5.3, since XP ⊆ {y ∈ Rn : A= y = b= } (see Exercise 5.2). The
thesis follows because
A({y ∈ Rn : A= y = b= }) = {y ∈ Rn : A= y = b= },
by Proposition 5.4 and Proposition 5.3.
(⊇) Take y ∈ {y ∈ Rn : A= y = b= }. Let x ∈ XP be such that xj > 0 for every
j ∈ N > (see Proposition 5.5). If y = x then y ∈ XP ⊆ A(XP ). Otherwise,
consider two cases. Assume that
{k ∈ N > : yk 6= xk } = ∅.
Then, yk = xk ≥ 0 for every k ∈ N > . Hence, y ∈ XP ⊆ A(XP ), by Exercise 5.2.
Otherwise, let
z β = βy + (1 − β)x
where
xk
max when xk − yk ≤ 0 for each k ∈ N >
k 6=yk } xk − yk
> :x
k∈{k∈N
β= xk
min otherwise.
k∈{k∈N :xk −yk >0} xk − yk
>
We start by showing that
z β ∈ XP ,
taking into account Exercise 5.2. It is immediate that A= z β = b= . It remains
to prove that zjβ ≥ 0 for every j ∈ N > . Let j ∈ N > . We have two cases:
(1) xk − yk ≤ 0 for each k ∈ N > . Observe that
{k ∈ N > : yk 6= xk } 6= ∅.
Then,
β < 0.
There are two cases:
(a) xj − yj < 0. Hence,
zjβ = βyj + (1 − β)xj
= β(yj − xj ) + xj
xj
≥ xj −yj (yj − xj ) + xj
= 0
(b) xj − yj = 0. Then,
zjβ = xj > 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.1. ADMISSIBILITY 125
(2) xk − yk > 0 for some k ∈ N > . Then,
β > 0.
There are three cases:
(a) xj − yj < 0. Thus,
β(yj − xj ) > 0.
On the other hand,
xj
(yj − xj ) < 0.
xj − yj
Hence,
xj
β(yj − xj ) > (yj − xj ).
xj − yj
Therefore,
zjβ > 0.
(b) xj − yj > 0. Observe that
xj
β≤ .
xj − yj
Thus,
xj
β(yj − xj ) ≥ (yj − xj ).
xj − yj
Therefore,
zjβ > 0.
(c) xj − yj = 0. Then,
zjβ = xj > 0.
It remains to show that
y ∈ A(XP ).
Observe that
β 6= 0
and
1 β β−1
y= z + x.
β β
Thus, y is an affine combination of x and z β in XP . So, y ∈ A(XP ). QED
We now define the dimension of a set in the context of affine spaces. Recall
the concept of dimension of an affine subspace in Definition 5.15.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
126 CHAPTER 5. GEOMETRIC VIEW
Definition 5.6
The dimension of W ⊆ Rn , denoted by
dim W,
is dim A(W ).
Remark 5.1
Observe that
dim ∅ = −1
taking into account Definition 5.15 and Example 5.1. Moreover, dim ∅ ≤
dim W for every W ⊆ Rn .
Proposition 5.7
Consider the standard optimization problem
min
x
cx
P = Ax = b
x ≥ 0,
where A is an m × n-matrix. Then,
(
n − rank(A= ) if XP 6= ∅
dim XP =
−1 otherwise.
Proof:
If XP = ∅ then dim XP = −1 (see Remark 5.1). Otherwise,
dim XP = dim A(XP ) = dim A(XP ) = dim {x ∈ Rn : A= x = b= }
by Proposition 5.3 and Proposition 5.6. Thus,
dim XP = dim ker(A= ) = n − rank(A= ),
by Proposition 5.4 and Theorem 4.3. QED
5.2 Optimizers
The main objective of this section is to show that basic admissible vectors are
vertices of a relevant convex polyhedron.
We now concentrate on the important notion of hyperplane relevant to
define polyhedron and faces.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.2. OPTIMIZERS 127
Definition 5.7
A hyperplane in Rn is an affine subspace of (Rn , Θ) with dimension n − 1.
Exercise 5.4
Show that any hyperplane is non-empty.
The next result is useful for providing a characterization of hyperplanes.
Proposition 5.8
Let U 6= ∅ be an affine subspace of (Rn , Θ) with dimension k. Then,
U = {x ∈ Rn : Ex = Et},
where E is an (n − k) × n-matrix, the lines of E are the vectors of a basis of
Θt (U )⊥ and t ∈ U .
Proof:
Observe that, for some t ∈ U , Θt (U ) is a subspace of Rn with dimension k.
Thus,
Θt (U )⊥
is a subspace of Rn with dimension n − k (Proposition 4.15). Let
BΘt (U )⊥
be a basis of Θt (U )⊥ (Proposition 1.5 and Proposition 1.12). Take
as the matrix (n − k) × n with the vectors of BΘt (U )⊥ as lines. Therefore, using
Proposition 4.15,
{v ∈ Rn : Ev = 0} = (Θt (U )⊥ )⊥ = Θt (U ). (†)
We are ready to show that
U = {x ∈ Rn : Ex = Et}.
(⊆) Let u ∈ U . Then, u = Θt (u) + t. Thus, Eu = EΘt (u) + Et. Hence, by (†),
Eu = Et.
(⊇) Take x ∈ Rn such that Ex = Et. Then, E(x − t) = 0. Hence, (x − t) ∈
Θt (U ) by (†). Therefore, x ∈ U . QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
128 CHAPTER 5. GEOMETRIC VIEW
Proposition 5.9
Let H ⊆ Rn be a hyperplane and t ∈ H. Then,
H = {x ∈ Rn : v · x = v · t},
where v is the unique vector in a basis of Θt (H)⊥ .
Proof:
Observe that Θt (H) is a subspace of Rn with dimension n − 1. Then,
Θt (H)⊥
has dimension 1, by Proposition 4.15. The thesis follows by Proposition 5.8.
QED
We now provide a characterization for the set of solutions of a particular
equation to be a hyperplane.
Proposition 5.10
Let v ∈ Rn and r ∈ R. Then, {x ∈ Rn : v · x = r} is a hyperplane if and only
if v 6= 0.
Proof:
(→) Assume that v = 0. There are two cases:
(1) r = 0. Then, {x ∈ Rn : v · x = r} = Rn is not a hyperplane, since
dim {x ∈ Rn : v · x = r} = dim Rn = n 6= n − 1.
(2) r 6= 0. Thus, {x ∈ Rn : v · x = r} = ∅ is not a hyperplane, since
dim {x ∈ Rn : v · x = r} = dim ∅ = −1 6= n − 1.
(←) Let H be {x ∈ Rn : v · x = r}, where v 6= 0. By Proposition 5.4, H is an
affine subspace of (Rn , Θ) such that Θt (H) = ker([v T ]) for some t ∈ H. Using,
Theorem 4.3,
n = dim ker([v T ]) + 1
since rank([v T ]) = 1. Thus, dim H = dim ker([v T ]) = n − 1. Therefore, H is a
hyperplane. QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.2. OPTIMIZERS 129
Proposition 5.11
Let H ⊆ Rn . Then, H is a hyperplane if and only if
H = {x ∈ Rn : v · x = r}
for some non-zero v ∈ Rn and r ∈ R.
Proof:
Immediate by Proposition 5.9 and Proposition 5.10. QED
We now introduce the concept of semi-space.
Definition 5.8
Let {x ∈ Rn : v · x = r} be a hyperplane. The sets
{x ∈ Rn : v · x ≥ r} and {x ∈ Rn : v · x ≤ r}
are the upper semi-space and the lower semi-space of the hyperplane, respec-
tively.
The following result states an important topological property of semi-spaces.
Proposition 5.12
The semi-spaces of a hyperplane are closed sets.
Proof:
Let {x ∈ Rn : v · x = r} be a hyperplane. The proof that
{x ∈ Rn : v · x ≥ r}
is a closed set is similar to the one in Proposition 2.9, by choosing the linear
map x 7→ v · x. Similarly for {x ∈ Rn : v · x ≤ r}. QED
Example 5.5
The set {x ∈ Rn : xj ≥ 0} is an upper semi-space of the hyperplane
{x ∈ Rn : xj = 0}
(see Proposition 5.11) and so is closed by Proposition 5.12.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
130 CHAPTER 5. GEOMETRIC VIEW
Exercise 5.5
Show that every semi-space is convex.
We now show that intersection preserves convexity. Afterwards, we provide
an illustration that it is not always the case that convexity is preserved by
union.
Proposition 5.13
The class of convex subsets of Rn is closed under intersection.
Proof:
Let {U k }k∈K be a family of convex sets and
\
U= U k.
k∈K
We consider two cases:
(1) U = ∅. Then, by Exercise 3.1, U is convex.
(2) U 6= ∅. Let u, v ∈ U and α ∈ [0, 1]. Then, u, v ∈ U k for every k ∈ K, hence,
since U k is convex,
(αu + (1 − α)v) ∈ U k
for every k ∈ K. Therefore, (αu + (1 − α)v) ∈ U . QED
Exercise 5.6
Show that every hyperplane is convex.
The next example shows that the union of convex sets is not always a convex
set.
Example 5.6
Let H1 and H2 be the hyperplanes
{x ∈ R2 : v · x = 2} and {x ∈ R2 : v · x = 1},
respectively, where v is (0, 1). Then, H1 and H2 are convex sets (see Exer-
cise 5.6). Observe that (1, 2), (1, 1) ∈ H1 ∪ H2 . Then,
3
1,
2
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.2. OPTIMIZERS 131
is a convex combination of (1, 2) and (1, 1) but it is not in H1 ∪ H2 . Hence,
H1 ∪ H2 is not convex.
Definition 5.9
A convex polyhedron is a finite intersection of semi-spaces. A convex polytope
is a bounded convex polyhedron.
Exercise 5.7
Show that a convex polyhedron is a convex set.
We now show that these notions provide a geometric characterization of the
set of admissible vectors.
Proposition 5.14
The set of admissible vectors of a standard optimization problem is a convex
polyhedron and so is a convex set.
Proof:
Let P be a standard optimization problem with m × n-matrix A. Observe that,
for each j = 1, . . . , n,
{x ∈ Rn : xj ≥ 0}
is a semi-space of Rn (see Example 5.5). On the other hand, for every i =
1, . . . , m, the i-th line of matrix A, denoted ai• , is non-zero (see Remark 1.2).
Thus, for each i = 1, . . . , m,
{x ∈ Rn : ai• x = bi }
is a hyperplane in Rn (see Proposition 5.10). Hence, for every i = 1, . . . , m,
{x ∈ Rn : ai• x ≤ bi } and {x ∈ Rn : ai• x ≥ bi }
are semi-spaces of Rn . Observe that XP is
n
\ m
\
{x ∈ Rn : xj ≥ 0} ∩ ({x ∈ Rn : ai• x ≤ bi } ∩ {x ∈ Rn : ai• x ≥ bi })
j=1 i=1
and so is the intersection of n + 2m semi-spaces. Therefore, XP is a convex
polyhedron and, as a consequence, by Exercise 5.7, is a convex set. QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
132 CHAPTER 5. GEOMETRIC VIEW
Before providing a geometric characterization of the basic admissible vec-
tors, it is necessary to define face.
Definition 5.10
Let X be a convex polyhedron in Rn . A set F ⊆ X is a face of X by the
hyperplane H = {x ∈ Rn : v · x = r} whenever
(
v · x = r for every x ∈ F
v · x > r for every x ∈ (X \ F ).
A vertex of X by H is a face of X by H with dimension 0. We say that x ∈ X
is a vertex of X whenever there exists an hyperplane H such that x is a vertex
of X by H.
The intuition that a vertex is a singleton set is justified by the following
result.
Proposition 5.15
Let U ⊆ Rn . Then, dim U = 0 if and only if U is a singleton set.
Proof:
(←) Observe that, by Example 4.7 and Example 5.9,
dim {u} = dim A({u}) = dim Θu ({u}) = dim {0}R = 0.
(→) Assume that dim U = 0. Then,
Θt (U ) = {0}R .
Taking into account that Θt is a bijection, U is a singleton set. QED
So, in the sequel, for simplifying the presentation, we may identify each
vertex with its unique element.
Example 5.7
Consider the standard optimization problem
min −2x1 − x2
x
3x1 − x2 + x4 = 6
P =
−x1 + 3x2 + x3 = 6
x≥0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.2. OPTIMIZERS 133
(see Example 4.6). Then,
v = (3, 3, 0, 0) ∈ XP
is a vertex of XP . Indeed, take the hyperplane
H = {x ∈ R4 : [0 0 1 1] x = 0}.
Observe that
[0 0 1 1] v = 0.
Hence, v ∈ H. It remains to show that
[0 0 1 1] y = y3 + y4 > 0
for every y ∈ (XP \ {v}). Let y ∈ (XP \ {v}). Observe that either y3 6= 0 or
y4 6= 0, since otherwise y = v. On the other hand, y3 ≥ 0 and y4 ≥ 0 since y is
an admissible vector. Therefore, either y3 > 0 or y4 > 0.
We now provide a geometric characterization of basic admissible vectors.
Proposition 5.16
Let P be a standard optimization problem. Then,
x is a vertex of XP iff x is a basic vector of XP .
Proof:
Indeed:
(→) Assume that x is a vertex of XP . Then, there exist a non-zero v ∈ Rn and
r ∈ R such that v · x = r and v · y > r for every y ∈ (XP \ {x}). Consider the
standard optimization problem P1 :
min
v·y
y
Ay = b
y ≥ 0.
Observe that XP1 = XP . Since v · x = min{v · y : y ∈ XP } = r, then
x ∈ SP1 .
Applying Theorem 4.1, because XP is non-empty and the objective map y 7→
v · y is bounded from below (by r) in XP , then there exists a basic admissible
vector x̃ ∈ XP such that
v · x̃ = min{v · y : y ∈ XP }.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
134 CHAPTER 5. GEOMETRIC VIEW
Since v · y > r for every y ∈ (XP \ {x}), then
x̃ = x.
Thus, x is basic.
(←) Assume that x is a basic vector of XP . Let B ⊆ N be a basis for x. Then,
xj = 0 for every j ∈ (N \ B). Let w ∈ Rn be such that
(
0 if j ∈ B
wj =
1 otherwise.
Then,
(1) w · x = 0 by the definition of w.
(2) w · y > 0 for each y ∈ XP \ {x}. Observe that
X
w·y = yj .
j∈N \B
Then, to show that w · y > 0, we must prove that there exists j ∈ N \ B such
that
yj > 0.
Assume, by contradiction, that yj = 0, for every j ∈ N \ B. Hence,
yj = xj , for every j ∈ N \ B.
On the other hand, since AB is nonsingular, the system
AB zB = b
has a unique solution. Therefore,
yB = xB .
So, y = x which contradicts the fact that y ∈ XP \ {x}. QED
To calculate the vertices of XP , we now provide a characterization of a
relevant hyperplane containing a vertex.
Definition 5.11
A hyperplane {x ∈ Rn : v · x = r} is m-basic, for 0 < m < n, whenever v has
m components with value 0 and n − m components with value 1 and r is 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.2. OPTIMIZERS 135
Proposition 5.17
Let P be a standard optimization problem with an m × n-matrix. Then, x is
a vertex of XP if and only if x is a vertex of XP by an m-basic hyperplane.
Proof:
(→) Assume that x is a vertex of XP . Then, by Proposition 5.16, x is a basic
admissible vector. Let B be the corresponding (index) basis. Then, xj = 0 for
every j ∈ N \ B. Take v such that
(
1 if j ∈ N \ B
vj =
0 otherwise.
Hence, v · x = 0. Assume by contradiction that there is y ∈ XP \ {x} such that
v · y = 0. Then, yN \B = 0. Moreover,
AB yB = Ay = b = Ax = AB xB .
Thus, yB = xB since AB is a nonsingular matrix. So, y = x contradicting the
hypothesis.
(←) Immediate by the definition of vertex. QED
Example 5.8
Consider the standard optimization problem
min −2x1 − 5x2
x
−2x1 + 2x2 + x3 = 2
P =
x1 + x4 = 2
x ≥ 0.
By, Proposition 5.14, XP is a convex polyhedron. We now find the vertices of
XP .
Taking into account Proposition 5.17, the 2-basic hyperplanes associated to a
vertex are of the form
{x ∈ R4 : v · x = 0},
where v ∈ R4 is a vector having 2 components with value 0 and the others with
value 1. Observe that a possible vertice x of XP by the hyperplane is such that
xj = 0 whenever vj = 1. We have the following candidates for v:
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
136 CHAPTER 5. GEOMETRIC VIEW
(1) v = (1, 1, 0, 0). We start by investigating if there is x ∈ XP such that
v · x = 0. Then, x1 = x2 = 0. Moreover, the system of restrictions becomes
x3 = 2
x4 = 2
x3 , x4 ≥ 0.
Hence, x = (0, 0, 2, 2) ∈ XP . Furthermore, v · y > 0 whenever y ∈ XP \ {x}.
Hence, x is a vertex of XP .
(2) v = (1, 0, 1, 0). We start by analyzing if there is x ∈ XP such that v · x = 0.
Then, x1 = x3 = 0. Moreover, the system of restrictions becomes
x2 = 1
x4 = 2
x2 , x4 ≥ 0.
Thus, x = (0, 1, 0, 2) ∈ XP . Furthermore, v · y > 0 whenever y ∈ XP \ {x}.
Hence, x is a vertex of XP .
(3) v = (1, 0, 0, 1). We start by investigating if there is x ∈ XP such that
v · x = 0. Then, x1 = x4 = 0. Moreover, the system of restrictions becomes
2x2 + x3 = 2
0=2
x2 , x3 ≥ 0.
Hence, there is no vertex of XP by this hyperplane.
(4) v = (0, 1, 1, 0). We start by analyzing if there is x ∈ XP such that v · x = 0.
Then, x2 = x3 = 0. Moreover, the system of restrictions becomes
−2x1 = 2
x1 + x4 = 2
x1 , x4 ≥ 0.
Hence, there is no vertex of XP by this hyperplane.
(5) v = (0, 1, 0, 1). We start by investigating if there is x ∈ XP such that
v · x = 0. Then, x2 = x4 = 0. Moreover, the system of restrictions becomes
−2x1 + x3 = 2
x1 = 2
x1 , x3 ≥ 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.3. SOLVED PROBLEMS AND EXERCISES 137
Thus, x = (2, 0, 6, 0) ∈ XP . Furthermore, v · y > 0 whenever y ∈ XP \ {x}.
Hence, x is a vertex of XP .
(6) v = (0, 0, 1, 1). We start by analyzing if there is x ∈ XP such that v · x = 0.
Then, x3 = x4 = 0. Moreover, the system of restrictions becomes
−2x1 + 2x2 = 2
x1 = 2
x1 , x2 ≥ 0.
Thus, x = (2, 3, 0, 0) ∈ XP . Furthermore, v · y > 0 whenever y ∈ XP \ {x}.
Hence, x is a vertex of XP .
The following result is known as Vertex Minimizer Theorem.
Theorem 5.1
Let P be a standard optimization problem. Assume that XP is non-empty and
that the objective map is bounded from below in XP . Then, there is a vertex
of XP in SP .
Proof:
By Theorem 4.1, under the assumptions, there is a basic admissible vector in
SP . So, by Proposition 5.16, there is a vertex of XP in SP . QED
5.3 Solved Problems and Exercises
Problem 5.1
Show that
dim {x ∈ Rn : x ≥ 0} = n.
Solution:
We start by showing that
A({x ∈ Rn : x ≥ 0}) = Rn .
By definition, A({x ∈ Rn : x ≥ 0}) ⊆ Rn . For the other inclusion let u ∈ Rn .
We consider two cases:
(1) u ≥ 0. Then, u ∈ {x ∈ Rn : x ≥ 0}. Hence, u ∈ A({x ∈ Rn : x ≥ 0}).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
138 CHAPTER 5. GEOMETRIC VIEW
(2) u 6≥ 0. Then,
u = (−1)u + 2u,
where u, u ∈ Rn are such that
(
−uj if uj < 0
uj =
uj otherwise
and (
0 if uj < 0
uj =
uj otherwise
for each j = 1, . . . , n. Since u ≥ 0 and u ≥ 0, we have
u, u ∈ {x ∈ Rn : x ≥ 0} ⊆ A({x ∈ Rn : x ≥ 0}).
Hence, u is an affine combination of elements of A({x ∈ Rn : x ≥ 0}). Thus,
u ∈ A({x ∈ Rn : x ≥ 0})
by Proposition 5.2.
Therefore, the dimension of A({x ∈ Rn : x ≥ 0}) is n because dim Rn is n, see
Example 4.8. Hence, dim {x ∈ Rn : x ≥ 0} = n. /
Problem 5.2
Consider the following standard optimization problem
min −2x1 − 5x2
x
−2x1 + 2x2 + x3 = 2
P =
x1 + x4 = 2
x ≥ 0.
Show that XP is a polytope.
Solution:
By Proposition 5.14, XP is a convex polyhedron. We now show that XP is
bounded. Let x ∈ XP . Then,
• x1 ≤ 2 since x1 + x4 = 2 and 0 ≤ x4 ;
• x4 ≤ 2 because x1 + x4 = 2 and 0 ≤ x1 ;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.3. SOLVED PROBLEMS AND EXERCISES 139
• x2 ≤ 3 since −2x1 + 2x2 + x3 = 2, 0 ≤ x1 ≤ 2, 0 ≤ x3 and x2 has the
greatest value when x1 is 2 and x3 is 0;
• x3 ≤ 6 because −2x1 + 2x2 + x3 = 2, 0 ≤ x1 ≤ 2, 0 ≤ x2 and x3 has the
greatest value when x1 is 2 and x2 is 0.
Therefore,
p
kxk = x21 + x22 + x23 + x24
√
≤ 22 + 32 + 62 + 22 .
So, XP is a polytope. /
Problem 5.3
Let P be a standard optimization problem and x ∈ XP . Then, x is a vertex of
XP if and only if for every y, z ∈ XP , x = y = z whenever there exists α ∈]0, 1[
such that x = αy + (1 − α)z.
Solution:
Using Proposition 5.16, x is a vertex of XP if and only if x is basic vector of
XP . On the other hand, by Proposition 4.7, x is a basic vector of XP if and
only if for every y, z ∈ XP , x = y = z whenever there exists α ∈]0, 1[ such that
x = αy + (1 − α)z. /
Problem 5.4
Let j ∈ N . Show, using the definition, that
{x ∈ Rn : xj = 0}
is a hyperplane.
Solution:
(1) {x ∈ Rn : xj = 0} is an affine subspace of (Rn , Θ); that is,
V = Θt ({x ∈ Rn : xj = 0}) = {x − t : x ∈ Rn and xj = 0}
is a subspace of Rn with t ∈ {x ∈ Rn : xj = 0}. Indeed:
(a) 0 ∈ V since t − t ∈ V .
(b) Assume that x − t, y − t ∈ V . Then, x, y ∈ Rn and xj = yj = tj = 0. Thus,
x + y − t ∈ Rn , (x + y − t)j = 0. Therefore, (x + y) − 2t ∈ V .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
140 CHAPTER 5. GEOMETRIC VIEW
(c) Assume that x − t ∈ V and α ∈ R. Then, xj = tj = 0 and x ∈ Rn . Then,
(αx)j = (αt)j = 0. Observe that
(αx + (1 − α)t)j = 0.
Therefore, αx + (1 − α)t − t = α(x − t) ∈ V .
(2) dim{x ∈ Rn : xj = 0} = n − 1. Consider the set
E = {e1 , . . . , ej−1 , ej+1 , . . . , en } ⊆ V,
where ek is as in Definition 1.20 for k = 1, . . . , j − 1, j + 1, . . . , n. Then, E is a
linearly independent set, by Proposition 1.10. Moreover,
X
x−t= (xk − tk )ek .
k∈{1,...,j−1,j+1,...,n}
Hence, E is a basis for V . So, dim V = n − 1. /
Problem 5.5
Let W ⊆ Rn be a non-empty set, t ∈ W and V a subspace of Rn such that
Θt (W ) ⊆ V . Show that there is an affine subspace U such that Θt (U ) = V
and W ⊆ U .
Solution:
Take
U = {v + t : v ∈ V }.
Then, it is immediate that Θt (U ) = V . Moreover, W ⊆ U as we show now.
Assume that w ∈ W . Then, Θt (w) = w − t ∈ V since Θt (W ) ⊆ V . Hence,
w ∈ U because w = (w − t) + t. /
Problem 5.6
Let W ⊆ Rn . Show that
Θt (A(W )) = spanRn (Θt (W ))
for every t ∈ W .
Solution:
Let t ∈ W .
(⊆) Assume that v ∈ Θt (A(W )). Then, v = u − t for some u in A(W ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.3. SOLVED PROBLEMS AND EXERCISES 141
Observe that u ∈ U 0 for every affine subspace U 0 such that W ⊆ U 0 . Hence,
v ∈ Θt (U 0 ) for every affine subspace U 0 such that W ⊆ U 0 . The thesis follows
immediately since there is, by Problem 5.5, an affine subspace U such that
Θt (U ) = spanRn (Θt (W )) and W ⊆ U , because Θt (W ) ⊆ spanRn (Θt (W )), by
definition.
(⊇) Observe that Θt (W ) ⊆ Θt (A(W )) since W ⊆ A(W ). Thus,
spanRn (Θt (W )) ⊆ Θt (A(W ))
since spanRn (Θt (W )) is the smallest subspace of Rn containing Θt (W ), by
definition. /
Problem 5.7
Show that the set of minimizers for a standard optimization problem is convex.
Solution:
Let P be a standard optimization problem. If SP = ∅ then SP is a convex set
(see Exercise 3.1). Otherwise, assume that s0 , s00 ∈ SP are minimizers. That
is, s0 , s00 ∈ XP and
cs0 = cs00 = min{cx : x ∈ XP }.
Take s = αs0 + (1 − α)s00 with α ∈ [0, 1]. Observe that s ∈ XP since XP is a
convex set (see Proposition 5.14). On the other hand, for every x ∈ XP ,
cs = c(αs0 + (1 − α)s00 )
= αcs0 + (1 − α)cs00
≤ αcx + (1 − α)cx
= cx
using the fact that α and 1 − α are non-negative. Therefore, s ∈ SP . /
Exercise 5.8
Show that U ⊆ Rn is convex if and only if U is the set of all linear combinations
of the form
Xk
αj uj ,
j=1
where k ∈ N such that uj ∈ U , αj ∈ R+
0 for j = 1, . . . , k and
k
X
αj = 1.
j=1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
142 CHAPTER 5. GEOMETRIC VIEW
Exercise 5.9
Let X be a convex polyhedron in Rn . Show that F ⊆ X is a face of X by
hyperplane {x ∈ Rn : v · x = r} if and only if
(
v · x = r for every x ∈ F
v · x < r for every x ∈ (X \ F ).
Exercise 5.10
Consider the canonical optimization problem
max 2x1 + 5x2
x
4
− x1 − x2 ≤ −2
P = 3
2x 1 + x2 ≤ 10
x ≥ 0.
Find the vertices of XCS(P ) .
5.4 Relevant Background
We start by establishing some results about affine spaces. For more details,
see [60, 8, 63, 6, 54, 9, 59, 33].
Definition 5.12
Let V be a vector space over a field with characteristic 0.1 An affine space over
V is a pair
(A, Θ),
where A is a set and Θ : A × A → V is a map such that:
• the map Θa1 : A → V , with Θa1 (a) = Θ(a1 , a), is a bijection, for every
a1 ∈ A;
• Θ(a1 , a3 ) = Θ(a1 , a2 ) + Θ(a2 , a3 ), for every a1 , a2 , a3 ∈ A (known as
Chasles’ Relation).
The vector space V is the direction of A and each a ∈ A is called a point.
We now provide some useful identities.
1 See [47] for the definition of characteristic of a field. For instance, R and C have charac-
teristic 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.4. RELEVANT BACKGROUND 143
Proposition 5.18
Let (A, Θ) be an affine space over V . Then,
Θ(a, a) = 0 and Θ(a, b) = −Θ(b, a).
Proof:
Both equalities are consequence of Chasles’ Relation. Indeed, observe that
Θ(a, a) = Θ(a, a) + Θ(a, a).
So, Θ(a, a) is the null vector. Note also that
Θ(a, b) + Θ(b, a) = Θ(a, a).
Since Θ(a, a) = 0, then Θ(a, b) + Θ(b, a) = 0. QED
The following result is known as the Parallelogram Rule.
Proposition 5.19
Let (A, Θ) be an affine space over V . Then,
Θ(a, b) = Θ(a0 , b0 ) iff Θ(a, a0 ) = Θ(b, b0 ).
Proof:
Using Chasles’ Relation, we have that
Θ(a, b) = Θ(a, a0 ) + Θ(a0 , b)
and
Θ(a0 , b) = Θ(a0 , b0 ) + Θ(b0 , b).
Thus,
Θ(a, b) = Θ(a, a0 ) + Θ(a0 , b0 ) + Θ(b0 , b);
that is,
Θ(a, b) − Θ(a0 , b0 ) = Θ(a, a0 ) + Θ(b0 , b).
Therefore, by Proposition 5.18,
Θ(a, b) − Θ(a0 , b0 ) = Θ(a, a0 ) − Θ(b, b0 ).
So, the equivalence holds. QED
Affine subspaces play an important role in optimization.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
144 CHAPTER 5. GEOMETRIC VIEW
Definition 5.13
Let (A, Θ) be an affine space over V and U ⊆ A. We say that U is an affine
subspace of (A, Θ) if it is either empty or there is t ∈ U such that
Θt (U )
is a subspace of V .
Example 5.9
Observe that any singleton subset {u} of Rn is an affine subspace since
Θu ({u}) = {0}R
and {0}R is a subspace by Example 1.24.
The following remark points out that it is possible to generate an affine
subspace from a subspace and a vector.
Remark 5.2
Let V be a subspace of Rn and t ∈ Rn . Then,
Θ−1
t (V ) = {v + t : v ∈ V }
is an affine subspace of (Rn , Θ).
We now show that the class of affine subspaces of an affine space is closed
under intersection.
Proposition 5.20
Let {Uj }j∈J be a family of affine subspaces of the affine space (A, Θ) over a
vector space V . Then,
\
U= Uj
j∈J
is also an affine subspace of (A, Θ). Moreover. if U 6= ∅ then
\
Θu (U ) = Θu (Uj )
j∈J
for each u ∈ U .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
5.4. RELEVANT BACKGROUND 145
Proof:
Let \
U= Uj .
j∈J
We have two cases:
(1) U = ∅. Then, U is an affine subspace of (A, Θ) by definition.
(2) U 6= ∅. Let u ∈ U . Then, u ∈ Uj for every j ∈ J. Then,
Θu (Uj )
is a subspace of V . Hence, \
Θu (Uj )
j∈J
is a subspace of V (see Proposition 1.6). It remains to prove that
\
Θu (U ) = Θu (Uj ).
j∈J
(⊆) Since U ⊆ Uj for every j ∈ J, then
Θu (U ) ⊆ Θu (Uj ), for every j ∈ J.
So, \
Θu (U ) ⊆ Θu (Uj ).
j∈J
T
(⊇) Let v ∈ j∈J Θu (Uj ). Then,
v ∈ Θu (Uj ), for every j ∈ J.
That is, for each j ∈ J, there is uj ∈ Uj such that Θu (uj ) = v. Because
Θu : A → V is injective, uj1 = uj2 = u0 for every j1 , j2 ∈ J. So, u0 ∈ U .
Therefore, v ∈ Θu (U ). QED
On the other hand, the union of affine subspaces is not always an affine
subspace.
Example 5.10
Consider the affine subspaces H1 and H2 introduced in Example 5.6. We
showed that H1 ∪ H2 is not closed under convex combinations. Therefore,
H1 ∪ H2 is not closed under affine combinations and so is not an affine subspace
by Proposition 5.2.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
146 CHAPTER 5. GEOMETRIC VIEW
We now concentrate on defining the dimension of affine spaces and sub-
spaces.
Definition 5.14
The dimension of an affine space (A, Θ) over V is the dimension of V .
Definition 5.15
Let U be an affine subspace of the affine space (A, Θ) over V . The dimension
of U , denoted by
dim U
is the dimension of the subspace Θt (U ) of V when U 6= ∅ and is set to be −1
otherwise.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 6
Duality
In this chapter, we introduce the notion of dual of a linear optimization problem
(the first duality statement in optimization is attributed to John von Neumann
see [21, 62, 61, 27, 23, 26]). The objective is to show that duality provides
yet a new technique for finding optimizers of the original problem. We start
by providing two main results, the Weak and the Strong Duality Theorems.
Then, we concentrate on slacks for the linear optimization problem and prove
the Slack Complementarity Theorem. Moreover, a necessary and sufficient
condition for the existence (and calculation) of the optimizers is stated, as well
as, the Equilibrium Theorem. Finally, we prove a sufficient condition for the
existence of a unique minimizer of a standard optimization problem.
6.1 Weak and Strong Duality
Towards defining the dual of a canonical optimization problem, we parameterize
the restriction Ax ≤ b and move it to the objective map as a penalty. This goal
is attained by extending the Lagrange multiplier technique used in Analysis
(see [31]) for optimizing a map under equality constraints (see [12]).
Definition 6.1
Let P = (A, b, c) be a canonical optimization problem where A is an m × n-
matrix and y ∈ Rm . The parameterized canonical optimization problem over P
and y is defined as follows:
(
max cx + y T (b − Ax)
x
x ≥ 0.
147
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
148 CHAPTER 6. DUALITY
Proposition 6.1
Let P = (A, b, c) be a canonical optimization problem where A is an m × n-
matrix and y ∈ Rm . For every z ∈ XP , we have
if y ≥ 0 then cz ≤ max cx + y T (b − Ax).
x≥0
Proof:
Let z ∈ XP . Hence,
cz ≤ cz + y T (b − Az) ≤ max cx + y T (b − Ax)
x≥0
since y T (b − Az) ≥ 0, because b − Az ≥ 0 and y ≥ 0. QED
Thus, each y ≥ 0 leads to an upper bound for the maximum value of the
objective map of P . The main result in duality theory states that the minimum
value over y of
max cx + y T (b − Ax),
x≥0
under the condition y ≥ 0, is the maximum value of the objective map of P .
Notation 6.1
Let P = (A, b, c) be a canonical optimization problem. We denote by
P
the optimization problem
min max cx + y T (b − Ax)
(
y x≥0
y ≥ 0.
Observe that although the objective map of P involves variables x and y,
the optimization is done on y (x is a bound variable).
Proposition 6.2
Let P = (A, b, c) be a canonical optimization problem and Q the linear opti-
mization problem
T
min b y
y
AT y ≥ cT
y ≥ 0.
Then, the set of optimizers of Q is equal to the set of optimizers of P .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.1. WEAK AND STRONG DUALITY 149
Proof:
(1) SQ ⊆ SP . Let r ∈ SQ . Then, AT r ≥ cT , r ≥ 0 and
bT r ≤ bT y
for every y ∈ XQ . Let y ≥ 0. Consider two cases:
(a) AT y ≥ cT . Thus,
max cx + rT (b − Ax) = bT r + max (c − rT A)x
x≥0 x≥0
= bT r
≤ bT y
= bT y + max (c − y T A)x
x≥0
= max cx + y T (b − Ax).
x≥0
(b) AT y 6≥ cT . Then, there is j = 1, . . . , n such that
cj − (AT y)j > 0.
Therefore,
max (c − y T A)x = +∞.
x≥0
Then,
max cx + y T (b − Ax) = bT y + max (c − y T A)x = +∞.
x≥0 x≥0
Thus,
max cx + rT (b − Ax) ≤ max cx + y T (b − Ax).
x≥0 x≥0
(2) SP ⊆ SQ . Let r ∈ SP . Then, r ≥ 0 and
max cx + rT (b − Ax) ≤ max cx + y T (b − Ax)
x≥0 x≥0
for every y ≥ 0. Let y ∈ XQ . Hence, y ≥ 0 and AT y ≥ cT . We now prove that
AT r ≥ cT .
Suppose, by contradiction, that AT r 6≥ cT . Then, there is j = 1, . . . , n such
that
cj − (AT r)j > 0.
Therefore,
max (c − rT A)x = +∞.
x≥0
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
150 CHAPTER 6. DUALITY
Thus,
max cx + rT (b − Ax) = +∞.
x≥0
Hence,
bT y + max (c − y T A)x = max cx + y T (b − Ax) = +∞.
x≥0 x≥0
Therefore,
c − y T A 6≤ 0
which contradicts the hypothesis that y ∈ XQ .
Thus,
bT r = bT r + max (c − rT A)x
x≥0
= max cx + rT (b − Ax)
x≥0
≤ max cx + y T (b − Ax)
x≥0
= bT y + max (c − y T A)x
x≥0
= bT y.
So, r ∈ SQ . QED
Notation 6.2
In the sequel, we refer to the problem Q introduced in Proposition 6.2, by the
dual of P . The set of admissible vectors for Q is
YQ
rather that XQ and the set of optimizers for Q is
RQ
rather that SQ .
Example 6.1
Consider the canonical optimization problem P introduced in Example 1.12.
The dual problem Q of P is as follows:
min 6y1 + 6y2
y
3y − y ≥ 2
1 2
−y1 + 3y2 ≥ 1
y ≥ 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.1. WEAK AND STRONG DUALITY 151
Exercise 6.1
Let Q be the dual of a canonical optimization problem. Show that if YQ 6= ∅
and the objective map of Q is bounded from below in YQ then RQ 6= ∅.
The next result, called the Canonical Weak Duality Theorem, shows that
the value of the objective map of Q for each vector in YQ is an upper bound of
the value of the objective map of P for any vector in XP .
Theorem 6.1
Let P be a canonical optimization problem and Q the dual of P . Then,
cx ≤ bT y
for every x ∈ XP and y ∈ YQ .
Proof:
Let y ∈ YQ and x ∈ XP . Then, y ≥ 0 and cT ≤ AT y. Hence, c ≤ y T A. On
the other hand, x ≥ 0 and Ax ≤ b. Thus, cx ≤ y T Ax and y T Ax ≤ y T b. Hence,
cx ≤ y T b. Thus, cx ≤ bT y, since bT y ∈ R. QED
Proposition 6.3
Let P be a canonical optimization problem and Q the dual of P . Then,
• if YQ 6= ∅ then the objective map of P in XP has an upper bound;
• if XP 6= ∅ then the objective map of Q in YQ has a lower bound.
Proof:
We only prove the first statement since the other one is similar. Assume that
YQ 6= ∅. There are two cases to consider:
(1) XP = ∅, in which case {cx : x ∈ XP } = ∅. Hence, each real number is an
upper bound of that set.
(2) XP 6= ∅, in which case the thesis follows from Theorem 6.1. QED
The next result, called the Optimality Criterion, provides a sufficient con-
dition for the existence of optimizers.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
152 CHAPTER 6. DUALITY
Proposition 6.4
Let P be a canonical optimization problem and Q the dual of P . If x ∈ XP ,
y ∈ YQ and cx = bT y then x ∈ SP and y ∈ RQ .
Proof:
Let x ∈ XP and y ∈ YQ be such that cx = bT y. Then, by Theorem 6.1, the
following hold:
1. cx ≤ bT y 0 for every y 0 ∈ YQ ;
2. cx0 ≤ bT y for every x0 ∈ XP .
In other words:
1. bT y ≤ bT y 0 for every y 0 ∈ YQ ;
2. cx0 ≤ cx for every x0 ∈ XP .
Then, y ∈ RQ and x ∈ SP . QED
Example 6.2
Consider the canonical optimization problem P introduced in Example 1.12.
Recall the dual problem Q of P in Example 6.1. It is immediate that (3, 3) ∈
XP . We want to use duality to conclude that (3, 3) ∈ SP . To find an admissible
vector of Q, observe that the system of linear equations:
(
3y1 − y2 = 2
−y1 + 3y2 = 1
7 5 7 5
has , as the solution. Hence, , ∈ YQ since this vector is also
8 8 8 8
non-negative. Moreover,
" # " 7 #
3 8
c = 9 = bT 5 .
3 8
7 5
Therefore, (3, 3) ∈ SP and , ∈ RQ by Proposition 6.4.
8 8
The following result, called the Lemma of Existence of Dual , states that
there is always an admissible vector y of the dual problem with objective value
as close as envisaged of cs when s is an optimizer of the canonical optimization
problem.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.1. WEAK AND STRONG DUALITY 153
Proposition 6.5
Let P be a canonical optimization problem, Q the dual of P and s ∈ SP . Then,
for each ε > 0 there exists y ∈ YQ such that
cs ≤ bT y < (cs + ε).
Proof:
Let ε > 0. Given s ∈ SP , observe that the system
(
Ax ≤ b
cx ≥ cs
has a non-negative solution (the vector s), whereas the system
(
Ax ≤ b
cx ≥ cs + ε
has no non-negative solution, since cs is the maximum value of the objective
map x 7→ cx in XP . Let A be the (m + 1) × n-matrix
" #
A
−c
ε
and b ∈ Rm+1 the vector " #
b
.
−cs − ε
Denote by P ε the canonical optimization problem
max
c0 x
x
ε
Ax ≤ b
x ≥ 0,
where c0 is an arbitrary row vector in Rn . Observe that
XP ε = ∅.
Then, by Proposition 3.11, we conclude that there are v ∈ (R+
0)
m
and z ∈ R+
0
such that:
• (v, z) 6= 0;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
154 CHAPTER 6. DUALITY
T
• (v, z) A ≥ 0T , and so AT v ≥ zcT ;
T ε
• (v, z) b < 0, and so bT v < z(cs + ε).
On the other hand, the problem
max c00 x
x
" #
b
Ax ≤
−cs
x ≥ 0
has a non-empty set of admissible vectors. So, by Proposition 3.11, we have
" #
T b
(v, z) ≥ 0.
−cs
Hence,
bT v ≥ zcs.
Thus,
zcs ≤ bT v < z(cs + ε).
Therefore, z > 0. Take
1
y= v.
z
Then,
• y ∈ YQ , since
– y ≥ 0, because v ≥ 0 and z > 0;
– AT y ≥ cT , noticing that AT v ≥ zcT .
• cs ≤ bT y < (cs + ε), since zcs ≤ bT v < z(cs + ε).
Hence, for each > 0, we are able to find a y ∈ YQ in the conditions required
by the statement. QED
As a direct consequence of the above result, if SP 6= ∅ then YQ 6= ∅.
Proposition 6.6
Let P be a canonical optimization problem and Q the dual of P . Then,
• if XP 6= ∅ and YQ = ∅ then the objective map of P in XP has no upper
bound;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.1. WEAK AND STRONG DUALITY 155
• if XP = ∅ and YQ 6= ∅ then the objective map of Q in YQ has no lower
bound.
Proof:
We only show the first statement since the proof of the second statement is sim-
ilar. Let XP 6= ∅ and YQ = ∅. Assume, by contradiction, that there is an upper
bound of the objective map of P in XP . Then, from Theorem 4.2, we conclude
that SP 6= ∅. Therefore, invoking Proposition 6.5, YQ 6= ∅, contradicting the
hypothesis. QED
The following result is known as the Canonical Strong Duality Theorem.
Theorem 6.2
Let P be a canonical optimization problem and Q the dual of P . Then,
1. if XP = ∅ and YQ = ∅ then SP = ∅ and RQ = ∅;
2. if XP 6= ∅ and YQ = ∅ then SP = ∅ and RQ = ∅ and the objective map
of P in XP has no upper bound;
3. if XP = ∅ and YQ 6= ∅ then SP = ∅ and RQ = ∅ and the objective map
of Q in YQ has no lower bound;
4. if XP 6= ∅ and YQ 6= ∅ then SP 6= ∅ and RQ 6= ∅ and
cs = bT r
for every s ∈ SP and r ∈ RQ .
Proof:
(1) It is immediate since SP ⊆ XP and RQ ⊆ YQ .
(2) In this case, Proposition 6.6 implies that the objective map of P in XP has
no upper bound. Therefore, SP = ∅. On the other hand, RQ = ∅ because YQ
is empty.
(3) The argument is similar to that of the previous case.
(4) Since YQ 6= ∅, by Theorem 6.1, it follows that the objective map of P in
XP has an upper bound. Hence, by Theorem 4.2, we conclude that SP 6= ∅.
Similarly, we can show that RQ 6= ∅. It remains to prove that cs = bT r for
every s ∈ SP and r ∈ RQ .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
156 CHAPTER 6. DUALITY
Let s and r be arbitrary elements of SP and RQ , respectively. By Theorem 6.1,
cs ≤ bT r.
We prove that
cs ≥ bT r
by contradiction. Indeed, assume that cs < bT r. Take ε = bT r − cs > 0. Then,
by Proposition 6.5, there exists y ∈ YQ such that bT y < cs + ε. That is, there
exists y ∈ YQ such that
bT y < bT r,
contradicting the hypothesis that r is a minimizer of Q. QED
Although the previous results about duality were established for canonical
optimization problems, they can be adapted to standard optimization prob-
lems.
Exercise 6.2
Show that the dual of a standard optimization problem is the linear optimization
problem
max bT y
(
y
AT y ≤ cT .
Example 6.3
Consider the standard optimization problem P introduced in Example 1.13.
The dual problem Q of P is as follows:
max 6y1 + 6y2
(y1 ,y2 )
3y1 − y2 ≤ −2
−y1 + 3y2 ≤ −1
y1 ≤ 0
y ≤ 0.
2
The following exercise establishes the Standard Weak Duality Theorem.
Exercise 6.3
Let P be a standard optimization problem and Q the dual of P . Show that
bT y ≤ cx
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.1. WEAK AND STRONG DUALITY 157
for every x ∈ XP and y ∈ YQ .
Exercise 6.4
Let P be a standard optimization problem and Q the dual of P . Show that for
every x ∈ XP and y ∈ YQ , if
bT y = cx
then x ∈ SP and y ∈ RQ .
Example 6.4
Consider the standard optimization problem P introduced in Example 1.13 and
recall the dual problem Q of P in Example 6.3. We want to use Exercise 6.4 to
conclude that (3, 3, 0, 0) ∈ SP . Recall that (3, 3, 0, 0) ∈ XP (see Example 4.6).
To find an admissible vector of Q, observe that the system of linear equations:
(
3y1 − y2 = −2
−y1 + 3y2 = −1
7 5 7 5
has − , − as the solution. Hence, − , − ∈ YQ , since this vector is
8 8 8 8
also non-negative. Moreover,
3 " 7 #
3 T
−8
0 = −9 = b
c .
− 58
0
7 5
Hence, (3, 3, 0, 0) ∈ SP and − , − ∈ RQ , by Exercise 6.4.
8 8
The following exercise establishes the Standard Strong Duality Theorem.
Exercise 6.5
Let P be a standard optimization problem and Q the dual of P . Show that
1. if XP = ∅ and YQ = ∅ then SP = ∅ and RQ = ∅;
2. if XP 6= ∅ and YQ = ∅ then SP = ∅ and RQ = ∅ and the objective map
of P in XP has no lower bound;
3. if XP = ∅ and YQ 6= ∅ then SP = ∅ and RQ = ∅ and the objective map
of Q in YQ has no upper bound;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
158 CHAPTER 6. DUALITY
4. if XP 6= ∅ and YQ 6= ∅ then SP 6= ∅ and RQ 6= ∅ and
cs = bT r
for every s ∈ SP and r ∈ RQ .
Exercise 6.6
Show that the dual of the linear optimization problem P
(
max 0x
x
Ax ≤ b
is the problem Q defined as follows:
min bT y
y
AT y = 0
y ≥ 0.
Moreover, show that bT y ≥ 0 for every y ∈ YQ . Finally, show that
1. if XP = ∅ and YQ = ∅ then SP = ∅ and RQ = ∅;
2. if XP 6= ∅ and YQ = ∅ then SP = ∅ and RQ = ∅ and the objective map
of P in XP has no upper bound;
3. if XP = ∅ and YQ 6= ∅ then SP = ∅ and RQ = ∅ and the objective map
of Q in YQ has no lower bound;
4. if XP 6= ∅ and YQ 6= ∅ then SP 6= ∅ and RQ 6= ∅ and
cs = bT r
for every s ∈ SP and r ∈ RQ .
6.2 Complementarity
The objective of this section is to provide techniques for deciding whether or
not an admissible vector is an optimizer, using duality and slack variables.
Definition 6.2
Let P be a canonical optimization problem and Q the dual of P . Then,
d = x 7→ b − Ax : Rn → Rm and e = y 7→ AT y − cT : Rm → Rn
are called the slack maps for P and Q, respectively.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.2. COMPLEMENTARITY 159
Note that d(x) ≥ 0 whenever x ∈ XP and e(y) ≥ 0 whenever y ∈ YQ .
Remark 6.1
Observe that when x ∈ XP and d(x)i = 0, then the i-th line of A is active in
x (see Definition 3.5).
Notation 6.3
When x ∈ XP , d(x) is the slack of x. On the other hand, when y ∈ YQ , e(y)
is the slack of y.
The next result is known as the Slack Lemma.
Proposition 6.7
Let P be a canonical optimization problem, Q the dual of P , x ∈ XP and
y ∈ YQ . Then,
• d(x)T y + e(y)T x = bT y − cx;
• d(x)T y ≥ 0;
• e(y)T x ≥ 0.
Proof:
(1) d(x)T y + e(y)T x = bT y − cx since
d(x)T y + e(y)T x = d(x)T y + xT e(y)
= (b − Ax)T y + xT (AT y − cT )
= bT y − xT AT y + xT AT y − xT cT
= bT y − xT cT
= bT y − cx.
(2) d(x)T y ≥ 0 and e(y)T x ≥ 0, since x, y, d(x), e(y) ≥ 0. QED
The following result establishes one of the implications of the Slack Com-
plementarity Theorem.
Proposition 6.8
Let P be a canonical optimization problem, Q the dual of P , s ∈ SP and
r ∈ RQ . Then,
d(s)T r = 0 and e(r)T s = 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
160 CHAPTER 6. DUALITY
Proof:
Observe that, by Theorem 6.2, cs = bT r. So, by Proposition 6.7, the thesis
follows. QED
This result implies that if the i-th component of r is non-zero then the i-th
line of A is active in s; that is, d(s)i = 0 (see Remark 6.1). Similarly for e(r).
The following characterization of optimizers is known as the Slack Comple-
mentarity Theorem.
Theorem 6.3
Let P be a canonical optimization problem, Q the dual of P , x ∈ XP and
y ∈ YQ . Then,
x ∈ SP and y ∈ RQ if and only if d(x)T y = 0 and e(y)T x = 0.
Proof:
(1) Assume that x ∈ SP and y ∈ RQ . Then, by Proposition 6.8, d(x)T y = 0
and e(y)T x = 0.
(2) Assume that d(x)T y = 0 and e(y)T x = 0. Then, by Proposition 6.7,
cx = bT y. Thus, by Proposition 6.4, x ∈ SP and y ∈ RQ . QED
Theorem 6.3 gives a way to test whether or not x ∈ XP is an optimizer.
The idea is to solve the system
(
d(x)T y = 0
e(y)T x = 0.
Then, if there is a solution y such that y ∈ YQ , we conclude that x ∈ SP and
y ∈ RQ .
Example 6.5
Consider the canonical optimization problem P in Example 1.12 and the dual
Q of P in Example 6.1. We check whether or not (3, 3) ∈ SP . Observe that
d(3, 3) = 0. On the other hand, the system
3 3
e(y)T
= 3y1 − y2 − 2 −y1 + 3y2 − 1 =0
3 3
implies that
3
(∗) y1 = −y2 + .
2
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.3. EQUILIBRIUM 161
On the other hand, y should be in YQ . Hence, substituting y1 by (*) in
3y1 − y2 ≥ 1,
we obtain the following inequality
7
y2 ≤ .
8
Moreover, substituting y1 by (*) in
−y1 + 3y2 ≥ 2,
we obtain the following inequality
7
y2 ≥ .
8
Therefore, we conclude that
7
y2 = .
8
Thus, using (*) again we get
5
y1 = .
8
5 7
So, by Theorem 6.3, (3, 3) ∈ SP and , ∈ RQ .
8 8
Exercise 6.7
Consider the following optimization problem:
max 2x1 + x2
x
−x1 + x2 ≤ 1
x1 + x2 ≤ 3
x1 − x2 ≤ 1
x ≥ 0.
Show, using duality, that (2, 1) is an optimizer and that (1, 2) is not an opti-
mizer.
6.3 Equilibrium
The objective of this section is to provide another way to check whether or
not an admissible vector is an optimizer. Moreover, we address the problem of
knowing if there is a unique optimizer. Recall Notation 4.3.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
162 CHAPTER 6. DUALITY
Definition 6.3
Let P be a standard optimization problem, Q the dual of P , x ∈ Rn and
y ∈ Rm . The system of equations
(APx )T y = cT
Px
is called the Equilibrium Condition for x and y.
The following result, known as the Equilibrium Theorem, relates optimizers
and the Equilibrium Condition.
Theorem 6.4
Let P be a standard optimization problem, Q the dual of P and x ∈ XP . Then,
the following statements hold:
• if x ∈ SP then RQ 6= ∅ and the Equilibrium Condition for x and y holds
for every y ∈ RQ .
• if there exists y ∈ YQ such that the Equilibrium Condition holds for x
and y then x ∈ SP .
Proof:
(1) Assume that x ∈ SP . Then, SP 6= ∅. Thus, by Exercise 6.5, RQ 6= ∅. Let
y ∈ RQ . Using, Exercise 6.5, bT y = cx. That is,
cx − xT AT y = 0.
Therefore,
n
X m
X
xj (cj − yi aij ) = 0.
j=1 i=1
Hence,
X m
X
(†) xj (cj − yi aij ) = 0.
j∈Px i=1
On the other hand,
m
X
(cj − yi aij ) ≥ 0, for j ∈ Px
i=1
since y ∈ YQ . Thus, from (†), we conclude that
m
X
xj (cj − yi aij ) = 0, for j ∈ Px .
i=1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.3. EQUILIBRIUM 163
Furthermore,
m
X
(cj − yi aij ) = 0, for j ∈ Px .
i=1
Hence,
X m
X
(‡) (cj − yi aij ) = 0.
j∈Px i=1
But (‡) is the Equilibrium Condition for x and y.
Pm
(2) Let y ∈ YQ be such that, for every j ∈ N , if xj > 0, then i=1 yi aij = cj .
Then, for every j ∈ N
Xm
xj (cj − yi aij ) = 0.
i=1
Hence, similarly to the proof of (1), bT y = cx. Therefore, by Exercise 6.4,
x ∈ SP . QED
Exercise 6.8
Let P be a standard optimization problem, Q the dual of P and x ∈ XP . Show
that
• if RQ 6= ∅ and the Equilibrium Condition for x and y holds for every
y ∈ RQ then x ∈ SP ;
• if x ∈ SP then there exists y ∈ YQ such that the Equilibrium Condition
for x and y holds.
Theorem 6.4 provides a new way to conclude that a given admissible vector
x is an optimizer. The idea is to solve the system of equilibrium equations
(APx )T y = cT
Px
and after that check whether or not the solution is an admissible vector of the
dual problem.
Example 6.6
Consider the standard optimization problem P in Example 1.13 and its dual Q
described in Example 6.3. Take the vector (3, 3, 0, 0) ∈ XP (see Example 4.6).
Then, P(3,3,0,0) = {1, 2} and the Equilibrium Condition is as follows:
3 −1 −2
y= .
−1 3 −1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
164 CHAPTER 6. DUALITY
The solution of this system of equations is
7 5
− ,− .
8 8
Moreover, this vector is admissible for Q. Indeed, the system of inequalities
3 −1 −2
7
−1 3 − 85 ≤ −1
1 0 −8 0
0 1 0
is satisfied. Hence, by Proposition 6.4, the vector (3, 3, 0, 0) is in SP .
The following result is known as the Uniqueness Theorem and is applied to
non-degenerate problems (see Definition 4.4).
Theorem 6.5
Let P be a non-degenerate standard optimization problem, Q the dual of P ,
s ∈ SP and r ∈ RQ . Assume that:
• s is basic;
• rT a•j < cj for each j ∈ (N \ Ps ).
Then, SP = {s} and RQ = {r}.
Proof:
(1) |SP | = 1. Assume, by contradiction, that there is s0 ∈ SP such that
s0 6= s.
Take v = s0 − s. Then,
Av = A(s0 − s) = As0 − As = b − b = 0.
Hence, X X X
(rT a•j )vj + (rT a•j )vj = (rT a•j )vj
j∈Ps j∈(N \Ps ) j∈N
= rT (Av)
= 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.3. EQUILIBRIUM 165
Observe that, for every j ∈ Ps , by Proposition 6.4,
rT a•j = cj
since s ∈ SP and r ∈ RQ . Therefore,
X X
cj vj + (rT a•j )vj = 0.
j∈Ps j∈(N \Ps )
On the other hand, for every j ∈ (N \ Ps ), we have:
• rT a•j < cj , by hypothesis
• vj = s0j ≥ 0, since sj = 0 and s0 ≥ 0.
Thus, X X
cj vj + cj vj ≥ 0.
j∈Ps j∈(N \Ps )
Furthermore, as we show in (*), there exists j ∈ (N \ Ps ) such that vj > 0.
Hence, it follows that
X X
cj vj + cj vj > 0.
j∈Ps j∈(N \Ps )
So,
cv > 0,
0
contradicting s, s ∈ SP .
(*) We now show that there exists j ∈ (N \ Ps ) such that vj > 0. Assume, by
contradiction, that
(†) vj = 0
for every j ∈ (N \ Ps ). Since the problem is non-degenerate and s is basic, by
Proposition 4.9,
|Ps | = m.
Therefore, by Proposition 4.6,
APs is a basis for s.
Thus, APs is a nonsingular m × m-matrix. On the other hand, from Av = 0, it
follows that
APs vPs + AN \Ps vN \Ps = 0.
Since vN \Ps = 0 by (†), then
APs vPs = 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
166 CHAPTER 6. DUALITY
Therefore, vPs = 0 because APs has a unique solution. Hence, v = 0 contra-
dicting s0 6= s.
(2) |RQ | = 1. By Proposition 6.4, there is r ∈ RQ such that
(APs )T r = cT
Ps .
Since matrix APs is nonsingular, then (APs )T is also nonsingular. Hence, the
Equilibrium Condition
(APs )T y = cT
Ps
has a unique solution, which is r. Thus, by Proposition 6.4, |RQ | = 1. QED
Example 6.7
Consider the standard optimization problem P in Example 1.13 and its dual Q
described in Example 6.3. We know, from Example 6.4, that (3, 3, 0, 0) ∈ SP
and (− 87 , − 58 ) ∈ RQ . We now show that (3, 3, 0, 0) is the unique optimizer of P .
Since (3, 3, 0, 0) is basic (see Example 4.6), it remains to check, by Theorem 6.5,
that the system of inequalities:
7 5 1 0 0
− − <
8 8 0 1 0
is satisfied. Indeed, this is the case and so, by Theorem 6.5, we conclude that
(3, 3, 0, 0)
is the unique optimizer of P and that
7 5
− ,−
8 8
is the unique optimizer of Q.
6.4 Logic of Inequalities
As seen in Section 3.1, Farkas’ Lemma (see Proposition 3.6) plays a key role in
the theory of linear optimization. Therefore, it is worthwhile to analyze it in
more detail, including its pure variant (see Proposition 6.9). The pure variant
has a logical interpretation. Considering inequalities as formulas, the pure
variant corresponds to saying that a calculus for reasoning with inequalities is
sound and complete.
We start by stating the Pure Variant of Farkas’ Lemma.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.4. LOGIC OF INEQUALITIES 167
Proposition 6.9
Let P be the optimization problem introduced in Exercise 6.6 and Q the dual
of P . Then,
XP 6= ∅ if and only if for every w ≥ 0, if wT A = 0T then wT b ≥ 0.
Proof:
(→) Assume that XP 6= ∅. Let w ∈ (R+ 0)
m
be such that wT A = 0T . Observe
that w ∈ YQ . Then, by Exercise 6.6,
bT w ≥ 0.
(←) Note that y T b = bT y ≥ 0 for every y ∈ YQ . Thus, 0 ∈ RQ , since 0 ∈ YQ .
Then, by Exercise 6.6, XP 6= ∅. QED
To define the Logic of Inequalities, we start by introducing the set of for-
mulas.
Definition 6.4
An m × n-formula is a system of inequalities of the form Ax ≤ b, where b ∈ Rm
and A is an m × n-matrix.
We now introduce a calculus for deriving a formula from another formula.
Definition 6.5
Let A0 x ≤ b0 be an m0 × n-formula and Ax ≤ b be an m × n-formula. We say
that A0 x ≤ b0 is derived from Ax ≤ b, written
Ax ≤ b ` A0 x ≤ b0
if there exists an m0 × m-matrix D such that
• D has non-negative components;
• A0 = DA and b0 = Db.
Example 6.8
We show that from the formula
4 3 10
−3 x 1
1 ≤ 6
x2
0 −1 −4
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
168 CHAPTER 6. DUALITY
we can derive the formula
x1
1 0 ≤ 0 .
x2
We must find a matrix D with non-negative coefficients such that
4 3 10
D −3 1 = 1 0 and D 6 = 0 .
0 −1 −4
Observe that D has the form:
d1 d2 d3
where (d1 , d2 , d3 ) is a solution of the system of equations
4d1 − 3d2 + 0d3 = 1
3d1 + d2 − d3 = 0
10d + 6d − 4d = 0.
1 2 3
Since (d1 , d2 , d3 ) = (1, 1, 4) is a solution, the derivation holds.
Definition 6.6
An m × n-formula Ax ≤ b is inconsistent when
Ax ≤ b ` 0T x ≤ −1.
Otherwise, is consistent.
Exercise 6.9
Show that the following system is inconsistent:
2x1 − 5x2 ≥ 0
x1 ≤ −1
x2 ≥ 3.
We now present the semantics of the logic. For that, we define satisfaction
of a formula in Rn .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.4. LOGIC OF INEQUALITIES 169
Definition 6.7
An m × n-formula Ax ≤ b is satisfied by u ∈ Rn , written
u Ax ≤ b,
if Au ≤ b holds. In this case, we also say that u is a model of Ax ≤ b.
Definition 6.8
The m × n-formula Ax ≤ b is satisfiable if it has a model.
The following result is called the Soundness of the Logic of Inequalities
Theorem.
Theorem 6.6
Let A0 x ≤ b0 be an m0 × n-formula, Ax ≤ b an m × n-formula and u ∈ Rn .
Assume that u
(Ax ≤ b) and Ax ≤ b ` A0 x ≤ b0 . Then, u
A0 x ≤ b0 .
Proof:
The hypothesis Ax ≤ b ` A0 x ≤ b0 yields the existence of an m0 × m-matrix
D with non-negative components such that A0 = DA and b0 = Db. We show
that u
A0 x ≤ b0 . This amounts to prove that
(DAu)k ≤ (Db)k
for k = 1, . . . , m0 .
Observe that
m
X
• (DAu)k = dki (Au)i ;
i=1
m
X
• (Db)k = dki bi .
i=1
From the hypothesis u Ax ≤ b it follows that Au ≤ b; that is, (Au)i ≤ bi for
i = 1, . . . , m. Therefore, since each dij is non-negative, we conclude that
(DAu)k ≤ (Db)k
for k = 1, . . . , m0 . QED
The following result is called the Logical Variant of the Farkas’ Lemma.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
170 CHAPTER 6. DUALITY
Proposition 6.10
Let Ax ≤ b be an m × n-formula. Then, Ax ≤ b is satisfiable if and only if
Ax ≤ b is consistent.
Proof:
(→) Let u ∈ Rn be such that u
Ax ≤ b. Assume, by contradiction, that
Ax ≤ b is not consistent; that is,
(Ax ≤ b) ` (0T x ≤ −1).
Then, by Theorem 6.6, u (0T x ≤ −1), which is a contradiction.
(←) This implication is proved by contraposition. Assume that Ax ≤ b is not
satisfiable; that is, the system Ax ≤ b has no solution. Then, by Proposi-
tion 6.9, there exists w ∈ (R+0)
m
such that wT A = 0T and wT b < 0. Therefore,
T
formula (0 x ≤ −1) is derived from formula (Ax ≤ b) using matrix
1
wT .
|wT b|
Hence, by definition, (Ax ≤ b) is inconsistent. QED
6.5 Solved Problems and Exercises
Problem 6.1
Let P be the following canonical optimization problem
1
max x1 + x2
x 2
x1 + 2x2 ≤ 12
x1 ≤ 4
x ≥ 0.
(1) Find the dual problem Q of P .
(2) Show that SP 6= ∅ and RQ 6= ∅.
(3) Define the slack maps.
(4) Show using duality that (1, 1) ∈
/ SP .
(5) Show using duality that (3, 29 ) ∈ SP .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.5. SOLVED PROBLEMS AND EXERCISES 171
Solution:
Observe that
1 2 12 1
A= b= and c = 1 .
1 0 4 2
(1) The dual Q of P is:
min 12y1 + 4y2
y
bT y
min
y1 + y2 ≥ 1
y
2
AT y ≥ cT =
y ≥ 0
2y1 ≥ 1
y ≥ 0.
(2) Observe that XP 6= ∅ because, for example, (1, 1) ∈ XP and YQ 6= ∅ since,
for instance, ( 21 , 0) ∈ YQ . Then, by Theorem 6.2, SP 6= ∅ and RQ 6= ∅.
(3) The slack maps are defined as follows:
" #
12 − x1 − 2x2
d = x 7→ b − Ax = x 7→ : R2 → R2
4 − x1
and
1
y1 + y2 − 2
e = y 7→ AT y − cT = y 7→ : R2 → R2 .
2y1 − 1
(4) We start by showing that there is no y ∈ R2 such that
d(1, 1)T y = 0
" #
1
T
e(y) =0
1
y ∈ YQ .
Indeed, the system of equations
3y1 + y2 = 0
3
3y1 + y2 =
2
does not have a solution. Taking into account (2), let y ∈ RQ . Therefore, it is
not the case that
1
d(1, 1)T y = 0 and e(y)T = 0.
1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
172 CHAPTER 6. DUALITY
Hence, by Theorem 6.3, we conclude that (1, 1) ∈
/ SP .
(5) By Proposition 6.4, for proving that x = (3, 92 ) ∈ SP , it is enough to show
that there is y ∈ YQ such that
9
c 3, = bT y.
2
Take y = ( 21 , 0) ∈ YQ . Then,
bT y = 6 = cx.
Therefore, x and y are optimizers of P and Q, respectively. /
Problem 6.2
Let A be an m × n-matrix and b ∈ Rm . Show that
{x ∈ Rn : Ax ≤ b} = ∅ iff {y ∈ Rm : y T A = 0, y T b < 0, y ≥ 0} 6= ∅.
Solution:
(→) The result is shown by contraposition. Assume that
(†) {y ∈ Rm : y T A = 0, y T b < 0, y ≥ 0} = ∅.
Since v = 0 satisfies v T A = 0 and v ≥ 0, the linear optimization problem Q
min
yT b
y
yT A = 0
y ≥ 0
has v as an optimizer. Indeed, if y ∈ YQ , then y T A = 0 and y ≥ 0. Therefore,
by (†), y T b ≥ 0. By Exercise 6.6, we conclude that the problem
(
max 0x
x
Ax ≤ b
also has an optimizer. So, {x ∈ Rn : Ax ≤ b} 6= ∅.
(←) The result is shown by contradiction. Assume that
• {y ∈ Rm : y T A = 0, y T b < 0, y ≥ 0} 6= ∅;
• {x ∈ Rn : Ax ≤ b} 6= ∅.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.5. SOLVED PROBLEMS AND EXERCISES 173
Then, there exist y ∈ Rm and x ∈ Rn such that y T A = 0, y T b < 0, y ≥ 0 and
Ax ≤ b. Therefore,
0 = y T Ax ≤ y T b < 0 ,
which is a contradiction. /
Problem 6.3
Let P be the following canonical optimization problem
max
−cx
x
−Ax ≤ −b
x ≥ 0,
Q the dual of P , s ∈ SP and r ∈ RQ . Show that s ∈ SP 0 , where P 0 is the
problem (
min cx − rT Ax
x
x ≥ 0.
Solution:
Note that Q is the problem:
min
−bT y
y
−AT y ≥ −cT
y ≥ 0
taking into account Proposition 6.2. By Proposition 6.8,
(†) 0 = d(s)T r = (−b + As)T r
and
(‡) 0 = e(r)T s = (−AT r + cT )T s.
From (‡), we have
cs − rT As = (c − rT A)s = (−AT r + cT )T s = 0.
It remains to check that cs − rT As ≤ cx − rT Ax for every x ≥ 0; that is, we
have to check that
(c − rT A)x ≥ 0
for every x ≥ 0. Indeed, c − rT A ≥ 0, since r ∈ YQ .
Therefore, s ∈ SP 0 since s ≥ 0. /
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
174 CHAPTER 6. DUALITY
Exercise 6.10
Consider the optimization problem
max 2x1 + x2
x
−x1 + x2 ≤ 1
P = x1 + x2 ≤ 3
x1 − x2 ≤ 1
x ≥ 0.
Show, using the Equilibrium Condition, that (2, 1, 2, 0, 0) is in SCS(P ) .
Exercise 6.11
Consider the optimization problem
max x1 + 2x2
x
x1 + x2 ≤ 1
P =
2x1 + x2 ≤ 3
2
x ≥ 0.
Find the dual of P and show directly that both have the same optimizer.
Exercise 6.12
Prove the Canonical Variant of Farkas’ Lemma in Proposition 3.11, from the
Pure Variant of Farkas’ Lemma in Proposition 6.9.
Exercise 6.13
Consider the standard optimization problem
min 2x1 + x2
x
−x1 − x2 + x3 = −1
x1 + x2 + x4 = 3
P =
x1 − x2 + x5 = 1
−x1 + x2 + x6 = 1
x ≥ 0.
Let Q be the dual of P . Check whether or not SP and RQ are singleton sets.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
6.5. SOLVED PROBLEMS AND EXERCISES 175
Exercise 6.14
Let P be a canonical optimization problem. Relate the duals of P and CS(P ).
Exercise 6.15
Find the dual of the problem introduced in Example 1.1 and interpret it.
Exercise 6.16
Show that the dual of the dual problem is equivalent to the original problem.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
176 CHAPTER 6. DUALITY
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 7
Complexity
This chapter should be taken as a gentle introduction to complexity theory
with examples in linear optimization. In Section 7.1, we present the rigorous
definition of the decision problems associated to linear optimization and in Sec-
tion 7.2, we discuss the representation of vectors and matrices. In Section 7.3,
we prove that the standard decision problem is in NP. A deterministic algo-
rithm for getting an optimizer, based on a brute-force method, is discussed in
Section 7.4. The complexity of this algorithm is shown not to be polynomial.
Finally, in the relevant background section, we introduce the main notions and
techniques that are necessary for assessing the computational efficiency of an
algorithm.
7.1 Optimization Decision Problem
In this section, we present the concepts that are needed to formulate the op-
timization decision problem. As introduced in Section 7.6, a decision problem
is a map h : D → {0, 1}, where D is a denumerable set. So, we restrict our
problems to have rational components.
Notation 7.1
For each m, n ∈ N+ with m < n, we denote by
DSF
mn ,
the set of triples (A, b, c) such that:
• A is a non-empty m × n matrix with rational coefficients;
• c ∈ Qn ;
177
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
178 CHAPTER 7. COMPLEXITY
• b ∈ Qm such that bi 6= 0 for each i = 1, . . . , m.
Observe that each (A, b, c) ∈ DSF
mn can be seen, according to Notation 1.10,
as a standard optimization problem.
Notation 7.2
Furthermore, let
[
DSF = DSF
mn ;
m, n ∈ N+
m<n
[
Q∪ = Qn .
n∈N+
The following result tells us that there exists an optimizer with rational
components for a standard optimization problem provided that the problem
has rational numbers as parameters and a non-empty set of optimizers.
Proposition 7.1
Let P ∈ DSF
mn be a standard optimization problem such that SP 6= ∅. Then,
SP ∩ Qn 6= ∅.
Proof:
Since SP 6= ∅, then x 7→ cx is bounded from below in XP and XP 6= ∅. Hence,
by Theorem 4.1, there is a basic admissible vector s ∈ SP . Then, there is
B ⊂ N such that AB is nonsingular, sN \B = 0, and sB is the unique solution
of
AB = b.
Thus, by Cramer’s Rule (see Proposition 4.23), each component of s is given
by
det (AB )jb
sj = .
det AB
Clearly, each sj is in Q since det (AB )jb and det AB are rational numbers as
can be seen by the Leibniz’s Formula (see Proposition 4.19). QED
We now introduce the relevant optimization decision problem.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.2. REPRESENTATION 179
Definition 7.1
The Standard Optimization Decision Problem is the map
SDP : DSF → {0, 1}
such that
SDP(P ) = 1 if and only if SP 6= ∅
for every standard optimization problem P . Similarly for the Canonical Opti-
mization Decision Problem
CDP : DSF → {0, 1}.
7.2 Representation
Before analyzing the complexity of the standard optimization decision prob-
lem, we need to discuss the representation of vectors and matrices. Recall the
representation of natural and rational numbers in Section 7.6.
Notation 7.3
An n-dimensional vector w of rational numbers is represented by the following
string of bits
ŵ = ŵ1 010 · · · 010 ŵn ,
where ŵj is the encoding of the rational number wj , as explained in Remark 7.3,
for j = 1, . . . , n. Moreover, we choose 010 as the separator of the encodings of
the rational numbers. An m × n-matrix A of rational numbers is represented
by
 = â1• 011 · · · 011 âm• ,
where âi• is the representation of the i-th row of the matrix. Moreover,
(A, b, c) ∈ DSF is represented by
\
(A, b, c) = Â 100 b̂ 100 ĉ.
The reader may think of alternative ways of representing vectors and ma-
trices of arbitrary dimensions and ponder on the fact that any reasonably eco-
nomic representation would serve the purpose at hand.1
The representation adopted here is not minimal since it contains some ac-
ceptable redundancy that simplifies the computation of the dimension.
1 Redundancy is allowed as long as it does not grow faster then polynomial on the sum of
the sizes of the entries of the vector and of the matrix.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
180 CHAPTER 7. COMPLEXITY
input : (k, u)
1. scan k and û simultaneously until one ends;
2. if the scan of û ends before the scan of k then return 1;
3. if the scan of k ends before the scan of û then return 0;
4. return 1.
Figure 7.1: An algorithm for comparing sizes.
In the sequel, it is useful to know the complexity of comparing the size of
a vector with a given natural number presented in unary notation (see Defini-
tion 7.11).
Proposition 7.2
There is an algorithm for the decision problem
sizeGE : {1}∗ × Q∪ → {0, 1}
such that sizeGE(k, u) = 1 if and only if the size of k is greater than or equal
to the size in bits of u, which is polynomial on the first argument.
Proof:
It is immediate that the algorithm in Figure 7.1 computes sizeGE and is poly-
nomial on the first argument. QED
Notation 7.4
In the sequel, we denote the algorithm presented in Figure 7.1 by asizeGE .
For the next result, see Definition 7.5.
Proposition 7.3
There exists a polynomial algorithm for dim : Q∪ → N+ .
Observe that, for the chosen representation of vectors and matrices, for any
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.2. REPRESENTATION 181
non-empty matrix A and vector w,
dim
!
Xw
pwp = (3 + pwi p) −3
i=1
and !
m
X
pAp = (3 + pai• p) − 3,
i=1
where m is the number of rows of matrix A. Furthermore, for each (A, b, c) ∈
DSF ,
p(A, b, c)p = pAp + pbp + pcp + 6.
In the sequel, it also becomes handy to use
p(A, b)p = pAp + pbp + 3
and
p(A, c)p = pAp + pcp + 3.
The following result establishes a bound on the size of the determinant of
a square matrix with rational entries.
Proposition 7.4
For every non-empty square matrix A with rational entries,
p det Ap ≤ 3pAp4 .
Proof:
First, we show that, for every square matrix Z with integer entries represented
as rational numbers with unit denominator
p det Zp ≤ 2pZp2 .
To this end, we prove by induction on the number m of rows of Z that
p det Zp ≤ 2mpZp.
(Basis) In this case, det Z = z11 . Thus, p det Zp = pZp < 2pZp.
(Step) Recall Laplace’s Expansion Formula for j = 1 (see Definition 4.8),
m
X
det Z = (−1)i+1 zi1 det Zi,1 .
i=1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
182 CHAPTER 7. COMPLEXITY
By the induction hypothesis,
p det Zi,1 p ≤ 2(m − 1)pZi,1 p
and, so, since pzi1 p, pZi,1 p ≤ pZp and the size of a product is bounded by the sum
of the size of the factors,
p(−1)i+1 zi1 det Zi,1 p = pzi1 det Zi,1 p ≤ pZp + 2(m − 1)pZi,1 p ≤ 2mpZp − pZp.
Therefore, since the size of a sum of m integer terms is bounded by the sum of
m and a bound of the size of each term,
p det Zp ≤ 2mpZp − pZp + m ≤ 2mpZp − pZp + pZp = 2mpZp ≤ 2pZp2
taking into account that m ≤ pZp. For the general case of a rational matrix A,
first observe that
1
A= Q Z,
i,j aij
where Z is an integer matrix of the same dimension with each
Y
zij = aij aij
i,j
(see Notation 7.11). Clearly,
pZp ≤ pAp2
and, so,
p det Zp ≤ 2pAp4 .
aij p + p det Zp ≤ pAp + 2pAp4 ≤ 3pAp4 .
Q
Moreover, p det Ap ≤ p i,j QED
We now provide a bound on the size of an optimizer (with rational compo-
nents) for a standard optimization problem in DSF in terms of the size of the
problem.
Proposition 7.5
For every P = (A, b, c) ∈ DSF such that SP 6= ∅, there is s ∈ SP with rational
components such that
5
psp ≤ 9p(A, b)p .
Proof:
Recall from the proof of Proposition 7.1 that among the rational optimizers of
P there is a basic vector s with each component given by
det (AB )bj
sj = .
det AB
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.2. REPRESENTATION 183
Therefore, by Proposition 7.4,
4 4
psj p ≤ 3p(AB )bi p + 3pAB p4 ≤ 6p(A, b)p .
Finally, since n ≤ p(A, b)p,
4
psp ≤ 6p(A, b)p n + 3n
5
≤ 9p(A, b)p
as required. QED
The previous result is extended to general linear optimization problems
(recall Notation 1.6).
Proposition 7.6
Let P = (A0 , A00 , A000 , b0 , b00 , b00 , c) be a linear optimization problem with rational
coefficients in all components such that SP 6= ∅. Then, there is s ∈ SP with
rational components such that
psp ≤ d(pA0 p + pA00 p + pA000 p + pb0 p + pb00 p + pb000 p)5
for some d ∈ N+ .
Proof:
Assume that A0 is an m0 × n-matrix, A00 is an m00 × n-matrix and A000 is an
m000 ×n-matrix. Observe that CS(LC(P )) is the standard optimization problem
h i
min −c c 0 y
y
Ay = b
y ≥ 0,
where A is the (m0 + m00 + 2m000 ) × (2n + m0 + m00 + 2m000 )-matrix
A0 −A0 Im0
0 0 0
−A00 A00 0 Im00 0 0
A000 −A000
0 0 Im000 0
−A000 A000 0 0 0 Im000
and b is the vector
b0
−b00
b000 .
−b000
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
184 CHAPTER 7. COMPLEXITY
Therefore,
SCS(LC(P )) 6= ∅
by Proposition 1.1 and Proposition 1.3. Hence, by Proposition 7.1, there is
s ∈ SCS(LC(P )) with rational components such that
5
psp ≤ 9p(A, b)p .
Thus, by Proposition 1.3,
s|2n ∈ SLC(P )
has only rational components and
5
ps|2n p ≤ 9p(A, b)p .
Finally, by Proposition 1.1,
s|n − s|n+1
2n ∈ SP
and has only rational components and
5
ps|n − s|n+1
2n p ≤ 9p(A, b)p .
The thesis follows since
p(A, b)p ≤ d0 pA0 p + d00 pA00 p + d000 pA000 p + e0 pb0 p + e00 pb00 p + e000 pb000 p + e
for some d0 , d00 , d000 , e0 , e00 , e000 , e > 0. QED
7.3 Non-Deterministic Approach
We now analyze the complexity of the standard optimization decision problem
SDP (see Definition 7.1) and show that it is non-deterministically polynomial
(see Definition 7.12).
Proposition 7.7
The standard optimization decision problem SDP is in NP.
Proof:
Take W to be Q∪ × Q∪ × N+ and consider the algorithm d in Figure 7.2 with
Dd = DSF × (Q∪ × Q∪ × N+ ). Let P = (A, b, c) ∈ DSF and Q be the dual of
P . Then,
(1) We show that if SDP(P ) = 1 then the execution of d on (P, x, y, d) returns
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.3. NON-DETERMINISTIC APPROACH 185
input : ((A, b, c), x, y, d)
5 5
1. if asizeGE (6p(A, b)p , x) ∧ asizeGE (dp(A, c)p , y) ∧
dim c = dim x ∧ dim b = dim y ∧ x ≥ 0 ∧
Ax = b ∧ AT y ≤ cT ∧ cx = bT y then:
return 1;
else:
return 0.
Figure 7.2: Algorithm d.
1 for some (x, y, d) ∈ Q∪ × Q∪ × N+ .
Assume that SDP(P ) = 1. Then, SP 6= ∅. Hence, by Exercise 6.5,
RQ 6= ∅
and cs = bT r for every s ∈ SP and r ∈ RQ . Let x be chosen as in Proposition 7.5
for P and y and d chosen as in Proposition 7.6 with respect to Q. Hence,
5
pxp ≤ 6p(A, b)p
and
5
pyp ≤ d(pAT p + pcT p)5 ≤ dp(A, c)p .
Then, the execution of algorithm d on (P, x, y, d) returns 1 since the conditions
of the if in step 1 are true.
(2) We show that if SDP(P ) = 0 then for every (x, y, d) ∈ Q∪ × Q∪ × N+ the
execution of d on (P, x, y, d) returns 0.
Assume that SDP(P ) = 0. Thus, SP = ∅. Assume, by contradiction, that there
is (x, y, d) ∈ Q∪ × Q∪ × N+ such that the execution of d on (P, x, y, d) returns
1. Then, all the conditions of the if in step 1 are true. Therefore, x ∈ XP and
y ∈ YQ . Moreover, by Exercise 6.4, x ∈ SP and y ∈ RQ , contradicting SP = ∅.
(3) Observe that the evaluation of each of the conditions of the if is performed
in polynomial time on the size of (A, b, c):
5 5
• asizeGE (6p(A, b)p , x) and asizeGE (dp(A, c)p , y), by Proposition 7.2;
• dim x = dim c and dim w = dim b, by Proposition 7.11, Proposition 7.3
and the first item;
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
186 CHAPTER 7. COMPLEXITY
input : (A, b, c)
1. s := 0;
2. o := +∞;
3. for each subset B of {1, . . . , n} with m elements do:
(a) if there is a (unique) solution z of AB z = b and z > 0 then:
• set x such that (
zj if j ∈ B
xj :=
0 otherwise;
• if cx < o then:
o := cx;
s := x;
4. return s.
Figure 7.3: Basic algorithm b.
• the other conditions, by Proposition 7.12, Exercise 7.4 and the first item.
Thus, d is polynomial on the first part of the input; that is, on DSF . QED
7.4 Deterministic Approach
In this section, we analyze the basic algorithm mentioned in Section 4.2, that
relies on a brute-force method that goes through all basic admissible vectors,
to get an optimizer whenever there is one (recall Theorem 4.1).
Notation 7.5
We say that b in Figure 7.3 is the basic algorithm.
Proposition 7.8
Let P = (A, b, c) ∈ DSF be a standard optimization problem such that SP 6= ∅.
Then, the basic algorithm when executed on P returns a vector in SP .
Proof:
The algorithm is correct since it finds a basic admissible vector which is an
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.4. DETERMINISTIC APPROACH 187
optimizer (see Theorem 4.1). QED
Although b is correct, it is, nevertheless, very inefficient due to the high
number of times that the body of the cycle in Step 3 is executed. Indeed,
observe that the body of the cycle at Step 3 is executed
n n!
=
m m!(n − m)!
times assuming that A is an m × n-matrix. This fact allows us to show that b
is not polynomial. To this end, we have to show that there is no polynomial
map f : N → R+ 0 such that ωb ∈ O(f ). That is, for every such f we have to
provide an unbounded subsequence of the sequence
ωb (ν)
ρ = ν 7→ .
f (ν)
Definition 7.2
Let µ : N → N be the map defined as follows:
µ(ν) = p(A, b, c)p,
where
• (A, b, c) ∈ DSF ;
n
• A is a 2 × n-matrix and n is the (ν + 1)-th positive even number;
• each rational entry of A, b and c has 13 bits.
Note that 13 bits is the minimum size of a rational number (see Remark 7.3).
Notation 7.6
We denote by
Pν
the optimization problem corresponding to µ(ν) for every ν ∈ N. Moreover, we
denote by
nν
the number of columns of the matrix in Pν .
Proposition 7.9
For each ν ∈ N, √ p
2 µ(ν) + 21 3
nν = − .
4 2
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
188 CHAPTER 7. COMPLEXITY
Proof:
Observe that
• pcp = 16nν − 3;
• pbp = 8nν − 3;
• pAp = 8n2ν − 3.
For each ν ∈ N,
pPν p = µ(ν) = p(A, b, c)p = pAp + pbp + pcp + 6 = 8n2ν + 24nν − 3.
Observe that, the positive solution of the equation
µ(ν) = 8n2ν + 24nν − 3
is √ p
2 µ(ν) + 21 3
−
4 2
for every ν ∈ N. QED
Proposition 7.10
The basic algorithm is not efficient.
Proof:
Observe that (see [19])
!
nν nν
nν ≥ 22
2
since nν is a positive even number. Hence, by Proposition 7.9, we obtain:
√ p
2 µ(ν) + 21 3
−
nν
nν ≥ 2 8 4.
2
Therefore, since the cycle in step 3 is executed
nν
nν
2
times when b is run on input Pν , we get
√ p
2 µ(ν) + 21 3
−
τb (Pν ) ≥ 2 8 4
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.5. SOLVED PROBLEMS AND EXERCISES 189
and, so, √ p
2 µ(ν) + 21 3
−
ωb (µ(ν)) ≥ τb (Pν ) ≥ 2 8 4.
ωb (µ(ν))
Hence, limν→∞ ρ(µ(ν)) = limν→∞ f (µ(ν)) = +∞. QED
7.5 Solved Problems and Exercises
input : (m, n)
1. while scanning m̂ and n̂ simultaneously until both end:
• if the scan of m̂ ends before that of n̂ then return 1;
• if the scan of n̂ ends before that of m̂ then return 0;
2. o := 1;
3. while scanning m̂ and n̂ simultaneously:
• if the current bit of m̂ is greater than the current bit of n̂ then
o := 0;
• if the current bit of m̂ is less than the current bit of n̂ then o := 1;
4. return o.
Figure 7.4: An algorithm for ≤N .
Problem 7.1
Show that ≤N ∈ P.
Solution:
Given m, n ∈ N. Consider the algorithm in Figure 7.4 that returns the result
in o. The first loop performs at most
2 min{pmp, pnp} + 3
operations. On the other hand, the second loop performs at most
6 min{pmp, pnp}
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
190 CHAPTER 7. COMPLEXITY
input : (m, n)
1. b := 0;
2. o := 0;
3. while scanning m̂ and n̂ simultaneously until both end:
• if the scan of m̂ has ended then
x := 0;
else:
set x to the current bit of m̂;
• if the scan of n̂ has ended then:
y := 0;
else:
set y to the current bit of n̂;
• set the current bit of ô to the xor of b, x and y;
• if there are at least two 1’s among b, x, y then:
b := 1;
else:
b := 0;
4. if b 6= 0 then ô := 1ô;
5. return o;
Figure 7.5: An algorithm for +N
operations. So, the total number of operations is
8 min{pmp, pnp} + 5.
Therefore, the algorithm is linear and so ≤N ∈ P. /
Problem 7.2
Show that +N ∈ P.
Solution:
Given m, n ∈ N, consider the algorithm in Figure 7.5 that returns the result in
o and uses b for the carry-over bit. The algorithm computes m +N n in at most
12 max{pmp, pnp} + 5
bit operations. Therefore, the algorithm is linear and so +N ∈ P. /
Problem 7.3
Show that ×N ∈ P.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.5. SOLVED PROBLEMS AND EXERCISES 191
input : (m, n)
1. if m = 0 ∨ n = 0 then return 0;
2. o := 0;
3. while scanning n̂:
• if the current bit of n̂ is 1 then o := o +N m;
• m̂ := m̂0;
4. return o.
Figure 7.6: An algorithm for ×N .
Solution:
Given m, n ∈ N, consider the algorithm in Figure 7.6. In short, the algorithm
notices that, !
X X
i
m×n=m× 2 = m × 2i ,
i∈I i∈I
where
I = {i : 0 ≤ i < pnp and n̂pnp−i = 1}.
Moreover, recall that multiplying m by 2i in binary is equivalent to appending
to m̂ a sequence of size i composed only by 0’s. The multiplication algorithm
calls the algorithm for +N , and therefore, it is relevant to give a description on
how this algorithm is called, to understand its complexity. The main idea is to
copy the inputs of the algorithm to some free area of the computer memory,
and there, perform the +N algorithm. Finally, we copy the output back to the
program being run.2 It is easy to see that the loop in step 3 runs for pnp times.
Moreover, the summation in the loop adds numbers with at most pnp + pmp bits.
Therefore, the number of bit operations in the execution of the algorithm is
at most pnp(pnp + pmp) modulo a multiplicative factor. Hence, the worst-case
execution time of the algorithm is in O(ν 7→ ν 2 ). Thus, it is quadratic and so
×N ∈ P. /
Problem 7.4
Show that ≤ ∈ P.
Solution:
Given q, r ∈ Q, consider the algorithm in Figure 7.7 that returns the required
2 We stress that there are more efficient multiplication algorithms than the one presented
here, namely those based on the Fast Fourier Transform (see for instance [19]).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
192 CHAPTER 7. COMPLEXITY
input : (q, r)
1. if σq = − ∧ σr = + then return 1;
2. if σq = + ∧ σr = − then return (0 ≤N q) ∧ (q ≤N r) ∧ (r ≤N 0);
3. if σq = σr = + then return q ×N r ≤N q ×N r;
4. if σq = σr = − then return q ×N r ≤N q ×N r.
Figure 7.7: An algorithm for ≤.
result using the algorithms for ≤N and ×N (see Notation 7.10). The execution
times of the algorithms implementing the operations are polynomial on the size
of the input. Therefore, the algorithm is efficient. So, ≤ ∈ P. /
input : (q, r)
1. if σq = σr then:
σo := +;
else:
σo := −;
2. (o, o) := red(q ×N r, q ×N r);
3. return (σo , o, o).
Figure 7.8: An algorithm for ×.
Problem 7.5
Show that × ∈ P.
Solution:
Consider the algorithm in Figure 7.8 using the algorithms for ×N and red (see
Notation 7.10). Clearly, the algorithm is polynomial. So, × ∈ P. /
Problem 7.6
Show that + ∈ P.
Solution:
Given q, r ∈ Q, consider the algorithm in Figure 7.9 that returns the required
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.5. SOLVED PROBLEMS AND EXERCISES 193
input : (q, r)
1. if σq = σr then:
• σo := σq ;
• (o, o) := red(q ×N r +N q ×N r, q ×N r);
2. if σq = + ∧ σr = − then:
• if q ×N r ≤N q ×N r then:
σo := +;
else:
σo := −;
• if σo = + then:
N
(o, o) := red(q ×N r−̇ q ×N r, q ×N r);
else:
N
(o, o) := red(q ×N r−̇ q ×N r, q ×N r);
3. if σq = − ∧ σr = + then:
• if q ×N r ≤N q ×N r then:
σo := +;
else:
σo := −;
• if σo = − then:
N
(o, o) := red(q ×N r−̇ q ×N r, q ×N r);
else:
N
(o, o) := red(q ×N r−̇ q ×N r, q ×N r);
4. return (σo , o, o).
Figure 7.9: An algorithm for +.
N
result (σo , o, o) using the algorithms for ≤N , +N , −̇ , ×N and red (see Nota-
tion 7.10). It is immediate that the algorithm is polynomial on the size of the
input. So, + ∈ P. /
Problem 7.7
Consider the decision problem
Prime : N → {0, 1}
such that Prime(x) = 1 if and only if x is a prime number. Show that Prime
is in P (see Definition 7.9).
Solution:
There are several algorithms, usually called primality tests for this problem
based on different characterization of a prime number. The Wilson Theorem
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
194 CHAPTER 7. COMPLEXITY
(see [39]) states that given p ∈ N with p 6= 1
p is a prime number iff (p − 1)! ≡ −1 (mod p).
For instance, 5 is a prime number since 4! + 1 = 24 + 1 is a multiple of 5. There
is a straightforward algorithm based on this result. However, such an algorithm
is not polynomial. On the other hand, a generalization to polynomials of the
Fermat’s Little Theorem (see [39]) states that for p ≥ 2,
p is prime iff there is q coprime p with (x + q)p ≡ (xp + q) (mod p).
For instance, let p = 2 and q = 1. Then, 2x is always a multiple of 2. Hence,
2 is a prime number.
It was shown in [1] that there is an efficient algorithm, that we refer to as
aPrime , based on this result for Prime. Therefore, Prime is in P. /
Problem 7.8
Consider the decision problem
Factor : N3 → {0, 1}
such that Factor(d, k1 , k2 ) = 1 if and only if d has a prime factor in {k1 , . . . , k2 }.
For instance, Factor(21, 2, 5) = 1 and Factor(21, 10, 11) = 0. Show that
Factor ∈ NP
(see Definition 7.12).
Solution:
Let
W=N
and
aFact : N3 × W → {0, 1}
be the algorithm in Figure 7.10, using the efficient algorithm aPrime mentioned
in Problem 7.7. Assume that
Factor(d, k1 , k2 ) = 1.
So, there is a prime number p such that k1 ≤ p ≤ k2 and d is a multiple of
p. Take p as the witness. Then, the execution of algorithm aFact on the input
((d, k1 , k2 ), p) terminates with value 1. Indeed:
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.5. SOLVED PROBLEMS AND EXERCISES 195
input : ((d, k1 , k2 ), p)
1. if p > k2 ∨ p < k1 ∨ aPrime (p) = 0 then:
return 0;
else:
(a) if mod(d, p) = 0 then:
return 1;
else:
return 0.
Figure 7.10: Algorithm aFact .
• conditions p > k2 , p < k1 and αPrime (p) = 0 are false;
• condition mod(d, p) = 0 is true;
• the output is 1.
Assume that
Factor(d, k1 , k2 ) = 0.
So, there no prime number p such that k1 ≤ p ≤ k2 and d is a multiple of p.
Take q ∈ N as an arbitrary witness. Then, the execution of algorithm aFact on
the input ((d, k1 , k2 ), q) terminates with value 0. Indeed:
• either q > k2 or q < k1 or aPrime (q) = 0;
• for any of the cases above the output is 0.
Since aPrime is efficient, it is very easy to show that aFact is polynomial on N3 .
Thus, Factor is a NP problem. /
Problem 7.9
Consider the maps f = ν 7→ an ν n + · · · + a0 and g = ν 7→ ν n . Show that
f ∈ O(g).
Solution:
We have
an ν n + · · · + a0 ≤ ν n (|an | + · · · + |a0 |).
The thesis follows taking α = |an | + · · · + |a0 | and ν0 = 0. /
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
196 CHAPTER 7. COMPLEXITY
Exercise 7.1
Show that =N , ÷N , rem, gcd, red ∈ P (see Notation 7.10).
Exercise 7.2
Show that ·, ·, σ· , p · p, =, −, ·−1 ∈ P (see Notation 7.12).
Exercise 7.3
Show that there exists a polynomial algorithm for each of the following opera-
tions:
• w 7→ pwp : Q∪ → N+ ;
• A 7→ pAp : Q∪ × Q∪ → N+ ;
• (A, b, c) 7→ p(A, b, c)p : DSF → N+ ;
• (A, b) 7→ p(A, b)p : {(A, b) : ∃ c (A, b, c) ∈ DSF } → N+ ;
• (A, c) 7→ p(A, c)p : {(A, c) : ∃ b (A, b, c) ∈ DSF } → N+ .
Exercise 7.4
Show that there is a polynomial algorithm for computing each of the following
linear algebra operations over the field Q:
• inner product of two vectors;
• transpose of a matrix;
• product of a matrix by a vector;
• entry-wise comparisons (≤ and =) between vectors.
Exercise 7.5
Let CVP : DSF × Q∪ → {0, 1} be a map such that
CVP(P, x) = 1 if and only if x ∈ SP
for every pure canonical optimization problem P . Use the active lines technique
(see Section 3.2) to conclude that CVP ∈ P.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.6. RELEVANT BACKGROUND 197
Exercise 7.6
Let g : N → R+
0 . Show that
f (ν)
O(g) = f : N → R+
0 : lim sup < ∞ .
ν→∞ g(ν)
Exercise 7.7
Let f, g, h : N → R+
0 be maps. Show that
• O(g + h) = O(max{g, h});
• O(βg) = O(g), for each β ∈ R+ ;
• If fj ∈ O(gj ) for j = 1, 2 then f1 + f2 ∈ O(g1 + g2 );
• O(g) is closed for the sum of maps;
• f + O(g) = {f + h : h ∈ O(g)} ⊆ O(f + g);
• O(g) ∪ O(h) ⊆ O(max{g, h});
• O(min{g, h}) ⊆ O(g) ∪ O(h);
where
• βf for ν 7→ βf (ν);
• f + g for ν 7→ f (ν) + g(ν);
• max{f, g} for ν 7→ max{f (ν), g(ν)};
• min{f, g} for ν 7→ min{f (ν), g(ν)}.
7.6 Relevant Background
To analyze the efficiency of algorithms for solving linear optimization problems,
we need some concepts and results from the theory of computational complex-
ity [34, 5]. The efficiency of an algorithm is measured by the computational
resources that are required for its execution as a map of the size of the input
data. These resources are the execution time and the memory space.
Notation 7.7
Given an algorithm a, we denote by
Da
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
198 CHAPTER 7. COMPLEXITY
the input data space.
Remark 7.1
In the sequel, we assume that Da is a denumerable set.
Notation 7.8
For each d ∈ Da , we denote by
pdp
the size of d; that is, the number of bits used to represent d.
Notation 7.9
Given an algorithm a, we denote by
τa : Da → R+
0
the map assigning to each d the execution time of a on the input d; that is, the
number of bit operations that a executes on d.
Definition 7.3
The worst-case execution time of an algorithm a is the map
ωa = ν 7→ max{τa (d) : d ∈ Da , pdp ≤ ν} : N → R+
0.
In general, the idea is not to obtain the map ωa explicitly but rather to
identify a class of maps where this map belongs. In particular, from an efficiency
point of view, it is desirable that ωa belongs to a polynomial class.
For this purpose, we need to define the big O notation (also called the
Landau or the Bachman-Landau notation and in some cases also called the
asymptotic notation, see [19]).
Definition 7.4
The class of maps with asymptotic upper bound g : N → R+
0 up to a multiplica-
tive constant, denoted by
O(g),
is the set of maps f : N → R+
0 such that
∃ α ∈ R+ ∃ ν0 ∈ N ∀ ν > ν0 f (ν) ≤ αg(ν).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.6. RELEVANT BACKGROUND 199
Definition 7.5
An algorithm a is polynomial or algebraic or efficient (in time), written
ωa ∈ P or even a∈P
if
∃k ∈ N ωa ∈ O(ν 7→ ν k )
(see Remark 1.1). When a is polynomial, we denote by
ka
the minimum value of k satisfying the condition.
Definition 7.6
We say that a polynomial algorithm a is linear when ka = 1 and say that it is
quadratic when ka = 2.
Definition 7.7
An algorithm a is exponential if
k
∃k ∈ N ωa ∈ O(ν 7→ 2ν ).
Definition 7.8
A computation problem is a map h : D → E where D is a denumerable set
and E is a countable set. A decision problem is a computation problem where
E = {0, 1}.3
Definition 7.9
A computation problem h : D → E is polynomial, written
h ∈ P,
if there exists an efficient algorithm a such that:
• Da = D;
• for each d ∈ D, the execution of a on input d returns h(d).
3 The terminology of decision problem was first introduced by the mathematician David
Hilbert in 1928 (see [37]) when asking whether or not Mathematics is decidable.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
200 CHAPTER 7. COMPLEXITY
Sometimes, a problem is difficult to solve but checking if a candidate is
indeed a solution of the problem is much easier. This intuition leads to another
relevant class of complexity composed of the problems whose solutions can be
verified in polynomial time.
Before defining the class of such problems, we need to introduce the notion
of an algorithm being polynomial only on part of the input.
Definition 7.10
Let a be an algorithm with input data space factorized as follows:
Da = Da0 × Da00 .
The worst-case execution time based on the first argument of a, denoted by
ωa1 ,
is the map
+
ν 7→ sup{τa (d0 , d00 ) : (d0 , d00 ) ∈ Da , pd0 p ≤ ν} : N → R0 .
Definition 7.11
An algorithm a with Da = Da0 × Da00 is polynomial on Da0 if
∃k ∈ N ωa1 ∈ O(ν 7→ ν k ).
Definition 7.12
A decision problem h : D → {0, 1} is non-deterministically polynomial, written
h ∈ NP,
if there exist a set W and an algorithm a with Da = D × W polynomial on D
and such that:
• if h(d) = 1 then there exists w ∈ W such that the execution of a on input
(d, w) returns 1;
• if h(d) = 0 then, for every w ∈ W, the execution of a on input (d, w)
returns 0.
In this situation, the elements of W are called witnesses or certificates.
Clearly, every decision problem h : D → {0, 1} in P is in NP. The converse is
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.6. RELEVANT BACKGROUND 201
still an open problem,4 although the majority of mathematicians believe in the
conjecture5
P 6= NP.
To assess the efficiency of algorithms and establish the complexity of prob-
lems within the realm of rational numbers, we need first to analyze the time
complexity of the basic operations (including addition, multiplication and com-
parison) on rational numbers represented with infinite precision by pairs of
coprime natural numbers.
Moreover, we assume a binary representation for each natural number n,
denoted by
n̂ = n̂1 . . . n̂(n) ,
where each n̂i ∈ {0, 1}, (n) is the length of the binary representation of n, and
n̂1 is its most significant bit. Recall that either n̂1 = 1 or n = 0. In the latter
case (n) = 1. From this point on, we assume that the representation of n is n̂
and, so, that pnp = (n).
We assume also that there is independent and direct access to the least
significant bit of each of the inputs of an operation and that afterwards each
input can be sequentially scanned from bit to bit until the most significant bit
is reached.
We start by analyzing the time complexity of the following relevant basic
operations on natural numbers.
Notation 7.10
The following operations on natural numbers are relevant in the chapter:
≤N : m, n 7→ m less than or equal to n : N2 → {0, 1};
=N : m, n 7→ m equal to n : N2 → {0, 1};
+N : m, n 7→ sum of m and n : N2 → N;
N
−̇ : m, n 7→ maximum between 0 and n − m : N2 → N;
×N : m, n 7→ product of m and n : N2 → N;
÷N : m, n 7→ integer division of m by n : N2 → N;
rem : m, n 7→ reminder of division of m by n : N2 → N;
gcd : m, n 7→ greatest common divisor of m and n : N2 → N;
red : m, n 7→ (m ÷N gcd(m, n), n ÷N gcd(m, n)) : N × N+ → N × N+ .
4 Problem 4 of the Millenium Problems from the Clay Mathematics Institute.
5 Those interested in the problem should read Stephen Cook’s article [18] for a very nice
presentation of the main issues related to this problem.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
202 CHAPTER 7. COMPLEXITY
The reader will expect that adding two natural numbers with a few bits will
take less time than making the same operation on two natural numbers with
many bits. The key assumption here is that each single-bit operation, including
scanning a bit, comparing bits, summing two bits, assigning a bit to a variable,
among others, takes a unit of time.
Proposition 7.11
There exists a polynomial algorithm for each operation in Notation 7.10.
The proof of the previous proposition is spread across Problem 7.1, Prob-
lem 7.2, Problem 7.3 and Exercise 7.1.
Actually, for showing that all these operations can be computed in polyno-
mial time, we need only sufficiently good upper bounds on the time complexity
of these operations (see Problem 7.1, Problem 7.2 and Problem 7.3). Never-
theless, when available, we mention en passant better upper bounds that one
can easily find in the literature but that are more difficult to establish.
Capitalizing on the efficient algorithms presented in the proof of Propo-
sition 7.11, it is straightforward to provide efficient algorithms for the basic
operations on rational numbers. To this end, we start with the concept of
coprime natural numbers.
Definition 7.13
Two natural numbers are coprime if their greatest common divisor is 1.
Remark 7.2
Observe that the map
(m, n) 7→ m/n : {(m, n) ∈ N × N+ : m and n are coprime} → Q+
0
is a bijection since the coprimality requirement guarantees that each non-
negative rational number has a unique representation. This representation
is extended as expected to Q by adding a sign bit. Note that this extension is
no longer bijective because 0 has two signed representations.
Notation 7.11
Given a non-negative rational number q, we refer to its unique coprimal repre-
sentation by
(q, q).
Moreover, each q ∈ Q is identified with the triple
(σq , q, q).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
7.6. RELEVANT BACKGROUND 203
For representing a rational number
(σq , q, q),
in the computer memory, we need to encode the sign σq , the natural numbers
q and q, and the separators between the natural numbers.
Remark 7.3
The encoding
σ̂q
of the sign is 1 if σq is + and is 0 otherwise. To encode the comma “,”, we use
three bits, namely 001. Finally, for the representation of natural numbers, we
need to distinguish them from the separator. Hence, we triplicate each bit in
its binary representation. If
pqp
X
q= 2i−1 bi
i=1
then, we use the following string
q̂ = bpqp bpqp bpqp · · · b2 b2 b2 b1 b1 b1
to represent q. Similarly, for q. In this way, a rational is represented by the
string
q̂ = σ̂q 001 q̂ 001 q̂.
Observe that this string requires pqp = 7 + 3pqp + 3pqp bits (including one bit for
the sign and six bits for the two separators 001). Thus, the minimum size of a
rational number is 13 bits.
Notation 7.12
The following operations on rational numbers are relevant in the chapter:
· : q 7→ q : Q → N;
· : q 7→ q : Q → N+ ;
σ· : q 7→ σq : Q → {+, −};
p · p :7→ pqp : Q → N+ ;
≤ : q, r 7→ q less than or equal to r : Q2 → {0, 1};
= : q, r 7→ q equal to r : Q2 → {0, 1};
+ : q, r 7→ sum of q and r : Q2 → Q;
× : q, r 7→ product of q and r : Q2 → Q;
− : q 7→ symmetric of q : Q → Q;
·−1 : q 7→ 1/q : Q+ ∪ Q− → Q.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
204 CHAPTER 7. COMPLEXITY
The idea is to reduce operations on rational numbers to operations on nat-
ural numbers. In this way, we can take advantage on the algorithms introduced
above on natural numbers (see Proposition 7.11) to provide algorithms on ra-
tional numbers.
Proposition 7.12
There exists a polynomial algorithm for each of the operations in the rational
numbers.
The proof of the previous proposition is spread across Problem 7.4, Prob-
lem 7.5, Problem 7.6 and Exercise 7.2.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 8
The Simplex Algorithm
Herein, we present the Simplex Algorithm that provides a (positive or negative)
answer to the existence of optimizers of standard optimization problems and
provides an optimizer in the positive case (see [22, 24, 25, 49, 2, 32]). We prove
the soundness and completeness of the algorithm. Finally, we show that the
Simplex Algorithm is not polynomial.
8.1 Algorithm
In this section we discuss the Simplex Algorithm. We assume given a standard
optimization problem resulting from a canonical one. This is not a limitation
since we can always get a canonical optimization problem from a general linear
optimization problem (see Proposition 1.1). When an optimization problem is
such that b 6> 0, we can use either Exercise 1.4 or Exercise 1.5 to obtain an
equivalent optimization problem with a positive restriction vector.
Definition 8.1
Let P ∈ CS(C) be a non-degenerate problem with b > 0. The Simplex Algo-
rithm for P is the algorithm in Figure 8.1.
The variables b and s contain the output results. The variable b is equal
to 0 when the problem has no optimizers and is 1 otherwise. The variable
s returns an optimizer whenever there is one. The variables x, P, y, k, t, o
and j are auxiliary. The auxiliary variable x contains at each point the basic
admissible vector under analysis. The variable P is the set Px . The variable y is
a potential admissible vector of the dual problem. When y is admissible, then
205
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
206 CHAPTER 8. THE SIMPLEX ALGORITHM
input : (A, b, c) where A is an m × (n + m)-matrix
1. x := (0, . . . , 0, b1 , . . . , bm );
2. P := {j ∈ {1, . . . , n + m} : xj > 0};
3. y := solution of the system (AP )T y = (cP )T ;
4. if AT y ≤ cT then:
(a) (b, s) := (1, x);
(b) return (b, s);
else:
(a) k := some k such that (AT y)k > (cT )k ;
(b) t := t with tP = (AP )−1 a•k , tk = −1, t{1,...,n+m}\(P∪{k}) = 0;
(c) if t ≤ 0, then:
i. b := 0;
ii. return (b, x);
else:
xj
i. o := min : tj > 0 ;
tj
ii. x := x − ot;
iii. go to 2.
Figure 8.1: Simplex Algorithm s.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.1. ALGORITHM 207
x is an optimizer. Otherwise, we take note in k of a component that witnesses
that y is not admissible. The variables t and o are used to change x to another
basic admissible vector.
We now illustrate the application of the Simplex Algorithm.
Example 8.1
Recall the standard optimization problem P
min −2x1 − x2
x
3x1 − x2 + x3 = 6
−x1 + 3x2 + x4 = 6
x ≥ 0
introduced in Example 1.13. Observe that P ∈ CS(C) (see Example 1.12).
This problem is shown to be non-degenerate in Example 4.5. Morever, b > 0.
Hence, we can apply algorithm s to P . Let
0
0
x= 6
P = {3, 4}
6
By solving the system of equations (AP )T y = (cP )T we get
0
y= .
0
The vector y does not satisfy
AT y ≤ cT .
since it fails to satisfy both inequalities. Then, either k = 1 or k = 2. Assume
we choose k = 1. Then,
−1
0
t= .
3
−1
6
Since it is not the case that t ≤ 0, then, o = min = 2. Hence,
3
2
0
x= 0 .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
208 CHAPTER 8. THE SIMPLEX ALGORITHM
In this case, P = {1, 4}. By solving the system of equations (AP )T y = (cP )T
we get
2
−
y = 3 .
0
The vector y does not satisfy
AT y ≤ cT
since the second inequality does not hold. Thus, k = 2. Then,
1
−
3
−1
t= .
0
8
3
Since it is not the case that t ≤ 0, then, o = 3. Hence,
3
3
x= 0 .
So, P = {1, 2}. By solving the system of equations (A)M y = (c)T
M we get
7
−
8
y=
5
−
8
Vector y satisfies
AT y ≤ cT .
Therefore, s outputs b = 1 and s = (3, 3, 0, 0).
As we have seen in the example above, it may be the case that there are 2 or
more inequalities in AT y ≤ cT that do not hold, for a particular y. Therefore,
there are several possible choices for k. One of the most well known choices
is the gradient choice (which was used by George Dantzig in his seminal pa-
per [22]) consisting in choosing k = k where k is the smallest element in
{j ∈ N : cj − (AT y)j < 0}.
The choice of k in Example 8.1 was done according to this criterion.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.2. SOUNDNESS, COMPLETENESS AND COMPLEXITY 209
8.2 Soundness, Completeness and Complexity
The objective of this section is to prove the soundness and completeness of the
Simplex Algorithm. We start by showing some auxiliary lemmas.
Proposition 8.1
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. Then,
(0, . . . , 0, b1 , . . . , bm ), in Step 1 of the Simplex Algorithm in Figure 8.1, is a
basic admissible vector.
Proof:
We start by observing that (0, . . . , 0, b1 , . . . , bm ) ∈ XP , since b > 0 by hypothe-
sis. Moreover, (0, . . . , 0, b1 , . . . , bm ) is a basic vector by Proposition 4.9, because
b > 0. QED
Observe that the system of equations in Step 3 of the Simplex Algorithm
has always a unique solution. This is the case since AP is a nonsingular ma-
trix because the problem is non-degenerate and, in each iteration, x is a basic
admissible vector (see Proposition 8.5).
Proposition 8.2
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. When
x ∈ XP and the guard of Step 4 of the Simplex Algorithm in Figure 8.1 holds,
then x ∈ SP .
Proof:
Let Q be the dual of P (see Exercise 6.2). Since AT y ≤ cT , then y ∈ YQ .
Furthermore, after assignment at Step 3,
AP T y = cP T .
Hence,
T
xT T T
P AP y = xP cP .
Thus,
(AP xP )T y = xT T
P cP .
Therefore,
(Ax)T y = xT cT .
So,
bT y = cx,
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
210 CHAPTER 8. THE SIMPLEX ALGORITHM
since Ax = b. Taking into account that x ∈ XP and y ∈ YQ , by applying
Proposition 6.4, we conclude that x ∈ SP . QED
Proposition 8.3
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. When
x ∈ XP , the guard of Step 4 of the Simplex Algorithm in Figure 8.1 does not
hold and t ≤ 0 then SP = ∅.
Proof:
Consider the family
{wδ }δ∈R+ ,
0
where
wδ = x − δt.
We start by showing that k ∈
/ P. Indeed,
(AT y)k > (cT )k .
On the other hand,
T T
(AP ) y = (cP ) .
That is,
(AT y)j = (cT )j
for every j ∈ P. Thus,
k∈
/ P.
Then, for every j ∈ N , we have:
δ
if j = k
δ
wj = xj − δtj if j ∈ P
0 otherwise.
We now show that
cwδ < cx
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.2. SOUNDNESS, COMPLETENESS AND COMPLEXITY 211
for every δ > 0. Indeed,
cwδ = ck δ + cP xP − δ cP tP
= cP xP + δ(ck − cP tP )
= cP xP + δ(ck − yT AP tP ) (∗)
T −1
= cP xP + δ(ck − y AP (AP ) a•k ) (∗∗)
= cP xP + δ(ck − yT a•k ) (†)
< cP xP (∗ ∗ ∗)
= cx,
where the justifications are as follows:
(∗) since (AP )T y = (cP )T .
(∗∗) because tP = (AP )−1 a•k .
(∗ ∗ ∗) since (AT y)k > (cT )k .
Moreover, we prove that
wδ ∈ XP
for every δ > 0. In fact,
wδ = x − δt ≥ 0
because t ≤ 0 and x ≥ 0. Furthermore:
Awδ = AP (xP − δtP ) + A{k} δ + A{1,...,n+m}\({k}∪P) 0
= AP xP + δ(A{k} − AP tP )
= AP xP + δ(A{k} − AP (AP )−1 a•k )
= AP xP + δ(A{k} − a•k )
= AP xP
= b.
We now show that, for each z ∈ XP there is δ > 0 such that
cwδ < cz.
Consider two cases:
(1) cz < cP xP . Let
cz − cP xP
δ> > 0.
ck − yT a•k
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
212 CHAPTER 8. THE SIMPLEX ALGORITHM
Then,
cwδ = cP xP + δ(ck − yT a•k ) by (†)
cz − cP xP
< cP xP + (ck − yT a•k )
ck − yT a•k
= cz.
(2) cz ≥ cP xP . Then, cwδ < cz since cwδ < cx for every δ ∈ R+ , as seen above.
Therefore, SP = ∅. QED
Proposition 8.4
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. Assume
that x ∈ XP . Then, in Step (4c)(ii) of the Simplex Algorithm in Figure 8.1,
x − ot ∈ XP and c(x − ot) < cx.
Proof:
Let j be such that
xj
o= .
tj
Note that tj > 0. Then,
j ∈ P,
because tk = −1 and t{1,...,n+m}\(P∪{k}) = 0. Thus,
xj > 0.
Thus,
o > 0.
Note that
x − ot is wo ,
where wδ was defined in the proof of Proposition 8.3. Then,
x − ot ∈ XP
and
c(x − ot) < cx,
as already established in the proof of Proposition 8.3. QED
Proposition 8.5
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. Then,
in Step 1 and in each execution of Step (4c)(ii) of the Simplex Algorithm in
Figure 8.1, x is a basic admissible vector.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.2. SOUNDNESS, COMPLETENESS AND COMPLEXITY 213
Proof:
The proof follows by induction on the number k of assignments on x during the
execution of s.
(Base) k = 1. Then, x is basic by Proposition 8.1.
(Step) Assume by induction hypothesis that x is a basic vector in the (k − 1)-th
execution of Step (4c)(ii) of the Simplex Algorithm in Figure 8.1. We must
show that
x − ot
is a basic vector. Observe that x − ot ∈ XP by Proposition 8.4. By Proposi-
tion 4.9, because P is non-degenerate, it suffices to establish
|{j ∈ {1, . . . , n + m} : xj − otj > 0}| = m.
Observe that |Px | = m since x is basic. On the other hand,
Px−ot = {j ∈ {1, . . . , n + m} : wjo > 0}
taking into account the proof of Proposition 8.4. Let j be such that
xj
o= .
tj
We now prove that
Px−ot = {k} ∪ (Px \ {j}).
Since
k 6∈ Px
(see the proof of Proposition 8.3), then
wko = o > 0
(see the proof of Proposition 8.4). Hence,
k ∈ Px−ot .
On the other hand, since j ∈ Px (taking into account that o > 0), then
xj
xj − otj = xj − tj = 0.
tj
Thus,
j∈
/ Px−ot .
Note that
x` − ot` = 0,
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
214 CHAPTER 8. THE SIMPLEX ALGORITHM
for every ` ∈ {1, . . . , n + m} \ (Px ∪ {k}). Therefore,
` ∈ ({1, . . . , n + m} \ Px−ot )
for every ` ∈ {1, . . . , n + m} \ (Px ∪ {k}). Hence,
|{1, . . . , n + m} \ Px−ot | ≥ n − 1.
On the other hand,
j∈
/ {1, . . . , n + m} \ (Px ∪ {k})
since j ∈ Px . Thus,
|{1, . . . , n + m} \ Px−ot | ≥ n.
since j ∈
/ Px−ot . So,
|Px−ot | ≤ m.
Assume, by contradiction, that
|Px−ot | < m = |Px |.
Since x − ot ∈ XP , by Proposition 8.4, then
X
b= a•` (x − ot)` .
`∈Px−ot
Thus, b would be a linear combination of less than m columns of A contradicting
the non-degeneracy assumption. QED
Proposition 8.6
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. Then, the
execution of the Simplex Algorithm in Figure 8.1 on P terminates.
Proof:
Let xi be the basic vector of the i-th iteration of the execution of s on P .
Observe that
|{xi : i ≤ j}| = j
for every j ∈ N+ , since
cxi+1 < cxi ,
by Proposition 8.4. Assume, by contradiction, that the execution of s on P
does not terminate. Then,
|{xi : i ∈ N+ }| = |N|.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.2. SOUNDNESS, COMPLETENESS AND COMPLEXITY 215
Therefore, {xi : i ∈ N+ } is not finite. Moreover, by Proposition 8.5,
{xi : i ∈ N+ } ⊆ BP ,
where BP is the set of basic vectors of P . Therefore, BP is infinite contradicting
Proposition 4.5. QED
We are ready to prove the Soundness of the Simplex Algorithm Theorem.
Theorem 8.1
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. Then, the
execution of the Simplex Algorithm in Figure 8.1 on P terminates
• either with b = 0 and then SP = ∅;
• or with b = 1 and then s ∈ SP .
Proof:
(1) Assume that s terminates with b = 0. Then, x ∈ XP , t ≤ 0 and the guard
of the Simplex Algorithm in Step 4 is false. Therefore, by Proposition 8.3,
SP = ∅.
(2) Assume that s terminates with b = 1. Then, x ∈ XP and the guard of Step
4 of the Simplex Algorithm is true. Thus, by Proposition 8.2, x ∈ SP and so
s ∈ SP . QED
We now prove the Completeness of the Simplex Algorithm Theorem.
Theorem 8.2
Let P = (A, b, c) ∈ CS(C) be a non-degenerate problem with b > 0. Then,
• if SP = ∅ then the execution of s on P terminates with b = 0;
• if SP 6= ∅ then the execution of s on P terminates with b = 1 and s ∈ SP .
Proof:
Let Q be the dual of P (see Exercise 6.2).
(1) Assume that SP = ∅. Note that XP 6= ∅ since the initial x is an admissible
vector, by Proposition 8.1. Using Exercise 6.5, we conclude that:
YQ = ∅.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
216 CHAPTER 8. THE SIMPLEX ALGORITHM
x2
(0, 1)
(1, 1 − 2ε)
XP2ε
(0, 0) (1, 0)
x1
Figure 8.2: Set of admissible vectors of P2ε .
Observe that the guard of Step 4 tests whether or not y is an admissible vector
of Q. Hence, the guard of Step 4 never holds since YQ = ∅. Hence, the execution
of s on P never terminates with (b, s) = (1, x). Since the execution of s on P
terminates, by Proposition 8.6, then b = 0.
(2) Assume that SP 6= ∅. By Theorem 8.1, the execution of s on P does not
terminate with b = 0 but with b = 1 and with s ∈ SP . QED
The Simplex Algorithm is not efficient in general. Indeed, it is enough
to consider a family of problems such that the execution of the simplex on
the n-th problem of the family requires 2n − 1 iterations (along the lines of
Proposition 7.10). We use a variant of the Klee-Minty example, see [44, 52], as
the family of problems.
Definition 8.2
Let 0 < ε ≤ 31 and n ∈ N+ . The standard optimization problem Pnε is defined
as follows:
n
X
εn−j xj
min −
x
j=1
x1 + xn+1 = 1
i−1
X
εi−j xj + xn+i = 1,
2 i = 2, . . . , n
j=1
x ≥ 0.
For instance, the set of admissible vectors of the canonical optimization
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.3. SOLVED PROBLEMS AND EXERCISES 217
problem corresponding to P2ε is depicted in Figure 8.2 as well as the basic
vectors. As shown in [52], Pnε has 2n basic admissible vectors and the Simplex
Algorithm when executed on Pnε requires 2n − 1 iterations using an appropriate
choice of k. Therefore, the Simplex Algorithm visits all the basic admissible
vectors.
8.3 Solved Problems and Exercises
Problem 8.1
Let P be the following optimization problem:
h i
min −2 −1 0 0 x
x
" # " #
−1 1 1 0 1
x =
1 −1 0 1 1
x ≥ 0.
Show that SP = ∅ using the Simplex Algorithm in Figure 8.1.
Solution:
We start by noting that P ∈ CS(C), P is non-degenerate and b > 0. Hence, we
can apply the Simplex Algorithm. Let Q be the dual of P . The initial basic
admissible vector x is
0
0
.
1
1
Moreover, P = {3, 4}. The unique solution of
(AP )T y = (cP )T
is y = (0, 0). Since
AT y 6≤ cT
for the first and the second components, then y ∈
/ YQ . Take k = 1. Then,
−1
0
t= −1 .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
218 CHAPTER 8. THE SIMPLEX ALGORITHM
x2
XP
x1
Figure 8.3: Set XP 0 where CS(P 0 ) = P with P in Problem 8.1.
Since it is not the case that t ≤ 0, then o = 1. Hence,
1
0
x= 2 .
0
Then, P = {1, 3}. The unique solution of
(AP )T y = (cP )T
is y = (0, −2). Since
AT y 6≤ cT
for the first and the second components, then y ∈
/ YQ . Hence, take k = 2.
Then,
−1
−1
t= 0 .
0
Since it is the case that t ≤ 0, then b = 0. So, by Theorem 8.1, SP = ∅. /
Problem 8.2
Let P be the following optimization problem:
min −2x1 − x2
x
x1 + x3 = 5
4x1 + x2 + x4 = 25
x ≥ 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.3. SOLVED PROBLEMS AND EXERCISES 219
Using the Simplex Algorithm provide an optimizer of P .
Solution:
It is immediate that P is non-degenerate and b > 0. Hence, we can apply the
Simplex Algorithm. Let Q be the dual of P . The initial basic admissible vector
x is
(0, 0, 5, 25).
Then, P = {3, 4}. The unique solution of
(AP )T y = (cP )T
is y = (0, 0). Since
AT y 6≤ cT
for the first and the second components, then y ∈
/ YQ . Hence, take k = 1.
Then,
−1
0
t= 1 .
4
Moreover,
25
o = min 5, = 5.
4
Therefore,
x = (5, 0, 0, 5).
Then, P = {1, 4}. The solution of
(AP )T y = (cP )T
is y = (−2, 0). Since
AT y 6≤ cT
for the second component, then y ∈
/ YQ . Hence, take k = 2. Then,
0
−1
t= 0 .
Moreover,
o = 5.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
220 CHAPTER 8. THE SIMPLEX ALGORITHM
x2
XP
x1
Figure 8.4: Basic vectors visited in Example 8.2.
Thus,
x = (5, 5, 0, 0).
Hence, P = {1, 2}. The solution of
(AP )T y = (cP )T
is y = (2, −1). Since
AT y 6≤ cT
for the third component, then y ∈
/ YQ . Hence, take k = 3. Then,
1
−4
t= −1 .
Moreover,
o = 5.
Thus,
x = (0, 25, 5, 0).
Hence, P = {2, 3}. The solution of
(AP )T y = (cP )T
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
8.3. SOLVED PROBLEMS AND EXERCISES 221
is y = (0, −1). This vector is in YQ . Therefore, b = 1 and s = (0, 25, 5, 0) is an
optimizer, by Theorem 8.1. /
Exercise 8.1
Let P be the following optimization problem:
min 3x1 + 3x2 + 2x3
x
x1 + 2x2 + 3x3 = 3
4x1 + 5x2 + 6x3 = 9
x ≥ 0.
Using reductions and the Simplex Algorithm find an optimizer of P , if there is
one.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
222 CHAPTER 8. THE SIMPLEX ALGORITHM
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Chapter 9
Integer Optimization
An integer optimization problem is a linear optimization problem where the
components of the admissible vectors are natural numbers and the components
of A, b and c are rational numbers (recall Definition 1.7). More precisely, an
n-dimensional integer standard optimization problem is a linear optimization
problem of the form:
min cx
x
Ax = b
x ∈ Nn ,
where the components of A, b and c are assumed to be rational numbers. Sim-
ilarly, we can define an n-dimensional integer canonical optimization problem
as
max
x
cx
Ax ≤ b
x ∈ Nn ,
where the components of A, b and c are assumed to be rational numbers.
The reader can get a picture of the history of integer optimization in [41].
Example 9.1 (Knapsack Problem)
The Knapsack Problem consists in maximizing the value of objects that can be
put in a knapsack assuming that objects have a value and a weight and that
the weight of all the objects that can be carried in the knapsack has an upper
bound. Suppose that we have different kinds of objects, say j = 1, . . . , n. The
223
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
224 CHAPTER 9. INTEGER OPTIMIZATION
problem can be modeled as follows:
max
x
cx
ax ≤ b
x ∈ Nn ,
where
• xj is the number of objects of kind j;
• aj is the weight of each object of kind j;
• cj is the value of each object of kind j;
• b is the upper bound on the weight;
and aj and cj are positive rational numbers for j = 1, . . . , n and b is a positive
rational number.
9.1 Relaxed Problem
Some of the techniques for solving an integer optimization problem employ the
use of a companion optimization problem.
Definition 9.1
Let
max
x
cx
P = Ax ≤ b
x ∈ Nn
be an integer canonical optimization problem. The relaxed canonical optimiza-
tion problem of P , denoted by
R(P ),
is
max
x
cx
Ax ≤ b
x ≥ 0.
Similarly for an integer standard optimization problem.
It is worthwhile to compare the sets of admissible vectors and optimizers of
P and R(P ).
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.1. RELAXED PROBLEM 225
Proposition 9.1
Let P be an integer canonical optimization problem. Then,
1. XP ⊆ XR(P ) ;
2. if s ∈ SP and s0 ∈ SR(P ) then cs ≤ cs0 ;
3. if SR(P ) ∩ Nn 6= ∅ then SP ⊆ SR(P ) .
Proof:
(1) Let x ∈ XP . Then, x ≥ 0 since x ∈ Nn , and Ax ≤ b. Therefore, x ∈ XR(P ) .
(2) Assume that s ∈ SP and s0 ∈ SR(P ) . Observe that, by (1),
max{cx : x ∈ XR(P ) } ≥ max{cx : x ∈ XP }.
x x
Then,
cs0 = max{cx : x ∈ XR(P ) } ≥ max{cx : x ∈ XP } = cs.
x x
n
(3) Assume that s ∈ SR(P ) ∩ N . Hence, s ∈ XP . Moreover, s ∈ SP by (2).
Let s0 ∈ SP . Then, cs = cs0 and so s0 ∈ SR(P ) . QED
Another way to compare the sets of optimizers is via the integrality gap.
Definition 9.2
Let P be an integer canonical optimization problem such that there is s0 ∈
SR(P ) with cs0 6= 0. The integrality gap between SP and SR(P ) , denoted by
IGP ,
is
cs
,
cs0
where s ∈ SP and s0 ∈ SR(P ) . Similarly, for integer standard optimization
problems.
Observe that (
≤1 if P is canonical
IGP
≥1 if P is standard.
The integrality gap is 1 when an optimizer of the relaxed problem is also an
optimizer of the integer problem. In Proposition 9.7, we will see that when the
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
226 CHAPTER 9. INTEGER OPTIMIZATION
restriction matrix of P is totally unimodular and P satisfies some additional
mild conditions then
IGP = 1
meaning that we do not need to introduce new techniques for dealing with such
integer optimization problems.
Example 9.2 (Knapsack Problem)
Recall the knapsack problem P introduced in Example 9.1. The difficulty of the
knapsack problem has to do with the x1 , . . . , xn ∈ N constraint. The relaxed
canonical optimization problem R(P ) is
max
x
cx
ax ≤ b
x≥0
and can be solved using duality techniques. Observe that the dual problem
QR(P ) is
min by
y
aT y ≥ cT
y ≥ 0.
Let k = 1, . . . , n be an index such that
ck c1 cn
= max ,...,
ak a1 an
(observe that the weight aj and the value cj of each object of kind j are
positive, see Example 9.1); that is, k is a kind of an object with the greatest
ratio value/weight. Let x be such that
0 if j 6= k
xj =
b otherwise
ak
(note that b is positive, see Example 9.1); that is, we only put in the knapsack
objects of kind k. Moreover, let
ck
y= .
ak
We show that x ∈ SR(P ) . Indeed:
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.2. TOTALLY UNIMODULAR PROBLEMS 227
• x ∈ XR(P ) ;
• y ∈ YQR(P ) ;
bck
• by = = cx.
ak
Therefore, by Proposition 6.4, x ∈ SR(P ) and y ∈ RQR(P ) .
9.2 Totally Unimodular Problems
In this section, we intend to show that when the matrix of an integer canonical
optimization problem is totally unimodular then, under suitable conditions, the
integrality gap of the problem is 1 and the set of maximizers is non-empty.
Definition 9.3
An integer square matrix is unimodular if its determinant is either 1 or −1. A
matrix is totally unimodular if each of its nonsingular submatrices (see Defini-
tion 1.23) is unimodular.
Proposition 9.2
The entries of a totally unimodular matrix are either 0 or 1 or −1.
Proof:
Let A be a totally unimodular matrix and aij a coefficient of A. Assume that
aij 6= 0. Then, [aij ] is a nonsingular matrix with determinant aij . Since A
is totally modular by hypothesis, then either aij = 1 or aij = −1. Otherwise
aij = 0. QED
Example 9.3
For instance, the matrix
−1 0 1
1 1 0
is totally unimodular. Indeed:
(1) all the nonsingular submatrices 1 × 1 have determinants in {−1, 1} since it
is the case that det [1] = 1 and det [−1] = −1.
(2) all the nonsingular submatrices 2 × 2 have determinants in {−1, 1} since it
is the case that the determinant of each of the submatrices
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
228 CHAPTER 9. INTEGER OPTIMIZATION
−1 0 0 1 −1 1
1 1 1 0 1 0
is −1.
Proposition 9.3
The collection of all unimodular n × n-matrices is a group.
Proof:
(1) Assume that A and B are unimodular n × n-matrices. Hence, det A and
det B are in {−1, 1}. Then, A × B is also unimodular since
det (A × B) = (det A) × (det B),
by Proposition 4.20. Therefore, det (A × B) ∈ {−1, 1}.
(2) I is unimodular since det I = 1.
(3) Assume that A is a unimodular n × n-matrix. Hence, det A ∈ {−1, 1}.
Then, A is nonsingular since det A 6= 0 (see Proposition 4.21). Then, A−1 is
also unimodular since
1
det A−1 = ,
det A
by Proposition 4.20. Therefore, det A−1 ∈ {−1, 1}. QED
Proposition 9.4
Let A be an m × m-matrix with integer coefficients. Then,
A is unimodular iff det A 6= 0 and for each b ∈ Zm if Ax = b then x ∈ Zm .
Proof:
(→) Observe that A is a nonsingular matrix. Since x = A−1 b, then, by Cramer’s
Rule (see Proposition 4.23),
det [A]jb
xj = .
det A
Observe that either det A = 1 or det A = −1 and det [A]jb ∈ Z. Therefore,
xj ∈ Z for every j = 1, . . . , m.
(←) Let b be the vector ej (see Definition 1.20) and x = A−1 ej . Then, by
hypothesis, x ∈ Zm . Therefore,
(A−1 )•j = A−1 ej ∈ Zm
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.2. TOTALLY UNIMODULAR PROBLEMS 229
for every j = 1, . . . , m. Thus, det A−1 ∈ Z. But
1
det A−1 =
det A
by Proposition 4.20. Therefore, either det A−1 = det A = 1 or det A−1 =
det A = −1. Hence, A is unimodular. QED
The following result states that total unimodularity is preserved under spe-
cific operations.
Proposition 9.5
Let A be a totally unimodular m×n-matrix with integer coefficients. Then, any
matrix obtained from A by adding a row or a column of the form (0, . . . , 1, . . . , 0)
is again totally unimodular. The same happens if we multiply any row or col-
umn of A by −1.
Proof:
Denote the coefficients of A by aij for any i = 1, . . . , m and j = 1, . . . , n. We
only consider the case where the matrix A0 is obtained from A by adding the
line (0, . . . , 1, . . . , 0) at the top. That is, A0 is the matrix:
0 ... 1 ... 0
a11 ... a1j . . . a1n
...
am1 ... amj . . . amn
Let B 0 be a nonsingular submatrix of A0 . There are two cases:
(1) B 0 is a submatrix of A. Then, det B 0 ∈ {−1, 1} since A is totally unimod-
ular.
(2) B 0 is not a submatrix of A. Observe that B 0 should include element a01j (= 1)
of matrix A0 because otherwise B 0 is singular. Hence,
det B 0 = ± det B,
where B is the nonsingular submatrix of A composed by all lines of B 0 which
the exception of line 1 and all columns of B 0 with the exception of column j of
A0 (see Definition 4.8). Since A is totally unimodular,
det B ∈ {−1, 1}.
Hence, det B 0 ∈ {−1, 1}. Therefore, A0 is also totally unimodular. QED
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
230 CHAPTER 9. INTEGER OPTIMIZATION
We now provide a sufficient condition for a matrix to be totally unimodular
(firstly introduced in [36]).
Proposition 9.6
Let A be a matrix such that:
• the entries of A are either 0 or 1 or −1;
• there are no more than two non-zero entries in each column;
• the set of the indices of the rows can be partitioned in two subsets with
the following properties:
– if a column has two entries of the same sign then the indices of the
rows of those entries belong to different subsets of the partition;
– if a column has two entries of different signs then the indices of the
rows of those entries belong to the same subset of the partition.
Then, A is totally unimodular.
Proof:
We must show that each nonsingular k × k-submatrix of A is unimodular, for
every k ∈ N. The proof is by induction on k.
Base: k = 0. Observe that there is a unique 0 × 0-submatrix of A, the empty
matrix, which has determinant 1 (see Definition 4.8).
Step: Let k = `+1. Assume, by the induction hypothesis, that each nonsingular
` × `-submatrix of A has determinant either equal to 1 or equal to −1. Let A0
be a nonsingular k × k-submatrix of A. Recall (see Remark 4.2) that
k
A0
X
det A0 = (−1)i+j a0ij Mi,j ,
j=1
0
A
where Mi,j is the (i, j)-minor of A0 . There are three cases to consider:
(1) A0 has a column with only zero entries. In this case, det A0 = 0. Hence,
A0 is a singular matrix. So, this case is not possible.
(2) A0 has a column j with a unique non-zero entry. So, a0ij ∈ {−1, 1}. Then,
k
A0 A0
X
0
det A = (−1)i+j a0ij Mi,j = (−1)i+j a0ij Mi,j .
i=1
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.2. TOTALLY UNIMODULAR PROBLEMS 231
0
Since A0 is nonsingular, then Mi,j
A
6= 0 and the matrix A0i,j is nonsingular.
A0
Thus, by the induction hypothesis, Mi,j ∈ {−1, 1}. Therefore,
det A0 ∈ {−1, 1}.
(3) All columns of A0 have two non-zero entries. Let {K1 , K2 } be the partition
of {1, . . . , k} corresponding to the given partition of A. We now show that
X X
(†) a0ij = a0ij
i∈K1 i∈K2
for every j = 1, . . . , k. Assume that a0i1 j and a0i2 j are the two non-zero entries
of column j. Then, there are two cases:
(a) a0i1 j and a0i2 j have the same sign. Assume without loss of generality, that
i1 ∈ K1 , i2 ∈ K2 and a0i1 j = a0i2 j = 1. Then,
X X
a0ij − a0ij = a0i1 j − a0i2 j = 1 − 1 = 0.
i∈K1 i∈K2
(b) a0i1 j and a0i2 j have different signs. Assume without loss of generality, that
i1 , i2 ∈ K1 , a0i1 j = −1 and a0i2 j = 1. Then,
X X X
a0ij − a0ij = a0ij = a0i1 j + a0i2 j = −1 + 1 = 0.
i∈K1 i∈K2 i∈K1
Hence, by (†),
X X
a0i• − a0i• = 0
i∈K1 i∈K2
meaning that the set of rows of A is linearly dependent. Thus, det A0 = 0.
0
Hence, A0 is a singular matrix. So, this case is not possible. QED
Example 9.4
The matrix
1 0 0
0 −1 1
1 1 −1
is totally unimodular. Indeed, it satisfies the first two conditions of Proposi-
tion 9.6. The partition {{1}, {2, 3}} of the indices of rows shows that the third
condition of Proposition 9.6 is also satisfied. Therefore, Proposition 9.6 allow
us to conclude that the matrix is totally unimodular.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
232 CHAPTER 9. INTEGER OPTIMIZATION
Example 9.5 (Assignment Problem)
The Assignment Problem consists in assigning to each individual in a collection
V1 of individuals a task from a collection V2 of tasks in such a way that no task
is performed by more than one individual and no individual executes more
than one task. Moreover, we want to maximize the global satisfaction of the
individuals on the assignment.
We assume that initially, each individual in V1 had expressed which tasks in
V2 he/she is willing to perform and the respective degree of satisfaction. This
can be modeled by
• a bipartite graph (V, E)1 induced by the partition {V1 , V2 } of V , where
an edge e ∈ E from v1 ∈ V1 to v2 ∈ V2 states that individual v1 is willing
to perform task v2 ;
• a satisfaction degree map c : E → N such that c(e) is the degree of
satisfaction of individual S(e) in performing task T (e).
Each feasible assignment of individuals to tasks can be seen as a map
x : E → {0, 1}
such that,
• if x(e) = 1 then x(e0 ) = 0 for every e0 ∈ E such that S(e) = S(e0 )
expressing that to each individual at most one task is assigned;
• if x(e) = 1 then x(e0 ) = 0 for every e0 ∈ E such that T (e) = T (e0 )
expressing that to each at task at most one individual is assigned.
So, the assignment problem consists in finding a feasible assignment of in-
dividuals to tasks that maximizes the degree of satisfaction of the whole set
of individuals. That is, the assignment problem consists in finding a feasible
assignment s such that for every other assignment x
X X
c(e)x(e) ≤ c(e)s(e).
e∈E e∈E
Therefore, representing the maps x and c as vectors with components xe
and ce , respectively, for each e ∈ E, the assignment problem can be presented
1 A graph is a pair (V, E), where V is a set whose elements are called vertices and E ⊆ V ×V
is a set whose elements are called edges. For each e ∈ E, S(e) gives the first component of
e and T (e) the second component. A finite graph is a graph where V is a finite set. A
bipartite graph (V, E) induced by the partition {V1 , V2 } of V is a finite graph such that:
(1) E ⊆ V1 × V2 ; (2) for each v ∈ V1 there is e ∈ E such that S(e) = v; (3) for each v ∈ V2
there is e ∈ E such that T (e) = v.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.2. TOTALLY UNIMODULAR PROBLEMS 233
as the following integer canonical optimization problem:
X
max ce x e
x
e∈E
X
xe ≤ 1, for each v1 ∈ V1
e∈{e∈E:S(e)=v1 }
X
xe ≤ 1, for each v2 ∈ V2
}
e∈{e∈E:T (e)=v 2
xe ∈ {0, 1} for each e ∈ E.
Therefore, assuming that V1 = {v1 . . . . , vk }, V2 = {vk+1 . . . . , vm } and E =
{e1 , . . . , en }, the restriction m × n-matrix A has the following entries:
1 if S(ej ) = vi and 1 ≤ i ≤ k
aij = 1 if T (ej ) = vi and k + 1 ≤ i ≤ n
0 otherwise,
b is a vector with m ones and c is a row vector with n componentes. Observe
that aij = 1, where i is such that 1 ≤ i ≤ k if and only if vi is the source of
edge ej and aij = 1, where i is such that k + 1 ≤ i ≤ m if and only if vi is the
target of edge ej . Hence, if ej = (vi1 , vi2 ) then
ai1 j = ai2 j = 1
and of course vi1 ∈ V1 and vi2 ∈ V2 . Furthermore,
aij = 0
for every i ∈ {1, . . . , m} \ {i1 , i2 }. Thus, in each column there are exactly
two entries equal to 1. So A is totally unimododular by Proposition 9.6 by
partitioning the set of indices of rows in those indexed by V1 and those indexed
by V2 .
Consider the following sets V1 = {v1 , v2 , v3 } and V2 = {v4 , v5 , v6 } of in-
dividuals and tasks, respectively and the set E = {e1 : v1 → v4 , e2 : v1 →
v5 , e3 : v2 → v5 , e4 : v2 → v6 , e5 : v3 → v6 , e6 : v3 → v4 } of preferences. More-
over, assume that the degree of satisfaction for each possible choice is 1. So,
the corresponding assignment problem is the following 0-1 integer canonical
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
234 CHAPTER 9. INTEGER OPTIMIZATION
optimization problem:
max [1 1 1 1 1 1] [xe1 xe2 xe3 xe4 xe5 xe6 ]T
x
1 1 0 0 0 0 x e1 1
0 0 1 1 0 0 x e2 1
0
0 0 0 1 1 x e3 ≤
1
1 0 0 0 0 1 x e4
1
0 1 1 0 0 0 x e5 1
0 0 0 1 1 0 x e6 1
x , x , x , x , x , x ∈ {0, 1}.
e1 e2 e3 e4 e5 e6
As noted above, the restriction matrix of the assignment problem is totally
unimodular by choosing the partition of V1 ∪ V2 as
{{1, 2, 3}, {4, 5, 6}}.
Observe that the assignment x = (0, 1, 1, 0, 1, 0) is not an admissible vector
since xe2 + xe3 = 2 6≤ 1. In this case, individuals v1 and v2 would both be
assigned to task v5 . The assignments
(1, 0, 1, 0, 1, 0) and (0, 1, 0, 1, 0, 1)
are maximizers of the problem corresponding to the following assignments
v1 7→ v4 , v2 7→ v5 , v3 7→ v6 and v1 7→ v5 , v2 7→ v6 , v3 7→ v4 ,
respectively. Indeed, they are admissible. Moreover, since
min(|V1 |, |V2 |) = 3
then each admissible vector should have at most three 1’s.
The reader interested in assignment problems may consult [13, 28, 56, 16].
Also of interest to this subject are the works [17, 51, 42].
We now provide a sufficient condition for an integer canonical optimization
problem to have optimizers.
Proposition 9.7
Let P = (A, b, c) be an integer canonical optimization problem. Assume that:
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.3. THE BRANCH AND BOUND TECHNIQUE 235
• A is a totally unimodular m × n-matrix;
• b ∈ Zm ;
• XP 6= ∅;
• x 7→ cx is bounded from above in XR(P ) .
Then, SP is non-empty and IGP = 1.
Proof:
The restriction matrix of CS(R(P )) is
[A I]
which is also totally unimodular by Proposition 9.5. Since x 7→ cx is bounded
from above in XR(P ) , then x 7→ [−c 0] x is bounded from below in XCS(R(P )) .
Observe that XR(P ) 6= ∅ by Proposition 9.1. Hence, XCS(R(P )) 6= ∅. Therefore,
by Theorem 4.1, we conclude that there is a basic admissible vector, say s =
(s1 , . . . , sn+m ), such that
s ∈ SCS(R(P )) .
Thus, there is B ⊆ {1, . . . , m} such that AB is nonsingular and
AB sB = b.
Since [A I] is totally unimodular, then AB is unimodular. Hence, by Propo-
sition 9.4, sB ∈ Zm . Moreover, since s ∈ XCS(R(P )) , then sB ∈ Nm . Further-
more, s ∈ Nn+m . Thus, (s1 , . . . , sn ) ∈ SP and so IGP = 1. QED
The result above tells us that integer optimization problems with totally
unimodular matrices under some mild conditions can be analyzed using generic
linear optimization techniques.
9.3 The Branch and Bound Technique
As an illustration of a technique for solving integer linear optimization prob-
lems, we present in this section, a version of the branch and bound technique
proposed in [20] based on [45]. For a survey on techniques for addressing this
problem, including the cutting-plane technique proposed in [35], the interest
reader may consult [58].
An algorithm based on the branch and bound technique can be found in
Figure 9.1. The algorithm invokes an auxiliary algorithm
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
236 CHAPTER 9. INTEGER OPTIMIZATION
input : (A, b, c)
1. (b, s, o) := a(A, b, c);
2. if b = 0 then return 0;
3. w := {((A, b, c), (b, s, o))};
4. while w 6= {} do
(a) let ((A0 , b0 , c), (b, s, o)) be the first element of w;
(b) remove the first element from w;
(c) if each component of s is in N then return (1, s);
(d) let j be such that sj ∈ / N;
(e) let A0≤ , b0≤ be obtained by adding ej to A0 , bsj c to b0 ;
(f) (b, s, o) := a(A0≤ , b0≤ , c);
(g) if b = 1 then add ((A0≤ , b0≤ , c), (b, s, o)) to w sorted by (*);
(h) let A0≥ , b0≥ be obtained by adding −ej to A0 , −dsj e to b0 ;
(i) (b, s, o) := a(A0≥ , b0≥ , c);
(j) if b = 1 then add ((A0≥ , b0≥ , c), (b, s, o)) to w sorted by (*);
5. return 0.
Figure 9.1: Algorithm based on the branch and bound technique
that when executed over a canonical optimization problem (A, b, c) returns
(b, s, o) such that b = 0 when SP = ∅ and b = 1, s ∈ SP and o = cs otherwise.
The variable w is a list and not a set (elements can occur repeatedly and the
position of an element in the list is relevant). So, it makes sense to refer to the
first element of w as well as to remove an element in a particular position of
the list. Moreover, in the algorithm, (*) means in descending order of o and
for the same value of o putting first the ones with only natural components.
Example 9.6
Herein, we illustrate the application of the branch and bound technique to the
following integer linear optimization problem
max −2x1 + 5x2
x
−4x + 7x ≤ 21
1 2
P = 2x1 + 5x2 ≤ 32
x1 ≤ 6
x ∈ N2 .
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.3. THE BRANCH AND BOUND TECHNIQUE 237
The corresponding relaxed linear optimization problem R(P ) is
max −2x1 + 5x2
x
−4x1 + 7x2 ≤ 21
2x1 + 5x2 ≤ 32
x1 ≤ 6
x≥0
which has the maximizer
7
,5
2
and objective value 18. Thus, the initial value of w is
7
R(P ), 1, , 5 , 18 .
2
Since the first component of the maximizer is not a natural number, we consider
the following subproblems:
max −2x1 + 5x2
x
−4x1 + 7x2 ≤ 21
2x1 + 5x2 ≤ 32
P1 =
x1 ≤ 6
x1 ≤ 3
x ≥ 0
which has the maximizer
33
3,
7
123
and objective value and
7
max −2x1 + 5x2
x
−4x 1 + 7x2 ≤ 21
2x1 + 5x2 ≤ 32
P2 =
x1 ≤ 6
x1 ≥ 4
x ≥ 0.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
238 CHAPTER 9. INTEGER OPTIMIZATION
which has the maximizer
24
4,
5
and objective value 16. Thus, the value of w is
33 123 24
P1 , 1, 3, , , P2 , 1, 4, , 16 .
7 7 5
In the next step consider the first element of w. Since the second component
of the maximizer of P1 is not a natural number, we can consider the following
two subproblems of P1 :
max
−2x1 + 5x2
x
−4x1 + 7x2 ≤ 21
2x1 + 5x2 ≤ 32
P11 =
x1 ≤ 3
x2 ≤ 4
x ≥ 0
which has the maximizer
7
,4
4
33
and objective value and
2
max
−2x1 + 5x2
x
−4x1 + 7x2 ≤ 21
2x1 + 5x2 ≤ 32
P12 =
x1 ≤ 3
x2 ≥ 5
x ≥ 0
which has XP12 = ∅. Thus, the value of w is
7 33 24
P11 , 1, ,4 , , P2 , 1, 4, , 16 .
4 2 5
In the next step consider the first element of w. Since the first component of
the maximizer of P11 is not a natural number, we can consider the following
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.3. THE BRANCH AND BOUND TECHNIQUE 239
two subproblems of P11 :
max −2x1 + 5x2
x
−4x1 + 7x2 ≤ 21
1 5x2 ≤ 32
2x +
P111 = x1 ≤ 3
x2 ≤ 4
x ≤1
1
x≥0
which has the maximizer
25
1,
7
111
and objective value and
7
max −2x1 + 5x2
x
−4x 1 + 7x2 ≤ 21
2x1 + 5x2 ≤ 32
P112 = x1 ≤ 3
x2 ≤ 4
x1 ≥ 2
x≥0
which has the maximizer
(2, 4)
and objective value 16. Thus, the value of w is
24
(P112 , (1, (2, 4) , 16)) , P2 , 1, 4, , 16 ,
5
25 111
P111 , 1, 1, , .
7 7
In the next step the execution of the algorithm ends returning (1, (2, 4)).
For details about the complexity of integer optimization algorithms the
reader may consult [50].
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
240 CHAPTER 9. INTEGER OPTIMIZATION
9.4 Solved Problems and Exercises
Problem 9.1 (Vertex Coloring Problem, see [40])
Let G = (V, E) be a finite graph. The map
g : V → {1, . . . , k}
is a k-coloring of G whenever g(v1 ) 6= g(v2 ) for every (v1 , v2 ) ∈ E. The graph G
is k-colorable if there is a k-coloring of G. The objective is to find the smallest
number of colors from {1, . . . , k} that provides a k-coloring of G. Model this
problem as a linear optimization problem.
Solution:
Assume that V = {v1 , . . . , v` }. An admissible vector has the form
(x1 , . . . , xk , z1v1 , . . . , zkv` )
meaning that
( (
1 if color j is used 1 if vertex v is colored with j
xj = zjv =
0 otherwise 0 otherwise.
Hence, the linear optimization problem
k
X
min xj
x,z v1 ,...,z v`
j=1
k
X
zjv = 1 for each v ∈ V
j=1
0
zjv + zjv ≤ 1 for each (v, v 0 ) ∈ E
zjv ≤ xj
for each v ∈ V and j = 1, . . . , k
xj ∈ {0, 1}
for each j = 1, . . . , k
v
zj ∈ {0, 1} for each v ∈ V and j = 1, . . . , k
models the vertex coloring problem. Observe that when there are no optimizers,
we conclude that the graph is not k-colorable. When there are optimizers, the
value of the objective map for each optimizer is the chromatic number of G
(the smallest number of colors needed to color the graph). /
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
9.4. SOLVED PROBLEMS AND EXERCISES 241
Exercise 9.1
Consider the problem
max 2x1 + 3x2
x
− 2 x1 + x2 ≤ 1
3
I=
8
3 x1 + x2 ≤ 16
x ∈ N.
Use the branch and bound algorithm to conclude whether or not this problem
has a maximizer.
Exercise 9.2 (Traveling Problem)
Consider a tourist that wants to visit a particular collection of cities minimizing
the number of kilometers traveling, in such a way that each city is visited once
and only once and return to the departure city. Model this problem as an
integer linear optimization problem.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
242 CHAPTER 9. INTEGER OPTIMIZATION
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Bibliography
[1] M. Agrawal, N. Kayal, and N. Saxena. Primes is in P. Annals of Mathe-
matics, 2:781–793, 2002.
[2] D. Alevras and M. W. Padberg. Linear Optimization and Extensions.
Springer, 2001.
[3] T. M. Apostol. Calculus. Vol. I: One-Variable Calculus, with an Introduc-
tion to Linear Algebra. Blaisdell, second edition, 1967.
[4] T. M. Apostol. Calculus. Vol. II: Multi-Variable Calculus and Linear Al-
gebra, with Applications to Differential Equations and Probability. Wiley,
second edition, 1991.
[5] S. Arora and B. Barak. Computational Complexity. Cambridge University
Press, 2009.
[6] M. Audin. Geometry. Springer, 2003.
[7] S. Axler. Linear Algebra Done Right. Springer, second edition, 1997.
[8] M. K. Bennett. Affine and Projective Geometry. Wiley, 1995.
[9] M. Berger. Geometry I. Springer, 2009.
[10] D. P. Bertsekas. Network Optimization: Continuous and Discrete Models.
Athena Scientific, 1998.
[11] D. P. Bertsekas. Convex Analysis and Optimization. Athena Scientific,
2003.
[12] D. Bertsimas and J. Tsitsiklis. Introduction to Linear Optimization.
Athena Scientific, 1997.
[13] G. Birkhoff. Three observations on linear algebra. Univ. Nac. Tucumán.
Revista A., 5:147–151, 1946.
243
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
244 BIBLIOGRAPHY
[14] T. S. Blyth and E. F. Robertson. Basic Linear Algebra. Springer, 2002.
[15] W. C. Brown. Matrices and Vector Spaces. Marcel Dekker, 1991.
[16] R. Burkard, M. Dell’Amico, and S. Martello. Assignment Problems. Soci-
ety for Industrial and Applied Mathematics, 2009.
[17] R. E. Burkard and E. Çela. Linear assignment problems and extensions.
In Handbook of Combinatorial Optimization, Supplement Vol. A, pages
75–149. Kluwer, 1999.
[18] S. Cook. The importance of the P versus NP question. Journal of the
ACM, 50(1):27–29, 2003.
[19] T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction
to Algorithms. MIT Press, third edition, 2009.
[20] R. J. Dakin. A tree-search algorithm for mixed integer programming prob-
lems. The Computer Journal, 8:250–255, 1965.
[21] G. B. Dantzig. A theorem on linear inequalities. Technical report, Pen-
tagon, 1948.
[22] G. B. Dantzig. Maximization of a linear function of variables subject
to linear inequalities. In Activity Analysis of Production and Allocation,
pages 339–347. Wiley, 1951.
[23] G. B. Dantzig. Reminiscences about the origins of linear programming.
Operations Research Letters, 1(2):43–48, 1982.
[24] G. B. Dantzig. Origins of the simplex method. In A History of Scientific
Computing, pages 141–151. ACM, 1990.
[25] G. B. Dantzig. Linear Programming and Extensions. Princeton University
Press, reprint of the 1968 corrected edition, 1998.
[26] G. B. Dantzig. Linear programming. Operations Research, 50(1):42–47,
2002.
[27] G. B. Dantzig and A. Orden. Notes on linear programming: Duality
theorems. Technical Report RM-1265, AD 114135, US Air Force Project
Rand, 1953.
[28] U. Derigs, O. Goecke, and R. Schrader. Monge sequences and a simple
assignment algorithm. Discrete Applied Mathematics, 15(2-3):241–248,
1986.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
BIBLIOGRAPHY 245
[29] I. Ekeland and R. Témam. Convex Analysis and Variational Problems.
Society for Industrial and Applied Mathematics, 1999.
[30] J. Farkas. Theorie der einfachen Ungleichungen. Journal für die Reine
und Angewandte Mathematik, 124:1–27, 1902.
[31] W. Fleming. Functions of Several Variables. Springer, second edition,
1977.
[32] J. N. Franklin. Methods of Mathematical Economics. Society for Industrial
and Applied Mathematics, 2002.
[33] J. Gallier. Geometric Methods and Applications: For Computer Science
and Engineering. Springer, second edition, 2013.
[34] M. R. Garey and D. S. Johnson. Computers and Intractability. W. H.
Freeman and Co., 1979.
[35] R. E. Gomory. An algorithm for integer solutions to linear programs. In
Recent Advances in Mathematical Programming, pages 269–302. McGraw-
Hill, 1963.
[36] I. Heller and C. B. Tompkins. An extension of a theorem of Dantzig’s. In
Linear Inequalities and Related Systems, volume 38 of Annals of Mathe-
matics Studies, pages 247–254. Princeton University Press, 1956.
[37] D. Hilbert and W. Ackermann. Grundzüge der theoretischen Logik.
Springer, 1959.
[38] F. E. Hohn. Elementary Matrix Algebra. Dover Publications, 2002.
[39] K. Ireland and M. Rosen. A Classical Introduction to Modern Number
Theory. Springer, 1990.
[40] T. R. Jensen and B. Toft. Graph Coloring Problems. Wiley, 1995.
[41] Jünger, T. M., Liebling, D. Naddef, G. Nemhauser, W. Pulleyblank,
G. Reinelt, G. Rinaldi, and L. Wolsey, editors. 50 Years of Integer Pro-
gramming 1958–2008. Springer, 2010.
[42] S. N. Kabadi and A. P. Punnen. A strongly polynomial simplex method
for the linear fractional assignment problem. Operations Research Letters,
36(4):402–407, 2008.
[43] Y. Katznelson and Y. R. Katznelson. A (Terse) Introduction to Linear
Algebra, volume 44. American Mathematical Society, 2008.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
246 BIBLIOGRAPHY
[44] V. Klee and G. J. Minty. How good is the simplex algorithm? In Inequal-
ities, III, pages 159–175. Academic Press, 1972.
[45] A. H. Land and A. G. Doig. An automatic method of solving discrete
programming problems. Econometrica, 28:497–520, 1960.
[46] S. Lang. Linear Algebra. Springer, third edition, 1989.
[47] S. Lang. Algebra. Springer, third edition, 2002.
[48] J. R. Munkres. Topology. Prentice-Hall, second edition, 2000.
[49] M. W. Padberg. Linear Optimization and Extensions. Springer, 1999.
[50] C. H. Papadimitriou. On the complexity of integer programming. Journal
of the Association for Computing Machinery, 28(4):765–768, 1981.
[51] C. Papamanthou, K. Paparrizos, N. Samaras, and K. Stergiou. Worst
case examples of an exterior point algorithm for the assignment problem.
Discrete Optimization, 5(3):605–614, 2008.
[52] K. Paparrizos, N. Samaras, and D. Zissopoulos. Linear programming:
Klee–Minty examples. In C. A. Floudas and P. M. Pardalos, editors,
Encyclopedia of Optimization, pages 1891–1897. Springer, 2009.
[53] P. Pedregal. Introduction to Optimization. Springer, 2003.
[54] M. Reid and B. Szendrői. Geometry and Topology. Cambridge University
Press, 2005.
[55] R. T. Rockafellar. Convex Analysis. Princeton University Press, 1970.
[56] A. E. Roth, U. G. Rothblum, and J. H. Vande Vate. Stable matchings,
optimal assignments, and linear programming. Mathematics of Operations
Research, 18(4):803–828, 1993.
[57] W. Rudin. Functional Analysis. McGraw-Hill, 1991.
[58] A. Schrijver. Theory of Linear and Integer Programming. Wiley, 2000.
[59] I. R. Shafarevich and A. O. Remizov. Linear Algebra and Geometry.
Springer, 2013.
[60] T. Terlaky and S. Z. Zhang. Pivot rules for linear programming: A sur-
vey on recent theoretical developments. Annals of Operations Research,
46/47(1-4):203–233, 1993.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
BIBLIOGRAPHY 247
[61] J. von Neumann. On a maximization problem. Technical report, Institute
for Advanced Studies, Princeton, 1947.
[62] J. von Neumann and O. Morgenstern. Theory of Games and Economic
Behavior. Princeton University Press, second edition, 1947.
[63] S. J. Wright. Primal-Dual Interior-Point Methods. Society for Industrial
and Applied Mathematics, 1997.
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
248 BIBLIOGRAPHY
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
List of symbols
=, 5 C, 13
[ ], 38 CS, 19
≥, 5
≤, 5 DSF , 178
DSF
mn , 177
{0}K , 32
Da , 197
(Rn , Θ), 117
det A, 111
ˆ·, 179, 201, 203
dim, 108, 126, 146, 180
·, 202
d(x), 158
p · p, 198
·, 202 e(y), 158
A= , 122 int(U ), 58
AB , 89 IGP , 225
Ai,j , 111 ker(A), 115
A(W ), 121
A(W ), 118 L, 7
A, 12 LC, 15
Ax , 74
M , 89
a•j , 73 A
Mi,j , 112
ai• , 73
N , 89
Bε (x), 57 N > , 122
Bε [x], 57 NP, 200
bn(U ), 58
b, 12 O(g), 198
b= , 122 P, 199
A
P , 148
Ci,j , 112
i
CDP, 179 Q|j , 15
C(A), 73 Q∪ , 178
249
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
250 LIST OF SYMBOLS
RQ , 150 τa , 198
rank(A), 114
Rn , 33 V ⊥ , 109
V1 + V2 , 34
SP , 3
SDP, 179 ωa1 , 200
span(A), 114 ωa , 198
spanV (U ), 34
s, 206 XP , 3
S, 14 XP◦ , 40
sizeGE, 180 ∂XP , 40
SC, 18
SP , 22 YQ , 150
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
Subject index
Active line, 74 Basis
Admissible vector, 2 admissible, 90
basic, 90 canonical, 36
set of, 3 index, 89
Affine standard, 36
combination, 118 vector space, 36
dimension of space, 146 Boundary point, 58
dimension of subspace, 146 Branch and Bound Technique, 236
direction, 142
hull, 121
Canonical Optimization Decision
point, 142
Problem, 179
smallest subspace, 118
Canonical optimization problem,
space, 142
11
subspace, 144
Algorithm dual, 150
efficient, 199 integer, 223
exponential, 199 parameterized, 147
linear, 199 pure, 12
polynomial, 199 relaxed, 224
polynomial on part of the Canonical Weak Duality Theorem,
input, 200 151
quadratic, 199 Cauchy–Schwarz Inequality, 56
Simplex, 205, 206 Chasles’ Relation, 142
Assignment Problem, 232 Closure
under affine combinations, 119
Ball Cofactor Expansion, 112
closed, 57 Combination
open, 57 affine, 118
Basic admissible vector, 90 convex, 63
Basic Minimizer Theorem, 97 linear, 35
251
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
252 SUBJECT INDEX
Completeness of the Simplex for Matrices, 73
Algorithm Theorem, 215 Geometric Variant, 70
Computation problem, 199 Logical Variant, 169
polynomial, 199 Pure Variant, 166
Cone Standard Variant, 79
convex, 64 Field, 31
generated by, 64
primitive, 65 Group, 30
Constraint, 1
Constructive Maximizer Theorem, Hyperplane, 127
98 m-basic, 134
Convergence of sequence, 59
Cramer’s Rule, 113 Inner product, 55
Euclidean, 55
Decision problem, 199 linearity conditions, 55
Canonical Optimization, 179 positive definiteness, 55
non-deterministically symmetry, 55
polynomial, 200 Input data space, 198
Standard Optimization, 179 Integer canonical optimization
Description, 1 problem, 223
Diet Problem, 9 relaxed, 224
Dimension Integer optimization problem, 10
affine space, 146 Branch and Bound Technique,
affine subspace, 146 236
set, 126 canonical, 223
vector space, 108 integrality gap, 225
Direction, 142 relaxed canonical, 224
Dual standard, 223
of canonical optimization Integer standard optimization
problem, 150 problem, 223
of standard optimization Integrality gap, 225
problem, 156 Interior point, 58
Equilibrium Condition, 162 Knapsack Problem, 2, 4, 5, 11,
Equilibrium Theorem, 162 223, 226
Execution time, 198
Existence of Maximizer Theorem, Laplace’s Expansion Formula, 111
50 Leibniz’s Formula, 112
Lemma
Face, 132 Canonical Variant of Farkas,
Farkas’ Lemma 79
Canonical Variant, 79 Existence of Dual, 152
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
SUBJECT INDEX 253
Geometric Variant of Farkas, empty, 38
70 kernel, 115
Logical Variant of Farkas, 169 minor, 112
Matrix Variant of Farkas, 73 nonsingular, 110
Non-Degeneracy, 100 rank, 114
Pure Variant of Farkas, 166 submatrix, 38
Slack, 159 totally unimodular, 227
Standard Variant of Farkas, unimodular, 227
79 Maximization problem, 3
Lemma of Existence of Dual, 152 general linear, 7
Linear linear, 7
combination, 35 Maximizer, 3
map, 37 Maximizer Theorem, 78
Linear optimization problem, 7 Maximum
canonical, 11 local, 75
compact form, 8 Minimization problem, 3
integer, 10 general linear, 7
matricial form, 8 linear, 7
set of, 7 Minimizer, 3
standard, 13
Local Maximizer Theorem, 75 Non-Degeneracy Lemma, 100
Logic of Inequalities, 167 Norm, 54
derivation, 167 Euclidean, 55
formula, 167 positive definiteness, 54
inconsistent formula, 168 positive homogeneity, 54
model of formula, 169 triangular inequality, 54
satisfaction of formula, 169
Soundness Theorem, 169 Objective map, 3
Logical Variant of the Farkas’ Optimality Criterion, 151
Lemma, 169 Optimization problem, 2
Optimizer, 3
Map set of, 3
bounded from above, 62 Orthogonal vectors, 56
bounded from below, 62
continuous, 61 Parallelogram Rule, 143
linear, 37 Perpendicular vectors, 56
maximum on a set, 62 Point, 142
minimum on a set, 62 Polyhedron, 131
slack, 158 face, 132
Matrix, 37 vertex, 132
cofactor, 112 Polytope, 131
determinant, 111 Problem
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
254 SUBJECT INDEX
Assignment, 232 of vector, 159
Diet, 9 Slack Complementarity Theorem,
Knapsack, 2, 4, 5, 11, 223, 226 160
Rectangle, 2, 3 Slack Lemma, 159
Transportation, 23 Slack maps, 158
Traveling, 241 Slack of vector, 159
Vertex Coloring, 240 Soundness of the Logic of
Pure Variant of Farkas’ Lemma, Inequalities Theorem, 169
166 Soundness of the Simplex
Algorithm Theorem, 215
Rank-Nullity Theorem, 115 Span
Rectangle Problem, 2, 3 matrix, 114
vector space, 34
Semi-space Standard Optimization Decision
lower, 129 Problem, 179
upper, 129 Standard optimization problem, 13
Set dual, 156
bounded, 60 integer, 223
closed, 59 non-degenerate, 99
compact, 60 restricted, 87
convex, 63 Standard Strong Duality Theorem,
dimension, 126 157
linearly dependent, 35 Standard Weak Duality Theorem,
linearly independent, 35 156
lower bound, 62 Strong Duality Theorem
of boundary points, 58 Canonical, 155
of interior points, 58 Subspace
open, 59 orthogonal complement, 109
upper bound, 62 vector space, 33
Set of admissible vectors, 3
boundary of, 40 Theorem
interior of, 39 Basic Minimizer, 97
Set of optimizers, 3 Canonical Strong Duality, 155
Simplex Algorithm, 205, 206 Canonical Weak Duality, 151
Completeness Theorem, 215 Completeness of the Simplex
gradient choice, 208 Algorithm, 215
Soundness Theorem, 215 Constructive Maximizer, 98
Slack Equilibrium, 162
Complementarity Theorem, Existence of Maximizer, 50
160 Local Maximizer, 75
Lemma, 159 Maximizer, 78
maps, 158 Rank-Nullity, 115
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
SUBJECT INDEX 255
Slack Complementarity, 160 canonical basis, 36
Soundness of the Logic of dimension, 108
Inequalities, 169 finite-dimensional, 35
Soundness of the Simplex induced by a field, 33
Algorithm, 215 linearly dependent set, 35
Standard Strong Duality, 157 linearly independent set, 35
Standard Weak Duality, 156 null vector, 33
Uniqueness, 164 span, 34
Vertex Minimizer, 137 subspace, 33
Transportation Problem, 23 sum, 34
Traveling Problem, 241 Vertex, 132
Vertex Coloring Problem, 240
Uniqueness Theorem, 164
Vertex Minimizer Theorem, 137
Vector space, 32
basis, 36 Worst-case execution time, 198
This book has been purchased as a PDF version from the Publisher and is for the purchaser's sole use.
You might also like
- Solution Manual For An Introduction To Optimization 4th Edition Edwin K P Chong Stanislaw H Zak100% (52)Solution Manual For An Introduction To Optimization 4th Edition Edwin K P Chong Stanislaw H Zak13 pages
- An Introduction To Continuous Optimization100% (4)An Introduction To Continuous Optimization400 pages
- Linear and Integer Optimization - Theory and Practice, 3rd Ed, 2015100% (2)Linear and Integer Optimization - Theory and Practice, 3rd Ed, 2015676 pages
- Applied Linear Algebra and Optimization Using MATLABNo ratings yetApplied Linear Algebra and Optimization Using MATLAB1,176 pages
- Optimization Models (Giuseppe C. Calafiore, Laurent El Ghaoui) (Z-Library)No ratings yetOptimization Models (Giuseppe C. Calafiore, Laurent El Ghaoui) (Z-Library)648 pages
- Métodos numéricos aplicados a Ingeniería: Casos de estudio usando MATLABFrom EverandMétodos numéricos aplicados a Ingeniería: Casos de estudio usando MATLAB5/5 (1)
- A First Course in Linear Optimi - Wei ZhiNo ratings yetA First Course in Linear Optimi - Wei Zhi306 pages
- JohnLee First Course in Linear OptimizationNo ratings yetJohnLee First Course in Linear Optimization172 pages
- Instant Ebooks Textbook Fundamentals of Optimization Theory With Applications To Machine Learning Gallier J. Download All Chapters100% (2)Instant Ebooks Textbook Fundamentals of Optimization Theory With Applications To Machine Learning Gallier J. Download All Chapters62 pages
- (A First Course in Linear Optimization) Jon Lee (B-Ok - CC) PDF100% (1)(A First Course in Linear Optimization) Jon Lee (B-Ok - CC) PDF188 pages
- Where can buy Fundamentals of optimization theory with applications to machine learning Gallier J. ebook with cheap price100% (2)Where can buy Fundamentals of optimization theory with applications to machine learning Gallier J. ebook with cheap price55 pages
- Download ebooks file (Ebook) Fundamentals of optimization theory with applications to machine learning by Gallier J., Quaintance J all chapters100% (9)Download ebooks file (Ebook) Fundamentals of optimization theory with applications to machine learning by Gallier J., Quaintance J all chapters54 pages
- An Introduction to Optimization 2nd Edition Edwin K. P. Chong instant download100% (1)An Introduction to Optimization 2nd Edition Edwin K. P. Chong instant download49 pages
- Get Fundamentals of Optimization Theory With Applications To Machine Learning Gallier J. Free All Chapters100% (3)Get Fundamentals of Optimization Theory With Applications To Machine Learning Gallier J. Free All Chapters52 pages
- Linear Optimization and Extensions - Problems and Solutions (PDFDrive)No ratings yetLinear Optimization and Extensions - Problems and Solutions (PDFDrive)450 pages
- (Ebook) Linear and Integer Optimization: Theory and Practice, Third Edition by Sierksma, G.;Zwols, Y. ISBN 9781498743129, 1498743129 instant download100% (1)(Ebook) Linear and Integer Optimization: Theory and Practice, Third Edition by Sierksma, G.;Zwols, Y. ISBN 9781498743129, 1498743129 instant download45 pages
- Fundamentals of optimization theory with applications to machine learning Gallier J. - The newest ebook version is ready, download now to exploreNo ratings yetFundamentals of optimization theory with applications to machine learning Gallier J. - The newest ebook version is ready, download now to explore70 pages
- Full Download Fundamentals of optimization theory with applications to machine learning Gallier J. PDF DOCX100% (2)Full Download Fundamentals of optimization theory with applications to machine learning Gallier J. PDF DOCX65 pages
- Miller S J - Mathematics of Optimization Pure and Applied Undergraduate Texts - 2017100% (2)Miller S J - Mathematics of Optimization Pure and Applied Undergraduate Texts - 2017350 pages
- (Ebook) A Gentle Introduction to Optimization by B. Guenin, J. Könemann, L. Tunçel ISBN 9781107658790, 1107658799 - Download the ebook now and own the full detailed content100% (1)(Ebook) A Gentle Introduction to Optimization by B. Guenin, J. Könemann, L. Tunçel ISBN 9781107658790, 1107658799 - Download the ebook now and own the full detailed content52 pages
- Download full (Ebook) Linear and Integer Optimization: Theory and Practice, Third Edition by Sierksma, G.;Zwols, Y. ISBN 9781498743129, 1498743129 ebook all chaptersNo ratings yetDownload full (Ebook) Linear and Integer Optimization: Theory and Practice, Third Edition by Sierksma, G.;Zwols, Y. ISBN 9781498743129, 1498743129 ebook all chapters81 pages
- Libro - An Introduction To Continuous OptimizationNo ratings yetLibro - An Introduction To Continuous Optimization399 pages
- Mathematical Optimization Linear ProgramNo ratings yetMathematical Optimization Linear Program167 pages
- Download full Solution Manual for An Introduction to Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak all chapters100% (8)Download full Solution Manual for An Introduction to Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak all chapters36 pages
- Solution Manual for An Introduction to Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak - Complete Set Of Chapters Available For One-Click Download100% (4)Solution Manual for An Introduction to Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak - Complete Set Of Chapters Available For One-Click Download50 pages
- Download full (Ebook) Linear Algebra and Optimization for Machine Learning: A Textbook by Charu C. Aggarwal ISBN 9783030403430, 3030403432 ebook all chapters100% (7)Download full (Ebook) Linear Algebra and Optimization for Machine Learning: A Textbook by Charu C. Aggarwal ISBN 9783030403430, 3030403432 ebook all chapters65 pages
- Linear and Integer Optimization (V3C1/F4C1) : Lecture NotesNo ratings yetLinear and Integer Optimization (V3C1/F4C1) : Lecture Notes129 pages
- Access Solution Manual for An Introduction to Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak All Chapters Immediate PDF Download100% (12)Access Solution Manual for An Introduction to Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak All Chapters Immediate PDF Download35 pages
- Instant download (Ebook) Optimization Theory: A Concise Introduction by Jiongmin Yong ISBN 9789813237643, 9813237643 pdf all chapter100% (4)Instant download (Ebook) Optimization Theory: A Concise Introduction by Jiongmin Yong ISBN 9789813237643, 9813237643 pdf all chapter81 pages
- Download full Linear programming with MATLAB 1st Edition Michael C. Ferris ebook all chapters100% (3)Download full Linear programming with MATLAB 1st Edition Michael C. Ferris ebook all chapters81 pages
- (Ebook) Linear programming with MATLAB by Michael C. Ferris, Olvi L. Mangasarian, Stephen J. Wright ISBN 9780898716436, 0898716438download100% (3)(Ebook) Linear programming with MATLAB by Michael C. Ferris, Olvi L. Mangasarian, Stephen J. Wright ISBN 9780898716436, 0898716438download59 pages
- (Ebook) Linear programming with MATLAB by Michael C. Ferris, Olvi L. Mangasarian, Stephen J. Wright ISBN 9780898716436, 0898716438 All Chapters Instant Download100% (3)(Ebook) Linear programming with MATLAB by Michael C. Ferris, Olvi L. Mangasarian, Stephen J. Wright ISBN 9780898716436, 0898716438 All Chapters Instant Download81 pages
- (Book) Fundas of OptTheory Applications To MLNo ratings yet(Book) Fundas of OptTheory Applications To ML832 pages
- Full Download (eBook PDF) Mathematics of Optimization: How to Do Things Faster PDF DOCX100% (9)Full Download (eBook PDF) Mathematics of Optimization: How to Do Things Faster PDF DOCX55 pages
- Large Scale Linear and Integer Optimization: A Unified ApproachNo ratings yetLarge Scale Linear and Integer Optimization: A Unified Approach739 pages
- Linear Algebra and Optimization for Machine Learning: A Textbook Charu C. Aggarwal - The full ebook with all chapters is available for download now100% (2)Linear Algebra and Optimization for Machine Learning: A Textbook Charu C. Aggarwal - The full ebook with all chapters is available for download now56 pages
- Full Solution Manual For An Introduction To Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak All Chapters100% (6)Full Solution Manual For An Introduction To Optimization, 4th Edition Edwin K. P. Chong Stanislaw H. Zak All Chapters44 pages
- Jean Gallier, Jocelyn Quaintance - Linear Algebra and Optimization With Applications To Machine Learning - Volume II - Fundamentals of Optimization Theory With Applications To Machine Learning. 2-Wor100% (1)Jean Gallier, Jocelyn Quaintance - Linear Algebra and Optimization With Applications To Machine Learning - Volume II - Fundamentals of Optimization Theory With Applications To Machine Learning. 2-Wor896 pages
- Finite Mathematics, Models, and Structure: Revised EditionFrom EverandFinite Mathematics, Models, and Structure: Revised EditionNo ratings yet
- Download Complete Diagnostic Ultrasound: Abdomen and Pelvis 2nd Edition Aya Kamaya PDF for All ChaptersNo ratings yetDownload Complete Diagnostic Ultrasound: Abdomen and Pelvis 2nd Edition Aya Kamaya PDF for All Chapters37 pages
- Approach To Infants and Children With Cyanotic Congenital Heart Disease100% (1)Approach To Infants and Children With Cyanotic Congenital Heart Disease6 pages
- External - Turbo HD DVR V4.21.000 Build190722 - Release Notes100% (1)External - Turbo HD DVR V4.21.000 Build190722 - Release Notes4 pages
- Bio Deterioration of Historic Buildings and MonumentsNo ratings yetBio Deterioration of Historic Buildings and Monuments2 pages
- CHAPTER 1 3 Rice Production FINAL RevisedNo ratings yetCHAPTER 1 3 Rice Production FINAL Revised18 pages
- AEROPUERTO PUCALLPA 5-Plataforma Con Mejoramiento-SectionReportNo ratings yetAEROPUERTO PUCALLPA 5-Plataforma Con Mejoramiento-SectionReport4 pages
- Few Questions-Difficulty Level High:: Select The Answer That Best Answers The QuestionNo ratings yetFew Questions-Difficulty Level High:: Select The Answer That Best Answers The Question8 pages
- ecgr2155-experiment-10-time-constant-of-an-rc-circuit (1)No ratings yetecgr2155-experiment-10-time-constant-of-an-rc-circuit (1)7 pages
- PDF Applied Computational Physics 1st Edition Joseph F. Boudreau download100% (1)PDF Applied Computational Physics 1st Edition Joseph F. Boudreau download57 pages
- Active Calculus Activities CH 1-4 (v.12.30.13)No ratings yetActive Calculus Activities CH 1-4 (v.12.30.13)113 pages
- Digital and Social Media Marketing: Assignment 1No ratings yetDigital and Social Media Marketing: Assignment 18 pages
- Homemade Rasam Powder - Rasam Podi Nalini'sKitchenNo ratings yetHomemade Rasam Powder - Rasam Podi Nalini'sKitchen11 pages
