Convex Optimization

Stephen Boyd Lieven Vandenberghe

Revised slides by Stephen Boyd, Lieven Vandenberghe, and Parth Nobel

8. Geometric problems
 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.1

 Minimum volume ellipsoid around a set
▶ Löwner-John ellipsoid of a set C: minimum volume ellipsoid E with C ⊆ E
▶ parametrize E as E = {v | ∥Av + b∥ 2 ≤ 1}; can assume A ∈ Sn++
▶ vol E is proportional to det A−1 ; to find Löwner-John ellipsoid, solve problem

minimize (over A, b) log det A−1

subject to supv∈C ∥Av + b∥ 2 ≤ 1

convex, but evaluating the constraint can be hard (for general C)

▶ finite set C = {x1 , . . . , xm }:

minimize (over A, b) log det A−1

subject to ∥Axi + b∥ 2 ≤ 1, i = 1, . . . , m

also gives Löwner-John ellipsoid for polyhedron conv{x1 , . . . , xm }

Convex Optimization Boyd and Vandenberghe 8.2

 Maximum volume inscribed ellipsoid
▶ maximum volume ellipsoid E with E ⊆ C, C ⊆ Rn convex
▶ parametrize E as E = {Bu + d | ∥u∥ 2 ≤ 1}; can assume B ∈ Sn++
▶ vol E is proportional to det B; can find E by solving

maximize log det B

subject to sup ∥u∥ 2 ≤1 IC (Bu + d) ≤ 0

(where IC (x) = 0 for x ∈ C and IC (x) = ∞ for x ∉ C)

convex, but evaluating the constraint can be hard (for general C)
▶ polyhedron {x | aTi x ≤ bi , i = 1, . . . , m}:

maximize log det B

subject to ∥Bai ∥ 2 + aTi d ≤ bi , i = 1, . . . , m

(constraint follows from sup ∥u∥ 2 ≤1 aTi (Bu + d) = ∥Bai ∥ 2 + aTi d)

Convex Optimization Boyd and Vandenberghe 8.3
 Efficiency of ellipsoidal approximations

▶ C ⊆ Rn convex, bounded, with nonempty interior

▶ Löwner-John ellipsoid, shrunk by a factor n (around its center), lies inside C
▶ maximum volume inscribed ellipsoid, expanded by a factor n (around its center) covers C
▶ example (for polyhedra in R2 )

√
▶ factor n can be improved to n if C is symmetric

Convex Optimization Boyd and Vandenberghe 8.4

 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.5

 Centering
▶ many possible definitions of ‘center’ of a convex set C

▶ Chebyshev center: center of largest inscribed ball

– for polyhedron, can be found via linear programming

▶ center of maximum volume inscribed ellipsoid

– invariant under affine coordinate transformations

xcheb xmve

Convex Optimization Boyd and Vandenberghe 8.6

 Analytic center of a set of inequalities

▶ the analytic center of set of convex inequalities and linear equations

fi (x) ≤ 0, i = 1, . . . , m, Fx = g

is defined as solution of
− m
Í
minimize i=1 log(−fi (x))
subject to Fx = g
▶ objective is called the log-barrier for the inequalities
▶ (we’ll see later) analytic center more easily computed than MVE or Chebyshev center

▶ two sets of inequalities can describe the same set, but have different analytic centers

Convex Optimization Boyd and Vandenberghe 8.7

 Analytic center of linear inequalities
▶ aTi x ≤ bi , i = 1, . . . , m
▶ xac minimizes 𝜙(x) = − m T
Í
i=1 log(bi − ai x)
▶ dashed lines are level curves of 𝜙

xac

Convex Optimization Boyd and Vandenberghe 8.8

 Inner and outer ellipsoids from analytic center

▶ we have
Einner ⊆ {x | aTi x ≤ bi , i = 1, . . . , m} ⊆ Eouter
where

Einner = {x | (x − xac ) T ∇2 𝜙(xac ) (x − xac ) ≤ 1}

Eouter {x | (x − xac ) T ∇2 𝜙(xac ) (x − xac ) ≤ m(m − 1)}
=
√︁
▶ ellipsoid expansion/shrinkage factor is m(m − 1)
(cf. n for Löwner-John or max volume inscribed ellpsoids)

Convex Optimization Boyd and Vandenberghe 8.9

 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.10

 Linear discrimination
▶ separate two sets of points {x1 , . . . , xN }, {y1 , . . . , yM } by a hyperplane
▶ i.e., find a ∈ Rn , b ∈ R with
aT xi + b > 0, i = 1, . . . , N, aT yi + b < 0, i = 1, . . . , M
▶ homogeneous in a, b, hence equivalent to
aT xi + b ≥ 1, i = 1, . . . , N, aT yi + b ≤ −1, i = 1, . . . , M
a set of linear inequalities in a, b, i.e., an LP feasibility problem

Convex Optimization Boyd and Vandenberghe 8.11

 Robust linear discrimination

(Euclidean) distance between hyperplanes

H1 = {z | aT z + b = 1}
H2 = {z | aT z + b = −1}

is dist(H1 , H2 ) = 2/∥a∥ 2

to separate two sets of points by maximum margin,

minimize (1/2) ∥a∥ 22

subject to aT xi + b ≥ 1, i = 1, . . . , N (2)
aT yi + b ≤ −1, i = 1, . . . , M

a QP in a, b

Convex Optimization Boyd and Vandenberghe 8.12

 Approximate linear separation of non-separable sets

minimize 1T u + 1T v
subject to aT xi + b ≥ 1 − ui , i = 1, . . . , N, aT yi + b ≤ −1 + vi , i = 1, . . . , M
u ⪰ 0, v ⪰ 0

▶ an LP in a, b, u, v
▶ at optimum, ui = max{0, 1 − aT xi − b}, vi = max{0, 1 + aT yi + b}
▶ equivalent to minimizing the sum of violations of the original inequalities

Convex Optimization Boyd and Vandenberghe 8.13

 Support vector classifier

minimize ∥a∥ 2 + 𝛾(1T u + 1T v)

subject to aT xi + b ≥ 1 − ui , i = 1, . . . , N
aT yi + b ≤ −1 + vi , i = 1, . . . , M
u ⪰ 0, v ⪰ 0
produces point on trade-off curve between inverse of margin 2/∥a∥ 2 and classification error,
measured by total slack 1T u + 1T v

example on previous slide, with 𝛾 = 0.1:

Convex Optimization Boyd and Vandenberghe 8.14

 Nonlinear discrimination

▶ separate two sets of points by a nonlinear function f : find f : Rn → R with

f (xi ) > 0, i = 1, . . . , N, f (yi ) < 0, i = 1, . . . , M

▶ choose a linearly parametrized family of functions f (z) = 𝜃 T F(z)
– 𝜃 ∈ Rk is parameter
– F = (F1 , . . . , Fk ) : Rn → Rk are basis functions

▶ solve a set of linear inequalities in 𝜃 :

𝜃 T F(xi ) ≥ 1, i = 1, . . . , N, 𝜃 T F(yi ) ≤ −1, i = 1, . . . , M

Convex Optimization Boyd and Vandenberghe 8.15

 Examples

▶ quadratic discrimination: f (z) = zT Pz + qT z + r, 𝜃 = (P, q, r)

▶ solve LP feasibility problem with variables P ∈ Sn , q ∈ Rn , r ∈ R

xiT Pxi + qT xi + r ≥ 1, yTi Pyi + qT yi + r ≤ −1

▶ can add additional constraints (e.g., P ⪯ −I to separate by an ellipsoid)

▶ polynomial discrimination: F(z) are all monomials up to a given degree d

▶ e.g., for n = 2, d = 3

F(z) = (1, z1 , z2 , z21 , z1 z2 , z22 , z31 , z21 z2 , z1 z22 , z32 )

Convex Optimization Boyd and Vandenberghe 8.16

 Example

separation by ellipsoid separation by 4th degree polynomial

Convex Optimization Boyd and Vandenberghe 8.17

 Outline

Extremal volume ellipsoids

Centering

Classification

Placement and facility location

Convex Optimization Boyd and Vandenberghe 8.18

 Placement and facility location

▶ N points with coordinates xi ∈ R2 (or R3 )

▶ some positions xi are given; the other xi ’s are variables

▶ for each pair of points, a cost function fij (xi , xj )

▶ placement problem: minimize

Í
i≠j fij (xi , xj )

▶ interpretations
– points are locations of plants or warehouses; fij is transportation cost between facilities i and j
– points are locations of cells in an integrated circuit; fij represents wirelength

Convex Optimization Boyd and Vandenberghe 8.19

 Example
▶ minimize (i,j) ∈ E h(∥xi − xj ∥ 2 ) , with 6 free points, 27 edges
Í

▶ optimal placements for h(z) = z, h(z) = z2 , h(z) = z4

1 1 1

0 0 0

−1 −1 −1
−1 0 1 −1 0 1 −1 0 1
▶ histograms of edge lengths ∥xi − xj ∥ 2 , (i, ) ∈ E
4 4 6
5
3 3 4
2 2 3
2
1 1
1
00 0.5 1 1.5 2 00 0.5 1 1.5 00 0.5 1 1.5
Convex Optimization Boyd and Vandenberghe 8.20