L.

Vandenberghe EE236C (Spring 2016)

15. Conic optimization

• conic linear program

• examples

• modeling

• duality

15-1
 Generalized (conic) inequalities

Conic inequality: a constraint x ∈ K where K is a convex cone in Rm

we will require that the cone K is proper:

• closed
• pointed: K ∩ (−K) = {0}
• with nonempty interior: int K 6= ∅; equivalently, K + (−K) = Rm

Notation (for proper K )

x K y ⇐⇒ x−y ∈K
x K y ⇐⇒ x − y ∈ int K

Conic optimization 15-2

 Inequality notation

we will use a different convention than in EE236B

Vector inequalities: for x, y ∈ Rm

• x > y, x ≥ y denote componentwise inequality

• x  y, x  y denote conic inequality for general (unspecified) proper cone K
• x K y, x K y denote conic inequality for specific K

Matrix inequality: for X, Y ∈ Sp

X  Y, X  Y mean X − Y is positive (semi-)definite

Conic optimization 15-3

 Properties of conic inequalities

preserved by nonnegative linear combinations: if x  y and u  v , then

αx + βu  αy + βv ∀α, β ≥ 0

define a partial order of vectors

• xx
• x  y  z implies x  z
• x  y and y  x imply y = x

in general, not a total order (requires that x  y or y  x for all x, y )

Conic optimization 15-4

 Conic linear program

minimize cT x
subject to Ax  b
Fx = g

• A ∈ Rm×n, F ∈ Rp×n; without loss of generality, we can assume

 
A
rank(F ) = p, rank( )=n
F

• K is a proper cone in Rm

• for K = Rm
+ , problem reduces to regular linear program (LP)

• by defining K = K1 × · · · × Kr , this can represent multiple conic inequalities

A1x K1 b1, ..., Ar x Kr br

Conic optimization 15-5

 Outline

• conic linear program

• examples

• modeling

• duality
 Norm cones

m−1

K = (x, y) ∈ R × R | kxk ≤ y

1
y

1
0 0
−1
0
1 −1 x2
x1

for the Euclidean norm this is the second-order cone (notation: Qm)

Conic optimization 15-6

 Second-order cone program

minimize cT x
subject to kBk0x + dk0k2 ≤ Bk1x + dk1, k = 1, . . . , r

Conic LP formulation: express constraints as Ax K b

   
−B10 d10
 −B11   d11 
   
.
 . 
K = Qm1 × · · · × Qmr ,
 
A= .
. , b= . 
 . 


 −B   d 
 r0   r0 
−Br1 dr1

(assuming Bk0, dk0 have mk − 1 rows)

Conic optimization 15-7

 Vector notation for symmetric matrices

• vectorized symmetric matrix: for U ∈ Sp

√
 
U11 U U
vec(U ) = 2 √ , U21, . . . , Up1, √22 , U32, . . . , Up2, . . . , √pp
2 2 2

• inverse operation: for u = (u1, u2, . . . , un) ∈ Rn with n = p(p + 1)/2

 √ 
2u1 √ u2 ··· up
1  u2 2up+1 · · · u2p−1 
mat(u) = √  ... ... ...

2 √

up u2p−1 ··· 2up(p+1)/2

√
coefficients 2 are added so that standard inner products are preserved:

tr(U V ) = vec(U )T vec(V ), uT v = tr(mat(u) mat(v))

Conic optimization 15-8

 Positive semidefinite cone

S p = {vec(X) | X ∈ Sp+} = {x ∈ Rp(p+1)/2 | mat(x)  0}

z 1

0
1 1
0
−1 0 x
y
  √  
x√ y/ 2
S 2 = (x, y, z)


 y/ 2 0
z

Conic optimization 15-9

 Semidefinite program

minimize cT x
subject to x1A11 + x2A12 + · · · + xnA1n  B1
...
x1Ar1 + x2Ar2 + · · · + xnArn  Br
with Aij , Bi ∈ Spi

Conic LP formulation

K = S p1 × S p2 × · · · × S pr
   
vec(A11) vec(A12) · · · vec(A1n) vec(B1)
 vec(A21) vec(A22) · · · vec(A2n)   vec(B2) 
A=
 ... ... ...
,
 b=
 .
..


vec(Ar1) vec(Ar2) · · · vec(Arn) vec(Br )

Conic optimization 15-10

 Exponential cone

the epigraph of the perspective of exp x is a non-proper cone

n o
K = (x, y, z) ∈ R3 | yex/y ≤ z, y > 0

the exponential cone is Kexp = cl K = K ∪ {(x, 0, z) | x ≤ 0, z ≥ 0}

1
z

2
y 1 0
−1
0−2 x

Conic optimization 15-11

 Geometric program

minimize cT x
ni
exp(aTik x + bik ) ≤ 0,
P
subject to log i = 1, . . . , r
k=1

Conic LP formulation

minimize cT x
 T 
aik x + bik
subject to  1  ∈ Kexp, k = 1, . . . , ni, i = 1, . . . , r
zik
ni
P
zik ≤ 1, i = 1, . . . , m
k=1

Conic optimization 15-12

 Power cone

m
P
Definition: for α = (α1, α2, . . . , αm) > 0 and αi = 1
i=1

Rm xα · · · xα
 1

Kα = (x, y) ∈ + × R | |y| ≤ 1 m
m

Examples for m = 2

α = ( 12 , 12 ) α = ( 23 , 31 ) α = ( 34 , 14 )

0.5 0.5 0.5

y

0 0 0

−0.5 −0.5 −0.5

0 0 0
0.5 0.5 1 0.5 0.5 1 0.5 0.5 1
1 0 1 0 1 0
x1 x2 x1 x2 x1 x2

Conic optimization 15-13

 Cones constructed from convex sets

Inverse image of convex set under perspective

K = {(x, y) ∈ Rn × R | y > 0, x/y ∈ C}

• K ∪ {(0, 0)} is a convex cone if C is a convex set

• cl K is proper if C has nonempty interior, does not contain straight lines

Consequence

any convex constraint x ∈ C can be represented as a conic inequality

x∈C ⇐⇒ (x, 1) ∈ K

(with minor modifications to make K proper)

Conic optimization 15-14

 Cones constructed from functions

Epigraph of perspective of convex function

K = {(x, y, z) ∈ Rn × R × R | y > 0, yf (x/y) ≤ z}

• K ∪ {(0, 0, 0)} is a convex cone if f is convex

• cl K is proper if int dom f 6= ∅, epi f does not contain straight lines

Consequence

can represent any convex constraint f (x) ≤ t as a conic linear inequality

f (x) ≤ t ⇐⇒ (x, 1, t) ∈ K

(with minor modifications to make K proper)

Conic optimization 15-15

 Outline

• conic linear program

• examples

• modeling

• duality
 Modeling software

Modeling packages for convex optimization

• CVX, YALMIP (MATLAB)

• CVXPY, PICOS (Python)
• MOSEK Fusion (different platforms)

assist in formulating convex problems by automating two tasks:

• verifying convexity from convex calculus rules
• transforming problem in input format required by standard solvers

Related packages

general-purpose optimization modeling: AMPL, GAMS

Conic optimization 15-16

 Modeling and conic optimization

Convex modeling systems

• convert problems stated in standard mathematical notation to conic LPs

• in principle, any convex problem can be represented as a conic LP

• in practice, a small set of primitive cones is used (Rn+, Qp, S p)

• choice of cones is limited by available algorithms and solvers (see later)

modeling systems implement set of rules for expressing constraints

f (x) ≤ t

as conic inequalities for the implemented cones

Conic optimization 15-17

 Examples of second-order cone representable functions

• convex quadratic

f (x) = xT P x + q T x + r (P  0)

• quadratic-over-linear function

xT x
f (x, y) = with dom f = Rn × R+ (assume 0/0 = 0)
y

• convex powers with rational exponent

xβ

α x>0
f (x) = |x| , f (x) =
+∞ x ≤ 0

for rational α ≥ 1 and β ≤ 0

• p-norm f (x) = kxkp for rational p ≥ 1

Conic optimization 15-18

 Examples of SD cone representable functions

• matrix-fractional function

f (X, y) = y T X −1y with dom f = {(X, y) ∈ Sn+ × Rn | y ∈ R(X)}

• maximum eigenvalue of symmetric matrix

• maximum singular value f (X) = kXk2 = σ1(X)

 
tI X
kXk2 ≤ t ⇐⇒ 0
XT tI

P
• nuclear norm f (X) = kXk∗ = i σi (X)
 
U X 1
kXk∗ ≤ t ⇐⇒ ∃U, V :  0, (tr U + tr V ) ≤ t
XT V 2

Conic optimization 15-19

 Functions representable with exponential and power cone

Exponential cone

• exponential and logarithm

• entropy f (x) = x log x

Power cone

• increasing power of absolute value: f (x) = |x|p with p ≥ 1

• decreasing power: f (x) = xq with q ≤ 0 and domain R++

• p-norm: f (x) = kxkp with p ≥ 1

Conic optimization 15-20

 Outline

• conic linear program

• examples

• modeling

• duality
 Dual cone

K ∗ = {y | xT y ≥ 0 for all x ∈ K}

Properties (if K is a proper cone)

• K ∗ is a proper cone
• (K ∗)∗ = K
• int K ∗ = {y | xT y > 0 for all x ∈ K , x 6= 0}

Dual inequality: x ∗ y means x K ∗ y for generic proper cone K

Note: dual cone depends on choice of inner product

Conic optimization 15-21

 Examples

• Rp+, Qp, S p are self-dual: K = K ∗

• dual of norm cone is norm cone for dual norm

• dual of exponential cone

∗ +

Kexp = (u, v, w) ∈ R− × R × R | −u log(−u/w) + u − v ≤ 0

(with 0 log(0/w) = 0 if w ≥ 0)

• dual of power cone is

Kα∗ Rm α1 αm

= (u, v) ∈ + × R | |v| ≤ (u1/α1) · · · (um/αm)

Conic optimization 15-22

 Primal and dual conic LP

Primal (optimal value p?)

minimize cT x
subject to Ax  b

Dual (optimal value d?)

maximize −bT z
subject to AT z + c = 0
z ∗ 0

Weak duality: p? ≥ d? (without exception)

Conic optimization 15-23

 Strong duality

Main theorem: p? = d? if primal or dual problem is strictly feasible

Other implications of strict feasibility

• if primal is strictly feasible, then dual optimum is attained (if d? is finite)

• if dual is strictly feasible then primal optimum is attained (if p? is finite)

Compare with linear programming duality (K = Rm

+)

• for an LP, only exception to strong duality is p? = +∞, d? = −∞

• strong duality holds if primal or dual is feasible
• if optimal value is finite then it is attained (in primal and dual)

Conic optimization 15-24

 Example with finite nonzero duality gap

Primal problem
minimize x1
 
0 x1
subject to 0
x1 x 2
x1 ≥ −1

optimal value p? = 0

Dual problem
maximize −z
 
Z11 Z12
subject to  0, z≥0
Z12 Z22
2Z12 + z = 1, Z22 = 0
optimal value d? = −1

Conic optimization 15-25

 Optimality conditions

if strong duality holds, then x and z are optimal if and only if

T
      
0 0 A x c
= + (1)
s −A 0 z b

s  0, z ∗ 0, zT s = 0

Primal feasibility: block 2 of (1) and s  0

Dual feasibility: block 1 of (1) and z ∗ 0

Zero duality gap: inner product of (x, z) and (0, s) gives

z T s = cT x + bT z

Conic optimization 15-26

 Strong theorems of alternative

Strict primal feasibility

exactly one of the following two systems is solvable

1. Ax ≺ b

2. AT z = 0, z 6= 0, z ∗ 0, bT z ≤ 0

Strict dual feasibility

if c ∈ R(AT ), exactly one of the following two systems is solvable

1. Ax K 0, Ax 6= 0, cT x ≤ 0

2. AT z + c = 0, z K ∗ 0

solution of one system is a certificate of infeasibility of the other system

Conic optimization 15-27

 Weak theorems of alternative

Primal feasibility

at most one of the following two systems is solvable

1. Ax  b

2. AT z = 0, z ∗ 0, bT z < 0

Dual feasibility

at most one of the following two systems is solvable

1. Ax  0, cT x < 0

2. AT z + c = 0, z ∗ 0

these are strong alternatives if a constraint qualification holds

Conic optimization 15-28

 Self-dual embeddings

Idea

embed primal, dual conic LPs into a single (self-dual) conic LP, so that:

• embedded problem is feasible, with known feasible points

• from the solution of the embedded problem, primal and dual solutions of
original problem can be constructed, or certificates of primal or dual infeasibility

Purpose: a feasible algorithm applied to the embedded problem

• can detect infeasibility in original problem

• does not require a phase I to find initial feasible points

used by some interior-point solvers

Conic optimization 15-29

 Basic self-dual embedding

minimize 0
T
    
0 0 A c x
subject to  s  =  −A 0 b  z 
κ −cT −bT 0 τ
s  0, κ ≥ 0, z ∗ 0, τ ≥0

• a self-dual conic LP

• has a trivial solution (all variables zero)

• z T s + τ κ = 0 for all feasible points (follows from equality constraint)

• hence, problem is not strictly feasible

Conic optimization 15-30

 Optimality condition for embedded problem

T
    
0 0 A c x
 s  =  −A 0 b  z 
κ −cT −bT 0 τ

s  0, κ ≥ 0, z ∗ 0, τ ≥0

zT s + τ κ = 0

• follows from self-dual property

• a (mixed) linear complementarity problem

Conic optimization 15-31

 Classification of nonzero solution

let s, κ, x, z, τ be a nonzero solution of the self-dual embedding

Case 1: τ > 0, κ = 0

ŝ = s/τ, x̂ = x/τ, ẑ = z/τ

are primal and dual solutions of the conic LPs, i.e., satisfy

T
      
0 0 A x̂ c
= +
ŝ −A 0 ẑ b

ŝ  0, ẑ ∗ 0, ŝT ẑ = 0

Conic optimization 15-32

 Classification of nonzero solution

Case 2: τ = 0, κ > 0; this implies cT x + bT z < 0

• if bT z < 0, then ẑ = z/(−bT z) is a proof of primal infeasibility:

AT ẑ = 0, bT ẑ = −1, ẑ ∗ 0

• if cT x < 0, then x̂ = x/(−cT x) is a proof of dual infeasibility:

Ax̂  0, cT x̂ = −1

Case 3: τ = κ = 0; no conclusion can be made about the original problem

Conic optimization 15-33

 Extended self-dual embedding

minimize θ
T
      
0 0 A c qx x 0
 s   −A 0 b qz 
 z  +  0 
   
 κ  =  −cT
subject to 
−bT
 
0 qτ   τ   0 
0 −qxT −qzT −qτ 0 θ 1

s  0, κ ≥ 0, z ∗ 0, τ ≥0

• qx, qz , qτ chosen so that

(s, κ, x, z, τ, θ) = (s0, 1, x0, z0, 1, z0T s0 + 1)

is feasible, for some given s0  0, x0, z0 ∗ 0

• a self-dual conic LP

Conic optimization 15-34

 Optimality condition

T
      
0 0 A c qx x 0
 s   −A 0 b qz 
 z  +  0 
   
 κ  =  −cT −bT
  
0 qτ   τ   0 
0 −qxT −qzT −qτ 0 θ 1

s  0, κ ≥ 0, z ∗ 0, τ ≥0

zT s + τ κ = 0

• follows from self-dual property

• a (mixed) linear complementarity problem

Conic optimization 15-35

 Properties of extended self-dual embedding

• problem is strictly feasible by construction

• if s, κ, x, z, τ, θ satisfy equality constraint, then

θ = sT z + κτ

(take inner product with (x, z, τ, θ) of each side of the equality)

• at optimum, θ = 0 and problem reduces to the embedding on p.15-30

• classification of p.15-32 also applies to solutions of extended embedding

Conic optimization 15-36

 Reference

A. Ben-Tal and A. Nemirovski, Lectures on Modern Convex Optimization. Analysis, Algorithms, and
Engineering Applications, (2001).

Conic optimization 15-37