Scribd
Upload
12 views

1_intro

The document provides an overview of convex optimization, detailing its mathematical formulation, applications, and the ease of solving such problems compared to nonconvex ones. It highlights the use of domain-specific languages for rapid prototyping and the historical development of algorithms in this field. Convex optimization is emphasized as a powerful tool applicable across various domains, including engineering and machine learning.

Uploaded by

賴裕芳
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views

1_intro

The document provides an overview of convex optimization, detailing its mathematical formulation, applications, and the ease of solving such problems compared to nonconvex ones. It highlights the use of domain-specific languages for rapid prototyping and the historical development of algorithms in this field. Convex optimization is emphasized as a powerful tool applicable across various domains, including engineering and machine learning.

Uploaded by

賴裕芳
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 19

Convex Optimization

Stephen Boyd Lieven Vandenberghe

Revised slides by Stephen Boyd, Lieven Vandenberghe, and Parth Nobel


1. Introduction
Outline

Mathematical optimization

Convex optimization

Convex Optimization Boyd and Vandenberghe 1.1


Optimization problem

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
gi (x) = 0, i = 1, . . . , p

▶ x ∈ Rn is (vector) variable to be chosen (n scalar variables x1 , . . . , xn )


▶ f0 is the objective function, to be minimized
▶ f1 , . . . , fm are the inequality constraint functions
▶ g1 , . . . , gp are the equality constraint functions

▶ variations: maximize objective, multiple objectives, . . .

Convex Optimization Boyd and Vandenberghe 1.2


Finding good (or best) actions

▶ x represents some action, e.g.,


– trades in a portfolio
– airplane control surface deflections
– schedule or assignment
– resource allocation
▶ constraints limit actions or impose conditions on outcome
▶ the smaller the objective f0 (x) , the better
– total cost (or negative profit)
– deviation from desired or target outcome
– risk
– fuel use

Convex Optimization Boyd and Vandenberghe 1.3


Finding good models

▶ x represents the parameters in a model


▶ constraints impose requirements on model parameters (e.g., nonnegativity)
▶ objective f0 (x) is sum of two terms:
– a prediction error (or loss) on some observed data
– a (regularization) term that penalizes model complexity

Convex Optimization Boyd and Vandenberghe 1.4


Worst-case analysis (pessimization)

▶ variables are actions or parameters out of our control


(and possibly under the control of an adversary)
▶ constraints limit the possible values of the parameters
▶ minimizing −f0 (x) finds worst possible parameter values

▶ if the worst possible value of f0 (x) is tolerable, you’re OK


▶ it’s good to know what the worst possible scenario can be

Convex Optimization Boyd and Vandenberghe 1.5


Optimization-based models

▶ model an entity as taking actions that solve an optimization problem


– an individual makes choices that maximize expected utility
– an organism acts to maximize its reproductive success
– reaction rates in a cell maximize growth
– currents in a circuit minimize total power

▶ (except the last) these are very crude models


▶ and yet, they often work very well

Convex Optimization Boyd and Vandenberghe 1.6


Basic use model for mathematical optimization

▶ instead of saying how to choose (action, model) x


▶ you articulate what you want (by stating the problem)
▶ then let an algorithm decide on (action, model) x

Convex Optimization Boyd and Vandenberghe 1.7


Can you solve it?

▶ generally, no
▶ but you can try to solve it approximately, and it often doesn’t matter

▶ the exception: convex optimization


– includes linear programming (LP), quadratic programming (QP), many others
– we can solve these problems reliably and efficiently
– come up in many applications across many fields

Convex Optimization Boyd and Vandenberghe 1.8


Nonlinear optimization

traditional techniques for general nonconvex problems involve compromises

local optimization methods (nonlinear programming)


▶ find a point that minimizes f0 among feasible points near it
▶ can handle large problems, e.g., neural network training
▶ require initial guess, and often, algorithm parameter tuning
▶ provide no information about how suboptimal the point found is

global optimization methods


▶ find the (global) solution
▶ worst-case complexity grows exponentially with problem size
▶ often based on solving convex subproblems

Convex Optimization Boyd and Vandenberghe 1.9


Outline

Mathematical optimization

Convex optimization

Convex Optimization Boyd and Vandenberghe 1.10


Convex optimization

convex optimization problem:

minimize f0 (x)
subject to fi (x) ≤ 0, i = 1, . . . , m
Ax = b

▶ variable x ∈ Rn
▶ equality constraints are linear
▶ f0 , . . . , fm are convex: for 𝜃 ∈ [0, 1] ,

fi (𝜃x + (1 − 𝜃)y) ≤ 𝜃fi (x) + (1 − 𝜃)fi (y)

i.e., fi have nonnegative (upward) curvature

Convex Optimization Boyd and Vandenberghe 1.11


When is an optimization problem hard to solve?

▶ classical view:
– linear (zero curvature) is easy
– nonlinear (nonzero curvature) is hard

▶ the classical view is wrong

▶ the correct view:


– convex (nonnegative curvature) is easy
– nonconvex (negative curvature) is hard

Convex Optimization Boyd and Vandenberghe 1.12


Solving convex optimization problems

▶ many different algorithms (that run on many platforms)


– interior-point methods for up to 10000s of variables
– first-order methods for larger problems
– do not require initial point, babysitting, or tuning
▶ can develop and deploy quickly using modeling languages such as CVXPY
▶ solvers are reliable, so can be embedded
▶ code generation yields real-time solvers that execute in milliseconds
(e.g., on Falcon 9 and Heavy for landing)

Convex Optimization Boyd and Vandenberghe 1.13


Modeling languages for convex optimization

▶ domain specific languages (DSLs) for convex optimization


– describe problem in high level language, close to the math
– can automatically transform problem to standard form, then solve

▶ enables rapid prototyping


▶ it’s now much easier to develop an optimization-based application
▶ ideal for teaching and research (can do a lot with short scripts)

▶ gets close to the basic idea: say what you want, not how to get it

Convex Optimization Boyd and Vandenberghe 1.14


CVXPY example: non-negative least squares

math: CVXPY code:

minimize ∥Ax − b∥ 22 import cvxpy as cp


subject to x⪰0
A, b = ...

▶ variable is x x = cp.Variable(n)
▶ A, b given obj = cp.norm2(A @ x - b)**2
constr = [x >= 0]
▶ x ⪰ 0 means x1 ≥ 0, . . . , xn ≥ 0 prob = cp.Problem(cp.Minimize(obj), constr)
prob.solve()

Convex Optimization Boyd and Vandenberghe 1.15


Brief history of convex optimization

▶ theory (convex analysis): 1900–1970

▶ algorithms
– 1947: simplex algorithm for linear programming (Dantzig)
– 1960s: early interior-point methods (Fiacco & McCormick, Dikin, . . . )
– 1970s: ellipsoid method and other subgradient methods
– 1980s & 90s: interior-point methods (Karmarkar, Nesterov & Nemirovski)
– since 2000s: many methods for large-scale convex optimization

▶ applications
– before 1990: mostly in operations research, a few in engineering
– since 1990: many applications in engineering (control, signal processing, communications,
circuit design, . . . )
– since 2000s: machine learning and statistics, finance

Convex Optimization Boyd and Vandenberghe 1.16


Summary

convex optimization problems


▶ are optimization problems of a special form
▶ arise in many applications
▶ can be solved effectively
▶ are easy to specify using DSLs

Convex Optimization Boyd and Vandenberghe 1.17

You might also like