Review of Optimization Methods

Prof. Manuela Pedio

20550– Quantitative Methods for Finance

August 2018
Outline of the Course
 Lectures 1 and 2 (3 hours, in class):
 Linear and non-linear functions on
 Limits, continuity, differentiability, rules to compute
derivatives, approximation with differentials
 Logarithmic and exponential functions
 Introduction to integration
 Lecture 3 (1.5 hours, in the lab):
 Review of matrix algebra with applications in excel
 Lectures 4 and 5 (3 hours, 1.5 of which in the lab):
 Introduction to optimization: functions of one variable
 Generalization: functions of several variables
 Use of Excel Solver to tackle constrained optimization
2
A Review of Optimization Methods
Optimization: Statement of the Problem (1/2)
 Optimization == maximizing (or minimizing) some objective
function, y = f(x), by picking one or more appropriate values of
the control (aka choice) variable x
 The most common criterion of choice among alternatives in
economics (and finance) is the goal of maximizing something
(like the profit of a firm) or minimizing something (like costs or
risks)
 For instance, think of a risk-averse investor who wants to
maximize a mean-variance objective by picking an
appropriate set of portfolio weights
 Maxima and minima are also called extrema and may be
relative (or local, that is, they represent an extremum in the
neighborhood of the point only) or global
 Key assumption: f(x) is n times continuously differentiable
3
A Review of Optimization Methods
Optimization: Statement of the Problem (2/2)
 In the leftmost graph, optimization is trivial: the function is a
constant and as such all points are at the same time maxima
and minima, in a relative sense
 In the second plot, f(x) is monotonically increasing, there is no
finite maximum, if the set of nonnegative real numbers is the
domain (as the picture implies)
 The points E and F on the right are examples of a relative
(local) extrema
 A function can well have several relative extrema, some of
which may be maxima while others are minima

4
A Review of Optimization Methods
Candidate points: The First-Derivative Test (1/2)
 As a first step we want to identify the “candidate” points to solve
the optimization problem, i.e., all the local extrema
 Indeed, global extrema must also be local extrema or end points
of f(x) on its domain
• If we know all the relative maxima, it is necessary only to select the
largest of these and compare it with the end points in order to
determine the absolute maximum
 Key Result 1 (First-Derivative Test): If a relative extremum of the
function occurs at x = x0, then either f'(x0) = 0, or f'(x0) does not
exist; this is a necessary condition (but NOT sufficient)

5
A Review of Optimization Methods
Candidate points: The First-Derivative Test (2/2)
 Key Result 1 Qualified: If f’(x0) = 0 then the value of f(x0) will be:
(a) A relative maximum if the derivative f'(x) changes its sign
from > 0 to <0 from the immediate left of the point x0 to its
immediate right
(b) A relative minimum if f'(x) changes its sign from negative
to positive from the immediate left of x0 to its immediate right
(c) Neither a relative maximum nor a relative minimum if f'(x)
has the same sign on both the immediate left and right of point
x0 (inflection point)
NOTE: we are assuming that the function
is continuous and possesses continuous
derivatives => for smooth functions,
relative extreme points can occur only
when the first derivative has a zero value Inflection point
6
A Review of Optimization Methods
One Example

7
A Review of Optimization Methods
Concavity, Convexity, and Second-Order Derivatives
 A strictly concave (convex)
function is such that if we pick any
pair of points M and N on the
function and join them by a Inflection point
straight line, the line segment MN
Concave Convex
must lie entirely below (above)
the curve, except at points M and
N

 If the second derivative is negative for all x then

the function is strictly concave
 If the second derivative is positive for all x then the
function is strictly convex
8
A Review of Optimization Methods
The Second-Order Derivative Test
 Key Result 2 (Second-Derivative Test): If the first derivative of a
function at x = x0 is f’(x0) = 0 (first-order, necessary condition),
then f(x0 0 ), will be:

(a) A relative maximum if f"(x0) < 0 Second-order,

(b) A relative minimum if f"(x0) > 0 sufficient condition

 This test is in general more convenient to use than the first-

derivative test, because it does not require us to check the
derivative sign to both the left and the right of x0
 Drawback: this test is inconclusive in the event that f"(x0) = 0
when the stationary value f(x0) can be either a relative
maximum, or a relative minimum, or even an inflection point
• This is what makes the condition sufficient only

Lecture 5 – Review of Optimization Methods 9

Two Examples

Lecture 5 – Review of Optimization Methods 10

Functions with more than one variable
 We are now going to generalize optimization problems to
functions of several variables, i.e.,
 Functions , i.e.,
 In fact, functions from to will be popping up very often
in your future studies
 For instance, the return of a portfolio is a linear function of the returns
of the n assets that compose the portfolio:
𝑟 𝑤 𝑟 𝑤 𝑟 ⋯ 𝑤 𝑟
 Another example is a utility function 𝑈 𝑥 , 𝑥 , … , 𝑥 of a bundle of
consumption goods
 However, we first need to generalize the concept of derivative
to the case of functions of several variables
 This leads us to the introduction partial derivatives and of
Jacobian derivatives
11
A Review of Optimization Methods
Partial derivatives and the Jacobian
Definition: Let . Then for each variable at each
point in the domain of f, the partial
derivative with respect to is

if the limits exists. Only the ith variable changes, while the others
stay constant

 The vector (more generally, matrix) 𝐱𝟎 that collects all

partial derivatives

𝐱𝟎

is called the Jacobian derivative of f at

12
A Review of Optimization Methods
Partial Derivatives: One Example
 Example: consider the function

 Let us compute the partial derivative with respect to

 Simply treat as it was a constant and apply the same rules
of one-variable calculus

 Now let compute the partial derivative with respect to

 The concept can be easily generalized to a function of more

than two variables

13
A Review of Optimization Methods
Second Order Derivatives and Hessians (1/2)
 If the n partial derivative functions of f are continuous
functions at the point in we say that f is continuously
differentiable at
 If all the n partial derivatives are themselves
differentiable we can compute their partial derivatives

 is called the -second order partial derivative of f

and it is generally denoted as

 When then we speak of cross (or mixed) partial

derivatives
 A function of n variables has second order partial
derivatives that are usually arranged into a n x n Hessian
matrix
14
A Review of Optimization Methods
Second Order Derivatives and Hessians (2/2)
 The Hessian matrix is typically denoted as or
and takes the form

 Young’s theorem: the Hessian matrix is a symmetric matrix,

i.e., for each pair of indices i and j
𝜕 𝑓
𝜕𝑥 𝑥
15
A Review of Optimization Methods
One Example
 Consider the function
 Let us compute the Hessian matrix; we already computed
,

 Now we need to compute

 ; ;

 Hessian matrix is

 You can check that

16
A Review of Optimization Methods
Optimization: the case of n-variable functions
 Now we are ready to generalize optimization to the case of n-
variable functions
 The strategy remains looking for critical points (relative
extrema) and then try to isolate global ones among them
 is a critical point for f if it fulfills
,
which means that
( , for each i

 If is an interior point which is a local maximum or

minimum then it is a critical point
 However, the reverse is not true, i.e., the condition is necessary
but not sufficient for an interior point to be a local extremum
17
A Review of Optimization Methods
Checking the sign of the Hessian matrix (1/2)
 As one may guess from the one-variable case, second order
conditions involve checking the sign of the Hessian matrix
 We need to add a definition to the matrix algebra review that
we discussed in the last lecture
• A principle minor of a square matrix A is the determinant of a
submatrix obtained by eliminating some rows and the corresponding
column; the order of a minor is the dimension of the considered
submatrix
• A leading principal minor 𝐴 is a principal minor obtained by
considering the first k rows and columns of the original matrix
• For instance,
𝐴 𝑎
𝑎 𝑎 𝑎
𝐴 𝑎 𝑎 𝑎 𝑎
𝑎 𝑎 𝑎
𝐴 ⇒ the determinant of the
𝑎 𝑎 𝑎
3x3 matrix itself
18
A Review of Optimization Methods
Checking the sign of the Hessian matrix (2/2)
 A square symmetric matrix is said to be
• Positive definite: if all its leading principal minors are
strictly positive
• Negative definite: if and then all its leading
principal minors alternate in sign (but are different from
zero)
• Indefinite: if we have a nonzero leading principal minor and
at least one leading principal minor does not follow the
patterns above
• Positive semidefinite: if every principal minor is
nonnegative
• Negative semidefinite: if and every principal minor
of odd order is and every principal minor of even order
is
19
A Review of Optimization Methods
Sufficient second order conditions
 The sufficient second order conditions for a local extremum
are as follows, given that is an interior critical point:
• If is negative definite => is a local maximum
point
• If is positive definite => is a local minimum
point
• If is indefinite => is a saddle point
 Semidefinite cases require further investigation and we shall
skip their discussion
 When the sign of the Hessian matrix does not depend on , the
local extrema are also global because when the Hessian is
positive (negative) definite over the entire domain the function
is strictly convex (concave)
20
A Review of Optimization Methods
Example of Unconstrained Optimization (1/2)
 Study the optimization of the following function:

 Step 1: find the internal critical points

 12𝑥 6𝑥𝑦 0

 3𝑥 3𝑦 0
 Solving that is non-trivial and time consuming
 You get three critical points:
 A(0,0); B( , ; C( , )
 Step 2: compute the Hessian matrix
36𝑥 6𝑦 6𝑥
 𝐷 𝑓
6𝑥 6𝑦
 We need to check it at A, B, and C
21
A Review of Optimization Methods
Example of Unconstrained Optimization (2/2)
 As an example, I will only check the sign of the Hessian at
C( )
9 3 3 6 3
𝐷 𝑓
3 3 3 3
 𝐴 6 0
 𝐴 6𝑥3 3 3 9 0
 Then the Hessian matrix is positive definite and the point is a local
minimum
 This is an easy problem and yet you see how computationally intense
it is
 Sometimes the solution shall ben find numerically anyway
 Things get even worse when we introduce constraints (equality
constraints, inequality constraints or both)
 We shall now introduce Excel solver
22
A Review of Optimization Methods
Excel Solver (1/3)
 The Solver is an analysis tool available as an additional package
into Excel
 If you do not have it already installed in your Excel you can
download it as Excel add-in from the Excel options
 Once you have installed it, you find it under the tab “Data”
 The Solver is able to solve optimization problems for you (even
with a number of equality/inequality constraints, as we shall
see later on)
 Essentially, it maximizes (minimizes) the value obtained into an
objective cell in which you have to specify a certain function…
 …by changing a set of cells (control variables) that you specify
elsewhere as an array in the worksheet
23
A Review of Optimization Methods
Excel Solver (2/3)
OBJECTIVE FUNCTION:
 In the cell A6 write a value (almost whatever) for x and in
B6 write a value for y
 The values are only used to initialize the search (in some
situations it may matter where you initialize the search, but
typically the nature of the problem that you are solving
suggests reasonable values)
 For instance, when finding optimal weights for a
portfolio I typically start from equal weights
 Then write the functions f
into the cell C6

24
A Review of Optimization Methods
Excel Solver (3/3)
 Now open the Solver:

OBJECTIVE CELL
Choose if you
want to look
CONTROLS
for a min or
for a max

For now we
are not
putting
constraints

SOLVE IT
25
A Review of Optimization Methods
Hints of Constrained Optimization (1/2)
 Up to these points, all control variables have been independent
of each other: the decision made regarding one variable does
not impinge upon the choices of the remaining variables
• E.g., a two-product firm can choose any value for Q1 and any Q2 it
wishes, without the two choices limiting each other
• If the firm in the example is somehow required to fulfill a restri-
ction (e.g., a production quota) in the form of Q1 + Q2 = k, how-
ever, the independence between the choice variables will be lost
• The new optimum satisfying
the production quota constitu-
tes a constrained optimum, which,
in general, may be expected to
differ from the free optimum
 Key Result : A constrained maxi-
mum can never exceed the free
maximum
26
A Review of Optimization Methods
Hints of Constrained Optimization (2/2)
 In general, a constrained maximum can be expected to achieve a
lower value than the free maximum, although, by coincidence, the
two maxima may happen to have the same value
o We had added another constraint intersecting the first constraint at
a single point in the xy plane, the two constraints together would
have restricted the domain to that single point
o Then the locating of the extremum would become a trivial matter
• In a meaningful problem, the number and the nature of the
constraints should be such as to restrict, but not eliminate, the
possibility of choice
o Generally, the number of constraints should be less than the
number of choice variables
 Under C < N equality constraints, when we can write a sub-set
of the choice variables as an explicit function of all others, the
former can be substituted out:
, ,…,
27
A Review of Optimization Methods
Hint: Lagrange Multiplier Method (1/2)
becomes:
, ,…,
an unconstrained problem
 However, the direct substitution method cannot be applied
when the C constraints do not allow us to re-write the objective
functions in N – C free control variables
• Even if some of the variables become implicit functions of others,
it would be complex to proceed because the objective would
become “highly composite”
 In such cases, we often resort to the method of Lagrange
(undetermined) multipliers
• The goal is to convert a constrained extremum problem into a
form such that the first-order condition of the free extremum
problem can still be applied
• For instance, consider an objective function z = f(x.y) subject to
the constraint g(x,y)=c where c is a constant
28
A Review of Optimization Methods
Lagrange Multiplier Method (2/2)
 The Lagrangian problem is:
max 𝑓 𝑥, 𝑦 𝜆𝑐 𝑔 𝑥, 𝑦
, ,

 The necessary FOC is then:

The stationary values of the
Lagrangian function Z will
automatically satisfy the constraint

The optimal value 𝜆*

provides a measure of
the sensitivity of the
Lagrangian function to a
shift of the constraint

29
A Review of Optimization Methods
Excel Solver for constrained optimization (1/2)
 Lagrange multiplier method gives us critical (candidate) points,
but also in this case we need to check second order conditions
 This requires to check positive (negative) definiteness of large
square matrices (bordered Hessian matrices) – we shall skip
the details
 Again, the Excel Solver may help us!
 First of all, the Solver easily solves “Kuhn-Tucker” type of
problems (that is, problems in which choice variables are
restricted to only take positive values)
 It is sufficient to tick the box “make all unconstrained variable
positive)
 Try with the problem we did before (the new minimum will be
at (0,0), quite intuitively)
30
A Review of Optimization Methods
Excel Solver for constrained optimization (2/2)
 Now let us add constraints to our problem


 Click on “add” to add equality / inequality constraints

31
A Review of Optimization Methods
Exercises
 Solve the following exercises using Excel Solver

o Maximize the function

o Minimize the function

o Minimize the function

32
A Review of Optimization Methods