Scribd
Upload
83 views

Lifted Newton Optimization

The document describes the lifted Newton method, which "lifts" optimization problems into higher dimensional spaces to exploit problem structure and improve convergence rates. It presents the idea, shows how derivatives can be computed efficiently via an auxiliary function, and applies the method to Gauss-Newton and sequential quadratic programming algorithms. Numerical tests on parameter estimation and PDE problems demonstrate faster convergence of the lifted approaches.

Uploaded by

Andrea Spencer
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
83 views

Lifted Newton Optimization

The document describes the lifted Newton method, which "lifts" optimization problems into higher dimensional spaces to exploit problem structure and improve convergence rates. It presents the idea, shows how derivatives can be computed efficiently via an auxiliary function, and applies the method to Gauss-Newton and sequential quadratic programming algorithms. Numerical tests on parameter estimation and PDE problems demonstrate faster convergence of the lifted approaches.

Uploaded by

Andrea Spencer
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 60

The Lifted Newton Method and Its

Use in Optimization
Moritz Diehl
Optimization in Engineering Center (OPTEC), K.U. Leuven, Belgium
joint work with Jan Albersmeyer (U. Heidelberg)
Overview

 Idea of Lifted Newton Method


 An algorithmic trick to exploit structure "for free"
 Application to optimization
 Convergence analysis
 Numerical Tests

Lifted Newton Method described in:


J.Albersmeyer & M.D. The Lifted Newton Method and Its Application in
Optimization. SIAM J. Opt. (2010) 20:3,1655-1684
Simplified Setting: Root Finding Problem

Aim: solve root finding problem

where is composed of several nonlinear subfunctions:


Simplified Setting: Root Finding Problem

Aim: solve root finding problem

where is composed of several nonlinear subfunctions:

Idea: "Lift" root finding problem into a higher dimensional space…


Lifted Problem

The equivalent, "lifted" problem is:

with
Why to lift and increase the system size?

 Lifting is a generalization of the well-known "multiple shooting"


technique for solution of
• boundary value problems [Osborne 1969]
• parameter estimation problems in ODE [Bock 1987,Schloeder1988]
• optimal control problems [Bock and Plitt 1984, Gill et al. 2000,
Schaefer2005]

 Lifting often offers advantages in terms of


• sparsity exploitation (not today's focus)
• larger region of convergence
• more freedom for initialization (e.g. measurements)
• faster local contraction rate
Motivating Toy Example

 Original scalar root finding problem:

 Lifted formulation yields a nonlinear equation system with 5


equations in 5 variables:

 Compare lifted and unlifted Newton iterates with same initial


guess. Obtain intermediate lifted variables by function evaluation.
Motivating Toy Example

 First iteration is identical, as we initialized identically


 Lifted Newton method converges in 7 instead of 26 iterations!
Lifted Problem is much larger ... is it more expensive?

In each lifted Newton iteration

we have to solve a large linear system:

It is large and structured,..

… but exploitation algorithms are difficult to develop and implement,


cf. Dissertations [Schloeder1988, Schaefer2005] in context of
multiple shooting
Overview

 Idea of Lifted Newton Method


 An algorithmic trick to exploit structure "for free"
 Application to optimization
 Convergence analysis
 Numerical Tests
Algorithmic Trick: Preliminaries 1
Original "user function":

"Lifted residual" (after minor code additions):


Algorithmic Trick: Preliminaries 2

Write

with

in each lifted Newton iteration, we have to solve:

How to solve this linear system efficiently ?


Well-Known Technique: "Condensing"

To solve

can eliminate

and solve "condensed" system in original


degrees of freedom only:
and expand solution again:

But how to compute a, b, A, B efficiently ?


Basis of new trick: an auxiliary function

Define

as implicit function satisfying a perturbed fixed point equation ( by vector d )

PROPOSITION: if then

and

Partial derivatives of Z deliver part of solution of Newton system ! But how to


obtain Z ?
Obtain Z by another minor code modification

 Can show that Z is obtained by one evaluation of the following


function:
Costs of generating and solving linear system
Using Z, can easily compute via directional derivates a, A, b, B as

and then compute the Newton step

Computational effort per iteration:


 computing A & B (in one combined forward sweep)
 factoring B
Extra efforts for lifting:
 computing vector a (one extra directional derivative)
 matrix vector product (could also be done w/o evaluating A)
Overview

 Idea of Lifted Newton Method


 An algorithmic trick to exploit structure "for free"
 Application to optimization
 Convergence analysis
 Numerical Tests
Two lifted algorithms

 Lifted Gauss-Newton Method

 Lifted SQP Method


Lifted Gauss-Newton

 Original Problem:

 Lifted Problem:

( obtained by lifting )
Lifted Gauss-Newton: Quadratic Subproblems
 Linearized lifted problem = structured QP subproblem:

 Condensed QP (solved by dense QP solver):

easily get as lifted derivatives of


Lifted Gauss-Newton: Quadratic Subproblems
 Linearized lifted problem = structured QP subproblem:

 Condensed QP (solved by dense QP solver):

easily get as lifted derivatives of


Lifted SQP Method: Problem Statement

 For notational simplicity, regard first only unconstrained


optimization problem:

 Aim: find root of first order necessary conditions for optimality:


Lifted Problem

 Lifted problem formulation:

 First order necessary conditions for optimality of lifted problem

 Lifted Newton = full space SQP method (with exact Hessian) ?


Gradient evaluation via adjoint differentiation

To compute perform the following code:

function value

gradient
Gradient evaluation via adjoint differentiation

To compute perform the following code:

function value

gradient
Lifted Newton = Full Space SQP

THEOREM: If we lift with respect to all


intermediate variables:

then full space SQP and lifted Newton iterations are identical, with

COROLLARY: Same equivalence holds for lifting of constrained problems:

if we lift Lagrange gradient and constraint:

(need full derivatives only w.r.t. u, not µ, due to symmetry of KKT systems)
Overview

 Idea of Lifted Newton Method


 An algorithmic trick to exploit structure "for free"
 Application to optimization
 Convergence analysis
 Numerical Tests
Why does lifting often improve convergence speed ?
Simple model problem: chain of scalar functions

 Regard a sequence of scalar functions:

 after affine transformations, can assume that solution is zero and

 under these circumstances, the non-lifted function is:


Convergence speed of Non-Lifted Newton

 Derivative is given by

 It is easy to show that non-lifted Newton iterations contract like:

i.e. quadratic convergence with contraction constant


Lifted Newton Convergence

 Lifted residual is

THEOREM: Lifted Newton iterations contract in a "staggered" way:

(if all have the same sign, last variable is "leader" and contracts fastest)
Convergence for motivating toy example
Practical Conclusions from Theorem

Two cases:

 Same curvature  lifted Newton better.


E.g. in toy example, or dynamic simulation codes where several times
• same time stepping function is called
• at similar values

 Opposite curvature  unlifted Newton better.


Example: , e.g. decomposed as

After first iteration, unlifted Newton is already converged, lifted not!


Toy Optimal Control Problem for Illustration

Mildly nonlinear and unstable system.


Use ODE solver with sensitivities (e.g. in ACADO or MUSCOD-II).
Compare unlifted and lifted Gauss-Newton method.
Unlifted Gauss-Newton
Unlifted: First Iteration
Unlifted: Second Iteration
Unlifted: Third Iteration
Unlifted: 4th Iteration
Unlifted: 5th Iteration
Unlifted: 6th Iteration
Unlifted: 7th Iteration (Solution)
Toy Example: Lifted Initialization
Lifted: First Iteration
Lifted: First Iteration
Lifted: Second Iteration
Lifted: 3rd Iteration (already solution!)
Overview

 Idea of Lifted Newton Method


 An algorithmic trick to exploit structure "for free"
 Application to optimization
 Convergence analysis
 Numerical Tests
Large scale Gauss-Newton example 1

 Regard 2-D heat equation with 3 unknown diffusion parameters c

 Discretize on 9 x 9 grid and perform 200 timesteps


( = 16200 hidden variables )

 Measure state u every 10 time steps ( = 1620 measurements )

 Aim: estimate unknown parameters


Compare three Gauss-Newton variants

 Non-lifted Newton with 3 variables only

 Lifted Newton (with 1620 lifted node variables), initialized like


non-lifted variant via a forward simulation

 Lifted Newton, but all nodes initialized with measurements


Convergence for PDE parameter estimation example 1
Large Scale Example 2

 Regard 2-D shallow water wave equation (with unknown parameters)

 Discretize all 3 states u, v, h on 30 x 30 grid and perform 10000


timesteps ( = 27 million variables ! )

 Measure only height h every 100 time steps ( = 90 000


measurements)

 Aim: estimate 2 unknown parameters, water depth H and viscous


friction coefficient b
Large Scale Example 2: Measurements
Large Scale Example 2: Measurements
Large Scale Example 2: Measurements
Large Scale Example 2: Measurements
Compare three Gauss-Newton variants for example 2

 Non-lifted Newton with 2 variables only

 Lifted Newton (with 90 000 lifted variables), initialized like non-


lifted variant via a forward simulation

 Lifted Newton, but initialize height with measurements


Performance of 3 variants for different initial guesses

CPU time per iteration: 8.9 s for unlifted, 11.8 s for lifted (~1 min total)
(Note: formulation e.g. in AMPL would involve 27 million variables)
Summary
 Lifting offers advantages for Newton type optimization:
• faster local convergence rate (proven in simplified setting)
• more freedom for initialization

 Structure exploiting "Lifted Newton Methods" can easily be generated for


any given user functions and any Newton type method:
• only minor code additions
• nearly no additional costs per iteration
• compatible with SQP, BFGS, Gauss-Newton, …
• compatible with any linear solver for condensed systems e.g.
GMRES, cf. [Shimizu, Ohtsuka & D. 2009]

 Code and examples available in C++ package LiftOpt [Albersmeyer&D. 2010]


References

 M.R. Osborne, On shooting methods for boundary value problems,


Journal of Mathematical Analysis and Applications, 27 (1969), pp.
417–433.
 H.G. Bock, Randwertproblemmethoden zur
Parameteridentifizierung in Systemen nichtlinearer
Differentialgleichungen, vol. 183 of Bonner Mathematische
Schriften, University of Bonn, Bonn, 1987.
 J.P. Schloeder, Numerische Methoden zur Behandlung
hochdimensionaler Aufgaben der Parameteridentifizierung, vol. 187 of
Bonner Mathematische Schriften, University of Bonn, Bonn, 1988.
 P. E. Gill, L. O. Jay, M. W. Leonard, L. R. Petzold, and V. Sharma. An
SQP method for the optimal control of large-scale dynamical systems.
Journal Comp. Appl. Math.,120:197–213, 2000.
 A.A.S. Schaefer, Efficient reduced Newton-type methods for solution of
large-scale structured optimization problems with application to
biological and chemical processes, PhD thesis, University of Heidelberg,
2005.
 Y. Shimizu, T. Ohtsuka & M.D, A real-time algorithm for nonlinear
receding horizon control using multiple shooting and continuation/Krylov
method. Int. J. Robust Nonlinear Control (2009) 19:919–936
 J. Albersmeyer & M. D., The Lifted Newton Method and Its
Application in Optimization. SIAM J. Opt. (2010) 20:3,1655-1684

You might also like