Advances in Numerical Mathematics

Edited by
Prof. Dr. Dres. h. c. Hans Georg Bock
Prof. Dr. Dr. h. c. Wolfgang Hackbusch
Prof. Mitchell Luskin
Prof. Dr. Dr. h. c. Rolf Rannacher
Andreas Potschka

A Direct Method for

Parabolic PDE Con-
strained Optimization
Problems
Andreas Potschka
Heidelberg, Germany

Dissertation Heidelberg University, Germany, 2011

ISSN 1616-2994
ISBN 978-3-658-04475-6 ISBN 978-3-658-04476-3 (eBook)
DOI 10.1007/978-3-658-04476-3

The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie;

detailed bibliographic data are available in the Internet at http://dnb.d-nb.de.

Library of Congress Control Number: 2013955622

Springer Spektrum
© Springer Fachmedien Wiesbaden 2014
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole
or part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical
way, and transmission or information storage and retrieval, electronic adaptation, compu-
ter software, or by similar or dissimilar methodology now known or hereafter developed.
Exempted from this legal reservation are brief excerpts in connection with reviews or schol-
arly analysis or material supplied specifically for the purpose of being entered and executed
on a computer system, for exclusive use by the purchaser of the work. Duplication of this
publication or parts thereof is permitted only under the provisions of the Copyright Law of
the Publisher’s location, in its current version, and permission for use must always be obtained
from Springer. Permissions for use may be obtained through RightsLink at the Copyright
Clearance Center. Violations are liable to prosecution under the respective Copyright Law.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this
publication does not imply, even in the absence of a specific statement, that such names are
exempt from the relevant protective laws and regulations and therefore free for general use.
While the advice and information in this book are believed to be true and accurate at the date
of publication, neither the authors nor the editors nor the publisher can accept any legal re-
sponsibility for any errors or omissions that may be made. The publisher makes no warranty,
express or implied, with respect to the material contained herein.

Printed on acid-free paper

Springer Spektrum is a brand of Springer DE.

Springer DE is part of Springer Science+Business Media.
www.springer-spektrum.de
Acknowledgments
I submitted this mathematical doctoral thesis in August 2011 to the faculty of nat-
ural sciences and mathematics of Heidelberg University and successfully passed
its disputation on December 19, 2011.
I acknowledge the support by Deutsche Forschungsgemeinschaft (DFG) within
the Internationales Graduiertenkolleg 710 Complex processes: Modeling, Simula-
tion and Optimization, under grant BO864/12-1 within the Schwerpunktprogramm
1253 Optimization with Partial Differential Equations, and within the Heidelberg
Graduate School for Mathematical and Computational Methods in the Sciences
(HGS). I am also grateful for support by Bundesministerium für Bildung und
Forschung (BMBF) under grant 03BONCHD. Furthermore I would like to ac-
knowledge support by the European Commission within the project Embedded
Optimization for Resource Constrained Platforms (EMBOCON).
I would especially like to thank: my advisers Hans Georg Bock and Rolf Ran-
nacher; my mentors Sebastian Sager and Johannes Schlöder; my collaborators
Jan Van Impe, Sebastian Engell, Moritz Diehl, Stefan Körkel, Ekaterina Kostina,
Achim Küpper, and Filip Logist; my research stay hosts Moritz Diehl, Andreas
Griewank, Matthias Heinkenschloss, Nick Trefethen, and Fredi Tröltzsch; my
colleagues Jan Albersmeyer, Dörte Beigel, Kathrin Hatz, Christian Hoffmann,
Christian Kirches, Simon Lenz, Andreas Sommer, Andreas Schmidt, and Leo
Wirsching; my rocking friends Tommy Binford, Eddie Castillo, Rick Castillo,
Roger Fletcher, Sean Hardesty, Denis Ridzal; my very close friends Tom Kraus,
Anita and Micha Strauß, Evelyn and Tobias Stuwe; my siblings Susanne and
Manuel; and my parents Maria and Michael.
I am indebted to Falk Hante, Christian Kirches, Mario Mommer, Sebastian
Sager, and Johannes Schlöder for valuable comments on a first draft of this thesis.

Andreas Potschka
Abstract
In this thesis we develop a numerical method based on Direct Multiple Shooting
for Optimal Control Problems (OCPs) constrained by time-periodic Partial Differ-
ential Equations (PDEs). The proposed method features asymptotically optimal
scale-up of the numerical effort with the number of spatial discretization points.
It consists of a Linear Iterative Splitting Approach (LISA) within a Newton-type
iteration with globalization on the basis of natural level functions. We investigate
the LISA-Newton method in the framework of Bock’s κ-theory and develop reli-
able a-posteriori κ-estimators. Moreover we extend the inexact Newton method to
an inexact Sequential Quadratic Programming (SQP) method for inequality con-
strained problems and provide local convergence theory. In addition we develop
a classical and a two-grid Newton-Picard preconditioner for LISA and prove grid-
independent convergence of the classical variant for a model problem. Based on
numerical results we can claim that the two-grid version is even more efficient
than the classical version for typical application problems. Moreover we develop
a two-grid approximation for the Lagrangian Hessian which fits well in the two-
grid Newton-Picard framework and yields a reduction of 68 % in runtime for a
nonlinear benchmark problem compared to the use of the exact Lagrangian Hes-
sian. We show that the quality of the fine grid controls the accuracy of the solution
while the quality of the coarse grid determines the asymptotic linear convergence
rate, i.e., Bock’s κ. Based on reliable κ-estimators we facilitate automatic coarse
grid refinement to guarantee fast convergence. For the solution of the occurring
large-scale Quadratic Programming Problems (QPs) we develop a structure ex-
ploiting two-stage approach. In the first stage we exploit the Multiple Shooting
and Newton-Picard structure to reduce the large-scale QP to an equivalent QP
whose size is independent of the number of spatial discretization points. For the
second stage we develop extensions for a Parametric Active Set Method (PASM)
to achieve a reliable and efficient solver for the resulting, possibly nonconvex QP.
Furthermore we construct three illustrative, counter-intuitive toy examples which
show that convergence of a one-shot one-step optimization method is neither nec-
essary nor sufficient for the convergence of the forward problem method. For three
regularization approaches to recover convergence our analysis shows that de-facto
loss of convergence cannot be avoided with these approaches. We have further
implemented the proposed methods within a code called MUSCOP which features
VIII Abstract

automatic derivative generation for the model functions and dynamic system solu-
tions of first and second order, parallelization on the Multiple Shooting structure,
and a hybrid language programming paradigm to minimize setup and solution time
for new application problems. We demonstrate the applicability, reliability, and
efficiency of MUSCOP and thus the proposed numerical methods and techniques
on a sequence of PDE OCPs of growing difficulty ranging from linear academic
problems, over highly nonlinear academic problems of mathematical biology to
a highly nonlinear real-world chemical engineering problem in preparative chro-
matography: The Simulated Moving Bed (SMB) process.
Contents

Acknowledgments V

Abstract VII

1 Introduction 1
1.1 Results of this thesis . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Thesis overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

I Theoretical foundations 9
2 Problem formulation 11
2.1 Dynamical models described by Partial Differential Equations . . 11
2.2 Coupled ODEs and PDEs . . . . . . . . . . . . . . . . . . . . . . 15
2.3 The Optimal Control Problem . . . . . . . . . . . . . . . . . . . 15

3 Direct Optimization: Problem discretization 19

3.1 Discretize-then-optimize approach . . . . . . . . . . . . . . . . . 19
3.2 Method Of Lines: Discretization in space . . . . . . . . . . . . . 20
3.3 Direct Multiple Shooting: Discretization in time . . . . . . . . . . 23
3.4 Discretization of path constraints . . . . . . . . . . . . . . . . . . 25
3.5 The resulting Nonlinear Programming Problem . . . . . . . . . . 25

4 Elements of optimization theory 27

4.1 Basic definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
4.2 Necessary optimality conditions . . . . . . . . . . . . . . . . . . 28

II Numerical methods 31
5 Inexact Sequential Quadratic Programming 33
5.1 Newton-type methods . . . . . . . . . . . . . . . . . . . . . . . . 33
5.2 Local convergence . . . . . . . . . . . . . . . . . . . . . . . . . 36
X Contents

5.3 Globalization of convergence . . . . . . . . . . . . . . . . . . . . 38

5.3.1 Generalized level functions . . . . . . . . . . . . . . . . . 39
5.3.2 The Newton path . . . . . . . . . . . . . . . . . . . . . . 40
5.3.3 The natural level function and the Natural Monotonicity Test 41
5.4 A Rosenbrock-type example . . . . . . . . . . . . . . . . . . . . 45
5.5 The Restrictive Monotonicity Test . . . . . . . . . . . . . . . . . 46
5.6 Natural Monotonicity for LISA-Newton methods . . . . . . . . . 46
5.6.1 Estimates for natural monotonicity . . . . . . . . . . . . . 47
5.6.2 A Linear Iterative Splitting Approach . . . . . . . . . . . 54
5.6.3 GINKO Algorithm . . . . . . . . . . . . . . . . . . . . . 67
5.7 Inequality constrained optimization problems . . . . . . . . . . . 68
5.7.1 SQP with exact derivatives . . . . . . . . . . . . . . . . . 68
5.7.2 Inexact SQP . . . . . . . . . . . . . . . . . . . . . . . . 72

6 Newton-Picard preconditioners 75
6.1 The Newton-Picard method for finding periodic steady states . . . 76
6.2 Discretization of the model problem . . . . . . . . . . . . . . . . 77
6.3 Newton-Picard for optimal control problems . . . . . . . . . . . . 80
6.3.1 General considerations . . . . . . . . . . . . . . . . . . . 81
6.3.2 Simultaneous Newton-Picard iteration . . . . . . . . . . . 81
6.3.3 Convergence for classical Newton-Picard . . . . . . . . . 85
6.3.4 Numerical solution of the approximated linear system . . 90
6.3.5 Pseudocode . . . . . . . . . . . . . . . . . . . . . . . . . 91
6.3.6 Algorithmic complexity . . . . . . . . . . . . . . . . . . 95
6.4 Extension to nonlinear problems and Multiple Shooting . . . . . . 97

7 One-shot one-step methods and their limitations 99

7.1 Illustrative, counter-intuitive examples in low dimensions . . . . . 101
7.1.1 Fast forward convergence, optimization divergence . . . . 102
7.1.2 Forward divergence, fast optimization convergence . . . . 102
7.2 Subproblem regularization . . . . . . . . . . . . . . . . . . . . . 103
7.3 Analysis of the regularized subproblems . . . . . . . . . . . . . . 104
7.3.1 Primal regularization . . . . . . . . . . . . . . . . . . . . 105
7.3.2 Primal-dual regularization . . . . . . . . . . . . . . . . . 106
7.3.3 Hemi-primal regularization . . . . . . . . . . . . . . . . . 107
7.3.4 Divergence and de facto loss of convergence for subprob-
lem regularizations . . . . . . . . . . . . . . . . . . . . . 109

8 Condensing 111
Contents XI

8.1 Multiple shooting structure . . . . . . . . . . . . . . . . . . . . . 111

8.2 Newton-Picard structure . . . . . . . . . . . . . . . . . . . . . . 115
8.3 Elimination of discretized PDE states . . . . . . . . . . . . . . . 118
8.4 Newton-Picard Hessian approximation . . . . . . . . . . . . . . . 120
8.5 Scaling invariance of the Newton-Picard LISA-Newton method . . 121

9 A Parametric Active Set method for QP solution 123

9.1 General remarks on Quadratic Programming Problems . . . . . . 123
9.1.1 Optimality conditions . . . . . . . . . . . . . . . . . . . 124
9.1.2 Existing methods . . . . . . . . . . . . . . . . . . . . . . 124
9.1.3 Existing software . . . . . . . . . . . . . . . . . . . . . . 125
9.2 Parametric Active Set methods . . . . . . . . . . . . . . . . . . . 127
9.2.1 The Parametric QP . . . . . . . . . . . . . . . . . . . . . 127
9.2.2 The Parametric Quadratic Programming algorithm . . . . 127
9.2.3 Far bounds . . . . . . . . . . . . . . . . . . . . . . . . . 132
9.3 Fundamental numerical challenges . . . . . . . . . . . . . . . . . 133
9.4 Strategies to meet numerical challenges . . . . . . . . . . . . . . 134
9.4.1 Rounding errors and ill-conditioning . . . . . . . . . . . . 134
9.4.2 Comparison with zero . . . . . . . . . . . . . . . . . . . 136
9.4.3 Cycling and ties . . . . . . . . . . . . . . . . . . . . . . 137
9.5 The code rpasm: A PASM in Matlab® . . . . . . . . . . . . . . . 139
9.6 Comparison with existing software . . . . . . . . . . . . . . . . . 139
9.6.1 Criteria for comparison . . . . . . . . . . . . . . . . . . . 139
9.6.2 Discussion of numerical results . . . . . . . . . . . . . . 140
9.7 Drawbacks of the proposed PASM . . . . . . . . . . . . . . . . . 142
9.8 Nonconvex Quadratic Programs . . . . . . . . . . . . . . . . . . 143

10 Automatic derivative generation 145

10.1 Algorithmic Differentiation . . . . . . . . . . . . . . . . . . . . . 146
10.2 The principle of IND . . . . . . . . . . . . . . . . . . . . . . . . 147
10.3 IND for implicit time-stepping with monitor strategy . . . . . . . 148
10.4 Numerical effort of IND . . . . . . . . . . . . . . . . . . . . . . 151

11 The software package MUSCOP 153

11.1 Programming paradigms . . . . . . . . . . . . . . . . . . . . . . 153
11.1.1 Hybrid language programming . . . . . . . . . . . . . . . 153
11.1.2 No data encapsulation . . . . . . . . . . . . . . . . . . . 154
11.1.3 Algorithm centered not model centered . . . . . . . . . . 155
11.1.4 Reverse Communication interface . . . . . . . . . . . . . 155
XII Contents

11.2 Orchestration of software components . . . . . . . . . . . . . . . 157

11.2.1 Function and derivative evaluation . . . . . . . . . . . . . 157
11.2.2 Condensing and condensed QP solution . . . . . . . . . . 158
11.2.3 Estimation of κ and ω . . . . . . . . . . . . . . . . . . . 159
11.2.4 Automatic mesh refinement . . . . . . . . . . . . . . . . 159

III Applications and numerical results 161

12 Linear boundary control for the periodic 2D heat equation 163
12.1 General parameters . . . . . . . . . . . . . . . . . . . . . . . . . 163
12.2 Euclidean vs. L2 projector . . . . . . . . . . . . . . . . . . . . . 164
12.3 Mesh independence . . . . . . . . . . . . . . . . . . . . . . . . . 167
12.4 Comparison with Schur complement preconditioning . . . . . . . 169

13 Nonlinear boundary control of the periodic 1D heat equation 171

13.1 Problem and algorithmical parameters . . . . . . . . . . . . . . . 171
13.2 Discussion of numerical convergence . . . . . . . . . . . . . . . 172
13.2.1 Grid refinements . . . . . . . . . . . . . . . . . . . . . . 172
13.2.2 Computation times . . . . . . . . . . . . . . . . . . . . . 174
13.2.3 Exact Hessian vs. two-grid approximation . . . . . . . . . 175
13.2.4 Refinement of control in time . . . . . . . . . . . . . . . 177

14 Optimal control for a bacterial chemotaxis system 181

14.1 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . 181
14.2 Numerical results . . . . . . . . . . . . . . . . . . . . . . . . . . 182

15 Optimal control of a Simulated Moving Bed process 187

15.1 Mathematical modeling of adsorption processes . . . . . . . . . . 188
15.1.1 General Rate Model . . . . . . . . . . . . . . . . . . . . 188
15.1.2 Mass balances . . . . . . . . . . . . . . . . . . . . . . . 191
15.1.3 Objective and constraints . . . . . . . . . . . . . . . . . . 192
15.2 Numerical results . . . . . . . . . . . . . . . . . . . . . . . . . . 192

16 Conclusions and future work 199

Bibliography 201
List of acronyms
AD Algorithmic Differentiation

BDF Backward Differentiation Formula

BVP Boundary Value Problem

ECOP Equality Constrained Optimization Problem

END External Numerical Differentiation

FDM Finite Difference Method

FEM Finite Element Method

FVM Finite Volume Method

IND Internal Numerical Differentiation

IRAM Implicitly Restarted Arnoldi Method

IVP Initial Value Problem

KKT Karush-Kuhn-Tucker

LICQ Linear Independence Constraint Qualification

LISA Linear Iterative Splitting Approach

MOL Method Of Lines

NDGM Nodal Discontinuous Galerkin Method

NLP Nonlinear Programming Problem

NMT Natural Monotonicity Test

OCP Optimal Control Problem

ODE Ordinary Differential Equation

XIV List of acronyms

OOP Object Oriented Programming

PASM Parametric Active Set Method

PCG Preconditioned Conjugate Gradient

PDE Partial Differential Equation

PQP Parametric Quadratic Programming

QP Quadratic Programming Problem

RMT Restrictive Monotonicity Test

SCC Strict Complementarity Condition

SMB Simulated Moving Bed

SOSC Second Order Sufficient Condition

SQP Sequential Quadratic Programming

VDE Variational Differential Equation

1 Introduction
“The miracle of the appropriateness of the language of mathematics
for the formulation of the laws of physics is a wonderful gift which
we neither understand nor deserve. We should be grateful for it and
hope that it will remain valid in future research and that it will extend,
for better or for worse, to our pleasure even though perhaps also to
our bafflement, to wide branches of learning.”
— E.P. W IGNER [164]

Mathematics today permeates an ever increasing part of the sciences far beyond
mathematical physics just as about 50 years ago Nobel Prize laureate Wigner has
hoped for. In particular mathematical methods for simulation and optimization of
quantitative mathematical models continue to face growing demand in disciplines
ranging from engineering, biology, economics, physics, etc. even to emerging ar-
eas of psychology or archeology (see, e.g., Sager et al. [139], Schäfer et al. [140]).
In this thesis we focus on mathematical and computational methods for the class
of Optimal Control Problems (OCPs) pioneered by Pontryagin and Bellman in the
middle of the 20th century. General mathematical optimization problems consist
of finding a solution candidate which satisfies a set of constraints and minimizes a
certain objective function. OCPs are optimization problems whose free variables
comprise states and controls from (usually infinite dimensional) function spaces
constrained to satisfy given differential equations. The differential equations de-
scribe the behavior of a dynamic system which can be controlled in a prescribed
way.
The treatment of constraints given by Partial Differential Equations (PDEs) is
one major challenge that we address in this thesis. PDEs appear when spatially
distributed phenomena need to be taken into account, e.g., when we describe the
diffusion of a substance in a liquid. Ordinary Differential Equations (ODEs),
which describe the evolution of a system in time, are not a satisfactory mathe-
matical tool for the description of spatial effects (although we shall use them to
approximate solutions of PDEs). A considerable amount of theory and practical
computational methods is available today for ODE OCPs. The presence of PDE
constraints causes additional difficulties both on the theoretical as well as on the

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_1,
© Springer Fachmedien Wiesbaden 2014
2 1 Introduction

numerical side and is a much younger field of research especially in the aspect of
methods which heavily rely on high computing power.
OCPs are inherently infinite problems because we seek solutions in function
spaces. We can divide numerical methods for OCPs into two main classes: Direct
and indirect methods. The defining line between the two is somewhat blurry, espe-
cially when we cross borders of mathematical communities. We base our classifi-
cation here on the sequence of discretization and optimization. In indirect methods
we first derive optimality conditions in function space which we discretize after-
wards. In direct methods we discretize the problem first and then find an optimizer
of the resulting Nonlinear Programming Problem (NLP). Moreover we often end
up with an implicit characterization of the control via artificially introduced co-
state or adjoint variables in indirect methods. This is in contrast to direct methods
for which the discretized control usually occurs explicitly as one or the only re-
maining variable. Indirect methods for ODE OCPs are mostly based on Dynamic
Programming (see, e.g., Bellman [15]) or Pontryagin’s Maximum Principle (see,
e.g., Pontryagin et al. [126]). Tröltzsch [152] in his introductory textbook for PDE
OCPs also treats only indirect methods. A discussion of direct and indirect meth-
ods for PDE OCPs is given in Hinze et al. [86, Chapter 3]. In the 1980’s the
endeavor to apply numerical optimization quickly to new application areas and
new problems led to the development of direct methods for ODE OCPs, most no-
tably collocation methods (see, e.g., Bär [8], Bock [23], Biegler [18]) and Direct
Multiple Shooting (Bock [22], Plitt [125], Bock and Plitt [25]). One advantage of
direct methods is that the optimality conditions of an NLP are generic, whereas
optimality conditions of undiscretized OCPs need to be reestablished for each new
problem and often require partial a-priori knowledge of the mathematical structure
of the solution which in general is not available for many application problems. At
the crux of creating an efficient direct optimization method is structure exploita-
tion in the numerical solution of the NLP. Usually either Sequential Quadratic
Programming (SQP) or Interior Point methods are employed (see, e.g., the text-
book of Nocedal and Wright [121]). These iterative methods require the compu-
tation of derivatives of the objective function and the constraints. Derivative free
methods (see, e.g., the introductory textbook by Conn et al. [34]) are typically not
suited because of the high number of unknowns and because nonlinear constraints
can only be treated with excessive computational effort.
It is our goal in this thesis to extend Direct Multiple Shooting for ODE OCPs in
order to make it applicable and continue its success story for a class of PDE OCPs.
The first hurdle on this venture is the large problem size of the discretized OCPs.
Schäfer [141] describes in his dissertation approaches to address this difficulty by
exploitation of the special mathematical structure of the discretized OCPs. His
1.1 Results of this thesis 3

approaches lead to a reduction in the number of needed directional derivatives for

the dynamical system. The Schäfer approach requires only a constant number of
directional derivatives per optimization iteration while the number of directional
derivatives for conventional Direct Multiple Shooting depends linearly on the num-
ber of spatial discretization points which typically grow prohibitively large.
However, the approach of Schäfer cannot be applied efficiently to OCPs with
boundary conditions in time, the treatment of which is another declared goal of
this thesis. PDE OCPs with time-periodicity conditions are even more difficult
because for each spatial discretization point one additional constraint arises. In
order to obtain an algorithm whose required number of directional derivatives is
independent of the spatial discretization we have developed a globalized inexact
SQP method in extension to ideas for inexact Newton methods (Ortega and Rhein-
boldt [122], Dembo et al. [41]), inexact SQP methods (Diehl et al. [48], Wirsching
[166]), the Newton-Picard approach (Lust et al. [110]), and globalization via nat-
ural level functions (Bock [22], Bock et al. [26], Deuflhard [44]).
Boundary conditions in time occur often in practical applications, most of the
time in form of a periodicity constraint. In this thesis we apply the investigated
methods to the optimization of a real-world chromatographic separation process
called Simulated Moving Bed (SMB). Preparative chromatography is one of vari-
ous examples in the field of process operations for which periodic operation leads
to a considerable increase in process performance compared to batch operation.
The complicated structure of optimal solutions makes mathematical optimiza-
tion an indispensable tool for the practitioner (see, e.g., Nilchan and Pantelides
[120], van Noorden et al. [156], Toumi et al. [151], de la Torre et al. [40], Kawajiri
and Biegler [93, 94], Agarwal et al. [1]).

1.1 Results of this thesis

For the first time we propose a method based on Direct Multiple Shooting for
time-periodic PDE OCPs which features optimal scale-up of the effort in the num-
ber of spatial discretization points. This result is based on grid-independence of the
number of inexact SQP iterations and a bound on the numerical effort for one in-
exact SQP iteration as essentially a constant times the effort for the solution of one
Initial Value Problem (IVP) of the dynamical system. We can solve a nonlinear
discretized large-scale optimization problem with roughly 700 million variables
(counting intermediate steps of the IVP solutions as variables) in under half an
hour on a current commodity desktop machine. Although developed particularly
for PDE OCPs with time-periodicity constraints in mind, the method can also be
4 1 Introduction

applied to problems with fixed initial conditions instead of time-periodicity con-

straints and is thus considerably versatile.
Based on an inner Linear Iterative Splitting Approach (LISA) for the linear sys-
tems we review a LISA-Newton method. It is well-known that the linear asymp-
totic convergence rate of a LISA-Newton method with l inner LISA iterations
coincides with the asymptotic convergence rate of the LISA method to the power
of l. We prove this result for the first time in the framework of Bock’s κ-theory.
Truncated Neumann series occur in the proof which yield a closed form for the
backward error of the inexact linear system solves. This backward error is of sig-
nificant importance not only for the linear system itself but also in Bock’s Local
Contraction Theorem (Bock [24]) which characterizes the local convergence of
Newton-type methods.
The previous result enables us to develop three novel a-posteriori κ-estimators
which are computed from the iterates of the inner LISA iterations. We highlight
the complications which result from the occurrence of non-diagonalizable iteration
matrices from a geometrical point of view with examples.
We further extend LISA-Newton methods to SQP methods and prove that limit
points satisfy a first order necessary optimality condition and that a second order
sufficiency condition transfers from the Quadratic Programming Problem (QP) in
the solution to the solution of the NLP. Moreover we describe the use of inexact
Jacobians and Hessians within a generalized LISA method based on QPs. We
also attempt an extension of a globalization strategy for LISA-Newton methods
using natural level functions for the case of inexact SQP methods. We discuss
important details of the numerical implementation and show that the developed
strategy works reliably on numerical examples of practical relevance.
For LISA methods for time-periodic PDE OCPs we develop Newton-Picard pre-
conditioners. We propose a classical variant based on Lust et al. [110] and a two-
grid variant. We show that it is of paramount importance for numerical efficiency
to modify the classical Newton-Picard preconditioner to use an L2 -based projector
instead of a Euclidean projector. Moreover we prove grid-independent conver-
gence of the classical Newton-Picard preconditioner on a linear-quadratic time-
periodic PDE OCP. We further give numerical evidence that the two-grid variant
is more efficient on a wide range of practical problems. For the extension of the
proposed preconditioners to the nonlinear case for use in LISA-Newton methods
we discuss several difficulties of the classical Newton-Picard preconditioner. We
also develop a new two-grid Hessian approximation which fits naturally in the
two-grid Newton-Picard framework and yields a reduction of 68 % in runtime for
an exemplary nonlinear benchmark problem. Moreover we show that the two-grid
Newton-Picard LISA-Newton method is scaling invariant. This property is of con-
1.1 Results of this thesis 5

siderable importance for the reliability of the method on already badly conditioned
problems.
The analysis reveals that the quality of the fine grid controls the accuracy of
the solution while the quality of the coarse grid determines the asymptotic lin-
ear convergence rate, i.e., Bock’s κ, of the two-grid Newton-Picard LISA-Newton
method. Based on the newly established reliable a-posteriori κ-estimates we de-
velop a numerical strategy for automatic determination of when to refine the coarse
grid to guarantee fast convergence.
We further develop a structure exploiting two-stage strategy for the solution of
QP subproblems in the inexact SQP method. The first stage is an extension of the
traditional condensing step in SQP methods for Direct Multiple Shooting which
exploits the constraint for periodicity or alternatively given fixed initial values for
the PDE in addition to the Multiple Shooting matching conditions. This strat-
egy reduces the large-scale QP to an equivalent QP whose size is independent of
the spatial discretization. The reduction can be efficiently computed because it
additionally exploits the (two-grid) Newton-Picard structure in the QP constraint
and Hessian matrices. For the second stage we develop a Parametric Active Set
Method (PASM) which can also treat nonconvex QPs with indefinite Hessian ma-
trices. This capability is required because we want to treat nonconvex NLPs using
accurate approximations for Lagrangian Hessians. We propose numerical tech-
niques for improving the reliability of our PASM code which outperforms several
other popular QP codes in terms of reliability.
The Newton-Picard LISA method can also be interpreted as a one-shot one-step
approach for a linear PDE OCP. The almost optimal convergence theorem which
we prove for the considered model problem supports the conjecture that such one-
step approaches will in general yield optimization algorithms which converge as
fast as the algorithm for the forward problem, which consists of satisfying the
constraints for fixed controls. Contrary to common belief, however, we have con-
structed three small-scale, equality constrained QPs which illustrate that the con-
vergence for the forward problem method is neither sufficient nor necessary for
the convergence of the one-step optimization method. Furthermore we show that
existing one-step techniques to enforce convergence might lead to de-facto loss of
convergence with contraction factors of almost 1. These examples and results can
serve as a warning signal or guiding principle for the choice of assertions which
one might want to attempt to prove about one-step methods. It also justifies that
we prove convergence of the Newton-Picard LISA only for a model problem.
We have put substantial effort into the implementation of the proposed ideas in
a new software package called MUSCOP. Based on a hybrid programming design
principle we strive to keep the code both easy to use and easy to maintain/develop
6 1 Introduction

further at the same time. The code features parallelization on the Multiple Shoot-
ing structure and automatic generation of derivatives of first and second order of
the model functions and dynamic systems in order to reduce setup and solution
time for new application problems to a minimum.
Finally we use MUSCOP to demonstrate the applicability, reliability, and effi-
ciency of the proposed numerical methods and techniques on a sequence of PDE
OCPs of growing difficulty: Linear and nonlinear boundary control tracking prob-
lems subject to the time-periodic linear heat equation in 2D and 1D, a tracking
problem in bacterial chemotaxis which features a strong nonlinearity in the con-
vective term, and finally a real-world practical example: Optimal control of the
ModiCon variant of the SMB process.

1.2 Thesis overview

This thesis is structured in three parts: Theoretical foundations, numerical meth-
ods, and applications and numerical results. In Chapter 2 we give a short intro-
duction to Bochner spaces and sketch the functional analytic setting for parabolic
PDE in order to formulate the PDE OCP that serves as the point of origin for all
further investigations in this thesis.
We present a direct optimization approach in Chapter 3. After a discussion of
the discretize-then-optimize and optimize-then-discretize paradigms we describe
a multi-stage discretization approach: Given a hierarchy of spatial discretizations
we employ the Method Of Lines (MOL) to obtain a sequence of large-scale ODE
OCPs which we subsequently discretize with Direct Multiple Shooting. We then
formulate the resulting NLPs and discuss their numerical challenges.
In Chapter 4 we give a concise review of elements of finite dimensional opti-
mization theory for completeness. This concludes Part 1, theoretical foundations.
We begin Part 2, numerical methods, with the development of a novel inexact
SQP method in Chapter 5. We commence the discussion with Newton-type meth-
ods and present Bock’s Local Contraction Theorem and its proof. Subsequently
we review popular methods for globalization of Newton-type methods and dis-
cuss their limits when it comes to switching from globalized mode to fast local
contraction mode. We then present the idea and several interpretations of global-
ization via natural level functions and explain how they overcome the problem of
impediment of fast local convergence. The natural level function approach leads
to computable monotonicity tests for the globalization strategy. A review of the
Restrictive Monotonicity Test (RMT) and a Natural Monotonicity Test (NMT) for
LISA-Newton methods then precedes an exhaustive discussion of the convergence
1.2 Thesis overview 7

of LISA and its connection with Bock’s κ-theory. On this basis we develop three
a-posteriori κ-estimators which are based on the LISA iterates. In addition we
propose an extension to SQP methods, prove that a first-order necessary optimal-
ity condition holds if the method converges, and further show that a second order
sufficiency condition transfers from the QP in the solution to the solution of the
NLP. Finally we present a novel extension to inexact SQP methods on the basis of
a generalized LISA for QPs.
In Chapter 6 we develop so-called Newton-Picard preconditioners for time-
periodic OCPs. We discuss a classical and a two-grid projective approach. For
the classical approach we show grid-independent convergence. We conclude the
chapter with a discussion of the application of Newton-Picard preconditioning in
a LISA-Newton method for nonlinear problems and for Multiple Shooting.
We present three counter-intuitive toy examples in Chapter 7 which show that
convergence of the forward problem method is neither sufficient nor necessary
for the convergence of a corresponding one-step one-shot optimization approach.
We furthermore analyze regularization approaches which are designed to enforce
one-step one-shot convergence and demonstrate that de-facto loss of convergence
cannot be avoided via these techniques.
In Chapter 8 we discuss condensing of the occurring large-scale QPs to equiva-
lent QPs whose size is independent of the number of spatial discretization points.
We further develop efficient numerical exploitation of the Multiple Shooting and
Newton-Picard structure. Moreover we propose a two-grid Hessian matrix ap-
proximation which fits well in the framework of the two-grid Newton-Picard pre-
conditioners. As a final remark we show scaling invariance of the Newton-Picard
LISA-Newton method for PDE OCPs.
The solution of the resulting medium-scale QPs via PASM is our subject in
Chapter 9. We identify numerical challenges in PASMs and develop strategies
to meet these challenges, in particular the techniques of drift correction and flip-
ping bounds. Furthermore we implement these strategies in a code called rpasm
and demonstrate that rpasm outperforms other popular QP solvers in terms of re-
liability on a well-known test set. We conclude the chapter with an extension to
nonconvex QPs which can arise when employing the exact Lagrangian Hessian or
the two-grid Newton-Picard Hessian approximation. The proposed PASM is also
considerably efficient because it can be efficiently hot-started.
In Chapter 10 we review numerical methods for automatic generation of deriva-
tives on the basis of Algorithmic Differentiation (AD) and Internal Numerical
Differentiation (IND). Furthermore we address issues with a monitor strategy in
implicit numerical integrators which can lead to violation of the IND principle for
8 1 Introduction

the example of a linear 1D heat equation. Then we conclude the chapter with a
short account on the numerical effort of IND.
We dedicate Chapter 11 to the design of the software package MUSCOP. Issues
we address include programming paradigms and description of the various soft-
ware components and their complex orchestration necessary for smart structure
exploitation. This concludes Part 2, numerical methods.
In Part 3 we present applications and numerical results which were generated
with MUSCOP. Linear boundary control for the periodic 2D heat equation is in the
focus of our presentation in Chapter 12. We give numerical evidence of the failure
of Euclidean instead of L2 projection in classical Newton-Picard preconditioners.
In accordance with the proof of mesh-independent convergence we give several
computational results for varying problem data and discuss why the two-grid vari-
ant is superior to the classical Newton-Picard preconditioner.
We extend the problem to nonlinear boundary control in 1D in Chapter 13 and
discuss numerical self-convergence. We can show that employing the two-grid
Hessian approximation leads to an overall reduction in computation time of 68 %.
We discuss parallelization issues and compare runtimes for different discretiza-
tions of the control in time. In all cases we give detailed information about the
runtime spent in different parts of the algorithm and show exemplarily that with
above 95 % most of the runtime is required for system simulation and IND.
In Chapter 14 we present a tracking type OCP for a (non-periodic) bacterial
chemotaxis model in 1D. The model is characterized by a highly nonlinear con-
vective term. We demonstrate the applicability of the proposed methods also to
this problem and discuss the self-convergence of the computation.
Chapter 15 is the last chapter of this thesis. In it we present the SMB process
and explain a mathematical model for chromatographic columns. We then present
numerical results for the ModiCon variant of the SMB process for real-world data.
We obtain optimal solutions with an accuracy which has not been achieved before.
This concludes Part 3 and this thesis.
Chapters 6, 7, 9, 12, and parts of Chapter 15 are based on own previously pub-
lished work. For completeness we reprint partial excerpts here with adaptions to
the unified nomenclature and structure of this thesis. We give the precise refer-
ences to the respective articles at the beginning of each of these chapters.
 Part I

Theoretical foundations
2 Problem formulation
The goal of this chapter is to introduce the Optimal Control Problem (OCP) for-
mulation which serves as the point of origin for all further investigations in this
thesis. To this end we recapitulate elements of the theory of parabolic Partial Dif-
ferential Equations (PDEs) in Section 2.1 and present a system of PDEs coupled
with Ordinary Differential Equations (ODEs) in Section 2.2. The coupled system
is one of the constraints among additional boundary and path constraints for the
OCP which we describe in Section 2.3. We emphasize the particular aspects in
which our problem setting differs and extends the setting most often found in PDE
constrained optimization.

2.1 Dynamical models described by Partial

Differential Equations
We treat processes which are modeled by a state u distributed in space and evolv-
ing over time. The evolution of u is deterministic and described by PDEs. The
behavior of the dynamical system can further be influenced by a time and possibly
space dependent control q.
Nonlinear instationary PDEs usually do not have solutions in classical function
spaces. We recapitulate the required definitions for Bochner spaces and vector-
valued distributions necessary for formulations which have solutions in a weak
sense. The presentation here is based on Dautray and Lions [37], Gajewski et al.
[57], and Wloka [167]. We omit all proofs which can be found therein. Through-
out this chapter let Ω ∈ Rd be a bounded open domain with sufficiently regular
boundary ∂ Ω, X be a Banach space, and dμ denote the Lebesgue measure in Rd .
We assume that the reader is familiar with basic concepts of functional anal-
ysis (see, e.g., Dunford and Schwartz [50]). We denote with L p (Ω), 1 ≤ p ≤ ∞,
the Lebesgue space of μ-measurable R-valued functions whose absolute value to
the p-th power has a bounded integral over Ω if p < ∞ or which are essentially
bounded if p = ∞. Functions which coincide μ-almost everywhere are consid-
ered identical. With W k,p (Ω), k ≥ 0, 1 ≤ p < ∞ we denote the Sobolev space of
functions in L p (Ω) whose distributional derivatives up to order k lie in L p (Ω).

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_2,
© Springer Fachmedien Wiesbaden 2014
12 2 Problem formulation

The spaces L p (Ω),W k,p (Ω) endowed with their usual norms are Banach spaces.
The spaces H k (Ω) := W k,2 (Ω) equipped with their usual scalar product are Hilbert
spaces. The construction of L p (Ω) and W k,p (Ω) can be generalized to functions
with values in Banach spaces:
Definition 2.1 (Bochner spaces). By L p (Ω; X), 1 ≤ p < ∞, we denote the space of
all measurable functions v : Ω → X satisfying

vXp dμ < ∞.
Ω

We identify elements of L p (Ω; X) which coincide μ-almost everywhere and equip

L p (Ω; X) with the norm
 1/p
p
vL p (Ω;X) = vX dμ .
Ω

Now we proceed in the following way: Generally we are interested in weak

solutions u ∈ W in an appropriate Hilbert space W ⊂ L2 ((t1 ,t2 ) × Ω) with finite
t1 ,t2 ∈ R. Functions in L2 ((t1 ,t2 ) × Ω) need not even be continuous and hence
we must exercise care to give well-defined meaning to derivatives and the traces
u(t1 , .) and u(t2 , .). This is not trivial because altering u on any set of measure zero,
e.g., {t1 ,t2 } × Ω, yields the same u in L2 ((t1 ,t2 ) × Ω). The traces are important for
the formulation of boundary value conditions. We address these issues concerning
the state space in three steps. In a first step, we write
L2 ((t1 ,t2 ) × Ω) = L2 ((t1 ,t2 ); L2 (Ω)),
i.e., we interpret u as an L2 function in time with values in the space of L2 functions
in space. Second, we can formulate the time derivative du/dt of u via the concept
of vectorial distributional derivatives.
Definition 2.2. Let Y be another Banach space. We denote the space of continuous
linear mappings from X to Y with L (X,Y ).
Definition 2.3. The space of vectorial distributions of the interval (t1 ,t2 ) ⊂ R with
values in the Banach space X is denoted by
D  ((t1 ,t2 ); X) := L (C∞ ([t1 ,t2 ]; R), X).
We can identify every u ∈ L2 ((t1 ,t2 ); X) ⊂ L1 ((t1 ,t2 ); X) with a distribution T ∈
D  ((t1 ,t2 ); X)
via the Bochner integral
 t2
Tϕ = u(t)ϕ(t)dt for all ϕ ∈ C∞ ([t1 ,t2 ]; R).
t1
2.1 Dynamical models described by Partial Differential Equations 13

Definition 2.4. The k-th derivative of T is defined via

 t2
dk T
ϕ = (−1)k u(t)ϕ (k) (t)dt.
dt k t1

Thus, dT /dt ∈ D  ((t1 ,t2 ); X). We assume now that X → Y , where → denotes
continuous embedding. Hence it holds that

D  ((t1 ,t2 ); X) → D  ((t1 ,t2 );Y ),

L p ((t1 ,t2 ); X) → L p ((t1 ,t2 );Y ).

Let u ∈ L2 ((t1 ,t2 ); X). We say that du/dt ∈ L2 ((t1 ,t2 );Y ) if there exists u ∈
L2 ((t1 ,t2 ),Y ) such that
 t2  t2
dT
u (t)ϕ(t)dt = ϕ =− u(t)ϕ (1) (t)dt for all ϕ ∈ C∞ ([t1 ,t2 ]; R),
t1 dt t1

and we identify du/dt := u . We also use the abbreviation ∂t u := du/dt.

In the third step, let V and H be separable Hilbert spaces and let V ∗ denote the
dual space of V . We assume throughout that (V, H,V ∗ ) is a Gelfand triple
d d
V → H → V ∗ ,

i.e., the embeddings of V in H and H = H ∗ in V ∗ are continuous and dense. Now

we choose X = V and Y = V ∗ in order to define the space of L2 functions over V
with time derivatives in L2 over the dual V ∗ according to

W (t1 ,t2 ) = {u ∈ L2 ((t1 ,t2 );V ) | ∂t u ∈ L2 ((t1 ,t2 );V ∗ )}.

Lemma 2.5. The space W (t1 ,t2 ) is a Hilbert space when endowed with the scalar
product
 t2  t2
(u, v)W (t1 ,t2 ) = (u(t), v(t))V dt + (∂t u(t), ∂t v(t))V ∗ dt.
t1 t1

Proof. See Wloka [167, Satz 25.4].

Theorem 2.6. We can alter every u ∈ W (t1 ,t2 ) on a set of measure zero to obtain
a function in C0 ([t1 ,t2 ]; H). Furthermore, if we equip C0 ([t1 ,t2 ]; H) with the norm
of uniform convergence then

W (t1 ,t2 ) → C0 ([t1 ,t2 ]; H).

14 2 Problem formulation

Proof. See Dautray and Lions [37, Chapter XVIII, Theorem 1].
Corollary 2.7. For u ∈ W (t1 ,t2 ) the traces u(t1 ), u(t2 ) have a well-defined mean-
ing in H (but not in V in general).
For the control variables we assume q ∈ L2 ((t1 ,t2 ); Q) where Q ⊆ L2 (Ω)nq or
Q ⊆ L2 (∂ Ω)nq for distributed or boundary control, respectively. We can then for-
mulate the parabolic differential equation
∂t u(t) + A(q(t), u(t)) = 0, (2.1)
with a nonlinear elliptic differential operator A : Q × V → V ∗.
In the numerical
approaches which we present in Chapters 5 and 6 we exploit that A is an elliptic
operator. We further assume that A is defined via a semilinear (i.e., linear in the
last argument) form a : (Q ×V ) ×V → R according to
A(q(t), u(t)), ϕ V ∗ ×V = a(q(t), u(t), ϕ) for all ϕ ∈ V. (2.2)
We consider Initial Value Problems (IVPs), i.e., PDE (2.1) subject to u(t1 ) = u0 ∈
H. The question of existence, uniqueness, and continuous dependence of solutions
on the problem data u0 and q cannot be answered satisfactorily in a general set-
ting. However, there are problem-dependent sufficient conditions (compare, e.g.,
Gajewski et al. [57] for the case A(q(t), u(t)) = Aq (q(t)) + Au (u(t))). A thorough
discussion of this question is beyond the focus of this thesis.
Example 1. For illustration we consider the linear heat equation with Robin bound-
ary control and initial values
∂t u = Δu in (0, 1) × Ω, (2.3a)
∂ν u + αu = β q on (0, 1) × ∂ Ω, (2.3b)

u = u0 ,
t=0
(2.3c)
where α, β ∈ L∞ (∂ Ω) and ∂ν
denotes the derivative in the direction of the outwards
pointing normal ν on ∂ Ω. We choose V = H 1 (Ω) and H = L2 (Ω). Multiplication
with a test function ϕ ∈ V and integration by parts transform equations (2.3a)
and (2.3b) into
 
0= ∂t u(t)ϕ − (Δu(t))ϕ (2.4a)
Ω Ω   
= ∂t u(t)ϕ + ∇u(t)T ∇ϕ − ∇u(t)T ν ϕ (2.4b)
Ω Ω ∂Ω
   
= ∂t u(t)ϕ + ∇u(t)T ∇ϕ + αu(t)ϕ − β q(t)ϕ (2.4c)
Ω Ω ∂Ω ∂Ω

=: ∂t u(t)ϕ + a(q(t), u(t), ϕ), (2.4d)
Ω
2.2 Coupled ODEs and PDEs 15

which serves as the definition for the semilinear form a and the corresponding
operator A. We immediately observe that a is even bilinear on (Q ×V ) ×V in this
example.

2.2 Coupled ODEs and PDEs

In some applications, e.g., in chemical engineering, the models consist of PDEs
which are coupled with ODEs. We denote the ODE states, which are not dis-
tributed in space, by v ∈ C0 ([t1 ,t2 ]; Rnv ). These states can for instance model the
accumulation of mass of a chemical species at an outflow port of a chromato-
graphic column (compare Chapter 15). We can formulate the coupled system of
differential equations as
∂t u(t) = −A(q(t), u(t), v(t)), v̇(t) = f ODE (q(t), u(t), v(t)), (2.5a)
where f ODE : Q × H × Rnv , subject to initial or boundary value conditions in time.
We restrict ourselves to an autonomous formulation because the non-autonomous
case can always be formulated as system (2.5) by introduction of an extra ODE
state v̇i = 1 with initial value vi (t1 ) = t1 .
The question of existence, uniqueness, and continuous dependence on the data
for the solution of IVPs with the differential equations (2.5) is even more challeng-
ing than for PDE IVPs and must be investigated for restriced problem classes (e.g.,
when A is not dependent on the v(t) argument). Again, a thorough discussion of
this question exceeds the scope of this thesis.

2.3 The Optimal Control Problem

We now state the OCP which is the point of origin for all further investigations of
this thesis:
minimize Φ(u(1), v(1)) (2.6a)
q∈L2 ((0,1);Q)
u∈W (0,1)
v∈C0 ([0,1];Rnv )

s. t. ∂t u = −A(q(t), u(t), v(t)), t ∈ (0, 1), (2.6b)

v̇ = f ODE
(q(t), u(t), v(t)), t ∈ (0, 1), (2.6c)
(u(0), v(0)) = r (u(1), v(1)),
b
(2.6d)
r (q(t), v(t)) ≥ 0,
c
t ∈ (0, 1), (2.6e)
re (v(1)) ≥ 0, (2.6f)
16 2 Problem formulation

with nonlinear functions

Φ : H × Rnv → R, r b : H × R n v → H × R nv ,
c e
r c : Q × R nv → R nr , re : Rnv → Rnr .

We now discuss each line of OCP (2.6) in detail.

The objective function Φ in line (2.6a) is different from what is typically treated
in PDE constrained optimization. Often, even for nonlinear optimal control prob-
lems, the objective functions are assumed to consist of a quadratic term for the
states, e.g., L2 tracking type in space or in the space-time cylinder, plus a quadratic
Tychonoff-type regularization term for the controls (see, e.g., Tröltzsch [152]) of
the type
 
1 1 
2
 γ 1
u(t) − udesired (t) dt + q(t)2Q dt. (2.7)
2 0 H 2 0
We remark that tracking type problems with objective (2.7) on the space-time
cylinder can always be cast in the form of OCP (2.6) by introduction of an ad-
ditional ODE state variable vi subject to
 2
 
v̇i (t) = u(t) − udesired (t) + γ q(t)2Q , vi (0) = 0,
H

with the choice Φ(u(1), v(1)) = vi (1)/2. The applications we are interested in,
however, can have economical objective functions which are not of tracking type.
Constraints (2.6b) and (2.6c) determine the dynamics of the considered system.
We have already described them in detail in Sections 2.1 and 2.2 of this chapter.
Initial or boundary value constraints are given by equation (2.6d). Typical ex-
amples are pure initial value conditions via constant

rb (u(1), v(1)) := (u0 , v0 )

or periodicity conditions

rb (u(1), v(1)) := (u(1), v(1)).

Compared to initial value conditions the presence of boundary value conditions

makes it more difficult to use reduced approaches which rely on a solution opera-
tor for the differential equations mapping a control q to a feasible state u. Instead
of solving one IVP, the solution operator would have to solve one Boundary Value
Problem (BVP) which is in general both theoretically and numerically more diffi-
cult. Thus we avoid this sequential approach in favor of a simultaneous approach
2.3 The Optimal Control Problem 17

in which the intermediate control and state iterates of the method may be infeasi-
ble for equations (2.6b) through (2.6d). Of course feasibility must be attained in
the optimal solution.
Inequality (2.6e) is supposed to hold for almost all t ∈ (0, 1) and can be used
to formulate constraints on the controls and ODE states. We deliberately do not
include PDE state constraints in the formulation which give rise to various theo-
retical difficulties and are currently a very active field of research. We allow for
additional inequalities on the ODE states at the end via inequality (2.6f). In the
context of chemical engineering applications, the constraints (2.6e) and (2.6f) can
comprise flow rate, purity, throughput constraints, etc.
Problems with free time-independent parameters can be formulated within prob-
lem class (2.6) via introduction of additional ODE states vi with vanishing time
derivative v̇i (t) = 0. Although the software package MUSCOP (see Chapter 11)
treats time-independent parameters explicitly, we refrain from elaborating on these
issues in this thesis in order to avoid notational clutter.
OCP (2.6) also includes the cases of free start and end time via a time transfor-
mation, e.g., τ(t) = (1 − t)τ1 + tτ2 ∈ [τ1 , τ2 ],t ∈ [0, 1]. This case plays an impor-
tant role in this thesis, e.g., in periodic applications with free period duration, see
Chapter 15.
Concerning regularity of the functions involved in OCP (2.6), we take a prag-
matic view point: We assume that the problem can be consistently discretized
(along the lines of Chapter 3) and that the resulting finite dimensional optimiza-
tion problem is sufficiently smooth on each discretization level to allow for em-
ployment of fast numerical methods (see Chapter 5).
3 Direct Optimization: Problem
discretization
The goal of this chapter is to obtain a discretized version of OCP (2.6). We discuss
a so-called direct approach and summarize its main advantages and disadvantages
in Section 3.1 in comparison with alternative approaches. In Sections 3.2 and 3.3
we discretize OCP (2.6) in two steps. First we discretize in space and obtain a
large-scale ODE constrained OCP which we then discretize in time to obtain a
large-scale Nonlinear Programming Problem (NLP) presented in Section 3.5. The
numerical solution of this NLP is the subject of Part II in this thesis.

3.1 Discretize-then-optimize approach

We follow a direct approach to approximate the infinite dimensional optimiza-
tion problem (2.6) by finite dimensional optimality conditions by first discretizing
the optimization problem to obtain an NLP (discretize-then-optimize approach).
Popular alternatives are indirect approaches where infinite dimensional optimality
conditions are formulated (optimize-then-discretize approaches). These two main
routes are displayed in Figure 3.1.
To give a detailed list and comprehensive comparison of direct and indirect ap-
proaches for OCPs is beyond the scope of this thesis. A short comparison for
ODE OCPs can be found, e.g., in Sager [138]. For PDE OCPs we refer the reader
to Hinze et al. [86, Chapter 3]. Both direct and indirect approaches have various
advantages and disadvantages which render one or the other more appropriate for a
concrete problem instance at hand. For the sake of brevity we restrict ourselves to
only state the main reason why we have decided to apply a direct approach: While
the most important property of indirect methods is certainly that they provide the
deepest insight into the mathematical structure of the solution of the particular
problem, the insight usually comes at the cost of heavy mathematical analysis,
which might be too time consuming or too difficult for a practitioner. The direct
approach that we lay out in this chapter enjoys the advantage that it can be applied
in a straight-forward, generic, and almost automatic way. We believe that this

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_3,
© Springer Fachmedien Wiesbaden 2014
20 3 Direct Optimization: Problem discretization

Necessary infinite
Infinite dimensional optimize
−−−−→ dimensional
optimization problem
optimality conditions
⏐ ⏐
⏐ ⏐
discretize discretize

Necessary finite
Finite dimensional
−−−−→ dimensional
optimization problem optimize
optimality conditions

Figure 3.1: The two main routes to approximate an infinite dimensional optimization prob-
lem (upper left box) with necessary finite dimensional optimality conditions (lower right
box) are direct approaches (discretize-then-optimize, lower left path) versus indirect ap-
proaches (optimize-then-discretize, upper right path).

property is paramount for successful deployment of the method in collaboration

with practitioners.
We have chosen a Direct Multiple Shooting approach for the following numeri-
cal advantages: First, it has been shown in Albersmeyer [2] and Albersmeyer and
Diehl [4] that Multiple Shooting can be interpreted as a Lifted Newton method
which might reduce the nonlinearity of a problem and enlarge thus the domain of
fast local convergence (see Chapter 5). Second, we can supply good initial value
guesses—if available—for iterative solution methods through the local state vari-
ables. Third, we can employ advanced numerical methods for the solution and
differentiation of the local IVPs (see Chapter 10). Fourth, due to the decoupling
of the IVPs we can parallelize the solution of the resulting NLP on the multiple
shooting structure (see Chapter 11).

3.2 Method Of Lines: Discretization in space

The first step in the discretization of OCP (2.6) consists of discretizing the func-
tion spaces V and Q, e.g., by Finite Difference Methods (FDMs), Finite Element
Methods (FEMs), Finite Volume Methods (FVMs), Nodal Discontinuous Galerkin
Methods (NDGMs), or spectral methods. Introductions to these methods are avail-
able in textbook form, e.g., LeVeque [109, 108], Braess [29], Hesthaven and
Warburton [84], and Hesthaven et al. [85]. The approach of discretizing first in
3.2 Method Of Lines: Discretization in space 21

space and then in time is called Method Of Lines (MOL) and is often applied for
parabolic problems (see, e.g., Thomée [150]). We must exercise care that the spa-
tial discretization is appropriate for the concrete problem at hand. For instance, an
FEM for advection dominated problems must be stabilized, e.g., by a Streamline
Upwind Petrov Galerkin formulation (see, e.g., Brooks and Hughes [31]), and an
NDGM for diffusion dominated problems must be stabilized by a jump penalty
term (see, e.g., Warburton and Embree [161]).
We assume that after discretization we obtain a finite dimensional space Qh ⊆ Q
and a hierarchy of finite dimensional spaces Vhl , l ∈ N, satisfying
Vh1 ⊂ Vh2 ⊂ · · · ⊂ V.
We choose this setting for the following reasons: For many applications, especially
in chemical engineering, an infinite dimensional control is virtually impossible to
implement on a real process. In the case of room heating for instance, the temper-
ature field is distributed in the three-dimensional room, but the degrees of freedom
for the control will still be the scalar valued position of the radiator operation knob.
In this case, a one-dimensional discretization Qh of Q is fully sufficient. For the
applications we consider, we always assume Qh to be a low dimensional space.
It is of paramount importance, however, to accurately resolve the system state u.
For this reason, we assume that the spaces Vhl are high dimensional for for large
l. The numerical methods we describe in Part II will rely on and exploit these
assumptions on the dimensionality of Qh and Vhl .
We can then use the finite dimensional spaces Vhl and Qh to obtain a discretiza-
tion of the semilinear form a and thus the operator A from equation (2.2). On each
level l we are led to an ODE of the form
Mul u̇l (t) = f PDE(l) (q(t), ul (t), v(t)),
l
with symmetric positive-definite NVl -by-NVl matrix Mul , ul (t) ∈ RNV , q(t) ∈ RNQ ,
l l
and f PDE(l) : RNQ × RNV × Rnv → RNV . In this way we approximate PDE (2.1)
with ODEs which are of large scale on finer discretization levels l. Let us illustrate
this procedure in an example.
Example 2. We continue the example of the heat equation (Example 1) on the
unit square Ω = (0, 1)2 with boundary control. For the discretization in space we
employ FEM. Let us assume that we have a hierarchy of nested triangular grids for
the unit square (compare Figure 3.2) with vertices ξil ∈ Ω, i = 1, . . . , NVl , on level
l ∈ N. Let the set of triangular elements on level l be denoted by T l . We define
the basis functions ϕil by requiring
ϕil (ξ jl ) = δi j , ϕil is linear on each element T ∈ T l ,
22 3 Direct Optimization: Problem discretization

Level 1 Level 2 Level 3 Level 4

Figure 3.2: First four levels of an exemplary hierarchy of nested triangular meshes for the
unit square obtained by uniform refinement with NVl = (2l−1 + 1)2 vertices on level l =
1, . . . , 4.

with δi j denoting the Kronecker Delta. This construction yields the well-known hat
functions. We then define the spaces Vhl simply as the span of the basis functions
ϕil , i = 1, . . . , NVl .
For the discretization of Q we assume that a partition of ∂ Ω in segments Sk ,
k = 1, . . . , NQ , is given and choose Qh as the span of their characteristic functions
ψk = χSk , which yields a piecewise constant discretization of the control on the
boundary of the domain.
Let

Nl Nl
ul = ∑i=1 wl = ∑i=1 q = ∑i=1
N
V
uli ϕil ∈ Vhl , V
wli ϕil ∈ Vhl , Q
qi ψ i ∈ Q h

denote arbitrarily chosen discretized functions and their coordinates (in bold type-
face) within their finite dimensional spaces. Then we obtain the following expres-
sions for the terms occurring in equation (2.4) which allow for the evaluation of
the integrals via matrices:

  
Nl
ul wl = ∑i, Vj=1 uli ϕil ϕ lj wlj =: (ul )T MVl wl ,
Ω Ω
  
l
(∇u ) ∇w = ∑i, j=1 ui
N
l T l V l
(∇ϕi ) ∇ϕ j wlj =: (ul )T Sl wl ,
l T l
Ω Ω
  
NVl
αwl q = ∑i=1 ∑ j=1 i ∂ Ω i j wlj =: qT MQl wl ,
NQ
q αψ ϕ l
∂Ω
  
Nl
β ul wl = ∑i, Vj=1 uli β ϕil ϕ lj wlj =: (ul )T M∂l wl .
∂Ω ∂Ω
3.3 Direct Multiple Shooting: Discretization in time 23

The occurring matrices are all sparse because each basis function has by construc-
tion a support of only a few neighboring elements. We now substitute u(t), ϕ and
q(t) in equation (2.4) with their discretized counterparts to obtain

u̇l (t)T MVl eli = −ul (t)T Sl eli − ul (t)T M∂l eli + q(t)T MQl eli , for i = 1, . . . , NVl ,

where eli denotes the i-th column of the N l -by-N l identity matrix. Exploiting sym-
metry of MVl , Sl , and M∂l yields the equivalent linear ODE formulation

MVl u̇l (t) = (−Sl − M∂l )ul (t) + (MQl )T q(t) =: f PDE(l) (q(t), ul (t), v(t)). (3.1)

It is well-known that the state mass matrix on the left hand side of equation (3.1) is
symmetric positive definite. To multiply equation (3.1) from the left with the dense
matrix (MVl )−1 is often avoided in order to preserve sparsity of the right hand side
matrices.
We want to conclude Example 2 with the remark that in a FVM or an NDGM,
where the basis functions are discontinuous over element boundaries, the mass
matrix has block diagonal form and hence sparsity is preserved for (MVl )−1 . For
spectral methods, all occurring matrices are usually dense anyway. In both cases,
the inverse mass matrix is usually formulated explicitly in the right hand side of
equation (3.1).

3.3 Direct Multiple Shooting: Discretization in time

After approximation of PDE (2.1) with large-scale ODEs as we have described
in Section 3.2, we can employ Direct Multiple Shooting (see the seminal paper
of Bock and Plitt [25]) to discretize the ODE constrained OCP. The aim of this
section is to give an overview of Direct Multiple Shooting.
To this end let
0 = t 0 < · · · < t nMS = 1
denote a partition of the time interval [0, 1], the so-called shooting grid. We further
employ a piecewise discretization of the semi-discretized control q(t) such that
q(t) is constant on the shooting intervals

I i = (t i−1 ,t i ), i = 1, . . . , nMS ,

with values
q(t) = ∑i=1
n
q χI i (t).
MS i−1
24 3 Direct Optimization: Problem discretization

Piecewise higher order discretizations in time are also possible, as long as the
shooting intervals stay decoupled. Otherwise we loose the possibility for structure
exploitation which is important for numerical efficiency reasons as we discuss
in Chapter 8. In this thesis we restrict ourselves to piecewise constant control
discretizations in time for reasons of simplicity.
We now introduce artificial initial values (sl,i , vi ), i = 0, . . . , nMS , for the semi-
discretized PDE states ul (t) and the ODE states v(t), respectively. We define

Nl
f ODE(l) (q(t), ul (t), v(t)) := f ODE (∑ j=1 q j (t)ψ j , ∑ j=1
N
Q V
ulj (t)ϕ j , v(t))

and assume that each local IVP

Mul u̇l (t) = f PDE(l) (qi−1 , ul (t), v(t)), t ∈ Ii, ul (t i−1 ) = sl,i−1 , (3.2a)
v̇(t) = f ODE(l) (qi−1 , ul (t), v(t)), t ∈ Ii, v(t i−1 ) = vi−1 . (3.2b)

has a unique solution, denoted by the pair

(ul,i (t; qi−1 , sl,i−1 , vi−1 ), vl,i (t; qi−1 , sl,i−1 , vi−1 )).

Local existence and uniqueness of (ul,i , vl,i ) are guaranteed by the Picard-Lindelöf
theorem if the functions f PDE(l) and f ODE(l) are Lipschitz continuous in the second
and third argument. By means of (ul,i , vl,i ) we obtain a piecewise, finite dimen-
sional parametrization of the state trajectories. To recover continuity of the entire
trajectory across the shooting grid nodes we have to impose matching conditions
 l,i i i−1 l,i−1 i−1   l,i 
u (t ; q , s ,v ) s
− = 0, i = 1, . . . , nMS .
vl,i (t i ; qi−1 , sl,i−1 , vi−1 ) vi

Remark 3.1. We introduce an additional artificial control variable qnMS on the last
shooting grid node in order to have the same structure of degrees of freedom in
each ti , i = 0, . . . , nMS . We shall always require qnMS = qnMS −1 . This convention
simplifies the presentation and implementation of the structure exploitation that
we present in Chapter 8.
Remark 3.2. It is also possible and numerically advantageous to allow for differ-
ent spatial meshes on each shooting interval I i in combination with a-posteriori
mesh refinement, see Hesse [83]. In that case the matching conditions have to
be formulated differently. This topic, however, is beyond the scope of this thesis.
We restrict ourselves to uniformly refined meshes which are equal for all shooting
intervals.
3.4 Discretization of path constraints 25

3.4 Discretization of path constraints

Before we can formulate the discretized optimization problem we have been aim-
ing at in this chapter, we need to repeat on each level l the construction of f ODE(l)
from f ODE for the remaining functions
Nl
Φl (sl,nMS , vnMS ) := Φ(∑ j=1
l,nMS
V
sj ϕ j , vnMS ),
Nl
rb(l) (sl,nMS , vl,nMS ) := rb (∑ j=1
l,nMS
V
sj ϕ j , vl,nMS ),

ri (q(t), v(t)) := rc (∑ j=1

N
Q
q j (t)ψ j , v(t)).

We observe that the path constraint containing ri is supposed to hold in infinitely

many points t ∈ [0, 1]. There are different possibilities to discretize such a con-
straint (see Potschka [127] and Potschka et al. [128]). For the applications we treat
in this thesis it is sufficient to discretize path constraint (2.6e) on the shooting grid

ri (qi−1 , vi−1 ) ≥ 0, i = 1, . . . , nMS .

3.5 The resulting Nonlinear Programming Problem

Finally we arrive at a finite dimensional optimization problem on each spatial dis-
cretization level l

minimize
n
Φl (sl,nMS , vnMS ) (3.3a)
(qi ,sl,i ,vi )i=0
MS

s. t. rb(l) (sl,nMS , vl,nMS ) − (sl,0 , vl,0 ) = 0, (3.3b)

u (t ; q
l,i i i−1
,s l,i−1
,v i−1
) − s = 0,
l,i
i = 1, . . . , nMS , (3.3c)
v (t ; q
l,i i i−1
,s l,i−1
,v i−1
) − v = 0,
i
i = 1, . . . , nMS , (3.3d)
nMS −1
qnMS
−q = 0, (3.3e)
r (q
i i−1
,v i−1
) ≥ 0, i = 1, . . . , nMS , (3.3f)
re (vnMS ) ≥ 0. (3.3g)

Throughout we assume that all discretized functions are sufficiently smooth to

apply the numerical methods of Part II. This includes that all functions need to be
at least twice continuously differentiable. In the case of the functions f PDE(l) and
f ODE(l) we might even need higher regularity to allow for efficient adaptive error
control in the numerical integrator.
26 3 Direct Optimization: Problem discretization

For the efficient solution of NLP (3.3) we have developed an inexact Sequential
Quadratic Programming (SQP) method which we describe in Part II. We conclude
this chapter with a summary of the numerical challenges:
 
Large scale. The NLPs have nNLP(l) = (nMS + 1) NVl + nv + nMS NQ variables
and are thus considered large-scale for finer levels l. The numerical methods
which we describe in Part II aim at the efficient treatment of NLP (3.3) for
large NVl ≈ 105 . The number of shooting intervals nMS will be between 101
and 102 which amounts to an overall problem size nNLP(l) ≈ 107 . We want
to remark that this calculation does not include the values of ul,i which have
to be computed in intermediate time steps between shooting nodes. There
can be between 101 and 102 time steps per interval.
Efficient derivative generation. It is inevitable for the solution of large-scale op-
timization problems to use derivative-based methods. Hence we need nu-
merical methods which deliver consistent derivatives of the functions occur-
ring in NLP (3.3), especially in the matching conditions (3.3c) and (3.3d). In
Chapter 10 we describe such a method which efficiently computes consistent
derivatives of first and second order in an automated way.
Structure exploitation. Efficient numerical methods must exploit the multiple
shooting structure of NLP (3.3). We present a condensing approach in Chap-
ter 8 which operates on linearizations of NLP (3.3). Furthermore we develop
preconditioners in Chapter 6 which exploit special spectral properties of the
shooting Wronksi matrices. These spectral properties arise due to ellipticity
of the operator A.
Mesh-independent local convergence. One of the main results of this thesis is
that these preconditioners lead to mesh-independent convergence of the in-
exact SQP method, i.e., the number of iterations is asymptotically bounded
by a reasonably small constant for l → ∞. We prove this assertion for a
model problem in Chapter 6 and the numerical results that we have obtained
on the application problems in Part III suggest that this claim also holds for
difficult real-world problems.
Global convergence. Often there is only few a-priori information available about
the solution of real-world problems. Hence it is paramount to enforce con-
vergence of the inexact SQP method also from starting points far away from
the solution. However, it must be ensured that an early transition to fast local
convergence is preserved. We describe such a globalization strategy based
on natural level functions in Chapter 5.
4 Elements of optimization theory
In this short chapter we consider the NLP
minimize
n
f (x) (4.1a)
x∈R
s. t. gi (x) = 0, i∈E, (4.1b)
gi (x) ≥ 0, i∈I, (4.1c)
where f : Rn → R and g : Rn → Rm
are twice continuously differentiable functions
and the sets E and I form a partition of {1, . . . , m} =: m = E ∪˙ I . In the case of
E = m, I = {}, NLP (4.1) is called Equality Constrained Optimization Problem
(ECOP).

4.1 Basic definitions

We follow Nocedal and Wright [121] in the presentation of the following basic
definitions.
Definition 4.1. The set
F = {x ∈ Rn | gi (x) = 0, i ∈ E , gi (x) ≥ 0, i ∈ I }
is called feasible set.
Definition 4.2. A point x ∈ F is called feasible point.
Definition 4.3. A point x∗ ∈ Rn is called global solution if x∗ ∈ F and
f (x∗ ) ≤ f (x) for all x ∈ F .
Definition 4.4. A point ∈ x∗ is called local solution if x∗ ∈ F and if there
Rn
exists a neighborhood U ⊂ R of x∗ such that
n

f (x∗ ) ≤ f (x) for all x ∈ U ∩ F .

Most algorithms for NLP (4.1) do not guarantee to return a global solution,
which is virtually impossible if F is of high dimensionality as is the case in PDE
constrained optimization problems. Thus we restrict ourselves to finding only local
solutions. Research papers on global optimization can be found in Floudas and
Pardalos [56] and in the Journal of Global Optimization.

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_4,
© Springer Fachmedien Wiesbaden 2014
28 4 Elements of optimization theory

4.2 Necessary optimality conditions

The numerical algorithms to be described in Part II for approximation of a local
solution of NLP (4.1) are based on finding an approximate solution to necessary
optimality conditions. We present these conditions after the following required
definitions.
Definition 4.5. The active set at a feasible point x ∈ F is defined as

A (x) = {i ∈ m | gi (x) = 0}.

Definition 4.6. The Linear Independence Constraint Qualification (LICQ) holds

at x ∈ F if the the active constraint gradients ∇gi (x), i ∈ A (x), are linearly inde-
pendent.
Remark 4.7. There are also weaker constraint qualifications (see, e.g., Nocedal
and Wright [121]). For our purposes it is convenient to use the LICQ.
Definition 4.8. The Lagrangian function is defined by

L (z) = f (x) − ∑ yi gi (x),

i∈m

where z := (x, y) ∈ Rn+m .

The following necessary optimality conditions are also called Karush-Kuhn-
Tucker (KKT) conditions [92, 99].
Theorem 4.9 (First-Order Necessary Optimality Conditions). Suppose that x∗ ∈
Rn is a local solution of NLP (4.1) and that the LICQ holds at x∗ . Then there is a
Lagrange multiplier vector y∗ ∈ Rm such that the following conditions are satisfied
at z∗ = (x∗ , y∗ ):

∇x L (x∗ , y∗ ) = 0, (4.2a)
gi (x∗ ) = 0, i∈E, (4.2b)
gi (x∗ ) ≥ 0, i∈I, (4.2c)
y∗i ≥ 0, i∈I, (4.2d)
y∗i gi (x ) = 0,
∗
i ∈ m. (4.2e)

Proof. See Nocedal and Wright [121].

Remark 4.10. The Lagrange multipliers y are also called dual variables, in contrast
to the primal variables x. We call z∗ = (x∗ , y∗ ) primal-dual solution.
4.2 Necessary optimality conditions 29

In the next definition we characterize a property which is favorable for the de-
termination of the active set in a numerical algorithm because small changes in the
problem data will not lead to changes in the active set at the solution.

Definition 4.11. Suppose z∗ = (x∗ , y∗ ) is a local solution of NLP (4.1) satisfy-

ing (4.2). We say that the Strict Complementarity Condition (SCC) holds if yi > 0
for all i ∈ I ∩ A (x∗ ).

A useful sufficient optimality condition is based on the notion of two cones:

Definition 4.12. Let x ∈ F . The cone of linearized feasible directions is defined

by

Fl (x) = {d ∈ Rn | d T ∇gi (x) = 0, i ∈ E , d T ∇gi (x) ≥ 0, i ∈ A (x) ∩ I }.

Definition 4.13. Let x ∈ F . The cone of critical directions is defined by

C (x, y) = {d ∈ Fl (x) | d T ∇gi (x) = 0, for all i ∈ A (x) ∩ I with yi > 0}.

The cone of critical directions plays an important role in the following sufficient
optimality condition.

Theorem 4.14. Let (x∗ , y∗ ) satisfy the KKT conditions (4.2). If furthermore the
Second Order Sufficient Condition (SOSC)

d T ∇2xx L (x∗ , y∗ )d > 0 for all d ∈ C (x∗ , y∗ ) \ {0}

holds then x∗ is a strict local solution.

Proof. See Nocedal and Wright [121].

 Part II

Numerical methods
5 Inexact Sequential Quadratic
Programming
In this chapter we develop a novel approach for the solution of inequality con-
strained optimization problems. We first describe inexact Newton methods in Sec-
tion 5.1 and investigate their local convergence in Section 5.2. In Section 5.3 we
review strategies for the globalization of convergence and explain a different ap-
proach based on generalized level functions and monotonicity tests. An example
in Section 5.4 illustrates the shortcomings of globalization strategies which are not
based on the so called natural level function. We review the Restrictive Monotonic-
ity Test (RMT) in Section 5.5 and propose a Natural Monotonicity Test (NMT) for
Newton-type methods based on a Linear Iterative Splitting Approach (LISA). This
combined approach allows for estimation of the critical constants which character-
ize convergence. We finally present how these results can be extended to global
inexact SQP methods. We present efficient numerical solution techniques of the
resulting sequence of Quadratic Programming Problems (QPs) in Chapters 8 and 9.

5.1 Newton-type methods

We consider the problem of finding a zero of a nonlinear function F : D ⊆ RN →
RN which we assume to be continuously differentiable with Jacobian denoted by
J. This case is important for computing local solutions of an ECOP because its
KKT conditions (4.2) reduce to a system of nonlinear equations
 
∇L (z)
F(z) := = 0,
g(x)

where N = n + m and z = (x, y) are the compound primal-dual variables. We shall

discuss extensions for the inequality constrained case in Section 5.7.
The numerical methods of choice for the solution of F(z) = 0 are Newton-type
methods: Given an initial solution guess z0 , we iterate

Δzk = −M(zk )F(zk ), zk+1 = zk + αk Δzk , (5.1)

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_5,
© Springer Fachmedien Wiesbaden 2014
34 5 Inexact Sequential Quadratic Programming

with scalars αk ∈ (0, 1] and matrices M(z) ∈ RN×N . The scalar αk is a damping or
underrelaxation parameter. We shall see in Sections 5.2 and 5.3 that the choice of
αk = 1 is necessary for fast convergence close to a solution but choices αk < 1 are
necessary to achieve convergence from initial guesses z0 which are not sufficiently
close to a solution.
Different choices of M lead to different Newton-type methods. Typical and im-
portant choices for M include Quasi-Newton methods (based on secant updates,
see, e.g., Nocedal and Wright [121]), the Simplified Newton method (M(z) =
J −1 (z0 )), and the Newton method (M(z) = J −1 (z)), provided the inverses exist.
We have developed a method which uses Linear Iterative Splitting Approach (see
Section 5.6.2) with a Newton-Picard deflation preconditioner (described in Chap-
ter 6) to evaluate M.
Before we completely dive into the subject we want to clarify the naming of
methods. We use Newton-type method as a collective term to refer to methods
which can be cast in the form of equation (5.1). If the linearized subproblems
are solved by an iterative procedure we use the term inexact Newton method. Un-
fortunately the literature is not consistent here: The often cited paper by Dembo
et al. [41] uses inexact Newton method in the sense of our Newton-type method and
Newton-iterative method in the sense of our inexact Newton method to distinguish
between solving

˜ k )Δzk = −F(zk )
J(z or J(zk )Δzk = −F(zk ) + rk , (5.2)

where rk ∈ RN is a residual and J(z

˜ k ) ≈ J(zk ). Some authors, e.g., Morini [115],
further categorize inexact Newton-like methods which solve

˜ k )Δzk = −F(zk ) + rk
J(z

in each step. From the point of view that equations (5.2) are merely characterizing
the backward and the forward error of Δzk for J(zk )Δzk = −F(zk ), we believe that
equations (5.2) should not be the basis for categorizing algorithms but rather be
kept in mind for the analysis of all Newton-type methods. The following lemma
shows that one can move from one interpretation to the other:

Lemma 5.1. Let Δz∗ solve J(zk )Δz∗ = −F(zk ). If Δzk is given via

Δzk = −M(zk )F(zk ) or ˜ k )Δzk = −F(zk )

J(z

then the residual can be computed as

rk = J(zk )Δzk + F(zk ).

5.1 Newton-type methods 35

 
Conversely, if rk and Δzk are given and Δzk 2 > 0 then one possible J(z
˜ k ) is given
by
˜ k ) = J(zk ) − rk (Δz ) .
k T
J(z
(Δzk )T Δzk
Moreover, if J(zk ) is invertible and (Δzk )T Δz∗ = 0 then
 
k −1 (Δzk − Δz∗ )(Δzk )T
M(z ) = J(z ) = I +
k ˜ J(zk )−1 .
(Δzk )T Δz∗

Proof. The first assertion is immediate. The second assertion can be shown via

˜ k )Δzk = −F(zk ) + rk − rk (Δz ) Δz = −F(zk ).

k T k
J(z
(Δzk )T Δzk

By virtue of the Sherman-Morrison-Woodbury formula (see, e.g., Nocedal and

Wright [121]) we obtain

rk (Δz )
k T
J(zk )−1 (Δz k −1
k )T Δzk J(z )
k −1 k −1
M(z ) = J(z
k ˜ ) = J(z ) + .
1 − (Δz(Δz )
k T
k −1 r
k )T Δzk J(z ) k

The last assertion then follows from J(zk )−1 rk = Δzk − Δz∗ .
In this thesis we focus on computing Δzk iteratively via the iteration

Δzki+1 = Δzki − M̂(zk )(J(zk )Δzki + F(zk )), (5.3)

with M̂(zk ) ∈ RN×N . We call iteration (5.3) Linear Iterative Splitting Approach
(LISA) to emphasize that the iteration (which we further discuss in Section 5.6.2)
is linear and based on a splitting

J(zk ) = J(z
ˆ k ) − ΔJ(zk ), ˆ k )−1 .
M̂(zk ) = J(z

ˆ k ) will be given by a Newton-Picard preconditioner (see Chap-

In this thesis J(z
ter 6). For other possible choices of J(zˆ k ) in this context, which include Jacobi,
Gauß-Seidel, Successive Overrelaxation, etc., we refer the reader to Ortega and
Rheinboldt [122] and Saad [135]. There is no consistent naming convention avail-
able in the literature: We can find names like generalized linear iterations (Ortega
and Rheinboldt [122]) or basic linear methods (Saad [135]) for what we call LISA.
A characterization of M(zk ) based on M̂(zk ) for LISA in terms of a truncated Neu-
mann series shall be given in Lemma 5.27.
36 5 Inexact Sequential Quadratic Programming

A linear iteration like (5.3) can in principle be accelerated by the use of Krylov-
space methods at the cost of making the iteration nonlinear. We abstain from
nonlinear acceleration in this thesis because the Newton-Picard preconditioners
are already powerful enough when used without acceleration (see Chapter 6).
In the following sections we review the theory for local and global convergence
of Newton-type methods.

5.2 Local convergence

We present a variant of the Local Contraction Theorem (see Bock [24]). Let the
set of Newton pairs be defined according to

N = {(z, z ) ∈ D × D | z = z − M(z)F(z)}

and let . denote a norm of RN . We need two conditions on J and M:

Definition 5.2 (Lipschitz condition: ω-condition). The Jacobian J together with
the approximation M satisfy the ω-condition in D if there exists ω < ∞ such that
for all t ∈ [0, 1], (z, z ) ∈ N
     
M(z ) J(z + t(z − z)) − J(z) (z − z ) ≤ ωt z − z 2 .

Definition 5.3 (Compatibility condition: κ-condition). The approximation M sat-

isfies the κ-condition in D if there exists κ < 1 such that for all (z, z ) ∈ N
   
M(z )(I − J(z)M(z))F(z) ≤ κ z − z  .

Remark 5.4. If M(z) is invertible, then the κ-condition can also be written as
   
M(z )(M −1 (z) − J(z))(z − z ) ≤ κ z − z  , ∀(z, z ) ∈ N .

With the constants from the previous two definitions we define

 
 
ck = κ + (ω/2) Δzk 

and for c0 < 1 the closed ball

 
D0 = B(z0 ; Δz0  /(1 − c0 )).

The following theorem characterizes the local convergence of a full step (i.e.,
αk = 1) Newton-type method in a neighborhood of the solution. Because of its
importance we include the well-known proof.
5.2 Local convergence 37

Theorem 5.5 (Local Contraction Theorem). Let J and M satisfy the ω-κ-condi-
tions in D and let z0 ∈ D. If c0 < 1 and D0 ⊆ D, then zk ∈ D0 and the sequence
(zk ) converges to some z∗ ∈ D0 with convergence rate
       2
 k+1       
Δz  ≤ ck Δzk  = κ Δzk  + (ω/2) Δzk  .

Furthermore, the a-priori estimate

  (ck ) j  
 k+ j   k  (c0 )k+ j  
Δz0 
z − z∗  ≤ Δz  ≤
1 − ck 1 − c0
holds and the limit z∗ satisfies

M(z∗ )F(z∗ ) = 0.

If additionally M(z) is continuous and nonsingular in z∗ , then

F(z∗ ) = 0.

Proof based on the Banach Fixed Point Theorem. The assumption c0 < 1 and the
Definition of D0 imply that z0 , z1 ∈ D0 . We assume that zk+1 ∈ D0 . Then
   
 k+1   
Δz  = M(zk+1 )F(zk+1 )

= M(zk+1 ) F(zk ) − J(zk )M(zk )F(zk )

+ M(zk+1 ) F(zk+1 ) − F(zk ) + J(zk )M(zk )F(zk ) 

    1 
   dF k 
≤ κ Δzk  + 
 M(zk+1
) (z + tΔzk
)dt − J(zk
)Δz k 

0 dt
   1 
   
≤ κ Δzk  + M(zk+1 ) J(zk + tΔzk ) − J(zk ) Δzk  dt
0
   2  
     
≤ κ Δzk  + (ω/2) Δzk  = ck Δzk  .

It follows that the sequence (ck ) is monotonically decreasing because

ω 

 ω 
 ω 

ck+1 = κ + Δzk+1  ≤ κ + ck Δzk  = ck − (1 − ck ) Δzk  ≤ ck .
2 2 2
Telescopic application of the triangle inequality yields zk+2 ∈ D0 due to
  k+1   k+1  
 k+2 0    Δz0 
z − z  ≤ ∑ Δz j  ≤ ∑ (c0 ) j Δz0  ≤ .
j=0 j=0 1 − c0
38 5 Inexact Sequential Quadratic Programming

By induction we obtain zk ∈ D0 for all k ∈ N. From

  k+ j−1    0
   
k Δz
j−1
 k+ j 
z − zk  = ∑ Δzi  ≤ ∑ 0 (c )k  i
Δz ≤ (c 0 )
i=k i=0 1 − c0

follows that (zk ) is a Cauchy sequence and thus converges to a fixed point z∗ ∈ D0 .
For the a-priori estimate consider
  ∞   ∞   (ck ) j  
 k+ j       k
z − z∗  ≤ ∑ Δzk+ j+i  ≤ ∑ (ck )i Δzk+ j  ≤ Δz  .
i=0 i=0 1 − ck

In the limit
z∗ = z∗ − M(z∗ )F(z∗ ) ⇒ M(z∗ )F(z∗ ) = 0
holds which shows the remaining assertions.
Remark 5.6. If F is linear we obtain ω = 0 and if furthermore M(z) is constant the
convergence theory is completely described by Theorem 5.26 to be presented.
Remark 5.7. Assume J is nonsingular throughout D0 . Then the full step Newton
method with M(z) = J −1 (z) converges quadratically in D0 due to κ = 0.
Remark 5.8. In accordance with Deuflhard’s algorithmic paradigm (see Deuflhard
[44]) we assume the constants κ and ω to be the infimum of all possible candidates
which satisfy the inequalities in their respective definitions. These values are in
general computationally unavailable. Within the algorithms to be presented we
approximate the infima from below with computational estimates denoted by [κ]
and [ω] by sampling the inequalities over a finite subset N ⊂ N which comprises
various iterates of the algorithm.

5.3 Globalization of convergence

Most strategies for enforcing global convergence of inexact SQP methods are
based on globalization techniques like trust region (see, e.g., Heinkenschloss and
Vicente [82], Heinkenschloss and Ridzal [81], Walther [158], Gould and Toint
[65]) or line search (see, e.g., Biros and Ghattas [20], Byrd et al. [33]). The ex-
plicit algorithmic control of Jacobian approximations is usually enforced via an
adaptively chosen termination criterion for an inner preconditioned Krylov solver
for the solution of the linearized system. In some applications, efficient precon-
ditioners are available which cluster the eigenvalues of the preconditioned system
and thus effectively reduce the number of inner Krylov iterations necessary to
5.3 Globalization of convergence 39

solve the linear system exactly (see Battermann and Heinkenschloss [9], Batter-
mann and Sachs [10], Biros and Ghattas [19]).
We shall show in Section 5.4 that both line search and trust region methods can
lead to unnecessarily damped iterates in the vicinity of the solution where fast local
convergence in the sense of the Local Contraction Theorem 5.5 is already possible.
Our aim in this section is to present the theoretical tools to understand this un-
desirable effect and to introduce the idea of monotonicity tests.
We begin the discussion on the basis of the Newton method and extend it to
Newton-type methods in Section 5.6 and to inexact SQP methods for inequality
constrained optimization problems in Section 5.7.

5.3.1 Generalized level functions

It is well known that the local Newton method with αk = 1 is affine invariant under
linear transformations in the residual and variable space:

Lemma 5.9. Let A, B ∈ GL(N). Then the iterates zk for the Newton method on
F(z) and the iterates z̃k for
F̃(z̃) := AF(Bz̃)
with z̃0 := B−1 z0 are connected via

z̃k = B−1 zk , k ∈ N.

Proof. Let k ∈ N. We assume z̃k = B−1 zk , obtain

˜ k )−1 F(z̃k ) = z̃k − B−1 J(Bz̃k )−1 A−1 AF(Bz̃k ) = B−1 zk+1 ,
z̃k+1 = z̃k − J(z̃

and complete the proof by induction.

It is desirable to conserve at least part of the invariance for the determination of
the damping parameters αk . Our goal here are globalization strategies which are
invariant under linear transformations in the residual space with A ∈ GL(N), B =
I. This type of invariance is called affine covariance (see Deuflhard [44] for a
classification of invariants for local and global Newton-type methods). We shall
elaborate in Section 5.4 why affine covariance is important for problems which
exhibit high condition numbers of the Jacobian J(z∗ ) in the solution. This is the
typical case in PDE constrained optimization problems.
We can immediately see that the Lipschitz constant ω in Definition 5.2 and the
compatibility constant κ in Definition 5.3 are indeed independent of A. Thus they
lend themselves to be used in an affine invariant globalization strategy.
40 5 Inexact Sequential Quadratic Programming

We conclude this section with a descent result for the Newton direction on gen-
eralized level functions
T (z|A) := 12 AF(z)22 , A ∈ GL(N).
Generalized level functions extend the concept of the classical level function T (z|I)
and play an important role in affine invariant globalization strategies. The follow-
ing simple but nonetheless remarkable lemma (see, e.g., Deuflhard [44]) shows
that the Newton direction is a direction of descent for all generalized level func-
tions.
Lemma 5.10. Let F(z) = 0. Then, for all A ∈ GL(N), the Newton increment
Δz = −J(z)−1 F(z) satisfies
ΔzT ∇T (z|A) = −2T (z|A) < 0.
 T
Proof. ΔzT ∇T (z|A) = −F(z)T J(z)−T F(z)T AT AJ(z) = −2T (z|A) < 0.
However, decrease T (z + αΔz|A) < T (z|A) might only be valid for α  1. We
shall illustrate this problem with an example in Section 5.4. For the construction of
efficient globalization strategies, A must be chosen such that the decrease condition
is valid for a maximal range of α, as we shall discuss in Sections 5.5 and 5.6.

5.3.2 The Newton path

The Newton path plays a fundamental role in affine invariant globalization strate-
gies for the Newton method. We present two characterizations of the Newton path,
one geometrical and one based on a differential equation.
For preparation let A ∈ GL(N) and define the level set associated with T (z|A)
according to
G(z|A) := {z ∈ D ⊆ RN | T (z |A) ≤ T (z|A)}.
Iterative monotonicity (descent) with respect to T (z|A) can the be written in the
form
zk+1 ∈ G̊(zk |A), if G̊(zk |A) = {}.
To achieve affine invariance of the globalization strategy, descent must be inde-
pendent of A. Thus we define

G(z) := G(z|A).
A∈GL(N)

The geometric derivation of the Newton path due to Deuflhard [42, 43, 44] and
the connection to the continuous analog of the Newton method characterized by
Davidenko [38] is given by the following result.
5.3 Globalization of convergence 41

Theorem 5.11. Let J(z) be nonsingular for all z ∈ D. For some A ∈ GL(N) let the
 in z0 be compact and contained in D. Then
path-connected component of G(z0 |A)
the path-connected component of G(z0 ) is a topological path z : [0, 2] → RN , the
so-called Newton path, which satisfies

F(z(α)) = (1 − α)F(z0 ), (5.4a)

T (z(α)|A) = (1 − α) T (z |A) ∀A ∈ GL(N),
2 0
(5.4b)
dz
(α) = −J(z(α))−1 F(z0 ), (5.4c)
dα
z(0) = z0 , z(1) = z∗ , (5.4d)

dz 
 = −J(z0 )−1 F(z0 ) = Δz0 . (5.4e)
dα α=0
Proof. See Deuflhard [44, Theorem 3.1.4].
Remark 5.12. The differential equation (5.4c) is derived from the homotopy

H(z, α) = F(z) − (1 − α)F(z0 ) = 0, (5.5)

which gives rise to the function z(α) upon invocation of the Implicit Function
Theorem. After solving equation (5.5) for F(z0 ) and using the reparametrization
α(t) = 1 − exp(−t) we can recover the so-called continuous Newton method or
Davidenko differential equation (Davidenko [38])

J(z(t))ż(t) = −F(z(t)), t ∈ [0, ∞), z(0) = z0 . (5.6)

Theorem 5.11 justifies that the Newton increment Δzk is a distinguished direc-
tion not only locally but also far away from a solution. It might only be too large,
hence necessitating the need for damping through αk .
In other words, the Newton path is the set of points generated from infinitesimal
Newton increments (denoted by ż(t) instead of Δzk ). Performing nonzero steps
αk Δzk in iteration (5.1) for the Newton method gives rise to a different Newton
path emanating from each zk , k ∈ N. It seems inevitable that for a proof of global
convergence based on the Newton path the iterates zk must be related to a single
Newton path, which we discuss in Section 5.5.

5.3.3 The natural level function and the Natural Monotonicity

Test
In this section we assemble results for the natural level function and the NMT.
For the presentation we follow Section 3.3 of Deuflhard [44]. At first, we restrict
42 5 Inexact Sequential Quadratic Programming

the presentation to the Newton method. Extensions to Newton-type methods with

iterative methods for the linear algebra shall be discussed in Section 5.6. The main
purpose of this section is the presentation of the natural level functions, defined by

Tk∗ (z) := T (z|J(zk )−1 ),

which have several attractive properties. We can already observe that descent in
Tk∗ can be evaluated by testing for natural monotonicity
   
 k+1   k 
Δz  < Δz  ,

where one potential step of the Simplified Newton method can be used to evaluate
Δzk+1 according to
J(zk )Δzk+1 = −F(zk+1 ).
As already mentioned, the Lipschitz constant ω plays a fundamental role in the
global convergence theory based on generalized level functions. In contrast to
Bock [24], however, Deuflhard [44] uses a different definition for the Lipschitz
constant:

Definition 5.13 (ω-condition for the Newton method). The Jacobian J satisfies

the ω-condition  < ∞ such that for all (z, z ) ∈ D × D
in D if there exists ω
     2
J(z)−1 J(z ) − J(z) (z − z) ≤ ω
 z − z .

 we define the set of

Remark 5.14. In order to compare the magnitudes of ω and ω
interpolated Newton pairs

z) ∈ D × D | t ∈ [0, 1], (z, z ) ∈ N ,

Nt = {(z, z = z + t(z − z), z = 
z}.

Then we can compute the smallest ω according to

  −1 
J(z ) (J(z + t(z − z)) − J(z))t(z − z)
ω= sup
(z,z )∈N ,z=z t 2 z − z2
t∈(0,1]
 
J(z − J(z)−1 F(z))−1 (J( z − z)
z) − J(z)) (
= sup ,
(z,
z)∈Nt z − z2


which coincides with Definition 5.13 of ω except for the evaluation of the weight-
ing matrix in the Lipschitz condition at a different point. Because Nt is much
smaller than D × D, the constant ω must be expected to be much larger than ω.
5.3 Globalization of convergence 43

This will lead to smaller bounds on the step sizes αk . Furthermore, in practical
computations with a Newton-type method, only ω can be estimated efficiently
 is not explicitly restricted to the set of Newton pairs
from the iterates because ω
N.
Remark 5.15. Most proofs which rely on ω  can also be carried out in a similar
fashion with ω but some theoretical results cannot be stated as elegantly. As an
example, the occurrence of the condition number cond(AJ(zk )) in Theorem 5.16
 We take up the pragmatic position that ω should be used for all
relies on using ω.
 if we can gain theoretical insight
practical computations but we also recede to ω
about qualitative convergence behavior of Newton-type methods.
The first theorem which relies on ω  characterizes step sizes αk which yield
optimal values for a local descent estimate of generalized level functions T (z|A)
in the Newton method.
Theorem 5.16. Let D be convex, J(z) nonsingular for all z ∈ D. Let furthermore J

satisfy the ω-condition in D, zk ∈ D, A ∈ GL(N), and G(zk |A) ⊂ D. Let Δzk denote
the Newton direction and define the Kantorovich quantities
 
 
hk := Δzk  ω, hk := hk cond(AJ(zk )).

Then we obtain for α ∈ [0, min(1, 2/hk (A))] that

   
   
AF(zk + αΔzk ) ≤ tk (α|A) AF(zk ) ,

where
tk (α|A) := 1 − α + (1/2)α 2 hk (A).
The optimal choice of the damping factor in terms of this local estimate is

α k (A) := min(1, 1/hk (A)).

Proof. See Deuflhard [44, Theorem 3.12].

Theorem 5.16 lends itself to the following global convergence theorem.
Theorem 5.17. In addition to the assumptions of Theorem 5.16 let D0 denote the
path-connected component of G(z0 |A) in z0 and assume that D0 ⊆ D is compact.
Then the damped Newton iteration with damping factors

αk ∈ [ε, 2α k (A) − ε]

and sufficiently small D0 -dependent ε > 0 converges to a solution point z0 .

44 5 Inexact Sequential Quadratic Programming

Proof. See Deuflhard [44, Theorem 3.13].

Theorem 5.17 is a theoretical result for global convergence based on descent in
any Generalized level function T (z|A) with fixed A. However, the “optimal” step
size chosen according to Theorem 5.16 is reciprocally proportional to the condition
number cond(AJ(zk )). Thus a choice of A far away from J(zk )−1 , e.g., A = I on
badly conditioned problems, will lead to quasi-stalling of the globalized Newton
method even within the domain of local contraction. Such a globalization strategy
is practically useless for difficult problems, even though there exists a proof of
global convergence.
This observation has led to the development of natural level functions Tk∗ =
T (z|J(zk )−1 ). The choice of Ak = J(zk )−1 yields the optimal value of

1 ≤ cond2 (Ak J(zk )) = 1,

and thus the largest value for the step size α k . As already mentioned at the begin-
ning of this section, we recall that descent in Tk∗ can be evaluated by the NMT
   
 k+1   k 
Δz  < Δz  ,

where Δzk+1 is the increment for one potential Simplified Newton step.
Natural level functions have several outstanding properties as stated by Deufl-
hard [44, Section 3.3.2]:

Extremal properties For A ∈ GL(N) the reduction factors tk (α|A) and the theo-
retical optimal damping factors α k (A) satisfy

tk (α|Ak ) = 1 − α + (1/2)α 2 hk ≤ tk (α|A),

α k (Ak ) = min(1, 1/hk ) ≥ α k (A).

Steepest descent property The steepest descent direction for T (z|A) in zk is

−∇T (zk |A) = −(AJ(zk ))T AF(zk ).

With A = Ak we obtain

Δzk = −∇T (zk |Ak ),

which means that the damped Newton method in zk is a method of steepest

descent for the natural level function Tk∗ .
5.4 A Rosenbrock-type example 45

Merging property Full steps and thus fast local convergence are guaranteed in
proximity of the solution
 
 k  ⇒ hk ≤ 1 ⇒ α k (Ak ) = 1.
Δz  ≤ 1/ω
2

Asymptotic distance function For F ∈ C2 (D), we verify that

T (z|J(z∗ )−1 ) = 12 z − z∗ 22 + O(z − z∗ 32 ).

Hence the NMT asymptotically estimates monotonicity in the distance to

the solution. The use of Ak can be considered a nonlinear preconditioner.

However, a straight-forward proof of global convergence similar to Theorem

5.17 is not possible because Ak is not kept fixed for all iterations. A Newton-type
method with global convergence proof based on the Newton path is outlined in
Section 5.5.

5.4 A Rosenbrock-type example

In order to appreciate the significance of the natural level function let us consider
the following example due to Bock [24] and the discussion therein. We use the
Newton method to find a zero of the function
 
z1
F(z) =
50z2 + (z1 − 50)2 /4

with starting guess z0 = (50, 1) and solution z∗ = (0, −12.5) (compare Figure 5.1).
We observe that the classical level set (contained within the dashed curve) is
shaped like a bent ellipse. The excentricity of the ellipse is due to the condi-
tion number of J(z0 ), which is cond2 (J(z0 )) = 50. The ellipse is bent because of
the mild nonlinearity in the second component of F. A short computation yields
ω ≤ 0.01.
We further observe that the direction of steepest descent for the classical level
function T (z|I) and for the natural level function T (z|J(z0 )−1 ) describe an angle of
87.7 degrees. Thus the Newton increment, which coincides with the direction of
steepest descent for the natural level function (see Section 5.3.3), is almost parallel
to the tangent on the classical level set. Consequently only heavily damped Newton
steps lead to a decrease in the classical level function. We obtain optimal descent
in T (z|I) with a stepsize of α0 ≈ 0.077, although the solution z∗ can be reached
46 5 Inexact Sequential Quadratic Programming

with two iterations of a full step Newton method. This behavior illustrates how
the requirement of descent in the classical level function impedes fast convergence
within the domain of local contraction.
In real-world problems the conditioning of J(z∗ ) is typically a few orders of
magnitude higher than 50, leading to even narrower valleys in the classical level
function. Additionally, nonlinear and highly nonlinear problems with larger ω
give rise to higher curvature of these valleys, rendering the requirement of descent
in the classical level function completely inappropriate.
Especially in the light of inexact Newton methods as we describe in Section 5.6,
the use of natural level functions T (z|J(zk )−1 ) is paramount: Even relatively small
perturbations of the exact Newton increment can result in the inexact Newton in-
crement being a direction of ascent in T (z|I).

5.5 The Restrictive Monotonicity Test

Bock et al. [26] have developed a damping strategy called the RMT which can
be interpreted as a step size control for integration of the Davidenko differential
equation (5.6) with the explicit Euler method. The Euler method can be extended
by a number of so called back projection steps which diminish the distance of the
iterates to the Newton path emanating from z0 . The quantities involved in the first
back projection must anyway be computed to control the step size. Numerical
experience seems to suggest that more than one back projection step does not
improve convergence considerably and should thus be avoided in all known cases.
However, repeated back projection steps provide the theoretical benefit of making
a proof of global convergence of the RMT possible. In particular, the RMT does
not lead to iteration cycles on the notorious example by Ascher and Osborne [7] in
contrast to the NMT.

5.6 Natural Monotonicity for LISA-Newton methods

In this section we give a detailed description of an affine covariant globalization
strategy for a Newton-type method based on iterative linear algebra. The linear
solver must supply error estimates in the variable space. This strategy is described
in Deuflhard [44, Section 3.3.4] for the linear solvers CGNE (see, e.g., Saad [135])
and GBIT (due to Deuflhard et al. [45]). We describe a LISA within the Newton-
type framework in Section 5.6.2. We have also developed a suitable preconditioner
for the problem class (3.3) which we present in Chapter 6.
5.6 Natural Monotonicity for LISA-Newton methods 47

60

40

20
z2 coordinate

-20

-40

-60

-80
-50 0 50
z1 coordinate

Figure 5.1: Rosenbrock-type example (adapted from Bock [24]): The Newton increment
(solid arrow) is almost perpendicular to the direction of steepest descent for the classi-
cal level function T (z|I) (dashed arrow) in the initial iterate (marked by ◦). Only heavily
damped Newton steps lead to descent in the classical level function due to the narrow and
bent classical level set (contained within the dashed curve). In contrast, the Newton step
is completely contained within the more circle-shaped natural level set (contained within
the solid curve) corresponding to the natural level function T (z|J(z0 )−1 ). Within two full
Newton steps, the solution (marked by ×) is reached.

5.6.1 Estimates for natural monotonicity

Let Δzk denote the exact Newton step and δ zk the inexact Newton step obtained
from LISA. Furthermore, define the residual

rk := J(zk )(δ zk − Δzk ).

48 5 Inexact Sequential Quadratic Programming

We want to characterize the error of LISA by the quantity

   
   
δk := δ zk − Δzk  / δ zk  .

The framework for global convergence is natural monotonicity subject to pertur-

bations due to the inexactness of δ zk . First, we study the contraction factors
   
   
Θk (α) = Δzk+1 (α) / Δzk 

in terms of the exact Newton steps Δzk and the exact Simplified Newton steps
Δzk+1 (α) defined via
J(zk )Δzk+1 (α) = −F(zk + αδ zk ).
We emphasize the occurrence of the inexact Newton step δ zk on the right hand
side.
 
Lemma 5.18. Let δk < 1 and define hδ := ω
2
 δ zk . Then we obtain the estimate
k
 
δk 1 hδ
Θk (α) ≤ 1 − 1 − α + α2 k . (5.7)
1 − δk 2 1 − δk
The optimal damping factor is
α k = min(1, (1 − 2δk )/hδk ).
If we implicitly define ρ via
ρ δ
δk =
αh (5.8)
2 k
and assume that ρ ≤ 1 we obtain the optimal damping factor

α k = min(1, 1/ (1 + ρ)hδk ). (5.9)

Proof. See Deuflhard [44, Lemma 3.17].

This test can and should not be directly implemented because the computation
of the constant hδk and the exact Simplified Newton step Δzk+1 is prohibitively
expensive.
An appropriate replacement for the nonrealizable Θk is the inexact Newton path

z(α), α ∈ [0, 1], which we can implicitly define via
z(α)) = (1 − α)F(zk ) + αrk .
F(
We immediately observe that 
z(0) = zk ,  z(1) = z∗ if rk = 0. We can
z˙(0) = δ zk , but 
now define an exact Simplified Newton step on the perturbed residual via
J(zk )Δzk+1 = −F(zk + αδ zk ) + rk . (5.10)
5.6 Natural Monotonicity for LISA-Newton methods 49

Lemma 5.19. With the current notation and definitions we obtain the estimate
   
 k+1   
Δz − (1 − α)δ zk  ≤ 12 α 2 hδk δ zk  .

Proof. See Deuflhard [44, Lemma 3.18].

For computational efficiency reasons we must also refrain from solving equa-
tion (5.10) exactly. Instead we use LISA which introduces another residual error,
denoted by  rk+1
i and defined for each inner iteration (LISA iteration) i. Then we
can define an i-dependent inexact Simplified Newton step via

J(zk )δ zk+1
i = (−F(zk + αδ zk ) + rk ) + 
rk+1
i .

As in the above formula, we need to keep the dependence of δ zk+1 i on α k in mind

but drop it in the notation for the sake of brevity. It is now paramount for the effi-
ciency of the Newton-type method to balance the accuracies of the inner iterations
with the nonlinearity of the problem.
Lemma 5.19 suggests to use a so-called cross-over of initial values [44] for
LISA according to
δ zk+1
0 = (1 − α)δ zk , δ zk0 = δ zk , (5.11)

which predict the solution to first order in α.

We substitute the nonrealizable contraction factor Θk now by
   
k = 
Θ
  
δ zk+1  / δ zk  ,

which can be computed efficiently. The following lemma characterizes the depen-
 k on α.
dence of Θ

Lemma 5.20. Assume that LISA for equation (5.10) with initial value crossing
(5.11) has been iterated until
 
 k+1 
Δz − δ zk+1
i 
ρi = 
 k+1
 < 1.
 (5.12)
Δz − δ zk+1
0 

Then we obtain the estimate

  1 + ρ  
 k+1  i 2 δ  k
δ zi − (1 − α)δ zk  ≤ α hk δ z  .
2
50 5 Inexact Sequential Quadratic Programming

Proof. The proof for the LISA case is the same as for the GBIT case, see Deuflhard
[44, Lemma 3.20].
The quantity ρi , however, cannot be evaluated directly because we must not
compute Δzk+1 exactly for efficiency reasons. Instead we define the computable
estimate  
 k+1 
Δz − δ zk+1i  εi

ρi =  ≈ , (5.13)
k+1   k+1 
δ zi − δ z0  δ zi − δ zk+1
k+1
0 

where 
ε i is an estimate for the error of LISA, see Section 5.6.2.

Lemma 5.21. With the notation and assumptions of Lemma 5.20 we have
   
 k+1    k+1 k+1 
Δz − δ zk+1 i  ≤ ρ i (1 + ρ i ) Δz − δ z0 .

Proof. We follow Deuflhard [44, Lemma 3.20 for GBIT and below]: The applica-
tion of the triangle inequality and assumptions (5.12) and (5.11) yield
     
 k+1     
δ zi − (1 − α)δ zk  ≤ Δzk+1 − (1 − α)δ zk  + δ zk+1 i − Δzk+1 
 
 
= (1 + ρi ) Δzk+1 − (1 − α)δ zk  .

Using definition (5.13) on the left hand side then delivers the assertion.
An immediate consequence of Lemma 5.21 is the inequality

ρi ≤ ρ i (1 + ρi ). (5.14)

Deuflhard [44] proposes to base the estimation of ρi on equating the left and right
hand sides of inequality (5.14) to obtain

ρ i = ρi /(1 + ρi ), or ρi = ρ i /(1 − ρ i ) for ρ i < 1. (5.15)

Then accuracy control for the inner iterations can be based on the termination
condition
ρi ≤ ρmax with ρmax ≤ 14 ,
or, following (5.15),
ρ i ≤ ρ max with ρ max ≤ 13 .
We feel urged to remark that this is heuristic insofar as from inequality (5.14) we
can only conclude
ρ i ≥ ρi /(1 + ρi ) (and not “≤”).
5.6 Natural Monotonicity for LISA-Newton methods 51

Thekoptimal
 damping factor αk from Lemma 5.18 depends on the unknown hδk =
 
 δ z which must be approximated. Using the [.] notation (see Remark 5.8) we
ω
approximate hδk with a so-called Kantorovich estimate
 
 
[hδk ] = [ω]

δ zk  ≤ hδk ,

which leads via equation (5.8) to a computable estimate of the optimal step size

[α k ] = min(1, (1 − 2δk )/[hδk ]) = min(1, 1/ (1 + ρ)[hδk ] ).

Based on Lemma 5.20 we obtain an a-posteriori Kantorovich estimate

   
   
2 δ zk+1
i − δ zk+1
0  2 δ zk+1
i − (1 − α)δ zk 
δ
[hk ]i = = ≤ hδk .
(1 + ρi )α 2 δ zk  (1 + ρi )α 2 δ zk 
Replacing ρi by ρ i yields a computable a-posteriori Kantorovich estimate
   
 k+1   k+1 k
2(1 − ρ i ) δ zk+1
i − δ z0  2(1 − ρ i )  δ z i − (1 − α)δ z 
[hδk ]i = = ≤ hδk .
α 2 δ zk  α 2 δ zk 
(5.16)
From the definition of [hδk ] we can also derive a computable a-priori Kantorovich
estimate  k+1 
δ z 
δ
[hk+1 ] = [hδ ]∗ , (5.17)
δ zk  k
where [hδk ]∗ denotes the Kantorovich estimate after the last inner iteration.
The following bit counting lemma finally supplies bounds for the exact and
inexact contraction factors.
Lemma 5.22. Let an inexact Newton method with step sizes α = [α k ] be realized.
Assume that the leading binary digit of [hδk ] is correct, i.e.,

0 ≤ hδk − [hδk ] < σ max(1/(1 + ρ), [hδk ]) for some σ < 1.

Then the exact natural contraction factor satisfies
 k+1 
Δz  1 − σ (1 + 2ρ)
Θk = < 1− α.
Δzk  2 + ρ(1 − σ )
The inexact natural contraction factor is bounded by
 
k = δ z
k+1 1 (1 + ρ)(1 + σ )
Θ < 1 − 1 − α. (5.18)
δ zk  2 1+ρ
52 5 Inexact Sequential Quadratic Programming

Proof. We recall bound (5.7) in the form

 
δk αhδk
Θk ≤ 1 − 1 − − α. (5.19)
1 − δk 2(1 − δk )

We now find a bound for αhδk with optimal realizable step size

α = [α k ] = min(1, 1/ (1 + ρ)[hδk ] ).

If α = 1 we obtain [hδk ] ≤ 1/(1 + ρ) and thus

1+σ
αhδk ≤ [hδk ] + σ max(1/(1 + ρ), [hδk ]) ≤ .
1+ρ
If α < 1 we obtain the same bound
hδk [hδk ] + σ max(1/(1 + ρ), [hδk ])
αhδk = <
(1 + ρ)[hδk ] (1 + ρ)[hδk ]
1 + σ max(α, 1) 1 + σ
= = .
1+ρ 1+ρ
Therefore δk can be bounded in ρ and σ according to
ρ δ ρ(1 + σ )
δk = αh ≤ ,
2 k 2(1 + ρ)
which in turn yields for the factor in parentheses in equation (5.19)
ρ(1+σ ) 1+σ
δk αhδk 2(1+ρ) 1+ρ
1− − ≥ 1− −
1 − δk 2(1 − δk ) 1 − ρ(1+σ )
2 1 − ρ(1+σ )
2(1+ρ) 2(1+ρ)
ρ(1 + σ ) + (1 + σ ) 2 + ρ(1 − σ ) − ρ(1 + σ ) − 1 − σ
= 1− =
2(1 + ρ) − ρ(1 + σ ) 2 + ρ(1 − σ )
1 − σ (1 + 2ρ)
= ,
2 + ρ(1 − σ )
which proves the first assertion. For the inexact natural contraction factor we use
the triangle inequality in combination with Lemma 5.20 to obtain
         
k = 
Θ
        
δ zk+1  / δ zk  ≤ (1 − α) δ zk  + δ zk+1 − (1 − α)δ zk  / δ zk 
 
1 + ρ 2 δ 1 (1 + ρ)(1 + σ )
≤ 1−α + α hk ≤ 1 − 1 − α,
2 2 1+ρ
which shows the second assertion.
5.6 Natural Monotonicity for LISA-Newton methods 53

Remark 5.23. With the additional assumption that σ < 1/(1 + 2ρ) we obtain Θk <
1.
Based on Lemma 5.22 we can now formulate an inexact NMT. To this end we
substitute the bound (5.18) by the computable inexact NMT
 
 k+1 
δ z  ρ − ρ
k =
Θ < 1− α, (5.20)
δ zk  1+ρ
by replacing σ with its upper bound σ = 1. In order to be a meaningful test we
additionally require ρ < ρ. The inexact NMT (5.20) becomes computable if we
select a ρ < 1 and further impose for the relative error of the inner LISA that
ρ ρ(1 + σ )
δk ≤ α[hδk ] ≤
2 2(1 + ρ)
which is substituted by the computable condition
ρ 1
δk ≤ =: δ ≤
2(1 + ρ) 4
for the case of α < 1.
If in the course of the computation the inexact NMT is not satisfied, we reduce
the step size on the basis of the a-posteriori Kantorovich estimate (5.16), denoted
by [hδk ]∗ , according to


αknew := max(min(1/ (1 + ρ)[hδk ]∗ , α/2), αmaxred α) .
α=αk
old

Taking the min and max is necessary to safeguard the stepsize adaption against
too cautious (reduction by at least a factor of two) and too aggressive changes
(reduction by at most a factor of αmaxred ≈ 0.1). Especially aggressive reduction
must be safe-guarded because the computation of the Kantorovich estimate [hδk ]
in equation (5.16) is inflicted with a cancellation error which is then amplified by
1/α 2 . Furthermore, the cancellation error gets worse for smaller α because then
δ zk+1 is closer and closer to (1 − α)δ zk as a consequence of Lemma 5.20.
For the initial choice for αk we recede to the a-priori Kantorovich estimate (5.17)
via
αk+1 = min(1, 1/ (1 + ρ)[hδk ] )
The initial step size α0 has to be supplied by the user. As a heuristic one can
choose α0 = 1, 0.01, 0.0001 for mildly nonlinear, nonlinear, and highly nonlinear
problems, respectively.
54 5 Inexact Sequential Quadratic Programming

5.6.2 A Linear Iterative Splitting Approach

The goal of this section is to characterize the convergence and to give error esti-
mates for LISA. Furthermore we address the connection of the convergence rate
of LISA with the asymptotic convergence of the LISA-Newton method. To this
end let J,ˆ M̂ ∈ RN×N and F̂ ∈ RN . We approximate ζ ∈ RN which satisfies
ˆ = −F̂
Jζ
via the iteration
 
ζi+1 = ζi − M̂ Jζ
ˆ i + F̂ = (I − M̂ J)ζ
ˆ i − M̂ F̂. (5.21)
The iteration is formally based on the splitting

Jˆ = M̂ −1 − ΔJ.
We have been using this setting in Section 5.6.1 with Jˆ = J(zk ) and F̂ = F(zk ) or
F̂ = F(zk + αk δ zk ) to approximate ζ = Δzk or ζ = Δzk+1 , respectively. The matrix
M̂ is a preconditioner which can be used in a truncated Neumann series to describe
the approximated inverse M(z). We address this issue later in Lemma 5.27.

5.6.2.1 Affine invariant convergence of LISA

Lemma 5.24. Let A, B ∈ GL(N) yield transformations of F̂, J,ˆ and M̂ which satisfy
F = AF̂, J= AJB,
ˆ  = B−1 M̂A−1 .
M
Then LISA is affine invariant under A and B.
Proof. Assume ζi = B−1 ζi . Then we have
 
ζi+1 = (I − M  ζi − M
 J)  F = I − B−1 M̂A−1 AJB
ˆ B−1 ζi − B−1 M̂A−1 AF̂
  
= B−1 I − M̂ Jˆ ζi − M̂ F̂ = B−1 ζi+1 .
Induction yields the assertion.
Corollary 5.25. A full-step LISA-Newton method is affine invariant under trans-
formations A, B ∈ GL(N) with
 z) = AF̂(B
F( z)
if the matrix function M̂(z) satisfies
 z) = B−1 M̂(B
M( z)A−1 .
5.6 Natural Monotonicity for LISA-Newton methods 55

Proof. Lemmata 5.9 and 5.24.

The Newton-Picard preconditioners in Chapter 6 satisfy this relation at least
partially which leads to scaling invariance of the Newton-Picard LISA-Newton
method (see Chapter 6).
The convergence requirements of LISA are described by the following theorem:

Theorem 5.26. Let

κlin := σr (I − M̂ J).
ˆ

If κlin < 1 then M̂ and Jˆ are invertible and LISA (5.21) converges for all F̂, ζ0 ∈ RN .
Conversely, if LISA converges for all F̂, ζ0 ∈ RN , then κlin < 1. The asymptotic
linear convergence factor is given by κlin .

Proof. See Saad [135, Theorem 4.1].

5.6.2.2 Connection between linear and nonlinear convergence

We now investigate the connection of the preconditioner M̂(z) with the approxi-
mated inverse M(z) of the Local Contraction Theorem 5.5. The results have al-
ready been given by Ortega and Rheinboldt [122, Theorem 10.3.1]. We translate
them into the framework of the Local Contraction Theorem 5.5.

Lemma 5.27. Let ζ0 = 0 and l ≥ 1. Then the l-th iterate of LISA is given by the
truncated Neumann series for M̂ Jˆ according to
 
l−1
ζl = − ∑ (I − M̂J)ˆ i M̂ F̂.
i=0

Proof. Let l ∈ N and assume that the assertion holds for ζl . Then we obtain
  
l−1
ζl+1 = (I − M̂ J)ζ
ˆ l − M̂ F̂ = − ∑ (I − M̂J)ˆ i+1
+ I M̂ F̂
i=0
 
l
=− ∑ (I − M̂J)ˆ
i
M̂ F̂.
i=0

For l = 1 we have ζ1 = −M̂ F̂ and we complete the proof by induction.

The following lemma shows that M, defined by l LISA steps, is almost the
inverse of J.
56 5 Inexact Sequential Quadratic Programming

Lemma 5.28. Let the approximated inverse be defined according to

   
l−1 l−1
M(z) = ∑ (I − M̂(z)J(z)) i
M̂(z) = M̂(z) ∑ (I − J(z)M̂(z))
i
.
i=0 i=0

Then it holds that

M(z)J(z) = I − (I − M̂(z)J(z))l ,
J(z)M(z) = I − (I − J(z)M̂(z))l .

Proof. With the abbreviation A := I − M̂(z)J(z) we obtain the first assertion

 
l−1 l−1 l
M(z)J(z) = ∑ Ai (I − A) = ∑ Ai − ∑ Ai = I − (I − M̂(z)J(z))l .
i=0 i=0 i=1

The second assertion follows with the same argument.

Theorem 5.29. Let z∗ ∈ D satisfy M̂(z∗ )F(z∗ ) = 0. For the LISA-Newton method
with continuous preconditioner M̂(z) and l steps of LISA with starting guess ζ0 =
0, the following two assertions are equivalent:
i) The LISA at z∗ converges for all starting guesses and right hand sides, i.e.,

σr (I − M̂(z∗ )J(z∗ )) ≤ κlin < 1.

ii) The matrices M̂(z∗ ) and J(z∗ ) are invertible and for every ε > 0 there exists
a norm .∗ and a neighborhood U of z∗ such that the κ-condition 5.3 for
M(z) in U based on .∗ is satisfied with

κ ≤ κlin
l
+ε (where κlin < 1).

Proof. i) ⇒ ii): By virtue of Theorem 5.26 it holds that

σr (I − M̂(z∗ )J(z∗ )) ≤ κlin < 1

and that M̂(z∗ ) and J(z∗ ) are invertible. Recall that we have

I − M(z∗ )J(z∗ ) = (I − M̂(z∗ )J(z∗ ))l

by Lemma 5.28. Let ε > 0. Then the Hirsch Theorem delivers a norm .∗ such
that
I − M(z∗ )J(z∗ )∗ ≤ σr (I − M̂(z∗ )J(z∗ ))l + ε/2 ≤ κlin
l
+ ε/2.
5.6 Natural Monotonicity for LISA-Newton methods 57

By continuity of F, J, M̂, .∗ , and the inverse we obtain the existence of a neigh-
borhood U of z∗ such that all z ∈ U satisfy
M(z − M(z)F(z))(I − J(z)M(z))F(z)∗
 
≤ M(z − M(z)F(z))(M −1 (z) − J(z))∗ M(z)F(z)∗
≤ (κlin
l
+ ε) M(z)F(z)∗
because
M(z∗ − M(z∗ )F(z∗ ))(M −1 (z∗ ) − J(z∗ )) = I − M(z∗ )J(z∗ ).
Comparison with the κ-condition 5.3 yields
κ ≤ κlin
l
+ ε.
ii) ⇒ i): Let ẑ ∈ RN be a nonzero vector, small enough such that
z(t) := z∗ + t ẑ ∈ U, t ∈ (0, 1]. (5.22)
Because M(z∗ ) is invertible we obtain F(z∗ ) = 0 and write F(z(t)) in the form
 1
F(z(t)) = F(z∗ + t ẑ) − F(z∗ ) = t J(z∗ + τt ẑ)ẑdτ,
0

which leads to
 1
z (t) − z(t) := −M(z(t))F(z(t)) = −tM(z(t)) J(z(τt))dτ ẑ.
0

From the κ-condition 5.3 we infer the inequality

  1 
   
  −1
t M(z (t)) M(z(t)) − J(z(t)) M(z(t)) J(z(τt))dτ ẑ
0

∗
  1 
 

≤ tκ M(z(t)) J(z(τt))dτ ẑ
0
 .
∗

After division by t we take the limit t → 0 and obtain

κ M(z∗ )J(z∗ )ẑ∗ ≥ M(z∗ )J(z∗ ) [I − M(z∗ )J(z∗ )] ẑ∗ . (5.23)
We now recede to a special choice for ẑ based on the real Jordan decomposition
⎛ ⎞
J1
⎜ ⎟ −1
I − M(z∗ )J(z∗ ) = V ⎝ ..
. ⎠V ,
Jp
58 5 Inexact Sequential Quadratic Programming

with an invertible matrix V ∈ RN×N and real Jordan blocks Ji , corresponding to

eigenvalues λi = ai + ibi , of the form
⎛ ⎞ ⎛ ⎞
ai 1 Ci I2
⎜ . ⎟ ⎜ . ⎟
⎜ ai . . ⎟ ⎜ Ci . . ⎟
Ji = ⎜
⎜
⎟ if bi = 0 and Ji = ⎜
⎟ ⎜
⎟ if bi = 0,
⎟
⎝ .. ..
. 1⎠ ⎝ . I2 ⎠
ai Ci

where  
ai bi
Ci = .
−bi ai
We assume that the eigenvalues are sorted to satisfy |λi | ≥ |λi+1 |. Let V j denote
the j-th column of V . If b1 = 0, we choose α = 0 small enough such that with
ẑ = αV1 assumption (5.22) holds. We then obtain

M(z∗ )J(z∗ ) [I − M(z∗ )J(z∗ )] ẑ = |λ1 | M(z∗ )J(z∗ )ẑ . (5.24)

If b1 = 0, then we choose α, β ∈ R such that ẑ = αV1 + βV2 = 0 and (5.22) holds.

Due to invertibility of M(z∗ ), J(z∗ ), and V , we can define the norm
    
 α   
  = M(z∗ )J(z∗ )(V1 V2 ) α 
 β   β ∗
V

and consider the estimate

M(z∗ )J(z∗ ) [I − M(z∗ )J(z∗ )] ẑ∗

    
 α   α 
= C
 1 β 
 ≥ |λ |
1 
  = |λ1 | M(z∗ )J(z∗ )ẑ .
 ∗ (5.25)
V
β V

Finally we can combine (5.23), (5.24), and (5.25) in order to establish

κ M(z∗ )J(z∗ )ẑ∗ ≥ M(z∗ )J(z∗ ) [I − M(z∗ )J(z∗ )] ẑ∗ ≥ |λ1 | M(z∗ )J(z∗ )ẑ∗ .

Thus we have

κlin
l
+ ε ≥ κ ≥ |λ1 | = σr (I − M(z∗ )J(z∗ )) = σr (I − M̂(z∗ )J(z∗ ))l ,

independently of .∗ . Letting ε → 0 yields assertion i).

Let us halt shortly to discuss the previous results: Far away from the solution the
use of LISA allows for adaptive control of the angle between the search direction
5.6 Natural Monotonicity for LISA-Newton methods 59

and the tangent on the Newton path according to Lemma 5.28. Theorem 5.29
guarantees that although a LISA-Newton method with larger number l of inner
iterations is numerically more expensive per outer iteration than using l = 1, the
numerical effort in the vicinity of the solution is asymptotically fully compensated
by less outer inexact Newton iterations.

5.6.2.3 Convergence estimates for LISA

To simplify the presentation we denote the LISA iteration matrix by

A = I − M̂ J.ˆ

Lemma 5.30. Assume A ≤ κ̂ < 1. Then the following estimates hold:

ζl+1 − ζl  ≤ κ̂ l ζ1 − ζ0  ,
κ̂ l
ζl − ζ∞  ≤ ζ1 − ζ0  .
1 − κ̂
Proof. Let l ≥ 1. The first assertion follows from
 
 
ζl+1 − ζl  = A(ζl − ζl−1 ) ≤ Al  ζ1 − ζ0  ≤ κ̂ l ζ1 − ζ0  .

Thus we obtain
∞ ∞
κ̂ l
ζl − ζ∞  ≤ ∑ ζk − ζk+1  ≤ ∑ κ̂ k ζ1 − ζ0  ≤ ζ1 − ζ0  ,
k=l k=l 1 − κ̂

which proves the second assertion.

5.6.2.4 Estimation of κ̂
In order to make use of Lemma 5.30 we need a computable estimate [κ̂] of κ̂. We
present three approaches which are all based on eigenvalue techniques. They differ
mainly in the assumptions on the iteration matrix A, in the required numerical
effort, and in memory consumption.
For l = 1, 2, . . . we define

δ ζl = ζl − ζl−1 .

We have already observed in the proof of Lemma 5.30 that

δ ζl+1 = Aδ ζl = Al δ ζ1 .
60 5 Inexact Sequential Quadratic Programming

Thus LISA behaves like a Power Method (see, e.g., Golub and van Loan [61]). The
common idea behind all three κ̂ estimators is to obtain a good estimate for σr (A)
by approximation of a few eigenvalues during LISA. Based on Theorem 5.29 we
expect σr (A) to be a good asymptotic estimator for the norm-dependent κ̂.

Lemma 5.31 (Rayleigh κ-estimator). Let A be diagonalizable and the eigenvalues

μi , i = 1, . . . , N be ordered according to

|μ1 | > |μ2 | ≥ · · · ≥ |μN |

with a gap in modulus between the first and second eigenvalue. If furthermore δ ζ1
has a component in the direction of the eigenvector corresponding to μ1 we obtain

δ ζlT δ ζl+1
[κ̂]l := → σr (A) for l → ∞.
δ ζlT δ ζl

Proof. The proof coincides with the convergence proof for the Power Method. For
a discussion of the convergence we refer the reader to Wilkinson [165] and Parlett
and Poole [124]. The quotient

δ ζlT δ ζl+1 δ ζ T Aδ ζl
= lT .
δ ζl δ ζl
T δ ζ l δ ζl

in the assertion is the Rayleigh quotient.

We observe that only the last iterate needs to be saved in order to evaluate the
Rayleigh κ-estimator which can thus be implemented efficiently with low mem-
ory requirements. The possibly slow convergence of δ ζl towards the dominant
eigenvector if |μ1 | is close to |μ2 | does not pose a problem for the Rayleigh κ-
estimator because we are only interested in the eigenvalue, not the corresponding
eigenvector. However, the Rayleigh κ-estimator is not suitable in many practical
applications because the assumption that A is diagonalizable is often violated.
We have developed the following κ-estimator for general matrices:

Lemma 5.32 (Root κ-estimator). Let σr (A) > 0 and let δ ζ1 have a component
in the dominant invariant subspace corresponding to the eigenvalues of A with
largest modulus. Then the quotient of roots

δ ζl+1 1/l
[κ̂]l+1 := , l ≥ 1,
δ ζ1 1/l

yields an asymptotically correct estimate of σr (A) for l → ∞.

5.6 Natural Monotonicity for LISA-Newton methods 61

Proof. Matrix submultiplicativity yields the upper bound

 l   l 1/l  1/l
δ ζl+1 1/l A δ ζ1 1/l A  δ ζ1 1/l  
= ≤ = Al  ,
δ ζ1 1/l δ ζ1 1/l δ ζ1 1/l

which tends to σr (A) for l → ∞.

We construct a lower bound in three steps: First, we write down a Jordan de-
composition

A = XΛX −1 ,

where X ∈ GL(N) and Λ is a block diagonal matrix consisting of m Jordan blocks

J p j (λ j ) of sizes p j , j = 1, . . . , m, corresponding to the eigenvalues λ j . From the
identity

Al = XΛl X −1

we see that the columns of X corresponding to each Jordan block span a cyclic
invariant subspace of Al . There exists a constant c > 0 such that
 
z ≥ c zX := c X −1 z2

because all norms on a finite dimensional space are equivalent. With z := X −1 δ ζ1

we obtain a reduction of the problem to Jordan form

 1/l  1/l
 l   
A δ ζ1  ≥ c1/l Λl z . (5.26)
2

Second, we reduce further to one Jordan block via

 2 m  2  2
 l 
Λ z =
2
∑ Jp j (λ j )z̃ j 2 ≥ Jp1 (λ1 )z̃1 2 , (5.27)
j=1

where z̃ j is the subvector of z corresponding to the Jordan block J p j (λ j ). Without

 
loss of generality we choose the ordering of the Jordan blocks such that λ 1  =
σr (A) and z̃1 = 0 due to the assumption of the lemma.
62 5 Inexact Sequential Quadratic Programming

Third, we investigate one single Jordan block J p1 (λ1 ). To avoid unnecessary

notational clutter we drop the j = 1 indices. Let l ≥ p. Then we obtain
⎛ ⎞ 
 λ l  l λ l−1 · · ·  l λ l−(p−1) 
 1 p−1 
⎜ ⎟ 
  ⎜ .. .. .. ⎟ 
  ⎜ . . . ⎟ 
σr (A)−l J p (λ )l z̃ = |λ |−l ⎜ ⎟ z̃
2 ⎜ ..  l  l−1 ⎟ 
⎝ . 1 λ
⎠ 
 
 λ l 
 2

 
 
⎛ ⎞ 
 ( l
) ( l
) 
 1 1
λ −1 · · · p−1
λ −(p−1) 
⎜ ( p−1
l
) ( l
) ( l
) ⎟ 
⎜ p−1 p−1 ⎟ 
  ⎜ .. .. .. ⎟ 
l ⎜ . . . ⎟ 
=  ⎜ ⎟ z̃
p−1  ⎜
⎜ . .. ( l
) −1
⎟ 
⎟ 
⎜
1
λ ⎟ 
⎝ ( l
p−1 ) ⎠ 
 1 
 ( p−1)
l 
  ! 
 
 =:Ã(l)

  2
≥ Ã(l)z̃2 .

To estimate the quotients of binomials we assume k > j and obtain

l 
j k!(l − k)! k(k − 1) · · · (k − j + 1)
l  = = ,
j!(l − j)! (l − j) · · · (l − k + 1)
k

which tends to zero  for l → ∞. This shows that the (1, p) entry of Ã(l) dominates
for large l. Thus Ã(l)z̃2 converges to
 
σr (A)1−p z̃ p  > 0.
 
(If z̃ p  = 0 we can use the same argument with the last non-vanishing component
of z̃.) Consequently we can find l0 ∈ N such that
   
  1
σr (A)−l J p (λ )l z̃ ≥ σr (A)1−p z̃ p  > 0 for all l ≥ l0 . (5.28)
2 2
We now combine equations (5.26), (5.27), and (5.28) and obtain for l ≥ l0 that
 l     1/l
A δ ζ1 1/l cσr (A)1−p z̃ p 
≥ σr (A) ,
δ ζ1 1/l 2 δ ζ1 
5.6 Natural Monotonicity for LISA-Newton methods 63

which tends to σr (A) for l → ∞.

We believe it is helpful at this point to investigate two prototypical classes of
non-diagonalizable matrices to appreciate the convergence of LISA from a geo-
metrical point of view. We restrict the discussion to the case F̂ = 0 because we can
then exploit the fact that ζl+1 = Aζl as well as δ ζl+1 = Aδ ζl .
Example 3 (Jordan matrices). In the proof of Lemma 5.32 we have already seen
that the cyclic invariant subspaces of A are spanned by the columns of X corre-
sponding to each Jordan block. Thus the convergence in the case where A is one
Jordan block is prototypical.
Let thus λ ∈ [0, 1) and
⎛ ⎞
−λ 1
⎜ .. .. ⎟
⎜ . . ⎟
A = JN (−λ ) := ⎜
⎜
⎟.
⎟
⎝ ..
. 1 ⎠
−λ

We immediately see that σr (A) = λ < 1 and so we obtain convergence of the

iteration for all starting values ζ0 and right hand sides F̂ by virtue of Theorem 5.26.
In particular, we now choose ζ0 = eN , the last column of the N-by-N identity
matrix. If λ = 0 we obtain
"
eN−l for l < N,
ζl =
0 for l ≥ N,

i.e., ζl circulates backwards through all basis vectors e j and then suddenly drops
to zero. Figure 5.2 depicts iterations with varying λ and N. We observe that the
Jordan structure leads to non-monotone transient convergence behavior in the first
iterations. Only in the diagonalizable case of N = 1 is the convergence monotone.
Figure 5.2 also suggests that the Root κ-estimator of Lemma 5.32 can grossly
overestimate σr (A) in a large preasymptotic range of iterations.

Example 4 (Multiple eigenvalues of largest modulus). Consider the N-by-N per-

mutation matrix
⎛ ⎞
1
⎜ .. ⎟
⎜ . ⎟
P=⎜ ⎟.
⎝ 1⎠
1
64 5 Inexact Sequential Quadratic Programming

λ = 0.9 λ = 0.7
103 102

102
100
Error

101

10−2
100

10−1 10−4
0 5 10 15 20 0 5 10 15 20

λ = 0.5 λ = 0.3
2
10 105

100
100
10−2
Error

10−5
−4
10
10−10
10−6

10−8 10−15
0 5 10 15 20 0 5 10 15 20
Iteration Iteration

Figure 5.2: The errors of the iterates of LISA in the Euclidean norm .2 with a Jordan
iteration matrix given by Example 3 exhibit non-monotone transient convergence behavior.
The subfigures depict different values for λ . The iterations are performed with values for
N = 1, . . . , 5, marked by •, , ∗, ◦, ×, respectively.

The eigenvalues of P are given by the complex roots of the characteristic polyno-
mial λ N − 1. Thus they all satisfy |λi | = 1. Let X ∈ GL(N) and κ ∈ [0, 1). Then
the matrix
A := κXPX −1
5.6 Natural Monotonicity for LISA-Newton methods 65

has all the eigenvalues satisfying |κλi | = κ = σr (A). Again, Theorem 5.26 yields
convergence of LISA for all starting values ζ0 . By virtue of PN = I we obtain

A jN = (κ N ) j I,

which results in monotone N-step convergence. The behavior between the first and
N-th step can be non-monotone in an arbitrary norm as displayed in Figure 5.3. If
we take instead the X-norm
 
zX := X −1 z2

we immediately obtain monotone convergence by virtue of

  ⎛ ⎞
 −1 
Aζ X ≤ sup AzX ζ X = ⎝ sup κ PX z2 ⎠ ζ X = κ ζ X .
zX =1 X −1 z2 =1

In practical computations, however, the construction of a Hirsch-type norm like

.X is virtually impossible and thus a κ-estimator should be norm-independent.
This leads us to a third approach for the estimation of κ̂.

Lemma 5.33 (Ritz κ-estimator). Let δ ζ1 have a component in the dominant in-
variant subspace corresponding to the eigenvalues of A with largest modulus. De-
fine
Z(i, j) = (δ ζi , . . . , δ ζ j )
and let R ∈ R p×p be an invertible matrix such that Z(1, p) = QR with orthonormal
Q ∈ RN×p and maximal p ≤ l. Then

[κ̂]l+1 := σr (R−T Z(1, p)T Z(2, p + 1)R−1 )

yields the exact σr (A) after at most N iterations.

Proof. Consider the Ritz values μ j , j = 1, . . . , p of A on the Krylov space

Kl (A, δ ζ1 ) := span(A0 δ ζ1 , . . . , Al δ ζl ).

The Ritz values solve the following variational eigenvalue problem: Find v ∈
Kl (A, δ ζ1 ) such that

wT (Av − μv) = 0, for all w ∈ Kl (A, δ ζ1 ). (5.29)

66 5 Inexact Sequential Quadratic Programming

102

Error 100

10−2

10−4

10−6
0 5 10 15 20
Iterations

Figure 5.3: The errors of the iterates of LISA with a 5-by-5 matrix A = 12 XPX −1 given by
Example 4 exhibit non-monotone cyclic convergence behavior in the Euclidean norm.2
(• marks). The convergence is monotone if measured in the X-norm zX := X −1 z2 (◦
marks). We chose the matrix X to be a random symmetric matrix with condition number
100.

Because Q spans an orthonormal basis of Kl (A, δ ζ1 ) equation (5.29) is equivalent

to the standard eigenvalue problem

QT (AQṽ − μQṽ) = QT AQṽ − μ ṽ = 0.

Recall that Q = Z(1, p)R−1 . Thus we substitute

H := QT AQ = R−T Z(1, p)T AZ(1, p)R−1 = R−T Z(1, p)T Z(2, p + 1)R−1 .

Hence, as soon as the dimension of the Krylov space Kl (A, δ ζ1 ) ⊆ RN becomes

stationary when l grows, we obtain σr (H) = σr (A).
The Ritz κ-estimator of Lemma 5.33 yields the most reliable estimates for the
spectral radius of A. However, the large memory requirement for storing Z(1, l) is
not feasible in practice. Our experience is that a moderate bound on p still provides
useful estimates for κ̂.
Remark 5.34. In the implementation MUSCOP, which we describe in Chapter 11,
we explicitly compute Q and R by a QR decomposition. This extra effort is neg-
5.6 Natural Monotonicity for LISA-Newton methods 67

ligible if the matrix vector products with A dominate the overall effort, which is
certainly the case in MUSCOP especially on finer spatial grids.

Remark 5.35. We further propose that one should build the matrix Q iteratively,
e.g., via an Arnoldi process with upper Hessenberg matrix R (see, e.g., Golub and
van Loan [61]). This raises a couple of further questions which would have to
be addressed and exceed the scope of this thesis: If an orthonormal basis of the
Krylov space is anyway available, is a different solver for the linear systems more
appropriate? GMRES by Saad and Schultz [136], e.g., is explicitly built on an
Arnoldi process but lacks the property of affine invariance in the residual space
and an error criterion in the variable space. Furthermore, a connection between
the nonlinear κ and a descriptive constant for convergence of the linear solver like
in Theorem 5.29 should be investigated.

5.6.2.5 Adaptive κ improvement

Based on the κ-estimators from Section 5.6.2.4 we can adaptively control the qual-
ity of the preconditioner M(z). The procedure is as follows: Let κmax < 1 and an
integer ipre be given. If in the i-th LISA iteration

i > ipre and [κ̂]i > κmax (5.30)

then we need to improve the quality of M(x) to decrease κ. The integer ipre is a
safeguard to discard preasymptotic estimates of κ̂ which have not come close to
the actual spectral radius of the iteration matrix yet. In#our numerical experience
with the applications that we present in Part III, κmax = 1/2 and ipre = 8 produce
reasonable results.
Depending on the type of preconditioner M(z) the improvement can consist of
different strategies: In an adaptive Simplified Newton Method, e.g., we keep M(z)
constant until condition (5.30) is satisfied which triggers a new evaluation of M
at the current iterate zk . In Chapter 6 we describe a two-grid preconditioner M(z)
which can be improved by refinement of the coarse grid if condition (5.30) holds.

5.6.3 GINKO Algorithm

We distill the algorithmic ingredients for the presented Global inexact Newton
method with κ and ω monitoring (GINKO) into concise form in Algorithm 1.
68 5 Inexact Sequential Quadratic Programming

5.7 Inequality constrained optimization problems

We have developed an approach how inequality constrained optimization problems
can be treated on the basis of an NMT LISA-Newton method (see Section 5.6). We
especially focus on the direct use of the GINKO Algorithm 1 for the solution of
NLP problem (4.1) which we have formulated in Chapter 4 as

minimize
n
f (x) s. t. gi (x) = 0, i ∈ E , gi (x) ≥ 0, i ∈ I ,
x∈R

with E ∪˙ I = {1, . . . , m} =: m.
The quintessence of our approach is to formulate the stationarity and primal
feasibility condition of the KKT conditions (4.2) in a function F and ensure that
dual feasibility and complementarity hold in the solution via suitable choice of M.
We treat the case with exact derivatives in Section 5.7.1 and the extension to
inexact derivatives in a LISA-Newton method in Section 5.7.2.

5.7.1 SQP with exact derivatives

SQP is a collective term for certain methods that find critical points of NLP prob-
lems. A critical point is a pair z = (x, y) ∈ Rn+m of primal and dual variables which
satisfies the KKT conditions (see Theorem 4.9). SQP methods approximate crit-
ical points via sequential solution of QPs which stem from some (approximated)
linearization of the NLP around the current iterate. Various variants exist which
differ mostly in the way how QP subproblems are formulated and which global-
ization strategy is used. For an introduction see, e.g., Nocedal and Wright [121].
We now present our novel SQP approach. Let N = n + m and the function
F : RN → RN be defined according to
   
F1 (z) ∇x L (x, y)
F(z) = := (5.31)
F2 (z) g(x)

with Jacobian
   
J (z) −J2 (z)T ∇2xx L (z) −∇g(x)
J(z) = 1 = .
J2 (z) 0 ∇g(x)T 0

We observe that J is in general singular because g is not restricted to only active

constraints. For instance if g contains upper and lower bounds on a variable then
the corresponding two columns in ∇g(x) are linearly dependent for all z ∈ RN . We
shall see later that J is invertible on a suitably defined subspace (see Remark 5.38).
5.7 Inequality constrained optimization problems 69

Now we generalize the use of an approximated inverse matrix M(z) in the step
computation to a nonlinear function J ⊕ : RN+N → RN to compute
Δz = J ⊕ (z, −F̂) instead of Δz = −M(z)F̂,
where we have dropped the iteration index k for clarity. We define J ⊕ implicitly
in two steps. The fist step consists of computation of the primal-dual solution
 x, y) ∈ Rn+m of the QP
z = (
1 T  T
minimize x + F̂1 − J1 (z)x + J2 (z)T y x
x J1 (z) (5.32a)
x∈R n 2
  
s. t. J2 (z)
x + F̂2 − J2 (z)x i = 0, i∈E, (5.32b)
  
J2 (z)
x + F̂2 − J2 (z)x i ≥ 0, i∈I, (5.32c)
which is not formulated in the space of increments Δz = (Δx, Δy) ∈ Rn+m but rather
in the space of variables 
z = z + Δz. In the second step we reverse this transforma-
tion and obtain Δz = z − z.
Lemma 5.36. Assume that QP (5.32) at z ∈ RN has a unique solution. If ẑ =
(x̂, ŷ) ∈ RN satisfies ŷi ≥ −yi for i ∈ I then
J ⊕ (z, J(z)ẑ) = ẑ.
Proof. Let ẑ = (x̂, ŷ) ∈ RN be given and define
 
−J1 (z)x̂ + J2 (z)ŷ
F̂ = −J(z)ẑ = .
−J2 (z)x̂
To prove the lemma we show that J ⊕ (z, −F̂) = Δz = ẑ. With aforementioned
choice of F̂ QP (5.32) becomes
1 T  T
minimize x − J1 (z)(x̂ + x) − J2 (z)T (ŷ + y) x
x J1 (z)
x∈Rn 2
s. t. J2 (z)i (
x − x̂ − x) = 0, i∈E,
J2 (z)i (
x − x̂ − x) ≥ 0, i∈I.
Its stationarity condition reads
J1 (z) (
x − x̂ − x) − J2 (z) (
y − ŷ − y) = 0.
We thus observe that  z = ẑ + z is stationary and primal feasible. Dual feasibility
holds due to yi = ŷi + yi ≥ 0 for i ∈ I by assumption. Complementarity is satisfied
by virtue of x− x̂ − x = 0. Thus Δz =  z − z = ẑ.
Lemma 5.36 reveals that under the stated assumptions J ⊕ operates linear on the
second argument like a generalized inverse of J.
70 5 Inexact Sequential Quadratic Programming

Theorem 5.37. Assume that αk−1 = 1 and that the solutions of QP (5.32) at
(zk−1 , −F(zk−1 )) and (zk , −F(zk )) share the same active set A and satisfy the
SOSC and the SCC. Then there exists a matrix M k and a neighborhood U of F(zk )
such that
−M k F̂ = J ⊕ (zk , −F̂) for all F̂ ∈ U.
Proof. We first notice that the solution 
zk−1 of QP (5.32) at (zk−1 , −F(zk−1 )) sat-
isfies
zk = zk−1 + Δzk−1 = zk−1 .
Thus we have for all inactive inequality constraints that

yki = yk−1
i =0 and yk+1
i = yki = 0 for i ∈ m \ A

by virtue of complementarity. It follows that Δyki = 0, i ∈ m \ A and thus we can

set all rows n + i, i ∈ m \ A of M k to zero. Due to invariance of the active set A
we obtain for the remaining variables the linear system
   k+1   
J1 (zk ) −J2 (zk )TA x F1 (zk )
+ = 0, (5.33)
J2 (zk )A 0 yk+1
A F2 (zk )A

whose solution depends linearly on F(zk ) and defines the submatrix of M k corre-
sponding to primal and active dual variables. We further notice that zk+1 does not
depend on F2 (zk )m\A . Thus we can set all remaining columns n + i, i ∈ m \ A of
M k to zero. This fully defines the matrix M k .
Because the SOSC and the SCC hold, the active set A is stable under perturba-
tions (see, e.g., Robinson [132]) which yields the existence of a neighborhood U
of F(zk ) such that

−M k F̂ = J ⊕ (zk , −F̂) for all F̂ ∈ U.

This completes the proof.

The proof of Theorem 5.37 explicitly constructs a matrix M(zk ) as the lineariza-
tion of J ⊕ (zk , .) around −F(zk ) which exists under the stated assumptions. Thus
we can invoke the Local Contraction Theorem 5.5 if the solution z∗ satisfies the
SOSC and the SCC.
Remark 5.38. In the case of varying active sets between two consecutive QPs the
action of J ⊕ (zk , −F(zk )) can be interpreted as an affine linear function consisting
of an offset for Δz plus a linear term −M k F(zk ), where M can be constructed like in
Theorem 5.37 with a small enough step size αk > 0 such that F(zk+1 ) ∈ U. From
a geometrical point of view the overall iteration takes place on nonlinear segments
5.7 Inequality constrained optimization problems 71

given by the QP active sets with jumps between these segments. The assumption
that the reduced Jacobian given in equation (5.33) is invertible on each segment is
now as unrestrictive as the assumption of invertibility of J(zk ) for the root finding
problem F(z) = 0.
Remark 5.39. Algorithmically, the evaluation of M k is performed in the follow-
ing way: If M k is evaluated for the first time, a full QP (5.32) is solved. For all
further evaluations the active (or working) set is kept fixed and a purely equality
constrained QP is solved.
Remark 5.40. We are not aware of results how the jumps due to J ⊕ can be analyzed
within the non-local theory developed in Section 5.6 for globalization based on an
NMT. We have not yet attempted an approach to fill this gap yet, either. However,
the numerical results that we present in Part III are encouraging to undertake such
a probably difficult endeavor.
The following theorem ensures that limit points of the SQP iteration with J ⊕ are
indeed KKT points or even local solutions if SOSC holds on the QP level.

Theorem 5.41. If the SQP method with J ⊕ converges to z∗ then z∗ is a KKT point
of NLP (4.1). Furthermore, the conditions SOSC and SCC transfer from QP (5.32)
at z∗ to NLP (4.1) (at z∗ ).

Proof. If the SQP method converges it must hold that

0 = −M(z∗ )F(z∗ ) = J ⊕ (z∗ , F(z∗ )),

z = z∗ is a solution of
i.e., 

1 T  T
minimize x J1 (z∗ )
x + F1 (z∗ ) − J1 (z∗ )x∗ + J2 (z∗ )T y∗ x
x∈Rn 2
s. t. (J2 (z∗ )
x + F2 (z∗ ) − J2 (z∗ )x∗ )i = 0, i∈E,
(J2 (z∗ )
x + F2 (z∗ ) − J2 (z∗ )x∗ )i ≥ 0, i∈I.

We immediately observe primal feasibility for F2 (z∗ ) = g(z∗ ). From QP stationar-

ity we obtain NLP stationarity by virtue of

0 = J1 (z∗ )
x + F1 (z∗ ) − J1 (z∗ )x∗ + J2 (z∗ )T y∗ − J2 (z∗ )T y = F1 (z∗ ) = ∇x L (z∗ ).

Dual feasibility and complementarity for the NLP as well as SOSC and SCC follow
directly from the QP.
72 5 Inexact Sequential Quadratic Programming

5.7.2 Inexact SQP

The goal of this section is to present how the application of an approximation of J ⊕
within a LISA-Newton method (see Section 5.6) can be evaluated. Let us assume
that we have an approximation of the Jacobian matrix (e.g., via a Newton-Picard
approximation described in Chapter 6) given by
 2   k 
∇xx L (zk ) −∇g(xk ) B −(Ck )T
J(zk ) = ≈ =: Jˆk .
∇g(xk )T 0 Ck 0

We perform the construction of a preconditioner M̂(zk ) for LISA based on Jˆk now
analogously to the construction of J ⊕ (zk , .) from J(zk ). The key point is that the
transformation now requires the sum of the current Newton and the current LISA
iterate zk + δ zkl . Dropping the index k we solve the QP

1 T  T
minimize x Bx + F̂1 −C(x + δ xl ) +CT (y + δ yl ) x (5.34a)
x∈Rn 2
  
s. t. C x + F̂2 −C(x + δ xl ) i = 0, i∈E, (5.34b)
  
C x + F̂2 −C(x + δ xl ) i ≥ 0, i∈I, (5.34c)

and reverse the transformation afterwards via Δzk = 

zk − zk − δ zkl .
k
Remark 5.39 about the evaluation of M is also valid in the context of inexact
SQP for M̂ k .
5.7 Inequality constrained optimization problems 73

Algorithm 1: Global inexact Newton method with κ and ω monitoring (GINKO)

evaluate F0 = F(z0 ), set δ z00 = 0, k = 0, l = 0
k if k ≥ kmax then Error: Maximum outer Newton iterations reached
l if l ≥ lmax then Error: Maximum iterations for κ improvement reached
set j = 0, κ = 0, δ zk = not found
j if δ zk = not found then
if αk < αmin then Error: Minimum step size reached
set zk+1 = zk + αk δ zk , δ zk0 = (1 − αk )δ zk
evaluate Fk+1 = F(zk+1 )
set i = 0
i if i ≥ imax then Error: Maximum inner iterations reached
if δ zk = not found then compute residual rik = −Fk − J(zk )δ zki
else compute residual rik = −Fk+1 − rk − J(zk )δ zki
refine increment iterate δ zki+1 = δ zki + M̂(zk )rik
if i < 1 then set i = i + 1 and goto i
estimate contraction [κ̂] ≈ κ̂
if i > ipre and [κ̂] > κmax then ameliorate M̂(zk ), set l = l + 1, and goto l
if δ zk = not found then     
estimate error δki+1 = [κ̂] δ zki+1 − δ zki  / (1 − [κ̂]) δ zki+1 
if [(α < 1) ∧ (δki+1 > ρ/(2(1 + ρ)))] ∨ [(α = 1) ∧ (δki+1 > (ρ/2)[hδk ])] then
set i = i + 1 and goto i
set δ zk = δ zki+1 and save i, δ zki , δ zki+1
 
if δ zk  < TOL then terminate with solution zk+1 = zk + δ zk
compute residual rk = −Fk − J(zk )δ zk
if k > 0 then    
adapt a-priori Kantorovich estimate [hδk ] = (δ zk  / δ zk−1 )[hδk−1 ]∗
adapt step size αk = max(min(1, 1/((1 + ρ)[hδk ])), αmaxred αk−1 )
   
set δ ∗ = δ zk − δ zk  / δ zk  , δ zk = δ zk , j = 0 and goto j
k i+1 i i+1 0
else
recheck accuracy of δ zk : δk = [κ̂]δk∗ /(1 − [κ̂])
if [(α < 1) ∧ (δk > ρ/(2(1 + ρ)))] ∨ [(α = 1) ∧ (δk > (ρ/2)[hδk ])] then
restore i, δ zki , δ zki+1 , set δ zk = not found, i = i + 1, and goto i
    
compute ρ̄i+1 = [κ̂] δ zk − δ zk  / (1 − [κ̂]) δ zk − (1 − αk )δ zk 
i+1 i i+1
if ρ̄i+1 > ρ̄max then set i = i + 1 and gotoi 
 k+1 
 
2  k
a-posteriori estimate [hδk ]∗ = 2(1 − ρ̄i+1 ) δ zk+1
i+1 − δ z0  /(αk δ z )
 k+1   k 
compute monitor Θk = δ z  / δ z 
if Θk ≥ 1 − (ρ − ρ)/(1 + ρ) then
adapt step αk = max(min(1, 1/((1 + ρ)[hδk ]∗ )), αmaxred αk−1 ) and goto j
set δ zk+1
0 = δ zki+1 , k = k + 1, l = 0, and goto k
6 Newton-Picard preconditioners
For completeness we give the following excerpt from the preprint Potschka et al.
[131] here with adaptions in the variable names to fit the presentation in this thesis.
We present preconditioners for the iterative solution of symmetric indefinite lin-
ear systems     
Jˆ1 Jˆ2T x F̂
Jz =
ˆ = − 1 =: −F̂,
Jˆ2 0 y F̂2
with Jˆ ∈ R(n+m)×(n+m) , z, F̂ ∈ Rn+m derived from equation (5.33) within the frame-
work of an SQP method (see Chapter 5). Note that we have swapped the sign of y
to achieve symmtery of J.ˆ It is well known (see, e.g., Nocedal and Wright [121])
that Jˆ is invertible if Jˆ2 has full rank and Jˆ1 is positive definite on the nullspace of
Jˆ1 . For weaker sufficient conditions for invertibility of Jˆ and a survey of solution
techniques we refer the reader to Benzi et al. [16].
We base our investigations on the following linear-quadratic model problem:
Let Ω ⊂ Rd be a bounded open domain with Lipschitz boundary ∂ Ω and let Σ :=
(0, 1) × ∂ Ω. We seek controls q ∈ L2 (Σ) and corresponding states u ∈ W (0, 1)
which solve the time-periodic PDE OCP
 
1 γ
minimize (u(1; .) − û)2 + q2 (6.1a)
q∈L2 (Σ),u∈W (0,1) 2 Ω 2 Σ

s. t. ∂t u = DΔu in (0, 1) × Ω, (6.1b)

∂ν u + αu = β q in (0, 1) × ∂ Ω, (6.1c)
u(0; .) = u(1; .) in Ω, (6.1d)
with û ∈ L2 (Ω), α, β ∈ L∞ (∂ Ω) non-negative a.e., αL∞ (∂ Ω) > 0, and D, γ > 0.
This problem is an extension of the parabolic optimal control problem presented,
e.g., in the textbook of Tröltzsch [152].
Our focus here lies on splitting approaches
Jˆ = J˜− ΔJ
with J,˜ ΔJ ∈ R(n1 +n2 )×(n1 +n2 ) and J˜ invertible. We employ these splittings in a
LISA (see Chapter 5) which has the form
zk+1 = zk − J˜−1 (Jz
ˆ k + F̂) = J˜−1 ΔJzk − J˜−1 F̂. (6.2)

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_6,
© Springer Fachmedien Wiesbaden 2014
76 6 Newton-Picard preconditioners

As a guiding principle, the iterations should not be forced to lie on the subset of
feasible (possibly non-optimal) points, which satisfy Jˆ2 xk = −F̂2 for all k, i.e., the
PDE constraints are allowed to be violated in iterates away from the optimal solu-
tion. Instead, feasibility and optimality are supposed to hold only at the solution.
The presence or absence of this property defines the terms sequential/segregated
and simultaneous/all-at-once/coupled method, whereby a method with only fea-
sible iterates is called sequential or segregated. The preconditioners we present
work on formulations of the problem which lead to simultaneous iterations. From
a computational point of view, simultaneous methods are more attractive because
it is not necessary to find an exact solution of Jˆ2 xk = −F̂2 in every iteration.
This chapter is organized as follows: In Section 6.1 we give a short review
of the Newton-Picard related literature. We discuss the discretization of prob-
lem (6.1) afterwards in Section 6.2. In Section 6.3 we present the Newton-Picard
preconditioners in the framework of LISA (see Chapter 5). For the discretized
problem we discuss the cases of classical Newton-Picard projective approximation
and of a coarse-grid approach for the constraint Jacobians. The importance of the
choice of the scalar product for the projection is highlighted. We establish a mesh-
independent convergence result for LISA based on classical Newton-Picard split-
ting. In this section we also outline the fast solution of the subproblems, present
pseudocode, and analyze the computational complexity. Moreover we discuss ex-
tensions to nonlinear problems and the Multiple Shooting case in Section 6.4.
In Chapter 12 of this thesis we present numerical results for different sets of
problem and discretization parameters for the Newton-Picard preconditioners. In
addition, we compare the indefinite Newton-Picard preconditioners with a sym-
metric positive definite Schur complement preconditioner in a Krylov method set-
ting.

6.1 The Newton-Picard method for finding periodic

steady states
In the context of bifurcation analysis of large nonlinear systems Jarausch and
Mackens [90] have developed the so-called Condensed Newton with Supported
Picard approach to solve fixed point equations which have a few unstable or slowly
converging modes. Their presentation is restricted to systems with symmetric
Jacobian. Shroff and Keller [147] extended the approach to the unsymmetric
case with the Recursive Projection Method by using more sophisticated numer-
ical methods for the identification of the slow eigenspace. There are two articles in
volume 19(4) of the SIAM Journal on Scientific Computing which are both based
6.2 Discretization of the model problem 77

on [90, 147]: Lust et al. [110] successfully applied the Newton-Picard method
for computation and bifurcation analysis of time-periodic solutions of PDEs and
Burrage et al. [32] develop the notion of deflation preconditioners. To our knowl-
edge the first paper on deflation techniques is by Nicolaides [119] who explicitly
introduces deflation as a modification to the conjugate gradient method and not
as a preconditioner in order to improve convergence. Fast two-grid and multigrid
approaches for the determination of time-periodic solutions for parabolic PDEs on
the basis of a shooting approach are due to Hackbusch [77].

6.2 Discretization of the model problem

A full space-time discretization of problem (6.1) would lead to prohibitively large
memory requirements for d = 3. Thus, we employ a shooting approach which
reduces the degrees of freedom for the state variables to only the initial value. Let
us recapitulate the discretization steps outlined in Chapter 3 and apply them to the
model problem (6.1). We discretize the controls in space with nq form functions
ψ̃l whose amplitude can be controlled in time, i.e.,
q(t, x) = ∑l=1
n
q
ql (t)ψ̃l (x), ql ∈ L2 (0, 1), ψ̃l ∈ L2 (∂ Ω).
In weak variational form a solution u ∈ W (0, 1) of PDE (6.1b) satisfies for all
ϕ ∈ H 1 (Ω) and almost all t ∈ [0, 1] the equation
  
ut (t)ϕ = −D ∇u(t)T ∇ϕ + D ∂ν u(t)ϕ (6.3a)
Ω
Ω ∂ Ω 
= −D ∇u(t)T ∇ϕ − D αu(t)ϕ + D β q(t)ϕ. (6.3b)
Ω ∂Ω ∂Ω
We continue with discretizing the state u in space using a Galerkin approach. Let
ϕi ∈ H 1 (Ω), i = 1, . . . , nu , denote linearly independent functions, e.g., FEM hat
functions on a mesh with nu vertices, and define the matrices S, Q, M ∈ Rnu ×nu ,U ∈
Rnu ×nq m and the vector û ∈ Rnu according to
  
Si j = D ∇ϕiT ∇ϕ j , Qi j = D αϕi ϕ j , Uil = D β ϕi ψ̃l ,
 Ω  ∂Ω ∂Ω

Mi j = ϕi ϕ j , ûi = ûϕi .
Ω Ω
It is well known that the mass matrix M is symmetric positive definite. We can
now discretize equation (6.3) with MOL: The matrix of the discretized spatial dif-
ferential operator is L = −S − Q which leads to the Ordinary Differential Equation
(ODE)
M u̇(t) = Lu(t) +U(q1 (t) · · · qnq (t))T , (6.4)
78 6 Newton-Picard preconditioners

where u(t) = ∑ni=1u

ui (t)ϕi . Then, we discretize each ql (t) using piecewise constant
functions on m intervals. We uniquely decompose i = it m+iq with it = (i−1)/m
and iq = i − it m. Thus, 0 ≤ it < m, 1 ≤ iq ≤ nq and we can define

ψi (t, x) = χ[it /m,(it +1)/m] (t)ψ̃iq (x) ∈ L2 (Σ)

as a basis for the discrete control space. Here χ denotes the characteristic function
of the subscript interval. We can then define the symmetric positive definite control
mass matrix N ∈ Rnq m×nq m according to

Ni j = ψi ψ j .
Σ

Moreover we denote the discretized controls by q ∈ Rnq m .

It is well known that the end state u(1) depends linearly on u(0) and q due
to linearity of ODE (6.4). Thus, there exist unique matrices Gu ∈ Rnu ×nu and
Gq ∈ Rnu ×nq m such that
u(1) = Gu u(0) + Gq q.
Now we construct formulas for Gu and Gq . We first consider solutions of ODE
(6.4) for initial value u(0) = u0 and controls q̃ ∈ Rnq which are constant in time.
We can easily verify that the solution is given by the expression
 
u(t) = exp(tM −1 L)u0 + exp(tM −1 L) − Inu L−1U q̃, (6.5)

where Inu denotes the nu -by-nu identity matrix. If we consider the special case
q̃ = 0 we immediately observe that matrix Gu is given by the matrix exponential

Gu = exp(M −1 L). (6.6)

Because ODE (6.4) is autonomous the matrix Gq can be composed piece by piece
on the control time grid based on the matrices ∂ Gu := exp((1/m)M −1 L) and
∂ Gq := (∂ Gu − Inu )L−1U for a single interval. We obtain
 
Gq = ∂ Gm−1
u ∂ Gq · · · ∂ G1u ∂ Gq ∂ G0u ∂ Gq . (6.7)

We now investigate spectral properties of Gu . We start by showing that the

unsymmetric matrix M −1 L has a basis of M-orthonormal real eigenvectors and
only real eigenvalues.
Lemma 6.1. There exists an invertible matrix Z ∈ Rnu ×nu and a diagonal matrix
Ẽ ∈ Rnu ×nu such that

Z T MZ = Inu and M −1 LZ = Z Ẽ.

6.2 Discretization of the model problem 79

Proof. The matrix L = −S − Q is symmetric as a sum of symmetric Galerkin ma-

trices. We decompose
M = RTM RM
with invertible RM ∈ Rnu ×nu , e.g., by Cholesky decomposition, and use matrix RM
for the equivalence transformation

RM (M −1 L)R−1 −T −1
M = RM LRM

of M −1 L to a symmetric matrix. Thus, there exists an invertible matrix Z̃ ∈ Rnu ×nu

of eigenvectors of R−T −1
M LRM satisfying Z̃ Z̃ = Inu and a diagonal real matrix of
T

eigenvalues Ẽ ∈ R n u ×n u such that

R−T −1
M LRM Z̃ = Z̃ Ẽ (or, equivalently, R−1 −T −1 −1
M RM LRM Z̃ = RM Z̃ Ẽ).

We define Z := R−1
M Z̃ and immediately obtain the assertions.
Now we prove a negative upper bound on the eigenvalues of M −1 L.

Lemma 6.2. There exists a grid-independent scalar μ̃¯ < 0 such that all eigenval-
ues μ̃ of M −1 L satisfy μ̃ ≤ μ̃¯ .

Proof. Let (v, μ̃) ∈ Rnu × R, v = 0, be an eigenpair of M −1 L

M −1 Lv = μ̃v,

and define v = ∑ni=1

u
vi ϕi ∈ H 1 (Ω). We now follow a step in a proof of Tröltzsch
[152, Satz 2.6]: By the assumption of αL∞ (∂ Ω) > 0 there exists a measurable
subset Γ ⊂ ∂ Ω with positive measure and a scalar δ > 0 with α ≥ δ a.e. on Γ. We
obtain

μ̃ v2L2 (Ω) = μ̃vT Mv = vT Lv

     
= −D ∇vT ∇v + αv 2
≤ −D ∇v22 + δ v 2
.
Ω ∂Ω Ω Γ

Then we apply the generalized Friedrichs inequality [152, Lemma 2.5] which
yields a Γ-dependent constant c(Γ) > 0 such that
  
2 2 D min(1, δ )
μ̃ vL2 (Ω) ≤ −D ∇v2 + δ v ≤2
− v2H 1 (Ω) .
Ω Γ c(Γ)

With − v2H 1 (Ω) ≤ − v2L2 (Ω) we finally obtain the assertion for the choice μ̃¯ :=
−D min(1, δ )/c(Γ) < 0.
80 6 Newton-Picard preconditioners

Lemma 6.3. Let μ ∈ C be an eigenvalue of Gu . Then μ is real and there exists a

grid-independent scalar μ̄ < 1 such that 0 < μ ≤ μ̄.

Proof. The matrix Gu has the same eigenvectors as the matrix M −1 L. Thus, the
assertion is a direct consequence of equation (6.6) and Lemma 6.2 with μ̄ =
exp(μ̃¯ ) ∈ (0, 1).
We now formulate the finite dimensional linear-quadratic optimization problem

1
minimize uT0 Mu0 − ûT u0 + γqT Nq (6.8a)
u0 ∈Rnu ,q∈Rnq m 2

s. t. M(Gu − Inu )u0 + MGq q = 0. (6.8b)

Lemma 6.4. Problem (6.8) has a unique solution.

Proof. Due to convexity of problem (6.8), necessary optimality conditions are also
sufficient, i.e., if there exists a multiplier vector λ ∈ Rnu such that for u0 ∈ Rnu , q ∈
Rnq m it holds that
⎛ ⎞⎛ ⎞ ⎛ ⎞
M 0 (GTu − Inu )M u0 û
⎜ ⎟⎝ ⎠ ⎝ ⎠
⎝ 0 γN GTq M ⎠ q = 0 (6.9)
M(Gu − Inu ) MGq 0 λ 0

then (u0 , q, λ ) is a primal-dual optimal solution and, conversely, all optimal solu-
tions must satisfy condition (6.9). The constraint Jacobian
 
M Gu − Inu Gq

has full rank due to Gu − Inu being invertible by virtue of Lemma 6.3. The Hes-
sian blocks M and γN are positive definite. Thus, the symmetric indefinite linear
system (6.9) is non-singular and has a unique solution.

6.3 Newton-Picard for optimal control problems

In this section we investigate how the Newton-Picard method for the forward prob-
lem (i.e., solving for a periodic state for given controls) can be exploited in a si-
multaneous optimization approach.
6.3 Newton-Picard for optimal control problems 81

6.3.1 General considerations

For large values of nu it is prohibitively expensive to explicitly form the matrix in
equation (6.9) because the matrix Gu is a large, dense nu -by-nu matrix. Thus, we
cannot rely on direct linear algebra for the solution of equation (6.9). However,
we observe that matrix-vector products are relatively economical to evaluate: The
cost of an evaluation of Gu v is the cost of a numerical integration of ODE (6.4)
with initial value v and controls q = 0. The evaluation of GTu v can be computed
using the identities

MGu = M exp(M −1 L) = exp(LM −1 )M = GTu M, GTu v = MGu M −1 v. (6.10)

Matrix vector products with Gq and GTq can then be evaluated based on equa-
tion (6.7).
The main difficulty here are the large and dense Gu blocks and thus approaches
based on the paper of Bramble and Pasciak [30] and also constraint preconditioners
(e.g., Gould et al. [67]), which do not approximate the blocks containing Gu but
only the M and γN blocks, do not attack the main difficulty of the problem and
will thus be not considered further in this thesis.

6.3.2 Simultaneous Newton-Picard iteration

LISA for the linear system (6.9) yields a simultaneous optimization method be-
cause the iterations will in general not satisfy the periodicity constraint before
convergence. The type of preconditioners we study here is of the following form:
Let G̃u denote an approximation of Gu and regard the exact and approximated
matrices
⎛ ⎞
M 0 (GTu − Inu )M
⎜ ⎟
Jˆ := ⎝ 0 γN GTq M ⎠,
M(Gu − Inu ) MGq 0
⎛ ⎞
M 0 (G̃Tu − Inu )M
⎜ ⎟
J˜ := ⎝ 0 γN GTq M ⎠.
M(G̃u − Inu ) MGq 0

We investigate two choices for G̃u : The first is based on the classical Newton-
Picard projective approximation [110] for the forward problem, the second is based
on a two-grid idea.
82 6 Newton-Picard preconditioners

6.3.2.1 Classical Newton-Picard projective approximation

The principle of the Newton-Picard approximation is based on observations about
the spectrum of the monodromy matrix Gu (see Figure 12.1 in Chapter 12 on page
164). The eigenvalues μi cluster around zero and there are only few eigenvalues
that are close to the unit circle. The cluster is a direct consequence of the dissi-
pativity of the underlying heat equation, i.e., high-frequency components in space
get damped out rapidly. Thus, the zero matrix is a good approximation of Gu in
directions of eigenvectors corresponding to small eigenvalues. The rationale be-
hind a Newton-Picard approximation consists of approximating Gu exactly on the
low-dimensional space of eigenvectors corresponding to large eigenvalues. To this
end, let the columns of the orthonormal matrix V ∈ Rnu ×p be the p eigenvectors of
Gu with largest eigenvalues μi such that
GuV = V E, E ∈ R p×p diagonal.
Now, we approximate the matrix Gu with
G̃u = Gu Π,
where Π is a projector onto the dominant subspace of Gu . Lust et al. [110] pro-
posed to use
Π = VV T , (6.11)
which is an orthogonal projector in the Euclidean sense. This works well for the
solution of the pure forward problem but in a simultaneous optimization approach,
this choice may lead to undesirable loss of contraction, as shown in Chapter 12.
We propose to use a projector that instead takes the scalar product of the infi-
nite dimensional space into account. The projector maps a vector w ∈ Rnu to the
closest point V v, v ∈ R p , of the dominant subspace in an L2 sense, by solving the
minimization problem
1 1 1
minimize w − v2L2 (Ω) = vTV T MV v − vTV T Mw + wT Mw,
v∈R p 2 2 2
where w = ∑ni=1
u
wi ϕi and v = ∑ni=1
u
(V v)i ϕi . The projector is therefore given by
Π = V M p−1V T M, where M p = V T MV ∈ R p×p . (6.12)
Thus, we approximate Gu with
G̃u = V EM p−1V T M.

To compute the inverse of G̃u − Inu we have the following lemma which we invoke
with P = V and R = M p−1V T M:
6.3 Newton-Picard for optimal control problems 83

Lemma 6.5. Let G̃u ∈ Rnu ×nu , P ∈ Rnu ×p , R ∈ R p×nu , and E ∈ R p×p satisfy

G̃u = PER and RP = I p .

If E − I p is invertible then the inverse of G̃u − Inu is given by

(G̃u − Inu )−1 = PXR − Inu , where X = (E − I p )−1 + I p .

Proof. Based on the Sherman-Morrison-Woodbury formula (see, e.g., Nocedal

and Wright [121]) we obtain

(G̃u − Inu )−1 = (−Inu + PER)−1 = −Inu − P(I p − ERP)−1 ER

= P(E − I p )−1 ER − Inu .

The result follows from the identity (E −I p )−1 (I p +E −I p ) = (E −I p )−1 +I p = X.

Computation of the inverse thus only needs the inversion of the small p-by-p
matrices E − I p and M p . For inversion of G̃Tu − Inu we obtain similar to equa-
tion (6.10)

M G̃u M −1 = MGuV M p−1V T = GTu MV M p−1V T = (Gu Π)T = G̃Tu

and consequently
 −1
(G̃Tu − Inu )−1 = M(G̃u − Inu )M −1 = M(G̃u − Inu )−1 M −1 . (6.13)

A dominant subspace basis V for the p-dimensional dominant eigenspace of

M −1 L and thus Gu can, e.g., be computed via an Implicitly Restarted Arnoldi
Method (IRAM), see Lehoucq and Sorensen [103], for the (generalized) eigen-
value problem
M −1 LV −V Ẽ = 0 ⇔ LV − MV Ẽ = 0.
On the basis of equation (6.6) we obtain E := exp(Ẽ).

6.3.2.2 Two-grid Newton-Picard

This variant is based on the observation that for the heat equation the slowly de-
caying modes are the low-frequency modes and the fast decaying modes are the
high-frequency modes. Low-frequency modes can be approximated well on coarse
grids. Thus we propose a method with two grids in which G̃u is calculated only
84 6 Newton-Picard preconditioners

on a coarse grid (cf. Hackbusch [77]), while the remaining computations are per-
formed on the fine grid. Let P and R denote the prolongation and restriction matri-
ces between the two grids and let superscripts c and f denote coarse and fine grid,
respectively. Then, Gfu is approximated by

G̃fu = PER, with E := Gcu ,

i.e., we first project from the fine grid to the coarse grid, evaluate the exact Gcu on
the coarse grid, and prolongate the result back to the fine grid. Note that in contrast
to classical Newton-Picard, E is now not a diagonal matrix.
We use conforming grids, i.e., the Finite Element basis on the coarse grid can
be represented exactly in the basis on the fine grid. Thus, the prolongation P can
f c
be obtained by interpolation. Let uf ∈ Rnu , uc ∈ Rnu and define

nf nc
uf = ∑i=1
u
ufi ϕif ∈ H 1 (Ω), uc = ∑i=1
u
uci ϕic ∈ H 1 (Ω).

We define the restriction R in an L2 sense, such that given uf on the fine grid we
look for the projector R : uf → uc such that

(ϕic , uc )L2 (Ω) = ϕic , uf for all i = 1, . . . , ncu ,

L2 (Ω)

or, equivalently,
M c uc = PT M f uf .

We then obtain
R = (M c )−1 PT M f .

Due to P being an exact injection, it follows that PT M f P = M c and thus RP = Incu .

Lemma 6.5 then delivers the inverse of G̃fu − Infu in the form
 
(G̃fu − Infu )−1 = P (Gcu − Incu )−1 + Incu R − Infu ,

which can be computed by only an inversion of a ncu -by-ncu matrix from the coarse
grid and the inversion of the coarse grid mass matrix in the restriction operator.
We obtain an expression for the inverse of the transpose similar to equation (6.13)
via
−1 −1
(G̃fu )T − Infu = M G̃fu − Infu M −1 .
6.3 Newton-Picard for optimal control problems 85

6.3.3 Convergence for classical Newton-Picard

In this section we show that for problem (6.1), LISA (6.2) with classical Newton-
Picard preconditioning converges with a grid-independent contraction rate.
For the proof of Theorem 6.7 we need the following lemma. The lemma asserts
the existence of a variable transformation which transforms the Hessian blocks to
identity, and furthermore reveals the structure of the matrices on the subspaces of
fast and slow modes.

Lemma 6.6. Let p ≤ nu , EV = diag(μ1 , . . . , μ p ), EW = diag(μ p+1 , . . . , μnu). Then,

there exist matrices V ∈ Rnu ×p and W ∈ Rnu ×(nu −p) such that with Z = V W
the following conditions hold:
(i) Z is a basis of eigenvectors of Gu , i.e., Gu Z = (V EV W EW ).
(ii) Z is M-orthonormal, i.e., Z T MZ = Inu .
(iii) There exists a non-singular matrix T such that

⎛ ⎞
Inu −p 0 0 0 −Inu −p
⎜ 0 Ip EV − I p 0 ⎟
⎜ 0 ⎟
˜ =⎜
T T JT ⎜ 0 0 γN GTq MV T ⎟
Gq MW ⎟ ,
⎜ ⎟
⎝ 0 EV − I p V T MGq 0 0 ⎠
−Inu −p 0 W T MGq 0 0
⎛ ⎞
0 0 0 0 −EW
⎜ 0 0 ⎟
⎜ 0 0 0 ⎟
⎜ ⎟
T T ΔJT = ⎜ 0 0 0 0 0 ⎟.
⎜ ⎟
⎝ 0 0 0 0 0 ⎠
−EW 0 0 0 0

Proof. The existence of the matrices V and W , as well as conditions (i) and (ii)
follow from Lemma 6.1. To show (iii), we choose

⎛ ⎞
W V 0 0 0
⎜ ⎟
T =⎝0 0 Inq m 0 0 ⎠.
0 0 0 V W
86 6 Newton-Picard preconditioners

Due to M-orthonormality (ii) of V , the Newton-Picard projector from equation

(6.12) simplifies to Π = VV T M. Using V T MW = 0,V T MV = I p , and GTu MV =
MGuV = MV EV we obtain

T T ΔJT
⎛   ⎞⎛ ⎞
0 0 ΠT − Inu GTu M W V 0 0 0
= TT⎝ 0 0 0 ⎠⎜ ⎟
⎝ 0 0 I nq m 0 0 ⎠
MGu (Π − Inu ) 0 0 0 0 0 V W
⎛ ⎞
W T 0 0
⎜V T 0 ⎛ ⎞
⎜ 0 ⎟⎟ 0 0 0 0 −MGuW
⎜ ⎟⎝ ⎠
= ⎜ 0 Inq m 0 ⎟ 0 0 0 0 0
⎜ ⎟
⎝ 0 0 V ⎠ −MGuW 0 0 0
T 0
0 0 WT
⎛ ⎞
0 0 0 0 −EW
⎜ 0 0 0 0 0 ⎟
⎜ ⎟
⎜ ⎟
= ⎜ 0 0 0 0 0 ⎟.
⎜ ⎟
⎝ 0 0 0 0 0 ⎠
−EW 0 0 0 0

Similarly, we obtain for J˜ the form

T T JT
˜
⎛ ⎞⎛ ⎞
M 0 (MVV T GTu − Inu )M WV 0 0 0
⎜ ⎟⎜ ⎟
= TT⎝ 0 γN GTq M ⎠ ⎝ 0 0 I nq m 0 0 ⎠
M(GuVV M − Inu ) MGq
T 0 0 0 0 VW
⎛ ⎞
WT 0 0 ⎛ ⎞
⎜ V T 0 0 ⎟ MW 0 MV (EV − I p ) −MW
⎜ ⎟ MV
⎜ ⎟⎜ ⎟
= ⎜ 0 Inq m 0 ⎟ ⎝ 0 0 γN GTq MV GTq MW ⎠
⎜ ⎟
⎝ 0 0 V ⎠ −MW MV (EV − I p ) MGq
T
0 0
0 0 WT
⎛ ⎞
Inu −p 0 0 0 −Inu −p
⎜ 0 Ip EV − I p 0 ⎟
⎜ 0 ⎟
⎜ ⎟
=⎜ 0 0 γN Gq MV Gq MW ⎟ .
T T
⎜ ⎟
⎝ 0 EV − I p V MGq
T 0 0 ⎠
−Inu −p 0 W T MGq 0 0
6.3 Newton-Picard for optimal control problems 87

We now state the central theorem of this section.

Theorem 6.7. Let μi , i = 1, . . . , nu , denote the eigenvalues of Gu ordered in de-

scending modulus, let 1 < p ≤ nu , and assume μ p > μ p+1 . We further assume the
existence of a linear operator Ḡu : L2 (Σ) → L2 (Ω) which is continuous, i.e.,
 
Ḡq q ≤ C1 qL2 (Σ) for all q ∈ L2 (Σ), (6.14)
L 2 (Ω)

and satisfies the discretization error condition

 
 nu  nq m
 
 ∑ (Gq q) j ϕ j − Ḡq q ≤ C2 qL2 (Σ) for all q ∈ Rnq m , q = ∑ qi ψi , (6.15)
 j=1  i=1
L2 (Ω)

with constants C1 ,C2 ∈ R. If

γ > (C1 +C2 )2 /(1 − μ1 )2

then LISA (6.2) with Newton-Picard preconditioning applied to problem (6.1) con-
verges with a contraction rate of at most μ p+1 /μ1 .

Proof. Due to Theorem 5.26 the contraction rate is given by the spectral radius
σr (J˜−1 ΔJ) = σr (T −1 J˜−1 T −T T T ΔJT ). We obtain the eigenvalue problem
 −1
T T JT
˜ T T ΔJT v − σ v = 0,

which is equivalent to solving the generalized eigenvalue problem

˜ v = 0,
−T T ΔJT v + σ T T JT

with the matrices given by Lemma 6.6. We prove the theorem by contradiction.
Assume that there is a complex eigenpair (v, σ ) such that |σ | ≥ μ p+1 /μ1 > μ p+1 .
Division by σ yields the system

(1/σ )EW v5 + v1 − v5 = 0, (6.16a)

v2 + (EV − I p ) v4 = 0, (6.16b)
γNv3 + GTq M (V v4 +W v5 ) = 0, (6.16c)
(EV − I p ) v2 +V MGq v3 = 0,
T
(6.16d)
(1/σ )EW v1 − v1 +W MGq v3 = 0,T
(6.16e)
88 6 Newton-Picard preconditioners

where v was divided into five parts v1 , . . . , v5 corresponding to the blocks of the
system. Because |σ | > μ p+1 we obtain invertibility of Inu −p − (1/σ )EW and thus
we can eliminate
v5 = (Inu −p − (1/σ )EW )−1 v1 , v4 = (I p − EV )−1 v2 , (6.17a)
v2 = (I p − EV )−1 V T MGq v3 , v1 = (Inu −p − (1/σ )EW )−1 V T MGq v3 . (6.17b)
Substituting these back in equation (6.16c) yields

γN + GTq MV (I p − EV )−2 V T MGq

 −2 T
+ GTq MW Inu −p − σ −1 EW W MGq v3 = 0.

We denote the complex valued matrix on the left hand side with A(σ ). The final
step of the proof consists of showing that A(σ ) is invertible if γ is large enough.
Since M and N are positive definite matrices they have Cholesky decompositions
M = RTM RM , N = RTN RN
with invertible RM ∈ Rnu ×nu , RN ∈ Rnq m×nq m . If we define
 −1  
I p − EV 0 V
B(σ ) := RTM RM Gq R−1
N ∈C
nq m×nq m
0 Inu −p − σ −1 EW W
we obtain
γ −1 R−T −1 −1
N A(σ )RN = Inq m + γ B(σ ) B(σ ).
T

We now estimate the two-norm

 
B(σ )T B(σ )
2
 −1 
 2  2
 I − EV 0 
≤ RM Gq RN 2 RM (V W )2  p
−1 2
 . (6.18)
 0 Inu −p − σ −1 EW 
2
We consider each of the norms on the right hand side of inequality (6.18) sepa-
rately. Due to Lemma 6.6 (ii) we obtain
RM (V W )22 = 1.
The matrix in the last term of inequality (6.18) is a diagonal matrix and so the
maximum singular value of it can be easily determined. Due to |σ | > μ p+1 /μ1 we
have that  −1 
 
 I p − EV 0  1
 −1  ≤ .
 0 Inu −p − σ EW  1 − μ1
2
6.3 Newton-Picard for optimal control problems 89

The first term of inequality (6.18) can be bounded considering

   
RM Gq R−1  = sup RM Gq q
N 2 2
RN q2 =1
q∈Rnq m
 1
2 2
∑
nu
= sup j=1
(Gq q) j ϕ j
nq m
q=∑i=1 qi ψi Ω
$$ 2
Σ q =1
 
 nu 
= sup ∑ j=1 (Gq q) j ϕ j − Ḡq q + Ḡq q
nq m L2 (Ω)
q=∑i=1 qi ψi
qL (Σ) =1
 
2

≤ C2 + sup Ḡq q
nq m L2 (Ω)
q=∑i=1 qi ψi
qL (Σ) =1
 
2

≤ C2 + sup Ḡq q
L2 (Ω)
q∈L2 (Σ)
qL (Σ) =1
2

≤ C1 +C2 .
 
If now γ > (C1 + C2 )2 /(1 − μ1 )2 then γ −1 B(σ )T B(σ )2 < 1 and thus A(σ ) is
invertible. It follows that v3 = 0, which implies v = 0 via equations (6.17). Thus,
(v, σ ) cannot be an eigenpair.
The main result of this section is now at hand:
Corollary 6.8. The asymptotic convergence rate of LISA with classical Newton-
Picard preconditioning on the model problem is mesh-independent, provided γ is
large enough.
Proof. For finer and finer discretizations the largest p + 1 eigenvalues of M −1 L
converge. Thus, also the eigenvalues μ1 and μ p+1 of Gu converge to some μ̄1 < μ̃
and μ̄ p+1 ≤ μ̄1 , with μ̄ given by Lemma 6.3. We construct Ḡq as the infinite
dimensional counterpart to Gq , i.e., Ḡq maps controls in L2 (Σ) to the end value of
the heat equation (6.1b)–(6.1c) with zero initial values for the state. This operator
is continuous (see, e.g., Tröltzsch [152]). Let ε > 0. We can assume that Gq
satisfies the discretization error condition (6.15) with C2 = ε for a fine enough and
also for all finer discretizations. Define
γ̄ = (C1 + ε)2 /(1 − μ̄)2 .
Theorem 6.7 yields that if γ > γ̄ then the asymptotic convergence rate of LISA is
below the mesh-independent bound μ̄ p+1 /μ̄1 .
90 6 Newton-Picard preconditioners

We remark here that our numerical experience suggests the conjecture that the
contraction rate is actually μ p+1 instead of μ p+1 /μ1 .

6.3.4 Numerical solution of the approximated linear system

The implicit inversion of the preconditioner J˜ can be carried out by block elimi-
nation. To simplify notation we denote the residuals by ri , i = 1, 2, 3. We want to
solve ⎛ ⎞⎛ ⎞ ⎛ ⎞
M 0 (G̃Tu − Inu )M u0 r1
⎜ ⎟⎝ ⎠ ⎝ ⎠
⎝ 0 γN GTq M ⎠ q = r2 .
M(G̃u − Inu ) MGq 0 λ r3

Solving the last block-row for u0 and the first for λ , the second block-row becomes
 −1  −1 −1
Hq = r2 − GTq G̃Tu − Inu r1 − M G̃u − Inu M r3 =: r̃, (6.19)

with the nq m-by-nq m symmetric positive-definite matrix

 −1  −1
H = γN + GTq G̃Tu − Inu M G̃u − Inu Gq .

If nq m is moderately small we can set up Gq as well as (G̃u − Inu )−1 Gq according

to Lemma 6.5 or the two-grid analog and thus form matrix H explicitly. Then,
equation (6.19) can be solved for q via Cholesky decomposition of H. Alterna-
tively, we can employ a Preconditioned Conjugate Gradient (PCG) method with
preconditioner N.

Lemma 6.9. Assume there exists a linear operator Ḡu : L2 (Σ) → L2 (Ω) which
satisfies assumptions (6.14) and (6.15). Then the spectral condition number of
matrix N −1 H is bounded by

(C1 +C2 )2
cond2 (N −1 H) ≤ 1 + ,
γ(1 − μ̄)2

with μ̄ from Lemma 6.3.

Proof. The spectral condition number of N −1 H is equal to the ratio of largest to

smallest eigenvalue of N −1 H. Let (q, σ ) ∈ Rnq m × R be an eigenpair of N −1 H,
i.e.,
Hq − σ Nq = 0
6.3 Newton-Picard for optimal control problems 91

n m
and define q = ∑i=1
q
qi ψi ∈ L2 (Σ). We obtain
 2
σ qT Nq = qT Hq = γqT Nq + RM (G̃u − Inu )−1 R−1 
M RM Gq q 2 (6.20a)
"
≥ γqT Nq ⇒ σ ≥ γ,
 2  2 (6.20b)
≤ γqT Nq + RM (G̃u − In )−1 R−1  RM Gq q ,
u M 2 2

By virtue of Lemma 6.3 the largest singular value of G̃u − Inu is bounded by 1 − μ̄
and thus we obtain
 
RM (G̃u − In )−1 R−1 2 ≤ 1/(1 − μ̄)2 . (6.21)
u M 2

For the remaining norm we consider

   
RM Gq q = ∑nq m (Gq q)i ψi  2 (6.22a)
2 i=1 L (Σ)
 nq m   
≤ ∑i=1 (Gq q)i ψi − Ḡq qL2 (Σ) + Ḡq qL2 (Σ) (6.22b)
≤ (C1 +C2 ) qL2 (Σ) = (C1 +C2 ) RN q2 . (6.22c)

We now combine inequalities (6.20), (6.21), and (6.22) to obtain the assertion.
As a consequence of Lemma 6.9 we obtain that the number of required PCG
iterations bounded by a grid-independent number. In our numerical experience
10–20 PCG iterations usually suffice for a reduction of the relative residual to
10−6 .
Solving for u0 and λ is then simple:
 −1  −1   −1
u0 = G̃u − Inu M r3 − Gq q , λ = M −1 G̃Tu − Inu (r1 − Mu0 ) .

Note that once Gq and G̃u (in a suitable representation) have been calculated no
further numerical integration of the system dynamics is required.

6.3.5 Pseudocode
In this section we provide pseudocode to sketch the implementation of the pro-
posed Newton-Picard preconditioners. We focus on the case nq m  nu in which
it is economical to solve equation (6.19) by forming H and using Cholesky de-
composition, but we also discuss implementation alternatives for the case of large
nq m. We use a Matlab® oriented syntax here and assume that the reader is familiar
with linear algebra routines in Matlab® . For readability purposes we further as-
sume that matrices L, M, Lc , M c , P, R, E, X, N and dimensions nu , nq , m are globally
92 6 Newton-Picard preconditioners

accessible in each function. We do not discuss the assembly of Galerkin matrices

L, M, Lc , M c , and U or grid transfer operators P and R here.
The first function computes the matrices needed to evaluate the classical New-
ton-Picard preconditioner later. For numerical stability it is advantageous to per-
form IRAM in eigs with the symmetrified version (see proof of Lemma 6.1) and
to explicitly set the option that the matrix is real symmetric (not shown in pseu-
docode).
Function [P, R, E, X] = classicalApprox
output: Matrices P ∈ Rnu ×p , R ∈ R p×nu with RP = I p , matrices E, X ∈ R p×p
RM = chol(M);
[Ṽ , Ẽ] = eigs(@(u) RTM \ (L * (RM \ u)), nu , p, ’la’);
P = RM \ Ṽ ;
R = PT * M;
E = diag(exp(diag(Ẽ)));
X = diag(1./(diag(E) − 1) + 1);

For the two-grid version, we assume that the prolongation P and restriction R
are given. Because the occurring matrices are small, we can employ the LAPACK
methods in eig instead of IRAM in eigs.

Function [E, X] = coarseGridApprox

output: Coarse grid approximation E ∈ Rnu ×nu of Gu , matrix X ∈ Rnu ×nu
c c c c

[Ṽ , Ẽ] = eig(full(Lc ), full(M c ));

E = Ṽ * diag(exp(diag(Ẽ))) / Ṽ ;
X = inv(E − Incu ) + Incu ;

Now P, R, E, X are known and we can formulate matrix vector and matrix trans-
pose vector products with G̃u .

Function u1 = Gup(u0 ) Function u0 = GupT(u1 )

input : u0 ∈ Rnu input : u1 ∈ Rnu
output: u1 = G̃u u0 ∈ Rnu output: u0 = G̃Tu u1 ∈ Rnu
u1 = (P * (E * (R * u0 ))); u0 = (RT * (E T * (PT * u1 )));

In the same way we can evaluate matrix vector and matrix transpose vector
products with the inverse of G̃u − Inu according to Lemma 6.5.
6.3 Newton-Picard for optimal control problems 93

Function u = iGupmI(r) Function r = iGupTmI(u)

input : r ∈ Rnu input : u1 ∈ Rnu
output: u = (G̃u − Inu )−1 r output: r = (G̃Tu − Inu )−1 u
u = (P * (X * (R * r))) − r; r = (RT *(X T *(PT *u))) − u;

We want to remark that we take the liberty to call the four previous functions
also with matrix arguments. In this case the respective function is understood to
return a matrix of the same size and to be evaluated on each column of the input
matrix. For the computation of matrix vector products with Gu and Gq we define
an auxiliary function which integrates ODE (6.4) for given initial state and control
variables. The control coefficients are constant in time.

Function ue = dG(Δt, us , q̃)

input : Duration Δt, initial value us ∈ Rnu , control coefficients q̃ ∈ Rnq
output: End state ue ∈ Rnu after time Δt
Solve ODE (6.4) with initial value us and constant control q̃, e.g., by ode15s;

Based on the previous function we can now assemble matrix Gq . There are
alternative ways for the assembly. We have chosen an approach for the case that
large intervals for dG can be efficiently and accurately computed through adaptive
step size control as in, e.g., ode15s.

Function Gq = computeGq
output: Matrix Gq ∈ Rnu ×nq m
for j = 1 : nq do
Gq (:, j + nq * (m − 1)) = dG(1 / m, 0, Inq (:, j));
for i = 1 : m − 1 do
for j = 1 : nq do
Gq (:, j + nq * (i − 1)) = dG(1 − i / m, Gq (:, j + nq * (m − 1)), 0);

We can alternatively compute matrix vector and matrix transpose vector prod-
ucts with Gq via the following functions. For the transpose we exploit the expres-
sion
 i T   i   −1
∂ Gu ∂ Gq = U T L−1 ∂ GTu − Inu ∂ GTu = U T L−1 M ∂ Gi+1 u − ∂ Gu M .
i
94 6 Newton-Picard preconditioners

Function u1 = Gq(q)
input : q ∈ Rnq m
output: u1 = Gq q ∈ Rnu
u1 = zeros(nu , 1);
for i = 0 : m − 1 do
u1 = dG(1 / m, u1 , q(i * nq + (1 : nq )));

Function q = GqT(λ )
input : λ ∈ Rnu
output: q = GTq λ ∈ Rnq m
q = zeros (nq * m,1); λ̃ + = M \ λ ;
for i = m − 1 : −1 : 0 do
λ̃ = λ̃ + ; λ̃ + = dG(1 / m, λ̃ , 0);
q(i * nq + (1 : nq )) = U T * (L \ (M * (λ̃ + − λ̃ )));

We can also formulate functions for matrix vector and matrix transpose vector
products with Gu .
Function u1 = Gu(u0 )
input : u0 ∈ Rnu
output: u1 = Gu u0 ∈ Rnu
u1 = dG(1, u0 , 0);

Function u0 = GuT(λ )
input : λ ∈ Rnu
output: u0 = GTu λ ∈ Rnu
u0 = M * dG(1, M \ λ , 0);

For the evaluation of the preconditioner we employ a Cholesky decomposition

of matrix H which can be obtained with the following function.
Function RH = decompH
output: Cholesky factor RH ∈ Rnq m×nq m of H = RTH RH
V = iGupmI(Gq );
RH = chol(γ * N +V T * M * V );

We can finally state the function for a matrix vector product with the symmetric
indefinite matrix J.ˆ For readability we split up the argument and result into three
subvectors.
6.3 Newton-Picard for optimal control problems 95

Function [r1 , r2 , r3 ] = J(u, q, λ )

input : u ∈ Rnu , q ∈ Rnq m , λ ∈ Rnu
output: [r1 ; r2 ; r3 ] = J[u; q; λ ] ∈ Rnu +nq m+nu
r1 = M * u + GuT(λ ) − λ ;
r2 = γ * N * q + GTq * λ ;
r3 = Gu(u) − u + Gq * q;

At last we present pseudocode for matrix vector products with the precondi-
tioner J˜−1 . Again, we split up the argument and result into three subvectors.
Function [u, q, λ ] = iJp(r1 , r2 , r3 )
input : r1 ∈ Rnu , r2 ∈ Rnq m , r3 ∈ Rnu
output: [u; q; λ ] = J˜−1 [r1 ; r2 ; r3 ] ∈ Rnu +nq m+nu
q = RH \ (RTH \ (r2 − iGupTmI(GTq * (r1 − M * iGupmI(r3 )))));
u = iGupmI(r3 − Gq * q);
λ = iGupTmI(r1 − M * u);

We can also substitute the functions Gq and GqT for the occurrences of Gq and
GTq in J and iJp.

6.3.6 Algorithmic complexity

In this section we discuss the algorithmic complexity of the proposed method. To
simplify analysis we only count the number of necessary (fine grid) system integra-
tions which are required when evaluating matrix vector or matrix transpose vector
products with Gu , Gq , or (Gu Gq ). We shall see that we need O(Jz ˆ 0 + F̂ /εO )
simulations to solve the optimization problem (4.1) up to an absolute tolerance of
εO > 0.
If we solve the reduced systems (6.19) exactly then we obtain a grid-independent
contraction bound κ = σr (J˜−1 ΔJ) by virtue of Corollary 6.8. By Lemma 6.9 we
know that we can solve the reduced system (6.19) up to a relative residual tol-
erance εH > 0 using PCG with a grid-independently bounded number k of itera-
tions.
 Abackward analysis in the sense of Lemma 5.1 yields a matrix H̃ such that
H − H̃  ≤ r̃ εH and such that the PCG iterate qk satisfies H̃qk = r̃. Addition-
ally, matrix vector products with the inverse mass matrix M −1 need to be evaluated.
This can also be done at linear cost using diagonal preconditioners with PCG (see
Wathen [162]) to tolerance εM > 0. A similar backward analysis as for H̃ yields a
perturbed mass matrix M̃. Because the eigenvalues of the now H̃ and M̃ dependent
iteration matrix (as a perturbation of J˜−1 ΔJ) depend continuously on the entries
of H̃ and M̃ we obtain that for each κ̃ ∈ (κ, 1) there exist εH , εM > 0 such that
96 6 Newton-Picard preconditioners

the contraction rate of the outer iteration is bounded by κ̃. Thus, we cansolve
the optimization problem (4.1) up to a tolerance εO > 0 within O(Jz ˆ 0 + F̂  /εO )
iterations.
We now count the number of system integrations per iteration under the as-
sumption that we perform nH inner PCG iterations per outer iteration. For the
evaluation of matrix vector products with Jˆ we need two system integrations for
maxtrix vector products with (Gu Gq ) and its transpose. Concerning the matrix
vector product with the preconditioner J˜−1 we observe that multiplications with
G̃u and (G̃u − Inu )−1 do not need any system integrations. However, the setup
of r̃ in equation (6.19) requires one integration for a matrix vector product with
Gq . Furthermore, each inner PCG iteration requires two additional simulations for
matrix vector products with Gq and its transpose. Thus we need 3 + 2nH system
simulations per outer iteration which yields an optimal complexity of O(1/εO )
system integrations for the solution of the optimization problem (4.1) up to toler-
ance εO .
When performed this way the additional linear algebra consists of matrix vector
multiplications with sparse matrices, lower order vector computations, and dense
linear algebra for system sizes bounded by a factor of the grid-independent num-
bers p or ncu , respectively.
In the case of classical Newton-Picard approximation we need to add the com-
plexity of IRAM for the one-time determination of the dominant subspace spanned
by V . A detailed analysis of the numerical complexity for this step is beyond the
scope of this thesis. Suffice it that based on Saad [135, Theorem 6.3 and Cheby-
shev polynomial approximation (4.49)] together with the approximation μnu ≈ 0
we assume that the tangent of the angle between the p-th eigenvector of Gu and
the l-th Krylov subspace decreases linearly in l with a factor depending on the
ultimately grid-independent ratio μ p /μ p+1 which we assume to be greater than
one. We need one matrix vector product with Gu per Arnoldi iteration. Because
this computation is only needed once independently of εO the asymptotic com-
plexity O(1/εO ) does not deteriorate. It does, however, have an effect for practical
computations (see Section 12.3) and can easily dominate the overall cost of the
algorithm already for modest values of p.
We have also found the approach with explicit solution of equation (6.19) via
Cholesky decomposition of H beneficial for the practical computations presented
in Chapter 12. Although we obtain a one-time cubic complexity in nq m and a
square complexity in nq m per outer iteration, the runtime can be much faster than
iterative solution of equation (6.19) because per outer iteration only two system
integrations are required instead of 3 + 2nH .
6.4 Extension to nonlinear problems and Multiple Shooting 97

We want to close this section with the remark that the optimal choice of p, κ̃,
εM , and εH is a complex optimization problem which exceeds the scope of this
thesis.

6.4 Extension to nonlinear problems and Multiple

Shooting
So far we have focused our investigation of Newton-Picard preconditioning on
the linear model problem (6.1). For nonlinear problems we have to deal with the
difficulty that the matrix Gu depends on the current iterate and that thus the domi-
nant subspace can change from one SQP iteration to another. The extension of the
two-grid Newton-Picard preconditioner to nonlinear problems is straight-forward
because the dominant subspace is implicitly given by the coarse grid approxima-
tion. For the classical Newton-Picard preconditioner, however, the situation is
more complicated.
Lust et al. [110] use a variant of the Subspace Iteration, originally developed by
Stewart [149], to update the basis V of the dominant subspace approximation in
each SQP iteration. The Subspace Iteration is an iterative method for the compu-
tation of eigenvalues and eigenvectors. Each iteration consists of three steps (see,
e.g., Saad [133]):
1. Compute V := GuV .
2. Orthonormalize V .
3. Use the QR algorithm on V T GuV to compute its Schur vectors Y and update
V := VY (Schur-Rayleigh-Ritz step).
Locking and shifting techniques can improve efficiency of the method in practical
implementations (see Saad [133]).
The Subspace Iteration is also used simultaneously such that only a few Sub-
space Iterations (ideally only one) are needed per SQP step. Potschka et al. [130]
present preliminary numerical results for a Newton-Picard inexact SQP method
without LISA and using the Euclidean projector.
The reader has surely noticed that we have so far in this chapter only considered
the case of Single but not Multiple Shooting. Again, the two-grid preconditioner
can be extended in a rather canonical way (see Chapter 8 for the remaining details).
For the classical Newton-Picard preconditioner we can sketch two approaches:

Sequential approach. We perform the Subspace Iteration on the product of the

n
local shooting matrices Gu := GuMS . . . G1u . The main drawback of the se-
98 6 Newton-Picard preconditioners

quential approach is that it is impossible to compute GuV in parallel because

the result of G1uV must be available to compute G2u G1uV and so forth.

Simultaneous approach. We introduce a local dominant subspace approximation

basis V i on each shooting interval and perform the Subspace Iterations in a
decoupled way. It is, however, at least unclear how the local error propa-
gates to the accumulated error in the product because GiuV i and V i+1 will
in general not span the same subspace. Furthermore, the convergence speed
of the Subspace Iteration decreases with shorter time intervals, which can
be seen for the linear model problem by considering the matrix exponential
for Giu on a shooting interval of length τ. We obtain for the eigenvalues that
μ ij = exp(τ μ̃ ij ). Thus for smaller τ we obtain larger eigenvalue moduli and
ratios which impair the convergence speed of the Subspace Iteration on each
interval.

Based on these considerations we have decided to develop only the two-grid

version fully for nonlinear problems and Multiple Shooting. For an appropriate
Hessian approximation for nonlinear problems we also refer the reader to Chap-
ter 8.
7 One-shot one-step methods and their
limitations
We want to address the basic question if the results of Chapter 6 for the conver-
gence of the Newton-Picard LISA can be extended to general one-shot one-step
methods. We shall explain this class of problems and see that in the general case
no such result as Theorem 6.7 for the model problem (6.1) is possible. For com-
pleteness we quote large passages of the article Bock et al. [28] with modifications
concerning references to other parts of this thesis.
Many nonlinear problems

g(xs , xc ) = 0, x = (xs , xc ) ∈ Rm+(n−m) , g ∈ C 1 (Rn , Rm ), (7.1)

with fixed xc can be successfully solved with Newton-type methods (see Chapter 5)

given xs0 , xsk+1 = xsk − G−1

k g(xs , xc ).
k
(7.2)

In most cases a cheap approximation Gk ≈ ∂∂xgs (xsk , xc ) with linear contraction rate
of, say, κ = 0.8 is already good enough to produce an efficient numerical method.
In general, cheaper computation of the action of G−1 k on the residual compared
to the action of ( ∂∂xgs )−1 must compensate for the loss of locally quadratic con-
vergence of a Newton method to obtain an overall performance gain within the
desired accuracy. It is a tempting idea to use the same Jacobian approximations
Gk from the Newton-type method in an inexact SQP method for the optimization
problem with the same constraint

min f (x) s.t. g(x) = 0. (7.3)

x∈Rn

From this point of view we call problem (7.1) the forward problem of optimization
problem (7.3) and we will refer to the variables xc as control or design variables
and to xs as state variables.
Using (inexact) SQP methods which do not satisfy g = 0 in every iteration
for (7.3) is usually called simultaneous, or all-at-once approach and has proved
to be successful for several applications, e.g., in aerodynamic shape optimiza-
tion Bock et al. [27], Hazra et al. [80], chemical engineering Potschka et al. [130],

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_7,
© Springer Fachmedien Wiesbaden 2014
100 7 One-shot one-step methods and their limitations

or for the model problem (6.1) in Chapter 6. Any inexact SQP method for equality
constrained problems of the form (7.3) is equivalent to a Newton-type method for
the necessary optimality conditions

∇x L(x, y) = 0, g(x) = 0,

as we have seen in Chapters 4 and 5. We are lead to a Newton-type iteration for

the primal-dual variables z = (x, y) ∈ Rn+m
 −1  
Hk ATk ∇z L(zk )
zk+1 = zk − , (7.4)
Ak 0 g(xk )

where Hk is an approximation of the Hessian of the Lagrangian L and Ak is an

approximation of the constraint Jacobian
dg  
≈ Ak = A1k A2k .
dx
Note that like in Chapter 6 we use a plus sign in the definition of the Lagrangian
here to obtain symmetry of the KKT matrices. If A1k = Gk holds the method
is called one-step because exactly one step of the solver for the forward and the
adjoint problem is performed per optimization iteration.
The discretized model problem (6.1) in Chapter 6 has exactly this structure
where Ak is implicitly given by a Newton-Picard preconditioner for the forward
problem of finding a periodic steady state. Theorem 6.7 shows that in the case of
the model problem we achieve almost the same contraction for the optimization
problem by simply reusing Gk in equation (7.4).
In the remainder of this chapter we illustrate with examples that in general only
little connection exists between the convergence of Newton-type methods (7.2) for
the forward problem and the convergence of simultaneous one-step inexact SQP
methods (7.4) for the optimization problem because the coupling of control, state,
and dual variables gives rise to an intricate feedback between each other within the
optimization problem.
Griewank [71] discusses that in order to guarantee convergence of the simulta-
neous optimization method this feedback must be broken up, e.g., by keeping the
design y fixed for several optimization steps, or by at least damping the feedback
in the update of the design y by the use of “preconditioners” for which he derives
a necessary condition for convergence based on an eigenvalue analysis.
We are interested in the different but important case where there exists a con-
tractive method for the forward problem (e.g., Bock et al. [27], Hazra et al. [80],
Potschka et al. [130], and Chapter 6). If applied to the linearized forward problem,
7.1 Illustrative, counter-intuitive examples in low dimensions 101

we obtain preconditioners which are contractive, i.e., the eigenvalues of the pre-
conditioned system lie in a ball around 1 with radius less than 1. The contraction
property suggests the use of a simultaneous one-step approach. However, we can
show that contraction for the forward problem is neither sufficient nor necessary
for convergence of the simultaneous one-step method.
The structure of this chapter is the following: Based on the Local Contraction
Theorem 5.5 we present in Section 7.1 illustrative, counter-intuitive examples of
convergence and divergence for the forward and optimization problem which form
the basis for the later investigations on recovery of convergence. We continue
with presenting a third example and three prototypical subproblem regularization
strategies in Section 7.2 and perform an asymptotic analysis for large regulariza-
tion parameters in Section 7.3. We also show de facto loss of convergence for one
of the examples and compare the regularization approaches to Griewank’s One-
Step One-Shot projected Hessian preconditioners.

7.1 Illustrative, counter-intuitive examples in low

dimensions
Consider the following linear-quadratic optimization problem

min 1 T
2 x Hx, s.t. (A1 A2 )x = 0 (7.5)
x=(xs ,xc )∈Rn

with symmetric positive-definite Hessian H and invertible A1 . The unique solution

is x∗ = 0. As before we approximate A1 with A 1 such that we obtain a contracting
method for the forward problem. Without loss of generality, let A1 = I (otherwise
−1 from the left). We shall now have a look
multiply the constraint in (7.5) with A 1
at instances of problem (7.5) with n = m = 2. We stress that there is nothing
obviously pathologic about the following examples. The exact and approximated
constraint Jacobians have full rank, the Hessians are symmetric positive-definite,
and A1 is always diagonalizable or even symmetric. We use the notation

   
  H AT H T
A
A = A1 A2 , = A
A 1 A2 , K= , =
K .
A 0 
A 0

 are below 600.

In all examples, the condition numbers of K and K
102 7 One-shot one-step methods and their limitations

7.1.1 Fast forward convergence, optimization divergence

As a first instance we investigate problem (7.5) for the special choice of
⎛ ⎞
0.67 0.69 −0.86 −0.13 1 −0.072
  ⎜ ⎜ 0.69 19 2.1 −1.6 −0.072 0.99 ⎟
⎟
H AT = ⎜ ⎟ , (Ex1)
⎝ −0.86 2.1 1.8 −0.33 −0.95 0.26 ⎠
−0.13 −1.6 −0.33 0.78 −1.1 −0.19

According to the Local Contraction Theorem 5.5 and Remark 5.6 the choice of
1 = I leads to a fast linear contraction rate for the forward problem of
A
−1 A1 ) = σr (I − A1 ) ≈ 0.077 < 1.
κF = σr (I − A1

However, for the contraction rate of the inexact SQP method with exact Hessian
and exact constraint derivative with respect to xc , we get

−1 K) ≈ 1.07 > 1.

κO = σr (I − K

Thus the full-step inexact SQP method does not have the property of linear local
convergence. In fact it diverges if the starting point z0 has a non-vanishing compo-
nent in the direction of any generalized eigenvector of I − K −1 K corresponding to
a Jordan block with diagonal entries greater than 1.

7.1.2 Forward divergence, fast optimization convergence

Our second example is
⎛ ⎞
17 13 1.5 −0.59 0.27 −0.6
  ⎜ ⎟
⎜ 13 63 7.3 −4.9 −0.6 0.56 ⎟
H AT = ⎜ ⎟ . (Ex2)
⎝ 1.5 7.3 1.2 −0.74 −0.73 −3.5 ⎠
−0.59 −4.9 −0.74 0.5 −1.4 −0.0032

We obtain
−1 A1 ) ≈ 1.20 > 1,
κF = σr (I − A −1 K) ≈ 0.014 < 1,
κO = σr (I − K
1

i.e., fast convergence of the method for the optimization problem but divergence
of the method for the forward problem. From these two examples we see that in
general only little can be said about the connection between contraction for the
forward and the optimization problem.
7.2 Subproblem regularization 103

7.2 Subproblem regularization without changing the

Jacobian approximation
We consider another example which exhibits de facto loss of convergence for
Griewank’s One-Step One-Shot method and for certain subproblem regulariza-
tions. By de facto loss of convergence we mean that although κF is well below 1
(e.g., below 0.5), κO is greater than 0.99. With
⎛ ⎞
0.83 0.083 0.34 −0.21 1.1 0
  ⎜ ⎜ 0.083 0.4 −0.34 −0.4 1.7 0.52 ⎟
⎟
H AT = ⎜ ⎟ (Ex3)
⎝ 0.34 −0.34 0.65 0.48 −0.55 −1.4 ⎠
−0.21 −0.4 0.48 0.75 −0.99 −1.8
we obtain
−1 A1 ) ≈ 0.48 < 1,
κF = σr (I − A −1 K) ≈ 1.54 > 1.
κO = σr (I − K
1

The quantities Nxx , Gy , Gu in the notation of Griewank [71] are

Nxx = μH, −1 A1 ,
Gy = I − A Gu = −A2 ,
1

where μ > 0 is some chosen weighting factor for relative scaling of primal and
dual variables. Based on
Z(λ ) = (λ I − Gy )−1 Gu , H(λ ) = (Z(λ )T , I)Nxx (Z(λ )T , I)T ,
we can numerically verify that the projected Hessian preconditioners H(λ ), λ ∈
[−1, 1], do not restore contraction. The lowest spectral radius of the iteration ma-
trix is 1.17 for λ = −0.57 and larger for all other values (compare Figure 7.1).
We now investigate three different modifications of the subproblems which do
not alter the Jacobian blocks of the KKT systems. These modifications are based
on
κO = σr (I − K−1 K) = σr (K
−1 (K − K)),
which suggests that small eigenvalues of K  might lead to large κO . Thus we regu-
  −1
larize K such that the inverse K does not have large eigenvalues in the directions
of inexactness of ΔK = K  − K.
We consider three prototypical regularization methods here which all add a pos-
 The regularizing matrices are
itive multiple α of a matrix Λ to K.
⎛ ⎞ ⎛ ⎞ ⎛ ⎞
I 0 0 I 0 0 0 0 0
Λp = ⎝0 I 0⎠ , Λpd = ⎝0 I 0 ⎠ , Λhp = ⎝0 I 0⎠ ,
0 0 0 0 0 −I 0 0 0
104 7 One-shot one-step methods and their limitations

Contraction κ
20

10

0
-1 -0.5 0 0.5 1
λ

Figure 7.1: Contraction rates for One-Step One-Shot preconditioning with the projected
Hessians H(λ ). The two gaps are due to the two eigenvalues of Gy rendering λ I − Gy
singular.

where the subscripts stand for primal, primal-dual, and hemi-primal (i.e., only in
the space of design variables), respectively.

7.3 Analysis of the regularized subproblems

We investigate the asymptotic behavior of the subproblem solution for α → ∞ for
the primal, primal-dual, and hemi-primal regularization. We assume invertibility
of the approximation A1k and drop the iteration index k. We generally assume that

H is positive-definite on the nullspace of the approximation A.
Consider the α-dependent linear system for the step determination of the inexact
SQP method
   
 + αΛ Δx(α) = − ,
K (7.6)
Δy(α) −r

where  is the current Lagrange gradient and r is the current residual of the equality
constraint. We use a nullspace method to solve the α-dependent system (7.6). Let
matrices Y ∈ Rn×m and Z ∈ Rn×(n−m) have the properties
 
 = 0, ZTZ 0
AZ (Z Y )T (Z Y ) = , det(Y Z) = 0.
0 Y TY

 These are completed to

In other words, the columns of Z span the nullspace of A.
form a basis of Rn by the columns of Y which are orthogonal to the columns of Z.
In the new basis, we have Δx = Y p + Zq, with (p, q) ∈ Rn .
7.3 Analysis of the regularized subproblems 105

7.3.1 Primal regularization

The motivation for the primal regularization stems from an analogy to the Le-
venberg-Marquardt method which, in the case of unconstrained minimization, is
equivalent to a trust-region modification of the subproblem (see, e.g., Nocedal and
Wright [121]). It turns out that the regularization with Λp bends the primal sub-
problem solutions towards the step of smallest Euclidean norm onto the linearized
feasible set. However, it leads to a blow-up in the dual solution. From the follow-
ing Lemma we observe that the primal step for large α is close to the step obtained
by the Moore-Penrose-Pseudoinverse A + = AT (A
AT )−1 for the underdetermined
system (7.1) and that the step in the Lagrange multiplier blows up for r = 0, and
thus convergence cannot be expected.

Lemma 7.1. Under the general assumptions of Section 7.3 the solution of equa-
tion (7.6) for the primal regularization for large α is asymptotically given by

+ r + (1/α)ZZ + H A
Δx(α) = −A + r −  + o(1/α),

A
Δy(α) = α(A + )T (H A
T )−1 r + (A + r − ) + o(1).

 is invert-
Proof. From the second block-row of equation (7.6) and the fact that AY
  −1
ible due to A having full rank we obtain p = −(AY ) r. Premultiplying the first
block-row of equation (7.6) with Z T from the left yields the α-dependent equation

Z T HY p + Z T HZq + αZ T Zq + Z T  = 0. (7.7)

Let α > 0 and β = 1/α. Solutions of equation (7.7) satisfy

 
 )−1 r = 0.
Φ(q, β ) := β Z T HZ + Z T Z q + β Z T  − HY (AY

It holds that Φ(0, 0) = 0 and ∂∂Φq (0, 0) = Z T Z is invertible, as Z has full rank.
Therefore the Implicit Function Theorem yields the existence of a neighborhood
U ⊂ R of 0 and a continuously differentiable function q̄ : U → Rm such that q̄(0) =
0 and
Ψ(β ) := Φ(q̄(β ), β ) = 0 ∀β ∈ U.
∂ Φ dq̄
Using 0 = dΨ
dβ = ∂ q dβ + ∂∂ Φ
β and Taylor’s Theorem we have

dq̄  )−1 r −  + o(β ),

q̄(β ) = q̄(0) + (0)β + o(β ) = β (Z T Z)−1 Z T HY (AY
dβ
106 7 One-shot one-step methods and their limitations

which lends itself to the asymptotic

 )−1 r + (1/α)Z(Z T Z)−1 Z T HY (AY

Δx(α) = −Y (AY  )−1 r −  + o(1/α) (7.8)

of the primal solution of equation (7.6) for large regularization parameters α.

Consider aspecial
 choice for the matrices Y and Z based on the QR decompo-
 = Q R B with unitary Q and invertible R. We define
sition A
   
−R−1 B RT T Q
Z= , Y= =A
I BT

 )−1 = A
and obtain Y (AY T QQ−1 (A AT )−1 = A
+ , which yields the first assertion of
the Lemma.
For the corresponding dual solution we multiply the first block-row of equa-
tion (7.6) with Y T from the left to obtain

 )T Δλ (α) +Y T  = 0.
Y T (H + αI)Δx(α) + (AY

After some rearrangements and with the help of the identity

(A  )−T Y T (I − Z(Z T Z)−1 Z T ) = (A

+ )T (I − ZZ + ) = (AY + )T

we obtain the second assertion of the Lemma.

7.3.2 Primal-dual regularization

The primal-dual regularization is motivated by moving all eigenvalues of the reg-
ularized KKT matrix away from zero. It is well known that under our assumptions
 has n + m positive and n negative eigenvalues (see Gould [63]). The
the matrix K
primal regularization method only moves the positive eigenvalues away from zero.
By adding the −I term to the lower right block, also the negative eigenvalues can
be moved away from zero while conserving the inertia of K.

Lemma 7.2. Under the general assumptions of Section 7.3 the solution of equa-
tion (7.6) for the primal-dual regularization with large α is asymptotically given
by
     
Δx(α) 1  1 −
= − Λpd + o(1/α) = + o(1/α).
Δy(α) α r α r
7.3 Analysis of the regularized subproblems 107

Proof. Define again β = 1/α, z = (Δx, Δy), and

 
 + Λpd )z + β 
Φ(z, β ) = (β K .
r

It holds that
∂Φ  + Λpd , ∂Φ
Φ(0, 0) = 0, = βK (0, 0) = Λpd .
∂z ∂z
The Implicit Function Theorem and Taylor’s Theorem yield the assertion.
Consider the limit case
   
Δx(α) 1 
= − Λpd
Δy(α) α r

 If all the
and the corresponding local contraction rate κpd = σr (I − (1/α)Λpd K).
 are larger
real parts of the (potentially complex) eigenvalues of the matrix Λpd K
than 0, contraction for large α can be recovered although contraction may be ex-
tremely slow, leading to de facto loss of convergence.

7.3.3 Hemi-primal regularization

 only on the design vari-
In this section we are interested in a regularization of K
ables xc with Λhp . From the following Lemma we observe that for large α, the
primal solution of the hemi-primal regularized subproblem tends toward the step
obtained from equation (7.2) for the underdetermined system (7.1) and that the
dual variables do not blow up for large α in the hemi-primal regularization.

Lemma 7.3. Under the general assumptions of Section 7.3 the solution of equa-
tion (7.6) for the hemi-primal regularization is for large α asymptotically given
by
    −1  
−A−1 r A r
Δx(α) = 1 + (1/α)ZZ T H 1 −  + o(1/α), (7.9a)
0 0
 −1 T   −1  

A  r
A
Δy(α) = 1 H 1 −  + o(1), (7.9b)
0 0

T
−1 A2 )T I
with the choice Z = (−A 1 A2 )T = A
and Y = (A T .
1
108 7 One-shot one-step methods and their limitations

Proof. By our general assumption A 1 is invertible and the previous assumptions on

 )−1 = A
Y and Z are satisfied. Again it holds that Y (AY + . We recover p as before.
Let β = 1/α. We can define an implicit function to determine q(β ) asymptotically
via
Φ(q, β ) = (β Z T HZ + I)q +Y2 p + β Z T (HY p + ) ,
where we used that Z2T Z2 = I. It holds that Φ(−AT2 p, 0) = 0 and ∂∂Φq (−AT2 p, 0) = I.
Thus the Implicit Function Theorem together with Taylor’s Theorem yields
 
q(β ) = −AT2 p − β Z T HY p +  − HZAT2 p + o(β ).

By resubstitution of p and q(1/α) by the use of the identity

(Y − ZAT2 )(AY −T

 )−1 = (A 1 0)
T

we recover the first assertion of the Lemma.

For the dual solution, we again multiply the first block-row of equation (7.6)
with Y T from the left to obtain
  
0 0  )T Δy(α) +Y T  = 0,
YT H +α Δx(α) + (AY
0 I
which after some rearrangements yields the second assertion.
Consider the limit case α → ∞. We recover from equation (7.9a) that
 −1 
−A  rk
Δzk = 1 .
0

Hence xck = xc∗ stays constant and xk converges to a feasible point x∗ with the con-
traction rate κF of the Newton-type method for problem (7.1). For the asymptotic
step in the dual variables we then obtain
T
−T
Δyk = −A −T −T
1 (∇x f (xk ) + ∇x g(xk )yk ) + A1 0 H A1 0 rk .

For the convergence of the coupled system with xk and yk let us consider the Jaco-
bian of the iteration (xk+1 , yk+1 ) = T (xk , yk ) (with suitably defined T )
 
dT I−A −1 ∇x g(x)T 0
= 1 .
d(x, y) ∗ I−A −T ∇x g(x)
1

Hence (xk , yk ) converges with linear convergence rate κF , and yk converges to

−T
1 ∇x f (x∗ , y∗ ).
y∗ = −A
7.3 Analysis of the regularized subproblems 109

2.5
Contraction κO 2
1.5
1
0.5
10−4 10−3 10−2 10−1 100 101 102 103 104 105 106
Regularization α
1.005
Contraction κO

0.995
10−4 10−3 10−2 10−1 100 101 102 103 104 105 106
Regularization α

Figure 7.2: Divergence of the primal regularization (− · ) and de facto loss of convergence
for primal-dual (−−) and hemi-primal (−) regularization for example Ex3 depending on
the regularization value α. The lower diagram is a vertical close-up around κO = 1 of the
upper diagram.

Thus the primal-dual iterates converge to a point which is feasible and stationary
with respect to xs but not necessarily to xc . Taking α large but finite we see that the
hemi-primal regularization damps design updates while correcting state and dual
variables with a contraction of almost κF .

7.3.4 Divergence and de facto loss of convergence for

subproblem regularizations
Figure 7.2 depicts the dependence of κO of the optimization method on the regu-
larization parameter α and the choice of regularization (primal, primal-dual, and
hemi-primal) on example Ex3. The example was specifically constructed to show
de facto loss of convergence for all three regularizations. Obviously the primal
regularization does not even reach κO = 1.
We want to remark that the above discussion is not a proof of convergence for the
primal-dual or hemi-primal regularization approach. Nonetheless we have given a
counter-example which shows failure of the primal regularization approach. With
110 7 One-shot one-step methods and their limitations

the de facto loss of convergence in mind we believe that a proof of convergence

for the other regularization strategies is of only limited practical importance.
8 Condensing

Especially on fine space discretizations we obtain large scale quadratic subprob-

lems (5.34) in the inexact SQP method described in Chapter 5. The goal of this
chapter is to present a condensing approach which is one of two steps for the so-
lution of these large scale QPs. It consists of a structure exploiting elimination of
all discretized PDE variables from the QP. The resulting equivalent QP is of much
smaller, grid-independent size and can then, in a second step, be solved by, e.g., a
Parametric Active Set Method (PASM) which we describe in Chapter 9.

In Section 8.1 of this chapter we describe the typical multiple shooting struc-
ture. We highlight the additional Newton-Picard structures in Section 8.2. Then
we present the exploitation of these structures for the elimination of the discretized
PDE states in a rather general way in Section 8.3 and develop a particular Newton-
Picard Hessian approximation which fits well in the condensing framework in Sec-
tion 8.4. Based on the introduced notation we end this chapter with a result of
scaling invariance of the Newton-Picard LISA-Newton method in Section 8.5.

8.1 Multiple shooting structure

In their seminal paper, Bock and Plitt [25] have described a condensing technique
for the quadratic subproblems arising in an SQP method for Direct Multiple Shoot-
ing. We specialize this approach for the case that either fixed initial values or
boundary value equality constraints are posed on the PDE states. In Section 8.2
we extend the approach with the exploitation of the Newton-Picard structure in the
approximated Hessian and Jacobian matrices (see Chapter 6).

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_8,
© Springer Fachmedien Wiesbaden 2014
112 8 Condensing

Recall the discretized NLP (3.3) on level l. To avoid notational clutter we drop
the discretization level index l. The NLP (3.3) then reads

minimize
n
Φ(snMS , vnMS ) (8.1a)
(qi ,si ,vi )i=0
MS

s. t. rbs (snMS , vnMS ) − s0 = 0, (8.1b)

rbv (snMS , vnMS ) − v0 = 0, (8.1c)
u (t ; q
i i i−1 i−1 i−1
,s ,v
i
) − s = 0, i = 1, . . . , nMS , (8.1d)
v (t ; q
i i i−1
,s i−1
,v i−1
) − v = 0,
i
i = 1, . . . , nMS , (8.1e)
nMS −1
q nMS
−q = 0, (8.1f)
r (q
i i−1
,v i−1
) ≥ 0, i = 1, . . . , nMS , (8.1g)
r (v
e nMS
) ≥ 0, (8.1h)

where we have split up the boundary condition rb into two parts rbs and rbv . We
abbreviate derivatives that occur in the remainder of this chapter according to

∂ rbs ∂ rbs ∂ rbv ∂ rbv

Rbss := , Rbsv := , Rbvs := , Rbvv := ,
∂ snMS ∂ vnMS ∂ snMS ∂ vnMS

∂ ui ∂ ui ∂ ui
Giq := , Gis := , Giv := ,
∂ qi−1 ∂ si−1 ∂ vi−1
∂ vi ∂ vi ∂ vi
Hqi := i−1 , Hsi := i−1 , Hvi := i−1 ,
∂q ∂s ∂v
∂ ri ∂ ri re
Ri,i
q := , Ri,i
v := , Re := n .
∂ qi−1 ∂ vi−1 ∂ v MS

Let the Lagrangian of NLP (8.1) be denoted by L . It is well-known (see, e.g.,

Bock and Plitt [25]) that due to the at most linear coupling between variables
corresponding to shooting nodes i and i + 1 the Hessian matrix of the Lagrangian
L has block diagonal form. Our goal is to eliminate all PDE state variables si and
so we regroup the variables in the order

(s0 , . . . , snMS , v0 , . . . , vnMS , q0 , . . . , qnMS ).

The elimination is based on the s0 -dependent part of boundary condition (8.1b) and
on the matching conditions (8.1d). Thus we also shuffle the constraint order such
8.1 Multiple shooting structure 113

that they are the first ns (nMS + 1) constraints. We are lead to consider NLP (8.1)
in the ordering
minimize f (x1 , x2 ) (8.2a)
(x1 ,x2 )∈Rn1 +n2

s. t. gi (x1 , x2 ) = 0, i ∈ E1 , (8.2b)
gi (x1 , x2 ) = 0, i ∈ E2 , (8.2c)
gi (x1 , x2 ) ≥ 0, i∈I, (8.2d)
where |E1 | = n1 . In Section 8.2 we describe how to exploit gi , i ∈ E1 , for partial re-
duction on the QP level. Now x1 contains only the discretized PDE state variables
and gi , i ∈ E1 , comprises the boundary and matching conditions (8.1b) and (8.1d).
Then the compound derivative of the constraints has the form
⎛ ⎞
C11 C12
C = ⎝C21 C22 ⎠
C31 C32
⎛ ⎞
−I Rbss Rbsv
⎜ 1 ⎟
⎜ Gs −I G1v G1q ⎟
⎜ ⎟
⎜ .. .. .. .. ⎟
⎜ . . . . ⎟
⎜ ⎟
⎜ G
nMS
−I G
nMS
G
nMS ⎟
⎜ s v q ⎟
⎜ R b −I R b ⎟
⎜ vs vv ⎟
⎜ H1 Hv1 −I Hq1 ⎟
⎜ s ⎟
⎜ ⎟
=⎜ ⎜
..
.
..
.
..
.
..
. ⎟.
⎟
⎜ ⎟
⎜ Hs
nMS
Hv
nMS
−I Hq
nMS
⎟
⎜ ⎟
⎜ I −I ⎟
⎜ ⎟
⎜ i,1 i,1 ⎟
⎜ Rv Rq ⎟
⎜ ⎟
⎜ . .. . .. ⎟
⎜ ⎟
⎜ i,nMS i,nMS ⎟
⎝ Rv Rq ⎠
e,nMS
Rv
We want to stress that contrary to the appearance the block C11 is several orders of
magnitude larger than the blocks C22 and C32 on fine spatial discretization levels.
The next lemma shows that under suitable assumptions C11 is invertible. We use
products of non-commuting matrices where the order is defined via
nMS 0
∏ Gis := Gns MS · · · G1s and ∏ Gis = I by convention.
i=1 i=1
114 8 Condensing

n
Lemma 8.1. Let MB = I − (∏i=1 MS
Gis )Rbss . If MB is invertible so is C11 and the
−1
inverse is given by C11 =
⎛   ⎞ ⎛ ⎞
I
0
i Rb ⎛ ⎞ I
⎜ ∏ G s ss ⎟ I ⎜ 1 ⎟
⎜ i=1 ⎟ ⎜ ⎟
⎟ ⎜ ∏ Gs I
i
⎜ .. .. ⎟⎜ .. ⎟
⎜ . . ⎟⎜ . ⎟ ⎜ i=1 ⎟
−⎜ n −1  ⎟ ⎜ ⎟⎜ . .. .. ⎟.
⎜ ⎟⎝ I ⎠ ⎜ .. . . ⎟
⎜ ⎟ ⎜ ⎟
MS
⎝ I ∏ Gs Rss ⎠
i b
−1 ⎝nMS nMS ⎠
i=1 MB ∏ sG i ···
∏ s G i I
I i=1 i=nMS

Proof. We premultiply C11 with the matrices in the assertion one after the other to
obtain
⎛   ⎞
⎛ ⎞ 0
⎛ ⎞
⎜−I ∏ Gs Rbss ⎟
i
I
⎜ G1 I ⎟ −I1 Rbss ⎜ i=1  ⎟
⎜ s ⎟ ⎜G −I ⎟ ⎜ 1 ⎟
⎜ . ⎟⎜ s ⎟ ⎜ −I ∏ Gis Rbss ⎟
⎜ .. .. ..
. . ⎟⎜ . . ⎟=⎜ ⎟,
⎜ ⎟⎝ .. .. ⎠ ⎜ i=1 ⎟
⎝nMS ⎠ ⎜ . . ⎟
nMS
∏ Gs · · · G s I
i nMS
Gs −I ⎝ .. .
. ⎠
i=1 −MB
⎛   ⎞ ⎛   ⎞
0 0
⎛ ⎞ −I ∏ G i Rb −I ∏ G i Rb
I ⎜ s ss ⎟ ⎜ s ss ⎟
⎜ i=1  ⎟ ⎜ i=1  ⎟
⎜ .. ⎟⎜ 1 ⎟ ⎜ 1 ⎟
⎜ . ⎟⎜ −I ∏ Gis Rbss ⎟ ⎜ −I ∏ Gis Rbss ⎟
⎜ ⎟⎜ ⎟ = ⎜ ⎟,
⎝ I ⎠⎜ i=1 ⎟ ⎜ i=1 ⎟
⎜ .. .. ⎟ ⎜ .. .. ⎟
−1 ⎝ . . ⎠ ⎝ . . ⎠
MB
−MB −I
⎛   ⎞⎛   ⎞
0 0
⎜−I − ∏ Gis Rbss ⎟ ⎜−I ∏ Gis Rbss ⎟
⎜ i=1 ⎟⎜  i=1  ⎟
⎜ .. .. ⎟⎜ 1 ⎟
⎜ . . ⎟⎜ −I ∏ G i Rb ⎟
ss ⎟ = I.
⎜ n −1  ⎟ ⎜ s
⎜ ⎟⎜ i=1 ⎟
⎜ ∏ Gis Rbss ⎟ ⎜ ⎟
MS
−I − .. ..
⎝ i=1
⎠ ⎝ . . ⎠
−I −I
This proves the assertion.
The assumption of invertibility of MB is merely that one is not an eigenvalue of
the matrix  nMS i  b
GB := ∏i=1 Gs Rss
which coincides with the monodromy matrix of the periodicity condition (8.1b) in
the solution of NLP (8.1).
8.2 Newton-Picard structure 115

8.2 Newton-Picard structure

Now we investigate the structures arising from the approximation of the blocks in
C via Newton-Picard (see Chapter 6). We want to stress that it does not matter if
the two-grid or classical version of Newton-Picard is applied. We only assume that
there exist a prolongation operator P and a restriction operator R which satisfy

RP = I. (8.3)

We now approximate the blocks in C. Let hatted matrices (ˆ) denote the evaluation
of a matrix on either a coarse grid (two-grid variant) or on the dominant subspace
(classical variant). Then we assemble the approximations () from the hatted ma-
trices preceded and/or succeeded by appropriate prolongation and/or restriction
matrices according to

Rbss = PR̂bss R, Rbsv = PR̂bsv , Rbvs = R̂bvs R, Rbvv = R̂bvv , (8.4a)

iq = PĜiq ,
G Gis = PĜis R, iv = PĜis ,
G (8.4b)
 iq = Ĥqi ,
H  is = Ĥsi R,
H  iv = Ĥvi ,
H (8.4c)
Ri,i
q = R̂q ,
i,i
Ri,i
v = R̂v ,
i,i
Re = R̂e . (8.4d)

The following lemma shows that the approximation of MB can be cheaply evalu-
ated and inverted because it only involves operations on the coarse grid or on the
low-dimensional dominant subspace.
Lemma 8.2. Let
   
nMS nMS
ĜB := ∏ Ĝis R̂bss , B :=
G ∏ is
G Rbss ,
i=1 i=1

M̂B := I − ĜB , M B .
 B := I − G

 B and it holds that

If M̂B is invertible so is M
   
 −1 = I − PĜB R −1 = I − P I − M̂ −1 R.
M B B

Proof. From equation (8.3) we obtain

 
nMS
B = ∏ PĜis R PR̂bss R = PĜB R
G
i=1

and thus
 B = I − PĜB R.
M
116 8 Condensing

The calculation
  %   &
 −1 = I − PĜB R I − P I − I − ĜB −1 R
BM
M B
 %  −1 &
= I − PĜB R − P I − ĜB I − I − ĜB R
 
= I − P ĜB + I − ĜB − I R = I
yields the assertion.
The structure of C and the assertion of Lemma 8.1 is also preserved if we use the
proposed approximations. Thus Lemma 8.2 suggests that it is possible to compute
the inverse of the approximation of the large block C11 in a cheap way. We prove
this supposition in Theorem 8.3 but before we need to introduce another notational
convention, the Kronecker product of two matrices. We only use the special case
where the left-hand factor is the identity and thus we have for an arbitrary matrix
A that
⎛ ⎞
A
⎜ .. ⎟
I p×p ⊗ A := ⎝ . ⎠ (p instances of A blocks on the diagonal).
A
Theorem 8.3. Define the projectors
Πslow = InMS ×nMS ⊗ (PR), Πfast = I − Πslow .
Then
C−1
11 Π
slow −1
= (I ⊗ P)Ĉ11 (I ⊗ R), C−1
11 Π
fast
= −Πfast .
−1
Proof. Lemma 8.1 yields a decomposition of C11 into a product of the three ma-
trices A1 A2 A3 . The same type of decomposition holds when the blocks in C are
substituted by their tilde or hat counterparts. We now show in three steps that
k (I ⊗ P) = (I ⊗ P)Âk ,
A k = 1, 2, 3,
from which we can immediately infer the assertion
C−1
11 Π
slow 1 A
=A 2 A
3 (I ⊗ (PR)) = (I ⊗ P)Â1 Â2 Â3 (I ⊗ R) = (I ⊗ P)Ĉ−1 (I ⊗ R).
11

The cases k = 1, 3 follow from

     
j2 j2 j2
∏ Gs P = ∏ (PĜs R) P = P ∏ Ĝs ,
i i i
i= j1 i= j1 i= j1

Rbss P = PR̂bss RP = PR̂bss .

8.2 Newton-Picard structure 117

The case k = 2 only involves the inverse given by Lemma 8.2

   
M −1 P = I − P I − M̂ −1 R P = PM̂ −1 .
B B B

The proof of the assertion for Πfast is based on equation (8.3) which yields

R(I − PR) = R − RPR = 0. (8.5)

We obtain
1 Πfast = −Πfast ,
A 2 Πfast = Πfast ,
A 3 Πfast = Πfast ,
A

because all off-diagonal blocks are eliminated due to equation (8.5) and
   
M −1 (I − PR) = I − P I − M̂ −1 R (I − PR) = I − PR.
B B

From equation (8.5) follows also immediately that

Πfast Πfast = I ⊗ [(I − PR)(I − PR)] = Πfast ,

i.e., Πfast is idempotent and thus indeed a projector. Hence we obtain

C−1
11 Π
fast
=A 2 A
1 A 3 Πfast = −Πfast

which shows the last assertion.

Corollary 8.4. If it exists, the Newton-Picard approximation of block C11 has the
inverse  −1 
C−1
11 = (I ⊗ P) Ĉ11 + I (I ⊗ R) − I.

Proof. Consider

C−1 −1 slow + C−1 Πfast = (I ⊗ P)Ĉ−1 (I ⊗ R) + (I ⊗ (PR)) − I

11 = C 11 Π 11 11
 −1 
= (I ⊗ P) Ĉ11 + I (I ⊗ R) − I,

which follows directly from Theorem 8.3.

Remark 8.5. The inversion of C11 via the formula from Corollary 8.4 also reduces
the number of needed restrictions to the minimum of nMS . This is important for
FEM discretizations where an L2 restriction involves the inversion of a reduced
mass matrix.
Thus we see that the condensing operations involving C11 can be efficiently
computed involving only operations on the coarse grid or on the small Newton-
Picard subspace plus nMS prolongations and restrictions.
118 8 Condensing

8.3 Elimination of discretized PDE states

We now consider QPs with a structure inherited from NLP (8.2)
 T      T  
1 x1 B11 B12 x1 b x1
minimize + 1 (8.6a)
(x1 ,x2 )∈Rn1 +n2 2 x2 B21 B22 x2 b2 x2
s. t. C11 x1 +C12 x2 = c1 , (8.6b)
C21 x1 +C22 x2 = c2 , (8.6c)
C31 x1 +C32 x2 ≥ c3 . (8.6d)

Imagine the variable vector x1 comprising all discretized PDE states and the small
variable vector x2 containing the remaining degrees of freedom. The proof of the
following theorem can be carried out via KKT transformation rules as in Leinewe-
ber [106]. We want to give a slightly shorter proof here which can be interpreted
as a partial null-space approach.
Theorem 8.6. Assume that C11 in QP (8.6) is invertible and define
 −1 
−C11 C12
Z= , B = Z T BZ,
I
c1 = C11
−1
c1 , b = B21 c1 + b2 −C12
T −T
C11 (B11 c1 + b1 ),
c2 = c2 −C21 c1 , C2 = C22 −C21C11
−1
C12 ,
c3 = c3 −C31 c1 , C3 = C32 −C31C11
−1
C12 .

Let furthermore (x2∗ , y∗E2 , y∗I ) ∈ Rn2 +m2 +m3 be a primal-dual solution of the QP

1 T 
minimize x B x2 + bT x2 s. t. C2 x2 = c2 , C3 x2 ≥ c3 . (8.7)
x2 ∈R
n 2 2 2
If we choose

x1∗ = C11
−1
(c1 −C12 x2∗ ), (8.8a)
 
y∗E1 = C11
−T
(B12 − B11C11 −1
C12 )x2∗ + B11 c1 + b1 −C21
T ∗ T ∗
yE2 −C31 yI (8.8b)

then (x∗ , y∗ ) := (x1∗ , x2∗ , y∗E1 , y∗E2 , y∗I ) is a primal-dual solution of QP (8.6).
Proof. We first observe that constraint (8.6b) is equivalent to equation (8.8a) and
that thus  ∗  
x1 ∗ c
∗ = Zx2 + 1
x2 0
8.3 Elimination of discretized PDE states 119

is satisfied. Let us define

 
I
Y= .
0
 
The unit upper triangular matrix Q := Y Z is invertible and we can multiply
the stationarity condition of QP (8.6) from the left with QT to obtain the equivalent
system of equations

0 = Y T BZx2∗ + B11 c1 + b1 −C11

T ∗ T ∗
yE1 −C21 T ∗
yE2 −C31 yI , (8.9a)
⎛ ⎞T ⎛ ∗ ⎞
   C11 C12 yE
∗ B11  T⎝ ⎠ ⎜ ∗1 ⎟
0 = Z BZx2 + Z
T T
c + b − Z C21 C22 ⎝yE2 ⎠ . (8.9b)
B21 1
C31 C32 y∗I

Expansion of Y T BZ yields that condition (8.9a) is equivalent to equation (8.8b)

and by virtue of
⎛ ⎞ ⎛ ⎞
C11 C12  −1  0
−C C
CZ = ⎝C21 C22 ⎠ 11 12
= ⎝C2 ⎠
I
C31 C32 C3

condition (8.9b) is equivalent to the stationarity condition of QP (8.7)

B x2∗ + b −C2 T y∗E2 −C3 T y∗I = 0.

Feasibility is guaranteed by

C21 x1∗ +C22 x2∗ = C21C11

−1
(c1 −C12 x2∗ ) +C22 x2∗ = c2 ⇔ C2 x2∗ = c2 ,
C31 x1∗ +C32 x2∗ = C31C11
−1
(c1 −C12 x2∗ ) +C32 x2∗ ≥ c3 ⇔ C3 x2∗ ≥ c3 .

Finally complementarity holds because the multipliers y∗I for the inequalities are
the same in the condensed QP (8.7) and in the structured QP (8.6).
The condensed QP (8.7) is of much smaller size than QP (8.6) and its size does
not depend on the spatial discretization level. It still exhibits the typical multiple
shooting structure in the ODE states and could thus be condensed one more time.
In the examples which we present in Part III, however, the computational savings
are only marginal between skipping the second condensing and solving QP (8.7)
directly with the method we describe in Chapter 9.
120 8 Condensing

8.4 Newton-Picard Hessian approximation

We can efficiently evaluate the quantities that must be computed to set up the
condensed QP (8.7) of Theorem 8.6 by once again exploiting the Newton-Picard
structure in the approximations of the constraint Jacobian: The partial null-space
basis can be evaluated purely on the slow subspace because
 −1   
−C11 C12 −1
−(I ⊗ P)Ĉ11 (I ⊗ R)(I ⊗ P)Ĉ12
Z = =
I I
 −1

−(I ⊗ P)Ĉ11 Ĉ12
= .
I

This observation suggests a projected Newton-Picard approximation of the Hes-

sian matrix via
 
 (I ⊗ Πfast )T B11 (I ⊗ Πfast ) 0
B =
fast
,
0 0
 
(I ⊗ R)T B̂11 (I ⊗ R) (I ⊗ R)T B̂12
Bslow = ,
B̂21 (I ⊗ R) B̂22
B = Bfast + Bslow .

Consequently we have
ZT Bfast Z = 0
and thus we can also compute the condensed Newton-Picard Hessian matrix purely
on the slow subspace according to
 −1 
 T  −Ĉ11 Ĉ12
B = Z BZ = Ẑ B̂Ẑ with Ẑ =
T
.
I

The Hessian term Bfast only plays a role in the evaluation of Bc

  . Thus we only
1
need to evaluate one matrix vector product with the exact Hessian matrix for the
solution of the large structured Newton-Picard quadratic subproblem. In the two-
grid variant all remaining matrix vector products with the approximated Hessian
only require the coarse-grid operations.
Numerical experience on the application problems of Part III has shown that
a pure coarse grid Hessian approximation leads to a substantial loss of contrac-
tion for the Newton-Picard LISA-Newton method while the contraction with the
Newton-Picard Hessian approximation yields contractions which are almost as
good as when a pure fine grid Hessian is used.
8.5 Scaling invariance of the Newton-Picard LISA-Newton method 121

8.5 Scaling invariance of the Newton-Picard

LISA-Newton method
Based on Corollary 5.25 we know that if a preconditioner respects the transfor-
mation property of Lemma 5.24 we obtain affine invariance of the LISA-Newton
method. We now show that the Newton-Picard preconditioners partly satisfy the
transformation property. As a result we obtain invariance of the Newton-Picard
LISA-Newton method with respect to scaling. To be precise let α, β ∈ R, and

a 1 ∈ R n1 , a1i = α, i ∈ 1, . . . , n1 , a2 ∈ R|E2 | , a3 ∈ R|I | , a = (a1 , a2 , a3 ),

d 1 ∈ R n1 , di1 = β , i ∈ 1, . . . , n1 , d 2 ∈ Rn+2 , d = (d 1 , d 2 ),

and assume that no entry of a and d vanishes. Moreover we define

Ai = diag(ai ), i = 1, 2, 3, A = diag(a), Di = diag(d i ), i = 1, 2, D = diag(d).

We now consider the family of scaled NLPs of the form of NLP (8.2)

minimize f (β x1 , D2 x2 ) (8.10a)
x=(x1 ,x2 )∈Rn1 +n2

s. t. αgi (β x1 , D2 x2 ) = 0, i ∈ E1 , (8.10b)
ai gi (β x1 , D2 x2 ) = 0, i ∈ E2 , (8.10c)
ai gi (β x1 , D2 x2 ) ≥ 0, i∈I. (8.10d)

With y = (y1 , y2 , y3 ) ∈ Rn1 +|E2 |+|I | we obtain the scaled Lagrangian

L sc (x, y) = f (Dx) − ∑i∈E αyi gi (Dx) − ∑i∈E ai yi gi (Dx)

1 2 ∪I

and its gradient

∇x L sc (x, y) = D∇ f (Dx) − ∑i∈E αyi D∇gi (Dx) − ∑i∈E ai yi D∇gi (Dx),

1 2 ∪I

After introducing the scaled variables xsc = D−1 x and ysc = A−1 y we can establish
for the function F of Section 5.7.1 in Chapter 5 and its scaled counterpart that
 
D∇x L (x, y)
F sc (xsc , ysc ) = = diag(D, A)F(x, y),
Ag(Dx)
   
dF sc (xsc , ysc ) D 0 dF(x, y) D 0
=
d(xsc , ysc ) 0 A d(x, y) 0 A
122 8 Condensing

According to Lemma 5.24 we need to verify that the Newton-Picard approximation

satisfies the same transformation rule. Let Â1 = diag(α) and D̂1 = diag(β ) denote
the scaling matrices corresponding to A1 and D1 on the coarse grid. Then we can
immediately see that the transformation rule for the blocks of C in equations (8.4)
and for the two-grid Newton-Picard Hessian approximation of Section 8.4 satisfies
the assumption for Lemma 5.24 due to

(I ⊗ P)Â1 = αI ⊗ P = A1 (I ⊗ P) and D̂1 (I ⊗ R) = β I ⊗ R = (I ⊗ R)D1 .

Thus the Newton-Picard LISA-Newton method is scaling invariant.

9 A Parametric Active Set method for QP
solution
Most of this chapter is an excerpt form the technical report Potschka et al. [129]
which we include here for completeness. The chapter is dedicated to the numerical
solution of the convex QP

1 T
minimize x Bx + bT x s. t. cl ≤ Cx ≤ cu , (9.1)
x∈Rn 2

with symmetric Hessian matrix B ∈ Rn×n , constraint matrix C ∈ Rm×n , gradient

vector b ∈ Rn , and lower and upper constraint vectors cl , cu ∈ Rm . For most of
this chapter we furthermore assume B to be positive semidefinite. We describe the
generalization to nonconvex problems with indefinite B in Section 9.8.
The structure of this chapter is the following: We start with recalling the Para-
metric Quadratic Programming (PQP) method [17] in Section 9.2 and identify its
fundamental numerical challenges in Section 9.3. In Section 9.4 we develop strate-
gies to meet these challenges. It follows a short description of the newly developed
Matlab® code rpasm in Section 9.5 which we compare with other popular aca-
demic and commercial QP solvers in Section 9.6. We continue in Section 9.7 with
a description of drawbacks of the reliability improving strategies. In Section 9.8
we discuss an extension to compute local minima of nonconvex QPs.

9.1 General remarks on Quadratic Programming

Problems
Although we are mainly concerned with QPs which arise as subproblems of the
inexact SQP method described in Chapter 5, in particular after a condensing step
according to Chapter 8, the class of convex QP problems is important in its own
right. Gould and Toint [66] have been compiling a bibliography of currently 979
publications which comprises many application papers from disciplines as diverse
as portfolio analysis, structural analysis, VLSI design, discrete-time stabilization,
optimal and fuzzy control, finite impulse response design, optimal power flow,

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_9,
© Springer Fachmedien Wiesbaden 2014
124 9 A Parametric Active Set method for QP solution

economic dispatch, etc. Several benchmark and application problems are col-
lected in a repository [111] which is accessible through the CUTEr testing en-
vironment [68].

9.1.1 Optimality conditions

For the characterization of solutions of QP (9.1) we partition the index set m =
{1, . . . , m} into four disjoint sets
A e (x) = {i ∈ m | cli = (Cx)i = cui }, A l (x) = {i ∈ m | cli = (Cx)i < cui },
A u (x) = {i ∈ m | cli < (Cx)i = cui }, A f (x) = {i ∈ m | cli < (Cx)i < cui }
of equality, lower active, upper active, and free constraint indices, respectively. It
is well known (see Chapter 4) that for any solution x∗ of QP (9.1) there exists a
vector y∗ ∈ Rm of dual variables such that
Bx∗ + b −CT y∗ = 0, cl ≤ Cx∗ ≤ cu , (9.2a)
(Cx − c )i y∗i
∗ l
= 0, i ∈ A (x ),
l ∗
y∗i ≥ 0, ∗
i ∈ A (x ),
l
(9.2b)
(Cx∗ − cu )i y∗i = 0, i ∈ A (x ),
u ∗
y∗i ≤ 0, ∗
i ∈ A (x ).
u
(9.2c)
Conversely, every primal-dual pair (x∗ , y∗ ) which satisfies conditions (9.2) is a
global solution of QP (9.1) due to semidefiniteness of the Hessian matrix B. The
primal-dual solution is unique if and only if the following two conditions are sat-
isfied:
1. The active constraint rows Ci , i ∈ A e ∪ A l ∪ A u , are linearly independent.
2. Matrix B is positive definite on the null space of the active constraints.

9.1.2 Existing methods

All existing methods for solving QPs are iterative and can be grossly divided into
Active Set and Interior Point methods. Interior Point methods are sometimes called
Barrier methods due to the possibility of different interpretations of the resulting
subproblems, see, e.g., Nocedal and Wright [121]. As a defining feature, Ac-
tive Set methods keep a working set of active constraints and solve a sequence
of equality constrained QPs. The working set must be updated between the iter-
ates according to exchange rules which are based on conditions (9.2). In contrast,
Interior Point methods do not use a working set but follow a nonlinear path, the
so-called central path, from a strictly feasible point towards the solution.
Active Set methods can be divided into primal, dual, and parametric methods,
of which the primal Active Set method is the oldest and can be seen as a direct
9.1 General remarks on Quadratic Programming Problems 125

extension of the Simplex Algorithm [36]. Dual Active Set methods apply the
primal Active Set method to the dual of QP (9.1) (which exists if B is semidefinite).
A relatively new variant of Active Set methods are Parametric Active Set Methods
(PASM), e.g., the PQP method due to Best [17], which are the methods of interest
in this thesis. PASMs are based on an affine-linear homotopy between a QP with
known solution and the QP to be solved. It turns out that the optimal solutions
depend piecewise affine-linear on the homotopy parameter and that along each
affine-linear segment the active set is constant. The iterates of the method are
simply the start points of each segment.
The numerical behavior of Active Set and Interior Point methods is usually quite
different: While Active Set methods need on average substantially more iterations
than Interior Point methods, the numerical effort for one iteration is substantially
less for Active Set methods. Often one or the other method will perform favorably
on a certain problem instance, underlining that both approaches are important.
We want to concisely compare the main advantages of the different Active Set
versus Interior Point methods. One advantage of Interior Point methods is the
regularizing effect of the central path which leads to well-defined behavior on
problems with nonunique solutions due to, e.g., degeneracy or zero curvature in
a feasible direction at the solution. An advantage of all Active Set methods is
the possibility of hot starts which can give a substantial speed-up when solving
a sequence of related QPs because the active set between the solutions usually
changes only slightly. A unique advantage of PASM is that the so-called Phase 1
is not needed. The term Phase 1 describes the solution of an auxiliary problem to
find a feasible starting point for primal and dual Active Set methods or a strictly
feasible starting point for Interior Point methods. The generation of an appropriate
starting point with Phase 1 can be as expensive as the subsequent solution of the
actual problem.

9.1.3 Existing software

The popularity of primal/dual Active Set and Interior Point methods is reflected
in the large availability of free and commercial software products. A detailed list
and comparison of the various implementations is beyond the scope of this thesis.
We restrict ourselves to citing a few implementations which we consider popular
in Table 9.1. We further restrict our list to implementations which are specifically
designed for QPs, although any NLP solver should be able to solve QPs.
The packages GALAHAD and FICO(TM) Xpress also provide the possibility of
using crossover algorithms which start with an Interior Point method to eventually
refine the solution by few steps of an Active Set method. CPLEX additionally
126 9 A Parametric Active Set method for QP solution

Interior primal/dual Parametric

Code/Package
Point Active Set Active Set
BPMPD [114] +
BQPD [55] +
COPL QP [168] +
CPLEX [89] + +
CVXOPT [35] +
GALAHAD [69] + +
HOPDM [62] +
HSL [6] + +
IQP [21] +
LOQO [157] +
MOSEK [116] +
OOQP [58] +
qpOASES [52] +
QPOPT [60] +
QuadProg++ [46] +
quadprog [112] +
QuadProg [153] +
rpasm [129] +
Xpress Optim. Suite [54] + +

Table 9.1: Software for convex Quadratic Programming (in alphabetical order).

offers the option of a concurrent optimizer which starts a Barrier and an Active Set
method in parallel and returns the solution which was found in the shortest amount
of CPU time.

For Parametric Active Set methods we are only aware of the code qpOASES (see
Ferreau [52], Ferreau et al. [53]). We have developed a prototype Matlab® code
called rpasm to demonstrate the efficacy of the proposed techniques and strategies
to fortify reliability of PASM.
9.2 Parametric Active Set methods 127

9.2 Parametric Active Set methods

9.2.1 The Parametric QP
The idea behind Parametric Active Set methods consists of following the optimal
solutions on a homotopy path between two QP instances. Figuratively speaking,
the homotopy morphs a QP with known solution into the QP to be solved. Let this
homotopy be parametrized by τ ∈ [0, 1]. We then want to solve the one-parametric
family of τ-dependent QP problems

1
minimize x(τ)T Bx(τ) + b(τ)T x(τ) s. t. cl (τ) ≤ Cx(τ) ≤ cu (τ), (9.3)
x(τ)∈R n 2

with b, cl , and cu now being continuous functions b(τ), cl (τ), cu (τ). For fixed τ,
let the optimal primal-dual solution be denoted by z(τ) = (x(τ), y(τ)) which nec-
essarily satisfies (9.2) (with b = b(τ), cl = cl (τ), cu = cu (τ)). If we furthermore
restrict the homotopy to affine-linear functions b ∈ H n , cl , cu ∈ H m , where

H k = { f : [0, 1] → R | f (τ) = (1 − τ) f (0) + τ f (1), τ ∈ [0, 1]},

it turns out that the optimal solutions z(τ) depend piecewise linearly but not nec-
essarily continuously on τ (see Best [17]). On each linear segment the active set
is constant. Parametric Active Set algorithms follow z(τ) by jumping from one
beginning of a segment to the next. We can immediately observe that this ap-
proach allows hot-starts in a natural way. As mentioned already in Section 9.1.2,
no Phase 1 is needed to begin the method: We can always recede to the homotopy
start b(0) = 0, cl (0) = 0, cu (0) = 0, x(0) = 0, y(0) = 0, although this is certainly
not the best choice as we discuss in Section 9.3 and Section 9.4.

9.2.2 The Parametric Quadratic Programming algorithm

A Parametric Active Set method was described by Best [17] under the name Para-
metric Quadratic Programming (PQP) algorithm. Algorithm 2 displays the main
steps. The lines preceded by a number deserve further explanation.
Step 1: Computation of step direction. The working set W is encoded as an m-
vector with entries 0 or ±1, where the i-th component indicates whether constraint
i is inactive (Wi = 0), active at the lower bound (Wi = −1), or active at the upper
bound (Wi = +1). Let CW denote the matrix consisting of the rows Ci with Wi = 0
and let cW (τ) denote a vector which consists of entries cli (τ) or cui (τ) depending
128 9 A Parametric Active Set method for QP solution

Algorithm 2: The Parametric Quadratic Programming Algorithm.

Data: B,C, b(1), cl (1), cu (1), and z(0) = (x(0), y(0)) optimal for b(0), cl (0), cu (0)
with working set W ∈ {−1, 0, 1}m
Result: z(1) = (x(1), y(1)) or infeasible or unbounded
τ := 0;
1 Compute step direction Δz = (Δx, Δy) with current working set W ;
2 Determine maximum homotopy step Δτ;
if Δτ ≥ 1 − τ then return solution z(1) := z(τ) + (1 − τ)Δz;
Set τ + := τ + Δτ, z(τ + ) := z(τ) + ΔτΔz, and W + := W ;
if constraint l is blocking constraint then
Set Wl+ := ±1;
3 Linear independence test for new working set W + ;
if linear dependent then
4 Try to find exchange index k;
if not possible then return infeasible;
5 Adjust dual variables y(τ + );
Set Wk+ := 0;
end
else (sign change of k-th dual variable is blocking)
Set Wk+ := 0;
6 Test for curvature of B on new working set W + ;
if nonpositive curvature then
7 Try to find exchange index l;
if not possible then return unbounded;
8 Adjust primal variables x(τ + );
Set Wl+ := ±1;
end
end
Set τ := τ + and W := W + ;
9 Possibly update matrix decompositions;
Continue with Step 1;

on which (if any) bound is marked active in Wi . We can then determine the step
direction (Δx, Δy) by solving
      
Δx B T
CW Δx −(b(1) − b(τ))
KW (τ) := = . (9.4)
−ΔyW CW 0 −ΔyW cW (1) − cW (τ)

The dual step Δy must be assembled from ΔyW by filling in zeros at the entries of
constraints i which are not in the working set (i.e., Wi = 0). For the initial working
9.2 Parametric Active Set methods 129

set W we assume matrix CW to have full rank and matrix B to be positive definite on
the null space of CW . Thus matrix KW (0) is invertible. As we shall see in Steps 3
and 6, the PQP algorithm ensures the full rank and positive definiteness properties
and thus invertibility of KW (τ) for all further steps through exchange rules for the
working set W . We shall discuss a null space approach for the factorization of
KW (τ) in Step 9.
Step 2: Determination of step length. We can follow z(τ) in direction Δz along
the current segment until either an inactive constraint becomes active (blocking
constraint) or until the dual variable of a constraint in the working set becomes
zero (blocking dual variable). Following the straight line with direction Δz beyond
this point would lead to violation of conditions (9.2). The step length Δτ can be
determined by ratio tests

RT : Rm+m → R ∪ {∞}, RT(u, v) = min{ui /vi | i ∈ m, vi > 0}. (9.5)

If the set of ratios is empty the minimum yields ∞ by convention. With the help of
RT, the maximum step towards the first blocking constraint is given by

tp = min{RT(Cx(τ) − cl , −CΔx), RT(cu −Cx(τ),CΔx)}, (9.6)

and towards the first blocking dual variable by

td = RT(W ◦ y(τ),W ◦ Δy), (9.7)

where ◦ denotes elementwise multiplication to compensate for the opposite signs

of the dual variables for lower and upper active constraints. The maximum step
allowed is therefore
Δτ = min{tp ,td }.
Best [17] assumes that each occurring minimization yields either ∞ or a unique
minimizer with corresponding index l ∈ m if Δτ = tp or index k ∈ m if Δτ = td from
the sets of the ratio tests. The occurrence of at least one nonunique minimizer is
called a tie. We can distinguish between primal-dual ties if tp = td , primal ties if l
is not unique, and dual ties if k is not unique. In case of a tie it is not clear which
constraint should be added or removed from the working set W and bad choices
can lead to cycling or even stalling of the method. Thus successful treatment of
ties is paramount to the reliability of Parametric Active Set methods and shall be
further discussed in Section 9.4.3.
Step 3: Linear independence test. The addition of a new constraint l to the
working set W can lead to rank deficiency of CW + and thus loss of invertibility of
130 9 A Parametric Active Set method for QP solution

matrix KW (τ + ). The linear dependence of Cl on Ci , i : Wi = 0 can be verified by

solving      T
B CW T s Cl
= . (9.8)
CW 0 ξW 0
Only if s = 0 then Cl is linearly dependent on Ci , i : Wi = 0 (see Best [17]). The lin-
ear independence test can be evaluated cheaply by reusing the factorization needed
to solve the step equation (9.4).
Step 4: Determination of exchange index k. It holds that s = 0. Let ξ be
constructed from ξW like Δy from ΔyW . Equation (9.8) then yields

Cl = ∑ ξiCi . (9.9)
i:Wi =0

Multiplying equation (9.9) by λWl+ with λ ≥ 0 and adding this as a special form
of zero to the stationarity condition in equations (9.2) yields

B(x(τ) + ΔτΔx) − b(τ + ) = ∑i:W =0 yi (τ + )CiT

i
(9.10)
= −λWl+ClT + ∑i:W =0 (yi (τ + ) + λWl+ ξi )CiT .
i

Thus all coefficients of Ci , i : Wi+

= 0 on the right hand side of equation (9.10) are
also valid choices ỹ for the dual variables as long as they satisfy the sign condition
Wi+ ỹi ≤ 0. Hence we can compute the largest such λ with the ratio test

λ = RT(−Wl+ (W ◦ y(τ + )),Wl+ (W ◦ ξ )). (9.11)

If λ = ∞ then the parametric QP does not possess a feasible point beyond τ + and
thus the QP to be solved (at τ = 1) is infeasible. Otherwise, let k be a minimizing
index of the ratio set.
Step 5: Jump in dual variables. Now let
"
−λWi+ for i = l,
ỹi = +
+
yi (τ ) + λWi ξi for i : Wi = 0,

and set y(τ + ) := ỹ. It follows from construction of λ that ỹk = 0 and thus, con-
straint k can leave the working set. As a consequence, matrix CW + \{k} preserves
the full rank property and has the same null space as CW , thus securing regularity
of matrix KW + (τ + ).
Step 6: Curvature test. The removal of a constraint from the working set can
lead to exposure of directions of zero curvature on the null space of CW + (which
9.2 Parametric Active Set methods 131

is larger than the null space of CW ) leading to singularity of matrix KW + (τ + ).

Singularity can be detected by solving
 T
   
B CW s 0
= , (9.12)
CW 0 ξW −(ek )W

where ek is the k-th column of the m-by-m identity matrix. Only if ξ = 0 then B
is singular on the null space of CW + (see Best [17]). As for the linear indepen-
dence test of Step 3, the curvature test can be evaluated cheaply by reusing the
factorization needed to solve the step equation (9.4).
Step 7: Determination of exchange index l. It holds that ξ = 0 and s solves

Bs = 0, Ck s = −1, CW + s = 0. (9.13)

Therefore all points x̃ = x(τ + ) + σ s are also solutions if x̃ is feasible. We can

compute the largest such σ = min{σ l , σ u } with the ratio tests

σ l = RT(Cx(τ + ) − cl , −Cs), σ u = RT(cu −Cx(τ + ),Cs). (9.14)

If σ = ∞ then the parametric QP is unbounded beyond τ + , including the QP to

be solved (at τ = 1). Otherwise, let l be a minimizing index of a ratio set which
delivers a final minimizer of σ .
Step 8: Jump in primal variables. Now set x(τ + ) := x(τ + )+σ s. By construc-
tion of σ we have that either Cl x(τ + ) = cll (if σ = σ l ) or Cl x(τ + ) = cul (otherwise).
Thus l can be added to the working set via Wl+ := −1 (if σ = σ l ) or Wl+ := +1.
Step 9: Update matrix decompositions. We summarize a null space approach
for the solution of systems (9.4), (9.8), and (9.12). A range space approach is in
general not possible if B is only semidefinite (see, e.g., Nocedal and Wright [121]).
A direct factorization of KW (τ) via LDLT factorization is also possible but update
formulae are in the general case not as efficient as for the null space approach (see
Lauer [101]). The alternative of iterative linear algebra methods for the indefinite
matrix KW (τ) are beyond the scope of this thesis.
The null space approach is based on a QR decomposition of the transposed
active constraint matrix
 
  R
CW = QR̃ = Y Z
T
= Y R, QT Q = I.
0

Thus the columns of Z constitute an orthonormal basis of the null space of CW .

The columns of Y are an orthonormal basis of the range space of CW T and the

upper triangular matrix R is invertible due to the full rank assumption on CW . By

132 9 A Parametric Active Set method for QP solution

assumption, the projected Hessian Z T BZ is positive definite and lends itself to a

Cholesky decomposition
Z T BZ = LLT

with invertible triangular matrix L. Exploiting CW Z = 0 and CW Y = R, the unitary

basis transformation
⎛ T ⎞
 T  T
  Y BY Y T BZ R
Y Z 0 B CW Y Z 0
= ⎝Z T BY LLT 0⎠
0 0 I CW 0 0 0 I
RT 0 0

yields a block-triangular system which can be solved via backsubstitution. When

the working set W changes by addition, removal, or substitution of constraints,
the QR decomposition of CW + and following the Cholesky decomposition can
be updated cheaply from the previous decompositions (see Gill et al. [59]). For
exploitation of special structures of B and C in large scale applications we refer
the interested reader to Kirches et al. [97, 96].
This concludes our presentation of the Parametric Quadratic Programming al-
gorithm.

9.2.3 Far bounds

Many applications lead to QPs where some of the constraint bounds cli , cuj , i = j,
are infinite to allow for one-sided constraints, e.g., 0 ≤ xi ≤ ∞. However, a ho-
motopy from finite to infinite cl (τ) and cu (τ) is not possible. The flipping bounds
strategy to be described in Section 9.4.1 relies on finiteness of cl (τ) and cu (τ).
We circumvent this problem by application of a so-called far bounds strategy. It
is based on the following idea: Let M > 0 be given. If M is large enough then a
solution (x, y) of (9.1) is also a solution of

1 T
minimize x Bx + bT x s. t. c̃l ≤ Cx ≤ c̃u , (9.15)
x∈R n 2

where c̃li = max(cli , −M), c̃ui = min(cui , M), i = 1, . . . , m. We call a constraint bound
which attains the value ±M a far bound. Algorithmically we solve a sequence of
QPs with growing far bounds value M, see Algorithm 3. The total solution time
will mostly be dominated by the solution of the first QP as consecutive QPs of the
form (9.15) can be efficiently hot-started.
9.3 Fundamental numerical challenges 133

Algorithm 3: The far bounds strategy.

Initialize M = 103 ;
repeat
Solve QP (9.15);
if no far bounds active then return QP solution;
Grow far bounds: M := 103 M;
until M > 1020 ;
if last QP infeasible then return QP infeasible;
else return QP unbounded;

9.3 Fundamental numerical challenges

In this section we describe the numerical challenges that occur in the PQP algo-
rithm. We shall develop countermeasures in Section 9.4.
One fundamental challenge in many applications is ill-posedness of problems:
Small changes in the data of the problem lead to large changes in the solution. This
challenge necessarily propagates through the algorithm and leads to ill-conditioned
matrices KW . As a consequence the results of the step computation (9.4), the
linear independence test (9.8), and the curvature test (9.12) can be erroneous up
to cond(KW ) times machine precision in relative error (see, e.g., Wilkinson [165]).
This, in turn, can lead to very instable ratio tests (9.5) and wrong choices for the
working set which can cause the algorithm to break down.
Rounding errors can also accumulate over several iterations and lead to the para-
metric “solution” z(τ) being optimal with an accuracy much less than machine
precision. We call this phenomenon drift. Large drift can also lead to breakdown
of the algorithm because the general assumption of optimality of z(τ) is violated.
Furthermore, the termination criterion must be adapted to work reliably on both
well- and ill-conditioned problems.
Ill-conditioning can also be introduced if the null space of CW captures two
eigenvalues of B with high ratio, leading to ill-conditioning of the Cholesky factors
L. In the extreme case, the next step for the dual variables can be afflicted with a
large error, causing again instability in the ratio tests.
The second fundamental challenge is the occurrence of comparisons with zero,
a delicate subject in the presence of rounding errors. These comparisons permeate
the algorithm from the sign condition in the ratio tests (9.5) to the tests for linear
dependence (9.8) or zero curvature (9.12).
The third fundamental challenge is the treatment of ties, i.e., nonuniqueness of
minimizers of the ratio tests (9.5). Consider the case mentioned in Section 9.2.1
134 9 A Parametric Active Set method for QP solution

of a homotopy starting at x(0) = 0, y(0) = 0, b(0) = 0, cl (0) = 0, cu (0) = 0,W = 0.

Clearly (x(0), y(0)) is an optimal solution at τ = 0 regardless of the choice of B
and C. Note that for the PQP algorithm the choice of W = 0 is only possible if B is
positive definite. The first step direction will then point towards the unconstrained
minimizer of the objective. If more than one constraint is active in the solution
at τ = 1 then the primal ratio test (9.6) for determination of the step length yields
a (multiple) primal tie with Δτ = t p = 0. Of all possible ratio test minimizers,
one has to be chosen. One approach seems to be to employ pricing heuristics
from primal/dual Active Set methods but we prefer a different approach which
we discuss in Section 9.4.3. In the following iterations primal-dual ties can occur
while still Δτ = 0. Thus cycling, the repeated addition and removal of the same
constraints without any progress, can be possible which leads to stalling of the
method. We are not aware of any pricing strategy which can avoid the problem of
cycling. Ties also occur naturally in the case of degenerate QPs, where the optimal
primal or dual variables are not uniquely determined.

9.4 Strategies to meet numerical challenges

We propose to employ the following strategies for Parametric Active Set methods
to meet the fundamental numerical challenges described in Section 9.3.

9.4.1 Rounding errors and ill-conditioning

The most effective countermeasure against the challenges of ill-conditioning is to
iteratively improve the quality of the linear system solutions via
Iterative Refinement (see, e.g., Wilkinson [165]). We have already mentioned
that the relative error in the solution z of

KW z = d

can be as high as cond(KW ) times machine precision. Thus if cond(KW ) ≈ 1010

and we perform computations in double precision, the solution z can have as little
as six valid decimal digits. Iterative Refinement

z0 = 0, rk = KW zk − d, KW δ zk = rk , zk+1 = zk − δ zk

recovers a fixed number of extra valid digits in each iteration. In the previous
example, the iterate z2 has at least twelve valid decimal digits after only one extra
step of Iterative Refinement. It is worth noticing that compared to the original
9.4 Strategies to meet numerical challenges 135

“solution” z1 each iteration only needs to perform one additional matrix-vector-

multiplication with KW and one backwards solve with the decomposition of KW
described in Section 9.2.2. In exact arithmetic zk+1 = zk for all k ≥ 1.
Drift correction. A very effective strategy to avoid drift can be formulated if
the PQP algorithm is cast in a slightly different framework. After each iteration,
we rescale the homotopy parameter to τ = 0, thus interpreting the iterate z(τ + ) as
a new starting value z(0). This does not avoid drift yet but allows for modifications
to restore consistency of the starting point via
"
Ci x(0) if Wi = −1,
cli (0) :=
min{cli (0),Ci x(0)} otherwise,
"
Ci x(0) if Wi = +1,
cui (0) :=
max{cui (0),Ci x(0)} otherwise,
⎧
⎪
⎨max{0, yi (0)} if Wi = −1,
⎪
yi (0) := min{0, yi (0)} if Wi = +1,
⎪
⎪
⎩0, otherwise,
b(0) := Bx(0) −CT y(0),

for i ∈ m. This annihilates the effects of drift after every iteration, at the cost of
splitting up the single homotopy into a sequence of homotopies which are, how-
ever, very close to the remaining part of the original homotopy. In exact arithmetic
the proposed modification does not alter any value.
Termination Criterion. It is tempting to use the homotopy parameter τ in the
termination criterion as proposed in Algorithm 2. However, this choice renders the
termination criterion dependent on the choice of the homotopy start, an undesirable
property. Instead we propose to use the relative distance δ in the data space

Δ0 = (b(τ), cl (τ), cu (τ)), Δ1 = (b(1), cl (1), cu (1)),

 
s j = (Δ1j − Δ0j )/ max{1, Δ1j }, j = 1, . . . , n + m + m, δ = s∞ .

This choice also renders the termination criterion independent of the condition
number of KW (1). We observe that the termination criterion can give no guarantee
for the distance to the exact solution. Instead a backwards analysis result holds:
The computed solution is the exact solution to a problem which is as close as δ to
the one to be solved. The numerical results presented in Section 9.6 were obtained
with the tolerance δ ≤ δterm = 1e7 eps.
136 9 A Parametric Active Set method for QP solution

Ill-conditioning of the Cholesky factors L. To avoid ill-conditioning of the

Cholesky factors L we have developed the so-called flipping bounds strategy. Flip-
ping bounds is similar to taking long steps in the dual Simplex method (see Kostina
[98], Sager [137]), where one variable changes in the working set from upper to
lower bound immediately without becoming inactive in between, i.e., it flips. Flip-
ping is only possible if cl (1) and cu (1) have only finite entries, which is guaranteed
by the far bounds strategy described in Section 9.2.3. We modify the PQP algo-
rithm in the following way: If a constraint l was removed without requiring another
constraint k to enter the active set, we monitor the size of the smallest entry i of
the diagonal of L in Step 9. If 2i < δcurv we have detected a small eigenvalue in L
which corresponds to a small eigenvalue of B now uncovered through the removal
of constraint l. To avoid ill-conditioning of LLT , we introduce a jump in the QP
homotopy by requiring that the other bound of constraint l is moved such that it
becomes active immediately (hence the name flipping bounds) through setting

c̃ll (τ + ) := cul (τ + ),Wl+ = −1, if Wl = +1,

c̃ul (τ + ) := cll (τ + ),Wl+ = +1, if Wl = −1.

The entries j = l of c̃l and c̃u are set to the corresponding entries of cl and cu .
Consequently the Cholesky decomposition from the previous step stays valid for
the current projected Hessian.
The numerical results presented in Section 9.6 were obtained with the curvature
tolerance δcurv = 1e4 eps.

9.4.2 Comparison with zero

Ratio tests. In the ideal ratio test (9.5) we take a minimum over a subset of m quo-
tients with strictly positive denominator. The presence of round-off error makes it
necessary to substitute the ideal ratio test by an expression with adjustable toler-
ances, e.g.,

i = max(ui , εcut ),
ucut i ∈ m,
RTr (u, v, εcut , εden , εnum ) = min{ucut
i /vi | i ∈ m, vi ≥ εden , ucut
i ≥ εnum }.

We now explain the purpose of the three tolerances: The denominator tolerance
εden > 0 describes which small but positive values of vi should already be consid-
ered less than or equal to zero. They are consequently discarded as candidates for
the minimum.
The cutting tolerance εcut and the numerator tolerance εnum offer the freedom of
two different treatments for numerators close to zero. If εcut > εnum then negative
9.4 Strategies to meet numerical challenges 137

numerators are simply cut off at εcut before the quotients are taken, yielding that
the minimum is greater or equal to εcut /εden . For instance, we set εcut = 0 in
the ratio tests for determination of the step length (9.6) and (9.7). This choice is
motivated by the fact that in exact arithmetic ui ≥ 0 for all i ∈ m with vi > 0. Thus
only values ui which are negative due to round-off are manipulated and the step
length satisfies Δτ ≥ 0 also in finite precision arithmetic.
If εcut ≤ εnum then cutting does not have any effect. We have found it beneficial
for the reliability of PASM to set εnum = εden in the ratio tests (9.11) and (9.14) for
finding exchange indices.
The numerical results presented in Section 9.6 were obtained with the ratio test
tolerances εden = −εnum = 1e3 eps and εcut = 0 for step length determination and
εden = εnum = −εcut = 1e3 eps for the remaining ratio tests.
Linear independence and zero curvature test. After solution of systems (9.8)
and (9.12) for s and ξW we must compare the norm of s or ξ with zero. Let
ζ = (s, ξ ). We propose to use the relative conditions
s∞ ≤ εtest ζ ∞ for the linear dependence test and (9.16)
ξ ∞ ≤ εtest ζ ∞ for the zero curvature test. (9.17)
We remark that ζ ∞ = ξ ∞ if s = 0 and ζ ∞ = s∞ if ξ = 0. Thus we can
replace ζ ∞ in the code by ξ ∞ in test (9.16) and by s∞ in test (9.17). The
numerical results presented in Section 9.6 were obtained with εtest = 1e5 eps.

9.4.3 Cycling and ties

Once ties have occurred, their resolution is a costly affair because of the combi-
natorial nature of the decision which subset of the possible constraints should be
chosen to leave or enter the working set. This decision can be based on the solution
of yet another QP of larger size than the original problem (see Wang [160]) or on
heuristics similar to anti-cycling rules in the Simplex method.
We prefer a different approach instead. The idea behind the strategy we propose
for ties is simple: Instead of trying to treat ties, we try to avoid them in the first
place. The strategy is as simple as the idea and exploits the homotopy framework
of PASM. Let a homotopy start b(0), cl (0), cu (0) with optimal solution (x(0), y(0))
and working set W be given. Then for every triple of m-vectors r0 , r1 , r2 ≥ 0 the
primal-dual pair (x(0), ỹ(0)) with
⎧
⎪
⎨yi (0) + ri if Wi = −1,
⎪
ỹi (0) = yi (0) if Wi = 0, i ∈ m,
⎪
⎪
⎩y (0) − r if W = +1,
i i i
138 9 A Parametric Active Set method for QP solution

is an optimal solution to the homotopy start b̃(0), c̃l (0), c̃u (0), where for i ∈ m
"
cl (0), if Wi = −1,
c̃i (0) = i
l
ci (0) − ri , otherwise,
l 1
"
cu (0), if Wi = +1,
c̃ui (0) = i
ci (0) + ri , otherwise,
u 2

b̃(0) = −(Bx(0) −CT ỹ(0)).

In other words, if we move the inactive constraint bounds further away from Cx(0)
and the dual variables of the active constraints further away from zero, x(0) stays
feasible and b(0) can be adapted to restore optimality of (x(0), ỹ(0)) with the same
working set W . Recall that the ratio tests depend exactly on the residuals of the in-
active constraints and the dual variables of the active constraints. In our numerical
tests, the simple choice of

rij = (1 + (i − 1)/(m − 1))/2, j = 0, 1, 2, i ∈ m,

has proved to work reliably. Because of the shape of r j , we call this strategy
ramping. It is important to avoid two entries of r j to have the same value because
many QP problems exhibit special structures, e.g., variable bounds of the same
value for several variables which lead to primal ties if the homotopy starts with
the same value for each of these variables. Of course, the choice of linear ramping
is somewhat arbitrary and if a problem happens to have variable bounds in the
form of a ramp, ties are again possible. However, this kind of structure is far less
common than equal variable bounds.
We employ ramping in the starting point of the homotopy and also after an
iteration which resulted in a zero step Δτ = 0. Of course, this can lead to large
jumps in the problem homotopy and practically catapult the current b(0) := b̃(0)
further away from b(1). However, a PASM is capable of reducing even a large
distance in the data space to zero in one step, provided the active set is correct.
Thus the distance of the working set W to the active set of the solution is a more
appropriate measure of the progress of a PASM. By construction, the active set is
preserved by the ramping strategy.
We further want to remark that ties can never be completely avoided. For in-
stance in case of a QP whose solution lies in a degenerate corner, a tie must occur
in (at least) one iteration of a PASM. In the numerical examples we have treated
so far, the ramping strategy effectively deferred these ties to the final step, where a
tie is not a problem any more because the solution at the end of the last homotopy
9.5 The code rpasm: A PASM in Matlab® 139

segment is already one of infinitely many solutions of the QP to be solved and no

ties must be resolved in the solution.

9.5 The code rpasm: A PASM in Matlab®

We have implemented the strategies proposed in Section 9.4 in a Matlab® code
called rpasm. The main purpose of the code is to demonstrate reliability and so-
lution quality on the test set. In favor of code simplicity we have refrained from
employing special structure exploiting linear algebra routines which could further
enhance the runtime of the code. The three main features in the C++ PASM code
qpOASES (see Ferreau [52], Ferreau et al. [53]) for runtime improvement in the
linear algebra routines are special treatment of variable bounds, updates for QR de-
compositions, and appropriate updates for Cholesky decompositions. Of the three,
rpasm only performs QR updates. Variable bounds are simply treated as general
inequality constraints. Cholesky decompositions are computed from scratch after
a change in the active set. Another feature which is common in most commercial
QP solvers is the use of a preprocessing/presolve step to reduce the problem size
by eliminating fixed variables and dispensable constraints and possibly scaling the
data. We shall see that rpasm works reliably even without preprocessing.

9.6 Comparison with existing software

From the codes contained in Table 9.1 we use the ones which are freely available
for academic purposes and come with a Matlab® interface, i.e., CPLEX, OOQP,
qpOASES, plus the Matlab® solver quadprog and the newly developed rpasm.
The programs cover the range of Primal Active Set (CPLEXP, quadprog), Dual
Active Set (CPLEXD), Barrier/Interior Point (CPLEXB, OOQP), and Parametric
Active Set (qpOASES, rpasm). For rpasm, we further differentiate between a ver-
sion without iterative refinement (rpasm0) and with one possible step of iterative
refinement (rpasm1). All codes were used with their default settings on all prob-
lems.

9.6.1 Criteria for comparison

We compare the runtime and the quality of the solution. Runtime was measured as
the average runtime of three runs on one core of an Intel® Core™ i7 with 2.67 GHz
140 9 A Parametric Active Set method for QP solution

and 8 MB cache in Matlab® 7.6 under Linux 2.6 (64 bit). The quality of solutions
(x∗ , y∗ ) was measured using a residual ρ of conditions (9.2) defined via
 
ρstat = Bx∗ + b −CT y∗ ∞ ,
ρfeas = max(0, cl −Cx∗ ,Cx∗ − cu ),
 
 
ρcmpl
l
= max{(Cx∗ − cl )i y∗i  | y∗i ≥ +10 eps},
ρcmpl
u
= max{|(Cx∗ − cu )i y∗i | | y∗i ≤ −10 eps},
ρ = max(ρstat , ρfeas , ρcmpl
l
, ρcmpl
u
).

We visualize the results for problems from the Maros-Mészáros test set [111] with
at most n = 1000 variables and m = 1001 two-sided inequality constraints (not
counting variable bound constraints) in the performance graphs of Figures 9.1
and 9.2. The graphs display a time factor on the abscissa versus the percentage
of problems that each code was able to solve within the time factor times the run-
time of the fastest method for each problem. Roughly speaking, the graph of a
fast method is close to the left hand side of the diagram, the graph of a reliable
method is close to the top of the diagram. We remark that the results for rpasm
were obtained using only dense linear algebra routines.
There is a certain arbitrariness in the notion of a “solved problem” between the
different codes. We choose to consider a problem as solved if ρ is less than or
equal to a certain threshold. This approach is not unproblematic either: A not
tight enough termination threshold of a code can lead to premature termination
and the problem would be considered “not solved” by our criterion, although the
method might have been able to recover a better solution with more iterations.
This is especially an issue for Interior Point/Barrier methods. Thus the graphs in
Figures 9.1 and 9.2 show reliability of the methods only in connection with their
default settings. However, we are not aware of any simple procedure which would
lead to a fairer comparison. Figure 9.1 shows the results with a relatively loose
threshold of ρ ≤ 1e-2 and Figure 9.2 with a tighter threshold of ρ ≤ 1e-8.

9.6.2 Discussion of numerical results

We first discuss the results depicted in Figure 9.1 and continue with the differences
to the tighter residual tolerance in Figure 9.2.
From Figure 9.1 we see that the newly developed code rpasm with iterative
refinement is the only code which solves all of the problems to the prescribed
accuracy. The version of rpasm without iterative refinement fails on three prob-
lems (95 %). Furthermore, both versions of rpasm dominate quadprog both in
9.6 Comparison with existing software 141

100
rpasm0
90 rpasm1
quadprog
Percentage of problems solved

OOQP
80 qpOASES
CPLEXP
70 CPLEXD
CPLEXB
60
50
40
30
20
10
0
100 101 102 103 104 105
Time factor

Figure 9.1: Performance comparison with loose residual threshold ρ ≤ 1e-2.

runtime and the number of solved problems (62 %). The primal and dual versions
of CPLEX are the second most reliable with 96 % and 97 %. CPLEX solves no
problem in less than 1.3 s, not even the small examples which are solved in a few
milliseconds by rpasm. We suspect that this is due to a calling overhead in CPLEX,
e.g., for license checking. This is also one reason why OOQP is much faster than
the Barrier version of CPLEX, albeit they both solve roughly the same number of
problems (70 % and 73 %, respectively). Even though the code qpOASES is only
appropriate for QPs with positive definite Hessian, which make up only 27 % of
the considered problems, it still solves 44 % of the test problems. Additionally,
we want to stress that those problems solved by qpOASES were indeed solved
quickly.
Now we discuss the differences between Figure 9.2 and Figure 9.1, i.e., when
switching to a tighter residual tolerance of ρ ≤ 1e-8: The ratio of solved prob-
lems drops dramatically for the Interior Point/Barrier methods (CPLEXB: 29 %,
OOQP: 37 %). This is a known fact and the reason for the existence of crossover
methods which refine the results of Interior Point/Barrier methods with an Active
Set method. The code qpOASES still solves 44 % of the problems, which indi-
cates that the solutions that qpOASES yields are of high quality. Furthermore,
142 9 A Parametric Active Set method for QP solution

100
rpasm0
90 rpasm1
quadprog
Percentage of problems solved

OOQP
80 qpOASES
CPLEXP
70 CPLEXD
CPLEXB
60
50
40
30
20
10
0
100 101 102 103 104 105
Time factor

Figure 9.2: Performance comparison with tight residual threshold ρ ≤ 1e-8.

qpOASES is fast: It solves 36 % of the problems within 110 % of the time of the
fastest method for each of these problems. The number of problems solved by
quadprog decreases to 53 %. The primal and dual Active Set versions of CPLEX
solve 78 % of the problems. Only the code rpasm is able to solve more than 80 %
of the problems to a residual of ρ ≤ 1e-8 (rpasm0: 82 %, rpasm1: 84 %).
We can conclude that the strategies proposed in Section 9.4 indeed lead to a
reliable method for the solution of convex QPs.

9.7 Drawbacks of the proposed PASM

Although the method has proved to work successfully on the test set, the improve-
ment in reliability is achieved only at the price of breaking the pure homotopy
paradigm which complicates an otherwise straightforward proof of convergence
for the method: Drift correction, ramping, and the flipping bounds strategy lead to
jumps in the trajectories of b(τ), cl (τ), and cu (τ) and thus to a sequence of (pos-
sibly nonphysical) homotopies. Proving the nonexistence or possibility of cycles
caused by these strategies is future work.
9.8 Nonconvex Quadratic Programs 143

9.8 Nonconvex Quadratic Programs

The flipping bounds strategy presented in Section 9.4.1 can also be extended to
the case of nonconvex QPs with indefinite Hessian matrix B. When the Cholesky
factorization or update breaks down due to a negative diagonal entry, we also flip
instead of remove the constraint l. Hence the projected Hessian always stays pos-
itive definite. By the second order necessary optimality condition, the projected
Hessian in every isolated local minimum of the nonconvex QP is guaranteed to
be positive semi-definite. Conversely, if the projected Hessian is positive definite
and strict complementarity holds at τ = 1 we obtain a local minimum because the
second order sufficient condition is satisfied. No guarantees can be given in the
case of violation of strict complementarity.
Finding a global minimum of a nonconvex QP is known to be an NP-hard prob-
lem, even if the Hessian has only one single negative eigenvalue (see Murty [118]).
However, a local solution returned by the PASM can be refined by flipping all com-
binations of active bounds whose removal would lead to an indefinite projected
Hessian and restarting the PASM for each of these flipped Active Sets, revealing
again the combinatorial nature of finding the global solution of the nonconvex QP.
In the context of SQP with indefinite Hessian approximations (e.g., symmetric
rank one updates, the exact Hessian, etc.), a local solution of a nonconvex QP is
sufficient because the SQP method can only find local minima anyway.
For proof of concept we seek a local solution of the nonconvex problem

1 k−2 1 k=1
minimize ∑
2 i=1
(xk+i+1 − xk+i )2 − ∑ (xk−i + xk+i + αk−i+1 )2
2 i=1
(9.18a)

s. t. xk+i − xi+1 + xi = 0, i = 1, . . . , k − 1, (9.18b)

αi ≤ xi ≤ αi+1 , i = 1, . . . , k, (9.18c)
0.4(αi+2 − αi ) ≤ xk+i ≤ 0.6(αi+2 − αi ), i = 1, . . . , k − 1, (9.18d)

with given constants αi = 1 + 1.01i , i = 1, . . . , k − 1. We have adapted problem

(9.18) from problem class 3 by Gould [64] by switching the sign in front of the sec-
ond sum and the α terms in the objective. We start the computation with an initial
guess of x(0) = 0, y(0) = 0, set the lower bounds of equations (9.18c) and (9.18d)
active in the initial working set, and adjust the variables via ramping (see Sec-
tion 9.4.3). The changes of the working set are depicted in Figure 9.3 for k = 100
and therefore n = 199, m = 298. Row l of the depicted image corresponds to the
working set in iteration l and column j corresponds to the status of constraint j
in the working set when the iterations advance. The shades indicate constraints
which are inactive (gray), active at the lower bound (black), or active at the upper
144 9 A Parametric Active Set method for QP solution

bound (white). Thus direct transitions from black to white or vice versa along a
vertical line indicate flipping bounds. We can observe that the chosen initial work-
ing set is completely different to the final working set in the solution. Still the
number of iterations is less than two times the number of constraints which indi-
cates that the proposed method works efficiently on this instance of the nonconvex
problem (9.18).
In the solution which corresponds to Figure 9.3, n = 199 out of m constraints
are active and strict complementarity is satisfied. Thus we indeed have obtained a
local optimum.

50

100

150
Iteration

200

250

300

350

50 100 150 200 250

Constraint number

Figure 9.3: Active set changes for nonconvex problem (9.18), k = 100. Each line of the
image corresponds to the working set in one iteration. The colors indicate constraints which
are inactive (gray), active at the lower bound (black), or active at the upper bound (white).
Direct transitions from black to white or vice versa along a vertical line indicate flipping
bounds.
10 Automatic derivative generation

The inexact SQP method which we describe in Chapter 5 requires first and second
order derivatives of the problem functions. There are several ways how deriva-
tives can be provided. The first is to have them provided along with the problem-
dependent model functions by the user. This can be cumbersome for the user and
it is impossible for the program to check whether the derivatives are free of errors,
even though consistency tests evaluated in a few points can somewhat mitigate the
problem. These are, however, severe drawbacks.
A second way is the use of symbolic calculation of derivatives. Although in
principle possible by the use of symbolic computer algebra systems, the resulting
expressions for the derivatives can become too large to be evaluated efficiently.
A third way is the use of numerical schemes. Finite differences can be computed
efficiently but they inevitably involve cancellation and truncation errors. While the
cancellation errors can be circumvented by using a complex step derivative (see
Squire and Trapp [148]) two further drawbacks still remain: First, complex step
derivatives without cancellation are limited to first order derivatives. Second, the
evaluation of the gradient of a scalar-valued function of many variables cannot be
carried out efficiently.
The aim of this chapter is to recapitulate an efficient and automated way to
compute derivatives from a given computer code. This fourth way does not suffer
from the drawbacks of the previous three. The main principles are Algorithmic
Differentiation (AD) and Internal Numerical Differentiation (IND). We refer the
reader to Griewank [70] and Bock [22, 23], respectively.
The chapter is structured in four sections. In Section 10.1 we give a concise
survey about the idea behind AD and in Section 10.2 about the principle of IND.
We continue with the discussion of a subtle difficulty in the application of the IND
principle to implicit time-stepping methods with monitor strategy in Section 10.3
and conclude the chapter in Section 10.4 with a short note on the numerical effort
needed for the first and second order derivative generation needed in the inexact
SQP method for NLP (3.3) described in Chapter 5.

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_10,
© Springer Fachmedien Wiesbaden 2014
146 10 Automatic derivative generation

10.1 Algorithmic Differentiation

Every computer code that approximates a mathematical function performs the cal-
culation by concatenating a possibly large number of evaluations of a few ele-
mental operations like +, −, ∗, /, sin, exp, etc., yielding an evaluation graph with
intermediate results as vertices and elemental operations as edges. The principle of
AD is to apply the chain rule to the concatenation of elemental operations. This is
possible because the elemental operations are (at least locally) smooth functions.
There are two main ways how AD can be applied to compute derivatives of a
function
F : Rnind → Rndep
to machine precision.
Forward mode. We traverse the evaluation graph from the independent input
variables towards the dependent output variables. The numerical effort of
the forward mode to compute a directional derivative at x ∈ Rnind in the di-
rection of s ∈ Rnind
∇F(x)T s
is only a small multiple of the evaluation of F(x).

Backward mode. We traverse the evaluation graph backwards from the depen-
dent variables to the independent variables while accumulating the deriva-
tive information. The numerical effort of the backward mode to compute an
adjoint directional derivative at x ∈ Rnind in the direction of s ∈ Rndep

∇F(x)s

is also only a small multiple of the evaluation of F(x). For the backward
mode, however, the function F(x) has to be evaluated in advance and all in-
termediate results must be accessible when traversing backwards through the
evaluation graph. This is usually accomplished by storing all intermediate
results to a contiguous memory block, the so called tape, or by a checkpoint-
ing strategy which stores only a few intermediate results and recomputes the
remaining ones on the fly. Both approaches have their value depending on
the ratio of computation speed and access time to the memory hierarchy on
a particular computer architecture.
The elemental operations can be formally generalized to operate on truncated
Taylor series. This approach makes the evaluation of arbitrary-order derivatives
possible in a unified framework. To circumvent the drawback of high memory
10.2 The principle of IND 147

requirements and irregular memory access patterns, Griewank et al. [74] suggest
to use only univariate truncated Taylor series from which mixed derivatives can be
obtained by an interpolation procedure. Univariate Taylor coefficient propagation
can also be used in forward and backward mode.

10.2 The principle of IND

For the solution of the NLP (3.3) we also need the derivatives of the state trajecto-
ries ui and vi with respect to initial values and controls on each multiple shooting
interval which we denoted by the G-matrices and H-matrices in Chapter 8. For nu-
merical efficiency reasons we compute the values of ui and vi with adaptive control
of accuracy. Two problems arise for the computation of derivatives in this case:
First, if we consider the differential equation solver as a black box and employ
finite differences or AD to the solver as a whole we inevitably also differenti-
ate the adaptive components of the solver. This approach of External Numerical
Differentiation (END) yields derivatives which are not consistent, i.e., in general
they do not converge to the derivative of the exact solution when we increase the
demanded accuracy. Even worse, the END-derivative is polluted with large errors
if the adaptive components are not differentiable. This is rather the common case
than the exception, e.g., if the adaptive components use conditional statements.
Second, we could try a differentiate-then-discretize approach: The derivative
of the exact solution is given by a Variational Differential Equation (VDE) which
exists in a forward and adjoint form (see, e.g., Hairer et al. [78], Hartman [79]). If
only forward derivatives are needed then we can use a solver for the combined sys-
tem of nominal and variational differential equations to obtain derivatives which
are also consistent on a discretized level. However, if we apply a solver to the
adjoint VDE we in general obtain a different discretization scheme due to adaptive
error control. Thus the derivatives are not consistent on the discrete level which can
severely impede the local convergence of the superordinate inexact SQP method.
IND solves these two problems. The principle of IND states:
1. The derivatives of an adaptive numerical procedure must be computed from
the numerical scheme with all adaptive components kept constant (frozen).
2. The numerical scheme must be convergent for the nominal value and the
derivative.
IND can be applied directly on the discrete level, e.g., by performing AD subject to
skipping the differentiation of adaptive components, or indirectly, e.g., by choos-
ing the same discretization scheme for the original and the variational differential
equation.
148 10 Automatic derivative generation

10.3 IND for implicit time-stepping with monitor

strategy
In this section we focus on a prototypical example which demonstrates the subtle
issue of stability of the scheme for the VDE with implicit time-stepping methods.
We restrict our presentation to the Backward Euler method although the results
transfer to other implicit methods like Backward Differentiation Formula (BDF)
methods with IND as described by Bauer et al. [13, 12], Bauer [11], Albersmeyer
and Bock [3], Albersmeyer [2].
Example 5. Let us consider the linear heat equation on Ω = (0, π) with homoge-
neous Dirichlet boundary conditions

∂t u = Δu in (0, 1) × Ω,
u=0 on (0, 1) × ∂ Ω,

u t=0
=u . 0

We discretize the problem with the FDM in space on the equidistant grid

x j = jh, j = 0, . . . , N, h = π/N,

and obtain the linear IVP

u̇(t) = Au(t), u(0) = u0 , (10.1)

where the matrix A is given by

⎛ ⎞
−2 1
⎜ .. .. ⎟
1 ⎜ 1 . . ⎟
A= 2⎜ ⎟ ∈ R(N−1)×(N−1) .
h ⎜⎝ .. .. ⎟
. . 1⎠
1 −2

To satisfy the boundary conditions the values in the nodes x0 and xN are implicitly
set to zero and are not a part of the discretized vector u(t).

Lemma 10.1. For k = 1, . . . , N − 1 define the pair (vk , λk ) ∈ RN−1 × R by

vkj = sin( jkh), λk = 2h−2 (cos(kh) − 1).

Then (vk , λk ) is an eigenpair of A.

10.3 IND for implicit time-stepping with monitor strategy 149

Proof. Because sin(0kh) = 0 and sin(Nkh) = sin(kπ) = 0 we obtain

h2 (Avk ) j = sin( jkh − kh) + sin( jkh + kh) − 2 sin( jkh)

= 2 sin( jkh) cos(kh) − 2 sin( jkh)
= 2(cos(kh) − 1)vkj ,

for j = 1, . . . , N − 1. This proves the assertion.

We see that the eigenvalue of smallest modulus is
∞
(−1)i 2i
λ1 = 2h−2 (cos(h) − 1) = 2h−2 ∑ h = −1 + O(h2 ),
i=1 (2i)!

and that the eigenvalue of largest modulus tends towards minus infinity

2 2 4N 2
λN−1 = N (cos(π(N − 1)/N) − 1) ≈ − 2 .
π 2 π
Thus ODE (10.1) is stiff and becomes stiffer for finer spatial discretizations.
Because the vectors vk are N − 1 eigenvectors to pairwise different eigenvalues
λ they form a basis of RN−1 . If we rewrite ODE (10.1) in the basis spanned by
k

{vk } we obtain the decoupled ODE of Dahlquist type

u˙ (t) = diag(λ1 , . . . , λN−1 )

 u(t). (10.2)

Consider now a Backward Euler method for ODE (10.2). Starting from  u0 at
t0 u } such that the value 
= 0 we compute a sequence {n u at t = t
n n n−1 + Δt solves
n

approximately

0 = (I − Δt n diag(λ1 , . . . , λN−1 )) un−1 =: M n

un −  un − 
un−1 . (10.3)

Although we could solve equation (10.3) exactly without much effort, this is not
the case for nonlinear ODEs. Efficient numerical integrators employ a Newton-
type method for the efficient solution of the nonlinear counterpart of equation
(10.3). The so called monitor strategy is a Simplified Newton method for equa-
tion (10.3) where the decomposition of the iteration matrix M k of a previous step
is utilized and the contraction of the iteration is monitored. Usually we update
the iteration matrix if the error has not been reduced satisfactorily within three
Newton-type iterations. For simplification of presentation we assume that we per-
form enough iterations to reduce the error to machine precision. In the case of
ODE (10.2) the iteration boils down to

un,i = 
 un,i−1 − (M k )−1 (M n
un,i−1 − 
un−1 ).
150 10 Automatic derivative generation

Thus we obtain for the j-th component of 

un,i the expression
 
1 − Δt n λ j 
un−1
j
un,i
 = 1 − 
u n,i−1
+ .
j
1 − Δt k λ j j
1 − Δt k λ j

un,0
This iteration is unstable if and only if there exists an index j with  j = 0 and
such that  
 
1 − (1 − Δt n λ j )/(1 − Δt k λ j ) > 1,
or, equivalently,
1 − Δt n λ j 1
>2 ⇔ 1 − Δt n λ j > 2 − 2Δt k λ j ⇔ Δt n >   + 2Δt k . (10.4)
1 − Δt k λ j λj

A numerically optimal step size controller for stiff systems must maximize the step
length such that the method is always on the verge of becoming unstable. We see
from condition (10.4) that the step size can be more aggressively increased if the
initial value does not contain modes which belong to eigenvalues of large modulus.
The following problem now becomes apparent: Assume that the initial value is
a linear combination of only the first p  N (low-frequency) modes, i.e.,
p
u0 = ∑ u0j v j .
j=1

Assume further that the monitor strategy keeps the iteration matrix at M 0 for the
first steps and that the step size controller chooses
1 1
Δt j =   + 2Δt 1 <   + 2Δt 1 , j = 2, . . . , n,
λ p+1  λ p 

which yields a stable scheme. Say we want to compute the derivative of u(t n )
with respect to u0 in direction d by using the chosen scheme on the VDE for
ODE (10.1), which is again ODE (10.1) with initial value d because ODE (10.1)
is linear. If d has nonzero components in the directions of vk , k = p + 2, . . . , N − 1,
then this computation is not stable and thus not convergent. Hence this approach
does not satisfy the IND principle.
One possible solution to this problem is to perform adaptive error control on the
nominal values simultaneously with the forward derivative values (see, e.g., Al-
bersmeyer [2, Section 6.7.5]). However, computing the gradient of the Lagrangian
with forward sweeps is prohibitively expensive for large nind . Thus this approach
is computationally not feasible for the problems considered in this thesis.
10.4 Numerical effort of IND 151

Our pragmatic approach, which has worked reliably for the examples in Part III,
is to tighten the required tolerance for the Simplified Newton method that approxi-
mates the solution of the implicit system (equation (10.3) in our previous example)
by a factor of 1000. This leads to more frequent reevaluation of the iteration matrix
within the monitor strategy and enlarges thus the numerical stability regions. We
have not yet performed a thorough comparison of the extra numerical effort of this
approach which seems to amount to roughly 20%.

10.4 Numerical effort of IND

We have computed all numerical applications in Part III with the adaptive BDF
method with IND derivative generation described by Albersmeyer [2]. The meth-
ods that he has developed enable us to evaluate matrix vector products of the form
 2  
∇xx L (xk , yk ) −∇g(xk ) v1
J(xk , yk )v =
∇g(xk )T 0 v2

occurring in Chapter 5 in only a small multiple of the numerical effort spent for
evaluation of F(zk ). Even though the upper left block contains second derivatives
of ui and vi they need only be evaluated in one adjoint direction given by the current
Lagrange multipliers in yk and the forward direction given by v1 .
11 The software package MUSCOP
We have implemented the inexact SQP method based on the GINKO algorithm
with two-grid Newton-Picard generalized LISA as described in Chapter 5. It is our
goal in this chapter to highlight the most important software design decisions and
features. We have named the software package MUSCOP, which is an acronym
for Multiple Shooting Code for PDEs. The name alludes to the successful software
package MUSCOD-II (see Leineweber [105], Leineweber et al. [107] with exten-
sions by Leineweber [106], Diehl [47], Schäfer [141], Sager [138]) because we
had originally intended to design it as a MUSCOD-II extension. In Section 11.1
we discuss why we have decided to develop the method in a stand-alone parallel
form and outline which programming paradigms have proved to be useful in the
development of MUSCOP. Afterwards we explain the orchestration of the different
software components in Section 11.2.

11.1 Programming paradigms

The programming paradigms of a software package should be chosen to support
the main goals and target groups of the project. We identify two equally important
target groups for MUSCOP: Practitioners and algorithm developers. Both groups
have different perspectives on the goals of MUSCOP. In our opinion the main goals
are:
1. Hassle-free setup of new application problems
2. Quick, accurate, and reliable solution of optimization problems
3. Fast development of algorithmical extensions
While goal (2) is of equal importance to both practitioners and developers, goal (1)
will be more important than goal (3) for a practitioner and vice versa for a devel-
oper. A user of MUSCOP is in most real-life cases partly practitioner and devel-
oper at the same time.

11.1.1 Hybrid language programming

Quick problem solution and fast development of extensions sometimes are dia-
metrically opposed goals: On the one hand the fastest runtimes might be achieved

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_11,
© Springer Fachmedien Wiesbaden 2014
154 11 The software package MUSCOP

by only writing assembler machine code for a specific computer architecture, but
such a code might soon become too complex and surpass a developer’s capacity to
maintain or extend it in a reasonable amount of time. On the other hand, the sole
use of a high-level numerical programming language like Matlab® or GNU Oc-
tave might result in a considerable loss of performance, especially if algorithms
are not easily vectorizable, while the development time of the code and its ex-
tensions might be dramatically reduced, mainly because debugging problems on
a numerical level is possible in a more accessible way by the use of well-tested
built-in methods like cond, eig, svd, etc., and data visualization.
We use hybrid language programming in the following way: All time-critical
algorithmic components should be implemented in lower-level programming lan-
guages and all other components in higher-level programming languages. This
concept is not new, GNU Octave being one example because it is written in C++
while most dense linear algebra components are based on an optimized BLAS and
LAPACK implementation called ATLAS (see Whaley et al. [163]).
In the case of MUSCOP the most time-critical component is the evaluation of
the shooting trajectories and their derivatives of first and second order (see Part III).
This task is performed by the software package SolvIND (see Albersmeyer and
Kirches [5]) which is entirely written in C++ and uses ATLAS, UMFPACK (see
Davis [39]), and ADOL-C (see Griewank et al. [72, 73], Walther et al. [159]). Most
of the remaining code is written in Matlab® /GNU Octave except for the interface
to SolvIND which is at the time of writing only available for GNU Octave and
not for Matlab® . This approach has proven to be beneficial for the development
speed of MUSCOP while only minor performance penalties have to be accepted
(see Part III).
The GNU Octave and C++ components of MUSCOP are separated on the left
hand and right hand side of Figure 11.1, respectively. We shall give a more detailed
explanation of Figure 11.1 in Section 11.2.

11.1.2 No data encapsulation

Figure 11.1 already indicates that the different software components of MUSCOP
are heavily interwoven. This is not a result of poor design of programming blocks
(or classes if you will). It is rather the inevitable consequence of intelligent struc-
ture exploitation in the MUSCOP algorithm. The efficiency of MUSCOP lies in
the reuse of intermediate data of one logic program block in another one, which is a
major efficiency principle not only but particularly in complex numerical methods.
Take for instance the use of the Newton-Picard Hessian approximation in Chap-
ter 8, Section 8.4. It can only be assembled after half of the condensing steps,
11.1 Programming paradigms 155

namely the computation of Ẑ, has been performed. Only then can we evaluate the
partially projected (coarse grid) Hessian B = Ẑ T B̂Ẑ.
We do not want to suggest that a functional encapsulation in logical blocks like
function evaluation, condensing, QP solution, etc. is impedimental. We believe,
however, that the encapsulation of the data of these blocks is. In our opinion the
structure of the code should indeed follow the logical structure of a mathemati-
cal exposition of the method but the exchange of data should not be artificially
obstructed by interfaces which follow the functional blocks.
Object Oriented Programming (OOP) raises the rigid coupling of function and
data interfaces (methods and private members in the language of OOP) to a design
principle. We believe that OOP is a valid approach for software which is supposed
to be used in a black-box fashion but we believe it to be more obstructive than help-
ful for the structure exploiting numerical methods we develop in this thesis. This is
the main reason why MUSCOP is not OOP and not an extension of MUSCOD-II.
In MUSCOP we use a global, hierarchical data structure which is accessible
in all software components, at least on the Matlab® /GNU Octave level. Hierar-
chical here means that the data structure consists of substructures which map the
functional blocks to data blocks without hiding them. The biggest disadvantage
of global variables is if course that users and developers have to know which vari-
ables they are allowed to write access and in what states the variables are when
performing a read access. This task is without doubt a difficult one to accomplish.
But the difficulty really stems from the complexity of the numerical method and
not from the choice of computer language or programming paradigm. No pro-
gramming paradigm can turn a complex and difficult-to-understand method into a
simple and easy-to-understand code.

11.1.3 Algorithm centered not model centered

Another distinguishing design decision is that MUSCOP is centered around the
GINKO Algorithm 1 of Chapter 5, in contrast to MUSCOD-II which is centered
around the OCP model. This enables us to use the GINKO algorithm in MUS-
COP also as a stand-alone inexact SQP method or LISA-Newton method. We
shall describe in Section 11.2 how MUSCOP orchestrates the different software
components around GINKO.

11.1.4 Reverse Communication interface

Reverse Communication seemed to be an antiquated method of interface design
until it regained acceptance in the mid 90’s within the linear algebra community
156 11 The software package MUSCOP

#  $%%

)  
# 
 &
 (   '(

  
  !
$

&#$ 

   !


 

 &* !"


&(  


+  

$ 

Figure 11.1: Schematic of the MUSCOP software architecture.

(see, e.g., Dongarra et al. [49], Lehoucq et al. [104]). Its most compelling features
are simplicity and flexibility, especially when multiple programming languages are
in use.
A program with Reverse Communication interface is called with an incomplete
set of input data first. If more input data is needed the program returns to the caller
indicating which input data is needed next. After computation of this data the user
calls the program again passing the new input data. This procedure is iterated until
the program signals termination to the user.
A typical example is an iterative linear algebra solver which returns to ask the
user for a matrix vector multiplication or a preconditioner vector multiplication.
Obviously the solver does not need to know the matrix or the preconditioner, nor
does it need to pose restrictions on how they are represented.
GINKO also uses Reverse Communication. When called without input param-
eters GINKO initializes a data structure which is then modified by the user and
passed back to GINKO. In this data structure there are two state flags, the flow
control and the action flag. The flow control flag tells GINKO which part of the
code is the next to be evaluated and the action flag tells the user on return which
variables in the data structure must be freshly computed before GINKO can be
11.2 Orchestration of software components 157

called again. As a side note we want to remark that Reverse Communication is

strongly coupled with a Finite State Machine programming paradigm.
The main advantage of Reverse Communication lies in the fact that GINKO
does not pose any requirements on the form of function representations which al-
lows for great flexibility and easy extensibility of the method. The disadvantage
is that the developer has the responsibility to provide input data in a manner con-
sistent with the method. But this is not a problem of programming but rather a
problem of the numerical computing: The developer must know what he or she is
doing (at least to a large extent).

11.2 Orchestration of software components

We now turn to the explanation of Figure 11.1 which depicts a schematic overview
of the software components of MUSCOP and how they interact. As mentioned
earlier the figure is divided into four areas: The lower area is MUSCOP code
written in GNU Octave on the left hand side (white background) and in C++ on
the right hand side (light gray background). The upper area depicts the user code
written in GNU Octave on the left (light gray background) and C++ code on the
right (dark gray background).
The two dashed boxes symbolize conceptual entities which do not necessarily
have one block of code but are rather a placeholder to signify special structure
in the data that flows along the connecting edges of the diagram. The Spatial
discretization box is located over the border of the GNU Octave/C++ regions to
indicate that the code for spatial discretization can be in either language (or even
both).

11.2.1 Function and derivative evaluation

The model functions, in particular the discretized ODE and PDE right hand side
functions f ODE(l) and f PDE(l) , need to be evaluated many times and thus they are
programmed in C++. We evaluate them via SolvIND either directly or via the IND
integrator DAESOL-II (see Albersmeyer and Bock [3], Albersmeyer [2]).
SolvIND uses ADOL-C (see Walther et al. [159]) to automatically obtain first
and second order derivatives of the model functions in forward and backward
mode.
In MUSCOP we also take special care to extend the principle of IND to the
function evaluations within the GINKO algorithm: When we evaluate the simpli-
fied Newton step in the monotonicity test we must freeze the integration scheme
158 11 The software package MUSCOP

to obtain monotonicity on the discretized level. This feature avoids numerical

pollution of the monotonicity test by effects of the adaptive error control of the
integrator. When a step is accepted, we recompute the function values with a new
adaptive scheme.
We parallelize the calls to the integrator on the Multiple Shooting structure so
that the numerically most expensive part, the solution and differentiation of the
local IVPs, can be evaluated concurrently. For parallelization we use the toolbox
MPITB (see Fernández et al. [51]) which provides access to the message pass-
ing interface MPI (see, e.g., Gropp et al. [75]) from Matlab® /GNU Octave. Our
manager-worker approach allows for parallel processing both on cluster computers
and multi-core workstations. An advantage of computation on cluster computers
is that the integration AD tapes can be stored locally and do not need to be ex-
changed between the cluster nodes, yielding an efficient memory parallelization
without communication overhead.
The functions which need to be provided in GNU Octave code are loading of
model libraries via SolvIND, grid prolongation and restriction, evaluation of vari-
able norms suitable for the PDE discretization, visualization, and preparation of
an initial solution guess.

11.2.2 Condensing and condensed QP solution

We carry out the solution of the large-scale QP subproblems via condensing and
a PASM for the resulting small-scale to medium-scale QP (see Chapter 8). These
QPs need to be solved within the generalized LISA (see Chapter 5). For the solu-
tion of the first QP in a major GINKO iteration, i.e., k loop, we need to compute the
coarse grid Hessian and constraint Jacobian matrices. There is no need to reeval-
uate them for the following QP solutions until we increment k. We do need to
reevaluate the fine grid Lagrange gradient and constraint residuals, plus one fine
grid Hessian-vector product for the condensing procedure for each QP subprob-
lem.
The QP solver rpasm (see Chapter 9) allows for efficient hot-starting of the
first QP of a major GINKO iteration by reusing the working set of the previous
iteration. MUSCOP then freezes the working set of the first QP for all following
QPs until we increment k to commence the next major iteration.
If the working set of the previous iteration leads to a projected Hessian matrix
which is not positive definite at the current iterate then we need to resort to a safe
cold start of the PASM with trivial 0-by-0 projected Hessian.
11.2 Orchestration of software components 159

11.2.3 Estimation of κ and ω

MUSCOP currently estimates κ in two ways: If the coarse grid and the fine grid are
identical then GINKO is explicitly run with the information that κ = 0. This allows
for performing only one step of LISA instead of the minimum of two steps required
for an a-posteriori estimate of κ. If κ = 0 then one step of LISA already delivers
the exact solution of the linearized system. If the fine grid does not coincide with
the coarse grid we employ the Ritz κ-estimator.
The nonlinearity constant ω is implicitly estimated in the inexact Simplified
Newton step via
   
  2(1 − ρ̄i+1 ) δ zk+1 k+1 
i+1 − δ z0
ω δ zk  ≈ [hδk ]∗ = . (11.1)
αk2 δ zk 

We observe that the right hand side of equation (11.1) is afflicted with a cancella-
tion error in the norm of the numerator. This error is then amplified by αk−2 causing
that if the step size αk drops below, say, 10−4 then [hδk ]∗ might be overestimated.
This in turn leads to even smaller step sizes
1
αk = .
(1 + ρ)[hδk ]∗

Thus GINKO gradually reduces αk to zero and the method stalls. We have imple-
mented an estimator for the cancellation error. The cancellation error is displayed
for each iteration of MUSCOP but does not influence the iterations of MUSCOP.
It rather serves as an indicator for failure analysis.

11.2.4 Automatic mesh refinement

In general the refinement is performed within the following framework: The user
provides a conforming hierarchy of grids. MUSCOP starts with both coarse  and
 
fine grids on level l = 0. If the inexact Simplified Newton increment δ zk  is
smaller than a given threshold then we either terminate if level l is already the
finest level or we increment l and use the prolongation of the current iterate as a
starting guess for NLP (4.1) on the next level.
The variable steps on the following grid level can be used as a rough a-posteriori
error estimation for the discretization error.
If in the course of computation GINKO signals that M k needs to be improved
because κ is too large then we automatically refine the coarse grid until κ is small
enough again.
160 11 The software package MUSCOP

As mentioned earlier it will surely be advantageous to perform adaptive a-

posteriori mesh refinement independently for the fine and the coarse grid and
seperately for each shooting interval. This aspect, however, is beyond the scope of
this thesis.
 Part III

Applications and numerical results

12 Linear boundary control for the periodic
2D heat equation
We presents numerical results for the model problem (6.1) in this chapter. The
computations have been published in Potschka et al. [131] and are given here for
completeness.
This chapter is structured as follows: In Section 12.1 we list the problem pa-
rameters for the computations. Afterwards we discuss the effects of the Euclidean
and the L2 projector in the projective approximation of the Jacobian blocks in
Section 12.2. In Section 12.3 we present numerical evidence for the mesh inde-
pendence result of Theorem 6.7. We conclude this chapter with a comparison of
the symmetric indefinite Newton-Picard preconditioners with a symmetric posi-
tive definite Schur complement preconditioner in a Krylov method setting in Sec-
tion 12.4.

12.1 General parameters

The calculations were performed on Ω = [−1, 1]2 . We varied the diffusion coeffi-
cient D ∈ {0.1, 0.01, 0.001} which results in problems with almost only fast modes
for D = 0.1 and problems with more slow modes in the case of D = 0.001. The
functions α and β were chosen identically to be a multiple of the characteristic
function of the subset
Γ = Γ1 ∪ Γ2 := ({1} × [−1, 1]) ∪ ([−1, 1] × {1}) ⊂ ∂ Ω,
with α = β = 100χΓ . Throughout, we used the two boundary control functions
ψ̃1 (x) = χΓ1 (x), ψ̃2 (x) = χΓ2 (x).
In other words, the two controls act each uniformly on one edge Γi of the domain.
With γ = 0.001, we chose the regularization parameter rather small such that
the objective function is dominated by the tracking term which penalizes devia-
tion of the state at the end of the period from the desired state û. We used the
discontinuous target function
 
û(x) = 1 + χ[0,1]×[−1,0] (x) /2.

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_12,
© Springer Fachmedien Wiesbaden 2014
164 12 Linear boundary control for the periodic 2D heat equation

1
Eigenvalue μi

0.5

0
0 50 100 150 200
Index i

Figure 12.1: The eigenvalues μi of the spectrum of the monodromy matrix Gu decay expo-
nentially fast. Only few eigenvalues are greater than 0.5. Shown are the first 200 eigenvalues
calculated with D = 0.01 and β = 100χ(Γ) on a grid of 8-by-8 elements of order 5.

The controls were discretized in time on an equidistant grid of m = 100 intervals.

For the discretization of the initial state u(0) we used quadrilateral high-order
nodal Finite Elements. The reference element nodes are the Cartesian product of
the Gauss-Lobatto nodes on the 1D reference element. We used part of the code
which comes with the book of Hesthaven and Warburton [84], and extended the
code with continuous elements in addition to discontinuous elements.
The evaluations of matrix-vector products with Gu and Gq were obtained from
the Numerical Differentiation Formula (NDF) time-stepping scheme implemented
in ode15s [146], which is part of the commercial software package Matlab® , with
a relative integration tolerance of 10−11 . Due to the discontinuities in the controls,
the integration was performed intervalwise on the control discretization grid. A
typical spectrum of the monodromy matrix Gu can be seen in Figure 12.1. The
approximations G̃u are calculated directly from the fundamental system projected
on the slow modes or on the coarse grid.
Figure 12.2 shows the solution states and controls (u0 , q).

12.2 Euclidean vs. L2 projector

Figure 12.4 summarizes the spectral properties of the iteration matrices J˜−1 ΔJ.
The spectrum of the iteration matrix can also be interpreted as the deviation of
the preconditioned system matrix from the identity. The discretization with 4-by-4
12.2 Euclidean vs. L2 projector 165

2
1
1.5
Controls q

1 0.5
0.5
0 0
1
1
-0.5 0 0
0 0.5 1 -1
x2 -1 x1
Time t

Figure 12.2: Optimal controls q (left) and optimal states u0 (right) for target function û,
calculated for D = 0.01 on a grid of 32-by-32 elements of order 5. The displayed mesh is
not the finite element mesh but an evaluation of the Finite Element function on a coarser
equidistant mesh.

1 100
0.9
10−2
0.8
0.7 10−4
Contraction

0.6
10−6
0.5
0.4 10−8
0.3 10−10
0.2
10−12
0.1
0
0 100 200 300 400 0 100 200 300 400
Subspace dimension p Subspace dimension p

Figure 12.3: LISA contraction with Newton-Picard preconditioning versus the subspace di-
mension p for the Euclidean projector (left) and the L2 projector. Note that the plot on the
right hand side is in logarithmic scale.
166 12 Linear boundary control for the periodic 2D heat equation

1 1

0.5 0.5

0 0

-0.5 -0.5

-1 -1
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1

1 1

0.5 0.5

0 0

-0.5 -0.5

-1 -1
-1 -0.5 0 0.5 1 -1 -0.5 0 0.5 1

Figure 12.4: Top row: Unit circle and spectrum of iteration matrix for the classical Newton-
Picard with p = 20 using Euclidean projector (left column) and L2 projector (right column).
Bottom row: Like top row with p = 45.

elements of order 5 is moderately fine in order to achieve reasonable computation

times for the spectra.
Figures 12.3 and 12.4 depict that the appropriate choice of the projector for
the Newton-Picard approximation leads to fast convergence which is monotoni-
cally decreasing in the dimension p of the slow subspace. In Figure 12.3 we see
that both the Euclidean and the L2 projector eliminate many large eigenvalues, but
the Euclidean projector leaves out a few large eigenvalues which belong to eigen-
vectors which exhibit mesh-specific characteristics. Numerically we observe that
the Euclidean projector leads to a non-monotone behavior of the contraction rate
with respect to the subspace dimension, and also exhibits clear plateaus. The L2
projector leads to an exponential decay of the contraction rate with respect to the
subspace dimension and is by far superior to the Euclidean projector. Thus, only
the L2 projector will be considered further.
12.3 Mesh independence 167

Contraction σr (J˜−1 ΔJ) 100

10−2 D = 0.01, nu = 441

D = 0.01, nu = 1681
D = 0.01, nu = 6561
10−4 D = 0.1, nu = 441
D = 0.1, nu = 1681
D = 0.1, nu = 6561
10−6 0
10 101 102
Subspace dimension p

Figure 12.5: Asymptotic contraction rate for classical Newton-Picard preconditioning ver-
sus subspace dimension p for varying diffusion coefficient D and spatial degrees of freedom
nu .

12.3 Mesh independence

Figure 12.5 shows the asymptotic contraction rate of the iteration matrix J˜−1 ΔJ of
the basic linear splitting approach (6.2) with the classical Newton-Picard precon-
ditioner for diffusion coefficients D ∈ {0.1, 0.01} and spatial degrees of freedom
nu ∈ {441, 1681, 6561} with respect to the subspace dimension. Figure 12.6 shows
the same quantities for the two-grid version of the preconditioner for diffusion co-
efficients of D = {0.1, 0.01, 0.001}. We can observe that the contraction rate is
independent of nu in accordance with Corollary 6.8.
If we compare Figures 12.5 and 12.6 we see that the contraction for classical
Newton-Picard is better than for two-grid Newton-Picard with subspace dimension
p = ncu . However, better contraction is outweighed by the effort for constructing
the dominant subspace spanned by V through IRAM already for rather small values
of p. In particular for the case of D = 0.001, computation of V with p > 10 is
prohibitively slow.
However, using a Krylov method like GMRES (see Saad and Schultz [136]) to
accelerate the basic linear splitting approach (6.2) yields acceptable iteration num-
bers also for low values of p even though there is almost no contraction due to
σr (J˜−1 ΔJ) > 0.99. For the extreme pure Picard case p = 0, we obtain a solution
168 12 Linear boundary control for the periodic 2D heat equation

100

10−1
Contraction σr (J˜−1 ΔJ)

10−2
D = 0.001, nfu = 6561
D = 0.001, nfu = 25921
10−3
D = 0.01, nfu = 6561
D = 0.01, nfu = 25921
10−4 D = 0.1, nfu = 6561
D = 0.1, nfu = 25921
10−5
101 102
Coarse grid degrees of freedom ncu

Figure 12.6: Asymptotic contraction rate for two-grid Newton-Picard preconditioning ver-
sus coarse grid degrees of freedom ncu for varying diffusion coefficient D and fine grid
degrees of freedom nfu .

within 11, 34, 98 iterations for D = 0.1, 0.01, 0.001, respectively, with a termina-
tion tolerance εO = 10−4 . We remark that for inexact inner solutions with εM , εH
much larger than machine precision, Flexible GMRES (see Saad [134]) should be
employed.

As we have seen in Section 6.3.6, the effort on the coarse grid for the two-
grid Newton-Picard preconditioner is negligible compared to the effort on the fine
grid. Thus, even medium scale coarse grid degrees of freedom ncu are possible in
practical computations and lead to fast contraction rates. In this case, acceleration
of LISA (6.2) by nonlinear Krylov subspace methods does not lead to considerable
savings in the number of iterations.
12.4 Comparison with Schur complement preconditioning 169

12.4 Comparison with Schur complement

preconditioning
In Murphy et al. [117] it was shown that the symmetric positive definite exact
Schur complement preconditioner
⎛ ⎞
M 0 0
⎜ ⎟
JMGW = ⎝ 0 N 0 ⎠
−1
0 0 (Gu − Inu )M (Gu − Inu ) + γ Gq N Gq
T −1 −1 T

−1
√
leads to JMGW Jˆ having exactly three different eigenvalues 1 and (1 ± 5)/2. As
a consequence, any Krylov subspace method with an optimality or Galerkin prop-
erty converges within 3 iterations for the preconditioned system. Inversion of the
lower right block of JMGW is computationally prohibitively expensive but we can
approximate this block by the Newton-Picard approach presented in Section 6.3
which leads with X̃ = (G̃u − Inu )M −1 (G̃Tu − Inu ) ∈ Rnu ×nu to the preconditioner
⎛ ⎞
M 0 0
⎜ ⎟
J˜MGW = ⎝ 0 N 0 ⎠.
−1
0 0 X̃ + γ Gq N Gq −1 T

Now we can invoke again the Sherman-Morrison-Woodbury formula to obtain

 −1
X̃ + γ −1 Gq N −1 GTq = X̃ −1 − X̃ −1 Gq (γN + GTq X̃ −1 Gq )−1 GTq X̃ −1
= X̃ −1 − X̃ −1 Gq HGTq X̃ −1 ,

with X̃ −1 = (G̃u − Inu )−1 M(G̃Tu − Inu )−1 . We observe that the occurring matrices
coincide with the matrices which need to be inverted for the indefinite Newton-
Picard preconditioner J˜ we have developed in Section 6.3. Thus, one iteration of
an iterative method with J˜MGW can be considered computationally as expensive as
one iteration with J.˜
Because the preconditioner J˜MGW is positive definite we can employ it within a
symmetry exploiting Krylov subspace method like MINRES (see Paige and Saun-
ders [123]), which is not possible with the indefinite preconditioner J.˜ On the
downside, it is not possible to use J˜MGW in the basic linear splitting approach (6.2)
−1
because the√real eigenvalues of the iteration matrix In1 +n2 − J˜MGW J cluster around
√
0 and (1 ± 5)/2. Since (1 + 5)/2 > 1 LISA does not converge.
In Figure 12.7 we compare the number of iterations for symmetry exploiting
MINRES preconditioned by J˜MGW with the number of iterations for GMRES pre-
conditioned by J˜ for varying fine and coarse grid degrees of freedom nfu and ncu .
170 12 Linear boundary control for the periodic 2D heat equation

100
MINRES with J˜MGW , nfu = 6561
MINRES with J˜MGW , nfu = 25921
Iterations

50 GMRES with J,˜ nfu = 6561

GMRES with J,˜ nfu = 25921

0
0 50 100 150 200 250 300
Coarse grid degrees of freedom ncu

Figure 12.7: Comparison of the iterations of MINRES with Newton-Picard Schur comple-
ment preconditioner J˜MGW and GMRES with the symmetric indefinite Newton-Picard pre-
conditioner J˜ for varying fine and coarse grid degrees of freedom nfu and ncu .

We observe that the indefinite preconditioner J˜ is superior to J˜MGW by a factor of

2–4 even though J˜ is not employed in a symmetry exploiting Krylov method.
We remark that the indefinite preconditioning approach taken by Schöberl and
Zulehner [143] does not work in a straight forward way without an approximation
of the M-block in the preconditioner by a matrix M̂ such that M̂ − M is positive
definite. Thus, we do not include a comparison here.
13 Nonlinear boundary control of the
periodic 1D heat equation
In this chapter we consider the problem of optimal nonlinear boundary control of
the periodic heat equation
 
1 γ
minimize (u(1; .) − û)2 + (q − q̂)2 (13.1a)
q∈L2 (Σ),u∈W (0,1) 2 Ω 2 Σ

s. t. ∂t u = DΔu, in (0, 1) × Ω, (13.1b)

∂ν u + αu = β q , in (0, 1) × ∂ Ω,
4 4
(13.1c)
u(0; .) = u(1; .), in Ω, (13.1d)

on Ω = (0, 1). We see that problem (13.1) is very similar to the model prob-
lem (6.1) except for the polynomial terms in the boundary control condition (13.1c)
of Stefan–Boltzmann type.

13.1 Problem and algorithmical parameters

For our computations the desired state and control profiles are

û(x) = 1 + cos(π(x − 1))/10, q̂(t, x) = 1.

The other problem parameters are given by

γ = 10−4 , D = 1, α(t, 0) = β (t, 0) = 1, α(t, 1) = β (t, 1) = 0,

effectively resulting in a homogeneous Neumann boundary condition without con-

trol at x = 1. The control acts only via the boundary at x = 0.
We performed all computations with a relative integrator tolerance of 10−5 . The
algorithm terminates if the primal-dual SQP step is below 10−4 in the suitable
norm
 2  2 1/2
z := xPDE I⊗M + xrem 22 + yPDE I⊗M−1 + yrem 22 ,
V V

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_13,
© Springer Fachmedien Wiesbaden 2014
172 13 Nonlinear boundary control of the periodic 1D heat equation

where for any symmetric positive definite matrix A we define x2A = xT Ax and
where xPDE denotes the composite vector of PDE states and yPDE denotes the com-
posite vector of dual variables for the PDE state dependent part of the time bound-
ary constraint (3.3b) and the PDE continuity condition (3.3c). The variables xrem
and yrem are placeholders for the remaining primal and dual variables. The occur-
rences of the mass matrices MV in the Kronecker products (see Chapter 8) make
sure that the PDE variables are measured in an L2 sense. For the correct weight-
ing of the dual PDE variables we have to consider that a dual PDE variable ỹ of
NLP (4.1) is from the canonical dual space of RNV . To obtain a discretized Riesz
representation ŷ ∈ RNV in an L2 sense we need to require that

ỹT x = ŷT MV x for all x ∈ RNV

and thus we obtain  

ŷMV = MV−1 ỹM = ỹM−1 .
V V

We performed the computations on a hierarchy of spatial FDM meshes with

NVl = 4 · 8l−1 + 1

equidistant grid points on levels l = 1, . . . , 5 and controls which are piecewise con-
stant on grids of nMS = 12, 24, 48 equally sized intervals.
Figure 13.1 depicts the solution on the finest grid for the case of nMS = 24.

13.2 Discussion of numerical convergence

13.2.1 Grid refinements
We display the numerical self-convergence in Figure 13.2 for the case nMS = 24.
The first 10 iterations are performed fully on grid level l = 1. In iterations 9–12 and
starting again from iteration 16 on only full steps are taken (αk = 1). MUSCOP
refines the fine grid for the first time after  iteration 11 because the norm of the
 
inexact Simplified Newton increment δ z11  is small enough. Iteration 12 is the
first iteration with κ̂ > 0. An estimate of [κ̂] = 1.21 signals MUSCOP in iteration
13 that the coarse grid must be refined, too. The next refinements of the fine grid
happen after iterations 18, 21, and 23. Only three iterations are performed on the
finest grid level.  
For the computations of the distances zk − z∗  of the iterates zk to the final
solution approximation z∗ we prolongate the iterates on coarser levels l < 5 to
the finest level l = 5 and evaluate the norm on the finest level. We can observe
13.2 Discussion of numerical convergence 173

1.1 1.8
1.6
1.05 1.4
1.2
1
1
0.95 0.8
0.6
0.9 0.4
0.2
0.85 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
space x time t

Figure 13.1: Solution of problem (13.1) with nMS = 24. In the left panel we depict the state
u(1; .) at the period end (solid line) and the desired state û (dashed line). In the right panel
we depict the optimal control q over one period.

   
that in contrast to the step norm δ zk  the error zk − z∗  forms clear plateaus
in iterations 10–12, 18–19, 21–22, and 23–24. These plateaus occur because the
error in these iterations is dominated by the interpolation error of the spatial grid
on the coarser levels. We thus suggest for efficiency reasons to couple the fine grid
refinements to the contraction κ: If we can reduce the error by a factor of κ in
one step then we should refine the grid such that the interpolation error is reduced
with a similar magnitude. Thus we perform an aggressive refinement leading to
eight times more grid points after each refinement. We observe a reduction in
the (interpolation) error of about 1/8 in Figure 13.2 between iterations 19–22 and
22–24. We can thus infer by extrapolation that the final error is dominated by the
spatial interpolation error and lies around 7 · 10−4 . The observed error reduction
of O(h) in the grid size h is optimal from a theoretical point of view because it
coincides with the error of the spatial discretization.

We also perform refinement of the coarse grid aggressively. The rationale be-
hind an efficient choice of the coarse grid is to choose the coarse grid fine enough
to get fast convergence and thus fewer iterations on the numerically more expen-
sive finer grids while maintaining moderate or negligible cost of the full derivatives
on the coarse grid compared to the few directional derivatives on the fine grid.
174 13 Nonlinear boundary control of the periodic 1D heat equation

102

100

10−2
Damping αk 
 k δ∗ z 
Step norm k

Error z − z 
10−4
Coarse grid refinement
Fine grid refinement

10−6
0 5 10 15 20 25
iteration k

Figure 13.2: Self-convergence plot for problem (13.1).

13.2.2 Computation times

The computations were performed on up to four cores of an Intel® Core™ i7 with
2.67 GHz, 8 MB cache, and 18 GB RAM. In Table 13.1 we list the computation
times of the different algorithmic parts of the code. We see that with 97.6 % most
of the runtime is spent in the simulation and IND derivative generation of the
dynamic systems. The evaluation and derivative generation for the non-dynamic
functions, i.e., Φl , rb(l) , ri , re , the solution of the QP subproblems, which comprises
the condensing step for matrices and vectors, the PASM solution of the medium-
scale QP, and the blow-up of the condensed solution to the uncondensed space all
take up a negligible amount of time.
The numbers underline the hybrid programming approach that we have cho-
sen. Most of the runtime is spent within the C++ code to generate solutions and
derivatives of the dynamic systems.
The runtime can be reduced from 3289.3 s to 1714.9 s by exploitation of four
cores. The resulting speedup of 1.9 is clearly suboptimal. There are two main
reasons: First, each pair of the four cores shares one L2 cache and thus there are
13.2 Discussion of numerical convergence 175

Task Time [s] % of total

Simulation/IND 3209.8 97.6
Non-dynamic functions/AD 22.5 0.7
QP matrix condensing 6.2 0.2
QP vector condensing 3.7 0.1
QP solution 0.5 0.0
QP solution blow-up 5.9 0.2
Norm computations 1.7 0.1
Grid prolongations 6.7 0.2
Grid restrictions 15.5 0.5
GINKO 1.4 0.0
Remaining computations 15.4 0.5
Total 3289.3 100.0

Table 13.1: Timings for serial computation with nMS = 24.

penalties in cache efficiency when running on four cores. Second, the adaptive
timestepping results in different integration times on each shooting interval espe-
cially when on some intervals fast transients have to be resolved and on others not
as we can observe in Figure 13.3. Such transients can for instance be caused by
large jumps in the control variables (compare Figure 13.1). We have only imple-
mented equal distribution of the shooting intervals to the different processes. This
leads to many processes being idle until the last one has finished all its work. Op-
timal distribution of the processes is known as the job shop scheduling problem.
One simple solution heuristic is, e.g., greedy work balancing which adaptively dis-
tributes the IVPs over the available processes by assigning the currently largest job
to the first free process. The relative sizes of the jobs can be assumed to be known
from the previous SQP step. In Figure 13.4 we can see that a greedy distribution
leads to improved parallelism. A rigorous investigation of efficient parallelization
is beyond the scope of this thesis.

13.2.3 Exact Hessian vs. two-grid approximation

In Tables 13.2 and 13.3 we compare the cumulative time spent in the simulation
and IND of the dynamic systems for two different types of Hessian approximation:
176 13 Nonlinear boundary control of the periodic 1D heat equation

300 8

Integration time [s]

Integration steps
6
200
4
100
2

0 0
0 10 20 0 10 20
Shooting interval Shooting interval

Figure 13.3: Steps and integration times per shooting interval on the finest level in the solu-
tion. The solid black lines indicate the average.
core number

4
3
2
1
0 5 10 15 20 25
core number

4
3
2
1
0 5 10 15 20 25
Integration time [s]

Figure 13.4: Comparison of regular distribution of IVPs to four processes/cores (upper) and
greedy scheduling (lower). The termination time (makespan) can be significantly reduced.

The exact Hessian and the two-grid version (see Chapter 8). The quality of the
two-grid Hessian approximation is so good that we obtain the solution after 25
major iterations in both cases. Usage of a two-grid approximation yields more
evaluations of matrix vector products with the Hessian on the coarser grids but
less on the finer grids. We observe that the two-grid Hessian approximation yields
13.2 Discussion of numerical convergence 177

Level Spatial Forward Jacobian Jacobian Hessian

l NVl simulation MVP transpose MVP MVP
1 5 11.5 2.3 2.9 40.8
2 33 24.7 15.0 7.6 110.7
3 257 11.9 3.2 9.0 151.6
4 2049 46.6 12.9 39.5 715.1
5 16385 584.8 187.1 498.6 10556.8

Table 13.2: Cumulative time [s] for simulation and IND on different mesh levels for exact
Hessian approximation with nMS = 24.

Level Spatial Forward Jacobian Jacobian Hessian

l NVl simulation MVP transpose MVP MVP
1 5 16.8 3.0 3.9 81.4
2 33 32.5 17.7 9.5 158.4
3 257 14.0 3.2 8.9 33.1
4 2049 54.3 12.9 39.5 151.5
5 16385 659.1 187.6 499.9 1994.7

Table 13.3: Cumulative time [s] for simulation and IND on different mesh levels for two-
grid Hessian approximation with nMS = 24.

a performance increase of 84 % for the Hessian evaluation on the finest grid. The
overall wall-time savings on four cores amount to 68 % in this example.

13.2.4 Refinement of control in time

In Table 13.4 we present the number of SQP iterations on each spatial discretiza-
tion level when we refine the control discretization in time. We observe that more
iterations are needed for finer control discretizations but they are all spent on the
coarsest levels l = 1, 2. The SQP iterations on the finer levels l = 3, 4, 5 coincide.
The overall runtime for finer control discretizations increases due to three main
reasons: First, the number of necessary grid transfer operations, i.e., prolonga-
tions and restrictions, increases linearly with number of shooting intervals nMS .
178 13 Nonlinear boundary control of the periodic 1D heat equation

Control time dis- SQP iters on level l = Runtime [s]

cretization nMS 1 2 3 4 5 QP grid transfers total
12 9 6 3 2 3 5.5 6.5 1042.4
24 11 7 3 2 3 20.7 19.0 1396.4
48 17 19 3 2 3 93.7 48.2 1842.0

Table 13.4: SQP iterations on each spatial discretization level and runtimes of selected parts
for varying time discretizations of the control computed on four cores.

Level Spatial Forward Jacobian Jacobian Hessian

l NVl simulation MVP transpose MVP MVP
1 5 9.2 1.6 1.9 30.3
2 33 21.6 11.6 5.4 72.9
3 257 10.9 2.7 7.6 28.0
4 2049 41.6 10.8 33.3 126.7
5 16385 508.0 162.0 419.4 1714.9

Table 13.5: Cumulative time [s] for simulation and IND on different mesh levels for two-
grid Hessian approximation with nMS = 12.

Second, the effort for the solution of the QP subproblems increases because the
amount of linear algebra operations for the condensing step (see Chapter 8) in-
creases quadratically with nMS and because the condensed QP grows linearly in
size with nMS . Third, we can see in Tables 13.5 and 13.6 that the effort for simula-
tion and IND increases. The reason lies in the adaptivity of the IVP solver because
every jump in the controls leads to transients in the dynamic system which require
finer time steps to be resolved to the requested accuracy.
13.2 Discussion of numerical convergence 179

Level Spatial Forward Jacobian Jacobian Hessian

l NVl simulation MVP transpose MVP MVP
1 5 16.9 4.6 6.7 176.5
2 33 60.6 45.8 22.3 720.2
3 257 18.2 3.8 10.7 39.5
4 2049 68.6 15.2 46.6 178.0
5 16385 836.7 231.5 604.2 2417.2

Table 13.6: Cumulative time [s] for simulation and IND on different mesh levels for two-
grid Hessian approximation with nMS = 48.
14 Optimal control for a bacterial
chemotaxis system

Chemotaxis is the phenomenon of single cells moving in a directed fashion in re-

action to a chemical substance in their environment, e.g., to seek for food. In the
seminal paper of Keller and Segel [95] a mathematical model for bacterial chemo-
taxis was proposed for the first time. For further information and bibliographical
references see Horstmann [87, 88].
Chemotaxis can be explained by two phases of bacterial movement: A phase
of tumbling movement similar to a random walk and a phase of directed move-
ment through propulsion by flagella rotation (see Figure 14.1). The duration of
each phase is controlled by a chemical substance in the environment, the so called
chemoattractant. In environments with low chemoattractant concentration tum-
bling movement prevails while directed movement prevails in environments with
higher chemoattractant concentration. The effect of this simple mechanism is that
for large numbers of bacteria the bacteria will on average move upwards gradients
of the chemoattractant concentration. This behavior leads to interesting dynamic
phenomena like pattern formation and traveling waves of bacteria.

Figure 14.1: Simplified schematic of E.coli with rotating flagella for directed movement.

14.1 Problem formulation

We use the model of Tyson et al. [154, 155] which has been also used in a opti-
mizing boundary control scenario by Lebiedz and Brandt-Pollmann [102] with the
software MUSCOD-II. In contrast to their results, our approach allows for a much
higher accuracy in the spatial discretization.

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_14,
© Springer Fachmedien Wiesbaden 2014
182 14 Optimal control for a bacterial chemotaxis system

More specifically we consider the tracking type boundary control problem

  
1 γc γq 1 2
minimize (z(1, · ) − ẑ)2 + (c(1, · ) − ĉ)2 + q (14.1a)
z,c,q 2 Ω 2 Ω 2 0
 
∇c
s. t. ∂t z = Dz Δz + α∇ · z , in (0, 1) × Ω, (14.1b)
(1 + c)2
z2
∂t c = Δc + w − ρc in (0, 1) × Ω, (14.1c)
(μ + z2 )
∂ν z = 0, in (0, 1) × ∂ Ω, (14.1d)
∂ν c = β (q − c), in (0, 1) × ∂ Ω, (14.1e)
z(0, .) = z0 , (14.1f)
c(0, .) = c0 , (14.1g)
qu ≥ q ≥ ql , in (0, 1) × ∂ Ω, (14.1h)

where ∂ν denotes the derivative in direction of the outwards pointing normal on Ω.

The objective (14.1a) penalizes the deviation of the cell density and the chemo-
attractant concentration from given desired distributions at the end of the time
horizon, and penalizes excessive use of the control. The governing system of
PDEs is nonlinear. The difficulty lies in the chemotaxis term (preceded by α)
of equation (14.1b) which is a convection term with nonlinear convection velocity
(1 + c)−2 ∇c for z. In equation (14.1c) we see that the chemoattractant evolves
due to diffusion, is produced proportional to a nonlinear function of the cell den-
sity, and decays with a factor ρ. There is no flux of bacteria over the domain
boundaries due to the Neumann condition (14.1d) but we can control the system
via chemoattractant influx over the boundary in a Robin-type fashion according to
condition (14.1e), where q describes a controllable chemoattractant concentration
outside of the domain Ω. The optimization scenario (14.1) is not periodic in time.
Instead, we prescribe initial values for the cell density and the chemoattractant
concentration in equations (14.1f)–(14.1g). Finally we require the control q to be
bounded between qu and ql .

14.2 Numerical results

We computed approximate solutions to the optimal control problem (14.1) with
the problem data listed in Table 14.1. We used a four level hierarchy of spatial
FDM grids with 17, 65, 257, and 1025 equidistant points for z and c and nMS = 36
multiple shooting intervals. The computation ran 31 iterations in 15 min 40 s on
14.2 Numerical results 183

Symbol Value Description

Ω [0, 1] spatial domain
ẑ(x) 2x target cell density distribution
γc 0.5 weight for concentration tracking
ĉ(x) 0 target chemoattractant distribution
γq 1e-3 weight for control penalization
Dz 0.33 cell diffusion
α 80 chemotaxis coefficient
w 1 chemoattractant production coefficient
μ 1 chemoattractant production denominator
ρ 0 chemoattractant decay coefficient
β 0.1 Robin boundary control coefficient
z0 (x) 1 initial cell density distribution
c0 (x) 0 initial chemoattractant distribution
qu 0.2 upper control bound
ql 0.0 lower control bound

Table 14.1: Chemotaxis boundary control model data.

four cores. The IVP integrator performed between 19 and 64 integration steps per
shooting interval with an average of 27.1 steps in the solution on the finest grid.
Figure 14.2 shows a self-convergence plot. We observe that after refinement of the
fine and the coarse grid, the globalization strategy needs to recede to damped steps
for iterations 15 and 16. Afterwards only full steps are taken. Only four iterations
are performed on the finest grid level with derivatives generated on the second
coarsest level. The error plateaus are even more prominent than for the example
in Chapter 13. From extrapolation of the error at iterations
  15, 23, and 27, we can
expect the final solution to be accurate to about zk − z∗  ≈ 0.1. This accuracy
is not satisfactory but on our current system and with our implementation, finer
spatial discretizations are not possible. We do not believe that this is a generic
problem of our approach because the main memory problem is that DAESOL-
II currently keeps all IND tapes (including right hand side Jacobians and their
decompositions) in main memory. This takes up the largest amount of memory
within GINKO. We are positive that this memory bottleneck can be circumvented
184 14 Optimal control for a bacterial chemotaxis system

by previsional swapping out and in of tape entries to hard disk or by checkpointing

techniques.
From Figure 14.2 we also see that many iterations on coarser grids, namely the
iterations on the plateaus, can be saved if a reliable estimator for the interpolation
error is available. For efficiency reasons the fine
 grid should be refined as soon as
 
the inexact Simplified Newton increment δ zk  is below the interpolation error of
the spatial grid. This aspect is, however, beyond the scope of this thesis.
Moreover, if we consider the error reduction between iterations 21–22, 25–26,
29–30, and 30–31 we observe that the error reduction in the last iterates on each
of the three finest grid levels is roughly (fine) grid independent.
We depict the optimal states at different snapshots in time in Figure 14.4 and
the corresponding controls in Figure 14.3. We observe that in order to achieve
the unnatural linear distribution ẑ the optimal solution consists of a control action
on the left boundary at the beginning, followed by a control action on the right
boundary shortly afterwards. This effects the formation of two cell heaps close to
the boundary. Finally a control action on the right boundary makes the left heap of
cells move to the middle of the domain and the right heap to grow further towards
the target cell distribution ẑ.
14.2 Numerical results 185

Damping αk
Step norm δ zk 
102 Error zk − z∗ 
Coarse grid refinement
Fine grid refinement

100

10−2

10−4

0 5 10 15 20 25 30
iteration k

Figure 14.2: Self convergence plot for the chemotaxis problem (14.1).

0.2 0.2
0.15 0.15
control q

0.1 0.1
0.05 0.05
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
time t time t

Figure 14.3: Optimal control profiles for the chemotaxis problem (14.1). The left hand
panel shows the control at the boundary x = 0 and the right hand panel at x = 1.
186 14 Optimal control for a bacterial chemotaxis system

t = 0.00 t = 0.17
2 2
dens. z, conc. c

1.5 1.5
1 1
0.5 0.5
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
t = 0.33 t = 0.50
2 2
dens. z, conc. c

1.5 1.5
1 1
0.5 0.5
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
t = 0.67 t = 0.92
2 2
dens. z, conc. c

1.5 1.5
1 1
0.5 0.5
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
t = 0.94 t = 1.00
2 2
dens. z, conc. c

1.5 1.5
1 1
0.5 0.5
0 0
0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1
x coordinate x coordinate

Figure 14.4: Optimal states for the chemotaxis problem (14.1). For different time points t
we plot the bacteria density z (solid line) the chemoattractant concentration c (dashed line)
and the bacteria target distribution (dash-dotted line).
15 Optimal control of a Simulated Moving
Bed process
In this chapter we describe a variant of the Simulated Moving Bed (SMB) process.
For completeness we quote in large parts from the article Potschka et al. [130].
In a chromatographic column, different components that are dissolved in a liquid
are separated due to different affinities to the adsorbent. As a result the different
components move with different velocities through the column and hence can be
separated into nearly pure fractions at the outlet. The SMB process consists of
several chromatographic columns which are interconnected in series to constitute
a closed loop (see Figure 15.1). An effective counter-current movement of the sta-
tionary phase relative to the liquid phase is realized by periodic and simultaneous
switching of the inlet and outlet ports by one column in the direction of the liq-
uid flow. Compared to batch operation of a single chromatographic column, the
SMB process offers great improvements of process performance in terms of des-
orbent consumption and utilization of the solid bed. In the basic SMB process all

Figure 15.1: SMB configuration with six columns and four zones.

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3_15,
© Springer Fachmedien Wiesbaden 2014
188 15 Optimal control of a Simulated Moving Bed process

flow rates are constant and the switching of the columns is simultaneous with a
fixed switching period. By introducing more degrees of freedom the efficiency of
the separation can be increased further. The flow rates for instance can be varied
during the switching periods (PowerFeed), the feed concentration can be varied
during the switching periods (ModiCon) or asynchronous switching of the ports
can be introduced (VariCol) (see Schramm et al. [144, 145]).

15.1 Mathematical modeling of adsorption processes

Accurate dynamic models of such multi-column continuous chromatographic pro-
cesses consist of the dynamic process models for each single chromatographic
column, the node balances which describe the connection of the columns, and the
port switching. The behavior of radially homogeneous chromatographic columns
is described by the General Rate Model (see Schmidt-Traub [142]).

15.1.1 General Rate Model

For both species i = 1, 2 the General Rate Model considers three phases, namely
the instationary phase ci which moves through the columns between the fixed bed
particles, the liquid stationary phase cp,i inside the porous fixed bed particles, and
the adsorbed stationary phase qp,i on the inner surface of the particles.
We assume that the columns are long and thin enough that radial concentration
profiles can be neglected. The fixed bed particles are assumed to be spherical and
the concentrations inside the particles are assumed to be rotationally symmetric.
The governing equations in non-dimensional form are
 
∂t ci = Pe−1
i ∂zz ci − ∂z ci − Sti ci − cp,i |r=1 , (t, z) ∈ (0, T ) × (0, 1), (15.1a)
   −2  2 
∂t (1 − εp )qp,i + εp cp,i = ηi r ∂r r ∂r cp,i , (t, r) ∈ (0, T ) × (0, 1), (15.1b)

together with the boundary conditions

∂z ci (t, 0) = Pei ci (t, 0) − cin (t) , ∂r cp,i (t, 0) = 0, (15.2a)

 
∂z ci (t, 1) = 0, ∂r cp,i (t, 1) = Bii ci (t, z) − cp,i (t, 1) ,
(15.2b)

with positive constants εp (porosity), ηi (nondimensional diffusion coefficient),

Pei (Péclet number), Sti (Stanton number), and Bii (Biot number). The stationary
15.1 Mathematical modeling of adsorption processes 189

phases are coupled by an algebraic condition, e.g., the nonlinear extended Lang-
muir isotherm equation

Hi2 cp,i
qp,i = Hi1 cp,i + , (15.3)
1 + (k1 cp,1 + k2 cp,2 )cref

with non-negative constants Hi1 , Hi2 (Henry coefficients), ki (isotherm parameters),

and reference concentration cref .
The model poses a number of difficulties:
1. The isotherm equations are algebraic constraints.
2. The time derivatives ∂t qp,i and ∂t cp,i are coupled on the left hand side of
equation (15.1b).
3. For each point z ∈ [0, 1] in the axial direction a stationary phase equation
(15.1b) is supposed to hold.
4. The stationary phase equation has a singularity for r = 0.
Regarding point (3), we should think of equation (15.1b) as living on the two-
dimensional (z, r) domain without any derivatives in the axial direction. The cou-
pling occurs through the boundary conditions and equation (15.1a). Gu [76] pro-
posed to address this issue by using a low order collocation discretization of the
stationary phase in each grid point of the mesh for the moving phase. We now
explain this procedure in detail.
We address points (1) and (2) by elimination of qp,i via substitution of the alge-
braic constraints (15.3) into equation (15.1b). After differentiation with respect to
t we obtain a system of the form
    −2  2 
∂t cp,1 −1 η1 r ∂r r ∂r cp,1 
= G(cp,1 , cp,2 ) ,
∂t cp,2 η2 r−2 ∂r r2 ∂r cp,2

where the coupling 2-by-2 matrix G depends nonlinearly on cp,i via

  
H 2 c k c
ref 1 p,1
G11 = (1 − ε p ) H11 + 1
1− + εp,
1 + cref ∑ j k j cp, j 1 + cref ∑ j k j cp, j
cref H12 cp,1 k2
G12 = (ε p − 1) ,
(1 + cref ∑ j k j cp, j )2
cref H22 cp,2 k1
G21 = (ε p − 1) ,
(1 + cref ∑ j k j cp, j )2
  
H22 cref k2 cp,2
G22 = (1 − ε p ) H2 +
1
1− + εp.
1 + cref ∑ j k j cp, j 1 + cref ∑ j k j cp, j
190 15 Optimal control of a Simulated Moving Bed process

This 2-by-2 matrix can be inverted with the closed formula

 
1 G22 −G12
G−1 = .
G11 G22 − G21 G12 −G21 G11

As proposed by Gu [76] we approximate C p,i (t, r) by a quadratic collocation

polynomial ϕ(r). We impose that ϕ(r) satisfies the two boundary conditions
(15.2). Thus we are left with one degree of freedom which we choose to be the
point value
bi (t, z) := ϕ(0.5).
We are lead to the form

ϕ(r) = 4σi (ci (t, z) − bi (t, z))r2 + bi (t, z) − σi (ci (t, z) − bi (t, z))

with the abbreviation

Bii
σi = .
8 + 3Bii
The properties ϕ(0.5) = bi (t, z) and ∂r ϕ(0) = 0 are readily verified. The second
boundary condition holds true due to

ϕ(1) = 4σi (ci − bi ) + bi − σi (ci − bi ) = 3σi (ci − bi ) + bi ,

Bii (ci − ϕ(1)) = Bii (ci − 3σi (ci − bi ) − bi )
Bii
= [(8 + 3Bii )(ci − bi ) − 3Bii (ci − bi )]
8 + 3Bii
= 8σi (ci − bi ) = ϕ  (1).

For completeness we assemble here the derivatives and surface values required for
the substitution of the cp,i terms by bi :

∂ϕ ∂ bi ∂ϕ
(0.5) = , (r) = 8σi (ci − bi )r,
∂t  ∂t ∂r
1 ∂ ∂ϕ
r2 = 24σi (ci − bi ), ϕ(1) = 3σi (ci − bi ) + bi .
r2 ∂ r ∂r

All in all we have transformed equations (15.1a) and (15.1b) to

∂t ci = Pe−1
i ∂zz ci − ∂z ci − Sti (ci − (3σi (ci − bi )) + bi ), (15.4a)
   
∂t b1 −1 η1 24σ1 (c1 − b1 )
= G(b1 , b2 ) , (15.4b)
∂t b2 η2 24σ2 (c2 − b2 )
15.1 Mathematical modeling of adsorption processes 191

with boundary conditions

∂z ci (t, 0) = Pei ci (t, 0) − cin (t) , ∂z ci (t, 1) = 0.

In the case of several connected columns we use one reference flow velocity uref
for the non-dimensionalization. For a flow velocity u j = uref in zone j = I, . . . , IV
we have to multiply the right hand sides of equations (15.1) or (15.4), respectively,
with the quotient u j /uref .

15.1.2 Mass balances

The model for the whole SMB process consists of a fixed number Ncol of columns
described by the General Rate Model and mass balances at the ports between the
columns. The concentrations of column j are denoted by a superscript j. In the
ModiCon variant, the process is controlled by the time-independent flow rates QDe
(desorbent), QEx (extract), QRec (recycle), QFe (feed), and the time-dependent feed
concentration cFe (t). The remaining flow rates, which are the raffinate flow rate
QRa and the zone flow rates QI , . . . , QIV , are fully determined by conservation of
mass via

QRa = QDe − QEx + QFe ,

QI = QDe + QRec , QII = QI − QEx ,
QIII = QII + QFe , QIV = QIII − QRa = QRec .

The inflow concentrations of each column are the outflow concentrations of the
preceding column, except for the column after the feed and after the desorbent
ports which can be calculated from the feed concentration cFe,i and from the out-
flow concentrations cout
.,i of the previous column according to

I,i QI = cIV,i QIV ,

cin III,i QIII = cII,i QII + cFe,i QFe ,
out
cin out

for i = 1, 2. With the port concentrations and the flow rates the feed, extract, and
raffinate masses, and the product purities can be calculated via
 t  t
mFe,i (t) = cFe,i (τ)QFe dτ, mEx,i (t) = I,i (τ, 1)QEx dτ,
cout
0 0
 t
mRa,i (t) = III,i (τ, 1)QRa dτ,
cout
0
mEx,1 (t) mRa,2 (t)
PurEx (t) = , PurRa (t) = .
mEx,1 (t) + mEx,2 (t) mRa,1 (t) + mRa,2 (t)
192 15 Optimal control of a Simulated Moving Bed process

15.1.3 Objective and constraints

We consider the optimization of an SMB process with variable feed concentration
(ModiCon process) which minimizes desorbent consumption
 T
QDe (t)dt
0

subject to purity constraints for the two product streams

PurEx (T ) ≥ Purmin and PurRa (T ) ≥ Purmin

at a constant feed flow QFe but varying feed concentration cFe (t). Over one period
T the average feed concentration must be equal to the given feed concentration
cSMB
Fe of a reference SMB process.
At the end of each period the switching of ports leads to a generalized periodicity
constraint of the form
succ( j) +
cij (0, .) − ci (T, .) = 0,
i = 1, 2, j = 1, . . . , Ncol ,
succ( j)
bij (0, .) − bi (T, .) = 0,

where succ( j) denotes the index of the column which is the successor of column
j in the investigated SMB configuration.
Furthermore we require the total feed mass of one period to satisfy

mFe (T ) = cSMB
Fe QFe T, (15.5)

where cSMB
Fe is a given feed concentration of a (non-ModiCon) SMB reference
process.
The remaining constraints bound the maximum and minimum feed concentra-
tion
cFe,max ≥ cFe (t) ≥ 0
and the flow rates

Qmax ≥ QDe , QEx , QFe , QRa , QRe , QI , QII , QIII , QIV ≥ Qmin .

15.2 Numerical results

The results in this chapter were computed for EMD–53986 enantiomer separa-
tion. EMD–53986, or 5-(1,2,3,4-tetra-hydroquinolin-6-yl)-6-methyl-3,6-dihydro-
1,3,4-thiadiazin-2-one (see Figure 15.2), is a chiral precursor for a pharmaceutical
15.2 Numerical results 193

Figure 15.2: Chemical structure of EMD–53986.

reagent (see, e.g., Jupke [91] as cited by Küpper [100]). Only the R-enantiomer
has pharmaceutical activity and needs to be separated from the S-enantiomer after
chemical synthesis. We list the model parameters (taken from Küpper [100]) in
Table 15.1. Further model quantities are derived from these parameters which we
display in Table 15.2.
We computed the solution with nMS = 24 shooting intervals on a two level hier-
archy of spatial FDM grids with 21 and 81 equidistant grid points for each of the
Ncol = 6 columns and each species. The relative accuracy for the time-stepping
scheme was set to 10−5 on the coarse and 10−6 on the fine level and the GINKO
termination tolerance was 5 · 10−3 .
The optimization problem becomes more and more difficult for higher values of
product purity Purmin . We had to successively generate primal starting values via a
naive homotopy approach using ascending values for Purmin = 0.8, 0.9, 0.93, 0.95
on the coarse level. For Purmin = 0.95, GINKO needed 9 iterations on the coarse
level and then 11 iterations on the fine level with coarse grid derivatives. The com-
puted κ-estimates suggest [κ̂] ≤ 0.66. For the last four iterations only two LISA
are needed for each inexact solution of the linearized systems. Table 15.3 shows
the optimal values for the ModiCon SMB separation of EMB–53986 enantiomers.
We display the optimal feed concentration profile in Figure 15.4 and the optimal
moving concentration fronts of the moving phase for one period in Figure 15.3.
We can observe that the two concentration profiles travel to the right with different
velocities and thus there is almost only slow substance present at the extract port
after column 1 and almost only fast substance present at the raffinate port after
column 5.
The solution was computed on four cores within a total wall time of 98 min. Due
to memory restrictions for the IND tape sizes, finer grid levels were not possible.
In the solution there are between 24 and 177 integration steps per shooting interval
with an average of 42.1 steps per interval.
194 15 Optimal control of a Simulated Moving Bed process

Symbol Value Unit Description

L 9.0 cm column length
D 2.5 cm column diameter
εp 0.567 – particle void fraction
εb 0.353 – bulk void fraction
dp 0.002 cm particle diameter
ρ 0.799 g/cm3 fluid density
ν 0.012 g/(cm s) fluid viscosity
Dp 0.001 cm2 /s particle diffusion (estimated)
kapp,1 1.5e-4 1/s apparent mass transfer coefficient
kapp,2 2.0e-4 1/s apparent mass transfer coefficient
H11 2.054 – Henry coefficient
H21 2.054 – Henry coefficient
H12 19.902 – Henry coefficient
H22 5.847 – Henry coefficient
k1 472.0 cm3 /g isotherm parameter
k2 129.0 cm3 /g isotherm parameter
cref 2.5e-3 g/cm3 reference concentration
Purmin 95 % minimum product purity
cSMB
Fe 2.5e-3 g/cm3 reference feed concentration
cFe,max 1.25e-2 g/cm3 maximum feed concentration
Qmax 300 ml/min maximum flow rate
Qmin 30 ml/min minimum flow rate

Table 15.1: Model and optimization parameters for the ModiCon SMB process to separate
EMD–53986 enantiomers.
15.2 Numerical results 195

Symbol Formula Description

6
keff,i kapp,i effective mass transfer coefficient
dp
dp keff,i 15εp Dp
kl,i mass transfer coefficient
6 15εp Dp − (dp /2)2 keff,i
kl,i dp
Bii Biot number
2εp Dp
4QIII
uref reference flow velocity
πD2 εb
4Q j
uj flow velocity
πD2 εb
ρu j dp
Re j Reynolds number
ν
0.2 + 0.011(Re j εb )0.48 L
Pe j Péclet number
εb dp
4εp Dp L
ηj particle diffusion coefficient
dp2 u j
1 − εb
Stij 3Bii η j Stanton number
εb

Table 15.2: Derived model quantities for the ModiCon SMB process. Index i = 1, 2 denotes
the species, index j = I, . . . , IV the zone.

Description Symbol Optimal value Unit

period duration T 10.68 min
desorbent flow QDe 86.33 ml/min
extract flow QEx 86.33 ml/min
feed flow QFe 30.00 ml/min
recycle flow Q 30.00 ml/min
$ T Re
objective 0 QDe (t)dt 921.8 ml

Table 15.3: Optimal values for the ModiCon SMB separation of EMB–53986 enantiomers.
196 15 Optimal control of a Simulated Moving Bed process

t = 0.00 min t = 1.78 min

4 4
Concentration [g/l]

3 3

2 2

1 1

0 0
0 1 2 3 4 5 6 0 1 2 3 4 5 6
t = 3.56 min t = 5.34 min
4 4
Concentration [g/l]

3 3

2 2

1 1

0 0
0 1 2 3 4 5 6 0 1 2 3 4 5 6
t = 7.12 min t = 8.90 min
4 4
Concentration [g/l]

3 3

2 2

1 1

0 0
0 1 2 3 4 5 6 0 1 2 3 4 5 6
Column index Column index

Figure 15.3: Traveling concentration profiles over one period t ∈ [0, T ]. The six columns
are arranged from left to right in each panel. The feed port is located after column 3, extract
after column 1, raffinate after colum 5 and desorbent after column 6.
15.2 Numerical results 197

Feed concentration cFe [g/l

15

10

0
0 2 4 6 8 10
Time t

Figure 15.4: Optimal feed concentration profile cFe for ModiCon SMB separation of EMB–
53986 enantiomers. All feed mass is injected at the end of the period with maximum con-
centration cFe,max , subject to satisfaction of total feed mass constraint (15.5).
16 Conclusions and future work
In this thesis we have developed a numerical method based on Direct Multiple
Shooting for OCPs with time-periodic PDE constraints. We have achieved an
asymptotically optimal scale-up of the numerical effort with the number of spa-
tial discretization points based on inexact SQP with an inner generalized Newton-
Picard preconditioned LISA which features extensive structure exploitation in a
two-stage solution process for the possibly nonconvex QPs for which we have
developed a condensing approach and a PASM. We have implemented a numeri-
cal code called MUSCOP and have demonstrated the applicability, efficiency, and
reliability of the proposed methods on PDE OCPs from illustrating academic to
challenging real-world applications.
Our research inspires a number of now exposed questions for future research
directions and projects. We want to conclude this thesis with a list of the most
obvious ones:

Convergence analysis for PASMs for nonconvex QPs. We have developed nu-
merical techniques for the solution of nonconvex QPs with PASMs. In our
experience the resulting QP solver works unexpectedly well on these diffi-
cult problems. We believe it worthwhile to construct examples for which
it does not work or investigate proofs if none can be found. These exam-
ples, if found, might also serve as the basis for further improvement of the
numerical method.
NMT for inexact SQP. We have presented a NMT globalization strategy for the
class of LISA-Newton based inexact SQP methods. However, no proof of
convergence exists for this approach. We conjecture such a proof is possible
even for the inexact SQP case on the basis of RMT techniques.
A-posteriori mesh error estimation and mesh refinement. We have computed
all numerical examples in this thesis on uniformly refined spatial meshes.
Obviously locally refined meshes promise a great improvement of the ratio
of numerical effort vs. accuracy (see, e.g., Becker and Rannacher [14], Mei-
dner and Vexler [113], Hesse [83]). Furthermore, we should refine the fine
and coarse grid according to two different goals: The fine grid for highest ac-
curacy (in terms of the question which inspires the problem) and the coarse

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3,
© Springer Fachmedien Wiesbaden 2014
200 16 Conclusions and future work

grid for best contraction, i.e., smallest κ. Moreover the required global er-
ror estimators should be exploited to trigger fine grid refinement as soon
as the inexact Simplified Newton increment becomes smaller than the grid
interpolation error.

DAESOL-II tape management. The tape management of DAESOL-II is cur-

rently the memory bottleneck of MUSCOP. We propose to implement asyn-
chronous swapping of tape entries in and out of main memory to hard disk.
Because the tape must always be read in sequential order either forward
or backward, it is possible to prompt swapping of the required blocks into
main memory in advance. This process does not require CPU cycles on
most current hardware platforms and can be performed concurrently with
the remaining required computations.

Load balancing techniques. For simplicity we have only implemented regular

distribution of Multiple Shooting IVPs for simulation and IND to worker
processes in parallel. As we have demonstrated in Chapter 13 even a sim-
ple adaptive greedy heuristic could considerably improve the speed-up of
MUSCOP.

Computations on a cluster computer. The numerical results which we have pre-

sented in this thesis were computed on a desktop machine on four CPU
cores. We have designed MUSCOP to also run on a distributed cluster com-
puter. This approach would also mitigate the memory bottleneck issue be-
cause memory consumption is also parallelized in the proposed algorithm
on the Multiple Shooting structure.

Nonlinear instationary 2D and 3D problems. As we can see from the analysis

(e.g., Theorem 6.7), the proposed methods are not generally restricted by the
dimensionality of the considered geometry. After completion of the projects
mentioned in the three previous paragraphs we believe it is possible to treat
even larger instationary problems in 2D. For 3D problems we anticipate that
numerical fill-in in the direct sparse linear algebra routines within DAESOL-
II can become a bottleneck and would have to be addressed before.
Bibliography
[1] A. Agarwal, L.T. Biegler, and S.E. Zitney. Simulation and optimization of
pressure swing adsorption systems using reduced-order modeling. Indus-
trial & Engineering Chemistry Research, 48(5):2327–2343, 2009.

[2] J. Albersmeyer. Adjoint based algorithms and numerical methods for

sensitivity generation and optimization of large scale dynamic systems.
PhD thesis, Ruprecht–Karls–Universität Heidelberg, 2010. URL http:
//www.ub.uni-heidelberg.de/archiv/11651/.

[3] J. Albersmeyer and H.G. Bock. Efficient sensitivity generation for large
scale dynamic systems. Technical report, SPP 1253 Preprints, University of
Erlangen, 2009.

[4] J. Albersmeyer and M. Diehl. The Lifted Newton method and its application
in optimization. SIAM Journal on Optimization, 20(3):1655–1684, 2010.

[5] J. Albersmeyer and C. Kirches. The SolvIND webpage, 2007. URL http:
//www.solvind.org.

[6] M. Arioli, I.S. Duff, N.I.M. Gould, J.D. Hogg, J.A. Scott, and H.S. Thorne.
The HSL mathematical software library, 2007.

[7] U. Ascher and M.R. Osborne. A note on solving nonlinear equations and the
natural criterion function. Journal of Optimization Theory and Applications,
55(1):147–152, 1987.

[8] V. Bär. Ein Kollokationsverfahren zur numerischen Lösung allgemeiner

Mehrpunktrandwertaufgaben mit Schalt- und Sprungbedingungen mit An-
wendungen in der optimalen Steuerung und der Parameteridentifizierung.
Diploma thesis, Rheinische Friedrich–Wilhelms–Universität zu Bonn,
1983.

[9] A. Battermann and M. Heinkenschloss. Preconditioners for Karush-Kuhn-

Tucker matrices arising in the optimal control of distributed systems. In

A. Potschka, A Direct Method for Parabolic PDE Constrained Optimization Problems,

Advances in Numerical Mathematics, DOI 10.1007/978-3-658-04476-3,
© Springer Fachmedien Wiesbaden 2014
202 Bibliography

Control and estimation of distributed parameter systems (Vorau, 1996), vol-

ume 126 of Internat. Ser. Numer. Math., pages 15–32. Birkhäuser, Basel,
1998.

[10] A. Battermann and E.W. Sachs. Block preconditioners for KKT systems
in PDE-governed optimal control problems. In Fast solution of discretized
optimization problems (Berlin, 2000), volume 138 of Internat. Ser. Numer.
Math., pages 1–18. Birkhäuser, Basel, 2001.

[11] I. Bauer. Numerische Verfahren zur Lösung von Anfangswertaufgaben und

zur Generierung von ersten und zweiten Ableitungen mit Anwendungen bei
Optimierungsaufgaben in Chemie und Verfahrenstechnik. PhD thesis, Uni-
versität Heidelberg, 1999. URL http://www.ub.uni-heidelberg.de/
archiv/1513.

[12] I. Bauer, H.G. Bock, S. Körkel, and J.P. Schlöder. Numerical methods for
initial value problems and derivative generation for DAE models with appli-
cation to optimum experimental design of chemical processes. In Scientific
Computing in Chemical Engineering II, pages 282–289. Springer, 1999.

[13] I. Bauer, H.G. Bock, and J.P. Schlöder. DAESOL – a BDF-code for the
numerical solution of differential algebraic equations, 1999.

[14] R. Becker and R. Rannacher. An optimal control approach to error estima-

tion and mesh adaptation in finite element methods. Acta Numerica 2000,
pages 1–101, 2001.

[15] R.E. Bellman. Dynamic Programming. University Press, Princeton, N.J.,

6th edition, 1957.

[16] M. Benzi, G.H. Golub, and J. Liesen. Numerical solution of saddle–point

problems. Acta Numerica, 14:1–137, 2005.

[17] M.J. Best. An algorithm for the solution of the parametric quadratic pro-
gramming problem. In H. Fischer, B. Riedmüller, and S. Schäffler, editors,
Applied Mathematics and Parallel Computing – Festschrift for Klaus Ritter,
chapter 3, pages 57–76. Physica-Verlag, Heidelberg, 1996.

[18] L.T. Biegler. Solution of dynamic optimization problems by successive

quadratic programming and orthogonal collocation. Computers and Chem-
ical Engineering, 8:243–248, 1984.
Bibliography 203

[19] G. Biros and O. Ghattas. Parallel Lagrange-Newton-Krylov-Schur methods

for PDE-constrained optimization. Part I: The Krylov-Schur solver. SIAM
Journal on Scientific Computing, 27(2):687–713, 2005.

[20] G. Biros and O. Ghattas. Parallel Lagrange-Newton-Krylov-Schur meth-

ods for PDE-constrained optimization. Part II: The Lagrange-Newton solver
and its application to optimal control of steady viscous flows. SIAM Journal
on Scientific Computing, 27(2):714–739, 2005.

[21] J. Blue, P. Fox, W. Fullerton, D. Gay, E. Grosse, A. Hall, L. Kaufman,

W. Petersen, and N. Schryer. PORT mathematical subroutine library, 1997.
URL http://www.bell-labs.com/project/PORT/.

[22] H.G. Bock. Numerical treatment of inverse problems in chemical reac-

tion kinetics. In K.H. Ebert, P. Deuflhard, and W. Jäger, editors, Modelling
of Chemical Reaction Systems, volume 18 of Springer Series in Chemical
Physics, pages 102–125. Springer, Heidelberg, 1981. URL http://www.
iwr.uni-heidelberg.de/groups/agbock/FILES/Bock1981.pdf.

[23] H.G. Bock. Recent advances in parameter identification techniques for

ODE. In P. Deuflhard and E. Hairer, editors, Numerical Treatment of Inverse
Problems in Differential and Integral Equations, pages 95–121. Birkhäuser,
Boston, 1983. URL http://www.iwr.uni-heidelberg.de/groups/
agbock/FILES/Bock1983.pdf.

[24] H.G. Bock. Randwertproblemmethoden zur Parameteridentifizierung in

Systemen nichtlinearer Differentialgleichungen, volume 183 of Bonner Ma-
thematische Schriften. Universität Bonn, Bonn, 1987. URL http://www.
iwr.uni-heidelberg.de/groups/agbock/FILES/Bock1987.pdf.

[25] H.G. Bock and K.J. Plitt. A Multiple Shooting algorithm for di-
rect solution of optimal control problems. In Proceedings of the
9th IFAC World Congress, pages 242–247, Budapest, 1984. Pergamon
Press. URL http://www.iwr.uni-heidelberg.de/groups/agbock/
FILES/Bock1984.pdf.

[26] H.G. Bock, E.A. Kostina, and J.P. Schlöder. On the role of natural level
functions to achieve global convergence for damped Newton methods. In
M.J.D. Powell and S. Scholtes, editors, System Modelling and Optimization.
Methods, Theory and Applications, pages 51–74. Kluwer, 2000.
204 Bibliography

[27] H.G. Bock, W. Egartner, W. Kappis, and V. Schulz. Practical shape opti-
mization for turbine and compressor blades by the use of PRSQP methods.
Optimization and Engineering, 3(4):395–414, 2002.

[28] H.G. Bock, A. Potschka, S. Sager, and J.P. Schlöder. On the connection be-
tween forward and optimization problem in one-shot one-step methods. In
G. Leugering, S. Engell, A. Griewank, M. Hinze, R. Rannacher, V. Schulz,
M. Ulbrich, and S. Ulbrich, editors, Constrained Optimization and Opti-
mal Control for Partial Differential Equations, volume 160 of International
Series of Numerical Mathematics, pages 37–49. Springer Basel, 2011.

[29] Dietrich Braess. Finite elements. Cambridge University Press, Cambridge,

3rd edition, 2007.

[30] J.H. Bramble and J.E. Pasciak. A preconditioning technique for indefinite
systems resulting from mixed approximations of elliptic problems. Mathe-
matics of Computation, 50(181):1–17, 1988.

[31] A.N. Brooks and T.J.R. Hughes. Streamline upwind/Petrov-Galerkin for-

mulations for convection dominated flows with particular emphasis on the
incompressible Navier-Stokes equations. Computational Methods in Ap-
plied Mechanics and Engineering, 32(1-3):199–259, 1982.

[32] K. Burrage, J. Erhel, B. Pohl, and A. Williams. A deflation technique for

linear systems of equations. SIAM Journal on Scientific Computing, 19(4):
1245–1260, 1998.

[33] R.H. Byrd, F.E. Curtis, and J. Nocedal. An inexact SQP method for equality
constrained optimization. SIAM Journal on Optimization, 19(1):351–369,
2008.

[34] A.R. Conn, K. Scheinberg, and L.N. Vicente. Introduction to derivative-

free optimization, volume 8 of MPS/SIAM Series on Optimization. Society
for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2009.

[35] J. Dahl and L. Vandenberghe. CVXOPT user’s guide, release 1.1.3, 2010.
URL http://abel.ee.ucla.edu/cvxopt/userguide/index.html.

[36] G.B. Dantzig. Linear Programming and Extensions. Princeton University

Press, 1963.
Bibliography 205

[37] R. Dautray and J.-L. Lions. Evolution problems I. In A. Craig, editor,

Mathematical analysis and numerical methods for science and technology,
volume 5. Springer, 1992.

[38] D.F. Davidenko. On a new method of numerical solution of systems of

nonlinear equations. Doklady Akademii nauk SSSR, 88:601–602, 1953.

[39] T.A. Davis. Algorithm 832: UMFPACK – an unsymmetric-pattern multi-

frontal method with a column pre-ordering strategy. ACM Transactions on
Mathematical Software, 30:196–199, 2004.

[40] V. de la Torre, A. Walther, and L.T. Biegler. Optimization of periodic ad-

sorption processes with a novel problem formulation and nonlinear pro-
gramming algorithm. In AD 2004 – Fourth International Workshop on Au-
tomatic Differentiation, July 19-23, 2004, Argonne National Laboratory,
USA, 2004. (extended conference abstract).

[41] R.S. Dembo, S.C. Eisenstat, and T. Steihaug. Inexact Newton methods.
SIAM Journal on Numerical Analysis, 19(2):400–408, 1982.

[42] P. Deuflhard. Ein Newton-Verfahren bei fastsingulärer Funktionalmatrix

zur Lösung von nichtlinearen Randwertaufgaben mit der Mehrzielmethode.
PhD thesis, Universität zu Köln, 1972.

[43] P. Deuflhard. A Modified Newton Method for the Solution of Ill-

conditioned Systems of Nonlinear Equations with Applications to Multiple
Shooting. Numerische Mathematik, 22:289–311, 1974.

[44] P. Deuflhard. Newton Methods for Nonlinear Problems. Affine Invariance

and Adaptive Algorithms, volume 35 of Springer Series in Computational
Mathematics. Springer, 2006.

[45] P. Deuflhard, R. Freund, and A. Walter. Fast secant methods for the iterative
solution of large nonsymmetric linear systems. IMPACT of Computing in
Science and Engineering, 2(3):244–276, 1990.

[46] L. Di Gaspero. QuadProg++, 2010. URL http://www.diegm.uniud.

it/digaspero/index.php/software/.

[47] M. Diehl. Real-Time Optimization for Large Scale Nonlinear Pro-

cesses. PhD thesis, Universität Heidelberg, 2001. URL http://www.ub.
uni-heidelberg.de/archiv/1659/.
206 Bibliography

[48] M. Diehl, A. Walther, H.G. Bock, and E. Kostina. An adjoint-based SQP

algorithm with quasi-Newton Jacobian updates for inequality constrained
optimization. Optimization Methods and Software, 2009.

[49] J. Dongarra, V. Eijkhout, and A. Kalhan. LAPACK working note 99:

Reverse communication interface for linear algebra templates for iterative
methods. Technical Report UT-CS-95-292, University of Tennessee, 1995.

[50] N. Dunford and J.T. Schwartz. Linear operators part I: General theory. In
R. Courant, L. Bers, and J.J. Stoker, editors, Pure and applied mathematics,
volume VII. Wiley, New York, 1958.

[51] J. Fernández, M. Anguita, E. Ros, and J.L. Bernier. SCE toolboxes for the
development of high–level parallel applications. In Proceedings of the 6th
International Conference Computational Science – ICCS 2006, Reading,
United Kingdom, Part II, volume 3992, pages 518–525, 2006.

[52] H.J. Ferreau. An online active set strategy for fast solution of para-
metric quadratic programs with applications to predictive engine control.
Diploma thesis, Ruprecht–Karls–Universität Heidelberg, 2006. URL http:
//homes.esat.kuleuven.be/~jferreau/pdf/thesisONLINE.pdf.

[53] H.J. Ferreau, H.G. Bock, and M. Diehl. An online active set strategy to
overcome the limitations of explicit MPC. International Journal of Robust
and Nonlinear Control, 18(8):816–830, 2008.

[54] FICO. FICO(TM) Xpress Optimization Suite, Xpress-Optimizer Reference

manual, Release 20.00. Fair Isaac Corporation, Warwickshire, UK, 2009.

[55] R. Fletcher. Resolving degeneracy in quadratic programming. Annals of

Operations Research, 46-47:307–334, 1993.

[56] C.A. Floudas and P.M. Pardalos, editors. State of the art in global opti-
mization: computational methods and applications. Springer, Dordrecht,
1995.

[57] H. Gajewski, K. Gröger, and K. Zacharias. Nichtlineare Operatorgleichun-

gen und Operatordifferentialgleichungen. Akademie-Verlag, Berlin, 1974.

[58] E.M. Gertz and S.J. Wright. Object-oriented software for quadratic pro-
gramming. ACM Transactions on Mathematical Software, 29:58–81, 2003.
Bibliography 207

[59] P.E. Gill, W. Murray, M.A. Saunders, and M.H. Wright. Procedures for op-
timization problems with a mixture of bounds and general linear constraints.
ACM Transactions on Mathematical Software, 10(3):282–298, 1984.

[60] P.E. Gill, W. Murray, and M.A. Saunders. User’s Guide For QPOPT 1.0: A
Fortran Package For Quadratic Programming, 1995. URL http://www.
sbsi-sol-optimize.com/manuals/QPOPT%20Manual.pdf.

[61] G.H. Golub and C.F. van Loan. Matrix Computations. Johns Hopkins Uni-
versity Press, Baltimore, 3rd edition, 1996.

[62] J. Gondzio. HOPDM (version 2.12) – a fast LP solver based on a primal-

dual interior point method. European Journal of Operational Research, 85
(1):221–225, 1995.

[63] N.I.M. Gould. On practical conditions for the existence and uniqueness of
solutions to the general equality quadratic programming problem. Mathe-
matical Programming, 32(1):90–99, 1985.

[64] N.I.M. Gould. An algorithm for large-scale quadratic programming. IMA

Journal of Numerical Analysis, 11(3):299–324, 1991.

[65] N.I.M. Gould and Ph.L. Toint. Nonlinear programming without a penalty
function or a filter. Mathematical Programming, Series A, 122:155–196,
2010.

[66] N.I.M. Gould and P.L. Toint. A quadratic programming bibliography. Tech-
nical Report 2000-1, Rutherford Appleton Laboratory, Computational Sci-
ence and Engineering Department, 2010.

[67] N.I.M. Gould, M.E. Hribar, and J. Nocedal. On the solution of equality
constrained quadratic programming problems arising in optimization. SIAM
Journal on Scientific Computing, 23(4):1376–1395, 2001.

[68] N.I.M. Gould, D. Orban, and P.L. Toint. CUTEr testing environment for
optimization and linear algebra solvers, 2002. URL http://cuter.rl.
ac.uk/cuter-www/.

[69] N.I.M. Gould, D. Orban, and Ph.L. Toint. GALAHAD, a library of thread-
safe Fortran 90 packages for large-scale nonlinear optimization. ACM
Transactions on Mathematical Software, 29(4):353–372, 2004.
208 Bibliography

[70] A. Griewank. Evaluating Derivatives, Principles and Techniques of Algo-

rithmic Differentiation. Number 19 in Frontiers in Applied Mathematics.
SIAM, Philadelphia, 2000.

[71] A. Griewank. Projected Hessians for preconditioning in One-Step One-Shot

design optimization. In Large-Scale Nonlinear Optimization, volume 83 of
Nonconvex Optimization and Its Applications, pages 151–171. Springer US,
2006.

[72] A. Griewank, D. Juedes, and J. Utke. Algorithm 755: ADOL-C: A pack-

age for the automatic differentiation of algorithms written in C/C++. ACM
Transactions on Mathematical Software, 22(2):131–167, 1996.

[73] A. Griewank, D. Juedes, H. Mitev, J. Utke, O. Vogel, and A. Walther.

ADOL-C: A package for the automatic differentiation of algorithms written
in C/C++. Technical report, Technical University of Dresden, Institute of
Scientific Computing and Institute of Geometry, 1999. Updated version of
the paper published in ACM Trans. Math. Software 22:131–167, 1996.

[74] A. Griewank, J. Utke, and A. Walther. Evaluating higher derivative tensors

by forward propagation of univariate Taylor series. Mathematics of Com-
putation, pages 1117–1130, 2000.

[75] W. Gropp, E. Lusk, and A. Skjellum. Using MPI: Portable Parallel Pro-
gramming with the Message Passing Interface. Scientific and Engineering
Computation Series. MIT Press, Cambridge, MA, USA, 2nd edition, 1999.

[76] T. Gu. Mathematical Modelling and Scale Up of Liquid Chromatography.

Springer Verlag, New York, 1995.

[77] W. Hackbusch. Fast numerical solution of time-periodic parabolic problems

by a multigrid method. SIAM Journal on Scientific and Statistical Comput-
ing, 2(2):198–206, 1981.

[78] E. Hairer, S.P. Nørsett, and G. Wanner. Solving Ordinary Differential

Equations I, volume 8 of Springer Series in Computational Mathematics.
Springer, Berlin, 2nd edition, 1993.

[79] P. Hartman. Ordinary differential equations, volume 38 of Classics in Ap-

plied Mathematics. SIAM, second edition, 2002.
Bibliography 209

[80] S.B. Hazra, V. Schulz, J. Brezillon, and N.R. Gauger. Aerodynamic shape
optimization using simultaneous pseudo-timestepping. Journal of Compu-
tational Physics, 204(1):46–64, 2005.

[81] M. Heinkenschloss and D. Ridzal. An Inexact Trust-Region SQP method

with applications to PDE-constrained optimization. In K. Kunisch, G. Of,
and O. Steinbach, editors, Proceedings of ENUMATH 2007, the 7th Euro-
pean Conference on Numerical Mathematics and Advanced Applications,
Graz, Austria, September 2007. Springer Berlin Heidelberg, 2008.

[82] M. Heinkenschloss and L.N. Vicente. Analysis of Inexact Trust-Region

SQP algorithms. SIAM Journal on Optimization, 12(2):283–302, 2001.

[83] H.K. Hesse. Multiple Shooting and Mesh Adaptation for PDE Constrained
Optimization Problems. PhD thesis, University of Heidelberg, 2008.

[84] J.S. Hesthaven and T. Warburton. Nodal Discontinuous Galerkin Methods,

volume 54 of Texts in Applied Mathematics. Springer New York, 2008.

[85] J.S. Hesthaven, S. Gottlieb, and D. Gottlieb. Spectral methods for time-
dependent problems, volume 21 of Cambridge Monographs on Applied
and Computational Mathematics. Cambridge University Press, Cambridge,
2007.

[86] M. Hinze, R. Pinnau, M. Ulbrich, and S. Ulbrich. Optimization with PDE

Constraints. Springer, New York, 2009.

[87] D. Horstmann. From 1970 until present: The Keller-Segel model in chemo-
taxis and its consequences I. Jahresbericht der DMV, 105(3):103–165,
2003.

[88] D. Horstmann. From 1970 until present: The Keller-Segel model in chemo-
taxis and its consequences II. Jahresbericht der DMV, 106(2):51–69, 2004.

[89] IBM ILOG. IBM ILOG CPLEX V12.1, User’s Manual for CPLEX. IBM
Corp., New York, USA, 2009.

[90] H. Jarausch and W. Mackens. Numerical treatment of bifurcation branches

by adaptive condensation. In Numerical methods for bifurcation problems
(Dortmund, 1983), volume 70 of Internat. Schriftenreihe Numer. Math.,
pages 296–309. Birkhäuser, Basel, 1984.
210 Bibliography

[91] A. Jupke. Experimentelle Modellvalidierung und modellbasierte Auslegung

von Simulated Moving Bed (SMB) Chromatographieverfahren, volume 807
of Fortschrittberichte VDI, Reihe 3. VDI-Verlag, Düsseldorf, 2004. Disser-
tation, Universität Dortmund.

[92] W. Karush. Minima of functions of several variables with inequalities as

side conditions. Master’s thesis, Department of Mathematics, University of
Chicago, 1939.

[93] Y. Kawajiri and L.T. Biegler. Optimization strategies for Simulated Moving
Bed and PowerFeed processes. AIChE Journal, 52(4):1343–1350, 2006.

[94] Y. Kawajiri and L.T. Biegler. Large scale nonlinear optimization for asym-
metric operation and design of simulated moving beds. Journal of Chro-
matography A, 1133:226–240, 2006.

[95] E.F. Keller and L.A. Segel. Model for chemotaxis. Journal of Theoretical
Biology, 30(2):225–234, 1971.

[96] C. Kirches, H.G. Bock, J.P. Schlöder, and S. Sager. A factorization

with update procedures for a KKT matrix arising in direct optimal con-
trol. Mathematical Programming Computation, 2010. URL http://www.
optimization-online.org/DB_HTML/2009/11/2456.html. Submit-
ted.

[97] C. Kirches, H.G. Bock, J.P. Schlöder, and S. Sager. Block structured
quadratic programming for the direct multiple shooting method for optimal
control. Optimization Methods and Software, 26(2):239–257, 2011.

[98] E. Kostina. The long step rule in the bounded-variable dual simplex method:
Numerical experiments. Mathematical Methods of Operations Research, 55
(3):413–429, 2002.

[99] H.W. Kuhn and A.W. Tucker. Nonlinear programming. In J. Neyman,

editor, Proceedings of the Second Berkeley Symposium on Mathematical
Statistics and Probability, pages 481–492, Berkeley, 1951. University of
California Press.

[100] A. Küpper. Optimization, State Estimation, and Model Predictive Control

of Simulated Moving Bed Processes, volume 2010,1 of Schriftenreihe des
Lehrstuhls für Systemdynamik und Prozessführung. Prof. Dr.-Ing. Sebastian
Engell, Dortmund, 2010. Dissertation.
Bibliography 211

[101] S. Lauer. SQP–Methoden zur Behandlung von Problemen mit indefiniter re-
duzierter Hesse–Matrix. Diploma thesis, Ruprecht–Karls–Universität Hei-
delberg, 2010.
[102] D. Lebiedz and U. Brandt-Pollmann. Manipulation of Self-Aggregation
Patterns and Waves in a Reaction-Diffusion System by Optimal Boundary
Control Strategies. Physical Review Letters, 91(20):208301, 2003.
[103] R.B. Lehoucq and D.C. Sorensen. Deflation techniques for an implicitly
restarted Arnoldi iteration. SIAM Journal on Matrix Analysis and Applica-
tions, 17(4):789–821, 1996.
[104] R.B. Lehoucq, D.C. Sorensen, and C. Yang. ARPACK Users’ Guide: Solu-
tion of Large–Scale Eigenvalue Problems with Implicitly Restarted Arnoldi
Methods. Society for Industrial and Applied Mathematics (SIAM), 1998.
[105] D.B. Leineweber. The theory of MUSCOD in a nutshell. IWR-Preprint
96-19, Universität Heidelberg, 1996.
[106] D.B. Leineweber. Efficient reduced SQP methods for the optimization of
chemical processes described by large sparse DAE models, volume 613 of
Fortschritt-Berichte VDI Reihe 3, Verfahrenstechnik. VDI Verlag, Düssel-
dorf, 1999.
[107] D.B. Leineweber, I. Bauer, A.A.S. Schäfer, H.G. Bock, and J.P. Schlöder.
An efficient multiple shooting based reduced SQP strategy for large-scale
dynamic process optimization (Parts I and II). Computers and Chemical
Engineering, 27:157–174, 2003.
[108] R.J. LeVeque. Finite volume methods for hyperbolic problems. Cambridge
Texts in Applied Mathematics. Cambridge University Press, Cambridge,
2002.
[109] R.J. LeVeque. Finite difference methods for ordinary and partial differen-
tial equations. Society for Industrial and Applied Mathematics (SIAM),
Philadelphia, PA, 2007.
[110] K. Lust, D. Roose, A. Spence, and A. R. Champneys. An adaptive Newton-
Picard algorithm with subspace iteration for computing periodic solutions.
SIAM Journal on Scientific Computing, 19(4):1188–1209, 1998.
[111] I. Maros and C. Mészáros. A repository of convex quadratic programming
problems. Optimization Methods and Software, 11:671–681, 1999.
212 Bibliography

[112] The Mathworks. Matlab optimization toolbox user’s guide, 2010.

[113] D. Meidner and B. Vexler. Adaptive space-time finite element methods for
parabolic optimization problems. SIAM Journal on Control and Optimiza-
tion, 46(1):116–142, 2007.
[114] C. Mészáros. The BPMPD interior point solver for convex quadratic prob-
lems. Optimization Methods and Software, 11(1):431–449, 1999.
[115] B. Morini. Convergence behaviour of inexact Newton methods. Mathemat-
ics of Computation, 68(228):1605–1613, 1999.
[116] MOSEK. The MOSEK optimization tools manual, version 6.0 (revision
85), 2010. URL http://www.mosek.com/.
[117] M.F. Murphy, G.H. Golub, and A.J. Wathen. A note on preconditioning
for indefinite linear systems. SIAM Journal on Scientific Computing, 21(6):
1969–1972, 2000.
[118] K.G. Murty. Some NP-complete problems in quadratic and nonlinear pro-
gramming. Mathematical Programming, 39:117–129, 1987.
[119] R.A. Nicolaides. Deflation of conjugate gradients with applications to
boundary value problems. SIAM Journal on Numerical Analysis, 24(2):
355–365, 1987.
[120] S. Nilchan and C. Pantelides. On the optimisation of periodic adsorption
processes. Adsorption, 4:113–147, 1998.
[121] J. Nocedal and S.J. Wright. Numerical Optimization. Springer Verlag,
Berlin Heidelberg New York, 2nd edition, 2006.
[122] J.M. Ortega and W.C. Rheinboldt. Iterative solution of nonlinear equations
in several variables. Academic Press, New York, 1970.
[123] C.C. Paige and M.A. Saunders. Solutions of sparse indefinite systems of
linear equations. SIAM Journal on Numerical Analysis, 12(4):617–629,
1975.
[124] B.N. Parlett and W.G. Poole, Jr. A geometric theory for the QR, LU and
power iterations. SIAM Journal on Numerical Analysis, 10:389–412, 1973.
[125] K.J. Plitt. Ein superlinear konvergentes Mehrzielverfahren zur direkten
Berechnung beschränkter optimaler Steuerungen. Diploma thesis, Rheinis-
che Friedrich–Wilhelms–Universität Bonn, 1981.
Bibliography 213

[126] L.S. Pontryagin, V.G. Boltyanski, R.V. Gamkrelidze, and E.F. Miscenko.
The Mathematical Theory of Optimal Processes. Wiley, Chichester, 1962.

[127] A. Potschka. Handling path constraints in a direct multiple shooting method

for optimal control problems. Diploma thesis, Universität Heidelberg, 2006.
URL http://apotschka.googlepages.com/APotschka2006.pdf.

[128] A. Potschka, H.G. Bock, and J.P. Schlöder. A minima tracking variant of
semi-infinite programming for the treatment of path constraints within di-
rect solution of optimal control problems. Optimization Methods and Soft-
ware, 24(2):237–252, 2009.

[129] A. Potschka, C. Kirches, H.G. Bock, and J.P. Schlöder. Reliable so-
lution of convex quadratic programs with parametric active set meth-
ods. Technical Report 2010–11–2828, Heidelberg University, Interdisci-
plinary Center for Scientific Computing, Heidelberg University, Im Neuen-
heimer Feld 368, 69120 Heidelberg, Germany, 2010. URL http://www.
optimization-online.org/DB_HTML/2009/02/2224.html.

[130] A. Potschka, A. Küpper, J.P. Schlöder, H.G. Bock, and S. Engell. Opti-
mal control of periodic adsorption processes: The Newton-Picard inexact
SQP method. In Recent Advances in Optimization and its Applications in
Engineering, pages 361–378. Springer Verlag, 2010.

[131] A. Potschka, M.S. Mommer, J.P. Schlöder, and H.G. Bock. A Newton-
Picard approach for efficient numerical solution of time-periodic parabolic
PDE constrained optimization problems. Technical Report 2010–03–
2570, Interdisciplinary Center for Scientific Computing (IWR), Heidelberg
University, 2010. URL http://www.optimization-online.org/DB_
HTML/2010/03/2570.html.

[132] S.M. Robinson. Perturbed Kuhn-Tucker points and rates of convergence for
a class of nonlinear programming algorithms. Mathematical Programming,
7:1–16, 1974.

[133] Y. Saad. Numerical methods for large eigenvalue problems. Algorithms and
Architectures for Advanced Scientific Computing. Manchester University
Press, Manchester, 1992.

[134] Y. Saad. A flexible inner-outer preconditioned GMRES algorithm. SIAM

Journal on Scientific Computing, 14(2):461–469, 1993.
214 Bibliography

[135] Y. Saad. Iterative Methods for Sparse Linear Systems. SIAM, Philadelpha,
PA, 2nd edition, 2003.

[136] Y. Saad and M.H. Schultz. GMRES: A generalized minimal residual algo-
rithm for solving nonsymmetric linear systems. SIAM Journal on Scientific
and Statistical Computing, 7:856–869, 1986.

[137] S. Sager. Lange Schritte im Dualen Simplex-Algorithmus. Diploma

thesis, Universität Heidelberg, 2001. URL http://mathopt.de/
PUBLICATIONS/Sager2001.pdf.

[138] S. Sager. Numerical methods for mixed–integer optimal control prob-

lems. Der andere Verlag, Tönning, Lübeck, Marburg, 2005. URL http:
//mathopt.de/PUBLICATIONS/Sager2005.pdf.

[139] S. Sager, C. Barth, H. Diedam, M. Engelhart, and J. Funke. Optimization

as an analysis tool for human complex problem solving. SIAM Journal on
Optimization, 2011. Accepted.

[140] A. Schäfer, H. Mara, J. Freudenreich, C. Bathow, B. Breuckmann, and H.G.

Bock. Large scale Angkor style reliefs: High definition 3D acquisition
and improved visualization using local feature estimation. In Proc. of 39th
Annual Conference of Computer Applications and Quantitative Methods in
Archaeology (CAA), Beijing, China, 2011. Accepted.

[141] A.A.S. Schäfer. Efficient reduced Newton-type methods for so-

lution of large-scale structured optimization problems with applica-
tion to biological and chemical processes. PhD thesis, Universität
Heidelberg, 2005. URL http://archiv.ub.uni-heidelberg.de/
volltextserver/volltexte/2005/5264/.

[142] H. Schmidt-Traub, editor. Preparative Chromatography of Fine Chemicals

and Pharmaceuticals. Harwell Report. Wiley-VCH, 2005.

[143] J. Schöberl and W. Zulehner. Symmetric indefinite preconditioners for sad-

dle point problems with applications to PDE-constrained optimization prob-
lems. SIAM Journal on Matrix Analysis and Applications, 29(3):752–773,
2007.

[144] H. Schramm, S. Grüner, and A. Kienle. Optimal operation of Simulated

Moving Bed chromatographic processes by means of simple feedback con-
trol. Journal of Chromatography A, 1006:3–13, 2003.
Bibliography 215

[145] H. Schramm, M. Kaspereit, A. Kienle, and A. Seidel-Morgenstern. Simu-

lated moving bed process with cyclic modulation of the feed concentration.
Journal of Chromatography A, 1006:77–86, 2003.
[146] L.F. Shampine and M.W. Reichelt. The MATLAB ODE suite. SIAM Jour-
nal on Scientific Computing, 18(1):1–22, 1997.
[147] G.M. Shroff and H.B. Keller. Stabilization of unstable procedures: the re-
cursive projection method. SIAM Journal on Numerical Analysis, 30(4):
1099–1120, 1993.
[148] W. Squire and G. Trapp. Using complex variables to estimate derivatives of
real functions. SIAM Review, 40:110–112, 1998.
[149] G.W. Stewart. Simultaneous iteration for computing invariant subspaces of
non-Hermitian matrices. Numerische Mathematik, 25:123–136, 1976.
[150] Vidar Thomée. Galerkin finite element methods for parabolic problems,
volume 25 of Springer Series in Computational Mathematics. Springer-
Verlag, Berlin, 2nd edition, 2006.
[151] A. Toumi, S. Engell, M. Diehl, H.G. Bock, and J.P. Schlöder. Efficient
optimization of Simulated Moving Bed Processes. Chemical Engineering
and Processing, 46 (11):1067–1084, 2007.
[152] F. Tröltzsch. Optimale Steuerung partieller Differentialgleichungen: Theo-
rie, Verfahren und Anwendungen. Vieweg+Teubner Verlag, Wiesbaden, 2nd
edition, 2009.
[153] B.A. Turlach. QuadProg (quadratic programming routines), release 1.4,
1998. URL http://school.maths.uwa.edu.au/~berwin/software/
quadprog.html.
[154] R. Tyson, S. R. Lubkin, and J. D. Murray. Model and analysis of chemotac-
tic bacterial patterns in a liquid medium. Journal of Biology, 38:359–375,
1999.
[155] R. Tyson, S. R. Lubkin, and J. D. Murray. A minimal mechanism for bac-
terial pattern formation. Proceedings of the Royal Society B: Biological
Sciences, 266:299–304, 1999.
[156] T.L. van Noorden, S.M. Verduyn Lunel, and A. Bliek. Optimization of
cyclically operated reactors and separators. Chemical Engineering Science,
58:4114–4127, 2003.
216 Bibliography

[157] R.J. Vanderbei. LOQO: An interior point code for quadratic programming.
Optimization Methods and Software, 11(1–4):451–484, 1999.

[158] A. Walther. A first-order convergence analysis of Trust-Region methods

with inexact Jacobians. SIAM Journal on Optimization, 19(1):307–325,
2008.

[159] A. Walther, A. Kowarz, and A. Griewank. ADOL-C: A package for the

automatic differentiation of algorithms written in C/C++. Technical report,
Institute of Scientific Computing, Technical University Dresden, 2005.

[160] X. Wang. Resolution of ties in parametric quadratic programming. Master’s

thesis, University of Waterloo, Ontario, Canada, 2004.

[161] T. Warburton and M. Embree. The role of the penalty in the local dis-
continuous galerkin method for Maxwell’s eigenvalue problem. Computer
Methods Applied Mechanical Engineering, 195:3205–3223, 2006.

[162] A.J. Wathen. Realistic eigenvalue bounds for the Galerkin mass matrix.
IMA Journal of Numerical Analysis, 7(4):449–457, 1987.

[163] R.C. Whaley, A. Petitet, and J.J. Dongarra. Automated empirical optimiza-
tion of software and the ATLAS project. Parallel Computing, 27(1–2):3–35,
2001.

[164] E.P. Wigner. The unreasonable effectiveness of mathematics in the natural

sciences. Communications on Pure and Applied Mathematics, 13:1–14,
1960. Richard Courant lecture in mathematical sciences delivered at New
York University, May 11, 1959.

[165] J.H. Wilkinson. The Algebraic Eigenvalue Problem. Clarendon Press, Ox-
ford, 1965.

[166] L. Wirsching. An SQP algorithm with inexact derivatives for a Direct Mul-
tiple Shooting method for optimal control problems. Diploma thesis, Uni-
versität Heidelberg, 2006.

[167] J. Wloka. Partielle Differentialgleichungen: Sobolevräume u. Randwertauf-

gaben. B.G. Teubner, Stuttgart, 1982.

[168] X. Zhang and Y. Ye. User’s guide of COPL QP, computational optimization
program library: Convex quadratic programming, 1998. URL http://
www.stanford.edu/~yyye/Col.html.

Download more eBooks here: http://avaxhm.com/blogs/ChrisRedfield