Nonlinear Least Squares for Inverse Problems

Scientific Computation
Editorial Board
J.-J. Chattot, Davis, CA, USA
P. Colella, Berkeley, CA, USA
Weinan E, Princeton, NJ, USA
R. Glowinski, Houston, TX, USA
M. Holt, Berkeley, CA, USA
Y. Hussaini, Tallahassee, FL, USA
P. Joly, Le Chesnay, France
H. B. Keller, Pasadena, CA, USA
J. E. Marsden, Pasadena, CA, USA
D. I. Meiron, Pasadena, CA, USA
O. Pironneau, Paris, France
A. Quarteroni, Lausanne, Switzerland
and Politecnico of Milan, Italy
J. Rappaz, Lausanne, Switzerland
R. Rosner, Chicago, IL, USA
P. Sagaut, Paris, France
J. H. Seinfeld, Pasadena, CA, USA
A. Szepessy, Stockholm, Sweden
M. F.Wheeler, Austin, TX, USA

For other titles published in this series, go to

http:/www.springer.com/series/718
G. Chavent

Nonlinear Least Squares

for Inverse Problems
Theoretical Foundations and
Step-by-Step Guide for Applications

With 25 Figures

123
Guy Chavent
Ceremade, Université Paris-Dauphine
75775 Paris Cedex 16
France
and
Inria-Rocquencourt
BP 105, 78153 Le Chesnay Cedex
France
[email protected]

ISSN 1434-8322
ISBN 978-90-481-2784-9 e-ISBN 978-90-481-2785-6
DOI 10.1007/978-90-481-2785-6
Springer Dordrecht Heidelberg London New York

Library of Congress Control Number: 2009927706

Springer
c Science+Business Media B.V. 2009
No part of this work may be reproduced, stored in a retrieval system, or transmitted in any form or by
any means, electronic, mechanical, photocopying, microfilming, recording or otherwise, without written
permission from the Publisher, with the exception of any material supplied specifically for the purpose
of being entered and executed on a computer system, for exclusive use by the purchaser of the work.

Cover design: eStudio Calamar S.L.

Printed on acid-free paper

Springer is part of Springer Science+Business Media (www.springer.com)

To my wife Annette
Preface

The domain of inverse problems has experienced a rapid expansion, driven

by the increase in computing power and the progress in numerical modeling.
When I started working on this domain years ago, I became somehow frus-
trated to see that my friends working on modeling where producing existence,
uniqueness, and stability results for the solution of their equations, but that
I was most of the time limited, because of the nonlinearity of the problem, to
prove that my least squares objective function was differentiable. . . . But with
my experience growing, I became convinced that, after the inverse problem
has been properly trimmed, the final least squares problem, the one solved
on the computer, should be Quadratically (Q)-wellposed, that is, both well-
posed and optimizable: optimizability ensures that a global minimizer of the
least squares function can actually be found using efficient local optimization
algorithms, and wellposedness that this minimizer is stable with respect to
perturbation of the data.
But the vast majority of inverse problems are nonlinear, and the classi-
cal mathematical tools available for their analysis fail to bring answers to
these crucial questions: for example, compactness will ensure existence, but
provides no uniqueness results, and brings no information on the presence or
absence of parasitic local minima or stationary points . . . .
This book is partly a consequence of this early frustration: a first ob-
jective is to present a geometrical theory for the analysis of NLS problems
from the point of view of Q-wellposedness: for an attainable set with finite
curvature, this theory provides an estimation of the size of the admissible
parameter set and of the error level on the data for which Q-wellposedness

vii
viii PREFACE

holds. The various regularization techniques used to trim the inverse problem
can then be checked against their ability to produce the desirable Q-wellposed
problems.
The second objective of the book is to give a detailed presentation of im-
portant practical issues for the resolution of NLS problems: sensitivity func-
tions and adjoint state methods for the computations of derivatives, choice of
optimization parameters (calibration, sensitivity analysis, multiscale and/or
adaptive parameterization), organization of the inversion code, and choice
of the descent step for the minimization algorithm. Most of this material is
seldom presented in detail, because it is quite elementary from the mathe-
matical point of view, and has usually to be rediscovered by trial-and-error!
As one can see from these objectives, this book does not pretend to give
an exhaustive panorama of nonlinear inverse problems, but merely to present
the author’s view and experience on the subject. Alternative approaches,
when known, are mentioned and referenced, but not developed. The book is
organized in two parts, which can be read independently:
Part I (Chaps. 1–5) is devoted to the step-by-step resolution and analysis
of NLS inverse problems. It should be of interest to scientists of various
application fields interested in the practical resolution of inverse problems,
as well as to applied mathematicians interested also in their analysis. The
required background is a good knowledge of Hilbert spaces, and some notions
of functional analysis if one is interested in the infinite dimensional examples.
The elements of the geometrical theory of Part II, which are necessary for
the Q-wellposedness analysis, are presented without demonstration, but in
an as-intuitive-as-possible way, at the beginning of Chap. 4, so that it is not
necessary to read Part II, which is quite technical.
Part II (Chaps. 6–8) presents the geometric theory of quasi-convex and
strictly quasi-convex sets, which are the basis for the results of Chaps. 4 and 5.
These sets possess a neighborhood where the projection is well-behaved, and
can be recognized by their finite curvature and limited deflection. This part
should be of interest to those more interested in the theory of projection
on nonconvex sets. It requires familiarity with Hilbert spaces and functional
analysis. The material of Part II was scattered in various papers with different
notations. It is presented for the first time in this book in a progressive
and coherent approach, which benefits from substantial enhancements and
simplifications in the definition of strictly quasi-convex sets.
To facilitate a top-to-bottom approach of the subject, each chapter starts
with an overview of the concepts and results developed herein – at the price of
PREFACE ix

some repetition between the overview and the main corpus of the chapter. . . .
Also, we have tried to make the index more user-friendly, all indexed words
or expressions are emphasized in the text (but not all emphasized words are
indexed!).
I express my thanks to my colleagues, and in particular to François
Clement, Karl Kunisch, and Hend Benameur for the stimulating discussions
we had over all these years, and for the pleasure I found interacting with
them.

March 2009 Guy Chavent

Lyon
Contents

Preface vii

I Nonlinear Least Squares 1

1 Nonlinear Inverse Problems: Examples and Difficulties 5
1.1 Example 1: Inversion of Knott–Zoeppritz Equations . . . . . . 6
1.2 An Abstract NLS Inverse Problem . . . . . . . . . . . . . . . . 9
1.3 Analysis of NLS Problems . . . . . . . . . . . . . . . . . . . . 10
1.3.1 Wellposedness . . . . . . . . . . . . . . . . . . . . . . . 10
1.3.2 Optimizability . . . . . . . . . . . . . . . . . . . . . . . 12
1.3.3 Output Least Squares Identifiability and Quadratically
Wellposed Problems . . . . . . . . . . . . . . . . . . . 12
1.3.4 Regularization . . . . . . . . . . . . . . . . . . . . . . . 14
1.3.5 Derivation . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.4 Example 2: 1D Elliptic Parameter Estimation Problem . . . . 21
1.5 Example 3: 2D Elliptic Nonlinear Source Estimation Problem 24
1.6 Example 4: 2D Elliptic Parameter Estimation Problem . . . . 26

2 Computing Derivatives 29
2.1 Setting the Scene . . . . . . . . . . . . . . . . . . . . . . . . . 30
2.2 The Sensitivity Functions Approach . . . . . . . . . . . . . . . 33
2.3 The Adjoint Approach . . . . . . . . . . . . . . . . . . . . . . 33
2.4 Implementation of the Adjoint Approach . . . . . . . . . . . . 38
2.5 Example 1: The Adjoint Knott–Zoeppritz Equations . . . . . . 41

xi
xii CONTENTS

2.6 Examples 3 and 4: Discrete Adjoint Equations . . . . . . . . . 46

2.6.1 Discretization Step 1: Choice of a Discretized
Forward Map . . . . . . . . . . . . . . . . . . . . . . . 47
2.6.2 Discretization Step 2: Choice of a Discretized
Objective Function . . . . . . . . . . . . . . . . . . . . 52
2.6.3 Derivation Step 0: Forward Map and Objective Function 52
2.6.4 Derivation Step 1: State-Space Decomposition . . . . . 53
2.6.5 Derivation Step 2: Lagrangian . . . . . . . . . . . . . . 54
2.6.6 Derivation Step 3: Adjoint Equation . . . . . . . . . . 56
2.6.7 Derivation Step 4: Gradient Equation . . . . . . . . . . 58
2.7 Examples 3 and 4: Continuous Adjoint Equations . . . . . . . 59
2.8 Example 5: Differential Equations, Discretized Versus
Discrete Gradient . . . . . . . . . . . . . . . . . . . . . . . . . 65
2.8.1 Implementing the Discretized Gradient . . . . . . . . . 68
2.8.2 Implementing the Discrete Gradient . . . . . . . . . . . 68
2.9 Example 6: Discrete Marching Problems . . . . . . . . . . . . 73

3 Choosing a Parameterization 79
3.1 Calibration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80
3.1.1 On the Parameter Side . . . . . . . . . . . . . . . . . . 80
3.1.2 On the Data Side . . . . . . . . . . . . . . . . . . . . . 83
3.1.3 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 84
3.2 How Many Parameters Can be Retrieved from the Data? . . . 84
3.3 Simulation Versus Optimization Parameters . . . . . . . . . . 88
3.4 Parameterization by a Closed Form Formula . . . . . . . . . . 90
3.5 Decomposition on the Singular Basis . . . . . . . . . . . . . . 91
3.6 Multiscale Parameterization . . . . . . . . . . . . . . . . . . . 93
3.6.1 Simulation Parameters for a Distributed Parameter . . 93
3.6.2 Optimization Parameters at Scale k . . . . . . . . . . . 94
3.6.3 Scale-By-Scale Optimization . . . . . . . . . . . . . . . 95
3.6.4 Examples of Multiscale Bases . . . . . . . . . . . . . . 105
3.6.5 Summary for Multiscale Parameterization . . . . . . . 108
3.7 Adaptive Parameterization: Refinement Indicators . . . . . . . 108
3.7.1 Definition of Refinement Indicators . . . . . . . . . . . 109
3.7.2 Multiscale Refinement Indicators . . . . . . . . . . . . 116
3.7.3 Application to Image Segmentation . . . . . . . . . . . 121
3.7.4 Coarsening Indicators . . . . . . . . . . . . . . . . . . . 122
3.7.5 A Refinement/Coarsening Indicators Algorithm . . . . 124
CONTENTS xiii

3.8 Implementation of the Inversion . . . . . . . . . . . . . . . . . 126

3.8.1 Constraints and Optimization Parameters . . . . . . . 126
3.8.2 Gradient with Respect to Optimization
Parameters . . . . . . . . . . . . . . . . . . . . . . . . 129
3.9 Maximum Projected Curvature: A Descent Step for Nonlinear
Least Squares . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
3.9.1 Descent Algorithms . . . . . . . . . . . . . . . . . . . . 135
3.9.2 Maximum Projected Curvature (MPC) Step . . . . . . 137
3.9.3 Convergence Properties for the Theoretical
MPC Step . . . . . . . . . . . . . . . . . . . . . . . . . 143
3.9.4 Implementation of the MPC Step . . . . . . . . . . . . 144
3.9.5 Performance of the MPC Step . . . . . . . . . . . . . . 148

4 Output Least Squares Identifiability and Quadratically

Wellposed NLS Problems 161
4.1 The Linear Case . . . . . . . . . . . . . . . . . . . . . . . . . 163
4.2 Finite Curvature/Limited Deflection Problems . . . . . . . . . 165
4.3 Identifiability and Stability of the Linearized Problems . . . . 174
4.4 A Sufficient Condition for OLS-Identifiability . . . . . . . . . . 176
4.5 The Case of Finite Dimensional Parameters . . . . . . . . . . 179
4.6 Four Questions to Q-Wellposedness . . . . . . . . . . . . . . . 182
4.6.1 Case of Finite Dimensional Parameters . . . . . . . . . 183
4.6.2 Case of Infinite Dimensional Parameters . . . . . . . . 184
4.7 Answering the Four Questions . . . . . . . . . . . . . . . . . . 184
4.8 Application to Example 2: 1D Parameter Estimation with H 1
Observation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
4.8.1 Linear Stability . . . . . . . . . . . . . . . . . . . . . . 193
4.8.2 Deflection Estimate . . . . . . . . . . . . . . . . . . . . 198
4.8.3 Curvature Estimate . . . . . . . . . . . . . . . . . . . . 199
4.8.4 Conclusion: OLS-Identifiability . . . . . . . . . . . . . 200
4.9 Application to Example 4: 2D Parameter Estimation,
with H1 Observation . . . . . . . . . . . . . . . . . . . . . . . 200

5 Regularization of Nonlinear Least Squares Problems 209

5.1 Levenberg–Marquardt–Tychonov (LMT)
Regularization . . . . . . . . . . . . . . . . . . . . . . . . . . . 209
5.1.1 Linear Problems . . . . . . . . . . . . . . . . . . . . . 211
5.1.2 Finite Curvature/Limited Deflection
(FC/LD) Problems . . . . . . . . . . . . . . . . . . . . 219
5.1.3 General Nonlinear Problems . . . . . . . . . . . . . . . 231
xiv CONTENTS

5.2 Application to the Nonlinear 2D Source Problem . . . . . . . 237

5.3 State-Space Regularization . . . . . . . . . . . . . . . . . . . . 246
5.3.1 Dense Observation: Geometric Approach . . . . . . . . 248
5.3.2 Incomplete Observation: Soft Analysis . . . . . . . . . 256
5.4 Adapted Regularization for Example 4: 2D Parameter
Estimation with H1 Observation . . . . . . . . . . . . . . . . . 259
5.4.1 Which Part of a is Constrained by the Data? . . . . . . 260
5.4.2 How to Control the Unconstrained Part? . . . . . . . . 262
5.4.3 The Adapted-Regularized Problem . . . . . . . . . . . 264
5.4.4 Infinite Dimensional Linear Stability
and Deflection Estimates . . . . . . . . . . . . . . . . . 265
5.4.5 Finite Curvature Estimate . . . . . . . . . . . . . . . . 267
5.4.6 OLS-Identifiability for the Adapted Regularized
Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 268

II A Generalization of Convex Sets 271

6 Quasi-Convex Sets 275
6.1 Equipping the Set D with Paths . . . . . . . . . . . . . . . . . 277
6.2 Definition and Main Properties of q.c. Sets . . . . . . . . . . . 281

7 Strictly Quasi-Convex Sets 299

7.1 Definition and Main Properties of s.q.c. Sets . . . . . . . . . . 300
7.2 Characterization by the Global Radius of Curvature . . . . . . 304
7.3 Formula for the Global Radius of Curvature . . . . . . . . . . 316

8 Deflection Conditions for the Strict Quasi-convexity

of Sets 321
8.1 The General Case: D ⊂ F . . . . . . . . . . . . . . . . . . . . 327
8.2 The Case of an Attainable Set D = ϕ (C) . . . . . . . . . . . 337

Bibliography 345

Index 353
 Part I

Nonlinear Least Squares

1
PART I: NONLINEAR LEAST SQUARES 3

Chapter 1 provides an overview of the book: it shows how various finite

and infinite dimensional inverse problems can be cast in the same NLS mould,
reviews the difficulties to be expected, and hints at the places in the book
where they are addressed.
Chapter 2 details the sensitivity function and adjoint state approaches for
the computation of derivatives (gradient of the objective function, Jacobian
of the forward map). Various examples of discrete adjoint state calculations
are given.
Chapter 3 is devoted to the issues of parameterization: choice of dimen-
sionless parameters and data, use of singular value decomposition of the
linearized problem to assess the number of parameters that can be retrieved
for a given error level on the data, choice of a parameterization to reduce
the number of unknown parameters, organization of the inversion code for
an easy experimentation with various parameterization, choice of an adapted
descent step to enhance robustness and performance of descent minimization
algorithms. Special care is given to the discretization of distributed para-
meters: multiscale parameterization is shown to restore optimizability for
certain class of “nicely nonlinear” inverse problems, and its adaptive vari-
ant based on refinement indicators is presented, which in addition allows
to explain the data with a small number of degrees of freedom, thus avoid-
ing overparameterization.
The Q-wellposedness of NLS problems is the subject of Chap. 4. It begins
with a summary of the properties of the linear case, which are to be gener-
alized. Two classes of NLS problems are then defined, based on properties
of the first and second directional derivatives of the direct map: the class of
finite curvature (FC) least squares problems, whose attainable set has a finite
curvature, and the subclass of finite curvature/limited deflection (FC/LD)
problems, whose attainable set is a strictly quasi-convex set introduced in
Chap. 7. Linearly stable FC/LD problems are shown to be Q-wellposed. Ap-
plication are given to the estimation of finite dimensional parameters, and
to the estimation of the diffusion parameter in 1D and 2D elliptic equation
Chapter 5 is devoted to the practically important problem of restor-
ing Q-wellposedness by regularization. The usual Levenberg–Marquardt–
Tychonov (LMT) regularization is considered first. After recalling the
convergence results to the minimum norm solution for the linear case,
when both data error and regularization parameter go to zero, we show that
these results generalize completely to the nonlinear case for FC/LD prob-
lems. For general nonlinear least square, where small  does not guarantee
4 PART I: NONLINEAR LEAST SQUARES

Q-wellposedness any more, we give an estimation of the minimum amount of

regularization that does so. Then it is shown how state-space regularization
can handle some cases where LMT regularization fails, when the attainable
set has an infinite curvature. Roughly speaking, state-space regularization
amounts to smooth the data before solving the inverse problem. Finally,
an example of desirable adapted regularization is given, where the a-priori
information brought by the regularization term can be chosen such that it
does not conflict with the information conveyed from the data by the model.
Chapter 1

Nonlinear Inverse Problems:

Examples and Difficulties

We present in this chapter the nonlinear least-squares (NLS) approach to

parameter estimation and inverse problems, and analyze the difficulties as-
sociated with their theoretical and numerical resolution.
We begin in Sect. 1.1 with a simple finite dimensional example of nonlin-
ear parameter estimation problem: the estimation of four parameters in the
Knott–Zoeppritz equations. This example will reveal the structure of inverse
problem, and will be used to set up the terminology.
Then we define in Sect. 1.2 an abstract framework for NLS problems,
which contains the structure underlying the example of Sect. 1.1. Next we
review in Sect. 1.3 the difficulties associated with the resolution of NLS prob-
lems, and hint at possible remedies and their location in the book.
Finally, Sects. 1.4–1.6 describe infinite dimensional parameter estimation
problems of increasing difficulty, where the unknown is the source or diffusion
coefficient function of an elliptic equation, to which the analysis developed
in Chaps. 2–5 will be applied.
Examples of time marching problems are given in Sects. 2.8 and 2.9 of
Chap. 2

G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 5

and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 1, c Springer Science+Business Media B.V. 2009
6 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

1.1 Example 1: Inversion of Knott–Zoeppritz

Equations
We begin with an example that is intrinsically finite dimensional, where the
model consists in a sequence of simple algebraic calculations. The problem
occurs in the amplitude versus angle (AVA) processing of seismic data, where
the densities ρj , compressional velocity VP,j , and shear velocities VS,j on each
side j = 1, 2 of an interface are to be retrieved from the measurement of the
compressional (P–P) reflection coefficient Ri at a collection θi , i = 1, . . . , q of
given incidence angles.
The P-P reflection coefficient R at incidence angle θ is given by the Knott–
Zoeppritz equations ([1], pp. 148–151). They are made of quite complicated
algebraic formulas involving many trigonometric functions. An in-depth anal-
ysis of the formula shows that R depends in fact only on the following four
dimensionless combinations of the material parameters [50]:
⎧
⎪ ρ1 − ρ2
⎪
⎪ eρ = (density contrast),
⎪
⎪ ρ1 + ρ2
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
2
VP,1 − VP,2
2
⎪
⎪ eP = 2 (P-velocity contrast),
⎪
⎪ 2
⎪
⎪ VP,1 + VP,2
⎨
(1.1)
⎪
⎪
⎪
⎪
2
VS,1 − VS,2
2
⎪
⎪ eS = 2 (S-velocity contrast),
⎪
⎪ 2
VS,1 + VS,2
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
⎪
2
VS,1 2
+ VS,2 1 1
⎪
⎩ χ= ( 2 + 2 ) (background parameter),
2 VP,1 VP,2
1.1. EXAMPLE 1: INVERSION OF KNOTT–ZOEPPRITZ 7

so that R is given by the relatively simple sequence of calculations:

⎧
⎪
⎪ e = eS + eρ
⎪
⎪
⎪
⎪ f = 1 − e2ρ
⎪
⎪
⎪
⎪ S = χ(1 + eP )
⎪ 1
⎪
⎪
⎪ S 2 = χ(1 − eP )
⎪
⎪
⎪
⎪ T1 = 2/(1 − eS )
⎪
⎪
⎪ T2 = 2/(1 + eS )
⎪
⎪
⎪
⎪
⎪ q 2 = S 2
1 sin θ
⎪
⎪
⎪
⎪ M1 = S1 − q 2
⎪
⎪
⎪
⎨ M2 =  S2 − q 2
N1 = T1 − q 2 (1.2)
⎪
⎪
⎪
⎪ N2 = T2 − q 2
⎪
⎪
⎪
⎪ D = eq 2
⎪
⎪
⎪
⎪ A = eρ − D
⎪
⎪
⎪
⎪ K =D−A
⎪
⎪
⎪
⎪ B =1−K
⎪
⎪
⎪
⎪ C =1+K
⎪
⎪
⎪
⎪ P = M1 (B 2 N1 + f N2 ) + 4eDM1 M2 N1 N2
⎪
⎪
⎪
⎪ Q = M2 (C 2 N2 + f N1 ) + 4q 2 A2
⎪
⎩
R = (P − Q)/(P + Q).

We call parameter vector the vector

x = (eρ , eP , eS , χ) ∈ IR4 (1.3)

of all quantities that are input to the calculation, and state vector the vector

y = (e, f, S1 , S2 , . . . , P, Q, R) ∈ IR19 (1.4)

made of all quantities one has to compute to solve the state equations (here
at a given incidence angle θ).
We have supposed in the above formulas that the incidence angle θ is
smaller than the critical angle, so that the reflection coefficient R computed
by formula (1.2) is real, but the least squares formulation that follows can
be extended without difficulty to postcritical incidence angles with complex
reflection coefficients.
8 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

If now we are given a sequence Rim , i = 1, . . . , q of “measured” reflection

coefficients corresponding to a sequence θ1 , . . . , θq of known (precritical)
incidence angles, we can set up a data vector
z = (R1m , . . . , Rqm ) ∈ IRq , (1.5)
which is to be compared to the output vector
v = (R1 , . . . , Rq ) ∈ IRq (1.6)
of reflection coefficients computed by formula (1.2). We can write
⎡ ⎤
y1
⎢ . ⎥
⎢ ⎥
v=M⎢ ⎢ . ⎥,
⎥ (1.7)
⎣ . ⎦
yq
with yi given by (1.4) for θ = θi , i = 1, . . . , q. The operator M that selects
the last component Ri of each vector yi in the state vector (y1 , . . . , yq ) ∈ IR19q
is called the measurement or observation operator. It is here a simple matrix
with q rows, p = 19q columns and 0 or 1 entries.
We can now set up the problem of estimating x ∈ IR4 from the knowledge
of z ∈ IRq . To compare v and z, we have to choose first for each i = 1, . . . , q
a unit Δzi to measure how much the model output vi = Ri deviates from the
data zi = Rim . The misfit of the model parameter x to the data z can then
be defined by
q
1 |vi − zi |2
J(x) = , (1.8)
2 i=1 Δzi2
with v given by (1.7). It is expected that minimization of J with respect to
x will allow to carry over to x some of the information we have on z. In
practice, one has also some a-priori information on the parameters eρ , eP , eS ,
and χ, at least some bounds. Hence, one restrains the minimization of J to
the set of admissible parameters
C = {x = (eρ , eP , eS , χ) | xi,min ≤ xi ≤ xi,max , i = 1 . . . 4}, (1.9)
where xi,min and xi,max are the given lower and upper bounds.
In the case where the measurement errors are independent Gaussian vari-
ables with known standard deviations σi , one can choose Δzi = σi , in which
case the minimization of J over C produces a maximum likelihood estimator
for x.
1.2. AN ABSTRACT NLS INVERSE PROBLEM 9

Remark
Other choices are possible forthe data misfit: one could, for example, replace
the Euclidean
q r 1/r norm v = ( q 2 1/2
1 i)
v on the data space by the r-norm v =
( 1 vi ) for some r > 1, as such a norm is known to be less sensitive to
outliers in the data for 1 < r < 2. The results of Chaps. 2 and 3 on derivation
and parameterization generalize easily to these norms. But the results on Q-
wellposedness and regularization of Chap. 4 and 5 hold true only for the
Euclidean norm.

1.2 An Abstract NLS Inverse Problem

The inversion of the Knott–Zoeppritz equations presented in Sect. 1.1 is a
special case of the following abstract setting, which will be used throughout
this book to handle NLS inverse problems:
• E will be the parameter space
• C the set of admissible parameters
• ϕ : C  F the forward or direct or input–output map
• F the data space
• z ∈ F the data to be inverted
The inverse problem consists then in searching a parameter x ∈ C whose
image ϕ(x) by the forward map ϕ is the data z. But in applications z will
never belong to the attainable set ϕ(C), because ϕ never represents abso-
lutely accurately the process under study (model errors), and because of the
measurement errors on z. The inverse problems has then to be solved in the
least squares sense as an NLS problem:
1
x̂ minimizes J(x) = ϕ(x) − z2F over C. (1.10)
2
This formulation contains the inversion of the Knott–Zoeppritz equations
(Sect. 1.1). It suffices for that to choose
⎧ 
⎪
⎪ E = IR4 with the norm xE = ( 41 x2i )1/2 ,
⎨
C given by (1.9),  (1.11)
⎪
⎪ F = IRq with the norm vF = ( q1 vi2 /Δzi2 )1/2 ,
⎩
ϕ : x  v with v given by (1.7).
10 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

But it is much more general, and allows also to handle infinite dimensional
problems, as we shall see in Sects. 1.4–1.6. So we shall suppose throughout
this book that the following minimum set of hypothesis holds:
⎧
⎪
⎪ E = Banach space, with norm  E ,
⎪
⎪
⎪
⎪ C ⊂ E with C convex and closed,
⎪
⎪
⎨ F = Hilbert space, with norm  F ,
z ∈ F (1.12)
⎪
⎪
⎪
⎪ ϕ : C  F is differentiable along segments of C,
⎪
⎪
⎪
⎪ and : ∃ αM ≥ 0 s.t. ∀x0 , x1 ∈ C, ∀t ∈ [0, 1]
⎩
Dt ϕ((1 − t)x0 + tx1 ) ≤ αM x1 − x0 .
These hypothesis are satisfied in most inverse problems – and in all those we
address in this book, including of course the inversion of the Knott–Zoeppritz
equations of Sect. 1.1 – but they are far from being sufficient to ensure good
properties of the NLS problem (1.10).

1.3 Analysis of NLS Problems

We review in this section the theoretical and practical difficulties associated
with the theoretical and numerical resolution of (1.10), and present the tools
developed in this book for their solution.

1.3.1 Wellposedness
The first question is wellposedness: does (1.10) has a unique solution x̂ that
depends continuously on the data z? It is the first question, but it is also a
difficult one because of the nonlinearity of ϕ. It is most likely to be answered
negatively, as the word ill-posed is almost automatically associated with in-
verse problem. To see where the difficulties arise from, we can conceptually
split the resolution of (1.10) into two consecutive steps:
Projection step: given z ∈ F , find a projection X̂ of z on ϕ(C)
Preimage step: given X̂ ∈ ϕ(C), find one preimage x̂ of X̂ by ϕ
Difficulties can – and usually will – arise in both steps:
1. There can be more than one preimage x̂ of X̂ if ϕ is not injective
2. Even if ϕ is injective, its inverse ϕ−1 : X̂ → x̂ may not be continuous
over ϕ(C)
1.3. ANALYSIS OF NLS PROBLEMS 11

3. The projection X̂ does not necessarily exist if ϕ(C) is not closed

4. The projection on ϕ(C) can be nonunique and not continuous as ϕ(C)

is possibly nonconvex because of the nonlinearity of ϕ

We consider first the unrealistic case of attainable data as where there

are no measurement or model error:

z = X̂, (1.13)

and recall the corresponding identifiability and stability properties, which

concern only the preimage step:

Definition 1.3.1 Let ϕ and C be given. The parameter x is

• Identifiable if ϕ is injective on C, that is,

x0 , x1 ∈ C and ϕ(x0 ) = ϕ(x1 ) =⇒ x0 = x1 (1.14)

• Stable if there exists a constant k ≥ 0 such that

x0 − x1  ≤ k ϕ(x0 ) − ϕ(x1 ) ∀x0 , x1 ∈ C (1.15)

The vast majority of available results concern the case of attainable

data only: most of them are identifiability results (see, e.g., [46] for the
estimation of coefficients in partial differential equations from distributed or
boundary observation), and a much smaller number are stability results (see,
e.g., [47, 70]).
But these results fall short to imply the wellposedness of NLS problem
(1.10), as they do not say anything when z is outside the attainable set ϕ(C).
In the more realistic case where one takes into account the model and
measurement errors, the data z can be outside the output set ϕ(C), and the
problem of the projection on ϕ(C) arises.
In the linear case, difficulty 3 (possible lack of existence of the projection)
already exists. But it can be cured simultaneously with difficulties 1 and 2 by
regularization (see LMT-Regularization in Sect. 1.3.4), because the projection
on the convex set ϕ(C), when it exists, is necessarily unique and Lipschitz
continuous!
But for nonlinear inverse problems, because of the possible nonunique-
ness and noncontinuity of the projection (difficulty 4), one cannot expect
12 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

wellposedness to hold for all z of F . So we have to limit the existence,

uniqueness, and stability requirement to some neighborhood ϑ of ϕ(C) in F .
To handle this situation, we introduce in Chap. 6 a first generalization
of convex sets to quasi-convex sets D ⊂ F , which precisely possess a neigh-
borhood ϑ on which the projection, when it exists, is unique and Lipschitz
stable [20].

1.3.2 Optimizability
The next question that arises as soon as one considers solving (1.10) on a
computer is the possibility to use performant local optimization techniques
to find the global minimum of the objective function J. It is known that
such algorithms converge to stationary points of J, and so it would be highly
desirable to be able to recognize the case where J has no parasitic stationary
points on C, that is, stationary points where J is strictly larger than its
global minimum over C. It will be convenient to call unimodal functions with
this property, and optimizable the associated least squares problem. Convex
functions, for example, are unimodal, and linear least squares problems are
optimizable.
Stationary points of J are closely related to stationary points of the dis-
tance to z function over ϕ(C), whose minimum gives the projection of z
on ϕ(C). Hence we introduce in Chap. 7 a generalization of convex sets to
strictly quasi-convex (s.q.c.) sets D ⊂ F , which possess a neighborhood ϑ on
which the distance to z function is unimodal over D [19].
Sufficient conditions for a set to be s.q.c. are developed in Chap. 8, in
particular, in the case of interest for NLS problems where D is the attainable
set ϕ(C).
As it will turn out (Theorem 7.2.10), s.q.c. sets are indeed quasi-convex
sets, and so requiring that the output set is s.q.c. will solve both the well-
posedness and optimizability problems at once.

1.3.3 Output Least Squares Identifiability

and Quadratically Wellposed Problems
Wellposedness and optimizability are combined in the following definition
[16, 17], which tries to carry over the good properties of the linear case to
the nonlinear case:
1.3. ANALYSIS OF NLS PROBLEMS 13

Definition 1.3.2 Let ϕ and C be given. The parameter x is OLS-identifiable

in C from a measurement z of ϕ(x) – or equivalently the NLS problem (1.10)
is Quadratically (Q-)wellposed – if and only if ϕ(C) possesses an open neigh-
borhood ϑ such that
(i) Existence and uniqueness: for every z ∈ ϑ, problem (1.10) has a unique
solution x̂
(ii) Unimodality: for every z ∈ ϑ, the objective function x  J(x) has no
parasitic stationary point
(iii) Local stability: the mapping z  x̂ is locally Lipschitz continuous from
(ϑ, · F ) to (C, · E )
The class of FC problems (Definition 4.2.1) represents a first step to-
ward Q-wellposedness. The subclass of finite curvature/limited deflection
(FC/LD) problems (Definition 4.2.2) represents a further step in this direc-
tion: in this class, the images by ϕ of the segments of C turn less than π/2
(Corollary 4.2.5) – which can always be achieved by reducing the size of the
admissible set C – and the attainable set is s.q.c. (Corollary 4.2.5). The pro-
jection on the attainable set is then, when it exists, unique, unimodal, and
stable (Proposition 4.2.7) for the arc length “distance” (4.27) on ϕ(C), and
the solution set is closed and convex.
Combining this with the linear identifiability and/or stability properties
of Sect. 4.3, one obtains Theorem 4.4.1, our main result, which gives a suf-
ficient conditions for Q-wellposedness or OLS-identifiability – and hence for
both wellposedness and optimizability. These sufficient conditions are writ-
ten in terms of the first and second directional derivatives of ϕ, and give an
estimation of the localization and stability constants, and of the size of the
neighborhood ϑ.
The case of finite dimensional parameters is considered in Sect. 4.5, where
it is shown in Theorem 4.5.1 that linear identifiability (i.e., identifiability
of the linearized problem) implies local (i.e., for small enough C) OLS-
identifiability or Q-wellposedness. This shows the importance of (linear) iden-
tifiability results for the analysis of finite dimensional NLS problems.
The techniques available for the determination of the curvature, deflec-
tion, and stability constants required for the application of Theorem 4.4.1
are reviewed in Sect. 4.7. These quantities can sometimes be determined by
calculus, but can also be approximated numerically when the number of pa-
rameters is small.
14 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

The sufficient conditions are then applied in Sects. 4.8 and 4.9 to analyze
the Q-wellposedness of the estimation of the diffusion coefficient in one- and
two-dimensional elliptic equations, as described in Sects. 1.4 and 1.6, when
an H 1 measurement of the solution is available.

1.3.4 Regularization
OLS-identifiability is a very strong property, so one cannot expect that
it will apply to the original NLS problem, when this one is known to be
ill-posed.
On the other hand, the only NLS problems that should reasonably be
solved on the computer at the very end are those where OLS-identifiability
holds, as this ensures the following:
– Any local optimization algorithm will converge to the global minimum
– The identified parameter is stable with respect to data perturbations
which are the only conditions under which one can trust the results of the
computation.
Hence the art of inverse problems consists in adding information of various
nature to the original problem, until a Q-wellposed NLS problem is achieved,
or equivalently OLS-identifiability holds: this is the regularization process.
The sufficient conditions for OLS-identifiability of Chap. 4 will then allow to
check whether the regularized problem has these desirable properties, and
help to make decisions concerning the choice of the regularization method.

Adapted Regularization
If the added a-priori information is (at least partly) incompatible with the
one conveyed from the data by the model, the optimal regularized param-
eter has to move away from its true value to honor both information. So
one should ideally try to add information that tends to conflict as little as
possible with the one coming from the data. Such an adapted regularization
is recommended whenever it is possible. But it requires a refined analysis of
the forward map ϕ, which is not always possible, and has to be considered
case by case. We give in Sect. 5.4 an example of adapted regularization for the
estimation of the diffusion coefficient in a two-dimensional elliptic equation.

But, in general, this hand-taylored approach is impossible, and the infor-

mation has to be added by brute force:
1.3. ANALYSIS OF NLS PROBLEMS 15

Regularization by Parameterization
This is one of the most commonly used approach: instead of searching for
the parameter x, one searches for a parameter xopt related to x by
x = ψ(xopt ), xopt ∈ Copt with ψ(Copt ) ⊂ C, (1.16)
where ψ is the parameterization map, and Copt is the admissible parameter
set for the optimization variables xopt (Sect. 3.3). Satisfying the inclusion
ψ(Copt ) ⊂ C is not always easy, and it has to be kept in mind when choosing
the parameterization mapping ψ.
Parameterization is primarily performed to reduce the number of un-
known parameters, but it has also the effect, when x is a function, to impose
regularity constraints, which have usually a stabilizing effect on the inverse
problem. The regularized problem is now
1
x
opt minimizes J ◦ ψ(xopt ) = ϕ(ψ(xopt )) − z2F over Copt . (1.17)
2
We address in Chap. 3 some practical aspects of parameterization:
– How to calibrate the parameters and the data
– How to use the singular value decomposition of the Jacobian D = ϕ(x)
to estimate the number of independent parameters that can be estimated for
a given level of uncertainty on the data
– How to use multiscale approximations and/or refinement indicators in
order to determine an adaptive parameterization, which makes the prob-
lem optimizable, explains the data up to the noise level, and does not lead
to over-parameterization
– How to organize the inversion code to make experimentation with var-
ious parameterizations more easy
– How to use in minimization algorithms a descent step adapted to the
nonlinear least square structure of the problem

Regularization by Size Reduction of C

Another natural way to add a-priori information is to search for the param-
eter x in a smaller set:
C ⊂ C, (1.18)
and to define the regularized problem simply by
1
x̂ minimizes J(x) = ϕ(x) − z2F over C. (1.19)
2
16 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

The set C can be defined as follows:

• Either by adding new constraint (s). when C is made of functions de-
fined over a domain Ω ∈ IRn , on can want to add the a priori infor-
mation that the parameter is bounded in a space of smooth functions.
This can be implemented by choosing a subspace E ⊂ E with the desired
smoothness, equipped with the stronger norm  · E , and by restricting
the search for x to the set

C = {x ∈ E | x ∈ C and xE ≤ M}. (1.20)

Such smoothness constraints are often used to ensure that the regular-
ized problem (1.19) is a FC least squares problem. FC problems were
introduced in [28] under the less descriptive name of “weakly nonlinear”
problems, and are defined in Sect. 4.2 (Definition 4.2.1). They represent
a first necessary step towards Q-wellposedness.
For example, in the estimation of the diffusion coefficient a in a two-
dimensional elliptic equation (Sect. 1.6), the natural admissible set C
is defined by (1.66):

C = {a : Ω  IR | 0 < am ≤ a(ξ) ≤ aM a.e. on Ω}.

The first regularization step we shall take in Sect. 4.9 will be to replace
C by the set defined in (4.106), which we rewrite here as:

C = { a ∈ E | am ≤ a(ξ) ≤ aM ∀ξ ∈ Ω, (1.21)
|a(ξ1 ) − a(ξ0 )| ≤ bM ξ1 − ξ0  ∀ξ0 , ξ1 ∈ Ω },

where E is a (sub)space of the space of uniformly Lipschitz-continuous

functions over Ω. These constraints will ensure (Theorem 4.9.4, point
two) that (1.19) is a finite curvature problem.
• Or by tightening existing constraints. This is particularly useful in the
case where (1.10) is already an FC problem. In this case, Corollary 4.2.5
shows that tightening the constraints – that is, reducing the size of C
– will produce a FC/LD problem (Definition 4.2.2), whose attainable
set ϕ(C ) is s.q.c., so that the projection on ϕ(C ) is wellposed on some
of its neighborhoods (Proposition 4.2.7). This will make the problem
amenable to LMT-regularization (see LMT regularization below and
Sect. 5.1.2).
1.3. ANALYSIS OF NLS PROBLEMS 17

For example, the second regularization step we shall take in Sect. 4.9
for the same diffusion coefficient problem as earlier will be to reduce,
in the admissible set (1.21), the interval between the lower and upper
bounds on a until the condition
π
aM − am ≤ am (1.22)
4
is satisfied. Proposition 4.9.3 will then ensure that the deflection condi-
tion is satisfied, and make the problem ready for LMT regularization.

Levenberg–Marquardt–Tykhonov (LMT) Regularization

This approach is the oldest one, as the work by Levenberg [54] goes back
to the forties, and that of Marquardt [62] to the sixties. It was popularized
in the seventies by Tikhonov and the Russian school [75, 63], who devel-
oped the theory for infinite dimensional systems. It consists in stabilizing the
original problem (1.10) by providing additional information in the form of a
regularizing functional to be added to the least-squares functional J(x). We
shall limit ourselves to the case of quadratic regularizing functionals, which
are by far the most widely used, and suppose that

the parameter space E is an Hilbert space, (1.23)

and that we have chosen

x0 ∈ C and  > 0, (1.24)

where x0 is an a priori guess of the parameter, and  > 0 is the regulariza-

tion parameter. The a priori guess x0 represents the additional information
provided, and  measures the confidence one has in this a priori guess. The
LMT-regularized version of (1.10) is

2
x̂ minimizes J (x) = J(x) + x − x0 2E over C. (1.25)
2
Its properties have been studied extensively when ϕ is linear, and we refer,
for example, to the monographs [8, 43, 59, 63, 38, 55, 12]. The main result for
linear problems is that the regularized problem (1.25) is wellposed as soon as
 > 0, and that the solution x̂,δ of (1.25) corresponding to noise corrupted
data zδ converges to the x0 -minimum norm solution of (1.10) – that is, the
solution of (1.10) closest to x0 , provided that such a solution exists, and
18 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

that  goes to zero more slowly than the noise level δ = zδ − z. We give
at the beginning of Chaps. 4 and 5 a short summary of properties of linear
least-squares problems.
We shall see that all the nice properties of LMT regularization for the
linear case extend to the class of FC/LC problems, where the attainable set
has a FC, and the size of the admissible set has been chosen small enough for
the deflection condition to hold (see Regularization by size reduction above).
LMT regularization of FC/LC problems is studied in Sect. 5.1 and is applied
in Sect. 5.2 to the the source identification problem of Sect. 1.5.
But for general nonlinear problems, the situation is much less brilliant.
In particular, there is no reason for the regularized problem (1.25) to be
Q-wellposed for small ’s! Hence when  → 0, problem (1.25) may have more
than one solution or stationary point: convergence results can still be ob-
tained for sequences of adequately chosen solutions [37, 67], but optimiz-
ability by local minimization algorithms is lost. We give in Sect. 5.1.3 an
estimation of the minimum amount of regularization to be added to restore
Q-wellposedness.

State-Space Regularization
When the original problem (1.10) has no finite curvature, which is alas almost
a generic situation for infinite dimensional parameters, it is difficult to find a
classical regularization process that ensures Q-wellposednes of the regularized
problem for small ’s. One approach is then to decompose (see Sect. 1.3.5)
the forward map ϕ into the resolution of a state equation (1.33) followed by
the application of an observation operator (1.34)
ϕ = M ◦ φ, (1.26)
where φ : x  y is the solution mapping of the state equation, and M :
y  v is the observation operator.
The state-space regularized version of problem (1.10) is then [26, 29]
2
x̂ minimizes J (x) = J(x) + φ(x) − ŷ 2 over C ∩ B , (1.27)
2
where B is a localization constraint, and ŷ is the solution of the auxiliary
unconstrained problem
1 2
ŷ minimizes M(y) − z2F + y − y0 2Y over Y, (1.28)
2 2
where y0 is an a priori guess for the state vector.
1.3. ANALYSIS OF NLS PROBLEMS 19

For example, in the parameter estimation problems of Sects. 1.4 and 1.6,
one can think of x as being the diffusion coefficient a, of y as being the solu-
tion of the elliptic equation in the Sobolev space H 1 (Ω), and of v as being a
measure of y in L2 (Ω). State-space regularization would then consist in using
first LMT regularization to compute a smoothed version y ∈ H 1 (Ω) of the
data z ∈ L2 (Ω) by solving the auxiliary problem (1.28), and then to use this
smoothed data as extra information in the regularized NLS problem (1.27).
Smoothing data before inverting them has been a long-time favorite in
the engineering community, and so state-space regularization is after all not
so unnatural. It is studied in Sect. 5.3. The main result is demonstrated in
Sect. 5.3.1: problems (1.27) and (1.28) remain Q-wellposed when  → 0, pro-
vided one can choose the state-space Y as follows:
– The problem of estimating x ∈ C from a measure of φ(x) in Y is
Q-wellposed (this requires usually that the the norm on Y is strong enough)
– The observation mapping M is linear and injective
This covers, for example, the case of the parameter estimation problems of
Sects. 1.4 and 1.6 when only L2 observations are available.
A partial result for the case of point and/or boundary measurements,
which do not satisfy the injectivity condition, is given in Sect. 5.3.2.

Common Features
All the regularized problems can be cast into the same form as the original
problem (1.10). It suffices to perform the following substitutions in (1.10):
⎧
⎪
⎪ C ← Copt
⎪
regularization ⎪ ⎨ ϕ(x) ← ϕ(ψ(xopt ))
by F ← F (1.29)
⎪
parameterization ⎪⎪
⎪   F ←   F
⎩
z ← z
⎧
⎪
⎪ C ← C
⎪
⎪
⎨ ϕ(x) ← ϕ(x)
reduction
F ← F (1.30)
of C ⎪
⎪
⎪
⎪  F ←  F
⎩
z ← z
20 CHAPTER 1. NONLINEAR INVERSE PROBLEMS
⎧
⎪
⎪ C ← C
⎪
⎪ ←
⎨ ϕ(x) (ϕ(x), x)
LMT
F ← F = F × E (1.31)
regularization ⎪
⎪ 2
⎪
⎪  F ←  F = ( 2F +  2E )1/2
⎩ 2
z ← (z, x0 )
⎧
⎪
⎪ C ← C
⎪
⎪ ←
⎨ ϕ(x) (M φ(x), φ(x))
state-space
F ← F = F × Y (1.32)
regularization ⎪
⎪ 2
⎪
⎪  F ←  F = ( 2F +  2Y )1/2
⎩ 2
z ← (z, y )

If more than one regularization technique are applied simultaneously, the

property still holds with obvious adaptions.
Hence the same abstract formulation (1.10) can be used to represent the
original NLS problem as well as any of its regularized versions (1.19), (1.17),
(1.25), or (1.27).

1.3.5 Derivation
The last difficulty we shall address is a very practical one, and appears as
soon as one wants to implement a local optimization algorithm to solve a
NLS problem of the form (1.10): one needs to provide the optimization code
with the gradient ∇J of the objective function, or the derivative or Jacobian
D = ϕ (x) of the forward map.
To analyze this difficulty, it is convenient to describe ϕ at a finer level by
introducing a state-space decomposition. So we shall suppose, when needed,
that v = ϕ(x) is evaluated in two steps:
⎧
⎨ given x ∈ C solve:
e(x, y) = 0 (1.33)
⎩
with respect to y in Y ,
followed by

set: v = M(y), (1.34)

where now

• y is the state, Y is an affine state-space with tangent vector space δY

1.4. EXAMPLE 2: 1D ELLIPTIC PARAMETER ESTIMATION 21

• e(x, y) = 0 is the state equation, where e maps C × Y into the right-

hand side space Z. For finite dimensional problems, δY = Z = IRp .
But it will be convenient to use distinct spaces δY and Z for infinite
dimensional problems
• M : Y  F is the (possibly nonlinear) measurement or observation
operator
We shall always suppose that the following minimum set of hypothesis holds:
⎧
⎪
⎪ Y is an affine space,
⎪
⎪
⎨ its tangent space δY is equipped with the norm  Y ,
∀x ∈ C, (1.33) has a unique solution y ∈ Y, (1.35)
⎪
⎪
⎪
⎪ ∀x ∈ C, the operator ∂e/∂y(x, y) ∈ L(δY , Z ) is invertible,
⎩
M is a continuously derivable application from Y to F .
For linear state equations, the invertibility of ∂e/∂y(x, y) is equivalent to
wellposedness of the state equation. For nonlinear problems, the condition
is required if one wants to solve the state equation by an iterative Newton
scheme. Hence it will be satisfied in practice as soon as the forward model
under consideration is stable and numerically solvable.
For example, if one wants that (1.33) and (1.34) define the forward map ϕ
associated with the Knott–Zoeppriz equations of Sect. 1.1, one has to choose
⎧
⎪
⎪ e(x, y) = 0 to be (1.2) repeated q times ,
⎨
y = (ei , fi . . . Qi , Ri , i = 1 . . . q),
(1.36)
⎪
⎪ Y = δY = Z = IR19q ,
⎩
M defined by (1.7).
We describe and compare in Chap. 2 the sensitivity functions and adjoint
state approaches for the computation of D and ∇J. Though calculating a
derivative is a simple mathematical operation, it can become quite intricate
when the function to derivate is defined through a set of equations, and so
we give a step-by-step illustration of these calculations on various examples.

1.4 Example 2: 1D Elliptic Parameter

Estimation Problem
Our second example is infinite dimensional: it consists in estimating the co-
efficient a as a function of the space variable ξ in the one-dimensional elliptic
22 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

equation:
− (auξ )ξ = gj δ(ξ − ξj ), ξ ∈ Ω, (1.37)
j∈J

over the domain Ω = [0, 1], with a right-hand side made of Dirac sources:
⎧
⎨ ξj denotes the location of the jth source,
gj ∈ IR denotes the amplitude of the jth source, (1.38)
⎩
J is a finite set of source indexes.
We complement (1.37) with Dirichlet boundary conditions:
u(0) = 0, u(1) = 0. (1.39)
Equations (1.37) and (1.38) models, for example, the temperature u in
a one-dimensional slab at thermal equilibrium, heated by point sources of
amplitude gi at locations xi , and whose temperature is maintained equal to
zero at each end, in which case a is the thermal diffusion coefficient.
A physically and mathematically useful quantity is
q(ξ) = −a(ξ) uξ (ξ), ξ ∈ Ω, (1.40)
which represents the (heat) flux distribution over Ω, counted positively in
the direction of increasing ξ’s. Equations (1.37) and (1.38) are simple in the
sense that the flux q can be computed by a closed form formula
 − H(ξ),
q(ξ) = H (1.41)
where  ξ
H(ξ) = gj δ(ξ − ξj ) dξ (1.42)
0 j∈J

is the primitive of the right-hand side, and where

 1 −1
 = 0 a (ξ) H(ξ) dξ ∈ IR
H (1.43)
1 −1
0
a (ξ) dξ

is an a−1 -weighted mean of H. The function H is piecewise constant on Ω,

and so is also q. The solution u is then given by
uξ (ξ) = a−1 (ξ)q(ξ), (1.44)
1.4. EXAMPLE 2: 1D ELLIPTIC PARAMETER ESTIMATION 23
 ξ
u(ξ) = a−1 (ζ)q(ζ) dζ. (1.45)
0
We see on (1.41) through (1.45) that uξ and u depend almost linearly on
a−1 , which suggests that the search for a−1 could be better behaved than the
search for a. So we decide to choose b = a−1 for parameter in this example,
and define the parameter space and admissible parameter set as
E = L2 (Ω), (1.46)
C = {b : Ω  IR | 0 < bm ≤ b(ξ) ≤ bM a.e. on Ω}, (1.47)
where bm and bM are given lower and upper bounds on b = a−1 .
Then for any b ∈ C, the state equations (1.37) and (1.39) have a unique
solution u in the state-space:
Y ={w ∈ L∞ (Ω) | w  ∈ L∞ (Ω) , w(0) = 0 , w(1) = 0}, (1.48)
and because of the boundary conditions w(0) = 0 , w(1) = 0, the Poincaré
inequality |v|L2 ≤ |v |L2 holds in Y . Hence we can equip Y with the norm
vY = |v  |L2 . (1.49)
To estimate b, we suppose we are given an element z of the data space F =
L2 (Ω), which provides us with some information on the temperature u.
We consider first the (quite unrealistic) case of a “distributed H 1 -
observation,” where one is able to measure the derivative of u all over
Ω, so that z is a measure of u . This corresponds to the observation operator
and forward map:
M : u ∈ Y  v = u ∈ F, (1.50)
ϕ : b ∈ C  v = u ∈ F. (1.51)
Here M is an isometry from Y onto F , and so the observation does not
incur any loss of information: it is the most favorable case. We shall prove in
Sect. 4.8 that the associated NLS problem (1.10) is a linearly stable FC/FD
problem, and hence Q-wellposed, on the subset D of C defined in (4.90),
provided the size of D is small enough (Sect. 4.8.4). This is an example of
regularization by size reduction of the admissible set.
The case of a weaker “distributed L2 -observation,” where one measures
only u over Ω:
M : u ∈ Y  v = u ∈ F, (1.52)
ϕ : b∈C  v=u∈F (1.53)
24 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

is considered in Sect. 5.3, where it is proven that the combination of the

previous Q-wellposedness results for H 1 observation with the state-space reg-
ularization approach (1.27) and (1.28) can still produces a sequence of Q-
wellposed problems.
For more realistic boundary or point measurements, the problem of the
Q-wellposedness of the NLS problems (1.10) or its regularized versions is
completely open.

1.5 Example 3: 2D Elliptic Nonlinear Source

Estimation Problem
We consider the nonlinear elliptic equation:
⎧
⎨ −Δu + k(u) = f in Ω,
⎪
u = 0 on a part ∂ΩD of ∂Ω, (1.54)
⎪
⎩ ∂u = g on ∂Ω = ∂Ω \ ∂Ω ,
N D
∂ν
where
• Δ is the Laplacian with respect to the space variables ξ1 · · · ξm

• Ω ⊂ IRm is a domain with boundary ∂Ω

• ∂ΩD and ∂ΩN form a partition of ∂Ω

• and ν is the outer normal to Ω

We make the following hypothesis on the nonlinearity k, the right-hand
sides f , g and the space domain Ω:

k ∈ W 2,∞ (IR), is nondecreasing,
(1.55)
with k(0) = 0, k  (ζ) ≥ 0 ∀ζ ∈ IR,

f ∈ L2 (Ω), g ∈ L2 (∂ΩN ) (1.56)


Ω ⊂ IRm , m = 1, 2, or 3, with C 1,1 boundary ∂Ω,
(1.57)
∂ΩD nonempty and both open and closed in ∂Ω.
The last condition in (1.57) means that there are no point on ∂Ω, where
the Dirichlet boundary ∂ΩD and the Neumann boundary ∂ΩN meet (this
1.5. EXAMPLE 3: 2D ELLIPTIC NONLINEAR SOURCE 25

excludes, e.g., the case where ∂ΩD and ∂ΩN are the two halves of the same
circle). In the two-dimensional case, it is satisfied, for example, if ∂ΩN is the
external boundary of Ω, and ∂ΩD is the boundary of a hole in the domain,
as the boundary of a source or sink, for example.
The elliptic equation (1.54) is nonlinear, but with a nonlinearity in the
lowest order term only. It admits a unique solution u ∈ Y [56], where
∂w
Y = {w ∈ L2 (Ω) | ∈ L2 (Ω) , i = 1 . . . m , w = 0 on ∂ΩD }. (1.58)
∂ξi
The source problem consists here in estimating the source terms f and/or
g of the elliptic problem (1.54) from a measurement z in the data space
F = L2 (Ω) of its solution u (the observation operator M is then simply the
canonical injection from the subspace Y of H 1 into L2 ). We shall denote by
def
C ⊂ E = L2 (Ω) × L2 (∂ΩN ), closed, convex, bounded (1.59)
the set of admissible f and g, to be chosen later, and by
def
z ∈ F = L2 (Ω), (1.60)
a (noisefree) measure of the solution u of (1.54).
Estimation of f and g in C from the measure z of u amounts to solve the
nonlinear least-squares problem
   
ˆ ĝ minimizes 1 ϕ(f, g) − z 2 2
f, over C, (1.61)
2 L (Ω)

where
ϕ : (f, g) ∈ C  u ∈ F = L2 (Ω) solution of (1.54) (1.62)
is the forward map, which is obviously injective. Hence one is here in the
case of identifiable parameters.
Given a sequence zn ∈ L2 (Ω), n = 1, 2 . . . , of noise corrupted measure-
ments of z and another sequence n > 0 of regularization coefficients such
that
|zn − z|L2 (Ω) → 0, n → 0,
and the LMT-regularization of (1.61) is
⎧  
⎪
⎪ fˆn , ĝn minimizes
⎨  2
1   2  2 2  2
ϕ(f, g) − zn  + n f − f0 L2 (Ω) + n g − g0 L2 (∂Ω ) (1.63)
⎪
⎪
⎩ 2 L2 (Ω) 2 2 N

over C,
26 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

We show in Sect. 5.2 that (1.61) is a FC problem. This will imply, under
adequate hypothesis on the size of C, that it is also a FC/LD problem.
 
Hence its LMT-regularization (1.63) is Q-wellposed for large n, and fˆn , ĝn
 
will converge to the solution fˆ, ĝ of (1.61).

1.6 Example 4: 2D Elliptic Parameter

Estimation Problem
In this last example, we consider a two-dimensional linear elliptic equation:
⎧
⎪
⎪ −∇·(a∇u) = 0 in Ω,
⎪
⎨ ∂u
u = 0 on ∂ΩD , a = 0 on ∂ΩN (1.64)
⎪ ∂ν
⎪
⎪  ∂u
⎩ a = Qj , u|∂Ωj = cj = constant, j ∈ J,
∂Ωj
∂ν
where
• a is the diffusion coefficient function to be estimated

• ∇· and ∇ are the divergence and gradient operators with respect to

the space variables ξ1 , ξ2

• Ω is a two-dimensional domain with boundary ∂Ω partitioned into ∂ΩD ,

∂ΩN and ∂Ωj , j ∈ J

• J is a finite set of indices

• Qj ∈ IR, j ∈ J is the total injection (Qj ≥ 0) or production (Qj ≤ 0)

rate through the boundary ∂Ωj

• the condition u|∂Ωj = cj = constant means that the solution u take

a constant, but unknown value cj on each boundary ∂Ωj : the values
of cj will change if the intensities Qj of the sources or the diffusion
coefficient a change

• ν is the exterior normal to Ω

We make the hypothesis that the Dirichlet boundary satisfies

∂ΩD is non empty. (1.65)

1.6. EXAMPLE 4: 2D ELLIPTIC PARAMETER ESTIMATION 27

In the context of fluid flow through porous media, the solution u of (1.64)
is the fluid pressure, ∂Ωj represents the boundary of the jth well, and Qj its
injection or production rate. In two-dimensional problems as the one we are
considering here, a is the transmissivity, that is, the product of the perme-
ability of the porous medium with its thickness. It is not accessible to direct
measurements, and so it is usually estimated indirectly from the available
pressure measurements via the resolution of an inverse problem, which we
describe now.
The natural admissible parameter set is
def
C = {a : Ω  IR | 0 < am ≤ a(ξ) ≤ aM a.e. on Ω}, (1.66)

where am and aM are given lower and upper bounds on a, Then for any
a ∈ C, (1.64) admits a unique solution u in the state-space:
∂w
Y = {w ∈ L2 (Ω) | ∈ L2 (Ω) , i = 1, 2 , w|∂ΩD = 0}. (1.67)
∂ξi
Because of the hypothesis (1.65), we can equip Y with the norm

wY = |∇w|IL2 (Ω) , (1.68)

where
def
IL2 (Ω) = L2 (Ω) × L2 (Ω), vIL2 (Ω) = (|v1 |2L2 (Ω) + |v2 |2L2 (Ω) )1/2 . (1.69)

As in the one-dimensional parameter estimation problem of Sect. 1.4, we

shall consider both H 1 and L2 observations.
The H 1 observation is obtained by choosing as observation space:

M : w ∈ Y  ∇w ∈ F. (1.70)

Here M is an isometry from Y to F , which means that no information is

lost in the measurement process. The NLS problem for the estimation of the
diffusion coefficient a ∈ C from a data z ∈ IL2 (Ω) is then
1
â minimizes |∇ua − z|2IL2 (Ω) over C, (1.71)
2
where
ua is the solution of the elliptic equation (1.64) . (1.72)
28 CHAPTER 1. NONLINEAR INVERSE PROBLEMS

The possibility of applying the sufficient condition for Q-wellposedness

of Chap. 4 to the case of H 1 observation is studied in Sect. 4.9: despite the
availability of this rich (and highly unrealistic) observation, we shall see that
there is not much hope to satisfy these sufficient conditions when a is infinite
dimensional. But when C is replaced by the smaller set made of uniformly
Lipschitz functions defined in (4.106), still denoted by C, one can prove the
identifiability of the linearized problem, but stability and Q-wellposedness
are out of reach.
A first approach is then to use brute force, and reduce the search to
finite dimensional subsets C to restore Q-wellposedness for (1.71): when the
dimension n of C goes to infinity, the size of C has to go to zero, as well as
the thickness of the neighborhood ϑ of the attainable set, and the stability
constant of the z  â inverse mapping over ϑ blows up to infinity.
A second approach is to determine which information on the parameter
cannot be retrieved from the data, and to supply this information by an
adapted regularization term. This is shown in Sect. 5.4 to restore stability
of a on C, and to provide Q-wellposedness on finite dimensional subsets C of
C. The size constraint on C and the stability constant of the z  â inverse
mapping over ϑ are now independent of the dimension of C , the only effect of
increasing this dimension being to decrease the thickness of the neighborhood
ϑ of the attainable set.
The case of an L2 observation is obtained by choosing as observation
space
F = L2 (Ω) equipped with the norm vF = vL2 (Ω) , (1.73)
and as measurement operator
M : w ∈ Y  w ∈ F. (1.74)
Here M is the canonical injection from H 1 into L2 , which means that we
have no information on the derivatives of the solution u. The NLS problem
for the estimation of a in C from a data z ∈ L2 (Ω) is then
1
â minimizes |ua − z|2L2 (Ω) over C, (1.75)
2
where ua is still given by (1.72). This case is studied in Sect. 5.3 by com-
bining the above mentioned results for an H 1 observation with the state-
space regularization technique introduced in State-space Regularization
of Sect. 1.3.4.
Chapter 2

Computing Derivatives

We address in this chapter a practical aspect of the numerical resolution

of NLS problems, namely the computation of the gradient of the objective
function or the Jacobian of the forward map, after discretization has occurred.
This calculation has to be computed both accurately, so that the optimization
algorithm has a chance to work properly, and efficiently, in order to keep
computation time as low as possible.
When the problem at hand is infinite dimensional, we suppose that it has
been reduced first to finite dimension by choosing the following:

• A discretization of the state equation e(x, y) = 0

• A discretization of the measurement operator M

• A discretization of the objective function J

• A finite dimensional parameterization of the unknown parameter x

We refer to Chap. 3 for the last point, but we do not discuss here the the
other discretizations, and we simply suppose that they are made according
to the state of the art, so that the resulting discrete objective function is a
reasonable approximation to the continuous one. We also do not discuss the
convergence of the parameter estimated with the discrete model to the one
estimated using the infinite dimensional model, and we refer for this to the
book by Banks, Kunisch, and Ito [7].
So our starting point is the finite dimensional NLS problem, which is to
be solved on the computer: the unknown parameter x is a vector of IRn , the

G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 29

and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 2, c Springer Science+Business Media B.V. 2009
30 CHAPTER 2. COMPUTING DERIVATIVES

state y a vector of IRP , and the output v = ϕ(x) a vector of IRq . We shall dis-
cuss different techniques, such as the sensitivity function and adjoint state
approaches, for the computation of the derivatives required by local opti-
mization methods, and give on various examples a step-by-step presentation
of their implementation.
The derivatives computed in this way are called discrete: they are the
exact derivatives of the discrete objective function. For infinite dimensional
problems, they are obtained by following the first discretize then differen-
tiate rule. When the discrete equations that describe ϕ are too complicated,
one can be tempted to break the rule, and to first calculate the derivatives
on the continuous model, which is usually simpler, at least formally, and
then only to discretize the resulting equations and formula to obtain the
derivatives. The derivatives computed in this way are called discretized, and
are only approximate derivatives of the discrete objective function. This is
discussed in Sect. 2.8 on an example.
The material of this chapter is not new, but it is seldom presented in a
detailed way, and we believe it can be useful to practitioners.

2.1 Setting the Scene

The evaluation of the objective function J(x) for a given parameter vector x
requires the computation of the forward map ϕ(x). In most applications, as
we have seen in Chap. 1, this involves the resolution of one or more possibly
nonlinear equations:
⎧
⎨ given x ∈ C solve
e(x, y) = 0 (2.1)
⎩
with respect to y in Y ,
followed by 
set ϕ(x) = M(y),
(2.2)
where M is the observation operator.
The subset C of IRn is the convex set of admissible parameters, the variable
y ∈ Y describes completely the state of the system under consideration, the
state-space Y is an affine space of dimension p with tangent space δY = IRp ,
and the observation operator M describes which (usually small) part of the
state variable can actually be measured.
The use of an affine state-space will allow to incorporate into Y some of
the linear conditions that define y, as, for example, the Dirichlet boundary
2.1. SETTING THE SCENE 31

condition in a partial differential equations (Sect. 2.6), or the initial condition

in an evolutionary problem (Sect. 2.9), and simplify the determination of the
adjoint state equation.
In the most frequent case where Y is a vector space, one has simply
Y = δY = IRp .
The above decomposition of the forward map ϕ as a state equation fol-
lowed by an observation operator is not uniquely defined. For example, one
can always enlarge the state-space to vectors of the form (y, v) ∈ Y × IRq
and consider that ϕ is obtained by the new state equation
⎧
⎨ given x ∈ C solve
e(x, y) = 0, v = M(y) (2.3)
⎩
with respect to (y, v) in Y × IR ,q

followed by
set ϕ(x) = v. (2.4)
Equation (2.4) corresponds to the observation operator (y, v) → v, which is
now simply a (linear) selection operator.
So we will always suppose in the sequel that the decomposition (2.1) and
(2.2) has been chosen in such a way that it corresponds to some computa-
tional reality, with the state equation being the “hard part” of the model,
where most of the computational effort rests, and the observation opera-
tor being the “soft part,” given by simple and explicit formulas. We shall
also suppose that the decomposition satisfies the minimum set of hypothesis
(1.35).
Once the decomposition (2.1) and (2.2) of the forward map and the norm
·F on the data space have been chosen, one is finally faced with the nu-
merical resolution of the inverse problem:
1 2
x̂ minimizes J(x) = ϕ(x) − z F over C. (2.5)
2
Among the possible methods of solution for problem (2.5), optimization
algorithms are in good place, although other possible approaches exist, as,
for example, the resolution of the associated optimality condition (see, e.g.,
[14, 52, 53]). We can pick from two large classes of optimization algorithms:

• Global algorithms [77], as simulated annealing or genetic algorithms,

perform a (clever) sampling of the parameter space. They converge to
32 CHAPTER 2. COMPUTING DERIVATIVES

the global minimum of J under quite general conditions, and are very
user-friendly as the only input they require is a code that computes
J(x). The price is the large number of function evaluation required
(easily over one hundred thousand iterations for as few as ten parame-
ters), and so the use of these algorithms tends to be limited to problems
where the product size of x by computation time of J is not too large.

• Local algorithms [13, 68] use additional information on the derivatives

of J to move from the current estimate to the next according to the local
shape of the graph of J. They converge only to the nearest stationary
point (Definition 4.0.9), but need much less iterations, which makes
them applicable to large size problems. They are also much less user-
friendly, as they require the user to provide the derivatives of J or ϕ,
which requires delicate calculations and coding.

As we have seen in Sect. 1.3.4, the art of regularization consists in re-

placing the original ill-posed inverse problem by a Q-wellposed regularized
problem, which can be solved by local optimization techniques (we refer to
Chap. 5 for examples of this process). Hence the final optimization problem
(2.5) is likely to be Q-wellposed, in which case local algorithms are guaran-
teed to converge to the global minimum. This is why we shall concentrate
on local algorithms, and try to make them a little more user-friendly by ad-
dressing the problem of the computation of the gradient ∇J or the Jacobian
D = ϕ (x) they require as input.
To begin with, notice that the calculation of the gradient by finite
difference
∂J J(x + h ej ) − J(x)
(x)  , (2.6)
∂xj h
where ej is the jth basis vector of IRn , cumulates all disadvantages: it is
computationally expensive (the number of evaluation of J required is pro-
portional to the size n of the parameter vector x), and it is not precise
(making it precise would require to adjust the size of the step h by trial and
error, for each component xj , until a compromise between truncation errors
– h too large – and rounding errors – h too small – is found, but this would
be still more computionally intensive, and is never done in practice).
We present now the two methods of choice for the computation of ∇J or
D: the sensitivity functions and the adjoint approaches.
2.3. THE ADJOINT APPROACH 33

2.2 The Sensitivity Functions Approach

In this approach, the Jacobian or sensitivity matrix D of the forward map
ϕ at x is computed column by column. The jth column sj of D gives the
sensitivity of the model output to the parameter xj , and it is called the jth
output sensitivity function. Derivation of the state equation (2.1) with respect
to xj gives
∂e ∂e ∂y
(x, y) + (x, y). = 0, j = 1 . . . n. (2.7)
∂xj ∂y ∂xj
The vectors ∂y/∂xj ∈ δY = IRp represent here the sensitivity of the state
y to each parameter xj , and they are called the state sensitivity functions.
They are obtained by solving the n linearized state equation (2.7). They can
then be combined with the derivative of the observation operator M  (x) to
compute the n output sensitivity functions sj :

∂y
sj = M  (y) , j = 1 . . . n. (2.8)
∂xj

With the Jacobian D = [s1 s2 . . . sn ] computed, the gradient of J is imme-

diately given by:
∇J = D T (ϕ(x) − z). (2.9)
This approach is the most widely used, as it involves only the natural task of
derivating the chain of equations and formula that define the forward map ϕ.
With the chosen “state-space” decomposition of ϕ, the computational cost
resides in the resolution of the n linear systems (2.7). Hence the sensitivity
function approach allows to compute both ∇J and D = ϕ (x) at an additional
cost proportional to the number n of parameters.

2.3 The Adjoint Approach

The adjoint approach provides an efficient way to compute the gradient with
respect to x of G(x, ϕ(x)):
– For any scalar valued differentiable function G(x, v), which is an explicit
function of its arguments x and v
– For any mapping ϕ(x) given by a state-space decomposition of the form
(2.1) and (2.2)
34 CHAPTER 2. COMPUTING DERIVATIVES

We shall use the shorthand notation ∇G for this gradient. Depending on the
derivative one wants to compute, different choices are possible for G(x, v):
• If one chooses
1
G(x, v) = v − z2F (independant of v !), (2.10)
2
then
G(x, ϕ(x)) = J(x) ∀x ∈ C, (2.11)
so that
∇G = ∇J,
and the adjoint approach computes the gradient of J.

• If one chooses
G(x, v) = v, eiF , (2.12)
where ei is the ith basis vector of IRq , and E = IRn and F = IRq are
equipped with usual Euclidian scalar products, then

G(x, ϕ(x)) = ϕ(x), ei F ∀x ∈ C,

so that
∇G = ϕ (x)T ei = D T ei = riT ,
where ri is the ith row of D = ϕ (x). In that case, the adjoint approach
computes the Jacobian D of ϕ row by row.

• If, given a vector gv ∈ IRq , one chooses

G(x, v) = v, gv F , (2.13)

where now E = IRn and F = IRq are equipped with scalar products
 , E and  , F , then similarly G(x, ϕ(x)) = ϕ(x), gv F , and

∇G = gx ∈ IRn , where gx = ϕ (x)T gv = D T , (2.14)

where gradient and transposition are relative to the chosen scalar prod-
ucts on E and F . Hence the adjoint approach will compute the result gx
of the action of the transposed Jacobian D T on any vector gv without
having to assemble the whole matrix D, transpose it, and perform the
matrix × vector product D T gv .
2.3. THE ADJOINT APPROACH 35

Remark 2.3.1 Because of formula (2.9), the choice (2.13) for G with gv =
ϕ(x) − z leads to the computation of ∇J, as did the choice (2.10). But both
choices will produce the same final formula for ∇J.

Remark 2.3.2 The adjoint approach with the choice (2.13) for G is used
when it comes to change parameters in an optimization problem: the gradient
of J with respect to the optimization parameter vector xopt is given by

∇xopt J = ψ  (xopt )T ∇xsim J, (2.15)

where xsim is the simulation parameter vector, and ψ is the xopt  xsim map-
ping (see Sect. 3.3). An example of such a calculation is given in Sect. 3.8.2.

We explain now how the adjoint approach computes ∇G once a state-

space decomposition (1.33) and (1.34) of ϕ has been chosen: knowing that
G is obtained by (1) solving for y the (possibly nonlinear) direct equation
e(x, y) = 0 and (2) evaluating the explicit real-valued function G(x, M(y)),
we want to show that the vector ∇G can always be obtained by (1) solving
for λ a (linear) adjoint equation and (2) evaluating ∇G from x, y, and λ by
simple explicit gradient formulas.
We remark for that purpose that minimizing G(x, M(yx )) with respect to
x amounts to minimize G(x, M(y)) with respect to (x, y) under the constraint
that e(x, y) = 0. This suggest to consider the Lagrangian function associated
to this constrained optimization problem:

L(x, y, λ) = G(x, M(y)) + e(x, y), λZ , (2.16)

where the Lagrange multiplier λ ∈ Z = IRp is called the adjoint variable of y.

The interest of this Lagrangian function is that it provides a convenient
way to calculate ∇G; we state the corresponding theorem in an abstract
framework, with E, δY, Z, F instead of IRn , IRp , IRp , IRq , as this will occasion-
ally allow us to use it for infinite dimensional problems:

Proposition 2.3.3 Let (1.33) and (1.34) be a state-space decomposition for

ϕ satisfying (1.12) and (1.35), let (x, v) ∈ E × F  G(x, v) ∈ IR be a given
real-valued differentiable function, and L(x, y, λ) be the associated Lagrangian
defined by (2.16).
36 CHAPTER 2. COMPUTING DERIVATIVES

Then x ∈ C  G(x, ϕ(x)) ∈ IR is differentiable, and its gradient ∇G is

given by the gradient equation:
∂L
∇G, δxE = (x, y, λ)δx ∀δx ∈ E, (2.17)
∂x
where
∗ y ∈ Y is the solution of the (direct) state equation
e(x, y) = 0, (2.18)
∗ λ ∈ F is the solution of the adjoint state equation
∂L
(x, y, λ)δy = 0 ∀δy ∈ δY. (2.19)
∂y
In this context, y is called the (direct) state, and λ the adjoint state.
Proof. Because of the third hypothesis in (1.35), the solution yx of the state
equation (1.33) is uniquely defined for any x ∈ C. For this value y = yx of the
state, one has M(yx ) = ϕ(x) and e(x, yx ) = 0, so that the definition (2.16)
of the Lagrangian reduces to
L(x, yx , λ) = G(x, ϕ(x)) ∀x ∈ C, ∀λ ∈ Z. (2.20)
But G and M are by hypothesis differentiable functions, and the implicit
function theorem implies, using hypothesis (1.35), the differentiability of x 
yx . This proves the differentiability of x ∈ C  G(x, M(yx )) = G(x, ϕ(x)) ∈
IR. So we can differentiate (2.20) with respect to x for a fixed λ:
∂L ∂L
(x, yx , λ) δx + (x, yx , λ) δyx = ∇G, δxE ,
∂x ∂y
which reduces to the gradient equation (2.17) as soon as λ is a solution of
the adjoint equation (2.19). So the theorem will be proved if we check that
the adjoint equation (2.19) defines uniquely λ. Using the Definition (2.16)
of the Lagrangian, the adjoint equation becomes (v = M(y)):
 
∂e
∇v G(x, v), δvF + (x, y)δy, λ = 0,
∂y
 Z

 ∂e
∇v G(x, v), M (y)δyF + (x, y)δy, λ = 0,
∂y
 Z

 ∂e
M (y) ∇v G(x, v), δyY +
T
δy, (x, y) λ T
= 0,
∂y Y
∂e
M  (y)T ∇v G(x, v) + (x, y)T λ = 0. (2.21)
∂y
2.3. THE ADJOINT APPROACH 37

Hypothesis (1.35) ensures that ∂e/∂y(x, y) is an isomorphism from δY to Z,

so that λ is uniquely defined by (2.21).
We return now to the finite dimensional case where E = IRn , δY =
Z = IRp , and F = IRq . The computational cost in the adjoint approach is
in the resolution of the adjoint equation, the gradient being then obtained
by simple explicit formulas. Hence we see that the adjoint approach allows
to compute ∇J (or any row ri of D = ϕ (x)) at the sole additional cost
of the resolution of the adjoint equation, independently of the number n of
parameters. This makes the resolution of problems with a very large number
of parameters possible by using gradient-based optimization algorithms. As
for the computation of the Jacobian D = ϕ (x), it is made in the adjoint
approach row-by-row, and so its cost is proportional to the number q of
observations, but independent of the number n of parameters. This feature
is extremely useful when it comes to determine the number of independant
parameters that can be retrieved for a given combination of model and data
(Sect. 3.2).
The above costs have been estimated under the assumption that storing
the direct state y is possible, and that the corresponding cost is negligible.
This is not always true, especially for large-size time-dependent problems, in
which case the adjoint approach may require to compromise between storing
and recomputing the direct state (see Grievank [40] for an optimal strategy).

Remark 2.3.4 We have made explicit in (2.21) the variational formulation

(2.19) for the adjoint equation. One could as well explicit the variational
formula (2.17) for ∇G using the Definition (2.16) of the Lagrangian:

∂e
∇G = ∇x G(x, M(y)) + (x, y)T λ. (2.22)
∂x
It is, however, not advisable to use formulas (2.21) and (2.22) in practice,
∂e ∂e
as they require to write down the matrices ∂y (x, y) and ∂x (x, y), which can
be very large. Moreover, matrix transposition requires some thinking when it
is made with respect to weighted scalar products.
Despite their abstract appearance, the variational formulations (2.17)
(2.19) of the adjoint and gradient equations, which are based on the ex-
plicit Lagrangian function (2.16), are the most convenient to use in the
applications.
38 CHAPTER 2. COMPUTING DERIVATIVES

2.4 Implementation of the Adjoint Approach

We give in this section a step-by-step presentation of the calculations required
for the determination of the adjoint equations and the gradient formula prior
to coding. These calculations are delicate, tedious, and error prone, and lot of
effort is currently developing in the automatic differentiation community to
develop the applicability of the “reverse mode” of automatic differentiation
codes, which generates automatically the code for the adjoint equation when
given the code for the forward map [41, 40]. A less ambitious approach toward
automatization is that of [32], where the state equation (not the code!) is the
input to a formal calculus code, whose output is made of the formula for the
adjoint and gradient equations, which have then to be coded. As of today,
these automated approaches work only on relatively simple problems, and
most of the adjoint equations have to be established by hand. In any case, it
is recommended to check that the gradient computed by the adjoint approach
coincides, up to a large number of digits, with the one computed carefully
by finite difference, or by automatic differentiation on a small size problem
[61]. Here is one possible organization of the adjoint calculations:

Step 0: Forward Map and Objective Function

Identify

• The map ϕ : x ∈ (IRn ,  , E )  v ∈ (IRq ,  , F ) to be inverted

• The objective function G(x, v) whose gradient is to be computed

The choice of G will depend on the derivative one wants to compute, see
(2.10), (2.12), and (2.13) for examples. It is also necessary to specify the scalar
products on IRn and IRq to make gradient and transposition well defined:

• The parameter space E = IRn is equipped with the Euclidean scalar

product. If needed, the gradient with respect to calibrated parameters
(Sect. 3.1.1 in next chapter) follows immediately by the chain rule.

• The scalar product  , F on the data space F = IRq determines the

norm ·F used to compare the model output ϕ(x) to the data z, and
to build up the data misfit objective function J. This norm can take
into account the uncertainty on each component of the observation,
2.4. IMPLEMENTATION OF THE ADJOINT APPROACH 39

as in formula (1.8) in the Knott–Zoeppritz example of Sect. 1.1. For

infinite dimensional problems where J involves an integral over space
and/or time, as in (1.61), (1.71), and (1.75), the norm ·F is usually
chosen so that the discrete objective function is an approximation of
the continuous one (see also Sect. 3.1.2 in next chapter).

Step 1: State-Space Decomposition

Dispatch the equations defining ϕ between an affine state-space Y of dimen-
sion p, a set of p state equation e(x, y) = 0, and an observation operator
M(y). One may have to choose among different equivalent formulations of
the state equation: for example, y−1/x = 0 and xy−1 = 0 are two equivalent
state equations, and one has to decide which one to call e(x, y) = 0 and enter
in Definition 2.25 of the Lagrangian. To get the simplest calculations, and
usually the most efficient code, a good rule is to choose the formulation that
is the easiest to differentiate (the second one in our example). As a result,
the ϕ : x  v mapping is given by

x ∈ IRn  y ∈ Y solution of e(x, y) = 0, (2.23)

y ∈ Y  v = M(y) ∈ IRq , (2.24)

where M is explicit and the computational effort is in the resolution of

e(x, y) = 0.
It is important to check that there are as many equations as unknowns:
the number of equations e(x, y) = 0 should be equal to the dimension p of the
state-space Y , that is, to the dimension of its tangent vector space δY = IRp .

Step 2: Lagrangian
Combine the objective function G(x, v) chosen in step 0 and the decomposi-
tion e(x, y) = 0 , v = M(y) of ϕ chosen in step 2 to build up the Lagrangian:

L(x, y, λ) = G(x, M(y)) + e(x, y), λZ , (2.25)

for any x ∈ IRn , y ∈ Y , and λ ∈ IRP .

The only thing left to choose in the above formula is the scalar product
 , Z on the right-hand side space of the state equation. When the equation
e(x, y) = 0 comes from the discretization of a continuous problem, this degree
40 CHAPTER 2. COMPUTING DERIVATIVES

of freedom can be used to make the e(x, y), λZ term in (2.25) to mimic the
corresponding term of the continuous Lagrangian. This will allow to interpret
the vector λ as a discretization of the continuous adjoint variable.
A useful check is the following:

• Write down explicitly what the parameter vector x, the state vector y,
the state-space Y , and its tangent space δY are in the problem under
consideration

• Make sure that the Lagrangian L is an explicit function of its argu-

ments: all quantities other than x, y, and λ appearing in the right-hand
side of (2.25) should be known.

From this point on, there is no more decision to make all calculations follow
from the formula (2.25) for the Lagrangian and the scalar product  , E on
the parameter space chosen in step 0.

Step 3: Adjoint Equation

Differentiate the Lagrangian (2.25) at point x, yx , λ, with respect to the state
y ∈ Y (so that δy ∈ δY = IRp ) in order to obtain the “variational form”
(2.19) of the adjoint equation, and reorganize the result by factorizing δyj
for all j = 1 . . . p:
p
∂L
(x, yx , λ) δy = hj (x, yx , λ) δyj ∀δyj ∈ IR, j = 1 . . . p. (2.26)
∂y j=1

This reorganization is the most delicate and tedious part. Once it is done,
the “computational form” of the adjoint equations for λ ∈ IRp are obtained
by equating to zero the coefficients of δyj in (2.26):

hj (x, yx , λ) = 0 ∀j = 1 . . . p. (2.27)

We call λx the solution of the above adjoint equation.

Step 4: Gradient Equation

Differentiate the Lagrangian (2.25), at point x, yx , λx , with respect to its
first argument x ∈ IRn , and reorganize the result by factorizing δx in the
2.5. EXAMPLE 1: ADJOINT EQUATIONS 41

parameter scalar product  , E in order to obtain the “variational form”

(2.17) of the gradient equation, which we recall here:

∂L
∇G, δxE = (x, yx , λx ) δx ∀δx ∈ IRn . (2.28)
∂x
When IRn is equipped with the usual Euclidean scalar product, the ith com-
ponent of ∇G is simply the coefficient of δxi in ∂L
∂x
(x, yx , λx ) δx.

2.5 Example 1: The Adjoint Knott–Zoeppritz

Equations
We compute here the derivatives of the forward map ϕ : x  v described in
Sect. 1.1: the parameter vector x ∈ IR4 is made of the dimensionless contrast
and background coefficients (eρ , eP , eS , χ) across the elastic interface, and
the output vector v ∈ IRq is made of the q reflection coefficients R1 , . . . , Rq
computed by the sequence of formula (1.2) for q different incidence angles
θ1 , . . . , θq .
We illustrate on this simple example the step-by-step adjoint approach of
Sect. 2.3.

Step 0: Forward Map and Objective Function

Because of the independence of the calculations performed for each incidence
angle, we only need to compute the derivative of the forward map ψ : x =
(eρ , eP , eS , χ) ∈ IR4  v = R ∈ IR for one given incidence angle θ.
Hence the forward map here is ψ : x  R, and the objective function is
G(x, R) = R.
We equip both parameter space IR4 and output space IR with the usual
Euclidean scalar products, so that the Jacobian ψ  of ψ and its gradient ∇ψ
are transposed matrices for the chosen scalar products.

Step 1: State-Space Decomposition

We use here the decomposition suggested in Sect. 1.1:

y = (e, f, S1 , S2 , . . . , P, Q, R) ∈ IR19 (state vector), (2.29)

M = [0 . . . 0 1] (observation operator), (2.30)
42 CHAPTER 2. COMPUTING DERIVATIVES

which has to be complemented by the state equation e(x, y) = 0. This requires

to rewrite the formula (1.2) in the form of a sequence of equations. If we want
to avoid the need to differentiate square roots or quotients, we can choose
for e(x, y) = 0:
⎧
⎪
⎪ e − eS − eρ = 0
⎪
⎪ f − 1 + e2ρ = 0
⎪
⎪
⎪
⎪ S1 − χ(1 + eP ) = 0
⎪
⎪
⎪
⎪
⎪
⎪ S2 − χ(1 − eP ) = 0
⎪
⎪
⎪
⎪ (1 − eS )T1 − 2 = 0
⎪
⎪ (1 + eS )T2 − 2 = 0
⎪
⎪
⎪
⎪
⎪
⎪ q 2 − S1 sin2 θ = 0
⎪
⎪
⎪
⎪ M12 + q 2 − S1 = 0 (M1 ≥ 0)
⎪
⎪
⎨ M2 + q − S2 = 0
2 2
(M2 ≥ 0)
N1 + q − T1 = 0
2 2
(N1 ≥ 0) (2.31)
⎪
⎪
⎪
⎪ N 2
+ q 2
− T2 = 0 (N 2 ≥ 0)
⎪
⎪
2
⎪
⎪ D − eq 2
= 0
⎪
⎪
⎪
⎪ A − eρ + D = 0
⎪
⎪
⎪
⎪ K −D+A=0
⎪
⎪
⎪
⎪ B−1+K = 0
⎪
⎪
⎪
⎪ C −1−K =0
⎪
⎪
⎪
⎪ P − M1 (B 2 N1 + f N2 ) − 4eDM1 M2 N1 N2 = 0
⎪
⎪
⎪
⎪ Q − M2 (C 2 N2 + f N1 ) − 4q 2 A2 = 0
⎩
(P + Q)R − (P − Q) = 0.
When the chosen equations have more than one solution, the condition in
parenthesis indicate which one is to be chosen. As expected, there are 19
equations for 19 state unknowns.

Step 2: Lagrangian
With the objective function G of step 0 and the state-space decomposition
of step 1, the Lagrangian reads
L(x, y, λ) = (2.32)
R +λ1 (e − eS − eρ )
+λ2 (f − 1 + e2ρ )
+λ3 (S1 − χ(1 + eP ))
2.5. EXAMPLE 1: ADJOINT EQUATIONS 43

+λ4 (S2 − χ(1 − eP ))

+λ5 ((1 − eS )T1 − 2)
+λ6 ((1 + eS )T2 − 2)
+λ7 (q 2 − S1 sin2 θ)
+λ8 (M12 + q 2 − S1 )
+λ9 (M22 + q 2 − S2 )
+λ10 (N12 + q 2 − T1 )
+λ11 (N22 + q 2 − T2 )
+λ12 (D − eq 2 )
+λ13 (A − eρ + D)
+λ14 (K − D + A)
+λ15 (B − 1 + K)
+λ16 (C − 1 − K)
+λ17 (P − M1 (B 2 N1 + f N2 ) − 4eDM1 M2 N1 N2 )
+λ18 (Q − M2 (C 2 N2 + f N1 ) − 4q 2 A2 )
+λ19 ((P + Q)R − (P − Q)).

At this point, it is useful to summarize the arguments of the Lagrangian

and the spaces where they belong:

x = (eρ , eP , eS , χ) ∈ IR4 (parameter), (2.33)

y = (e, f, S1 , S2 , . . . , P, Q, R) ∈ IR19 (direct state),(2.34)
λ = (λ1 , . . . , λ19 ) ∈ IR 19
(adjoint state), (2.35)

and to check the following:

– The direct and adjoint state vectors have the same dimension
– The vectors x, y, and λ contain all the information necessary to compute
the Lagrangian L as an explicit function (here the only quantity that appears
in the right-hand side of (2.32) and is not in the arguments of L is the
incidence angle θ, which is known).
44 CHAPTER 2. COMPUTING DERIVATIVES

Step 3: Adjoint Equation

The variational form (2.19) of the adjoint equation is obtained by differenti-
ating the 20 terms of the Lagrangian (2.32) with respect to the state variables
only, and equating the result to zero:
∂L
(x, y, λ) δy = (2.36)
∂y
δR + λ1 δe
+ λ2 δf
+ λ3 δS1
+ λ4 δS2
+ λ5 (1 − eS )δT1
+ λ6 (1 + eS )δT2
+ λ7 (δ(q 2 ) − δS1 sin2 θ)
+ λ8 (2M1 δM1 + δ(q 2 ) − δS1 )
+ λ9 (2M2 δM2 + δ(q 2 ) − δS2 )
+ λ10 (2N1 δN1 + δ(q 2 ) − δT1 )
+ λ11 (2N2 δN2 + δ(q 2 ) − δT2 )
+ λ12 (δD − δe q 2 − e δ(q 2 ))
+ λ13 (δA + δD)
+ λ14 (δK − δD + δA)
+ λ15 (δB + δK)
+ λ16 (δC − δK)
+ λ17 (δP − δM1 (B 2 N1 + f N2 )
−M1 (2B δB N1 + B 2 δN1 + δf N2 + f δN2 )
−4δe DM1 M2 N1 N2 − 4eδD M1 M2 N1 N2
−4eDδM1 M2 N1 N2 − 4eDM1 δM2 N1 N2
−4eDM1 M2 δN1 N2 − 4eDM1 M2 N1 δN2 )
+ λ18 (δQ − δM2 (C 2 N2 + f N1 )
−M2 (2C δC N2 + C 2 δN2 + δf N1 + f δN1 )
−4δ(q 2 ) A2 − 8q 2 A δA)
+ λ19 ((δP + δQ)R + (P + Q)δR − δP + δQ) = 0
∀δy = (δe, δf, δS1 , δS2 , . . . , δP, δQ, δR) ∈ IR19 .
2.5. EXAMPLE 1: ADJOINT EQUATIONS 45

The computational form of the adjoint equations is obtained by equating to

zero the coefficient of δe, δf, δS1 , δS2 , . . . , δP, δQ, δR in (2.36):
⎧
⎪ 0 = λ1 − q λ12 − 4DM1 M2 N1 N2 λ17
2
⎪
⎪
⎪ 0 = λ2 − M1 N2 λ17 − M2 N1 λ18
⎪
⎪
⎪
⎪ 0 = λ3 − λ8
⎪
⎪
⎪
⎪ 0 = λ4 − λ9
⎪
⎪
⎪
⎪
⎪
⎪ 0 = (1 − eS ) λ5 − λ10
⎪
⎪
⎪
⎪ 0 = (1 + eS ) λ6 − λ11
⎪
⎪
⎪
⎪ 0 = λ7 + λ8 + λ9 + λ10 + λ11 − e λ12 − 4A2 λ18
⎪
⎪
⎪
⎪ 0 = 2M1 λ8 − 4eDM2 N1 N2 λ17 − (B 2 N1 + f N2 ) λ17
⎪
⎪
⎨ 0 = 2M2 λ9 − 4eDM1 N1 N2 λ17 − (C 2 N2 + f N1 ) λ18
0 = 2N1 λ10 − M1 B 2 λ17 − 4eDM1 M2 N2 λ17 − M2 f λ18 (2.37)
⎪
⎪
⎪
⎪ 0 = 2N λ
2 11 − M 1 f λ 17 − 4eDM M N
1 2 1 17 λ − M 2 C 2
λ 18
⎪
⎪
⎪
⎪ 0 = λ 12 + λ 13 − λ 14 − 4eM 1 M 2 N 1 N2 λ 17
⎪
⎪
⎪
⎪ 0 = λ13 − λ14 − 8q 2 A λ18
⎪
⎪
⎪
⎪ 0 = λ14 + λ15 − λ16
⎪
⎪
⎪
⎪ 0 = λ15 − 2M1 BN1 λ17
⎪
⎪
⎪
⎪ 0 = λ16 − 2M2 CN2 λ18
⎪
⎪
⎪
⎪ 0 = λ17 − (1 − R) λ19
⎪
⎪
⎪
⎪ 0 = λ18 + (1 + R) λ19
⎩
0 = (P + Q) λ19 + 1.

Equations in (2.37) are the adjoint Knott–Zoeppritz equations. They are

solved easily backwards, computing first λ19 , then λ18 , etc....

Step 4: Gradient Equation

Differentiation of the Lagrangian (2.32) with respect to its first argument
x = (eρ ,eP ,eS ,χ) gives

∂L
(x, y, λ) = (2.38)
∂x
−λ1 (δeS + δeρ )
+λ2 2eρ δeρ
−λ3 (δχ (1 + eP ) + χ δeP )
46 CHAPTER 2. COMPUTING DERIVATIVES

−λ4 (δχ (1 − eP ) − χ δeP )

−λ5 δeS T1
+λ6 δeS T2
−λ13 δeρ .

The gradient equation (2.28) gives then the vector ∇G = ∇R:

∂R ∂R ∂R ∂R ∂L
δeρ + δeP + δeS + δχ = (x, yx , λx ) δx (2.39)
∂eρ ∂eP ∂eS ∂χ ∂x
∀δx ∈ IRn .
∂R
Comparing (2.38) and (2.39), we see that is the coefficient of δeρ in
∂eρ
(2.38), etc..., which gives

∂R
= −λ1 + 2λ2 eρ − λ13 (2.40)
∂eρ
∂R
= χ(λ4 − λ3 ) (2.41)
∂eP
∂R
= −λ1 − λ5 T1 + λ6 T2 (2.42)
∂eS
∂R
= −λ3 (1 + eP ) − λ4 (1 − eP ) (2.43)
∂χ

Remark 2.5.1 We are here in the favorable situation where the forward map
ϕ is the juxtaposition of q independent “component” maps ψ1 , . . . , ψq . So if
we value to 1 the computational cost of ϕ, the cost of each ψi , i = 1, . . . , q, is
1/q. Then each row ri of D = ϕ (x) can be computed as earlier as ∇ψiT at
the cost of 1/q (one adjoint equation for ψi ), so that the whole Jacobian D
can be evaluated at an additional cost of 1, that is, at the same cost as the
gradient ∇J in the general case.

2.6 Examples 3 and 4: Discrete Adjoint

Equations
The examples 3 and 4 of Sects. 1.5 and 1.6 are similar from the point of view
of differentiation. So we consider here one problem that contains them both:
2.6. EXAMPLES 3 AND 4: DISCRETE ADJOINT EQUATIONS 47
⎧
⎨ −∇·(a∇u) + k(u) = f in Ω,
⎪
u = ue on a part ∂ΩD of ∂Ω, (2.44)
⎪
⎩ a ∂u = g on ∂Ω = ∂Ω \ ∂Ω ,
N D
∂ν
where Ω is a domain of IR2 with boundary ∂Ω, ∂ΩD , and ∂ΩN form a partition
of ∂Ω, and ν is the outer normal to Ω. We suppose that k, f , g, and Ω satisfy
hypothesis (1.55)–(1.57), that ue is in H 1/2 (∂ΩD ), and that a belongs to the
admissible parameter set C defined in (1.66). Under these hypotheses, the
state equation (2.44) (as well as its variational formulation (2.47) below)
admits a unique solution u ∈ H 1 (Ω), and so we can consider the following
observations:
⎧
⎨ Z ∈ IL2 (Ω) measure of ∇u in Ω,
z ∈ L (Ω)2
measure of u in Ω, (2.45)
⎩
zN ∈ L (∂ΩN ) measure of
2
u on ∂ΩN ,
which correspond to the observation operator:

M : u ∈ H 1 (Ω)  (∇u, u, u|∂ΩN ) ∈ IL2 (Ω) × L2 (Ω) × L2 (∂ΩN )

and the objective function

  
J(a, k, f, g, ue) = Z − ∇u + |z − u| +
2 2
|zN − u|∂ΩN |2 . (2.46)
Ω Ω ∂ΩN

Though it is probably not realistic from a practical point of view to estimate

simultaneously a, k, f, g, and ue by minimization of J, the above formulation
is convenient as it will allow for the simultaneous determination of the partial
derivatives of J with respect to its five arguments by the adjoint approach.
To compute a discrete gradient, as explained in the introduction of this
chapter, we follow the rule first discretize, then differentiate: we first reduce
the problem to finite dimension, and only then apply the adjoint state ap-
proach to compute the gradient.

2.6.1 Discretization Step 1: Choice of a Discretized

Forward Map
We choose for this example a standard finite element approximations of the
state equation (2.44), which we describe now. It is based on the variational
formulation of (2.44):
48 CHAPTER 2. COMPUTING DERIVATIVES
⎧
⎨ find u ∈ H (Ω)
1
 with u|∂ΩD= ue such
 that
a∇u∇w + k(u)w = f w + gw (2.47)
⎩ Ω Ω Ω ∂ΩN
for all w ∈ H (Ω) with w|∂ΩD = 0,
1

and consists in choosing a finite dimensional subspace Wh of H 1 (Ω), where

the approximated solution uh and the test function wh will reside, quadra-
ture formula to approximate the integrals which appear in (2.47), and finite
dimensional approximations of the parameters a, k, f, g, and ue .

1. The space Wh . We suppose that Ω is a polyhedron of IR2 , and we cover

Ω by a triangulation Th , that is, a family of nondegenerated triangles K
called elements, whose union is Ω, and such that two distinct elements
of Th are either disjoints, or share only an edge or a vertex. Th is called
the simulation mesh. The index h refers to the size of the elements
of Th :
h = max diam(K),
K∈Th

but it will more generally be used to indicate the discretized quantity

associated to a continuous one. The triangulation Th is chosen adapted
to the boundary conditions, that is, such that ∂ΩD and ∂ΩN are made
of edges of elements of Th . We denote by Ωh the set of nodes of Th , that
is, the collection of all vertices M of the elements K.
We choose for Wh the lowest order Lagrange finite element space:

Wh = {wh ∈ C(Ω) | w|K ∈ P1 }, (2.48)

where C(Ω) is the space of continuous functions over the closure Ω of

Ω, and P1 is the space of polynomials of degree one in two variables.
The degrees of freedom of Wh are the values uM of uh at the nodes Ωh
of Th :
uh = (uM ∈ IR, M ∈ Ωh ). (2.49)

They are linked to the function uh by

uh (x) = uM wM (x) ∀x ∈ Ω, (2.50)

M ∈Ωh
2.6. EXAMPLES 3 AND 4: DISCRETE ADJOINT EQUATIONS 49

where the basis functions wM are defined by


0 if P = M
wM ∈ Wh , wM (P ) = ∀M, P ∈ Ωh . (2.51)
1 if P = M

The dimension of Wh is the number Nh of nodes in Ωh . So we have to

find Nh (nonlinear) equations to determine uh .

2. The quadrature formula. We use the “trapezoidal rule” to approximate

the integral of a regular function over an element K or one edge A of
Th . If we denote by ai , i = 1, 2, 3, the vertices of K and by ai , i = 1, 2,
the endpoints of an edge A, we obtain, for any function ψ continuous
on K,   3
ψ ≈ IK (ψ) = |K| i=1 ψ(ai ),
K
|A| 2
3 (2.52)
A
ψ ≈ IA (ψ) = 2 i=1 ψ(ai ).

The integrals over Ω and ∂ΩN are then approximated by

  
Ω ψ ≈ IΩ (ψ) = I (ψ| ),
K∈Th K K (2.53)
∂ΩN
ψ ≈ I∂ΩN (ψ) = A∈∂ΩN IA (ψ|A ),

◦
where ψ is any function on Ω whose restriction to the interior K of an
◦
element K of Th is continuous on K , and has a limit on the edges and
vertices of K. Hence the restrictions ψ|K to K and ψ|A to a boundary
edge A are well defined at the vertices of K and the endpoints of A.
This ensures that IK (ψ|K ) and I∂ΩN (ψ) in the right-hand sides of the
Definition (2.53) make sense.
We shall use in the sequel the quadrature formula (2.53) for functions
ψ, which are either continuous on Ω or are piecewise constant on Th .

3. The discretized parameters. We have to choose the finite dimensional

inputs corresponding to a, k, f, g, and ue . Because the computational
cost in the adjoint approach is independent of the number of parame-
ters, one computes the gradient with respect to the simulation param-
eters, which correspond to the most comprehensive discretization of
the parameters compatible with the chosen approximation of the state
equation (see Sect. 3.3 in the next chapter). Their number exceeds usu-
ally by far the number of parameters that could actually be identified
50 CHAPTER 2. COMPUTING DERIVATIVES

without further regularization. So the optimization has to be performed

with respect to a (usually smaller) number of optimization parameters.
The choice of these optimization parameters is addressed in the Chap. 3
on parameterization, and the calculation of the gradient with respect
to the optimization parameters from the gradient with respect to the
simulation parameters is discussed in Sects. 3.3 and 3.8.2.
We describe now a possible choice of simulation parameters approxi-
mating a, k, f, g, ue .
In the variational formulation (2.47), a is the coefficient of ∇u∇w in
an integral over Ω. As u and w are approximated in the space Wh of
functions which are piecewise linear on Th , their gradients are constant
vector fields on each element K, so that any variation of a inside an
element K is not going to be seen, only its mean value over K plays
a role. This suggests to choose for ah a piecewise constant parameter
on Th :
ah = ( aK ∈ IR, K ∈ Th ). (2.54)

The nonlinearity u  k(u) can be approximated by a function kh

depending on a finite number nk of coefficients: polynomial, continuous
piecewise linear function, closed form formula, etc.
The right-hand side f can be approximated by a function fh , which is
either piecewise constant on the triangles of Th or piecewise linear in
the same space Wh as uh . We choose the second solution, which leads
to slightly simpler formulas. The degrees of freedom of fh are then its
values on the nodes M of Ωh :

fh = (fM ∈ IR, M ∈ Ωh ). (2.55)

Similarly, the right-hand side g on the boundary ∂ΩN can be approxi-

mated by a function gh , which is either piecewise constant on the edges
A of ∂ΩN , or continuous on ∂ΩN and linear on each of its edge A. We
choose the second solution, so the degrees of freedom of gh are its values
on the subset ∂ΩN,h of nodes M of Ωh , which are located on ∂ΩN (we
do not include in Ωh the endpoints of ∂ΩN , when they exist, as it will
turn out that the value of gM at such points has no influence on the
solution):
gh = ( gM ∈ IR, M ∈ ∂ΩN,h ). (2.56)
2.6. EXAMPLES 3 AND 4: DISCRETE ADJOINT EQUATIONS 51

There is no choice for the discretization ue,h of ue : if one wants to satisfy

exactly the boundary condition uh = ue,h on ∂ΩD , then ue,h is neces-
sarily in the trace of Wh on ∂ΩD , that is, in the space of continuous
functions on ∂ΩD , which are linear over each edge A of ∂ΩD . The de-
grees of freedom of ue,h are then its values on the subset ∂ΩD,h of nodes
M of Ωh , which are located on ∂ΩD , including the endpoints of ∂ΩD
when they exist:

ue,h = ( ue,M ∈ IR, M ∈ ∂ΩD,h ). (2.57)

The sets of node ∂ΩN,h and ∂ΩD,h form a partition of the subset ∂Ωh of Ωh
made of nodes located on the boundary ∂Ω. All the terms in the variational
formulation (2.47) have now a finite dimensional counterpart, and so we can
define the finite dimensional variational formulation:
⎧
⎨ find uh ∈ Wh with uh|∂ΩD = ue,h such that
IΩ (ah ∇uh ∇wh )+IΩ (kh (uh )wh ) = IΩ (fh wh )+I∂ΩN (gh wh ) (2.58)
⎩
for all wh ∈ Wh with wh|∂ΩD = 0.

As the node of the quadrature formulas IΩ and I∂ΩN coincide with the nodes
M ∈ Ωh of the degrees of freedom of uh , it is a simple matter to deduce from
the above equation, where uh ∈ Wh is still a function, a system of nonlinear
equations for the vector of degrees of freedom uM ∈ Ωh :
⎧
⎪
⎪ uM = ue,M
  ∀M ∈ ∂ΩD,h ,
⎪
⎪ −
⎨ P ∈Ωh \∂ΩD,h M,P P
A u + α k (u
M h M ) = P ∈∂ΩD,h AM,P ue,P
+αM fM (2.59)
⎪
⎪
⎪
⎪ (+∂αM gM if M ∈ ∂ΩN,h ),
⎩
∀M ∈ Ωh\∂ΩD,h .

where
AM,P = IΩ (ah ∇wM ∇wP ) ∀M, P ∈ Ωh\∂ΩD,h , (2.60)
and where αM and ∂αM are geometric coefficients related to the triangula-
tion Th :
1 1
αM = |K|, ∂αM = |A|. (2.61)
3 K∈T ,KM 2 A⊂∂Ω ,AM
h N

The nonlinear state equations (2.59) have to be solved on the computer by

some iterative algorithm. However, one does not consider this algorithm as
52 CHAPTER 2. COMPUTING DERIVATIVES

being a part of the process to be differentiated: the vector uh produced by the

code is considered to be the unique solution of equation (2.59). Of course, this
is acceptable if the algorithm has been run up to a satisfactory convergence.
At this point, the “parameter-to-state” map ah , kh , fh , gh , ue,h  uh is
perfectly defined from a computational point of view.

2.6.2 Discretization Step 2: Choice of a Discretized

Objective Function
We choose here an approximation Jh to the continuous objective function J
defined in (2.46). Using the approximation of u and the approximation of the
integrals over Ω and ∂ΩN of the previous section, it is natural to define Jh by
⎧
⎨ Jh (ah , kh , fh , gh , ue,h) = IΩ (Zh − ∇uh 2 )
+ IΩ (|zh − uh |2 ) (2.62)
⎩ + I∂ΩN (|zN,h − uh|∂ΩN |2 ),

where the data Z = (Zh , zh , zN,h ) satisfy the following:

• Zh is a piecewise constant vector field on Th
• zh is a function of Wh , that is, continuous piecewise linear on Ω
• zN,h is a continuous piecewise linear function on ∂ΩN
The corresponding degrees of freedom are
⎧
⎨ Zh = (ZK ∈ IR2 , K ∈ Th ) (measure of ∇u),
zh = (zM ∈ IR, M ∈ Ωh ) (measure of u), (2.63)
⎩
zN,h = (zN,h,M ∈ IR, M ∈ ∂ΩN,h ) (measure of u on ∂ΩN ).
The discretization of the continuous problem is now completed, and so we can
move to the determination of the gradient of the discrete objective function
Jh with respect to all its arguments by the adjoint approach.

2.6.3 Derivation Step 0: Forward Map and Objective

Function
Our objective is to compute the partial derivatives of Jh with respect to the

n = CardTh + nk + CardΩh + Card∂Ωh

2.6. EXAMPLES 3 AND 4: DISCRETE ADJOINT EQUATIONS 53

degrees of freedom of the parameter vector x = (ah , kh , fh , gh , ue,h ), so we

equip the parameter space E = IRn with the usual Euclidean scalar product.
The discrete objective function (2.62) is of the form (2.11), with a forward
map ϕh and an objective function Gh given by:
ϕh : x = (ah , kh , fh , gh , ue,h )  v = (∇uh , uh , uh |∂ΩN ), (2.64)

where uh is solution of (2.58), and:

⎧
⎨ Gh (x, v) = IΩ (Zh − Vh 2 )
+ IΩ (|zh − vh |2 ) (2.65)
⎩
+ I∂ΩN (|zN,h − vN,h |2 ),
where v = (Vh , vh , vN,h ) is in the data space defined by (2.63). This data space
is equipped with the scalar product associated to the norm in the right-hand
side of (2.65). As required, Gh is an explicit function of x and v (it even does
not depend on x !), and Gh (x, ϕh (x)) coincides with Jh (x).

2.6.4 Derivation Step 1: State-Space Decomposition

We have to now dispatch (2.58) defining uh into a possibly affine state-space
Y and a set of state equations e(x, uh ) = 0, which defines uh ∈ Y . There
are two options here, depending on whether or not we include the Dirichlet
boundary condition in the state-space Y :
Option 1: Y does not include the boundary condition. We consider
here the vector state-space:
Y = δY = Wh .
The state equations e(x, y) = 0 which determine y = uh ∈ Y are then
(see (2.58))
uh|∂ΩD = ue,h , (2.66)
IΩ (ah ∇uh ∇wh )+IΩ (kh (uh )wh ) = IΩ (fh wh )+I∂ΩN (gh wh ) (2.67)
∀wh ∈ Wh s.t. wh|∂ΩD = 0,
which, together with the observation operator
M : uh  (∇uh , uh , uh|∂ΩN ), (2.68)

defines the forward map ϕh chosen in (2.64).

54 CHAPTER 2. COMPUTING DERIVATIVES

Option 2: Y includes the boundary condition. This option is avail-

able when one does not need to compute the gradient of Jh with
respect to the Dirichlet data ue,h . So we eliminate ue,h from the pa-
rameter vector
x = (ah , kh , fh , gh ),
and introduce it in an affine state-space
Y = {wh ∈ Wh | wh|∂ΩD = ue,h },
whose tangent vector space is
δY = {wh ∈ Wh | wh|∂ΩD = 0},
which happens now to coincide with the space of test functions of the
variational formulation for uh . The state equation e(x, y) = 0 which
determine y = uh ∈ Y is then (compare with (2.66) and (2.67))
IΩ (ah ∇uh ∇wh )+IΩ (kh (uh )wh ) = IΩ (fh wh ) (2.69)
+I∂ΩN (gh wh ) ∀wh ∈ δY,
which, together with the same observation operator M as in (2.68),
defines the forward map ϕh chosen in (2.64).

2.6.5 Derivation Step 2: Lagrangian

We take advantage here of the fact that the state equation e(x, y) = 0 is
partly or totally under variational form: the Lagrange multiplier associated
with a variational equation can always be chosen in the space of the test
functions of the variational formulation, in which case the corresponding
term e(x, y), λZ of the Lagrangian is immediately obtained by replacing
the test function by the Lagrange multiplier λ.
Option 1: In this case, the state is
uh ∈ Y = Wh
and there are two Lagrange multipliers:
– λD,h associated with the Dirichlet boundary condition (2.66), which
specifies the value of uh at each node M ∈ ∂ΩD,h . So we chose
λD,h = ( λD,M ∈ IR, M ∈ ∂ΩD,h ),
2.6. EXAMPLES 3 AND 4: DISCRETE ADJOINT EQUATIONS 55

and we define the scalar product between λD,h and ue,h − uh |∂ΩD via a
quadrature formula I∂ΩD defined in the same way as I∂ΩN as in (2.53).
– λh associated with the variational formulation (2.67), which one can
simply take in the space of the test functions
λh ∈ {wh ∈ Wh | wh |∂ΩD = 0}. (2.70)
With the objective function Gh of (2.65) and the state equations (2.66)
and (2.67), the Lagrangian for the first option is then
L(ah , kh , fh , gh , ue,h; uh ; λD,h , λh ) = (2.71)
IΩ (Zh − ∇uh  ) + IΩ (|zh − uh | ) + I∂ΩN (|zN,h − uh | )
2 2 2

+I∂ΩD ((ue,h − uh )λD,h )

+IΩ (ah ∇uh · ∇λh ) + IΩ (kh (uh )λh ) − IΩ (fh λh ) − I∂ΩN (gh λh ).
By construction, the dimension of the vector λD,h , λh is equal to the
number of equations in (2.66) and (2.67), and L is an explicit function
of all its arguments, as one can check on (2.71).
Option 2: The state satisfies now
uh ∈ Y = {wh ∈ Wh | wh|∂ΩD = ue,h},
and there is only one Lagrange multiplier λh associated with the vari-
ational state equation (2.69), which lives in its test function space
λh ∈ δY = {wh ∈ Wh | wh |∂ΩD = 0},
which is the same space as in (2.70). The Lagrangian for the second
option uses the same function Gh as option 1, and the single state
equation (2.69)
L(ah , kh , fh , gh ; uh ; λh ) = (2.72)
IΩ (Zh − ∇uh  ) + IΩ (|zh − uh | ) + I∂ΩN (|zN,h − uh | )
2 2 2

+IΩ (ah ∇uh · ∇λh ) + IΩ (kh (uh )λh ) − IΩ (fh λh ) − I∂ΩN (gh λh ).
The introduction of the boundary condition in the state-space leads
hence to a Lagrangian function with fewer arguments and fewer terms,
so that the adjoint state determination will be slightly simpler, but the
price to pay is that this approach will not give the gradient with respect
to the boundary condition ue,h.
56 CHAPTER 2. COMPUTING DERIVATIVES

2.6.6 Derivation Step 3: Adjoint Equation

Option 1: The equation for λh ∈ {wh ∈ Wh | wh |∂ΩD = 0} is obtained by
equating to zero the differential of (2.71) with respect to uh :

∂L
(ah , kh , fh , gh , ue,h ; uh ; λD,h , λh ) = (2.73)
∂uh
−2IΩ ((Zh −∇uh ) · ∇δuh )
−2IΩ ((zh −uh )δuh)
−2I∂ΩN ((zN,h −uh )δuh )
−I∂ΩD (δuh λD,h )
+IΩ (ah ∇δuh · ∇λh ) + IΩ (kh (uh )δuh λh )
= 0 ∀δuh ∈ δY,

where
δY = Wh .
Equations (2.70) and (2.73) define uniquely the adjoint states λD,h and
λh . Choosing successively δuh ∈ {wh ∈ Wh | wh |∂ΩD = 0} and δuh =
wM ∀M ∈ ∂Ωh,D , where wM is the basis function of Wh associated
with node M, we obtain the following decoupled adjoint equations for
λh and λD,h :
⎧
⎪
⎪ find λh ∈ Wh with λh |∂ΩD = 0 such that
⎪
⎪ IΩ (ah ∇λh · ∇wh )+IΩ (kh (uh )λh wh ) =
⎪
⎪
⎨ 2IΩ ((Zh −∇uh ) · ∇wh )
(2.74)
⎪
⎪ +2IΩ ((zh −uh )wh)
⎪
⎪ +2I∂ΩN ((zN,h −uh )wh )
⎪
⎪
⎩
for all wh ∈ Wh with wh|∂ΩD = 0,
⎧
⎪
⎪ find λD,h ∈ {λD,M ∈ IR, M ∈ ∂ΩD,h } such that
⎪
⎪
⎪
⎪ I∂ΩD (λD,h wM ) =
⎪
⎪
⎨ +IΩ (ah ∇λh · ∇wM ) + IΩ (kh (uh )λh wM )
−2IΩ ((Zh −∇uh ) · ∇wM ) (2.75)
⎪
⎪
⎪
⎪ −2IΩ ((zh −uh )wM )
⎪
⎪
⎪
⎪ −2I ∂ΩN ((zN,h −uh )wM )
⎩
for all basis functions wM ∈ Wh , M ∈ ∂ΩD,h .
Equation (2.74) is very similar to the variational formulation (2.58) for
uh , but with different right-hand sides: in the case where one does not
2.6. EXAMPLES 3 AND 4: DISCRETE ADJOINT EQUATIONS 57

observe the gradient of u, there is no 2IΩ ((Zh −∇uh ) · ∇wh ) term in

(2.74), and we obtain immediately that λh is an approximation to the
solution λ of an equation similar to the elliptic equation (2.44) defining
u, but with right-hand sides 2(z − u) instead of f in Ω and 2(zN − u)
instead of g on ∂ΩN .
We postpone the interpretation of (2.74) for λh in the general case,
and of (2.74) for λD,h , after we have established the continuous adjoint
equations at the end of Sect. 2.6.
The system of linear equations for the node values λM , M ∈ Ωh and
λD,M , M ∈ ∂ΩD,h resulting from (2.74) and (2.75) is (compare with
(2.59))
⎧
⎪
⎪ λ
M = 0 ∀M ∈ ∂ΩD,h
⎪
⎪ 
⎪
⎪ AM,P λP + αM kh (uM )λM =
⎨ P ∈Ωh \∂ΩD,h 
+2 KM |K| (ZK − ∇uh |K )·∇wM |K
⎪
⎪ +2αM (zM − uM )
⎪
⎪
⎪
⎪ (+2 ∂αM (zN,M − uM ) if M ∈ ∂ΩN,h )
⎩
∀M ∈ Ωh\∂ΩD,h ,
⎧ 
⎪
⎪ ∂αM λD,M = + P ∈Ωh \∂ΩD,h AM,P λP + αM kh (uM )λM
⎪
⎪ 
⎨ −2 KM |K| (ZK − ∇uh |K )·∇wM |K
−2αM (zM − uM )
⎪
⎪
⎪
⎪ (−2 ∂αM (zN,M − uM ) if M ∈ ∂ΩN,h )
⎩
∀M ∈ ∂ΩD,h .
Option 2: Similarly, we obtain the equation for λh ∈ δY by equating to
zero the differential of (2.72) with respect to uh :
∂L
(ah , kh , fh , gh ; uh ; λh ) = (2.76)
∂uh
−2IΩ ((Zh −∇uh ) · ∇δuh )
−2IΩ ((zh −uh )δuh)
−2I∂ΩN ((zN,h −uh )δuh )
+IΩ (ah ∇δuh · ∇λh ) + IΩ (kh (uh )δuh λh )
= 0 ∀δuh ∈ δY,
where now
δY = {wh ∈ Wh | wh |∂ΩD = 0},
so that (2.76) coincides with (2.74) of option 1.
58 CHAPTER 2. COMPUTING DERIVATIVES

Hence the two options for the boundary condition define the same adjoint
state λh , but λD,h is defined only in option 1.

2.6.7 Derivation Step 4: Gradient Equation

Option 1: Differentiation of the Lagrangian (2.71) with respect to all pa-
rameters ah , kh , fh , gh , ue,h for fixed direct and adjoint states uh , λD,h
and λh gives

δJh = I∂ΩD (δue,hλD,h ) (2.77)

+IΩ (δah ∇uh · ∇λh )
+IΩ (δkh (uh )λh )
−IΩ (δfh λh )
−I∂ΩN (δgh λh ).

If we expand the term I∂ΩD (δue,h λD,h ) using the definition of I∂ΩD , and
pick the coefficient of δue,M in the resulting formula, we see that

∂Jh
= ∂αM λD,M ∀M ∈ ∂ΩD,h . (2.78)
∂ue,M

We obtain similarly from the IΩ (δah ∇uh · ∇λh ) term

∂Jh
= |K|(∇uh · ∇λh )|K . (2.79)
∂aK
In order to determine the gradient with respect to the nonlinearity kh ,
let us denote by κ1 , . . . , κnk the coefficients that define the u  kh (u)
function. The differential of kh is then
∂kh
δkh (uh ) = (uh )δκj . (2.80)
j=1,...,nk
∂κj

We can now substitute into (2.77) the value of δkh given by (2.80) and
pick the coefficients of δκj , which gives the following expressions for
the partial derivatives of Jh with respect to κ1 , . . . , κnk :

∂Jh  ∂k 
h
= IΩ (uh ) λh . (2.81)
∂κh ∂κj
2.7. EXAMPLES 3 AND 4: CONTINUOUS ADJOINT EQUATIONS 59

In the case where kh is a polynomial of degree nk − 1 in the variable u

with coefficients κ0 , . . . , κnk −1 , we obtain from (2.80)

∂Jh
= IΩ (ujh λh ), j = 0, . . . , nk − 1.
∂κj

But kh can also be searched for as a continuous piecewise linear function

of u; in this case, the coefficients κj , j = 1, . . . , nk , are the values of kh
at a given set of values uj , j = 1, . . . , nk , of u, so that

∂Jh
= IΩ (wj (uh )λh ),
∂κj

where wj is the continuous piecewise linear function of u defined by

wj (ui ) = δi,j , with δi,j = 0 if i = j or 1 if i = j.
Finally, the two last terms of (2.77) give the gradient with respect to
the right-hand sides fh and gh :
∂Jh
= αM λM ∀M ∈ Ωh , (2.82)
∂fM
∂Jh
= ∂αM λM ∀M ∈ ∂ΩN,h . (2.83)
∂gM
Option 2: The only difference with option 1 is that the I∂ΩD ((ue,h −uh )λD,h )
term is missing in the Lagrangian (2.72) for option 2. Hence, the for-
mula (2.78) for the derivative with respect to the boundary condition
ue,h is not available in this option, but the formula (2.79), (2.81)–(2.83)
for the other derivatives are the same.

2.7 Examples 3 and 4: Continuous Adjoint

Equations
It is in fact possible to compute the derivative of the objective function
(2.46) of Examples 3 and 4 for the original infinite dimensional problem,
before any reduction to finite dimension is done. As we shall see in this
section, this determination is formally simpler than the determination of the
discrete gradient, as one does not need to consider all the formulas used
60 CHAPTER 2. COMPUTING DERIVATIVES

for the discretization. But it requires a good understanding of functional

analysis, distribution theory, and Green formulas, and from this point of
view the determination of the discrete gradient, which works only with real
numbers, is more elementary.
We present purposely the continuous gradient after the discrete one, to
emphasize the fact that the continuous gradient is not a preliminary step
to the discrete gradient. However, if they are determined first, the formulas
for the continuous Lagrangian can be used as a guideline for the choice of
the scalar products for the discrete Lagrangian, in order to ensure that the
discrete adjoint state is an approximation of the continuous one.
The forward map is (compare with (2.64))

ϕ : x = (a, k, f, g, ue)  v = (∇u, u, u|∂ΩN ), (2.84)

where u ∈ H 1 (Ω) is the solution of (2.47), and according to (2.46), the

objective function G(x, v) to be differentiated is (compare with (2.65))
  
G(x, v) = Z − ∇v + |z − v| +
2 2
|zN − v |2 .
Ω Ω ∂ΩN

This completes the step 0 of derivation. One can then choose the state-space
decomposition corresponding to option 1 above with the vector state-space:

Y = δY = H 1 (Ω),

so that the state equations (2.47) rewrite

u|∂ΩD = ue in H 1/2 (∂ΩD ), (2.85)

   
a∇u·∇w + k(u)w = fw + gw (2.86)
Ω Ω Ω ∂ΩN
∀w ∈ H 1 (Ω) such that w|∂ΩD = 0.

This, together with the observation operator

M : u  (∇u, u, u|∂ΩN ),

defines the forward map ϕ chosen in (2.84) and completes the step 1 of
derivation. In step 2, one introduces first the two Lagrange multipliers:
– λD associated with the Dirichlet boundary condition (2.85). The choice
of the function space for λD is a little technical: as λD is expected to define a
2.7. EXAMPLES 3 AND 4: CONTINUOUS ADJOINT EQUATIONS 61

linear functional on the dense subspace H 1/2 (∂ΩD ) of L2 (∂ΩD ), where (2.85)
holds, it is natural to require that
λD ∈ H −1/2 (∂ΩD ),
where H −1/2 (∂ΩD ) ⊃ L2 (∂ΩD ) is the dual space of H 1/2 (∂ΩD ) ⊂ L2 (∂ΩD ).
For any λD ∈ H −1/2 (∂ΩD ) and μ ∈ H 1/2 (∂ΩD ), we denote by λD , μH −1/2 ,H 1/2
the value of the linear functional λD on the function μ. In the case where λD
happens to be in the dense subset L2 (Ω) of H −1/2 (∂ΩD ), one has simply

λD , μH −1/2 ,H 1/2 = λD μ (2.87)
∂ΩD

– λ associated to the variational formulation (2.86), which one can simply

take in the space of the test functions of (2.86):
λ ∈ {w ∈ H 1 (Ω) | w|∂ΩD = 0}.
The corresponding Lagrangian function is then defined by (compare with
(2.71))
L(a, k, f, g, ue; u; λD , λ) = (2.88)
  
Z − ∇u + |z − u| +
2 2
|zN − u|2
Ω Ω ∂ΩN
+ue − u|∂ΩD , λD H 1/2 ,H −1/2
   
+ a∇u·∇λ + k(u)λ − fλ − gλ .
Ω Ω Ω ∂ΩN

Differentiation of the Lagrangian with respect to u gives, as above, two de-

coupled equations for λ and λD (compare with (2.74) and (2.75)):
⎧
 λ ∈ H (Ω) with
1
⎪
⎪ find λ|∂ΩD = 0 such that
⎪
⎪ 
λ w = +2 Ω (Z −∇u)·∇w
⎨ Ω a∇λ·∇w + Ω
k (u)
+2 Ω (z − u)w (2.89)
⎪
⎪ +2 ∂ΩN (zN − u)w
⎪
⎪
⎩
for all w ∈ H 1 (Ω) with w|∂ΩD = 0,
⎧
⎪
⎪ find λD ∈ H −1/2 (∂ΩD ) such that  
⎪
⎪ λ 
⎪
⎪ D , w | ∂Ω H −1/2 ,H 1/2 = +  a∇λ·∇w + k (u) λ w
⎨ D Ω Ω
−2 Ω (Z − ∇u)·∇w
(2.90)
⎪
⎪ −2 Ω (z − u)w
⎪
⎪
⎪
⎪ −2 ∂ΩN (zN − u)w
⎩
for all w ∈ H (Ω). 1
62 CHAPTER 2. COMPUTING DERIVATIVES

Proposition 2.7.1 The continuous adjoint equations (2.89) and (2.90)

have necessarily unique solutions λ and λD . These equations are a weak
formulation of the following set of partial differential equations:
⎧ 
⎨ −∇·(a∇λ) + k (u)λ = 2(z − u) − 2∇·(Z − ∇u) in Ω,
⎪
λ = 0 on ∂ΩD , (2.91)
⎪
⎩ a ∂λ = 2(z − u ) + 2(Z − ∇u )·ν on ∂Ω ,
N N
∂ν
∂λ
λD = a − 2(Z − ∇u )·ν on ∂ΩD . (2.92)
∂ν
Proof. The existence and uniqueness of the solution λ and λD of the
continuous adjoint equations (2.89) and (2.90) follow immediately from
Proposition 2.3.3.
We recall first the concept of weak formulation: let us call weak solution
a solution λ, λD of (2.89), (2.90), and classical solution a solution λ, λD
of (2.91), (2.92) in the usual sense of functions. The system of equations
(2.89) and (2.90) is then a weak formulation of the set of partial differential
equations (2.91) and (2.92) if and only if:
– Any classical solution is a weak solution
– Any regular weak solution is a classical solution
Without the regularity assumption, there is no hope for a weak solution to be
a classical solution, as the solution of (2.89) and (2.90) is not smooth enough
for all terms in (2.91) and (2.92) to make sense as functions: when λ is in
H 1 (Ω) and a is in L∞ (Ω), the vector field a∇u is in L2 (Ω) × L2 (Ω), so that,
for example,
– The term ∇·(a∇λ) does not make sense as a function on Ω, but only
as a distribution
– The term a ∂λ
∂ν
, which is by definition a∇u·ν, does not makes sense as
a function on ∂Ω (functions of L2 (Ω) have no trace on ∂Ω)
We check first that any classical solution is a weak one. Let λ, λD be a
classical solution of (2.91) and (2.92). Multiplying the first equation of (2.91)
by w ∈ H 1 (Ω), integrating over Ω and using the Green formula
⎧  
⎨ − Ω ∇·(a∇λ − 2(Z −∇u)) w = + Ω (a∇λ − 2(Z −∇u))·∇w
 (2.93)
⎩
− ∂Ω (a∇λ − 2(Z −∇u))·ν w,
2.7. EXAMPLES 3 AND 4: CONTINUOUS ADJOINT EQUATIONS 63

we obtain that
⎧    
⎪
⎪
⎪ Ω
a∇λ·∇w + Ω
k (u) λ w = 2 Ω (Z −∇u)·∇w
⎪
⎨ +2 Ω (z − u)w
+ ∂ΩD (a∇λ − 2(Z −∇u))·ν w (2.94)
⎪ 
⎪
⎪ + ∂ΩN (a∇λ − 2(Z −∇u))·ν w
⎪
⎩
for all w ∈ H 1 (Ω),

and, using (2.92) and the third equation of (2.91)

⎧   
⎪
⎪ a∇λ·∇w + Ω k  (u) λ w = 2 Ω (Z −∇u)·∇w
⎪
⎪
Ω
+2 Ω (z − u)w
⎨
+ ∂ΩD λD w (2.95)
⎪ 
⎪
⎪ +2 ∂ΩN (zN − u)w
⎪
⎩
for all w ∈ H 1 (Ω).

This equation is to be compared with the weak formulation (2.89) and (2.90):
– It reduces to (2.89) when the test function w is chosen such that w = 0
on ∂ΩD . Hence, the classical solution λ satisfies (2.89)
– Because λD is by hypothesis a function, formula (2.87) holds, so that
(2.95) coincides with (2.90). This shows that the classical solution λD satisfies
(2.90)
We conclude the proof by checking that any regular weak solution is a
classical solution. So let λ, λD be a regular solution of (2.89) and (2.90). Let
us choose w in (2.89) in the space D(Ω) of test functions of distributions.
This space is made of infinitely differentiable functions, which vanish over
some neighborhood of ∂Ω. Hence the integral over ∂ΩN disappears in (2.89),
which now reads, in the sense of distributions,

− ∇·(a∇λ) + k  (u)λ = 2(z − u) − 2∇·(Z − ∇u) in D  (Ω). (2.96)

But we have supposed that λ is a smooth function, so that (2.96) holds in

the sense of functions, everywhere on Ω, and hence coincides with the first
equation of (2.91). So we see that λ satisfies the first equation of (2.91). It
satisfies also trivially the second equation of (2.91) by definition of the weak
solution.
We prove now that λ and λD also satisfy the third equation of (2.91) as
well as equation (2.92). To do that, we multiply (2.96) – which now holds
everywhere on Ω – by a test function w ∈ H 1 (Ω), integrate over Ω, and use
64 CHAPTER 2. COMPUTING DERIVATIVES

the Green formula (2.93), which gives (2.94) as above. Subtracting (2.94)
from (2.89) for a w which vanishes over ∂ΩD gives
  
0 = 2 ∂ΩN (zN − u)w − ∂ΩN (a∇λ − 2(Z −∇u))·ν w
(2.97)
for all w ∈ H 1 (Ω) with w|∂ΩD = 0.

Because of hypothesis (1.57), when w spans the subspace of H 1 (Ω) made of

functions that vanish over ∂ΩD , its trace on ∂ΩN spans the dense subspace
H 1/2 (∂ΩN ) of L2 (∂ΩN ), and (2.97) implies that the coefficient of w is zero,
which shows that λ satisfies the third equation of (2.91).
Then adding (2.94) and (2.90) gives, using the third equation of (2.91),
 
λD , w|∂ΩD H −1/2 ,H 1/2 = ∂ΩD (a∇λ − 2(Z −∇u))·ν w
(2.98)
for all w ∈ H 1 (Ω).

Combining (2.98) with formula (2.87), which holds because of the smoothness
hypothesis made on λD , we obtain
  
λ w = ∂ΩD (a∇λ − 2(Z −∇u))·ν w
∂ΩD D
for all w ∈ H 1 (Ω).

Once again, because of hypothesis (1.57), when w spans H 1 (Ω), its trace on
∂ΩD spans the dense subspace H 1/2 (∂ΩD ) of L2 (∂ΩD ), which shows that λD
satisfies (2.92). This ends the proof of Proposition 2.7.1.
Now that the direct state u and the adjoint state λD , λ are known,
we simply have to differentiate the Lagrangian (2.88) with respect to x =
(a, k, f, g, ue) for fixed u, λD , λ to obtain the differential of the least squares
objective function J defined in (2.46)

δJ = δue , λD H 1/2 ,H −1/2 (2.99)

   
+ δa∇u·∇λ + δk(u) λ − δf λ − δg λ.
Ω Ω Ω ∂ΩN

This formula is the continuous equivalent of the “gradient equations” of the

discrete case: it shows that λD is the derivative with respect to the Dirichlet
boundary condition ue , that ∇u · ∇λ is the derivative with respect to the
diffusion coefficient a, that λ is the gradient with respect to the right-hand
side f , and that the trace of λ on ∂ΩN is the gradient with respect to the
2.8. EXAMPLE 5: DIFFERENTIAL EQUATIONS 65

Neumann condition g. When the adjoint state λ is regular enough so that its
level sets
Cv = {x ∈ IR2 | u(x) = v}, umin ≤ v ≤ umax
are regular curves, the differential with respect to k can be written as
 umax 
δJ = δk(v) λ,
umin Cv

so that the derivative of J with respect to the nonlinearity k for a value v of

u is the integral of λ along the level line Cv of u.

Remark 2.7.2 This example gives a first illustration of the difference be-
tween discrete and discretized gradient approaches mentioned in the intro-
duction of the chapter: they both require to choose a discretization of the
direct state equation (2.44) and the objective function (2.46), for example,
the ones described in Sects. 2.6.1 and 2.6.2. But once this is done, the dis-
crete adjoint equations (2.74) and (2.75) and the discrete derivative formulas
(2.77) follow unambiguously, whereas determination of the discretized adjoint
equations and gradient formula would require further to discretize the adjoint
equations (2.91) and (2.92) and the derivative formulas (2.99). There are
usually many different ways to do this, so that there is no guarantee that the
discretized adjoint equations will coincide with the discrete adjoint equations
(2.74) and (2.75), which are the only one that lead to the exact gradient of
Jh . For the problem under consideration, it is reasonable to think that a sea-
soned numerical analyst would choose the discrete adjoint equation (2.74) as
an approximation to (2.91), but it is most unlikely that he would have cho-
sen the intricate equation (2.75) as an approximation to the simple equation
(2.92), in which case the discretized gradient will be only an approximation
to the exact gradient of the discrete objective function.

2.8 Example 5: Differential Equations,

Discretized Versus Discrete Gradient
Consider the system of differential equations (state equation):
du
= f (u(t), a) for t > 0, u(0) = u0 , (2.100)
dt
66 CHAPTER 2. COMPUTING DERIVATIVES

where t  u(t) ∈ IRm is the (infinite dimensional) state variable, and where
the parameter vector a ∈ IRn and the initial data u0 ∈ IRm are to be
estimated.
Consider also that a measure z ∈ IRm of the state u(T ) at a given time
T is available for this (observation operator: u  u(T )). The least squares
objective function is then
1
J(a, u0 ) = z − u(T )2IRm . (2.101)
2
It is now an exercise to determine the continuous adjoint state and deriva-
tive formulas as we did for the elliptic problem in Sect. 2.7 (the Green formula
is replaced by integration by part). To compute the gradient with respect to
the initial condition u0 , one decides first not to include the initial condition
in an affine state-space, but rather to consider it as a state equation (this cor-
responds exactly to the Option 1 choice for the state space Y in Sect. 2.6).
The starting point is hence the following Lagrangian:
⎧
⎨ L(a, u0, u, λ, λ0) = 2 z − u(T )IRm
1 2
T
+ 0 (f (u(t), a) − du )·λ (2.102)
⎩ dt
+ (u0 − u(0)) · λ0 ,

the resulting adjoint equations are

 T
dλ ∂f
− = (u(t), a) λ(t) for t > 0, λ(T ) = u(T ) − z, (2.103)
dt ∂u
λ0 = λ(0), (2.104)

and the formulas for the derivatives (“gradient equations”) is

 T
∂f
δJ = (u(t), a)δa · λ(t) + λ0 · δu0 .
0 ∂a
The gradients of J with respect to a and u0 are then
 T  T
∂f
∇a J = (u(t), a) λ(t), (2.105)
0 ∂a

∇u0 J = λ0 = λ(0). (2.106)

2.8. EXAMPLE 5: DIFFERENTIAL EQUATIONS 67

To solve this problem on the computer, we discretize the differential equa-

tions (2.100) and the objective function (2.101). We introduce the times
tk , k = 0 . . . K, such that

0 = t0 < · · · < tk < · · · < tK = T,

and the corresponding time steps

hk+1/2 = tk+1 − tk .

Then we can decide, for example, to replace (2.100) by the discrete state
equation
uk+1 − uk
= f (uk+θ , a) , k = 1 . . . K , u0 = u0 (2.107)
hk+1/2
for some θ ∈ [0, 1], where we have used the convenient notation

uk+θ = (1 − θ)uk + θuk+1.

The scheme is explicit for θ = 0, and implicit for 0 < θ ≤ 1. We denote by

uh = (uk , k = 0 . . . K) ∈ IR(K+1)m

the solution of (2.107), which is supposed to exist. The vector uh is the state
and IR(K+1)m the state-space of the system.
The observation operator associated with final time data is M : uh  uK ,
and we can decide, for example, to approximate the objective function J by
1
Jh (a, u0 ) = z − uK 2IRm
2
At this point, we have the choice between
– Either take advantage of the fact that we have already computed the
continuous adjoint, and go for the discretized adjoint approach
– Or use the continuous adjoint only as a guideline for the choice of the
scalar products, and go for the discrete adjoint approach
We investigate now these two possibilities.
68 CHAPTER 2. COMPUTING DERIVATIVES

2.8.1 Implementing the Discretized Gradient

There are two more decisions to make in this approach:
– One has to choose one discretization of the adjoint equations (2.103) and
(2.104). By analogy with (2.107), it is natural to define the approximation

λh = (λk , k = K . . . 0) ∈ IR(K+1)m

of the solution t  λ(t) of (2.103) by

 T
λk−1 − λk ∂f
= (uk−θ , a) λk−θ , k = K . . . 1, λK = uK − z. (2.108)
hk−1/2 ∂u

It is then also natural to replace (2.104) for the multiplier λ0 associated with
the initial condition by
λ0,h = λ0 . (2.109)
– One has to discretize the formulas (2.105) and (2.106) for the gradients
of J. Replacing the integral by the trapezoidal rule, we obtain
K
   T 
T
∂f ∂f
(∇h )a Jh = hk−1/2 (uk−1, a) λk−1 + (uk , a) λk (2.110)
1
∂a ∂a
(∇h )u0 Jh = λ0 = λ0 . (2.111)

The index h in ∇h remembers us that this quantity is only in general an

approximation to the gradient.

2.8.2 Implementing the Discrete Gradient

The good side here is that there is no more decision to make, and that we
shall obtain the exact gradient of Jh . But the dark side is that we need to
redo the adjoint calculation at the discrete level. This usually leads to
cumbersome calculations, which we detail now.
We write first the discrete Lagrangian Lh (a, u0 ; uh ; λh , λ0,h ). We need to
choose a name for the Lagrange multiplier associated with the kth equation of
(2.107): as this equation computes uk+1 from uk , its multiplier is associated
with the [k, k + 1] interval. A natural solution would be to call it λk+1/2
in absence of further information. But it will appear on the final discrete
adjoint equations that this multiplier can be interpreted more precisely as
2.8. EXAMPLE 5: DIFFERENTIAL EQUATIONS 69

an approximation of λ(t) at the time tk+1−θ = (1 − θ)tk+1 + θtk , so that we

shall call it λk+1−θ . So we define the discrete Lagrange multipliers by
λh = (λ1−θ . . . λK−θ ) ∈ IRKm , λ0,h ∈ IRm . (2.112)
Then we need to choose a scalar product on IRKm and IRm . By analogy with
the continuous formula (2.102), we choose on IRKm, a scalar product that
mimics the integral from 0 to T of the scalar product in IRm , and we equip
IRm with the usual scalar product. The discrete Lagrangian is then
⎧
⎪
⎪ Lh (a, u0 ; uh ; λh , λ0,h ) = 12 z − uK 2IRm
⎪
⎪
⎨    k+1−θ
+ K−1 h k+1/2
f (u k+θ
, a) − (u k+1
− u k
)/hk+1/2
·λ
⎪
⎪
k=0
⎪
⎪
⎩
+(u0 − u0 ) · λ0,h ,

where as previously uk+θ is a notation for (1 − θ)uk + θuk+1.

The discrete adjoint equation is then obtained as usual by equating to
zero the partial derivative of Lh with respect to uh :
⎧
⎪
⎪ ∂Lh
⎪
⎪ δuh = (uK − z) · δuK
⎪
⎪ ∂u
⎪
⎪
h
⎨  
K−1 k+1/2 ∂f
⎪
⎪ + k=0 h
k+θ
(u , a) δu k+θ
− δu k+1
+ δu · λk+1−θ (2.113)
k
⎪
⎪ ∂u
⎪
⎪
⎪
⎪
⎩ −δu0 · λ ∀δuh = (δuk , k = 0 . . . K) ∈ IR(K+1)m .
0,h = 0

The K + 1 formulas for the computation of λh and λ0,h will be obtained by

equating to zero the coefficients of δuk in (2.113) for k = 0 . . . K. We perform
for this purpose a discrete
K integration by parts, which consists in rewriting
(2.113) in the form k=0 (. . . )δuk = 0. We call A, B, C, D, E the five terms of
(2.113), and reorganize them into the desired form. We notice first that δuK
is missing in the D term, but present in A. So we define λK+1−θ ∈ IRm by
λK+1−θ = uK − z, (2.114)
and rewrite the A + D terms as
K
A+D = δuk · λk+1−θ ,
k=0
70 CHAPTER 2. COMPUTING DERIVATIVES

which is of the desired form. Similarly, we see that δu0 is missing in the C
term, but present in E, so we define λ−θ ∈ IRm by
λ−θ = λ0,h , (2.115)
and rewrite the C + E terms as
K−1
C +E =− δuk+1 · λk+1−θ − δu0 · λ−θ ,
k=0

or, with an index shift

K
C +E =− δuk · λk−θ ,
k=0

which is also of the desired form. We work now on the B term:

K−1
∂f k+θ
B= hk+1/2 (u , a)δuk+θ · λk+1−θ ,
k=0
∂u

or, remembering that δuk+θ = (1 − θ)δuk + θδuk+1 ,

K−1
∂f k+θ
B = (1 − θ) hk+1/2 (u , a)δuk · λk+1−θ
k=0
∂u
K−1
∂f k+θ
+θ hk+1/2 (u , a)δuk+1 · λk+1−θ .
k=0
∂u
We see that δuK is missing in the sum of the first line. But if we define
tK+1 = tK , hK+1/2 = tK+1 − tK = 0,
we can extend this sum up to index k = K, as this adds only a zero term.
Similarly, δu0 is missing in the sum of the second line, but if we define
t−1 = t0 , h−1/2 = t0 − t−1 = 0,
we can extend the sum to k = −1. This gives, after an index shift,
K
∂f k+θ
B = (1 − θ) hk+1/2 (u , a)δuk · λk+1−θ
k=0
∂u
2.8. EXAMPLE 5: DIFFERENTIAL EQUATIONS 71

K
∂f k−1+θ
+θ hk−1/2 (u , a)δuk · λk−θ .
k=0
∂u
k k
If we define h and θ for k = 0 . . . K by

hk = tk+1−θ − tk−θ = (1 − θ)hk+1/2 + θhk−1/2
h θ = θhk−1/2 ,
k k
hk (1 − θk ) = (1 − θ)hk+1/2 .

we see that
tk = (1 − θk )tk−θ + θk tk+1−θ ,
and we can rewrite B as follows:
K  T
∂f k+θ
B= h (1 − θ )
k k
(u , a) λk+1−θ · δuk
k=0
∂u

K  T
∂f k−1+θ
+ k k
h θ (u , a) λk−θ · δuk ,
k=0
∂u
which is of the desired form.
The final discrete adjoint equations for the determination of λh =
(λ . . . λK−θ ) defined in (2.112), λK+1−θ and λ−θ defined in (2.114) and
1−θ

(2.115), and λ0,h are then

 T
λk−θ − λk+1−θ ∂f k+θ
= (1 − θ )
k
(u , a) λk+1−θ (2.116)
hk ∂u
 T
k ∂f k−1+θ
+θ (u , a) λk−θ , k = K ...0,
∂u
λK+1−θ = uK − z (2.117)
λ0,h = λ−θ (2.118)
Differentiation of the Lagrangian Lh with respect to the parameters a
and u0
K−1  
∂f k+θ
δLh = hk+1/2
(u , a) δa · λk+1−θ − δu0 · λ0,h
k=0
∂a
72 CHAPTER 2. COMPUTING DERIVATIVES

gives then the discrete gradient equations by picking up the coefficients

of δa and δu0 :
⎧  T
⎨ K−1 k+1/2 ∂f k+θ
∇a J h = k=0 h (u , a) λk+1−θ ,
∂a (2.119)
⎩
∇u0 Jh = −λ0,h .

Comparison of Discretized and Discrete Gradients

Once again, we see that the discrete adjoint equations (2.116)–(2.118) and
the formulas in (2.119) for the discrete gradient differ substantially from
their discretized counterparts (2.108)–(2.111). Moreover, the discrete formu-
las are far from being the most natural ones, which makes it very unlikely
for the discrete formulas to be chosen by chance when using the discretized
approach. The author remembers of a stiff parabolic problem, where the dis-
crete approach had been used for a time discretization with a constant time
step, and the correctness of the gradient produced by the code had been
validated by comparison with finite differences. But when it came to appli-
cation to real data, a variable time step was used to limit the computational
cost, and the code was simply modified by replacing everywhere the fixed
time step by the variable one, which amounted to a discretized approach. At
that point, the code blew up during the computations, and a lot of time was
spent looking up for a coding error; but the problem disappeared only after
the discrete adjoint and gradient formulas were implemented for the variable
time steps.

Adaptive Time Steps

Most simulators for time-dependent problems use adaptive time stepping to
adapt to the stiffness of the problem and reduce the computational costs.
When this happens, the time steps depend on the parameters, and should
be differentiated in the discrete gradient approach. But time stepping pro-
cedures are often not differentiable and poorly documented, and taking into
account this dependance would lead to still more tedious calculations. Luckily
enough, if the time steps are correctly estimated, small perturbations of these
time steps should have a minor influence on the solution, and it is the au-
thor’s experience that this influence can be usually neglected. Hence the first
reasonable thing to do is to calculate the discrete gradient with the variable,
2.9. EXAMPLE 6: DISCRETE MARCHING PROBLEMS 73

but fixed time steps determined by the time stepping procedure for the cur-
rent parameter value. Whenever the objective function has to be evaluated
for a different value of the parameters, for example, during the line search,
the time steps are determined again by the time stepping procedure.

2.9 Example 6: Discrete Marching Problems

It is sometime convenient to forget the continuous differential equation un-
derlying a marching problem. This can be the case when a simulation code
already exists for the marching problem of interest, and one wants to per-
form some optimization based on this code (see “Sensitivity versus adjoint”
below). Without loss of generality, the state equations e(a, u) = 0 for such a
problem can be written as

E k−1/2 (a, uk , uk−1) = 0, k = 1 . . . K, (2.120)

u0 = u0 , (2.121)
where a ∈ IRn is a vector of parameters, and u0 ∈ IRm is the initial value. For
sake of simplicity, we shall suppose here that u0 is known, and that a is the
parameter vector to be estimated, so that the parameter space is E = IRn .
But there is no difficulty in handling the case where u0 is unknown (see
Option 1 in Sect. 2.6).
When (2.120) is nonlinear with respect to uk , it has to be solved only ap-
proximately on the computer using an iterative scheme (a Newton algorithm,
for example). Such algorithms are governed by tests, and hence are not dif-
ferentiable. So it is practically impossible to include them in the definition
of the forward map, and, for the sake of discrete gradient computations, one
usually considers that (2.120) is the discrete equation, and that it is solved
exactly by the computer. This acceptable if the equation is solved precisely
enough.
As for the observation, let us consider, for example, the case where, at
each “time” index k, a measurement z k of

v k = M k (uk ) (2.122)

is available, where M k is the observation operator at index k:

M k : IRm  IRmk .
74 CHAPTER 2. COMPUTING DERIVATIVES

The observation space is then

F = IRq , where q = m1 + · · · + mK ,
and the forward map to be inverted is
ϕ : a ∈ E = IRn  (v 1 . . . v k ) ∈ F = IRq . (2.123)
Now, given a data vector
z = (z 1 . . . z K ) ∈ IRq ,
the data misfit function for the estimation of a is
K
1
J(a) = z k − M k (uk )2IRmk . (2.124)
2 k=1

A necessary step for the estimation of a knowing z is to compute the gradient

of J with respect to the parameter a. We compare now on this problem the
sensitivity functions and adjoint approaches. We recall that ∂E k−1/2 /∂uk
and ∂E k−1/2 /∂uk−1 denote the partial derivatives of E k−1/2 with respect to
its second and third arguments.

Sensitivity Functions Approach

According to Sect. 2.2, we differentiate (2.120)–(2.122) with respect to aj for
j = 1 . . . N. This gives the following formula for the jth column sj of the
Jacobian D of the forward map ϕ : a  (v 1 . . . v k ) (jth output sensitivity
function):
∂M(uk ) ∂uk
skj = = (M k ) (uk ) , k = 1 . . . K,
∂aj ∂aj
where the state sensitivity functions ∂uk /∂aj are given by (all partial deriva-
tives of E k−1/2 are evaluated at a, uk , uk−1 )
∂E k−1/2 ∂uk ∂E k−1/2 ∂uk−1 ∂E k−1/2
+ + = 0, (2.125)
∂uk ∂aj ∂uk−1 ∂aj ∂aj
∂u0
= 0. (2.126)
∂aj
The gradient of J is then given by
K
∂J
= (M k (uk ) − z k ) · skj j = 1 . . . N.
∂aj k=1
2.9. EXAMPLE 6: DISCRETE MARCHING PROBLEMS 75

Adjoint Approach
We follow the step-by-step approach of Sect. 2.4:
Step 0: The forward map ϕ is already defined in (2.123), and, as we want to
compute the gradient of J defined in (2.124), the objective function G(a, v) is
K
1
G(a, v) = z k − v k 2IRmk (independant of a). (2.127)
2 k=1

In absence of information on the need of parameter scaling, on the nature of

uncertainty on the data z, and on any continuous analogon of the problem,
we simply equip the parameter space IRn and the data space IRq with the
usual Euclidean scalar products.
Step 1: We do not need to compute the gradient with respect to the initial
condition u0 , as u0 is known, and so we can include this condition in an affine
state-space Y , according to Option 2 of Sect. 2.6:
Y = {u = (u0 . . . uK ) ∈ IR(K+1)m | u0 = u0 },
with the associated vector space:
δY = {δu = (0, δu1 . . . δuK ) ∈ IR(K+1)m } = IRp with p = Km.
The corresponding observation operator is
M : u = (u0 . . . uK ) ∈ Y  v = (M 1 (u1 ) . . . M K (uK )) ∈ IRq . (2.128)
Step 2: The Lagrangian is then, with G, is defined in (2.127) and M in
(2.128):
K
L(a, u, λ) = G(a, M(u)) + E k−1/2 (a, uk , uk−1) · λk−1/2 , (2.129)
k=1

where ⎧
⎨ a ∈ IRn
u = (u0 , u1 . . . uK ) ∈Y
⎩ 1/2
λ = (λ . . . λ K−1/2
) ∈ IRp .
Step 3: Differentiation of the Lagrangian with respect to the state u gives
the variational form of the adjoint equation:
∂L
δu = A + B + C = 0 ∀δu = (0, δu1 . . . δuK ) ∈ δY,
∂u
76 CHAPTER 2. COMPUTING DERIVATIVES

where
K
A= (M k (uk ) − z k ) · (M k ) (uk )δuk ,
k=1
K
∂E k−1/2
B= (a, uk , uk−1) δuk · λk−1/2 ,
k=1
∂uk
K
∂E k−1/2
C= (a, uk , uk−1 ) δuk−1 · λk−1/2 .
∂uk−1
k=1

We perform now a discrete integration by part to factorize the δuk terms:

one notices that the sum in the C term starts in fact at k = 2 (remember
that δu0 = 0), and that it can be extended to K + 1, provided we define
λK+1/2 ∈ IRp by λK+1/2 = 0. After an index shift, the C term becomes
K
∂E k+1/2
C= k
(a, uk+1, uk ) δuk · λk+1/2 .
k=1
∂u

Equating to zero successively the coefficients of δu1 . . . δuK gives the compu-
tational form of the adjoint equations (all partial derivatives of E k−1/2 are
evaluated at a, uk , uk−1, those of E k+1/2 are evaluated at a, uk+1, uk )
 k−1/2 T  k+1/2 T
∂E k−1/2 ∂E
k
λ + k
λk+1/2 (2.130)
∂u ∂u
 T
+ (M k ) (uk ) (M k (uk ) − z k ) = 0 for k = K . . . 1,
which can be solved backwards starting from the final condition
λK+1/2 = 0. (2.131)
Step 4: We differentiate now the Lagrangian (2.129) with respect to the
parameter vector a (partial derivatives of E k−1/2 are evaluated at a, uk , uk−1):
K
∂L ∂E k−1/2
δJ = δa = δa · λk−1/2 ,
∂a k=1
∂a

and pick the coefficient of δaj , which gives the gradient equations
K
∂J ∂E k−1/2 k−1/2
= ·λ j = 1 . . . N.
∂aj k=1
∂aj
2.9. EXAMPLE 6: DISCRETE MARCHING PROBLEMS 77

Sensitivity Versus Adjoint

The general pros and cons of sensitivity function and adjoint approaches for
the minimization of J apply here:

• The computational cost of the sensitivity function approach increases

with the number of parameters (there are as many linearized equations
(2.125) and (2.126) to solve as parameters), and changing the parame-
ters is not easy (it requires to recode the linearized equations). But it
gives not only the gradient of J, but also the Jacobian D of the forward
map ϕ, and allows to use Gauss–Newton or Levenberg–Marquardt op-
timization algorithms. It also does not require a large memory, as the
linearized equations can be solved along with the original marching
scheme.

• The computational cost of the adjoint approach is independent of the

number of parameters (one adjoint equation (2.130) and (2.131) in all
cases), which makes it well suited for inverse problems with large num-
ber of parameters; changing the parameterization using the chain rule is
easy once the gradient has been computed with respect to the (possibly
large) set of simulation parameters (see (2.15) and Sect. 3.3 below). But
it does not compute the Jacobian, and so one is limited to optimiza-
tion algorithms of Quasi-Newton type, which usually require a larger
number of iterations. Also, the memory requirement can be extremely
large, as one needs to store or recompute the direct state u1 . . . uK be-
fore solving backwards for the adjoint state λK−1/2 . . . λ1/2 (see [40]) for
an optimal compromise between computation and storage).

The adjoint approach is the method of choice for the sensitivity analysis
study, which is to be performed before the inversion itself to find out the num-
ber of parameters that can be retrieved from the data (Sect. 3.2 of Chap. 3):
it allows to compute the Jacobian D with respect to the (usually large)
number of simulation parameters, before any choice is made concerning the
parameterization and the optimization parameters. This is possible because
in the adjoint approach the Jacobian is computed row by row, with a cost
proportional to the number of observations, but independent of the number
of parameter (of course the function G(a, v) to be used there is no more
(2.127), but rather (2.12)).
We conclude this example with some implementation remarks.
78 CHAPTER 2. COMPUTING DERIVATIVES

First, given a forward modeling code that solves the marching problem
(2.120) and (2.121), the matrix ∂E k−1/2 /∂uk is necessarily formed somewhere
in the code to perform the corresponding Newton iterations. But this ma-
trix is also the matrix of the linearized equation (2.125) in the sensitivity
approach. Hence the additional work to implement the sensitivity approach
consists in forming the matrices ∂E k−1/2 /∂uk−1 and ∂E k−1/2 /∂a, and in solv-
ing the same system as in Newton’s iterations, but with n different right-hand
sides. As the matrix of the system is already formed, the computational effort
for the resolution of each linearized equation (2.125) is much less than the
one required for one simulation. This approach has been used to implement
sensitivity equations in a complex nonlinear reservoir simulation code [6].
Second, comparison of the sensitivity equations (2.125) and the adjoint
equation (2.130) shows that the same matrices ∂E k−1/2 /∂uk , ∂E k−1/2 /∂uk−1 ,
and ∂E k−1/2 /∂a appear at both places. So when given a modeling code with
sensitivity equations capabilities, one can consider developing an adjoint code
by identifying these matrices in the code, and recombining them into the
desired adjoint equation. This approach has been used for the same reservoir
simulation code as above in [79].
Chapter 3

Choosing a Parameterization

We address in this chapter practical aspects of parameterization for the same

finite dimensional inverse problem (2.5) as in Chap. 2, where the forward map
x ∈ IRn  v = ϕ(x) ∈ IRq is defined by the state-space decomposition (2.1)
and (2.2):
– Because of their possibly different physical nature, it is necessary to
calibrate (Sect. 3.1) the simulation parameters by adimensionalization, to
avoid artifacts in the conditioning of the problem
– The singular value decomposition of the Jacobian D = ϕ (x) allows
then to estimate the number of independent parameters that can be retrieved
for a given level of uncertainty on the data (Sect. 3.2)
– In the (usual...) case where the above number of retrievable parameters
is smaller than the number of simulation parameters, it is necessary to add
a priori information on the parameters to restore wellposedness – or better
Q-wellposedness, see Chap. 4 – of the least squares problem. The regulariza-
tion methods available to this purpose have been presented in Sect. 1.3.4. In
any case, one has to make a clear distinction (Sect. 3.3) between simulation
parameters (which are coefficients or source terms in the numerical simu-
lation model) and optimization parameters (the ones that are seen by the
optimizer, and will be actually estimated)
– The regularization by size reduction will be discussed in Chap. 4, and
the Levenberg–Marquardt–Tychonov and the state-space regularization in
Chap. 5. We present in this chapter the regularization by parameterization,
which proceeds by reduction of the number of unknowns: the information is
added by choosing a set of formulas to compute the simulation parameters
from a smaller number of optimization parameters. A good parameterization
G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 79
and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 3, c Springer Science+Business Media B.V. 2009
80 CHAPTER 3. CHOOSING A PARAMETERIZATION

should allow to explain the data up to the noise level and should not lead
to overparameterization (the dimension of the optimization vector should
be smaller than the number of retrievable parameters determined above).
We evaluate in this chapter four parameterization against these objectives:
closed form formula (Sect. 3.4), singular vector basis (Sect. 3.5), multiscale
approximation (Sect. 3.6), and adaptive parameterization (Sect. 3.7)
– Finally, we discuss in Sect. 3.8 some implementation issues:
• How to organize the inversion code to allow an easy experimentation with
various choices of optimization parameters (Sect. 3.8.1)
• How to compute the gradient with respect to optimization parameters once
the gradient with respect to numerical parameters is known (Sect. 3.8.2)
• And, in Sect. 3.9, we describe the maximum projected curvature (MPC) de-
scent step, which is specially designed to enhance the performance of descent
algorithms used for the resolution of nonlinear least squares problems.

3.1 Calibration
The unknown parameters correspond often to different physical quantities:
for example, hydraulic conductivities, porosities, acoustic impedances, etc.
Similarly, the available observations can involve temperatures, pressures, con-
centrations, etc.
Before trying to recover (part of) the parameters from the data, it is
necessary to eliminate the poor conditioning that can be caused by the dif-
ferent orders of magnitude associated with the different physical quantities
in the parameter and the data vectors. So one first has to calibrate the finite
dimensional inverse problem (2.5) by using dimensionless parameters and
data.

3.1.1 On the Parameter Side

Let X denote the unknown parameter as described by the physics. The prin-
ciple is to first adimensionalize X by choosing a reference value Xref for
each physical parameter, and then to choose the calibrated parameter vector
x ∈ IRn representing X in such a way that

1. The Euclidean norm of x represents a mean value of the adimensional-

ized parameters
3.1. CALIBRATION 81

2. And, in the case where X is a distributed parameter on a domain Ω,

the Euclidean scalar product on IRn is proportional to the L2 (Ω) scalar
product of the functions Xh approaching X

The first property is a matter of convenience, as it allows simply a more

direct interpretation of the results. The second property is meant to ensure
that the discretization does not change (in particular does not worsen. . . )
the conditioning of the least squares optimization problem. Hence,

• When X is a finite dimensional vector of scalar coefficients, one chooses

first, for each component Xi , a reference value Xi,ref > 0, which incorpo-
rates the best available knowledge on its magnitude (Xi,ref can be the
same for a group of components corresponding to the same physical
quantity). The calibrated parameter vector x is then defined by

1 Xi
xi = √ , i = 1 . . . n,
n Xi,ref
√
where the coefficient n ensures that
 n  12
def
xIRn = x2i = mean value of X/Xref .
i=0

• When X is a function of a variable ξ defined over some domain Ω,

the reduction to a finite number n of scalar parameters is usually done
by discretization (notations of Sect. 2.6.1): the domain Ω is covered
by a simulation mesh Th made of a finite number of cells or elements
K, where the index h > 0 denotes the largest cell size in Th . A finite
dimensional approximation space E h is then chosen for the functions
X; a function ξ  Xh (ξ) of E h is characterized by n “natural” degrees
of freedom X1 . . . Xn (cell or node values).

1. For a discontinuous piecewise constant approximation: The func-

tion ξ  Xh takes a constant value Xi on the ith cell Ki . The L2
scalar product of functions of E h is given by

X0,h X1,h = |Ki | X0,iX1,i ,
Ω i=1...n
82 CHAPTER 3. CHOOSING A PARAMETERIZATION

where |Ki| is the measure (length, area, or volume) of Ki . To

ensure that the L2 scalar product of the functions Xh is propor-
tional to the Euclidean scalar product in IRn of the calibrated
parameter vectors x, we define xi on the cell Ki by
 |K |  12 X
i i
xi = , i = 1 . . . n, (3.1)
|Ω| Xref
where |Ω| is the measure (length, area, or volume) of Ω, and where
Xref > 0 is the reference value used to adimensionalize X. The
coefficient |Ω| in (3.1) ensures, as above, that the Euclidean norm
of x represents a mean value of the adimensionalized parameters:
 n  1  1  X (ξ)2  12
def 2 2 h
xIRn = xi = dξ . (3.2)
i=1
|Ω| Ω X 2
ref

2. For a continuous piecewise linear approximation: The function

ξ  Xh is now continuous, and linear over each cell, as described
in Sect. 2.6.1. The degrees of freedom of Xh are its values XM at
the nodes M of Th . One approximate scalar product in L2 (Ω) is
then, with the notations (2.52) and (2.53),

def
IΩ (X0,h X1,h ) = αM X0,M X1,M ≈ X0,h X1,h , (3.3)
Ω
M node of Th

where the coefficients αM , defined in (2.61), satisfy

αM = |Ω|. (3.4)
M node of Th

So we can still define the calibrated parameters by (compare

with (3.1))
 α  12 X
Mi Mi
xi = , i = 1 . . . n. (3.5)
|Ω| Xref
As in the case of discontinuous approximation, this ensures that
the Euclidean norm of x is an (approximate) mean value of the
adimensionalized parameter (compare with (3.2))
 n  1  1  X (ξ)2  12
def 2 2 h
xIRn = xi = IΩ , (3.6)
i=1
|Ω| 2
Xref
3.1. CALIBRATION 83

With the above calibrations, the Euclidean scalar product in IRn corresponds,
2
up to the multiplicative constant 1/(|Ω|Xref ), to the (possibly approximate)
2
scalar product in L (Ω) for the physical parameters
⎧ 
n
1 ⎨ Ω Xh (ξ)Yh(ξ) dξ,
def
x, yIRn = xi yi = 2 ⎩
or (3.7)
|Ω|X ref
i=1 IΩ (Xh Yh ),

according to the type of approximation chosen for the parameter (discontin-

uous piecewise constant or continuous piecewise linear).

Remark 3.1.1 In the case where a continuously derivable approximation of

X is required, then necessarily some of the degrees of freedom will be deriva-
tives (instead of values) of Xh , and it will be necessary, to satisfy (3.2), (3.6),
and (3.7), to use a nondiagonal scalar product on the parameter space IRn . It
will then be necessary to exercise some caution for the gradient calculation,
as explained in Remark 3.1.2 below.

3.1.2 On the Data Side

Similarly, let Z denote the data vector given by the physics, and ΔZj denote
the uncertainty on its jth component. It is convenient to use this uncertainty
to define the calibrated data zj :

1 Zj
zj = √ , j = 1 . . . q.
q ΔZj

This amounts to use ΔZj as unit to measure the discrepancy between the
output of the model and the corresponding data x (as in (1.8), for example).
So if δZ ∈ IRq is a vector of data perturbations such that |δZj | ≤ ΔZj
for j = 1 . . . q, the vector δz of corresponding calibrated data perturbation
satisfies
√ def
|δzj | ≤ 1/ q for j = 1 . . . q and hence δz ≤ Δz = 1,

so that the uncertainty on the calibrated data vector z is Δz = 1.

When the Zj ’s are all independent Gaussian variables with standard de-
viation σj , one can take ΔZj = σj , in which case problem (2.5) coincides
with the maximum likelihood estimator.
84 CHAPTER 3. CHOOSING A PARAMETERIZATION

3.1.3 Conclusion
For the rest of the chapter, the parameter x and data z that appear in the
NLS inverse problem (2.5) will be the calibrated parameter and data defined
earlier, with the parameter space E and data space F equipped with the
Euclidean norms
 n
1/2  q
1/2
xE = x = x2i , vF = v = vj2 ,
i=1 j=1

which represent mean values of the adimensionalized parameters and data.

Remark 3.1.2 The calibration could have been taken into account as well
by introducing weighted scalar products and norms on the parameter space
IRn and the data space IRq . But this approach is error prone from a practical
point of view, as the transpose of a matrix is no more simply obtained by
exchanging rows and columns! For example, if ·, ·Λ is the scalar product
on IRq associated with a symmetric positive definite matrix Λ, the transposed
M TΛ , for this scalar product, of a q×q matrix M is Λ−1 M T Λ, where M T is the
usual transposed. This can be easily overseen in the numerical calculations.
This approach, however, cannot be avoided in the case where a full (i.e.,
not diagonal) covariance matrix of the data is available – but all the material
below can be easily adapted.

3.2 How Many Parameters Can be Retrieved

from the Data?
Before any attempt is made to minimize the least squares objective function
(2.5), the first thing to do is determine the largest number of independent
parameters that can be retrieved from the sole observations at a given noise
level on the data. This has to be done before any a-priori information is
added into the system: the parameters to be used for this determination are
the simulation parameters (Sect. 3.3 below), which are input to the simula-
tion code before any parameterization is chosen. In most cases, the original
unregularized problem is underdetermined, with a number n of simulation
parameters much larger than the number q of observations.
This analysis is performed on the linearized problem at a few (usually
one) nominal value xnom . It is based on the knowledge of the “noise level”
3.2. HOW MANY PARAMETERS CAN BE RETRIEVED 85

on the data, and on the singular values decomposition of the Jacobian ϕ (x).
There is a large amount of literature on this subject, which usually takes into
account the statistical properties of the errors on the data, see for instance
[74]. We give below an elementary approach based on uncertainty analysis,
but which is sufficient in the large number of cases where the statistical
information on the data is lacking or scarce.
So one chooses a nominal parameter value xnom ∈ C, and replaces the
forward map ϕ by its linearization ϕlin
nom :


∀δx ∈ IRn , ϕlin
nom (δx) = ϕ(xnom ) + ϕ (xnom ) δx.

Let znom = ϕ(xnom ) = ϕlin

nom (0) be the exact data corresponding to xnom ,
Δz > 0 the uncertainty level on the data, δz with δzF ≤ Δz a data
error vector, z = znom + δz the corresponding noise corrupted data, and
x̃ = xnom + δx the corresponding solution of the linearized unconstrained
inverse problem:
1 lin 1
δx minimizes ϕnom (δx) − z2F = ϕ (xnom ) δx − δz2F over IRn . (3.8)
2 2
We evaluate the size of the error δx induced by the error δz on the data znom
at the nominal value znom :

• Absolute uncertainty: One performs a singular value decomposition

(SVD) of the q × n matrix ϕ (xnom ). This produces two orthonormal
bases 
e1 , . . . , en = basis of parameter space IRn ,
(3.9)
1 , . . . , q = basis of data space IRq ,
and a sequence of r ≤ min{n, q} strictly positive numbers:

μ1 ≥ μ2 ≥ · · · ≥ μr > 0 (3.10)

called the singular values such that

 
ϕ (xnom )ei = μi i , i = 1, · · · , r,
(3.11)
ϕ (xnom )ei = 0, i = r + 1, · · · n.

It is then convenient to complement the above singular values μi , i =

1 · · · r, by defining:

μr+1 = μr+2 = · · · = μn = 0.
86 CHAPTER 3. CHOOSING A PARAMETERIZATION

The solution of (3.8) is then given by

μi δxi = δzi , i = 1 . . . r, (3.12)

where δxi and δzi are the coefficients of δx and δz on the singular bases:
 
δx = ni=1 δxi ei ,
q
δz = j=1 δzj j .

This shows first that the perturbations δxi , i = r + 1 · · · n, cannot be

retrieved from the available data, as they do not appear in (3.12), which
give the solutions of (3.8). Hence the uncertainty on the components
xi of xnom on singular vectors corresponding to zero singular values is
infinite:
Δxi = +∞, i = r + 1 · · · n.
Then for i = 1 · · · r, (3.12) gives, as |δzi | ≤ δzF ≤ Δz,

μi |δxi | ≤ Δz, i = 1 . . . r,

so that the uncertainty Δxi on the component xi of xnom on singular

vectors corresponding to nonzero singular values is given by
Δz
Δxi = , i = 1 . . . r. (3.13)
μi
Summing up, we see that the components of xnom on singular vectors
associated with zero singular values cannot be retrieved (the corre-
sponding uncertainty Δxi is infinite). For the components on the other
singular vectors, the larger the singular value, the smaller the uncer-
tainty! To determine which of these components can be retrieved in a
stable way, one has to estimate the relative uncertainty.

• Relative uncertainty: We suppose that the forward map ϕ satisfies

ϕ is defined at 0 and ϕ(0) = 0

(this condition can always be satisfied by translating the origin in the

parameter and data spaces).
3.2. HOW MANY PARAMETERS CAN BE RETRIEVED 87

Then, if the linearized model at xnom is valid, one has

0 = ϕ(0)  ϕ(xnom ) +ϕ (xnom )(0 − xnom ),

! "# $
znom
so that
znom  ϕ (xnom )xnom .
But the largest singular value μ1 is the norm of the Jacobian ϕ (xnom )
associated with the usual Euclidian norm, hence,
< μ1 xnom .
znom  ∼ (3.14)

Combining (3.13) and (3.14) gives

Δxi < μ1 Δz
, i = 1 . . . r. (3.15)
xnom  ∼ μi znom 
So we see that the relative uncertainty on the component xi of xnom
on one of the first r singular vectors is amplified at most by a factor
μ1 /μi from the relative uncertainty on the data. For the n−r remaining
components xi corresponding to singular values μi = 0, one has seen
that Δxi = + ∞, so that formula (3.15) remains valid in that case.

• Number of retrievable parameters: The component of xnom on the ith

singular vector ei is said to be retrievable if it is above the noise level,
that is, if its relative uncertainty is smaller than one.
Hence the number nr ≤ n of retrievable parameters for an uncertainty
level Δz on the data is given by
μi Δz
nr = number of singular values μi s.t.: ≥ .
μ1 znom 

The determination of the number nr of retrievable parameter is the first

numerical computation to perform for the solution of an inverse prob-
lem. It gives an upper bound to the number of independent parameters
that can be estimated from the data in a stable way.
Its practical determination requires the computation of the Jacobian
ϕ (xnom ). Because the number n of (simulation) parameters is usually
much larger than the number q of data, this is most efficiently done
88 CHAPTER 3. CHOOSING A PARAMETERIZATION

row-by-row by the adjoint state technique (Sect. 2.3 of Chap. 2). Of

course, in the rare case where n ≤ q, a column-by-column determination
of the Jacobian using the sensitivity functions approach (Sect. 2.2 of
Chap. 2) is the computationally less expensive approach. Examples of
application of this uncertainty analysis can be found in [50, 2, 80].

Remark 3.2.1 It can happen that the computation of the Jacobian is not
feasible when there are large number of parameters and data, so that nr
cannot be determined before the inversion is actually performed. However, the
determination of the gradient ∇J(x) is always feasible by adjoint state. It is
advisable in this case to use an adaptive parameterization (Sect. 3.7 below),
which can use the information given by ∇J(x) to take care automatically of
the tradeoff between data fitting and overparameterization.

3.3 Simulation Versus Optimization

Parameters
It is important to distinguish two (usually distinct) parameter vectors:
• The simulation parameters xsim (denoted by x in Sect. 1.3.4) are the
ones that are input to the simulation code. They should contain all
model parameters one might dream to invert for, and provide the most
comprehensive description of the numerical forward model used.
In particular, when the unknown parameter is a function, the numerical
computation of the forward model ϕ requires its discretization over
some mesh Th , and the simulation parameter xsim will be the (large)
vector made of all the calibrated degrees of freedom of this function
over Th . Its dimension nsim will hence in general be much larger than
the dimension q of the data vector z.
For example, if the unknown parameter is the diffusion coefficient a
in the inverse problem of Sect. 1.6, and if the resolution of the elliptic
equation is made, as proposed in Sect. 2.6.1, on a mesh Th of triangular
finite elements K, the natural choice for xsim is the vector of the cali-
brated values of the discretized parameter ah on Th , where ah is defined
in (2.54). The simulation parameter is hence given by (3.1), where now
Xi represents the (constant) value aKi of ah over the ith element of Th ,
and Xref the reference value chosen to adimensionalize a.
3.3. SIMULATION VERSUS OPTIMIZATION PARAMETERS 89

Not surprisingly, the analysis of the previous section, when applied to

the estimation of xsim from the data z, usually shows that the number nr
of retrievable parameters is strictly smaller than the dimension nsim of
xsim . Hence solving the optimization problem (2.5) with the parameters
x = xsim is purposeless, as it would be underdetermined. It is then
necessary to regularize the problem by adding a-priori information, as
discussed in Sect. 1.3.4, until all parameters are “above the noise level”
induced by the uncertainty on the data. We focus in the present chapter
on the regularization by parameterization approach, which proceeds by
reduction of the number of unknowns.

• The optimization parameters xopt are the ones that are input to the opti-
mization code, and hence which are actually estimated by minimization
of the objective function. The vector xopt is used to parameterize the
simulation parameter xsim (c.f. (1.16) of Sect. 1.3.4):

xsim = ψ(xopt ), xopt ∈ Copt with ψ(Copt ) ⊂ C, (3.16)

where ψ is the parameterization routine.

The dimension nopt of xopt has to be taken small enough to avoid over-
parameterization, that is, the Jacobian of ϕ as a function from IRnopt
to IRq should have all its singular values above the noise level (Fig. 3.1),
and large enough for the solution x opt of (2.5) to explain the data up to
the noise level. Moreover, the chosen parameterization should if possible
enhance the conditioning of the optimization problem (2.5).

The choice of optimization parameters is a critical, but delicate step of the

inversion, and one is very often led to experiment with it. It is hence practi-
cally important to organize the code in such a way that changing optimization
parameters incurs a minimum amount of changes in the code. When the in-
formation flux between the optimization routine and the forward+adjoint
modeling code is organized as in the flowchart of Fig. 3.8 below, changing
optimization parameters requires only to replace the two subroutines that
compute xsim = ψ(xopt ) (parameterization routine) and gopt = ψ  (xopt )T .gsim
(adjoint parameterization routine). This suggest that these two subroutine
should be developed and tested at the same time, prior to being inserted in
the inversion code. The necessary mathematical tools have been developed
in Chap. 2, and are applied in Sect. 3.8.2 below to an example.
90 CHAPTER 3. CHOOSING A PARAMETERIZATION

μi
μ1

Δz
znom

1 nr = 4 r=8 n i

Figure 3.1: Graphical determination of the number of retrievable

parameters: μ1 , . . . , μn = singular values, in decreasing order

Remark 3.3.1 Even in the case where all parameters in xsim are retrievable,
the choice xopt = xsim does not necessarily produce the best conditioning of the
optimization problem, and one can be led to choose a parameter xopt distinct
from xsim , see adapted multiscale basis in Sect. 3.6.

We describe below four approaches to reduce the number of unknown

parameters.

3.4 Parameterization by a Closed

Form Formula
When the information content of the data is very small, and the number
of simulation parameters is large, one solution is to express the simulation
parameters as a function of a small number of coefficients through a closed
form formula.
3.5. DECOMPOSITION ON THE SINGULAR BASIS 91

For example, suppose that the unknown parameter is a one-dimensional

temperature profile T on the [0, 1] interval. A natural simulation parameter
is then
xsim = (T1 /Tref , . . . , Tnsim /Tref ),
where Tref is a reference temperature used for calibration purpose (Sect. 3.1),
and where Ti , i = 1 . . . , nsim , represents the values of the temperature on a
discretization of [0, 1] into nsim intervals of length h = 1/nsim . Suppose that
the singular value analysis of Sect. 3.2 shows that only a few number nr,sim of
independent simulation parameters can be recovered from the data, and that
one expects from the physics of the problem that the temperature profile has
the shape of a bump. One can then incorporate this information into our
formulation by searching (for example) for a Gaussian temperature profile:
Tmax  1  (i − 1/2)h − ξ 2 
max
xsim,i = exp − , i = 1 . . . nsim ,
Tref 2 Δξ
where the optimization parameter is
T 
max
xopt = , ξmax , Δξ ∈ IR3 .
Tref
Even if nr,sim was larger than three, it is necessary to perform again the anal-
ysis of Sect. 3.1, this time with the xopt parameter, as there is no guarantee
that the directions of IRnsim “spanned” (the quotes come from the fact that
xopt  xsim is not linear) by the chosen parameterization are in the subspace
of IRnsim spanned by the singular vectors associated with the largest singular
values μsim .
The effect on conditioning of parameterizing by a closed form formula
is unpredictable. However the author’s experience is that it tends rather
to deteriorate, this being partly compensated by the very small number of
parameters.
In the present situation where the simulation parameters xsim are ex-
pressed by a simple formula in term of the optimization parameters xopt , the
determination of ∇xopt J from ∇xsim J is very easy (Sect. 3.8.2 below).

3.5 Decomposition on the Singular Basis

It is the first natural idea after one has performed the singular value decompo-
sition of the Jacobian, at some nominal value xnomsim , in order to determine the
number nr of retrievable parameters (Sect. 3.2): why not reduce the search to
92 CHAPTER 3. CHOOSING A PARAMETERIZATION

the nr coefficients of xsim on the singular basis vectors associated with singular
values that are above the noise level? With the notations of Sect. 3.2, this
amounts to choose for optimization parameters and parameterization map:
⎧
⎨ nopt = nr ≤ nsim ,
def
x = xSVD = x sim , ei IRnsim ,  i = 1 . . . nr , (3.17)
⎩ opt,i i
xsim = ψ(xopt ) = i=1 xopt,i ei + ni=n
nr sim
r +1
(xSVD
i )nom ei .

There is no need, with this choice, to redo the analysis of Sect. 3.2 with the
parameters xopt , as by construction they are all above the noise level!
This parameterization is interesting for the analysis of the problem: the
directions of the singular vectors 1 . . . nr indicate which part of the data
space actually carries useful information on the parameter x. Similarly the
singular vectors e1 . . . nr indicate which combination of the simulation pa-
rameters can be best estimated.
These nice properties are counterbalanced by the fact that the correspond-
ing physical interpretation is not always easy (as, e.g., in [36] for a large size
geophysical inverse problem). A nice exception is when the singular vectors
associated with the largest singular values happen to point in the direction of
axes of the simulation parameter space IRnsim . It is then possible to order the
simulation parameters according to decreasing singular values, so that the
nr first simulation parameters are retrievable, each with its own uncertainty
level, with the remaining ones having to be fixed (at their nominal values, for
example). This is the case for the inversion of the Knott–Zoeppriz equations
of Sect. 1.1, see [50].
Another difficulty comes from the fact that the Jacobian – and hence
its singular vectors – depends on the choice of the nominal parameter xnom sim
where it is evaluated. So the nice properties of the singular vector can get
lost when the current parameter changes during the course of the optimiza-
tion, which makes their use purposeless for uncertainty analysis.
For all these reasons, the choice (3.17) is seldom used to define the op-
timization parameters – but performing a SVD at a nominal value is never-
theless useful to estimate nr and gain insight into the problem.
3.6. MULTISCALE PARAMETERIZATION 93

3.6 Multiscale Parameterization

Multiscale parameterization concerns the case of distributed parameters,
where the unknown is a function ξ ∈ Ω  X(ξ) ∈ IR: most often ξ is
the space variable, in which case X describes a heterogeneity, but not nec-
essarily: it can also be a dependent variable (as temperature, concentration,
saturation, etc.), in which case X represents a nonlinearity.

3.6.1 Simulation Parameters for a Distributed

Parameter
According to Sect. 3.3, the natural simulation parameter xsim for a distributed
physical parameter X is the vector of the calibrated values xi of its approxi-
mation Xh ∈ E h over the simulation mesh Th :
xsim,i = xi , i = 1 . . . nsim . (3.18)
These calibrated values xi , defined in Sect. 3.1.1, are the coefficients of the de-
composition of Xh on a set of calibrated local basis functions ei , i = 1 . . . nsim ,
nsim
Xh = xsim,i ei .
i=1

The formulas for xi given in Sect. 3.1.1 show the following:

1. For a discontinuous piecewise constant approximation, xi is given by
(3.1) and ei by
⎧   12
⎪
⎪ |Ω|
⎨ ei = Xref |K i|
χKi , i = 1 · · · nsim ,
(3.19)
⎪
⎪
⎩ |e |2 2 = X 2 |Ω|,
i L (Ω) ref

where χKi denotes the characteristic function of the ith cell Ki of the
simulation mesh Th .
2. For a continuous piecewise linear approximation, xi is given by (3.5)
and ei by ⎧
⎪   12
⎪
⎨ ei = Xref α|Ω| ωMi , i = 1 · · · nsim ,
Mi
(3.20)
⎪
⎪
⎩ |e | 2 = X |Ω| = 2|e | 2 ,
2 2 2
i h,L (Ω) ref i L (Ω)
94 CHAPTER 3. CHOOSING A PARAMETERIZATION

where ωMi denotes the local basis function of E h associated to node

Mi (its value is 1 at the ith node Mi and 0 at all other nodes of Th ),
and where | · |h,L2 (Ω) is the norm on E h associated to the scalar product
def
Xh , Yh h,L2(Ω) = IΩ (Xh Yh ),

with IΩ defined in (3.3).

Finally, for both approximations,
• The simulation parameters xsim,i represent calibrated local (cell or node)
values of the unknown parameter function over the mesh Th
• The calibrated basis e1 . . . ensim of E h given by (3.19) or (3.20) is normal-
ized (to the value Xref |Ω|1/2 ) and orthogonal for the scalar product ·, ·L2(Ω)
(discontinuous piecewise constant functions) or ·, ·h,L2(Ω) (continuous piece-
wise linear functions)
• The Euclidean scalar product of IRnsim corresponds (up to the coefficient
2
Xref |Ω|) – see (3.7) to the scalar product ·, ·L2 (Ω) or ·, ·h,L2(Ω) .
Once the simulation parameters xsim have been chosen, the maximum
number of retrievable parameters nr can be determined as explained in
Sect. 3.2.

3.6.2 Optimization Parameters at Scale k

A mesh T  is a sub-mesh of the mesh T (one writes T  ⊃ T ), if T  is a
refinement of T , that is, if any element K of T is the union of elements K  of
T  . In a scale-by-scale approach, the parameter is searched for successively
over a sequence of embedded meshes T 0 ⊂ T 1 ⊂ · · · ⊂ T K , which allow to
represent more and more details.
In Sect. 3.6.1, we have used as simulation parameter space the approxi-
mation space E h introduced in Sect. 3.1.1 over the simulation mesh Th . The
same approximation can now be used over each coarse mesh T k to define a
sequence of embedded function spaces E k :

E0 ⊂ E1 ··· ⊂ EK with dim E k = nk , k = 0 . . . K.

Each space E k is the approximation space at scale k, with k = 0 correspond-

ing to the coarse scale, and k = K to the fine scale. The finer mesh T K is
chosen fine enough so that
nk ≥ nr ,
3.6. MULTISCALE PARAMETERIZATION 95

where nr is the number nr of retrievable parameters determined at the end

of Sect. 3.6.1. If nr is not known, one can choose T k of a size similar to Th –
but the optimization will stop at a scale k usually smaller than K.
Let eki ∈ E k , i = 1 . . . nk , be a collection of basis functions of E k at each
scale k = 0 . . . K. The choice of this basis (local or multiscale) will be dis-
cussed in paragraph 4 of Sect. 3.6.3. The optimization parameter xkopt ∈ IRnk
at scale k is then made of the coefficients on this basis of the approximation
xk of x in E k : n k
k
x = xkopt,i eki .
i=1

Given an optimization parameter xkopt ∈ IRnk , the simulation parameter

xsim ∈ IRnsim is computed (see (3.16)) by

xsim = ψ k xkopt , (3.21)

where the parameterization matrix ψ k represents the interpolation from the

optimization parameter space E k equipped with the chosen basis eki , i =
1 . . . nk , to the simulation parameter space E h equipped with the local basis
ei , i = 1 . . . nsim .
Choosing T K = Th for the fine scale mesh will simplify somewhat the
interpolation matrix ψ k , as the nodes of a coarse mesh T k are now also
nodes of the fine and simulation mesh T K = Th over which one interpolates.
But this is possible only in certain situations, for example, when Th is a
regular two-dimensional N1 × N2 rectangular mesh with N1 = 2K N10 and
N2 = 2K N20 , in which case T 0 is a N10 × N20 mesh.

3.6.3 Scale-By-Scale Optimization

The admissible sets at each scale are defined by
nk
k
C = {xkopt ∈ IR nk
| xkopt,i eki ∈ C}.
i=1

Then, given an initial guess xinit ∈ C 0 , the optimization problem (2.5) is

solved successively at scales k = 0, 1, . . . , the result of the optimization at
scale k−1 being used (after interpolation over the mesh T k ) as initial value for
96 CHAPTER 3. CHOOSING A PARAMETERIZATION

the optimization at scale k, until the data are explained up to the uncertainty
level
⎧ −1
⎨ set: k = 0 and x̂opt = xinit ,
opt ∈ C , find x̂opt = argminxopt ∈C k J(xopt ),
starting from x̂k−1 k k
(3.22)
⎩
increment k until the fit is down to uncertainty level.

In this approach, the final number of parameters that will be estimated is

not known before the end of the scale-by-scale optimization.
It was observed in a certain number of cases [57, 58, 15, 64, 69] that
this approach, which searches for the features of the unknown parameter
successively from coarse to fine scale.
– Allows local optimization algorithms to converge to (or near to) the
global minimum of the objective function, instead of stopping in a possibly
remote local minimum or stationary point
– And can enhance the conditioning of the optimization problems
There is to date no rigorous mathematical explanation of this phe-
nomenon, though it should not be out of reach to formalize the following
sketch of analysis:
– We first introduce a class of “nicely nonlinear” inverse problems, for
which scale-by-scale optimization is expected to perform well
– Then we consider a prototype “nicely nonlinear” problem with low
dimensions (two parameters and three data!) whose attainable set can be
represented on a picture
– We explain on this picture the properties of multiscale approach with
respect to local minima or stationary points and conditioning
– And finally we discuss the influence of the choice of basis functions at
each scale on the conditioning of the optimization problem.

1. Nicely nonlinear problems: At a point xsim ∈ C and in a unit direction

u ∈ IRnsim , we define

• s = V  = sensitivity of the forward map ϕ, where V =

Du ϕ(xsim )
• κ = A/V 2 = (upper bound to the) curvature of the forward
2
map ϕ at xsim in the direction u, where A = Du,u ϕ(xsim )

By construction, V is the velocity and A the acceleration at t = 0 along

the curve t  ϕ(xsim + tu) drawn on the attainable set ϕ(C). These
3.6. MULTISCALE PARAMETERIZATION 97

quantities play an important role in the analysis of the nonlinear inverse

problem (2.5) given in Chap. 4, where they are discussed in detail.
Let then (eki , i = 1 . . . nk ) denote, for each scale k, a normalized and
local basis of E k . This basis can be defined, over the mesh T k , in the
same way the basis (3.19) or (3.20) of E h was defined over Th .
We can introduce now a rather loosely defined class of problems, where
the scale-by-scale decomposition of the parameter space structures the
sensitivity and curvature levels:

Definition 3.6.1 The NLS inverse problem (2.5) is “nicely nonlinear”

if the sensitivity decreases and the curvature increases in the directions
of the local basis function eki when k moves from coarse to fine scales.

In short, “nicely nonlinear” problems are more sensitive but less nonlin-
ear at coarse scales, and less sensitive but more nonlinear at fine scales.
Hence the scale-by-scale resolution (3.22), which solves the coarse scale
problem first, can be expected to perform well.
An example of “nicely nonlinear” problem (see [57, 58]) is the estima-
tion of a one-dimensional diffusion parameter, defined in Sect. 1.4 and
analyzed for wellposedness in Sect. 4.8. One can also conjecture that
the estimation of the two-dimensional diffusion parameter (defined in
Sect. 1.6 and analyzed in Sects. 4.9 and 5.4) is “nicely nonlinear,” as
suggested in [30] and in Remark 4.9.5.
At the opposite end of the spectrum, one finds the problem of estimat-
ing the sound velocity from wavefield measurements: large scale per-
turbations of the sound velocity produce phase-shifts ( translation in
time) in the observed wavefield, which correspond to a high curvature
because of the high frequency content of the geophysical signals – a very
difficult problem, which cannot benefit from the multiscale approach
described here [24].

2. A prototype nicely nonlinear problem: Consider a forward map ϕ :

IR2  IR3 and an admissible set C = [−1, +1] × [−1, +1] ⊂ IR2 ,
where the x1 -axis corresponds to coarse scale (the space E 0 above),
and the x2 -axis to the fine details (a supplementary subspace W 1 of
98 CHAPTER 3. CHOOSING A PARAMETERIZATION

E 0 in E 1 , see (3.25) below). Let us suppose that this problem is “nicely

nonlinear”:

• On the subset C 0 = C ∩ {x|x2 = 0} of C, the curvature of ϕ is

small enough for the attainable set ϕ(C 0 ) to be the piece of curve
shown in Fig. 3.2, top
• On the segments of C 1 = C parallel to the second axis {x|x1 = 0},
the curvature of ϕ increases significantly, but at the same time its
sensitivity also decreases significantly, so that the attainable set
ϕ(C 1 ) might look like the piece of surface in Fig. 3.2, bottom.

3. Stationary points and scale-by-scale resolution: Let the data z ∈ IR3 be

given as in Fig. 3.3, and xinit = (−1, 0) ∈ C 0 ⊂ C be the given initial
guess of x. We see the following in Figs. 3.3 and 3.4:

z3

ϕ(C 0)

0
z2
z1
z3

ϕ(C 1)

0
z2
z1

Figure 3.2: A representation of the attainable set for the prototype

example at scale 0 and 1
3.6. MULTISCALE PARAMETERIZATION 99

ϕ(xinit)
z3
ϕ(x̂0opt)
ϕ(C 0)
ϕ(x̂sim) ϕ(C 1 = C)
ϕ(x̂1opt)

0
z2
z1 z

Figure 3.3: Comparison of local vs. multiscale parameterization for

the convergence of gradient algorithms

x2
+1

−1 xinit x̂0opt +1
x1

−1
x̂sim x̂1opt

Figure 3.4: Optimization paths in C for the local (dashed line) and
the multiscale (full line) resolution. The thin dashed lines corre-
spond to one level line of the objective function
100 CHAPTER 3. CHOOSING A PARAMETERIZATION

• If the inversion is performed by solving the usual formulation (2.5)

by a local gradient method with respect to the local variables
xsim = (x1 , x2 ), the algorithm will stop, because of the oscillations
of the attainable set ϕ(C), at a local minimizer x̂sim far from the
global one.
• Let now the inversion be performed according to the scale-by-scale
formulation (3.22). At the scale k = 0, any local algorithm will
reach the global minimum x̂0opt , as z is at a distance of ϕ(C 1 )
smaller than its smallest radius of curvature. Then at scale k = 1,
the local algorithm will start at x̂0opt , and stop at one nearby local
minimum x̂1opt , which by construction is much closer to the global
minimum – it coincides with it on the chosen example.

We see on this example that the ability of the multiscale resolution to

overcome the problem of local minima or stationary points for a nicely
enough nonlinear problem relies solely on a correct resolution in IRnk
of the optimization problems (3.22), in the sense that, at each scale k,
the optimization algorithm converges to the nearest local minimum or
stationary point.

4. Choosing optimization parameters at scale k: To perform the scale-by-

scale optimization (3.22), it is necessary to decide whether the opti-
mization vector xkopt at scale k is made of the coefficients of Xh on a
local or a multiscale basis of E k :

(A) On a local basis: In most cases, using at each scale k = 0 . . . K

the simplest local basis (eki , i = 1 . . . nk ) will produce satisfactory
results – but maybe not with the best computational efficiency:
(a) For a discontinuous piecewise constant approximation: E k is
made of piecewise constant functions over the elements of T k ,
and the optimization parameters and the associated local ba-
sis functions at scale k are given by
⎧  |K |  12 X
⎪
⎪ i i
⎪
⎪ x k
= i = 1 · · · nk ,
⎨ opt,i |Ω| Xref
 |Ω|  12
(3.23)
⎪
⎪ iek
= X ref χKi i = 1 · · · nk ,
⎪
⎪ |K |
⎩ 1
i
|ei |L2 (Ω) = Xref |Ω| 2 i = 1 · · · nk ,
3.6. MULTISCALE PARAMETERIZATION 101

where now Ki is the ith cell of the coarse mesh T k (compare

with the definitions (3.1), (3.18), and (3.19) of the simula-
tion parameters and basis defined over the fine simulation
mesh) Th .
Because of the L2 (Ω) orthogonality and normalization of the
functions eki in (3.23), the Euclidean scalar product for xkopt in
IRnk coincides (up to the coefficient Xref 2
|Ω| ) with the L2 (Ω)
scalar product in E k .
(b) For a continuous piecewise linear approximation: Here E k
is made of continuous piecewise linear functions over coarse
triangular elements of T k . The optimization parameters and
the local basis functions at scale k are then given by
⎧  α  12 X
⎪
⎪ Mi Mi
⎪
⎪ xi = i = 1 · · · nk ,
⎨ |Ω| Xref
 |Ω|  2
1

⎪
⎪ e k
= X ref ωMk
i = 1 · · · nk ,
⎪
⎪
i
α i
⎩ k2 M i
|ei |k,L2(Ω) = Xref 2
|Ω| = 2|eki |2L2 (Ω) i = 1 · · · nk ,
(3.24)
where now ωM is the function of E with value 1 at the ith
k k

node Mi and value 0 at all other nodes of the coarse mesh

T k , and where | · |k,L2(Ω) denotes the approximate L2 (Ω)-norm
on E k evaluated using the quadrature formula (3.3) over the
triangulation T k instead of Th (compare with the definitions
(3.5), (3.18), and (3.20) of the simulation parameters and
basis).
As in (3.20), the basis (3.24) is not any more orthogonal in
L2 (Ω), but it is orthogonal for the approximate scalar product
·, ·k,L2(Ω) . Hence the interpretation of the Euclidean scalar
product in IRnk as the scalar product in L2 (Ω) is retained for
this approximate scalar product.

The implementation of the scale-by-scale optimization with a local basis

at each scale is relatively simple. Because of the proportionality of the
Euclidean scalar product in IRnk and the L2 (Ω) scalar product in E k ,
the optimization problem retains at each scale the conditioning of the
continuous problem.
102 CHAPTER 3. CHOOSING A PARAMETERIZATION

(B) On an adapted multiscale basis: When computation time is at stakes,

one can take advantage of the link between scale and sensitivity to
enhance the conditioning of the problem at scale k by using for xkopt an
j , j = 1 . . . p ,  = 0 . . . k) of E k , which we
adapted multiscale basis (˜
define now.
LetW k , k = 1 . . . K denote a supplementary space of E k−1 in E k .
A function of W k represents details that are seen at scale k but not
at scale k − 1 – hence the name detail space given to W k . The corre-
sponding multiscale decomposition of E K into a direct sum of subspaces
is
E K = E 0 ⊕ W 1 ⊕ · · · ⊕ W K. (3.25)
The space E 0 is sometimes called the background space, as it corre-
sponds to the absence of any details. Depending on the choice made
for W k , the decomposition (3.25) can be orthogonal (if W k ⊥E k−1 for
k = 1 . . . K), or oblique in the other cases. Define
⎧  def 
⎨ 0j , j = 1 · · · p0 = n0 = normalized local basis of E 0 ,
and, for k = 1 · · · K, (3.26)
⎩ k
j , j = 1 · · · pk = normalized local basis of W ,
k

where p0 is the dimension n0 of E 0 and pk = nk −nk−1 is the dimension

of the detail space W k . A normalized multiscale basis eki , i = 1 . . . nk ,
of E k is then obtained by retaining only the first nk vectors in the
list (3.26) above. However, most optimization algorithms are invariant
with respect to a change of orthonormal basis. There is hence nothing
to expect from choosing for E k an orthonormal multiscale basis instead
of the simple local basis! But one can take advantage of the properties
of “nicely nonlinear” problems and use an adapted multiscale basis,
which boosts the sensitivity of the variables according to the size of the
details they represent:

• For the determination of the background (k = 0), the natural

choice is to use the optimization parameters xopt and the local
basis 0j = e0j defined by (3.23) or (3.24). So the first step of a scale-
by-scale resolution is the same for an adapted multiscale basis or
a local basis.
3.6. MULTISCALE PARAMETERIZATION 103

• For the determination of the details at a scale k ∈ {1 . . . K}, let

skj denote the sensitivity in the directions of the multiscale basis
(3.26) at some nominal parameter xnom sim :

skj = Vjk  = Dkj ϕ(xnom

sim ),

and define a mean relative sensitivity of details at scale k:

1 1/2 % 1 1/2
skr = (ski )2 (s0i )2 . (3.27)
pk i=1...pk
n0 i=1...n0

When k increases, the support of the basis functions kj becomes

smaller, and so one can expect that, according to Definition 3.6.1
of a “nicely nonlinear” problem,

1 > s1r > · · · > sK

r .

So one can define an adapted multiscale basis for E K (and hence

for each E k !) by


kj = kj /skr , k = 1 . . . K, (3.28)

where the coefficient 1/skr tries to compensates for the loss of sensi-
tivity of the forward model in the directions of finer details. Using
this adapted multiscale basis tends to spherize the level lines of the
objective function at scale k, and hence speeds up the resolution.
If one does not want to estimate numerically skr by (3.27), one can
experiment with skr = (hk /h0 )α for various α > 0, where hk is the
size of the cells of T k .

Remark 3.6.2 When W k ⊥ E k−1 (the decomposition (3.25) is orthogonal),

basis functions kj of W k are orthogonal to the local basis functions eki of
E k−1 defined in (3.23) or (3.24). Hence allowing the parameter to have a
nonzero component on kj does not change its mean values at scale k − 1. In
many applications (as the estimation of a diffusion coefficient for example),
the model output is sensitive to the mean value of the parameter, so that
opening of new degrees of freedom in W k will not deteriorate, at first order,
the quality of the fit to the data obtained at scale k − 1. For such problems,
the optimization routine, when searching for x̂kopt in E k , will have to work
104 CHAPTER 3. CHOOSING A PARAMETERIZATION

out its way in practice in the much smaller space W k , as the coefficients on
E k−1 are already almost at their optimal value.
Oppositely, when the multiscale basis is associated with an oblique decom-
position, allowing the parameter to have a nonzero component on a detail
direction kj of W k does change its mean values at scale k − 1. Hence it
will be necessary, during the course of the optimization at scale k, to change
coordinates both in W k and in E k−1 to maintain the correct mean values
determined at scale k − 1.
Multiscale bases associated to orthogonal decompositions should hence be
preferred, as they tend to ensure a faster resolution of the optimization prob-
lem at each scale k, by concentrating on the smaller problem of determining
the details in W k . But – with the exception of the Haar basis, which is limited
to piecewise constant functions on rectangular meshes – they are difficult to
construct, as the basis vectors kj of W k cannot in general be determined ex-
plicitly over the finite domain Ω. There exists a large amount of literature on
wavelets [48], which are finite support orthogonal multiscale bases – but they
are defined over an infinite domain, and cannot be used directly on bounded
domains. Another approach is to use a numerical orthogonalization proce-
dure as the Gram–Schmitt procedure, but this is computationally expensive,
and the finite support property of the basis functions is lost.

Remark 3.6.3 Instead of performing the scale-by-scale optimization (3.22),

one can minimize simultaneously with respect to the whole set of multiscale
parameters, starting from a first initial guess xinit ∈ C 0 :

starting from xinit ∈ C 0 , find x̂K

opt = argminxopt ∈C K J(xopt ).

Because of the invariance of optimization algorithms with respect to or-

thonormal change of basis, no improvement is to be expected concerning the
stationary points and conditioning problems if an orthonormal multiscale ba-
sis is used.
Using the same adapted basis (3.28) as in the scale-by-scale resolution is
not a good idea: boosting the sensitivity of fine scales to enhance condition-
ing will at the same time increase the nonlinearity, and hence worsen the
stationary points problem instead of relieving it!
But it was observed in [57, 58] that going the other way, that is, using a
multiscale basis that damps the sensitivity of fine scales, can allow to over-
come the stationary points problem – at the price of a worse conditioning.
3.6. MULTISCALE PARAMETERIZATION 105

This is the case, for example, of the Haar basis (Remark 3.6.4 below), or of
the adapted basis:

kj = skr kj , k = 1 . . . K.
When such a basis is used, the optimization algorithm works first in the
subspace of the most sensitive coordinates, that is, those corresponding to
coarse scales, thus allowing to overcome the stationary points problems. It is
only when these coordinates have been adjusted that the algorithm feels the
need to adjust the coordinates for the finer scales, which have been made much
less sensitive. The graph of the objective function as a function of the iteration
number has then a stair shape, where going down one stair corresponds to
grasping to the adjustment of the next finer scale.

3.6.4 Examples of Multiscale Bases

We describe now the simplest multiscale bases for discontinuous piecewise
constant and continuous piecewise linear parameters.

Approximation by a discontinuous piecewise constant function.

We consider the case of parameters that are, at scale k, piecewise constant

functions over a two-dimensional rectangular mesh T k obtained by k divisions
of a rectangular background mesh T 0 . The dimension of the corresponding
parameter space E k is then nk = 2k × 2k n0 , and the dimension of a supple-
mentary space W k is pk = 4k n0 − 4k−1n0 = 3nk−1 . The mesh T k is hence
obtained by dividing each rectangular cell L of T k−1 into four smaller rect-
angular cells A, B, C, D. The kth refinement is called regular if all cells L of
T k are divided into four equal rectangles.
We associate to L the three functions kL,lr , kL,tb , and kL,sp defined in
Fig. 3.5 (the indexes lr, tb, sp stand, respectively, for left-right, top-bottom,
and saddle-point), with

|L| |L| |L| |L|

a=m , b=m , c=m , d=m ,
4|A| 4|B| 4|C| 4|D|

where m > 0 is a normalization coefficient.

One checks easily that:
  
k
L,lr = 0, kL,tb = 0, kL,sp = 0,
L L L
106 CHAPTER 3. CHOOSING A PARAMETERIZATION

A B
L
C D

a cell of T k−1 four cells of T k

+a −b +a +b

+c −d −c −d

klr ktb

−a +b

+c −d

ksp

Figure 3.5: Definition of the three basis functions of W k associated

to the cell L of T k−1

and
|L|
|kL,lr |L2 (L) = |kL,tb |L2 (L) = |kL,sp |L2 (L) = m 1 ,
¯2
where ¯ is equal to four times the harmonic mean of the areas of rectangles
A, B, C, and D (¯ = |L| if the kth refinement is regular!):
4 1 1 1 1 1 
= + + + .
¯ 4 |A| |B| |C| |D|
The functions kL,lr , kL,tb , kL,sp are linearly independent and orthogonal to
E k−1 . So the orthogonal supplementary space W k to E k−1 in E k is
W k = span{kL,lr , kL,tb , kL,sp , L ∈ T k−1 }.
3.6. MULTISCALE PARAMETERIZATION 107

The basis functions kL,lr , kL,tb , kL,sp of W k associated to the same cell L are
not orthogonal in general (except for a regular refinement, see Remark 3.6.4).
But basis functions associated to different cells (at same or different scales)
are orthogonal. This basis of W k can be normalized independently of the
scale by a proper choice of m over each cell L:
⎧ 1 1
⎨ ¯2 |Ω| 2
m = Xref implies (compare with (3.23)):
|L|
⎩ k 1
|L,lr |L2 (L) = |kL,tb |L2 (L) = |kL,sp |L2 (L) = Xref |Ω| 2 .

With this normalization, the multiscale basis functions (3.26) of E k have

1
the same norm Xref |Ω| 2 as the local basis functions (3.23). The local basis
is orthogonal, and the multiscale basis is orthogonal by blocks, one block
being made of the three basis functions associated to one cell L of a meshes
T k , k = 1 · · · K.

Remark 3.6.4 When the refinement is regular, one has a = b = c = d = m

and ¯ = |L|, and the functions kL,lr , kL,tb , kL,sp are orthogonal. The case
m = 1 corresponds then to the classical Haar basis – notice that the norm
of the basis functions of the Haar basis decreases as |L|1/2 when the scale is
refined.

Approximation by a continuous piecewise linear function.

We consider now the case where the parameter space E k at scale k is made
of continuous piecewise linear (on triangles) or bilinear (on rectangles) func-
tions, over a mesh T k obtained by k refinements of a background mesh T 0
made of triangles and rectangles. It is supposed that the initial mesh and its
subsequent refinements are done in such a way that all meshes T k are regular
meshes in the sense of finite elements (the intersection of two cells of such a
mesh is either void, or an edge, or a vertex). Let
& '
V k = M such that M is a node of T k but not of T k−1 .

By construction, V k contains pk = nk − nk−1 nodes M.

So a first choice is to choose for kM the local basis functions (3.24) of E k
associated to the nodes M of V k :

kM = ekM , M ∈ Vk.

108 CHAPTER 3. CHOOSING A PARAMETERIZATION

With this choice, the space

W k = span{kM , M ∈ V k }
is an oblique supplementary space of E k−1 in E k . Hence the correspond-
ing multiscale basis kM , k = 0 · · · K, M ∈ V k is not orthogonal, neither be-
tween scales, nor inside a given supplementary
√ space. But it is normalized, as
1
its elements have the same L -norm 2 Xref |Ω| 2 as the local basis functions
2 2

eki , i = 1 · · · nK of E k (see (3.24)).

The corresponding parameterization matrix ψ k (see 3.21) involves a lot of
interpolation, so that the change of variable can be computationally expen-
sive; its implementation and that of its transposed are relatively delicate and
error-prone, though following the procedure described in Sect. 2.4 with the
forward map ϕ = ψ k and the objective function G defined in (2.13) should
limit the chances of error in the determination of (ψ k )T .
This basis suffers moreover from the defaults of oblique multiscale bases
mentioned in Remark 3.6.2, and so its use – or that of its more intricate or-
thogonalized version – should be considered only in special cases, for example,
when scale-by-scale resolution with the local basis (3.24) at each scale fails.

3.6.5 Summary for Multiscale Parameterization

Multiscale parameterization has the desirable property that a scale-by-scale
resolution will converge to or near to the global minimum for “nicely non-
linear” problems. However, a larger and larger number of d.o.f. is added
each time the introduction of a new scale is required: as long as the fit to the
data resulting from the optimization at scale k is above the uncertainty level,
one has to grasp to the finer scale k + 1, and introduce in the optimization
space IRnopt all the d.o.f. of E k+1 , which are not in E k , that is, those of
W k+1 . It is then likely that part of them will have little or no influence the
output of the model, which shows that the multiscale approach can lead to
overparameterization.

3.7 Adaptive Parameterization: Refinement

Indicators
We consider in this section an adaptive approach for the estimation of dis-
tributed parameters, which tries to retain the good properties of multiscale
parameterization with respect to stationary points, but avoids the pitfall of
3.7. ADAPTIVE PARAMETERIZATION 109

overparameterization: one starts with a small number of degrees of freedom

(often one), and adds (or substitutes) degrees of freedom one at a time, by
using refinement/coarsening indicators to choose, among a large number of
tentative degrees of freedom, one which incurs, at first order, the strongest
decrease of the objective function; the algorithm stops when the data are
explained up to the noise level.
Adaptive parameterization is one way of regularizing the inverse problem:
it adds the information that one searches for the simpler (in the sense of
number of degrees of freedom) parameter function in the chosen class.
Moreover, if the unknown parameter is a function and the tentative de-
grees of freedom are chosen at each step at a scale finer than the current one,
one can expect that the algorithm will retain, when the problem is “nicely
nonlinear,” the good properties of scale-by-scale optimization with respect
to stationary points (Sect. 3.6.3).

3.7.1 Definition of Refinement Indicators

We define in this section refinement indicators independently of the nature
of the unknown parameter (vector or function) and of the objective function
(provided it is derivable). We make the assumption that the choice of a
“good” parameterization depends primarily on the forward map ϕ to be
inverted, and we neglect the constraints in the construction of our indicators.
We use the notations (see Sect. 3.3)
⎧
⎪
⎪ E = IRnsim : space of simulation parameter,
⎨
J : IRnsim  IR+ : a derivable objective function,
(3.29)
⎪
⎪ E ⊂E : the current optimization space,
⎩
x∈E : current optimization parameter.

The space E and its subspace E are equipped with the Euclidean scalar
product in IRnsim , which approximates, up to a multiplicative constant, the
L2 -scalar product when x is a function (see Sect. 3.1.1). The current solution
and value of the problem are (we ignore the constraints...)

x̂ = arg min J(x), J( = J(x̂). (3.30)

x∈E

When the minimum value J( is not satisfying (e.g., if it is above the noise level
in the case of least squares), the question arises of which degree of freedom to
110 CHAPTER 3. CHOOSING A PARAMETERIZATION

add to the current optimization space E such that it produces a significant

decrease of the objective function J.
So let T ⊂ E, T ∩ E = ∅ be a set of tentative new basis vectors 
available at the current parameterization step. We shall refer to T as the
set of tentative degrees of freedom. It can be very large, as its vectors need
not to be linearly independent. We shall suppose that its elements have been
normalized:
∀ ∈ T , T = 1 (3.31)
for some norm  · T . This norm is not necessarily the Euclidean norm on
E = IRnsim : for example, when the unknown parameter is a real valued
function and the vector x ∈ E is made of the values of the function over the
simulation mesh Th (Sect. 3.7.2 below), one often uses the ∞ norm:
T = ∞ = max |i |. (3.32)
i=1...nsim

To select one – or a few – degree(s) of freedom  in T , one associates to

any tentative  a refinement indicator λ in such a way that large |λ|’s are
associated to ’s, which have a strong potential for the decrease of J:
1. Nonlinear indicators: One defines

λNL = ΔJ = J( − J,

def
(3.33)
where the J is the new value of the problem, computed with the full
nonlinear model:
J = J(x̃ + ỹ), (x̃, ỹ) = arg min J(x + y). (3.34)
x ∈ E , y ∈ IR
This is the most precise indicator, as it ensures by definition that the
 associated to the largest λNL produces the strongest decrease of the
objective function!
But it is also the most computationally intensive one, as its computa-
tion requires the solution of the full nonlinear optimization problem.
Hence it is impossible to use λNL = ΔJ as an indicator to choose 
among a large number of degrees of freedom.
2. Gauss–Newton indicators: In the case of nonlinear least squares prob-
lems, where J(x) = 12 ϕ(x) − z2 , one can define

λGN = ΔJ GN = J( − JGN ,
def
(3.35)
3.7. ADAPTIVE PARAMETERIZATION 111

where the JGN is the new value of the problem, computed with the
Gauss–Newton approximation to the forward map ϕ:

JGN = J GN (δx̃GN ), δx̃GN = arg min J GN (δx̃), (3.36)

δx̃ ∈ span{E , }
where
1
J GN (δx) = ϕ (x̂)δx − Δz2 with Δz = z − ϕ(x̂). (3.37)
2
The evaluation of λGN requires now one resolution of a linear least
squares problem for each tentative degree of freedom . This approach
can be used (see, e.g., [42, 60]) when the size and computational cost
of the problem makes its resolution possible by a Gauss–Newton opti-
mization algorithm (one evaluation of the refinement indicator is then
computationally equivalent to one Gauss–Newton iteration). But the
number of tested degrees of freedom is still limited in this approach.

3. First order indicators: We return now to the general situation (3.29).

The optimal objective function τ ∈ IR  Jτ∗ ≥ 0 in the direction  ∈ T
is defined by

Jτ∗ = J(x∗τ + τ ), x∗τ = arg min J(x + τ ). (3.38)

x∈E

Comparison with (3.34) shows that

 ∗ 
x0 = x̂, x∗ỹ = x̃,
J0∗ = J,( 
Jỹ∗ = J,

and (3.33) becomes

dJτ∗
λNL = ΔJ = Jỹ∗ − J0∗ = |τ =0 ỹ + . . . (3.39)
dτ
In absence of information on ỹ, one can take a chance and chose 
according to the modulus of dJτ∗ /dτ in (3.39). This choice is comforted
by the remark that, because of (3.31), perturbing x̂ of a given amount
y  in the direction  will produce a large decrease, at first order, of
the optimal objective function for those ’s that exhibit dJτ∗ /dτ ’s with
large modulus. Hence one is led to the . . .
112 CHAPTER 3. CHOOSING A PARAMETERIZATION

Definition 3.7.1 The first order refinement indicator associated to the

tentative degree of freedom  ∈ T at the current optimal parameter x̂ is
dJτ∗
λ= |τ =0 . (3.40)
dτ
It is given by
λ = ∇J(x̂), E = D J(x̂), (3.41)
where ∇J(x̂) denotes the gradient of the objective function with respect
to simulation parameters x ∈ E.

Proof. Derivation of (3.38) with respect to τ at τ = 0 gives, when x∗τ is

a derivable function of τ ,
 ∗

∗ dxτ
λ = ∇J(x0 ), + ∇J(x∗0 ), E . (3.42)
dτ E

But dx∗τ /dτ ∈ E and ∇J(x̂), δxE = 0 ∀δx ∈ E (c.f. the definition
(3.30) of x̂), so that the first term vanishes in the right-hand side of
(3.42) and (3.41) is proved.
Hence the evaluation of λ for a tentative degree of freedom  ∈ T
requires only the scalar product of , in the simulation parameter space
IRnsim , with the known vector ∇J(x̂)! This makes it easy to test a very
large number of tentative degrees of freedom before making up one’s
mind for the choice of a new one.

Remark 3.7.2 One could argue that, because of the potentially large
dimension nsim of x, the gradient ∇J(x̂) can be computationally unaf-
fordable. However, when the implementation of the inversion is based,
according to the recommendations of Sect. 3.8.1 below, on the adjoint
state approach of Sect. 2.3, the gradient ∇J(x̂) with respect to simu-
lation parameters is available as a byproduct of the computation of x̂
(search for ∇xsim J in the lower right corner of Fig. 3.8).

Remark 3.7.3 The first order refinement indicators λ were originally

defined as Lagrange multipliers in the context of estimation of piecewise
3.7. ADAPTIVE PARAMETERIZATION 113

constant parameter functions [23, 10]. We have used a more direct –

and hopefully simpler – presentation above, but the two definitions co-
incide: (3.38) can be seen as a constrained optimization problem:

(xτ , yτ ) = arg min J(x + y ),

x ∈ E , y ∈ IR
y−τ =0

whose Lagrangian is

Lτ (x, y, λ) = J(x + y ) − λ(y − τ ).

The necessary Lagrange optimality conditions are then, for a given

τ ∈ IR,
⎧
⎪ ∂L
⎪
⎪ ((x, y, λ)δx = ∇J(x + y ), δxE = 0 ∀δx ∈ E ,
⎪
⎪ ∂x
⎪
⎪
⎪
⎨ ∂L
((x, y, λ) = ∇J(x + y ), E − λ = 0,
⎪
⎪ ∂y
⎪
⎪
⎪
⎪
⎪
⎪
⎩ ∂L ((x, y, λ) =y−τ = 0.
∂λ
For τ = 0, the second equation gives (3.41) and property (3.40), which
we use now as definition, is the well-known property that a Lagrange
multiplier is the derivative of the optimal objective function with respect
to the right-hand side of the corresponding constraint.

We return now to the case of nonlinear least-squares problems, where

1
J(x) = ϕ(x) − z2F ,
2
and investigate the relation between the first order indicator λ and the Gauss–
Newton indicator λGN . We use notations (3.35) through (3.37), and denote by

F the data or observation space IRq ,

(3.43)
Φ the q × nsim matrix associated to ϕ (x̂),

when E = IRnsim and F = IRq are equipped with the Euclidean scalar prod-
ucts. Because the principle of adaptive parameterization is to stop adding
114 CHAPTER 3. CHOOSING A PARAMETERIZATION

degrees of freedom before overparameterization is attained, it is not a re-

striction to suppose that:
Φ is injective over span{E , } (3.44)
By definition one has
λGN = J( − JGN ,
1 1
= Δz2F − Φ δx̃GN − Δz2F ,
2 2
1
= Φ δx̃GN , ΔzF − Φ δx̃GN 2F . (3.45)
2
The Euler equation for the Gauss–Newton optimization problem (3.36) is

find δxGN ∈ span{E , } ⊂ IRnsim such that
(3.46)
Φ δx̃GN − Δz, Φ δxF = 0 ∀δx ∈ span{E , }.
Choosing δx = δx̃GN in (3.46) and combining with (3.45) gives first
1
λGN = Φ δx̃GN 2F . (3.47)
2
Subtracting then from (3.46), the Euler equation for x̂
−Δz, Φ δxF = 0 ∀δx ∈ E
and the Definition (3.41) of λ
−Δz, Φ F = λ
gives
Φ δx̃GN , Φ δxF = 0 ∀δx ∈ E ,
Φ δx̃ , Φ F = −λ.
GN

Hence we see that

δx̃GN = −λ η(),
where η() ∈ span{E , } is the unique solution (which exists because of
hypothesis (3.44)) of
Φ η, Φ δxF = 0 ∀δx ∈ E , (3.48)
Φ η, Φ F = 1. (3.49)
Combining this result with (3.47) gives
3.7. ADAPTIVE PARAMETERIZATION 115

Proposition 3.7.4 Let notation (3.43) and hypothesis (3.44) hold. Then the
Gauss–Newton and first order indicators are related by

λ2
λGN = Φ η()2F , (3.50)
2
where η() is defined by (3.48) and (3.49).

This confirms that the first order indicator carries one part of the information
on the variation of λGN with the tentative degree of freedom .
It is possible to explicit the calculations needed for the evaluation of the
coefficient Φ η()2F . We return for this to the optimization variables. The
dimension of the current optimization space E is nopt , and let

e1 . . . enopt ∈ E ⊂ E = IRnsim

be the current optimization basis, and Ψ() be the nsim × (nopt + 1) parame-
terization matrix (3.16) for the tentative optimization space associated to 

Ψ() = [e1 . . . enopt ]. (3.51)

Then the vectors η ∈ span{E , }, δx ∈ E and  ∈ T , which appear in (3.48)

and (3.49), are of the form

η = Ψ() ηopt for some ηopt ∈ IRnopt +1 ,

δx = Ψ() (δxopt , 0) for some δxopt ∈ IRnopt ,
 = Ψ() ( !0 ."#
. . 0$ , 1).
nopt times

Hence (3.48) and (3.49) become

ΦΨ() ηopt , ΦΨ() (δxopt 0)F = 0 ∀δxopt ∈ IRnopt ,

ΦΨ() ηopt , ΦΨ() ( !0 ."#
. . 0$ 1)F = 1.
nopt times

This rewrites

ΦΨ() ηopt , ΦΨ() δxopt F =

. . 0$ 1), δxopt IRnopt +1
( !0 ."# ∀δxopt ∈ IRnopt +1 ,
nopt times
116 CHAPTER 3. CHOOSING A PARAMETERIZATION

that is,
Ψ()T ΦT Φ Ψ() ηopt = ( !0 ."#
. . 0$ 1).
nopt times

Solving for ηopt and substituting in (3.50) gives an alternative expression of

λGN :
λ2  2
λGN = ( !0 ."# . . 0$ 1)(Ψ()T ΦT Φ Ψ())−1 . (3.52)
2
nopt times

This formula is derived in [11] starting from the Lagrange multipliers defi-
nition of first order indicators. It will be useful in Sect. 3.7.3 when applying
refinement indicators to segmentation of black and white images.

3.7.2 Multiscale Refinement Indicators

We consider in this section the case where the unknown parameter is a piece-
wise constant real valued function a : ξ ∈ Ω  a(ξ) ∈ IR over some domain
Ω ⊂ IR2 . First order indicators have been first introduced [23, 10], and used
[69, 35] in this context, and some convergence results have been obtained in
[9]. But multiscale refinement indicators are not limited to this case:
• They can be adapted to the estimation of smooth functions (see [33]
for the determination of a continuously differentiable function of two
variables)

• The method generalizes naturally to domains of higher dimension

• And results for the case of vector valued parameters can be found in
[44, 11]
Let Th be the simulation mesh covering Ω, made of nsim cells K. We
approximate the function a : Ω  IR by a function ah which takes a constant
value aK over each cell K of Th , and choose as simulation space E = IRnsim ,
and as simulation parameter :

asim = (asim,K , K ∈ Th ) ∈ IRnsim with asim,K = aK . (3.53)

For simplicity, we have chosen not to adimensionalize a, as all parameters aK

correspond to the same physical quantity.
Because of the large size of the discretization mesh Th , the data are usually
not sufficient to estimate the value of the unknown parameter in each cell K.
3.7. ADAPTIVE PARAMETERIZATION 117

We consider here the case where the additional information that is added to
regularize the problem is that a is constant over a number nopt of subdomains
(much) smaller than the number nsim of elements K (this is “regularization
by parameterization” as described in Sect. 1.3.4, see also [38]). We call such
a partition T of Th into nopt subsets Tj zonation, each one made of cells K of
Th (with the notations of Sect. 3.6.2, Th is a sub-mesh of T ), and we associate
to a function a : Ω  IR, which takes a constant value aj on each zone Tj the
vector aopt ∈ IRnopt of its values on each zone calibrated according to (3.1) –
once again we do not adimensionalize for simplicity:
 |T |  12
j
aopt = (aopt,j , j = 1 . . . nopt ) ∈ IRnopt with aopt,j = aj . (3.54)
|Ω|

The parameterization map Ψ : aopt  asim ∈ IRnnsim defined in (1.16) is hence

the nsim × nopt matrix:
Ψ = [eT1 . . . eTnopt ], (3.55)
where the vectors eTj ∈ IRnsim , j = 1 . . . nopt , are defined by (compare with
(3.19))
⎧
⎨ 0 1 if K ∈ / Tj ,
T
ej,K = |Ω| 2 (3.56)
⎩ 1 aopt,j if K ∈ Tj ,
|Tj | 2
and the current optimization space (see (3.29)) is E = span{eT1 . . . eTnopt }.
With this choice of optimization parameter, the Euclidean scalar product
on E ⊂ IRnopt is proportional to the L2 -scalar product of functions over
Ω, which will preserve the conditioning of the original problem, and the
Euclidean norm aopt  is a mean value of the function a over Ω (Sect. 3.1.1).
The “easy” – or at least more classic – part of the problem is the mini-
mization of the NLS objective function aopt  J(Ψ aopt ) over the admissible
parameter set for a given zonation T ; but the difficult part is that the zona-
tion T itself is not known a priori! Hence solving the regularized problem
amounts to find an “optimal” zonation T( of Ω such that:
(
1. The data are explained up to the noise level (i.e., the minimum of J ◦ Ψ
is small enough)

2. The number of zones is small enough to avoid overparameterization

( is injective)
(i.e., the Jacobian of ϕ ◦ Ψ
118 CHAPTER 3. CHOOSING A PARAMETERIZATION

The determination of the zonation is a difficult geometric problem. Var-

ious approaches to its solution have been proposed: displacement of the
boundaries between zones using geometric gradient methods, level set meth-
ods, etc. We develop below the multiscale refinement indicator approach,
which is particularly well suited for “nicely nonlinear problems” of Defi-
nition 3.6.1, (e.g., the estimation of the diffusion coefficient or the source
term in Sects. 1.4–1.6 of Chap. 1), where a coarse-to-fine scale-by-scale res-
olution has been shown in Sect. 3.6.3 to relieve the stationary point prob-
lem. The platform Ref-indic for the use of these indicators is available
at http://refinement.inria.fr/ref-indic/.
The multiscale refinement indicators proceed as follows: one starts with
T made of one or two zones, and increases the number of zones one at a
0

time, by splitting one zone of the current zonation T k into two subzones,
thus producing a new zonation T k+1 . This ensures at each step that the new
degree of freedom is (locally) at a finer scale than the current ones, hence the
name “multiscale” given to these indicators. The zone to be split and the way
it is split are chosen according to refinement indicators. By construction of
the refinement indicators, this ensures that each new optimization problem is
likely to produce the best decrease of the objective function among all tested
degrees of freedom.
The scale-by-scale optimization is stopped, as described in Sect. 3.6.2,
at the first k for which the data are explained up to the noise level. But
the parsimony with which the degrees of freedom are introduced ensures
that the Jacobian of ϕ ◦ Ψ has a full rank at all successive minimizers âkopt ,
and hence overparameterization is avoided.
Let T = T k denote the current zonation, and Ψ and E denote the cor-
responding current parameterization matrix and optimization space. We de-
scribe now the construction of T k+1 . Cutting one domain Tj of T into two
subdomains Tj,+ and Tj,− amounts to add to the basis eT1 . . . eTnopt of E defined
in (3.56) the vector  = (K , K ∈ Th ) defined by
⎧
⎨ 0 if K ∈/ Tj ,
 = (K , K ∈ Th ) ∈ IR nsim
with K = +1 if K ∈ Tj,+ , (3.57)
⎩
−1 if K ∈ Tj,− .

The (first order) refinement indicator (3.41) associated to this cut is

   
λ = ∇xsim J, IRnsim = ∇xsim J K − ∇xsim J K , (3.58)
K∈Tj=+ K∈Tj=−
3.7. ADAPTIVE PARAMETERIZATION 119

where ∇xsim J is evaluated at the point ΨâTopt of IRnsim . So we see that, once
∇xsim J has been made available as by-product of the minimization over the
current zonation, the first order indicator associated to any cut of any domain
Tj is obtained by a simple summation of the components of ∇xsim J over the
new subdomains. This makes it possible to test a very large set T of tentative
refinements of the current zonation T = T k . Once the new zonation T k+1
has been chosen, the optimization is performed with respect to the local (at
the zonation level) variables aopt,j , j = 1 . . . nk+1
opt as defined in (3.54).
For example, if the simulation mesh Th is made of rectangles (see Fig. 3.6),
one can choose, for the set T of tentative degrees of freedom, the collection of
the cuts dividing each current zone Zj by a vertical or horizontal line located
on one edge of the mesh, and by a “checkerboard cut” centered at each node
of the zone (see Remark 3.7.9 below). Using these cuts will lead to zones

+ ++ - + + ++
+++- ++ ++
+ ++ - - - - - + + + + + ++ +
+++ - - - - - + + +++ + ++
++ - - - - - - - - - - - -
+- - - - - - - - - - -
- - - - - - - - - -
Tj Tj

++++
++++
++ +++ - - -
++ ++ + - - -
- - - - + ++
- - - +++
- - ++ +
Tj
Figure 3.6: Examples of vertical, horizontal, and checkerboard cuts
for one zone Tj of a mesh Th made of rectangles. The vector  takes
value +1 in cells with a plus sign, and −1 in cells with a minus
120 CHAPTER 3. CHOOSING A PARAMETERIZATION

that have “simple” shapes, which is often an implicit constraint. We refer to

[10] for more details and numerical results. But the above list of cuts is not
exhaustive, and one can test any cut suggested to the user by its experience
or intuition.
If the complexity of the zones is not a problem, one can use a set T
containing only one cut in each zone Tj , namely the one that produces the
largest refinement indicator λj,max :

λj,max = |(∇xsim J)K |. (3.59)

K∈Tj

It corresponds to the vector  given by


0 if K ∈
/ Tj ,
K = (3.60)
sign(∇xsim J)K if K ∈ Tj .

This largest refinement indicator approach has been used in [23] for an oil field
modeling problem, and is applied in Sect. 3.7.3 below to the segmentation of
black and white images.
Notice that the vectors  associated to these cuts by (3.57) and (3.60)
satisfy the normalization (3.32) for T .

Remark 3.7.5 The refinement indicators have been defined in the context
of unconstrained optimization. In this case, the optimum âTopt at the current
zonation T satisfies

∂J/∂xopt,j = ∇xsim J, eTj IRnsim = 0 ∀j = 1 . . . nopt , (3.61)

so that formula (3.58) is equivalent to

  
λ = 2 K∈Tj=+ ∇xsim J K
   (3.62)
= −2 K∈Tj=− ∇xsim J K .

In practical applications, however, constraints are taken into account for the
determination of âTopt , so that (3.61) does not necessarily hold for all zones
(unless all constraints are taken into account via regularization, see Sect. 3.8.1
below). It is hence recommended to compute the refinement indicators by
formula (3.58) rather than by (3.62).
3.7. ADAPTIVE PARAMETERIZATION 121

3.7.3 Application to Image Segmentation

Let Ω be a rectangle covered by a large uniform mesh Th made of nsim rect-
angular (usually square) elements K called pixels, and z = (zK , K ∈ Th ) be
a black and white image on the domain Ω, where zK ∈ [0, 256], for example,
represents the gray level of pixel K. The segmentation of this image consists
in finding a zonation T of Th , that is, a partition of Th into a small number
nopt of subsets Tj , and an image â with a constant gray level in each zone
Tj , j = 1 . . . nopt , such that the number nopt of zones of T is small and â is
close to the given image z. This fits clearly in the framework of multiscale
refinement indicators of Sect. 3.7.2, with the particularly simple forward map:

ϕ : a ∈ C = [0, 256]nopt ⊂ E = IRnsim  a ∈ F = IRnsim . (3.63)

Because of the linearity of the forward map, the nonlinear refinement indica-
tor λNL coincides here with the Gauss–Newton indicator λGN . The question
that arises then naturally is: for such a simple inverse problem, does the
largest first-order indicator λ also hint at the largest decrease of the objec-
tive function? To answer this question, we explicit the relation (3.52) between
λNL = λGN and λ in the case of image segmentation. The matrix Φ is here
the nsim × nsim identity matrix, and so we are left to determine Ψ()T Ψ().
The resolution of the minimization problem (2.5) for a given zonation is
straightforward when ϕ = Id , as it amounts to affect to each zone the mean
gray level of the zone in the image! So there is no need here to calibrate
the optimization parameters to maintain the conditioning of the optimiza-
tion problem, and we replace the Definition 3.54 of optimization parameters
simply by

aopt = (aopt,j , j = 1 . . . nopt ) ∈ IRnopt with aopt,j = aj . (3.64)

Formula (3.56) for the definition of the basis eTj ∈ IRnsim , j = 1 . . . nopt , of E
becomes now 
T 0 if K ∈ / Tj ,
ej,K = (3.65)
1 if K ∈ Tj .
The parameterization matrix Ψ() associated to the tentative splitting of the
j-zone Tj into two subzones Tj,+ and Tj,− is given by (3.51), where eTj and
 are given by (3.65) and (3.57). Hence, the (nopt + 1) × (nopt + 1) matrix
122 CHAPTER 3. CHOOSING A PARAMETERIZATION

Ψ()T Ψ() is (only nonzero elements are represented):

⎡ ⎤
p1
⎢ .. ⎥
⎢ . ⎥
⎢ − ⎥
⎢ pj j − pj
p+ ⎥
Ψ() Ψ() = ⎢
T
.. ⎥, (3.66)
⎢ . ⎥
⎢ pnopt ⎥
⎣ ⎦
−
pj − pj
+
pj
where p1 . . . pnopt are the numbers of pixels in each zone of the current zona-
tion T , and pj,+ (respectively, pj,− ) is the number of pixels in the subzone
Tj,+ (respectively, Tj,− ) of the tentative cut. It is now a simple calculation to
deduce from (3.52) and (3.66) that
λ2j pj
λNL
j = λGN
j = . (3.67)
8 pj,+ pj,−
This shows that the largest first order indicator λj does not always corre-
spond to the largest decrease of the objective function among all tentative
refinements T , even when the forward map is the identity. But for large val-
ues of α, the function τ  α/(τ (α − τ )) is almost flat near α/2, so there tend
to be a good correlation between λ2 and the actual decrease of the objective
function, at least during the first iterations of the algorithm.
It is hence natural, when the complexity of the zonation is not a problem,
which is usually the case in image segmentation, to search at each iteration
for the new degree of freedom in a tentative set T containing only, in each
zone Tj of the current parameterization, the degree of freedom  given by
(3.59), which corresponds to the largest refinement indicator λj,max defined
by (3.60).
This approach is developed in [11] for the segmentation of B and W im-
ages, together with a generalization to segmentation of color images. The
platform Ref-image for the application of refinement indicators to image seg-
mentation is available at http://refinement.inria.fr/ref-image/.

3.7.4 Coarsening Indicators

We return to the general context of the estimation of piecewise constant
parameters. Suppose that the refinement indicators of Sect. 3.7.2 have sug-
gested to split one current zone Tj into two subzones Tj,+ and Tj,− . Before
3.7. ADAPTIVE PARAMETERIZATION 123

deciding to add this new degree of freedom, one can ask whether aggregating
one subzone, say Tj,+ , to one adjacent neighboring zone, say T , would not
be already beneficial to the objective function, thus leaving unchanged the
number of degrees of freedom.
Coarsening indicators that evaluate the interest of such an aggregation
have been defined in [10] via Lagrange multipliers. We give here a more direct
definition:
Definition 3.7.6 The coarsening indicator for the aggregation of the sub-
zone Tj,+ with one adjacent zone T is (see Fig. 3.7)

ΔJj∗+ , = −μ+ (aopt, − aopt,j ),

where μ+ is the first order indicator associated by Definition 3.7.1 to the

direction

0 if K ∈/ Tj,+ ,
η = (ηK , K ∈ Th ), where ηK =
+1 if K ∈ Tj,+ .

T

Tj,+
Tj,−

Figure 3.7: Coarsening indicators: the current zonation T (thick

lines, 5 zones), the tentative division of zone Tj proposed by the
refinement indicators (thin line), and one zone T (adjacent to Tj,+ )
to be tested for aggregation with Tj,+
124 CHAPTER 3. CHOOSING A PARAMETERIZATION

It is given by
 
μ+ = ∇xsim J K (3.68)
K∈Tj,+

According to Definition 3.7.1, the coarsening indicator ΔJj∗+ , gives the first
order decrease of the objective function incurred by changing, on Tj,+ , the
parameter from its current value aopt,j to the value aopt, in the adjacent
zone T .
ΔJj∗+ , can be positive or negative, but if, for some adjacent zone T , one
finds that ΔJj∗+ , /J ∗ is larger than some a-priori given percentage, one can
consider aggregating the zones Tj,+ and T , (rather than splitting the zone Tj
into two). In this case, the new zonation T k+1 will have the same number of
zones than T k , but will nevertheless produce a better fit to the data.

Remark 3.7.7 For the aggregation of Tj,− one has to compute (see (3.68))
 
μ− = ∇xsim J K
.
K∈Tj,−

Hence μ− and μ+ are linked to refinement indicators by

λ = μ+ − μ− .

Also, x̂Topt minimizes J over the zonation T , so that

  ∂J
μ+ + μ− = ∇xsim J K
= (âTopt )  0,
K∈Tj
∂xopt,j

which shows that μ+ and μ− are likely to be of opposite signs.

3.7.5 A Refinement/Coarsening Indicators Algorithm

We describe here an example of an adaptive optimization algorithm that uses
the refinement and coarsening indicators of Sects. 3.7.2 and 3.7.4 to estimate
a function a defined over a domain Ω covered by a simulation mesh Th .
This algorithm uses a conservative strategy for the choice of the refine-
ment: because the refinement indicators are only first order ones, the indi-
cator(s) with maximum absolute value do(es) not always produce the larger
decrease of the objective function. Hence the algorithm takes a better chance
3.7. ADAPTIVE PARAMETERIZATION 125

by computing the actual decrease of the objective function for a set of indi-
cators with large absolute values before taking a decision. The situation is
∗
different for the coarsening indicators ΔJj±, , which give directly an estima-
∗
tion of ΔJ for a given aggregation: the algorithm will decide to aggregate
or not at the sole view of the coarsening indicators.
Of course, many variants of this algorithm are possible: for example, one
could postpone the decision to aggregate after the actual decrease of J ∗ has
been computed for a family of potentially interesting aggregations, which
would require the resolution of a few more minimizations problems at each
step. We describe now the algorithm.
As a preliminary step, one has to choose the family T of tentative refine-
ments of the current zonation T , which will be explored by the algorithm
to define the next zonation (e.g., the set of cuts of Fig. 3.6 for each zone Tj
of T ).
Once this is done, the algorithm can go as follows:
1. Choose an initial zonation T .
2. Do until data are satisfactorily fitted:
3. Estimate a on the current zonation T by minimizing J with respect
to aTopt .
4. Compute the refinement indicators λ for all tentative refinements T :
For every zone Tj of T and for every cut of Tj do
compute the corresponding refinement indicator λ
Enddo
5. Compute |λ|max the largest absolute value of all computed refinement
indicators. Select a subset T 80% of cuts corresponding to refinement
indicators, which are larger than 80% of |λ|max (this percentage can
be adjusted)
6. Minimize J successively over the zonations associated to the cuts of
∗
T 80% , and let (ΔJcuts )max denote the best decrease obtained.
7. Compute the coarsening indicators for all selected cuts of T 80% :
For every cut of T 80% ,
every subzone Tj,± ,
every adjacent zone T do
∗
compute the coarsening indicator ΔJj±,
Enddo
126 CHAPTER 3. CHOOSING A PARAMETERIZATION

∗ ∗
8. If (ΔJj±, )max is larger than 50% of (ΔJcuts )max (this percentage can
be adjusted)
then aggregate Tj,± and T
else refine according to the best cut found at step 6.
Endif
9. Update the current zonation according to the retained cut or aggre-
gation.
Enddo
Remark 3.7.8 The above procedure is by nature interactive: after comple-
tion of step 7, one can ask the algorithm to display the selected cuts and
aggregations, and the user can mark those that seem of particular interest to
him, or even figure out a new refinement pattern that incorporates his or her
insight of the problem. The minimizations of step 8 can then be limited to the
marked items.
Remark 3.7.9 If one wants the parameter to be constant on zones Tj made
of a single connected component (i.e., made of “one piece”), it is necessary
to add a step 5 as
If some cuts of T 80% generate subdomains with more than one con-
nected component then
compute the refinement indicators corresponding to the subcuts
associated to each connected component (this will be the case
each time a checker board cut is selected!).
Update the set T 80% of selected cuts according to the 80% rule.
Endif

3.8 Implementation of the Inversion

As we have seen in Sect. 3.3, it is important for the inversion code to be flex-
ible with the choice of optimization parameters. The organization proposed
in this section takes this into account.

3.8.1 Constraints and Optimization Parameters

Constrained optimization is a delicate and difficult matter as soon as con-
straints other than box constraints are imposed. It is hence often unpractical
3.8. IMPLEMENTATION OF THE INVERSION 127

to require that the optimization routine takes in charge the constraints on

xopt , which would ensure that xsim ∈ C. An unconstrained optimization rou-
tine is hence often used. We consider below the case where the optimization
routine requires only as input the parameter x, the values J(x) of the ob-
jective function, and ∇J(x) of its gradient. But the results can be easily
adapted to the case where the Jacobian ϕ (x) is also required (e.g., in Confi-
dence Regions methods). We refer to [13, 68] for a presentation and analysis
of optimization algorithms.
The input xsim to the direct+adjoint routine is then computed from the
output xopt of the optimizer in two steps (left part of fig. 3.8):
• Return to simulation parameters: A provisory simulation vector x̃sim is
first computed from xopt using the chosen parameterization formula ψ:
x̃sim = ψ(xopt ). (3.69)

• Take care of the “explicit” constraints: Because of the unconstrained

optimizer used, the vector x̃sim does not necessarily belong to the ad-
missible parameter set C. Let us denote by c ,  ∈ L the constraints
defining C:
C = {x ∈ IRnsim | c (x) ≤ 0 ∀ ∈ L}.
Some of these constraints – say  ∈ Lexplicit – correspond to an explicit
projection operator P : they are simply taken into account by projecting
x̃sim on Cexplicit = {x ∈ IRn | c (x) ≤ 0 ∀ ∈ Lexplicit } before using it as
input to the direct+adjoint routine:
xsim = P (x̃sim ). (3.70)
An important practical case is that of the box constraints:
xj,min ≤ xj ≤ xj,max , j = 1 . . . N,
which correspond to the very simple projection operator
⎧
  ⎨ xj,min if xj ≤ xj,min,
P (x) j = xj if xj,min ≤ xj ≤ xj,max j = 1 . . . N,
⎩
xj,max if xj ≥ xj,max .
The direct+adjoint routine will work properly only when supplied with
simulation parameters that satisfy some sign and/or range conditions,
and it is a safe decision to impose at least these constraints through
the projection operator P .
128 CHAPTER 3. CHOOSING A PARAMETERIZATION

The remaining constraints – say  ∈ Limplicit – correspond to an implicit

projection operator, which cannot be evaluated by simple calculations. For
example, the projection operator associated to linear constraints (other than
box constraints!) would require the resolution of a quadratic programming
problem for each evaluation of P (x), which is something one usually cannot
afford. These constraints can be taken care of in a soft way by adding to the
objective function a penalization term Jηpen (see center part of Fig. 3.8):
1
Jηpen (x) = (c (x)+ )2 , η > 0, (3.71)
η2 ∈Limplicit

xinit
opt OPTIMIZER x̂opt

xopt J + Jηpen + Jreg ∇xopt (J + Jηpen + Jreg )

ψ ψ (xopt)t

x̃sim ∇x̃sim (J + Jηpen + Jreg )

P P (xsim)t

xsim ∇xsim (J + Jηpen + Jreg )

+Jηpen +∇xsim Jηpen

+Jreg +∇xsim Jreg

J ∇xsim J

DIRECT + ADJOINT MODEL

Figure 3.8: Organization of a nonlinear inversion code. Bot-

tom: direct and adjoint simulation code, top: optimization code,
ψ: parameterization routine, P : projection on box constraints,
ψ  (xopt )t : adjoint parameterization routine, P  (xsim )t : adjoint pro-
jection routine
3.8. IMPLEMENTATION OF THE INVERSION 129

where 
+ c (x) if c (x) ≥ 0,
c (x) =
0 if c (x) ≤ 0.
Using a small penalization parameter η will produce a small violation of the
constraints, and also a poorly conditioned optimization problem, in practice
experimentation is needed to choose η.
Introduction of additional a-priori information via LMT-regularization is
easily taken into account by addition of a regularizing functional Jreg (see
center part of Fig. 3.8). For example, when the information is that the pa-
rameter is “not too far” from some a-priori value x0 , Jreg has the form (see
Sect. 5.1 in Chap. 5, and (1.25) in Chap. 1)
2
Jreg = x − x0 2E .
2
Remark 3.8.1 It can happen that the regularizing functional depends on the
parameter x not only explicitly, but also through the state vector yx , as, for
example, in the adapted regularization considered in Sect. 5.4.3, or in the
state-space regularization approach of Chap. 5. For a functional Jreg (x, yx )
like this, the gradient has to be computed together with that of J by the ad-
joint state technique: this leads to add ∇y Jreg (x, yx ) in the right-hand side of
the adjoint equation, and a term ∇x Jreg (x, yx ) in the formula that give the
gradient with respect to x.

3.8.2 Gradient with Respect to Optimization

Parameters
As we can see in the right part of Fig. 3.8, the gradient with respect to opti-
mization parameters is obtained from the gradient with respect to simulation
parameters in two steps:
• Sum up the gradients of J, Jηpen , and Jreg with respect to xsim
• Apply, in reverse order, the transpose of the derivatives of the trans-
formations that had been applied to the parameter in the left part of
Fig. 3.8.
We detail in this section the implementation of the subroutine that computes
ψ  (xopt )T gsim for any given vector gsim ∈ IRnsim , where ψ : xopt  xsim is the
parameter change in the left part of Fig. 3.8.
130 CHAPTER 3. CHOOSING A PARAMETERIZATION

The first thing is to choose a convenient “direct” sequence of calculations

for the computation of xsim once xopt is given (block ψ in the left part of
the diagram). For example, if the same quantity is used at different points of
the calculations, it can be computationally efficient to make it an intermediate
variable, which will be evaluated only once, and used later as many times as
needed.
The second thing is to determine a convenient “adjoint” sequence of calcu-
lations to compute the result gopt ∈ IRnopt of the action of the block ψ  (xopt )T
(in the right part of the diagram) on any vector gsim ∈ IRnsim :

gopt = ψ  (xopt )T gsim , (3.72)

(the adjoint sequence of calculations will be applied to gsim = ∇x̃sim (J +

Jηpen + Jreg ) to produce gopt = ∇xopt (J + Jηpen + Jreg )). We shall distinguish
two cases:

• The direct sequence of calculations required for the evaluation of xsim

from xopt does not involve intermediate variables. In this case, the ad-
joint sequence of calculations can be determined simply by application
of the definition of transposition (as in Example 7 below)

• The direct sequence of calculations required for the evaluation of xsim

from xopt does involve intermediate variables. It is then necessary to use
the adjoint state technique of Chap. 2 to determine the adjoint sequence
of calculations (Example 8 below)

Example 7: Continuous Piecewise Linear Parameterization on a

Coarse Mesh
We consider here the case where the unknown parameter x is a function
a : ξ ∈ Ω ⊂ IR2  IR, and where the domain Ω is covered by two triangular
meshes:

• A fine simulation mesh Th made of triangles K. We shall denote by ∂Th

the set of nodes M of Th

• A coarse optimization mesh Topt made of triangles L. Similarly, we shall

denote by ∂Topt the set of nodes P of Topt
3.8. IMPLEMENTATION OF THE INVERSION 131

According to Sect. 3.1.1, the simulation and optimization parameters are de-
fined, for continuous piecewise linear approximations, by
α  12 a
sim,M M
asim,M = , ∀M ∈ ∂Th ,
|Ω| aref
α  12 a
opt,P P
aopt,P = , ∀P ∈ ∂Topt ,
|Ω| aref
where αsim,M and αopt,M are defined, with obvious adaptation, by (2.61),
and aM and aP are the value of the parameter a at nodes M and P . The
simplest parameterization map ψ : aopt  asim is obtained by computing aM
by interpolation between the values aP on the mesh Topt :

aM = ζM,P aP , ∀M ∈ ∂Th ,
P ∈∂Topt (M )

where

∂Topt (M) = {vertices of the triangle L of Topt containing M},

and where (ζM,P , P ∈ ∂Topt (M)) are the barycentric coordinates of M in

the triangle L of Topt containing M. They are given, for every M ∈ ∂Th , by

M= ζM,P , P 1= ζM,P .
P ∈∂Topt (M ) P ∈∂Topt (M )

The parameterization map ψ is then

asim,M = ψM,P aopt,P , ∀M ∈ ∂Th , (3.73)

P ∈∂Topt (M )

where the nonzero coefficients of ψ are given by

α  12
sim,M
ψM,P = ζM,P ∀M ∈ Th and ∀P ∈ ∂Topt (M).
αopt,P

The direct sequence of calculation for this parameterization is given by

(3.73): it does not involve any intermediate variable (the coefficients ψM,P
are computed once for all).
132 CHAPTER 3. CHOOSING A PARAMETERIZATION

So we shall simply use the definition of transposition to determine the

adjoint sequence of calculation: given gsim ∈ IRnsim , the vector gopt = ψ T gsim
satisfies by definition, for any δaopt ∈ IRnopt :
gopt , δaopt IRnopt = gsim , ψ δaopt IRnsim
 
= gsim,M ψM,P δaopt,P
M ∈∂Th P ∈∂Topt (M )
 
= gsim,M ψM,P δaopt,P ,
P ∈∂Topt M ∈∂Th (P )
! "# $
where gopt,P

∂Th (P ) = {nodes of Th in the elements L of Topt surrounding P }.

Hence the sought after adjoint sequence of calculation is
gopt,P = gsim,M ψM,P .
M ∈∂Th (P )

Example 8: Polar Coordinates

We compute (3.72) by the step-by-step approach of Sect. 2.4.

Step 0: Forward Map and Objective Function: The forward map is

here
ψ : θ = (θ1 · · · θn−1 )  x = (x1 · · · xn ),
and it is conveniently determined by the sequence of calculation:
x01 = 1, (3.74)
uk = cos θk ∀k = 1 · · · n − 1, (3.75)
xkj = xk−1
j uk ∀k = 1 · · · n − 1 ∀j = 1 · · · k, (3.76)
xkk+1 = sin θk ∀k = 1 · · · n − 1, (3.77)
followed by
xj = xn−1
j ∀j = 1 · · · n.

To compute gθ = ψ (θ) gx for a given gx ∈ IRn , one defines the objective
T

function θ  G(θ, ψ(θ)) by (c.f. (2.13)):

n
G(θ, x) = x, gx  IRn = xj gx,j .
j=1
3.8. IMPLEMENTATION OF THE INVERSION 133

Step 1: State-Space Decomposition:

parameter : (θ1 · · · θn ),
state vector : y = (uk , xkj , k = 1 · · · n − 1, j = 1 · · · k + 1),
state equation : (3.75), (3.76), and(3.77),
observation operator : M : y  x = xn−1 ∈ IRn ,

Step 2: Lagrangian Formula (2.25) gives here

L(θ, y, Λ) = G(θ, M(y)) − e(θ, y), Λ >

n n−1
= xn−1
j gx,j − (uk − cos θk ) λk
j=1 k=1
n−1 k
− (xkj − xk−1
j uk ) μkj
k=1 j=1
n−1
− (xkk+1 − sin θk ) μkk+1 ,
k=1

where the Lagrange multiplier vector is

Λ = (λ, μ) = (λk , μkj , k = 1 · · · n − 1, j = 1 · · · k + 1).

As expected, the Lagrangian is an explicit function of all its arguments.

Step 3: Adjoint Equation

n n−1
∂L
δy = δxn−1
j gx,j − δuk λk
∂y j=1 k=1
n−1 k
− (δxkj − δxk−1
j uk − xk−1
j δuk ) μkj
k=1 j=1
n−1
− δxkk+1 μkk+1 .
k=1
134 CHAPTER 3. CHOOSING A PARAMETERIZATION

Factorizing δy = (δuk , δxkj , k = 1 · · · n − 1, j = 1 · · · k + 1) gives

n−1 k
∂L
δy = − δuk (λk − xk−1
j μkj )
∂y k=1 j=1
n n−1 k+1 n−1 k
+ δxn−1
j gx,j − δxkj μkj + δxk−1
j uk μkj .
j=1 k=1 j=1 k=1 j=1

Making the index shift k  k − 1 in the last summation gives, as δx01 = 0,

n−1 k
∂L
δy = − δuk (λk − xk−1
j μkj ) (3.78)
∂y k=1 j=1
n
+ δxn−1
j (gx,j − μn−1
j )
j=1
n−2 k+1
− δxkj (μkj − uk+1 μk+1
j ),
k=1 j=1

which is of the form (2.26). Following (2.27), the adjoint state Λθ is the
solution obtained by equating to zero the coefficients of δxn−1
j , δxkj , and δuk
in (3.78):

μn−1
j = gx,j ∀j = 1 · · · n,
μkj = u k+1
μk+1
j ) ∀k = n − 2 · · · 1 , ∀j = 1 · · · k + 1,
k
λ k
= xk−1
j μkj ∀k = n − 1 · · · 1.
j=1

Step 4: Gradient Equation According to (2.28), gθ = ψ  (θ)T gx is given

by
∂L
gθ,k = (θ, yθ , Λθ ) ∀k = 1 · · · n − 1,
∂θk
= − sin θk λk + cos θk μkk+1 ∀k = 1 · · · n − 1,
! "# $ ! "# $
xkk+1 uk

gθ,k = −xkk+1 λk + uk μkk+1 ∀k = 1 · · · n − 1.

3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 135

3.9 Maximum Projected Curvature: A

Descent Step for Nonlinear Least Squares
We have seen in Sect. 3.8.1 how to transform the original constrained non-
linear least squares problem into the formally unconstrained problem:
def 1
x̂opt minimizes J(xopt ) = F (xopt )2 over IRnopt , (3.79)
2
where (see (3.69) and (3.70) or left part of Fig. 3.8):
def  
F (xopt ) = ϕ P (Ψ(xopt )) − z. (3.80)
For the rest of this section, we shall drop the index opt, so that x will denote
the optimization parameter.
We have seen that the determination of the gradient ∇J of the objective
function by adjoint state was required both in the preliminary study of the
problem (for the determination of the number of independant parameters
that can be retrieved from the data, in Sect. 3.2), and in adaptive parameter-
ization algorithms (for the determination of first order refinement indicators,
Sect. 3.7, Remark 3.7.2).
It is hence natural to use gradient-based descent optimization algorithms
for the solution of (3.79), and it becomes the only possible choice when the
number of optimization parameters is large enough to make the resolution of
the linearized problem – and hence confidence regions – unaffordable [78, 79].
But the efficiency of descent methods is hampered by the back-tracking –
and the many function evaluations – required at each iteration to obtain an
acceptable descent step. We describe in this section the Maximum Projected
Curvature (MPC) step [22, 2, 3], which takes advantage of the nonlinear least
squares structure of the problem: it uses the first and second derivatives of
the forward map F in the descent direction to determine a descent step that
is always acceptable.

3.9.1 Descent Algorithms

We recall first the organization of a descent algorithm for the resolution of
problem (3.79). Given the current estimate xk , the algorithm
1. Determines a descent direction yk , such that
yk , ∇J(xk ) < 0. (3.81)
136 CHAPTER 3. CHOOSING A PARAMETERIZATION

It is the way the descent direction yk is chosen, which gives its name
to the optimization algorithm class: Steepest Descent, Quasi-Newton,
Gauss–Newton, Levenberg–Marquardt, etc.

2. Chooses a search curve gk : IR+ → IRn in the parameter space, such

that
gk (0) = xk , g (0) = yk . (3.82)
In practice, all descent algorithms perform a straight line search, along
the half-line originating at xk pointing in the direction yk :

gk (α) = xk + α yk α ≥ 0. (3.83)

The reason behind this choice is mostly its simplicity. But we shall
also consider other search curves, in particular, if one can afford to
compute the Jacobian F  (xk ), the more intrinsic geodesic search curves
(Definition 3.9.1 below).

3. Computes an acceptable descent step αk , such that

• αk decreases sufficiently the objective function:

J(gk (αk )) ≤ J(xk ) + ω αk yk , ∇J(xk ) (3.84)

(Armijo condition),

where 0 < ω ≤ 1/2 is a given number (the same through all iterations
of course),
• αk is not too small:
⎧ 
⎪
⎪ g (αk ), ∇J(gk (αk )) ≥ ω yk , ∇J(xk )
⎪
⎪
⎪
⎪ (Wolfe condition),
⎨
or
⎪
⎪
⎪
⎪
⎪
⎪ J(gk (αk )) ≥ J(xk ) + ω  αk yk , ∇J(xk )
⎩
(Goldstein condition),

where ω < ω  < 1 is also a given number. The determination of αk is

the linesearch part of the optimization algorithm. It differs from one
implementation to the other inside the same class of algorithm, and is
largely responsible for its performance.
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 137

An example of acceptable descent step is the Curry step ᾱ: one moves
on the search curve g in the parameter space until the first stationary
point of J = 1/2F 2 is encountered:
d
ᾱ = Inf{α ≥ 0 such that F (gk (α))2 = 0}. (3.85)
dα
However, the Curry step cannot be used in practice, as its determina-
tion would require to many function evaluations.
4. Once αk is accepted, the k + 1 iterate is defined by
xk+1 = gk (αk ). (3.86)

3.9.2 Maximum Projected Curvature (MPC) Step

We define in this subsection the MPC step αk for given xk , yk , and gk , and so
we drop the iteration index k, with the exception of xk and xk+1 for obvious
reasons.
Let P be the curve of the data space, which is the image by F of the
search curve g of the parameter space:
P = F ◦ g : α ∈ IR+  F (g(α)) ∈ IRq .
We make the hypothesis that
P ∈ W 2,∞ (]0, ᾱ[, IRq ), (3.87)
where ᾱ is the Curry step defined in (3.85). Hence the two first distributional
derivatives of P with respect to α are in L∞ (]0, ᾱ[, IRq ), and we can define,
for almost every α ∈]0, ᾱ[, the residual r, the velocity V , the acceleration A,
the arc length ν, and the Curry arc length ν̄ along P by
r= P , (3.88)
V = dP/dα = Dg F (g), (3.89)
A = d2 P/dα2 = Dg2 ,g F (g) + Dg F (g), (3.90)
 α
ν(α) = V (α) dα, ν̄ = ν(ᾱ) . (3.91)
0

which satisfy
V ∈ C 0 ([0, ᾱ], IRq ), A ∈ L∞ (]0, ᾱ[, IRq ).
138 CHAPTER 3. CHOOSING A PARAMETERIZATION

Definition 3.9.1 Let F be derivable with F  (x) injective for all x ∈ IRn .
We shall say that g is a geodesic search curve if it satisfies the differential
equation
F  (g)T F  (g) g  + F  (g)T Dg2 ,g F (g) = 0 ∀α ≥ 0. (3.92)
The acceleration A is hence orthogonal to the hyperplane {δy ∈ IRq | δy =
F  (g)δx} tangent to the attainable set F (IRn ) at P (g), which shows that P
is a geodesic of F (IRn ). The velocity V and acceleration A satisfy moreover

dV 2 /dα = 0 =⇒ ν(α) = αV (0) ∀α ≥ 0,

A2 = Dg2 ,g F (g)2 − Dg F (g)2. (3.93)

We shall consider in the sequel only the straight search curves (3.83), or the
geodesic search curves of Definition 3.9.1.
The Definition (3.85) of the Curry step ᾱ along P implies that

V (α) = 0 ∀α ∈ [0, ᾱ[, (3.94)

and Proposition 8.2.1 shows that the first and second derivatives v and a of
P with respect to the arc length ν are given, for almost every α ∈]0, ᾱ[, by

v = V /V , v = 1 with v, a = 0, (3.95)

   
a = A − A, vv /V  , a2 = A2 −A, v2 /V 4 . (3.96)
2

We make now the additional hypothesis that

∃ 1/R > 0 : A(α) ≤ 1/R V (α)2 for almost every α ∈]0, ᾱ[, (3.97)

so that the radius of curvature ρ(α) along P satisfies (Proposition 8.2.2)

def
1/ρ(α) = a(ν(α)) ≤ 1/R for almost every α ∈]0, ᾱ[. (3.98)

The idea behind the MPC step is then to use the lower bound R on the
radius of curvature to find a calculable “worst case” lower bound ᾱW to the
computationally unaffordable Curry step ᾱ.
We give first an intuitive presentation of the MPC step in the case of a
two-dimensional data space IRq = IR2 . Then P is a plane curve which, as one
can see in Fig. 3.9, starts at F (xk ) (arc length ν = 0) in the direction v = v(0).
After that, the only thing known about P is, according to (3.98), that its
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 139

ν̄L 0
(target)

( x k) 1)
r̄L =F (x k
+
0)

r̄
F
P( )=
ᾱ W
P(

r̄W
ν̄

v
ν̄W
first stationary point

R
curve P : α  F ◦ g(α)

Figure 3.9: Geometrical representation of MPC step and

Theorem 3.9.4 in a two-dimensional data space, where projected
curvature and curvature coincide. The two thick dashed lines have
the same arc length

curvature remains smaller than 1/R. Hence there is no way, as expected, to

determine in a computationally efficient way the Curry arc length ν̄!
But as one can see in Fig. 3.9, the curve that leads to the smallest sta-
tionary arc length ν̄W among all curves of curvature smaller than 1/R that
leave F (xk ) in the direction v is the arc of circle of radius R (lower thick
dashed line) that “turns away” from the target 0. This curve is the “worst”
(hence the subscript W ) from the optimization point of view, as it produces
the largest stationary residual r̄W !
The MPC step is then ᾱW = ν −1 (ν̄W ): it consists in moving on the search
curve g in the parameter space up to the point ᾱW such that xk+1 = g(ᾱW )
has its image F (xk+1 ) at an arc length distance ν̄W of the origin F (xk ) of P
(thick dashed line part of P on Fig. 3.9).
140 CHAPTER 3. CHOOSING A PARAMETERIZATION

An important property that can be seen on the figure is that the residual
rk+1 = F (xk+1 ) at the point obtained by the MPC step is better (smaller)
than the worst case stationary residual r̄W . This will ensure for the MPC
step a useful “guaranteed decrease property.”
We turn now to the rigorous presentation of the MPC step in the general
case of a data space of any dimension. The curve P does not necessarily
remain anymore in the plane defined by P (0) and v(0), and the curvature
that matters for the determination of the worst-case Curry step turns out
be the projected curvature 1/ρproj (α) of P on the plane defined by the two
vectors P (α) and v(α). But this plane is defined only when the two vectors
do not point in the same direction, and so we define a set of exceptional
points:

Z = {α ∈ [0, ᾱ] such that P (α) − P (α), v(α)v(α) = 0}. (3.99)

The measure of Z can be strictly positive, and so one has to give special
attention to its points in the definition of projected curvature.
Definition 3.9.2 The projected radius of curvature along P is defined by

1/ρproj (α) = |a, n| for a.e. α ∈ [0, ᾱ] \ Z, (3.100)

1/ρproj (α) = 0 for a.e. α ∈ Z, (3.101)

where n is a unit vector orthogonal to v in the P, v plane, when it is defined

n = (P − P, vv)/(P − P, vv) (3.102)

(P, v, a are evaluated at α).

One expects the projected curvature of P to be smaller than its curvature,
this is confirmed by the:
Proposition 3.9.3 Let P satisfy the hypothesis (3.87) and (3.97). Then

1/ρproj (α) ≤ 1/ρ(α) for a.e. α ∈ [0, ᾱ] \ Z, (3.103)

1/ρproj (α) = 1/ρ(α) = 0 for a.e. α ∈ Z, (3.104)

and the following equality holds:

|a, P | = P − P, vv/ρproj for a.e. α ∈]0, ᾱ[, (3.105)

where P, v, a, ρproj are evaluated at α.

3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 141

The proof of the proposition is given in Appendix 1. The next theorem proves
the intuitive properties of the MPC step seen on Fig. 3.9:
Theorem 3.9.4 Let xk , y, g, ᾱ be the current estimate, descent direction,
search curve, and Curry step at iteration k of a descent optimization al-
gorithm for the resolution of problem (3.79). Let r(α) denote the residual
along P as defined in (3.88), and suppose that the curve P : α  F (g(α))
satisfies (3.87) and (3.97).
Then one can choose as descent step αk the MPC step ᾱW
αk = ᾱW defined by ν(ᾱW ) = ν̄W , (3.106)
which satisfies
0 < ᾱW ≤ ᾱ (3.107)
def def def
rk = r(0) > r̄W ≥ rk+1 = r(ᾱW ) ≥ r̄ = r(ᾱ), (3.108)
where
• ν̄W and r̄W are the worst case stationary arc length and residual:
ν̄L
ν̄W = R tan−1 , (3.109)
R + r̄L
1
r̄W = ((R + r̄L )2 + ν̄L2 ) 2 − R, (3.110)

• ν̄L and r̄L are the linear case stationary arc length and residual:
 
 V 

ν̄L =  F (xk ), , (3.111)
V  
)
r̄L = rk2 − ν̄L2 , (3.112)

where V = V (0) = Dy F (xk ) and rk = r(0) = F (xk ).

• 1/R is an upper bound to the projected curvature of P over the [0, ᾱ]
interval:
1/ρproj (α) ≤ 1/R for almost every α ∈ [0, ᾱ]. (3.113)

The proof of Theorem 3.9.4 is given in Appendix 2. The theorem remains

true when the data space IRq is replaced by an infinite dimensional Hilbert
space – the proof goes over without change.
142 CHAPTER 3. CHOOSING A PARAMETERIZATION

Proposition 3.9.5 Let P satisfy the hypothesis (3.87) and (3.97). Then the
MPC step αk :
1. Ensures a guaranteed decrease of the objective function:
1 2  1
J(xk ) − J(xk+1 ) = rk − rk+1
2
≥ ν̄W ν̄L > 0, (3.114)
2 2

2. Satisfies the Armijo condition (3.84) with the ratio:

 αk
1 V (αk ) 1
ω= where V (αk ) = V (α) dα. (3.115)
2 V (0) αk 0
In particular, one can always pretend one has reparameterized the
search curve g by the arc length ν along P , as this does not change
xk+1 ! Then α = ν, V (α) = v(ν), and V (α) = v(ν) = 1, so that
the Armijo condition is satisfied with ω = 1/2, which corresponds to a
quadratical decrease property.

The proof of the proposition is given in Appendix 3. The guaranteed decrease

property (3.114) is sharp: for a linear problem, one has R = +∞, so that
ν̄W = ν̄L , and equality holds in (3.114).
Remark 3.9.6 Choice of the search curve g. One sees from (3.114)
that the decrease of the objective function at iteration k is proportional to ν̄W ,
which itself is an increasing function, by formula (3.109), of the upper bound
1/R on the projected curvature. Let 1/ρstraight (0) and 1/ρgeodesic (0) denote
the curvature at α = 0 of the curves Pstraight and Pgeodesic associated to the
straight and geodesic search curves. Pgeodesic has the smallest curvature at
F (xk ) among all curves of F (IRn ) passing through F (xk ) in the direction Vk .
One of them is Pstraight , hence,

1/ρgeodesic (0) ≤ 1/ρstraight (0).

This does not imply necessarily that the projected curvatures satisfy the same
inequality, as it depends on the relative position of F (xk ), the acceleration
astraight (0) (orthogonal to Vk ), and the acceleration ageodesic (0) (orthogonal to
the tangent plane). But over a large number of problems and iterations, one
can expect that

in the mean: 1/ρproj,geodesic (0) ≤ 1/ρproj,straight (0).

3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 143

Similarly, there is no reason for 1/ρproj,geodesic (α) to remain smaller than

1/ρproj,straight (α) up to the Curry step ᾱ! But one can expect that the upper
bounds on the projected curvature satisfy

in the mean: 1/Rgeodesic ≤ 1/Rstraight =⇒ ν̄W,geodesic ≥ ν̄W,straight .

Hence using the geodesic search curve amounts giving oneself a better chance
of making a larger descent step!

3.9.3 Convergence Properties for the Theoretical

MPC Step
We recall first the definition of convergence for a minimization algorithm:
Definition 3.9.7 A minimization algorithm is convergent if and only if it
produces a sequence of iterates that satisfy ∇J(xk ) → 0 when k → +∞.
Convergent descent algorithm compute only stationary point – one has to rely
on Q-wellposedness results (Chap. 4) to ensure that this stationary point is
actually a minimizer.

Proposition 3.9.8 Define

 *
1
D = x ∈ IR | J(x) = F (x) ≤ J(x0 ) ,
n 2
(3.116)
2
where x0 is the initial guess for the resolution of problem (3.79), and suppose
that F  (x) and Dv,v
2
F (x) exist ∀x ∈ D, ∀y ∈ IRn and satisfy

F  (x)v ≤ Mv ∀x ∈ D, ∀y ∈ IRn , (3.117)

Dv,v
2
F (x) ≤ 1/R F  (x)v2 ∀x ∈ D, ∀y ∈ IRn , (3.118)

for some M ≥ 0 and 1/R ≥ 0. Then for any current estimate xk ∈ D and
any descent direction yk , the curve Pk = F ◦ gk , where gk is the associated
straight or geodesic search curve, satisfies

Pk ∈ W 2,∞ (]0, ᾱk [, IRq ), (3.119)

Ak (α) ≤ 1/Rk Vk (α) a.e. on ]0, ᾱk [ for some Rk ≥R. (3.120)
2

Hence one can apply to the resolution of the nonlinear least squares problem
(3.79) the descent algorithm (3.81), (3.82), and (3.86) with the MPC step
144 CHAPTER 3. CHOOSING A PARAMETERIZATION

(3.106) and (3.109) along the straight search curve (3.83) or the geodesic
search curve (3.92). The iterates xk and descent directions yk satisfy then

∇J(xk ) cos2 θk < ∞, (3.121)

k∈IN

where θk ∈ [0, π/2] is the angle between −∇J(xk ) and the descent direction
yk at iteration k, and the algorithm converges for the following choices of the
descent directions:

• yk + Fk Fk = 0, (Steepest descent)

• Fk T Fk yk + Fk Fk = 0, (Gauss–Newton)

provided ∃β > 0 s.t. Fk y ≥ βy ∀y ∈ IRn ∀k = 0, 1, 2 . . .

• Mk yk + Fk Fk = 0, (Quasi-Newton)
provided the matrices Mk are uniformly bounded and coercive,

• Fk T Fk yk + λk yk + Fk Fk = 0, (Levenberg–Marquardt)

provided λk is chosen such that λk ≥ c/(1 − c)Fk .

The proof is given in Appendix 4 below. If C is a closed convex subset of D,

hypothesis (3.117) and (3.118) show that F, C is a FC problem in the sense of
Definition 4.2.1. These hypothesis are satisfied in quite general situations: for
the finite dimensional problems considered here, for example, if D is bounded,
F smooth, and F  (x) injective for all x of D (see Sect. 4.5).

3.9.4 Implementation of the MPC Step

The exchange of information between the optimization and modeling routine
required for the MPC step is as follows:

1. Knowing the current estimate xk , the modeling routine returns

def
• either ∇Jk = ∇J(xk ) (Steepest Descent or Quasi Newton)
def
• or Fk = F  (xk ) (Gauss–Newton or Levenberg–Marquardt)

2. Using these informations, the optimization routine determines a descent

direction yk
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 145

3. Knowing yk , the modeling routine returns:

• Dy2k ,yk F (xk ) second directional derivative of the forward model

• Dyk F (xk ) in the case where only ∇Jk is available in step 1

4. Finally, the optimization routine determines

• The search curve gk

• An MPC descent step αk along gk

and sets xk+1 = gk (αk )

The computation of one or two directional derivatives of the forward model
in step 3 represents the additional computational burden (compare with
Fig. 3.8) required for the implementation of the MPC step; it is largely
compensated by a much smaller amount of back-tracking, as the computed
MPC step will be directly admissible most of the time (remember that the
theoretical MPC step defined in the previous section is always admissible,
Proposition 3.9.5).
We detail in this subsection the calculations involved in step 4. One com-
putes first
Vk = Dyk F (xk ) = Fk yk , vk = Vk /Vk , (3.122)
using either Dyk F (xk ), provided in step 3 when only ∇Jk is available, or the
Jacobian Fk when it is available in step 1.

• Determination of gk
We shall use for gk a second degree polynomial curve in the parameter space:
α2 
gk (α) = xk + α yk + g , (3.123)
2 k
where gk is to be chosen.
1. Case of a straight search curve. This choice is the only one for Steepest
Descent or Quasi Newton algorithms, as the modeling routine returns
only ∇Jk , but not Fk , so that no information is available on the tangent
plane to the attainable set at F (xk ). One can of course use for gk the
linear function (3.83), which corresponds to gk = 0. But one can also
take advantage of the Proposition 3.9.5, which shows that the MPC
step satisfies the Armijo condition with ω = 1/2 when the arc length ν
146 CHAPTER 3. CHOOSING A PARAMETERIZATION

is proportional to α, and use for gk the second order polynomial (3.123),

where gk is chosen such that d2 ν/dα2 (0) = 0:

gk + vk , Dy2k ,yk F (xk )yk = 0.

The computational cost is negligible, as Dy2k ,yk F (xk ) has in any case
to be computed for the determination of the MPC step αk , and this
choice increases the chances that the first guess of αk to be determined
below in (3.129) satisfies the Armijo condition.
2. Case of a geodesic search curve. This choice is possible only for the
Gauss–Newton or the Levenberg–Marquardt algorithms, and in general
for all algorithms where the Jacobian F  (xk ) is available. The numerical
resolution of the differential equation (3.92) is feasible, but computa-
tionally intensive. So we shall use for gk the second degree polynomial
(3.123), where gk is chosen such that the associated curve Pk has a
second order contact with the geodesic at F (xk ) = Pk (0):

Fk T Fk gk + Fk T Dy2k ,yk F (xk ) = 0. (3.124)

The additional cost here is the resolution of the linear system (3.124),
which turns out to be the same linear system as the Gauss–Newton
equation for the descent direction yk in Proposition 3.9.8, but with a
different right-hand side. By construction, this (approximated) geodesic
search curve has the same advantages as the straight line search, plus
the prospect of producing in the mean larger steps (Remark 3.9.6), and
hence a faster decrease of J.
In both cases, the arc length along Pk satisfies d2 ν/dα2 (0) = 0, so that the
second order development of ν at 0 reduces to

ν(α) = αVk  + o(α2 ). (3.125)

• Determination of αk
There are two points in the determination of αk by Theorem 3.9.4, where an
exact calculation unfeasible, and which require an approximation:
1. Determination of 1/Rk . A precise determination of the upper bound
1/Rk to the (unknown) projected curvature along Pk up to the (un-
known) Curry step ᾱk (condition (3.113)) would require the evaluation
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 147

of 1/ρproj at many points along Pk up to ᾱk , and would be at least

as expensive as the numerical determination of the Curry step ᾱ itself!
But there is one thing available at almost no cost, namely the projected
curvature at the origin of Pk . So one can define a tentative Rk by

Rk = κk ρproj,k with 0 ≤ κk ≤ 1, (3.126)

where κk is a security factor that accounts for the possible increase

of the projected curvature along the path, and ρproj,k is given by
Proposition 3.9.3:

1/ρproj,k = |ak , Fk |/Fk − Fk , vk vk ,

with ak given by (3.96) and (3.90):

 
ak = Ak − Ak , vk vk /Vk 2 , (3.127)
Ak = Dy2k ,yk F (xk ) + Dgk F (xk ), (3.128)

where Dgk F (xk ) is given by


−vk , Dy2k ,yk F (xk )Vk (straight line search)
Dgk F (xk ) =
Fk gk (geodesic search)

2. Determination of αk . An accurate determination of αk by (3.106) would

require the use of a quadrature formula to evaluate the arc length func-
tion ν(α), and hence a large number of evaluations of F , something one
cannot afford. So one replaces ν(α) by its second order approximation
(3.125), and define a tentative MPC step by

αk = ν̄W,k /Vk , (3.129)

where ν̄W,k is given by (3.109)

ν̄L,k
ν̄W,k = Rk tan−1 with (3.130)
Rk + r̄L,k
)
ν̄L,k = |Fk , vk |, r̄L,k = Fk 2 − ν̄L,k
2
. (3.131)

Because of the above approximations, there is no guarantee that, at the

difference of the theoretical MPC step, this tentative MPC step satisfies the
148 CHAPTER 3. CHOOSING A PARAMETERIZATION

Armijo condition (3.84) with ω  1/2 as stated by Proposition 3.9.5! Hence

a tentative αk will be accepted according to the (less demanding) following
Armijo condition, for some ω ∈]0, 1/2] (often ω = 10−4):

J(gk (αk )) ≤ J(xk ) + ω αk yk , ∇J(xk ). (3.132)

• If (3.132) is violated, then Rk is reduced by a factor τ ∈]0, 1[:

Rk ← τ Rk ,

and the calculations (3.129), (3.130), and (3.131) are repeated.

• If (3.132) is satisfied, then αk is accepted, and xk is updated:

xk+1 = gk (αk ),

with gk given by (3.123).

3.9.5 Performance of the MPC Step

A comparison of the MPC step (for both straight and geodesic search) with
the Armijo, Goldstein, and Wolfe line search is available in [2] for the steep-
est descent, Gauss–Newton, and Levenberg–Marquardt algorithms, together
with a comparison of the best performers with the confidence region and
Levenberg Marquardt algorithms of the Matlab library.
The comparison of the algorithms have been conducted on a set of 13
test problems, and for various performance criterions (convergence, number
of function evaluations, value of objective function at convergence, etc). We
refer to [2] for the presentation of the performance profiles used for this
comparison, and for a detailed account of the comparisons. We summarize
here the overall result of this study:
• Straight vs. geodesic search: The comparison turns to the advantage
of the geodesic search whenever it is available (Gauss–Newton and
Levenberg–Marquardt algorithms)

• Steepest descent algorithm: The MPC step (with straight line search
necessarily) performs better than the classical Armijo and Goldstein
stepping algorithms, but worse than the Wolfe stepping (see [13, 2] for
the definition of the stepping algorithms)
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 149

• Gauss–Newton algorithm: The MPC step with geodesic search performs

better than the Armijo, Wolfe, and Goldstein stepping algorithms

• Levenberg–Marquardt algorithms: The MPC step with geodesic search

performs better than the Armijo and Goldstein stepping algorithms,
but worse than Wolfe stepping

• Best performers: Gauss–Newton with MPC step and geodesic search

performs better than the Matlab Levenberg–Marquardt and confidence
region algorithms

Appendix 1: Proof of Proposition 3.9.3

Reparameterization of P by Arc Length
The curve α  P (α) of the data space satisfies (3.87) and (3.97), and
Proposition 8.2.2 part (i) shows that the reparameterization p : ν ∈ [0, ]ν̄] 
IRq of P as a function of the arc lenth ν is a W 2,∞ ([0, ν̄], IRq ) function: the first
and second derivatives v and a of p given by (3.95) and (3.96), the curvature
1/ρ defined by (3.98), and the projected curvature 1/ρproj defined by (3.100)
and (3.101) exist almost everywhere on [0, ν̄], and belong to L∞ ([0, ν̄], IRq ).
The α  ν change of variable is continuously derivable and strictly in-
creasing (see (3.94)), so that zero measure sets of [0, ᾱ] correspond to zero
measure sets of [0, ν̄]. Hence Proposition 3.9.3 will be proved if we show that

1/ρproj (ν) ≤ 1/ρ(ν) a.e. on [0, ν̄] \ Z, (3.133)

1/ρproj (ν) = 1/ρ(ν) = 0 a.e. on Z, (3.134)

where Z is now defined by

Z = {ν ∈ [0, ν̄] such that p(ν) − p(ν), v(ν)v(ν) = 0}. (3.135)

Comparison of Curvature and Projected Curvature

The inequality (3.133) follows immediately from the Definitions (3.98) of ρ
and (3.100) and (3.101) of ρproj for α ∈
/ Z. Then (3.134) will be proved if we
show that
1/ρ(ν) = a(ν) = 0 for a.e. ν ∈ Z. (3.136)
150 CHAPTER 3. CHOOSING A PARAMETERIZATION

So let ν ∈ Z be given. Two cases can happen:

1. ν is an isolated point of Z: There exists η > 0 such that ]ν − η, ν +

η[∩Z = {ν}. But the set of isolated points of Z has a zero measure,
and so we can ignore these points.

2. ν is not an isolated point of Z: ∀n ∈ IN−{0}, there exists νn ∈ Z, νn = ν

such that |νn − ν| ≤ 1/k. Hence (p = p(ν), pn = p(νn ) etc.):
   
0 = pn − pn , vn vn − p − p, vv (3.137)
= pn − p − pn − p, vn vn
−p, vn − vvn
−p, v(vn − v).

Because p and v are derivable almost everywhere on [0, ν̄], we can sup-
pose that p (ν) = v(ν) and v  (ν) = a(ν) exist. Then dividing (3.137) by
νn − ν and passing to the limit gives, as p and v are continuous at ν,

p, av + p, va = 0,

which implies, as v and a are orthogonal, that p, av = 0 and

p, va = 0. By definition of the Curry step ν̄, one has p, v < 0,
and hence a = 0, and (3.136) is proved.

Appendix 2: Proof of Theorem 3.9.4

We suppose for simplicity that the Curry step satisfies

ᾱ < +∞,

which corresponds to the usual practical situation. But the proof remains
true if ᾱ = +∞, provided one defines r̄ by

r̄ = lim r(α).
α→ᾱ

As we have seen in Appendix 1, the reparameterization p of P as a function

of the arc length ν belongs to W 2,∞ ([0, ν̄], IRq ).
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 151

Reparameterization by the Squared Residual Decrease

Following (3.99), we denote by r the residual along p and by r̄ its Curry
value:
r(ν) = p(ν) for all ν ∈ [0, ν̄], r̄ = r(ν̄).
By definition of the Curry step ν̄, the ν  r function is strictly decreasing
over the [0, ν̄] interval. Hence we can use as new parameter along the curve
the decrease t of the squared residual:

t = r02 − r(ν)2 , where r02 = r(0)2 , (3.138)

which satisfies
def
0 ≤ t ≤ t̄ = r02 − r̄ 2 .
Derivation of (3.138) gives

dt/dν = −2p, v = 2 |p, v|, (3.139)

and the Curry arc length ν̄ we want to minorate can be written as

 t̄
1 dt
ν̄ = . (3.140)
2 0 |p, v|

Notice that this integral is singular, as p, v → 0 when t → t̄.

Stationary Linearized Residual and Projected

Curvature
Let now r̄L (where L stands for “linearized”) denote the linearized stationary
residual, that is, the stationary residual along the tangent to p at arc length ν:

r̄L (ν)2 = p(ν) − p(ν), v(ν)v(ν)2 = p2 − p, v2. (3.141)

For a linear problem, p is a straight half line of the data space IRq , and so
r̄L is constant. We calculate in this section |dr̄L /dt|, which will provide the
adequate measure of nonlinearity for our purpose.
Derivation of (3.141) with respect to ν gives, together with (3.139) and
(3.105),
 dr̄ 2 
 L r̄L
  = |p, a| = for a.e. t ∈ [0, t̄ ].
dt ρproj
152 CHAPTER 3. CHOOSING A PARAMETERIZATION

The function t  r̄L is derivable a.e. on [0, t̄ ] \ Z (as the square root of
the strictly positive, a.e. derivable function t  r̄L2 ), where (c.f. (3.99) and
(3.135))
Z = {t ∈]0, t̄ [ such that r̄L (t) = 0}.
Hence  dr̄ 
 L 1 1
 = ≤ for a.e. t ∈ [0, t̄ ] \ Z. (3.142)
dt 2ρproj 2R
We prove now that this property remains true a.e. on Z. Let t ∈ Z be a
point where dr̄L /dt exists. Then r̄L (t) = 0 and r̄L (t ) ≥ 0 for a.e. t ∈ [0, t̄ ],
which implies that dr̄L /dt = 0. Hence (3.142) holds necessarily true at any
such point, where by definition 1/ρproj = 0 ≤ 1/R, and we are left to prove
that r̄L is derivable a.e. on [0, t̄ ].
We define for that purpose a sequence of functions
ηk = max{r̄L , 1/k} k = 1, 2, . . . ,
which converges simply to r̄L , and hence in the sense of distributions
ηk −→ r̄L in D  (]0, t̄ [) when k −→ +∞. (3.143)
This sequence is bounded independently of k in the L∞ (0, t̄) norm:
ηk ∞ ≤ max{r̄L ∞ , 1} ≤ max{p∞ , 1} ≤ max{p(0), 1}. (3.144)
The functions ηk are derivable a.e. on [0, t̄ ], as the square root of the a.e.
derivable functions max{r̄L2 , 1/k 2 } ≥ 1/k 2 > 0. Hence,
⎧ 
⎪  dr̄L  1 1 1
⎪  (t) = ≤
 ⎪
for a.e. t such that r̄L (t) > ,
 dη ⎨ dt 2ρproj 2R k
 k 
 (t) =
dt ⎪
⎪
⎪
⎩ 0 1
for a.e. t such that r̄L (t) ≤ ,
k
where we have used (3.142) to evaluate dr̄L /dt when r̄L > 1/k. This implies
 dη 
 k 1
  ≤
dt ∞ 2R
which, together with (3.144), shows that the sequence ηk , k = 1, 2 . . . , is
bounded in, say, H 1 (]0, t̄ [). Hence there exists a subsequence, still denoted
by ηk , and w ∈ H 1 (]0, t̄ [) such that
ηk  w weakly in H 1 (]0, t̄ [) ⊂ D (]0, t̄ [) when k −→ +∞. (3.145)
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 153

Comparison of (3.143) and (3.145) implies that r̄L = w ∈ H 1 (]0, t̄ [), which
proves the desired result: r̄L is a.e. derivable over [0, t̄ ]. Hence
 dr̄ 
 L 1 1
 = ≤ for a.e. t ∈ [0, t̄ ], (3.146)
dt 2ρproj 2R
and the following majoration holds for the continuous function r̄L :
t def
r̄L (t) ≤ r̄L (0) + = r̄L,W (t) for all t ∈ [0, t̄ ], (3.147)
2R
where r̄L,W is the worst case stationary linearized residual. As we shall prove
in the Sect. 3.9.5, and as one can guess from Fig. 3.9, r̄L,W is actually the
stationary linearized residual along the arc of circle pW of radius R (thick
lower dashed line on the figure), which turns away from the target 0 in the
plane containing p(0) and v(0). But for the time being, one can simply take
(3.147) as the definition of r̄L,W .

Comparison with the Worst Case

Let r̄L and r̄L,W be defined by (3.141) and (3.147), and define, for t ≥ 0

μ(t) = t + r̄L (t)2 = r02 − p(t), v(t)2 , (3.148)

μW (t) = t + r̄L,W (t)2 , (3.149)

(the graph of μW is a parabola), which satisfy (Fig. 3.10)

μ(0) = μW (0) = r̄L (0)2 < r02 , (3.150)

μ(t) ≤ μW (t) for all t ≥ 0, (3.151)

where we have used the fact that v(0), p(0) < 0 (v(0) is a descent direction),
and the majoration (3.147).
First, the right equality in (3.148) shows that t corresponds to a stationary
point of r 2 = p(t)2 , where p(t), v(t) = 0, if and only if μ(t) = r02 . Hence
the first stationary residual r̄ 2 and the corresponding parameter t̄ are given by

t̄ = inf{t > 0 | μ(t) = r02 }, r̄ 2 = r02 − t̄ = r̄L (t̄)2 .

Next, the worst case function μW is monotonous, so we can define uniquely

t̄W and a worst case stationary residual r̄W by (Fig. 3.10),

μW (t̄W ) = r02 , 2
r̄W = r02 − t̄W = r̄L,W (t̄W )2 , (3.152)
154 CHAPTER 3. CHOOSING A PARAMETERIZATION

μ
r02
μW μ

r̄L(0)2

0 tW t̄W t0 t t̄ r02 t

Figure 3.10: Properties of the μW and μ functions

which satisfy, using (3.151),

t̄W ≤ t̄, r̄W ≥ r̄. (3.153)

Then combining (3.140) with (3.148), one can express ν̄ and define ν̄W by
 
1 t̄ dt 1 t̄W dtW
ν̄ = , ν̄W = . (3.154)
2 0 (r0 − μ(t))
2 1/2 2 0 (r0 − μW (tW ))1/2
2

We are now close to our objective, which is to show that ν̄ ≥ ν̄W . We remark
first, using (3.151), that for a given t, the first integrand is larger than the
second one – this goes in the right direction, but it does not allow to conclude,
as the two domains of integrations are not the same! So we make a change
of variable in order to obtain integrals defined over the same interval. Define
first t0 by (Fig. 3.10):

t0 = max{t ∈ [0, t̄[ | μ(t) = r̄L (0)2 }.

The range of μ over the [t0 , t̄] interval is then [r̄L (0)2 , r02 ], the same as the
range of μW over [0, t̄W ]. Hence, we can define a (nonmonotonous) change of
variable t  tW from the [t0 , t̄] interval onto the [0, t̄W ] interval by

μW (tW ) = μ(t) ∀t ∈ [t0 , t̄],

3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 155

where tW is uniquely defined because of the strict monotonicity of μW . Be-

cause of (3.151), t and tW satisfy, as in (3.153),

tW ≤ t, r̄L,W (tW ) ≥ r̄L (t) ∀t ∈ [t0 , t̄]. (3.155)

We can now use this change of variable in the integral that gives ν̄W :
 
1 t̄W
dtW 1 t̄
μ (t) dt
ν̄W = = 
,
2 0 (r0 − μW (tW ))
2 1/2 2 t0 μW (tW ) (r0 − μ(t))1/2
2

where μW (tW ) > 0 and μ (t) can be positive or negative. But differentiation
of (3.148) gives, using (3.155) and (3.146),

dr̄L
|μ (t)| = |1 + 2r̄L (t) (t)|
dt
dr̄L
≤ 1 + 2r̄L (t)| (t)|
dt
1
≤ 1 + r̄L,W (tW ) = μW (tW ).
R
Hence ν̄W satisfies
 
1 t̄
|μ (t)| dt 1 t̄
dt
ν̄W ≤ 
≤ ,
2 t0 μW (tW ) (r0 − μ(t))
2 1/2 2 t0 (r02 − μ(t))1/2
 t̄
1 dt
≤ = ν̄,
2 2
0 (r0 − μ(t))1/2

and (3.107) is proved. We turn now to the proof of (3.108). Let t(ν̄W ) be the
squared residual decrease along p at the worst case stationary arc length ν̄W .
Formulas (3.154) for the arc length give
 t(ν̄W )  tW
1 dt 1 dt
ν̄W = = .
2 0 (r0 − μ(t))
2 1/2 2 0 (r02 − μW (t))1/2

The first integrand is smaller than the second, so t(ν̄W ) has to be larger than
tW , which gives, by definition of t and r̄W ,

r02 − r(ν̄W )2 ≥ r02 − r̄W

2
,

which is (3.108).
156 CHAPTER 3. CHOOSING A PARAMETERIZATION

Determination of the Worst Case

It remains now to prove formula (3.109) and (3.110) for ν̄W and r̄W . We
calculate first the integral (3.154), which gives ν̄W . Formula (3.147) for r̄L,W
gives, with the notations r̄L = r̄L (0) , ν̄L2 = r02 − rL (0)2 :
 t 2
r02 − μW (t) = r02 − t + (r̄L + ) ,
2R
t 2
= ν̄L2 + (R + r̄L )2 − (R + r̄L + )
2R
= (1 − u2(t))(ν̄L2 + (R + r̄L )2 ),

where u is defined by
t
R + r̄L +
u(t) =  2R
2
1/2 .
ν̄L + (R + r̄L )2
With this change of variable, formula (3.154) becomes
 1
du
ν̄W = R
u(0) (1 − u )
2 1/2
&π '
= R − sin−1 u(0)
2
ν̄L
= R tan−1 ,
R + r̄L
which is (3.109). We calculate now the worst residual r̄W . The definitions
(3.152) of t̄W , r̄W , and (3.147) of r̄L,W show that

t̄W
r̄W = r̄L,W (t̄W ) = r̄L + .
2R
Hence the first equation of (3.152) rewrites
2
t̄W + r̄W = r02
2R(r̄W − r̄L ) + r̄W
2
= r02 .

The positive root of this second order equation for r̄W is

r̄W = −R + (R2 + 2Rr̄L + r02 )1/2 ,

which is (3.110).
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 157

The above expressions for ν̄W and r̄W are those of the first stationary arc
length and residual along a circle pW of radius R (thick lower dashed line in
Fig. 3.10), which turns away from the target 0 in any plane containing p(0)
and v(0) (there can be many if these two vectors are colinear). This proves
that the worst case is achieved by the circle(s) pW .

Appendix 3: Proof of Proposition 3.9.5

We prove first the guaranteed decrease property (3.114). One has
rk2 = r(0)2 = (ν̄L2 + r̄L2 ), (by definition)
2
rk+1 ≤ r̄W
2
. (formula (3.108))
Let γ ∈ [0, π/2[ be defined by (Fig. 3.9):
1
sin γ = ν̄L /((R + r̄L )2 + ν̄L2 ) 2 , (3.156)
tan γ = ν̄L /(R + r̄L ). (3.157)
Using first the Definition (3.110) of r̄W , then (3.156) and (3.157), and finally
(3.109) one finds
1 2 1 2  1 2 
(rk − rk+1
2
) ≥ ν̄L + r̄L2 − ((R + r̄L )2 + ν̄L2 ) 2 − R
2 2
1 2 
= r̄L − (R + r̄L )2 − R2 + 2R ν̄L / sin γ
2
= −Rr̄L − R2 + R ν̄L / sin γ
= −R(ν̄L / tan γ − R) + R ν̄L / sin γ
1 − cos γ
= R ν̄L
sin γ
1 − cos γ
= ν̄W ν̄L
γ sin γ
The function γ  (1 − cos γ)/(γ sin γ) increases from 1/2 to 2/π when γ goes
from 0 to π/2, which proves (3.114). Then the Definition (3.106) of αk gives
 αk
ν̄L = ν(αk ) = V (α)dα = αk V (αk ),
0

and (3.111) gives, as F (xk ), V  < 0,

ν̄L = −F (xk ), V /V (0) = −∇J(xk ), yk /V (0). (3.158)
158 CHAPTER 3. CHOOSING A PARAMETERIZATION

Hence the guaranteed decrease writes

1 1 V (αk )
ν̄W ν̄L = − αk ∇J(xk ), yk  = −ωαk ∇J(xk ), yk ,
2 2 V (0)
which proves with (3.114) that the Armijo condition (3.84) is satisfied for ω
given by (3.115), and ends the proof of Proposition 3.9.5.

Appendix 4: Proof of Proposition 3.9.8

Properties (3.119) and (3.120) follow immediately from (3.117) and (3.118)
using either (3.90) with g  = 0 when gk is a straight search curve, or (3.93)
when gk is a geodesic search curve.
Hence the hypothesis of Theorem 3.9.4 are satisfied, and one can use for
all k the MPC step αk = ᾱW computed from Rk by (3.106) and (3.109). The
guaranteed decrease property (3.114) writes then at iteration k
1
J(xk ) − J(xk+1 ) ≥ ν̄W,k ν̄L,k
2
1 ν̄W,k 2
= ν̄
2 ν̄L,k L,k
1 Rk  ν̄L,k  2
= tan−1 ν̄ .
2 ν̄L,k Rk + r̄L,k L,k
The last right-hand side is a decreasing function of r̄L,k and an increasing
function of Rk . Hence one obtains using r̄L,k ≤ rk ≤ r0 and (3.120)
1 R  ν̄L,k  2
J(xk ) − J(xk+1 ) ≥ tan−1 ν̄
2 ν̄L,k R + r0 L,k
1 R  r0  2
≥ tan−1 ν̄ ,
2 r0 R + r0 L,k
R  r 
where we have used ν̄L,k ≤ rk ≤ r0 and the fact that tan−1 is a
r R + r0
2
decreasing function of r over the [0, R + r0 ] interval. We replace now ν̄L,k by
its expression (3.158) and use hypothesis (3.117)
1 R r0 ∇J(xk ), yk 2
J(xk ) − J(xk+1 ) ≥ tan−1
2 r0 R + r0 Vk 2
1 R r0
≥ 2
tan−1 ∇J(xk )2 cos2 θk
2M r0 R + r0
3.9. MAXIMUM PROJECTED CURVATURE (MPC) STEP 159

Summing up the last inequalities for k = 0, · · · K gives

J(x0 ) ≥ J(x0 ) − J(xK+1 ) ≥ C(M, R, r0 ) ∇J(xk )2 cos2 θk ,

k=1···K

which proves the convergence of the series (3.121). Finally, the conditions
associated to each choice of descent direction ensure that cos2 θk ≥ cmin > 0.
Together with the convergence of the series (3.121), this implies that

∇J(xk )2 → 0,

which is the definition of convergence of the algorithm. This ends the proof
of Proposition 3.9.8.
Chapter 4

Output Least Squares

Identifiability
and Quadratically Wellposed
NLS Problems

We consider in this chapter the nonlinear least squares (NLS) problem (1.10),
which we recall here for convenience:
1
x̂ minimizes J(x) = ϕ(x) − z2F over C. (4.1)
2
As we have seen in Chap. 1, this inverse problem describes the identification
of the parameter x ∈ C from a measurement z of ϕ(x) in F . We suppose
that the minimum set of hypothesis (1.12) of Chap. 1 holds:
⎧
⎪
⎪ E = Banach space, with norm  E ,
⎪
⎪
⎪
⎪ C ⊂ E with C convex and closed,
⎪
⎪
⎨ F = Hilbert space, with norm  F ,
z ∈ F (4.2)
⎪
⎪
⎪
⎪ ϕ : C  F is differentiable along segments of C,
⎪
⎪
⎪
⎪ and : ∃ αM ≥ 0 s.t. ∀x0 , x1 ∈ C, ∀t ∈ [0, 1],
⎩
Dt ϕ((1 − t)x0 + tx1 )F ≤ αM x1 − x0 E ,

and we recall the definition of stationary points:

G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 161
and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 4, c Springer Science+Business Media B.V. 2009
162 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Definition 4.0.9 A point x̃ ∈ C is a stationary point of problem (4.1) if it

satisfies the first-order necessary optimality condition:

ϕ(x̃) − z, Dt=0 ϕ((1 − t)x̃ + tx)F ≥ 0 ∀x ∈ C. (4.3)

A point x̃ is a parasitic stationary point if x̃ satisfies (4.3) but x̃ is not a

solution of (4.1).
A local minimum of J on C is a stationary point.

Our objective is to find additional conditions on C, ϕ, and z which ensure

that (4.1) is both wellposed and optimizable:

• A wellposed problem in the sense of Hadamard has a unique solution,

which depends continuously on the right-hand side of the equation. For
the optimization problem (4.1) of interest, uniqueness of the solution
x̂ can hold only if the data z has a unique projection on D = ϕ(C),
which cannot be true for any z when D is not convex! So the existence,
uniqueness, and stability properties will be required only to hold for
data z that stay in some neighborhood ϑ of D = ϕ(C) in F .

• Optimizability means the absence of parasitic stationary points in the

quadratic objective function. It is an extremely useful property for non-
linear inverse problems, as it make the least squares formulation nu-
merically constructive, by ensuring that local optimization algorithms
will not stop prematurely in a parasitic stationary point.

To ensure constructiveness of the NLS formulation (4.1), these two prop-

erties are combined in the Definition 1.3.2 of quadratic (Q)-wellposedness, or,
in parameter estimation terms, of output least squares (OLS)-identifiability
of x, which we recall here:

Definition 4.0.10 Let ϕ and C be given. The parameter x is OLS-identifiable

in C from a measurement z of ϕ(x) if and only if the NLS problem (4.1) is
Q-wellposed, that is, if ϕ(C) possesses an open neighborhood ϑ such that

(i) Existence and uniqueness: For every z ∈ ϑ, problem (1.10) has a

unique solution x̂

(ii) Unimodality: For every z ∈ ϑ, the objective function x  J(x) has no

parasitic stationary point
4.1. THE LINEAR CASE 163

(iii) Local stability: The mapping z  x̂ is locally Lipschitz continuous

from (ϑ, · F ) to (C, · E ).
In the nonlinear least-squares problem (4.1), the distance of the data z ∈
F to the attainable set D = ϕ(C) represents the modeling and measurement
errors, and we would like to be sure that keeping these errors below a certain
level r will ensure that z belongs to the set ϑ on which existence, uniqueness,
optimizability, and stability hold.
This is why we seek in this chapter sufficient conditions that ensure that
ϑ contains an open enlargement neighborhood of the attainable set
+ ,
ϑ ⊃ z ∈ F | d(z, D) < r (4.4)

for some r > 0, which then represents the upper limit of the noise and
modeling error level for which the NLS problem is well posed.
We recall first in Sect. 4.1 the results for the linear case we would like to
generalize to nonlinear problems, and define in Sect. 4.2 the class of FC/LD
problems that retain some useful properties of the linear case. We introduce
in Sect. 4.3 the notions of linearized identifiability and stability. These ingre-
dients are used in Sect. 4.4 to state a sufficient conditions for Q-wellposedness
of NLS problems, based on the results of Chaps. 7 and 8 on strictly quasi-
convex (s.q.c.) sets. The case of finite dimensional parameters is studied in
Sect. 4.5, where it is shown that Q-wellposedness holds locally as soon as
the linearized problem is identifiable. The remaining sections are devoted to
examples of Q-wellposed parameter estimation problems in elliptic equations.

4.1 The Linear Case

We recall here the case where ϕ is linear:
ϕ(x) = Bx with B ∈ L(E; F ), (4.5)
which will serve as a guideline for the study of the nonlinear case.
The output set ϕ(C) = B.C is then a convex set D, and we can take
advantage of the good properties of the projection on convex sets:
Proposition 4.1.1 Let D be a convex set of the Hilbert space F . Then
(i) Uniqueness: For any z ∈ F , there exists at most one projection X̂ of z
on D
164 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

(ii) Unimodality: If z ∈ F admits a projection X̂ on D, the “distance to z”

function has no parasitic stationary point on D (its unique stationary
point is X̂)
(iii) Stability: If z0 , z1 , ∈ F admit projections X̂0 , X̂1 , on D, then
X̂0 − X̂1 F ≤ z0 − z1 F ,

(iv) Existence: If z ∈ F , any minimizing sequence Xn ∈ D of the “distance

to z” function over D is a Cauchy sequence for the distance X − Y F ,
and Xn → X̂ = unique projection of z on the closure C of C.
If moreover D is closed, then X̂ ∈ C.
It is then straightforward to combine the above properties of the projection
with the hypothesis that the parameter x is identifiable (respectively, stable)
according to Definition 1.3.1:
Proposition 4.1.2 Let hypothesis (4.2) and (4.5) hold.
1. If x is identifiable, that is,
By = 0 =⇒ y = 0, (4.6)
and if the attainable set B.C is closed, then x is OLS-identifiable for
the “ arc length distance”
δ(x0 , x1 ) = B(x0 − x1 )F (4.7)
in data space: the linear least squares problem (4.1) is Q-wellposed with
ϑ = F , and for any z0 and z1 in F , the unique solutions x̂0 and x̂1 of
(4.1) satisfy the stability estimate
δ(x̂0 , x̂1 ) ≤ z0 − z1 F . (4.8)

2. If the estimation of x from Bx is stable, that is,

∃αm > 0 s.t. αm yE ≤ ByF ∀y ∈ E, (4.9)
then B.C is necessarily closed, and x is OLS-identifiable and problem
(4.1) Q-wellposed for the norm  · E , with the same ϑ = F , and the
stability estimate
αm x̂0 − x̂1 E ≤ z0 − z1 F . (4.10)
4.2. FINITE CURVATURE/LIMITED DEFLECTION PROBLEMS 165

For finite dimensional parameters, the stability inequality (4.9) follows

immediately from the identifiability property (4.6) (that is, the injectivity
of B).
Proof. The attainable set B.C is closed either by hypothesis (part 1), or by
a completeness argument using (4.9) and the fact that C is closed (part 2).
It is also convex, as the image of a convex set by a linear operator. The
proposition follows then immediately from the properties of the projection
on a closed convex sets recalled in Proposition 4.1.1.
The above proposition may seem to be a complicated way to state simple
results, but its interest is that it gives precisely the stencil of the results we
shall be able to generalize to some nonlinear inverse problems in the rest of
this chapter.
Linear inverse problems that do not satisfy the hypotheses of Proposi-
tion 4.1.2 may or may not have a solution, and the usual cure is to bring
additional information by regularization (Sect. 1.3.4). When the L.M.T. reg-
ularization is used, the original problem (4.1) is replaced by (1.25), which
satisfies the hypotheses of Proposition 4.1.2 as soon as  > 0, and exhibits
nice convergence properties when  → 0 (this will be studied in detail in
Sect. 5.1.1).
We define in the next section a class of nonlinear problems that retains
the properties of linear problems recalled in Propositions 4.1.1 and 4.1.2.

4.2 Finite Curvature/Limited Deflection

Problems
The good properties of the linear case followed from the convexity of the
output set ϕ(C). To generalize these properties to the nonlinear case, we
introduce the class of FC/LD problems, which all have in common the fact
that their attainable set is strictly quasiconvex (s.q.c.) – instead of being
convex as in the linear case.
The generalization of convex sets to s.q.c. sets is the subject of Chaps. 6–8.
These chapters are quite technical, but the only thing we need in the present
chapter is the combination of the practical sufficient condition for strict qua-
siconvexity developed in Chap. 8 with the properties of s.q.c. sets regarding
projection developed in Chaps. 6 and 7 and summarized in Theorem 7.2.11.
We shall refer to the above chapters for the proofs, and give in this section
166 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

a self-contained and hopefully intuitive presentation of the definition and

properties of FC/LD problems.
We define first the class of FC problems. FC problems were originally
called weakly nonlinear when they were introduced in [28]; but we prefer here
the denomination FC, which describes them better as we shall see below.
To any pair of points x0 , x1 of C, we associate a curve P on ϕ(C) by

P : t ∈ [0, 1]  ϕ((1 − t)x0 + tx1 ) ∈ ϕ(C). (4.11)

Under the minimum set of hypothesis (4.2), P (t) has a derivative V (t) =
Dt ϕ((1 − t)x0 + tx1 ), and the arc length of the curve P of the output set
ϕ(C) is  1
L(P ) = V (t)F dt. (4.12)
0

When t  P (t) is constant, then V (t)F = 0 ∀t ∈ [0, 1] and hence

L(P ) = 0, so that the curve P is reduced to one point of ϕ(C) and does not
provide any information on the shape of ϕ(C).
On the contrary, the curves P for which L(P ) > 0 stay by construction
on the attainable set ϕ(C), and so it is understandable that an upper bound
on their curvature brings some information on the shape of ϕ(C):

Definition 4.2.1 Let C and ϕ satisfy the minimum set of hypothesis (4.2).
Problem (4.1) is a finite curvature least squares problem (in short: a FC
problem) if the following conditions are satisfied:
⎧
⎪
⎪ there exists R > 0 such that
⎪
⎨ ∀x0 , x1 ∈ C, the curve P : t  ϕ((1 − x0 )t + tx1 ) satisfies
1 (4.13)
⎪
⎪ P ∈ W 2,∞ ([0, 1]; F ) and A(t)F ≤ V (t)2F for a.e. t ∈ [0, 1],
⎪
⎩ R
where V (t) = P (t), A(t) = P  (t).

Of course, linear problems are FC problems, as they satisfy obviously

the definition with 1/R = 0. The “finite curvature” name given to property
(4.13) of problem (4.1) is justified in Chap. 8 (Propositions 8.2.1 and 8.2.2),
where it is proved that, when it holds

• Either P (t) =constant, so that L(P ) = 0 and the curve P is reduced

to one point of ϕ(C) – its curvature is not defined
4.2. FINITE CURVATURE/LIMITED DEFLECTION PROBLEMS 167

• Or V (t) = 0 ∀t ∈ [0, 1], so that L(P ) > 0 and the radius of curvature
ρ(t) along the curve P satisfies

1 A(t)F
≤ for a.e. t in [0, 1], (4.14)
ρ(t) V (t)2F

so that the radius of curvature of the curve P and of the attainable

setϕ(C) (Definition 7.2.8) satisfy
1 def 1 1 def 1 1
= sup ≤ = sup ≤ < +∞,
R(P ) t∈[0,1] ρ(t) R(ϕ(C)) x0 ,x1 ∈C , t∈[0,1] ρ(t) R
(4.15)
which explains the FC name given to this class of problems.

The lower bound R to the radius of curvature R(ϕ(C)) of the attainable set
is called a radius of curvature of the inverse problem (4.1) (Definition 4.2.3
below).
Notice that (4.13) is only a sufficient condition for the radius of curvature
along P to be larger than R (compare (4.13) with the formula (8.66) for the
curvature, and/or think of ϕ : [0, 1]  IR2 defined by ϕ(x) = (x2 , x2 )).

Without further constraints, the attainable set ϕ(C) of a FC problem

may

• Fold over itself, which prevents property (i) of Proposition 4.1.1 to hold
on any neighborhood of ϕ(C) (think of the greek letter α as attainable
set, or consider the simple case where C = [0, diam C] ⊂ IR, and ϕ(x) =
(cos x, sin x) ∈ IR2 , so that ϕ(C) is an arc of circle of radius 1 and arc
length diam C, when diam C > 2π).

• Or possess no neighborhood on which the unimodality property (ii)

of Proposition 4.1.1 holds (think again of the arc of circle with π ≤
diam C < 2π: the point z = (cos diam C, sin diam C) is on ϕ(C), so
the “distance to z” function attains its global minimum on ϕ(C) at
z = ϕ(diam C)) (with value 0!), but it has also a stationary point (in
fact, a local minimum when π < diam C) at (0, 0) = ϕ(0) (with a
strictly positive value)).

It is hence necessary to constrain further ϕ(C) if the properties of the pro-

jection listed in Proposition 4.1.1 for the linear case are to be extended to
168 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

V (t) θ(t, t)

P (t)
P (t) P

V (t)

Figure 4.1: The deflection θ between two points of a curve P

the nonlinear case. A natural thing is to impose a constraint on the deflec-

tion Θ of the curves P , defined as the largest angle θ(t, t ) ∈ [0, π] between
any two tangent vectors V (t) and V (t ) to P (Fig. 4.1 and Definition 8.0.4).
For example, the previous arc-of-circle example with radius R = 1 has a
deflection bounded by Θ = diam C. Let us check whether limiting Θ can
prevent for this simple case the attainable set to fold over, and can restore
the unimodality of the projection:

• If Θ < 2π, the arc of circle can obviously not fold over itself!

• If Θ ≤ π/2, all points of IR2 at a distance of the arc of circle strictly

smaller than R = 1 have the uniqueness and unimodality properties (i)
and (ii) of Proposition 4.1.1 (see Fig. 7.3, top).

So the condition Θ ≤ π/2 seems to ensure a nice behavior of the attainable

set – at least for the arc-of-circle example. This is in fact general, as we shall
see in Proposition 4.2.7 below, so we make a definition:

Definition 4.2.2 Let C and ϕ satisfy the minimum set of hypothesis (4.2).
A FC problem (4.1) is a Limited Deflection least squares problem (in short,
a FC/LD problem) if it satisfies the Deflection Condition:
π
Θ≤ . (4.16)
2
The attainable set of an FC/LD problems is s.q.c. The upper bound Θ to
the deflection of the curves P is called the deflection of the inverse problem
(4.1).
4.2. FINITE CURVATURE/LIMITED DEFLECTION PROBLEMS 169

Proof. The strict quasi-convexity property of the attainable set follows from
Theorem 8.1.6, which gives a sufficient condition for a set to be s.q.c.
We explain now how to estimate the deflection Θ of a FC problem. We
remark that when P is an arc of circle, Θ is simply equal to the length of the
arc divided by its radius, that is, the product of the size of the arc by its curva-
ture. For a curve P other than a circle, a similar formula holds at the infinites-
imal level: the deflection dθ between the endpoints of the arc corresponding
to parameters t and t + dt is bounded (not equal in general) by the length
of the arc V (t)F dt multiplied by its curvature 1/ρ(t) (Proposition 8.1.2):
V (t)F dt
dθ ≤ , (4.17)
ρ(t)
(with the equality if P is a plane curve!). Combining with (4.14) gives
A(t)F V (t)F
dθ ≤ dt ≤ dt. (4.18)
V (t)F R
If we denote by t0 and t1 the values of t corresponding to the points of
the curve P where the deflection is maximum, we obtain, for the maximum
deflection Θ(P ) along the curve P ,
 t1  1  1
A(t)F
Θ(P ) = dθ ≤ dθ ≤ dt ≤ L(P )/R ≤ L/R, (4.19)
t0 0 0 V (t)F
1
where L(P ) = 0 V (t)F dt denotes the arc length of P and L an upper
bound to L(P ), This shows that any number Θ which satisfies
 1
A(t)F
dt ≤ Θ ≤ L/R for all x0 , x1 ∈ C (4.20)
0 V (t)F

is an upper bound to the deflection of all curves P defined in (4.11). Of

course, the angle between two vectors belongs always to the [0, π] interval, so
this upper bound on the deflection constrains actually the deflection of the
curves P of ϕ(C) only if Θ happens to be strictly smaller than π!
The above geometric approach to the estimation of the deflection is
only meant to provide an intuitive support to formula (4.20). We refer to
Theorem 8.2.3 in Chap. 8 for a rigorous proof.
We summarize the geometric quantities of interest introduced so far for
the analysis of FC problems:
170 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Definition 4.2.3 The geometric attributes of the FC problem (4.1) are the
following:
1. Its radius of curvature R > 0, defined as a lower bound to the radius
of curvature along all curves P of the form (4.11) with L(P ) > 0. It is
estimated by (4.13)
2. Its (arc length) size L ≥ 0, defined as an upper bound to the arc length
L(P ) of all curves P of the form (4.11). It is estimated by
 1
L(P ) = V (t)F dt ≤ L ≤ αM diam C ∀x0 , x1 ∈ C, (4.21)
0

where diam C is the diameter of C:

diam C = supx,y∈C x − yE , (4.22)

3. Its deflection Θ ≥ 0 defined as an upper bound to the deflection Θ(P )

of all curves P of the form (4.11) with L(P ) > 0. It is estimated by
(4.23) below
Then formula (4.20) gives
Proposition 4.2.4 Let (4.1) be a FC problem with curvature 1/R and size
L ≤ αM diam C. Then any angle Θ that satisfies
⎧ 1
⎨ 0 θ(t) dt ≤ Θ ≤ L/R,
where: (4.23)
⎩
A(t)F ≤ θ(t)V (t)F for a.e. t ∈ [0, 1] and all x0 , x1 ∈ C
is an upper bound to the deflection of the FC problem (4.1).
The majoration Θ ≤ L/R in (4.23) is sharp: consider once again the arc-of-
circle example. Then R = 1 is a lower bound to the radius of curvature of
the curves (4.11), and L = diam C is the upper bound to their arc length.
When diam C ≤ π, the largest deflection of the curves (4.11) is precisely
Θ = diam C, so that Θ = L/R.
Corollary 4.2.5 The FC problem (4.1) with curvature 1/R and size L ≤
αM diam C is a FC/LD problem as soon as one of the following sufficient
conditions is satisfied:
 1
A(t)F π
dt ≤ for all x0 , x1 ∈ C, (4.24)
0 V (t)F 2
4.2. FINITE CURVATURE/LIMITED DEFLECTION PROBLEMS 171

or
π
A(t)F ≤ V (t)F for a.e. t ∈ [0, 1] and all x0 , x1 ∈ C, (4.25)
2
or
π
L≤ R. (4.26)
2
Proof. it follows immediately from Proposition 4.2.4.
Condition (4.26) shows that the deflection condition can be enforced by
a size×curvature condition, that is, by requiring that the product of the size
L of ϕ(C) by its curvature 1/R is bounded, here by π/2 .
Though the deflection condition Θ ≤ π/2 can be satisfied for an un-
bounded set ϕ(C) (think, e.g., of the graph of a sine function in IR2 ),
Corollary 4.2.5 shows that Θ ≤ π/2 can be ensured by limiting the size of
the admissible parameter set C (A(t) is proportional to x1 − x0 2 and
V (t) is proportional to x1 − x0 !). Hence the deflection condition will act
in practice as a localization constraint: this is an example of regularization
by size reduction mentioned in Sect. 1.3.4.
We can now state the nonlinear counterpart of Proposition 4.1.1 concern-
ing the properties of the projection on ϕ(C). The stability of the projection
will hold for the “arc length distance” δ(X0 , X1 ) on the attainable set ϕ(C),
defined by
∀X0 , X1 ∈ ϕ(C), δ(X0 , X1 ) = supx0 ∈ϕ−1 (X0 ),x1 ∈ϕ−1 (X1 ) L(P ), (4.27)
where L(P ) is the length, defined in (4.12), of the curve P associated to x0
and x1 by (4.11).
Remark 4.2.6 The quantity δ(X0 , X1 ) is positive. But without further hy-
pothesis, it can happen that δ(X0 , X1 ) = 0 and X0 = X1 . However, for a
FC/LD problem, as it is the case in Proposition 4.2.7 below, δ(X0 , X1 ) = 0
implies X0 = X1 (Proposition 6.2.5), so that the first axiom of a distance
is satisfied. The second axiom of a distance δ(X0 , X1 ) = δ(X1 , X0 ) is always
satisfied, but we do not know whether the third axiom (triangular inequality)
is satisfied, this is why we use the word distance between quotes.
Proposition 4.2.7 Let (4.1) be a FC/LD problem with curvature 1/R < ∞,
and let ϑ be the enlargement neighborhood of ϕ(C) defined by
+ ,
ϑ = z ∈ F | d(z, ϕ(C)) < R . (4.28)
172 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Then the projection on the attainable set ϕ(C) has the following properties:
(i) Uniqueness: For any z ∈ ϑ, there exists at most one projection of
z on ϕ(C)

(ii) Unimodality: If z ∈ ϑ admits a projection X ( on ϕ(C), the “distance

to z” function has no parasitic stationary point on ϕ(C)

(iii) Local stability: If z0 , z1 ∈ ϑ admit projections X (0 , X

(1 on ϕ(C) and
are close enough so that there exists d ≥ 0 satisfying

z0 − z1 F + max d(zj , ϕ(C)) ≤ d < R, (4.29)

j=0,1

then one has

-
(0 − X
X (1 F ≤ L(P() ≤ (1 − d R(P())−1 z0 − z1 F , (4.30)

where P( is the curve associated by (4.11) to any a couple x (1 of the

(0 , x
( (
preimage of X0 , X1 .
This implies, as L(P() ≤ δ(X (1 ) and R(P() ≥ R > 0, a stability
(0 , X
property for the arc length “distance” δ in the attainable set
-
X(0 − X(1 F ≤ δ(X(0 , X
(1 ) ≤ (1 − d R)−1 z0 − z1 F . (4.31)

(iv) Existence: If z ∈ ϑ, any minimizing sequence Xn ∈ ϕ(C) of the “dis-

tance to z” function over ϕ(C) is a Cauchy sequence for both the dis-
tance X − Y F and the arc length “distance” δ(X, Y ). Hence Xn con-
( of z onto ϕ(C).
verges in F to the (unique) projection X
( ∈ ϕ(C), and δ(Xn , X)
If ϕ(C) is closed, then X ( → 0 when n → 0.

Proof. Equation (4.1) is a FC/LD problem, and so its attainable set ϕ(C)
is s.q.c. (Definition 4.2.2) and the proposition follows from the properties of
the projection on s.q.c. sets summarized in Theorem 7.2.11.
We investigate now the shape of the preimage sets, that is, the sets of
parameters x ∈ C that have the same image by ϕ. For a linear problem
ϕ(x) = B.x, the preimage of X ∈ B.C is the intersection of the closed affine
subspace {x ∈ E such that B.x = X} of E with the admissible parameter
set C, it is hence closed and convex. For FC/LD problems, the following
result holds:
4.2. FINITE CURVATURE/LIMITED DEFLECTION PROBLEMS 173

Proposition 4.2.8 Let (4.1) be a FC/LD problem. Then the preimage

ϕ−1 (X) is a closed and convex set for all X ∈ ϕ(C).

Proof. let X ∈ ϕ(C) be given. The finite curvature hypothesis implies that
C is closed and ϕ continuous (second and last properties of (4.2)), and hence
that ϕ−1 (X) is closed. Then the condition Θ ≤ π/2 on the deflection implies
that ϕ(C) is s.q.c. (Proposition 4.2.7). Let then x0 , x1 be two pre-image of
X, and P the curve (4.11) image by ϕ of the [x0 , x1 ] segment of C. The
function t  X − P (t)2 is s.q.c. (use (7.4) with D = ϕ(C) and z = X in
the Definition 7.1.2 of s.q.c. sets), positive, and takes the value 0 for t = 0
and t = 1: this can be possible only if X − P (t)2 = 0 ∀t ∈ [0, 1]. Hence
xt = (1 − t)x0 + tx1 belongs to ϕ−1 (X) for all t ∈ [0, 1], which shows that
ϕ−1 (X) is convex.
Propositions 4.2.7 and 4.2.8 (and also 4.3.3 below) show that FC/LD
problems are a direct generalization of linear least squares problems.

Remark 4.2.9 All known examples of FC/LD problems correspond to for-

ward maps ϕ, which are injective over C, for which ϕ(X)−1 = {x}, which is
trivially convex. The existence of non injective function ϕ that produce FC
problems – and hence FC/LD problems by reduction of the size of C – is an
open problem.

Remark 4.2.10 When the deflection Θ is larger than π, we have seen in

the arc-of-circle example that the conclusions of Proposition 4.2.7 cannot be
true. But what when π/2 < Θ ≤ π? Theorem 8.1.6, which serves to prove
Proposition 4.2.7, shows that ϕ(C) is still s.q.c., but with a smaller regular
neighborhood + ,
ϑ = z ∈ F | d(z, ϕ(C)) < RG , (4.32)

where RG ≤ R is the global radius of curvature (Sect. 7.2), as soon as the ge-
ometric attributes 1/R, L , and Θ of the FC problem (4.1) (Definition 4.2.3)
satisfy the extended deflection condition
def
RG = R(sin Θ + (L/R − Θ) cos Θ) > 0. (4.33)

Figure 8.2 shows that the set of deflection Θ and size×curvature product L/R
that satisfy (4.33) is of the form

Θ ≤ Θmax (L/R) (or < depending on the value of L/R), (4.34)

174 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

with Θmax given by (8.19). This shows that the range of authorized deflections
can be extended beyond π/2. Examples of simple sets that have a deflection
larger than π/2 but are nevertheless s.q.c. can be seen in Figs. 7.3 and 8.3.
Hence the use of conditions (4.33) or (4.34) allows to enlarge the size of
the admissible parameter set C for which ϕ(C) is s.q.c. – so that Propositions
4.2.7 and 4.2.8 above and 4.3.3 below still hold – at the price of reducing the
size of the neighborhood ϑ on which the projection is well-behaved.
For example, in the case where only the worst deflection estimate Θ =
L/R (see (4.23)) is available, one sees that (4.33) is satisfied for any π/2 <
Θ < π, but on the smaller neighborhood (4.32) of size RG = R sin Θ.
For sake of simplicity, we shall not attempt, in the rest of this chapter
and in Chap. 5, which deal with the regularization of inverse problems, to
produce the least constraining conditions on the size of C. So we shall not
use the extended deflection condition (4.34), but only the simple deflection
condition (4.16), which will ensure that the output set ϕ(C) is s.q.c. with a
neighborhood (4.28) of size R independent of the deflection Θ.

4.3 Identifiability and Stability

of the Linearized Problems
We introduce in this section identifiability and stability properties of the lin-
earized problems, and look into their relation with the corresponding prop-
erties of the original nonlinear problem (Definition 1.3.1).
Definition 4.3.1 Let C, ϕ satisfy the minimum set of hypothesis (4.2). The
parameter x is linearly identifiable over C if
x0 , x1 ∈ C , t ∈ [0, 1] and V (t)F = 0 =⇒ x0 = x1 , (4.35)
where V (t) is defined in (4.13).

Remark 4.3.2 Identifiability and linear identifiability coincide of course for

linear problems, but also for the class of so-called “bilinear” problems where
the ϕ mapping admits a state-space decomposition (see (1.33) and (1.34) in
Sect. 1.3.5 of Chap. 1) of the form:
⎧
⎨ e(x, y) = b(x, y) + c(y) + d with b bilinear, c linear, d constant,
M(y) = linear and injective,
⎩
b, c, M = continuous.
4.3. LINEARIZED IDENTIFIABILITY AND STABILITY 175

This can be seen easily: given x0 , x1 ∈ C and t ∈ [0, 1], subtraction of the
state equations written at x1 and x0 gives

b(x1 − x0 , y1 ) + b(x0 , y1 − y0 ) + c(y1 − y0 ) = 0, (4.36)

and derivation with respect to t of the equation at xt = (1 − t)x0 − tx1 gives

∂yt ∂yt
b(x1 − x0 , yt ) + b(xt , ) + c( ) = 0, (4.37)
∂t ∂t
where yt is the solution of e(x, y) = 0 for x = xt .
Identifiability holds as soon as ϕ(x1 ) − ϕ(x0 ) = 0 =⇒ x1 = x0 , that is,
using the injectivity of M, as soon as y1 = y0 =⇒ x1 = x0 , or, using (4.36),

b(x1 − x0 , y1 ) = 0 =⇒ x1 = x0 . (4.38)

Similarly, linear identifiability will hold as soon as V = M ∂yt /∂t = 0 =⇒

x1 = x0 , or, using the injectivity of M and (4.37), as soon as

b(x1 − x0 , yt ) = 0 =⇒ x1 = x0 ,

which is equivalent to (4.38).

The class of “bilinear” problems is relatively large, as it contains all prob-
lems governed by a linear equation, where the unknown parameter appear as
a coefficient, see the examples in Sects. 1.4 and 1.6 of Chap. 1.

But for nonlinear problems, identifiability does not imply in general lin-
earized identifiability (think, e.g., of ϕ(x) = x3 on C = [−1, +1], but this is
not a FC problem), and conversely linearized identifiability does not imply
identifiability. However, for FC/LD problems the following result holds:
Proposition 4.3.3 Let (4.1) be a FC/LD problem. Then

linear identifiability (4.35) =⇒ identifiability (1.14)

Proof. Let X ∈ ϕ(C) be given, and x0 , x1 be two preimage of X. We know

from Proposition 4.2.8 that the curve P defined in (4.11) satisfies P (t) = X
for all t ∈ [0, 1]. Derivation with respect to t gives V (t) = 0 ∀t ∈ [0, 1], and
hence x0 = x1 using (4.35).
To define the linearized version of the stability property (4.41), we replace
now the length ϕ(x1 ) − ϕ(x0 ) of the [ϕ(x0 ), ϕ(x1 ] segment of F by the arc
length L of the curve P : t  ϕ((1 − t)x0 − tx1 ):
176 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Definition 4.3.4 Let C, ϕ satisfy the minimum set of hypothesis (4.2). The
parameter x is linearly stable on C if

∃αm > 0 such that ∀x0 , x1 ∈ C :
1 (4.39)
αm x0 − x1 E ≤ L(P ) = 0 V (t)F dt,

where L(P ) is the arc length of the curve P image by ϕ of the [x0 , x1 ] segment,
as defined by (4.13), (4.11), and (4.21). A sufficient condition for (4.39) to
hold is 
∃αm > 0 such that ∀x0 , x1 ∈ C, ∀t ∈ [0, 1] :
(4.40)
αm x0 − x1 E ≤ V (t)F .
Linear stability is weaker than stability:
Proposition 4.3.5 Let C, ϕ satisfy the minimum set of hypothesis (4.2).
Then
stability =⇒ linear stability.
Proof. The stability condition (1.15) implies (4.39) with αm = 1/k, which
proves the implication.

4.4 A Sufficient Conditions

for OLS-Identifiability
We establish in this section the additional condition that ensures that a
FC/LD problem is Q-wellposed, or equivalently that x is OLS-identifiable.
Finite dimensional problems are studied in Sect. 4.5, and an infinite dimen-
sional example is given in Sect. 4.8 of this chapter.
The use of regularization to produce a FC/LD problem, which meets this
condition – and is hence Q-wellposed – is studied in Chap. 5, for FC/LD
problems in Sect. 5.1.2, for general nonlinear problems in 5.1.3, and for a
special family of infinite curvature problems in Sect. 5.3.
As we have seen in (4.21) and (4.23), the deflection Θ can always be chosen
smaller than L/R ≤ αM diamC/R. Hence the projection on the output set of
FC problems can be made well-behaved by reducing the size of the admissible
parameter set C until the deflection condition Θ ≤ π/2 is met. But as in the
linear case, these nice properties of the projection are not enough to ensure
the OLS-identifiability of x in problem (4.1): one has to add the nonlinear
counterpart of the stability condition (4.9).
4.4. A SUFFICIENT CONDITION FOR OLS-IDENTIFIABILITY 177

The first and most natural way to do that consists in considering that B
in (4.9) is the forward map ϕ, and to replace (4.9) by the stability condition
(1.15) of Definition 1.3.1, which we recall here for convenience:

∃k ≥ 0 such that
(4.41)
x0 − x1 E ≤ k ϕ(x0 ) − ϕ(x1 )F ∀x0 , x1 ∈ C.

Of course, combining (4.41) with the stability property (4.30) of the projec-
tion onto ϕ(C) would ensure the desired OLS-identifiability property. But it
would not take full advantage of (4.30), where the stability of the projection
is achieved not only for the usual distance X (0 − X(1  of the projections
F

of z0 and z1 , but also for their larger “arc length distance” δ(X(0 , X
(1 ) mea-
sured along the curves P(. This distance is defined in term of the directional
derivative V (t) of ϕ (see (4.31)). This leads us to consider that B in (4.9) is
rather the derivative of the forward map ϕ, and to replace (4.9) by the linear
stability condition of Definition 4.3.4:

Theorem 4.4.1 Let (4.1) be a FC/LD problem with curvature 1/R < ∞.
If the linear stability condition (4.39) is satisfied, the parameter x is OLS-
identifiable, that is, the NLS problem (4.1) is Q-wellposed on a neighborhood
+ ,
ϑ = z ∈ F |d(z, ϕ(C)) < R , (4.42)

and the following local Lipschitz stability result holds for the inverse problem:
for all z0 , z1 ∈ ϑ close enough so that there exists d ≥ 0 satisfying

z0 − z1 F + max d(zj , ϕ(C)) ≤ d < R, (4.43)

j=0,1

the corresponding solutions x(0 , x(1 of (4.1) satisfy

-
αm x(0 − x(1 E ≤ L(P() ≤ (1 − d R(P())−1 z0 − z1 F , (4.44)

where

• L(P() is the arc length of the curve P( : t  ϕ((1 − t)x(0 + tx(1 ) image by
ϕ of the [x(0 , x(1 ] segment

• R(P() ≥ R is a lower bound to the radius of curvature along P(

178 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Proof. The linear stability hypothesis implies of course the linear identifia-
bility of x and, because the problem has finite curvature and satisfies the
deflection condition, identifiability of x on C (Proposition 4.3.3). Hence ϕ is
injective on C: any X ∈ ϕ(C) has a unique preimage x = ϕ−1 (X) in C.
Let then Xj ∈ ϕ(C), j = 0, 1, be two points on the output set, and
xj = ϕ−1 (Xj ), j = 1, 2, be the corresponding (unique) preimage. Combining
the Definition (4.27) of δ(X, Y ) with the linear stability condition (4.39)
shows that
 1
δ(X0 , X1 ) = L(P ) = V (t)F dt ≥ αm x0 − x1 E . (4.45)
0

We prove now point 1 of the Definition 4.0.10 of OLS-identifiability. Let

z ∈ ϑ be given. The uniqueness of the solution of problem (4.1) follows
from the injectivity of ϕ over C (see above) combined with the uniqueness
of the projection of z onto ϕ(C) (property i) of Proposition 4.2.7). As for
the existence of a solution, let Xn ∈ ϕ(C) be a minimizing sequence of the
“distance to z” function over ϕ(C), and xn = ϕ−1 (Xn ) the corresponding
sequence of preimage. Because the problem has finite curvature and satisfies
the deflection condition (4.16), Proposition 4.2.7 applies, so that Xn is a
Cauchy sequence for the “distance” δ(X, Y ) and hence, using (4.45), xn is a
Cauchy sequence in the Banach space E. But C is closed, so there exists
x̂ ∈ C such that xn → x̂.Then the last property of (4.2) implies that ϕ is
Lipschitz continuous, so that Xn → X ( = ϕ(x̂). This proves that X ( is the
projection of z onto ϕ(C), and that x̂ is the (unique) solution of problem (4.1).
Then point 2 of Definition 4.0.10 follows immediately from property (ii)
of Proposition 4.2.7, combined with the existence of a (unique) projection on
ϕ(C) for all z ∈ ϑ.
Finally, we notice that the stability property (4.43) (4.44) follows immedi-
ately from the property (iii) of the projection on ϕ(C) in Proposition 4.2.7,
combined with the stability result (4.45). This implies the local Lipschitz
continuity of the z  x̂ mapping from (ϑ, · F ) to (C, · E ), which shows
that point 3 of Definition 4.0.10 is satisfied.
4.5. THE CASE OF FINITE DIMENSIONAL PARAMETERS 179

4.5 The Case of Finite Dimensional

Parameters
Using the results of the previous sections, we derive in this section simple
sufficient conditions for Q-wellposedness (Definition 4.0.10) of problem (4.1)
when the unknown parameter x is finite dimensional. These conditions will
allow to check whether the additional information brought to the original ill-
posed problem by regularization (Sect. 1.3.4) and, if necessary, discretization
is sufficient to restore Q-wellposedness.
We shall suppose that the following finite dimension (FD) minimum set
of hypothesis holds (compare with (4.2))
⎧
⎪ E = finite dimensional vector space, with norm  E ,
⎪
⎪
⎪
⎪ C = closed, convex subset of E,
⎪
⎪
⎪
⎪ Cη = convex open neighborhood of C in E,
⎪
⎪
⎨ F = Hilbert space, with norm  F ,
z ∈ F, (4.46)
⎪
⎪
⎪
⎪ ϕ : Cη  F is twice differentiable along segments of Cη ,
⎪
⎪
⎪
⎪ ∂ ∂2
⎪
⎪ and : V = ϕ((1 − t)x0 + tx1 ), A = 2 ϕ((1 − t)x0 + tx1 )
⎪
⎩ ∂t ∂t
are continuous functions of x0 , x1 ∈ Cη and t ∈ [0, 1].

For example, when C is defined by

C = {x ∈ E | c (x) ≤ 0 ∀ ∈ L}, (4.47)

where c ,  ∈ L are continuous and convex constraints, the neighborhood Cη

can be defined simply by

Cη = {x ∈ E | c (x) < η ∀ ∈ L } (4.48)

for some η > 0.

The differentiability and continuity results required in (4.46) happen to
hold in most of the applications, so that the FD minimum set of hypothesis
(4.46) is satisfied for the reduction to finite dimension of a large number of
problems. But it is obviously not sufficient to ensure the Q-wellposedness of
the inverse problem (4.1) over the finite dimensional set C. The next theorem
indicates which additional properties are sufficient:
180 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Theorem 4.5.1 Let the FD minimum set of hypothesis (4.46) hold. Then
ϕ is continuously differentiable over Cη , and the following properties hold:

1. If C is bounded, then

the attainable set ϕ(C) is compact – and hence closed,

the minimum set of hypothesis (4.2) holds on C.

2. If moreover x is linearly identifiable (Definition 4.3.1) over Cη , then

x is linearly stable over C (Definition 4.3.4),

the NLS problem (4.1) is a FC problem (Definition 4.2.1).

3. If moreover C is small enough for the deflection condition Θ ≤ π/2 to

hold, then

x is OLS-identifiable on C, or equivalently:
the NLS problem (4.1) is Q-wellposed on C.

Proof. Let S denote the unit sphere of E. For any x0 , x1 ∈ Cη , x0 = x1 , and

for any t ∈ [0, 1], we can define
x1 − x0
x = (1 − t)x0 + tx1 ∈ Cη , h= ∈ S. (4.49)
x1 − x0 E

Then V and A defined in (4.46) satisfy

V = x1 − x0 E Dh ϕ(x), A = x1 − x0 2E Dh,h

2
ϕ(x), (4.50)
2
where Dh ϕ(x) and Dh,h ϕ(x) are the first and second derivatives of ϕ at x
in the direction h, which exist by hypothesis. This shows that V /x1 − x0 E
and A/x1 − x0 2E depend only on the point x ∈ Cη and on the direction
h ∈ S.
Conversely, given x ∈ Cη and h ∈ S, there are many ways to find x0 , x1 ∈
Cη , x0 = x1 , and t ∈ [0, 1], which satisfy (4.49), for example,
⎧
⎨ x0 = x − d(x, E \ Cη )h/2 ∈ Cη ,
x1 = x + d(x, E \ Cη )h/2 ∈ Cη , (4.51)
⎩
t = 1/2 ∈ [0, 1],
4.5. THE CASE OF FINITE DIMENSIONAL PARAMETERS 181

where d(x, E \ Cη ) denotes the distance of x to the complementary set of Cη

in E. The function x  d(x, E \ Cη ) is continuous, and satisfies
d(x, E \ Cη ) > 0 ∀x ∈ Cη . (4.52)
Hence by composing the continuous mappings (x, h)  (x0 , x1 , t) defined by
(4.51) and (x0 , x1 , t)  (V /x1 − x0 E , A/x1 − x0 2E ) defined by (4.50), we
see that the first and second directional derivatives of ϕ are given by
V A
Dh ϕ(x) = 2
, Dh,h ϕ(x) = ∀(x, h) ∈ Cη × S, (4.53)
x1 − x0 E x1 − x0 2E
and satisfy
2
Dh ϕ(x) and Dh,h ϕ(x) are continuous functions of x, h over Cη × S. (4.54)
In particular, the partial derivatives ∂ϕ(x)/∂xi = Dei ϕ(x) exist and are
continuous functions of x over the open set Cη (ei denotes the ith basis vector
of E), which implies that ϕ is continuously differentiable over Cη .
We begin with point one of the theorem: C is now bounded. As we have
just seen, ϕ is continuous over the open neighborhood Cη of the closed,
bounded – and hence compact – set C, which implies that ϕ(C) is compact,
and hence closed. As for the minimum set of hypothesis (4.2), the only prop-
erty that does not follow immediately from (4.46) is the existence of an αM
in the last line. The best (smallest) αM is by definition
def V F
αM = sup
x0 ,x1 ∈C , x0 =x1 , t∈[0,1] x1 − x0 E

≤ sup Dh ϕ(x)F (use (4.53) left)

x∈C , h∈S
< +∞ (use (4.54) left and C × S compact).
We prove now the second point of the theorem. The linear identifiability
hypothesis over Cη implies that, for any x ∈ Cη and h ∈ S, the numerator V
in Dh ϕ(x) = V /x1 − x0 E is nonzero, as it corresponds to points x0 , x1 ∈ Cη
such that x0 − x1 E = d(x, E \ Cη ) > 0, c.f. (4.51) (4.52). The best (largest)
αm that satisfies (4.40) is, by definition,
def V F
αm = inf (4.55)
x0 ,x1 ∈C , x0 =x1 , t∈[0,1] x1 − x0 E

≥ inf Dh ϕ(x)F (use (4.53) left)

x∈C , h∈S
> 0 (use (4.54) left and Dh ϕ(x) = 0 over C ×S compact),
182 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

which proves the linear stability of x over C. As for the finite curvature of the
problem, the only thing in Definition 4.2.1 that does not follows immediately
from the FD minimum set of hypothesis (4.46) is the inequality (4.13). The
proof is similar: the best (smallest) 1/R that satisfies (4.13) is by definition
def AF
1/R = sup (4.56)
x0 ,x1 ∈C , x0 =x1 , t∈[0,1] V 2F
Dh,h
2
ϕ(x)
≤ sup (use (4.53) left and right)
x∈C , h∈S Dh ϕ(x)
2

< +∞ (use (4.54) and Dh ϕ(x) = 0 over C ×S compact),

,
which proves the finite curvature property.
Finally, the last point of the theorem is simply a rewriting of the sufficient
condition for OLS-identifiability of Theorem 4.4.1.

4.6 Four Questions to Q-Wellposedness

We summarize here the successive steps required to prove Q-wellposedness
by application of Proposition 4.2.7, Theorems 4.4.1 or 4.5.1. These steps can
(and ideally should) be performed before any attempt to minimize the least
square objective function is done. Let the minimum set of hypotheses (4.2)
be satisfied, x0 , x1 be two admissible parameters, and denote by V(t) (for
velocity) and A(t) (for acceleration) the first and second derivatives with
respect to t of the forward map ϕ at xt = (1 − t)x0 + tx1 along the segment
[x0 , x1 ].
A full analysis requires the answer to four questions:

1. Linear identifiability:
does V = 0 imply x1 = x0 ?
2-a. Closedness:
is the attainable set ϕ(C) closed?
or alternatively,

2-b. Linear stability:

does there exists αm > 0 such that, for all x0 , x1 ∈ C,
1
αm x1 − x0 E ≤ 0 V (t)F dt? (4.57)
4.6. FOUR QUESTIONS TO Q-WELLPOSEDNESS 183

3. Deflection condition:
does one have, for all x0 , x1 ∈ C and t ∈ [0, 1],
1
A(t)F ≤ θ(t) V (t)F with Θ = 0 θ(t) ≤ π/2? (4.58)

4. Finite curvature:
does there exist R > 0 such that,
for all x0 , x1 ∈ C and t ∈ [0, 1], A(t)F ≤ 1/R V (t)2F ? (4.59)

The theoretical and numerical tools available (or not available...) to an-
swer these questions are described in Sect. 4.7 below.

4.6.1 Case of Finite Dimensional Parameters

When the FD minimum set of hypothesis (4.46) is satisfied and the admissible
set C is bounded, then the minimum set of hypothesis (4.2) is verified, and a
positive answer to question 1 (linear identifiability) implies a positive answer
to questions 2-a (closedness), 2-b (linear stability, for any norm on E), and
4 (finite curvature):
⎫
finite dimensional parameters ⎬
bounded parameter set =⇒ local OLS-identifiability,
⎭
linear identifiability
where local OLS-identifiability is a short name for closed attainable set, lin-
early stable parameter, finite curvature problem. Local OLS-identifiability im-
plies that x is OLS-identifiable over any convex and closed subset D of C
small enough to ensure a positive answer to question 3 (deflection condition):
*
local OLS-identifiability
=⇒ OLS-identifiability.
deflection condition Θ ≤ π/2
The deflection condition (4.58) provides an estimation of the size of D, which
ensures OLS-identifiability. It is automatically satisfied if D satisfies
π
αM diamD ≤ R, (4.60)
2
but this last estimation is usually less precise than the ones derived directly
from (4.58).
184 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

4.6.2 Case of Infinite Dimensional Parameters

When the answer to questions 1, 2-a, 3, and 4 above is positive, the NLS
problem (4.1) is Q-wellposed for all data z in the neighborhood
ϑ = {z ∈ F | d(z, ϕ(C)) < R} (4.61)
of the attainable set, for the “distance” δ on C defined by
 1
δ(x0 , x1 ) = V (t)F dt (arc length in the attainable set),
0

with the stability property

 d −1
δ(x0 , x1 ) ≤ 1 − z1 − z0 F (4.62)
R
for any z0 , z1 ∈ ϑ and d > 0 such that
z1 − z0 F + max d(zj , ϕ(C)) ≤ d < R.
j=0,1

In the case where one is able to give a positive answer to question 2-b
for some norm  · E , the closedness question 2 is also answered positively,
and one has
αm x̂1 − x̂0 E ≤ δ(x0 , x1 ), (4.63)
so that the the NLS problem (4.1) is still Q-wellposed on the same neigh-
borhood (4.61), but for the stronger norm  · E on the parameter space for
which one has been able to prove linear stability.

Remark 4.6.1 In practice, the infinite dimensional problem has to be re-

duced at some point to finite dimension for computational purpose, by search-
ing for x in a finite dimensional subspace E of E (see Chap. 3). When the
infinite dimensional parameter x happens to be linearly identifiable over C,
the results of previous Sect. 4.6.1 apply automatically to the resulting finite
dimensional problem.

4.7 Answering the Four Questions

Most often, the original inverse problem does not satisfy, rigorously or even
approximately, any of the sets of sufficient conditions for Q-wellposedness
recalled in the previous section. Because the conditions are only sufficient,
4.7. ANSWERING THE FOUR QUESTIONS 185

this does not mean that it is ill-posed, only that one cannot decide between
well- and ill-posedness. A reasonable thing to do is then to add information,
either by reducing the number of parameters (Chap. 3), or by using L.M.T.
regularization or strengthening the observation in the perspective of using
state-space regularization (Chap. 5), until the regularized problem satisfies
one set of sufficient conditions. Hence these sufficient conditions provide a
guideline for the design of a wellposed and optimizable regularized problem,
and it is only at that point that numerical attempts to minimize the least
squares objective function should be made, now with a reasonable hope of
producing meaningful results.
We discuss for the rest of this section the available tools for checking these
condition under the minimum set of hypotheses (4.2): one can try to answer
questions 1–4 either rigorously, by proving the corresponding property or
giving a counterexample, or approximately, when a proof is out of reach,
by checking numerically the property – but this becomes computationally
intensive when there are more than a few parameters.
In any cases, it is useful, whenever possible, to estimate even crudely
the linear stability constant αm , and the lower bound R to the radius of
curvature of the problem, which provide useful practical information: αm , R
appear in the stability estimate (4.62) and (4.63) of the inverse problem,
and R gives the size of the neighborhood of the attainable set on which the
inverse problem is stable, and hence provides an upper bound on the size of
the admissible measurement and model errors.
For problems of small size, one can combine formula (4.65), (4.70), and
(4.73) for the numerical determination of αm , Θ, and R with Theorem 4.5.1
and the stability property (4.44) to perform a nonlinear stability analysis,
see, for example, [80].

Checking for Linear Identifiability (Question 1)

This is the first step on the way to OLS-identifiability. For the full nonlinear
inverse problem, identifiability has received much attention in the recent
years, see, for example, [46, 34, 47]. Unluckily, the availability of a nonlinear
identifiability result does not imply automatically linear identifiability (x 
x3 is injective, but its derivative at x = 0 is not injective...), and, when
a nonlinear identifiability result is available for the problem at hand, it is
necessary to check wether the nonlinear proof goes over to the linearized
186 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

problem. This suggests that the identifiability studies should be devoted to

the linearized problems rather than to the original nonlinear problem.
However, for bilinear problems, where the inverse problem is governed by
a linear state equation with the unknown parameter being one coefficient of
the equation, and with a linear injective observation operator, identifiability
coincides with linear identifiabiliy (see Remark 4.3.2), and the existence of a
nonlinear identifiability result implies automatically linear identifiability.
If none of this works, one is left with the challenge of proving directly
linear identifiability – not necessarily easy, nor necessarily true!
When the theoretical approach fails, one has to resort to numerical de-
termination. This makes sense only after the parameter has been reduced, if
necessary, to finite dimension. The approach is similar to that of Sect. 3.2: a
nominal parameter xnom ∈ C is chosen, and the singular value decomposition
(SVD) (3.9)–(3.11) of the q × n Jacobian ϕ (xnom ) is performed. Linear iden-
tifiability is achieved in theory as soon as the number r of strictly positive
singular values μ is equal to n for all xnom ∈ C, and it fails if r < n for some
xnom ∈ C.
Of course, this is impossible to check rigorously on the computer:

• One has to limit the computation of ϕ (xnom ) and its SVD to a finite
set CN of nominal parameters, and then cross fingers that things do not
change too much away from the chosen nominal value(s). The number
of points in CN depend on the computational power one can afford, it
can be reduced to one for computationally intensive problems. . .

• Testing that a floating number in the computer is strictly positive is a

difficult task. In practice, the test μ > 0 is replaced, for the determina-
tion of r, by
μ ≥ μmin > 0,
where μmin is a threshold determined by the level of noise or error on
the data, as explained in Sect. 3.2 of Chap. 3.

Remark 4.7.1 For infinite dimensional parameters, the existence of a the-

oretical linear identifiability result does not eliminate the need of performing
an SVD analysis after the problem has been reduced to finite dimension: some
of the singular values may (and usually will...) be zero or below the threshold
μmin .
4.7. ANSWERING THE FOUR QUESTIONS 187

Checking for Closedness (Question 2-a)

The attainable set is closed as soon as one can equip the parameter space
E with a norm that makes the admissible parameter set C compact and the
forward map ϕ continuous. This holds true in particular as soon as E is finite
dimensional and C bounded.
Also, the attainable set of a linearly stable FC/LD problem is closed as
soon as the parameter space E is complete.

Checking for Linear Stability (Question 2-b)

Analytical proofs of (4.57) are rare, and when they exist, give a pessimistic
(too small) estimate of the constant αm > 0, or even show only its existence,
without any information on its value.
So one has to evaluate αm numerically, which will necessarily produce an
optimistic (too large) value. There are two ways to do that:
• One can go along the line used to check numerically for linear identifia-
bility: discretize the admissible set C into a finite set CN , and perform
the SVD decomposition (3.9)–(3.11) of ϕ (xnom ) for all xnom ∈ CN .
When linear identifiability holds, one has r = n ∀xnom ∈ CN , and αm
can be estimated by
αm = min μn,nom . (4.64)
xnom ∈CN

The estimate (4.64) is optimistic because one performs the SVD only
at a finite number of points of C, but it can also be pessimistic because
performing the SVD amounts to investigate the stability in all direc-
tions of IRn , some of them pointing outside of C when the dimension
of C is strictly smaller than n.
• One can also go along the line of the next sections on the deflection and
finite curvature conditions: one discretizes both points and directions
of the admissible set C. Figure 4.2 shows two naturals way to do that:
– On the left, a very coarse coverage where only one (or a few) point(s)
and the directions of the coordinate axis are investigated
– On the right a more comprehensive coverage, based on a discretization
of the boundary ∂C into a finite set ∂CN (circles), where each [x0 , x1 ]
interval is in turn discretized into N intervals by N + 1 points xk/N , k =
0, 1 · · · N (black dots)
188 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

x0

xk/N

x0 x1/2 x1

x1
C C

Figure 4.2: Very scarce (left) and comprehensive (right) coverage

of the points and directions in C

Then αm is estimated by

Vk−1/2 
αm = min , (4.65)
x0 ,x1 ∈∂CN , k=1···N x1 − x0 

where Vk−1/2 (see (4.13)) is the derivative of the forward map ϕ at

def
x(k−1/2)/N = (x(k−1)/N + xk/N )/2 along the vector x1 − x0 . The deriva-
tive Vk−1/2 is evaluated either analytically, if the corresponding code is
available, or numerically if not, for example,

Vk−1/2 ϕ(xk/N ) − ϕ(x(k−1)/N )

= . (4.66)
x1 − x0  xk/N − x(k−1)/N 

Checking the Deflection Condition (Question 3)

The deflection Θ of the inverse problem (4.1) can sometimes be estimated
analytically, in particular for bilinear problems (see Remark 4.3.2) with full
observation. The state equation for bilinear problems is of the form:

e(x, y) = b(x, y) + c(y) + d = 0, (4.67)

4.7. ANSWERING THE FOUR QUESTIONS 189

where b is bilinear and c linear, and by full observation property we mean that
the observation operator M is linear and satisfies, for some κM ≥ κm > 0,

κm yY ≤ MyF ≤ κM yY for all y ∈ Y. (4.68)

The velocity and acceleration are then given by

∂yt ∂ 2 yt
V (t) = M and A(t) = M ,
∂t ∂t2
where yt is the solution of the state equation (4.67) for x = xt = (1−t)x0 +tx1 .
Deriving (4.67) twice gives the equations for ∂yt /∂t and ∂ 2 yt /∂t2 :
 ∂y   ∂y 
t t
b xt , +c + b(x1 − x0 , yt ) = 0,
∂t ∂t
 ∂2y   ∂2y   ∂yt 
t t
b xt , 2 + c + 2b x1 − x 0 , = 0. (4.69)
∂t ∂t2 ∂t
The same arguments that ensure that the state equation (4.67) has a unique
solution depending continuously on c will, in general, imply that (4.69) has
a unique solution, with
 2   
 ∂ yt   ∂yt 
   
 ∂t2  ≤ κ(C)  ∂t  ,
Y Y

where the constant κ(C) can be expressed in terms of the bounds imposed
on x in C. The deflection condition is then satisfied as soon as
κM
κ(C) ≤ π/2.
κm
An example of this situation can be found in the deflection estimates of
Proposition 4.9.3 (respectively, 5.4.5) for the estimation of the diffusion co-
efficient in a two-dimensional elliptic equation with H 1 -observation (respec-
tively, H 1 -observation with adapted regularization).
As for the numerical evaluation of the deflection Θ, it can be performed
on a coverage of points and directions of C that takes into account the size
of C, as the one at the right of Fig. 4.2:

Θ= max θ(k, k  ), (4.70)

x0 ,x1 ∈∂CN , k,k  =0,1···N, k =k 
190 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

where θ(k, k  ) is the deflection between the points xk/N and xk /N :
Vk , Vk 
θ(k, k  ) = cos−1 .
Vk  Vk 
Here Vk is the velocity at xk along x1 − x0 . It can be evaluated either ana-
lytically, or numerically by
Vk = (Vk+1/2 + Vk−1/2 )/2, (4.71)
for example, where Vk−1/2 is defined in (4.66).
Formula (4.70), which is based on the definition of Θ, is the more precise
one, but is computationally expensive, as it requires the comparison of θ(k, k  )
for all couples k, k  . A formula with a simple summation in k only, which is
based on the majoration of Proposition 4.2.4 but requires an estimation of
the acceleration A, is given in the next section in (4.74).
Remark 4.7.2 The deflection condition Θ ≤ π/2 is only a sufficient condi-
tion for the more precise extended deflection condition RG > 0 on the global
radius of curvature, which characterizes s.q.c. sets (see Remark 4.2.10 and
Chap. 7).
So in a situation where one can afford to compute the deflection by (4.70),
one should rather compute, at a similar computational burden, the global
radius of curvature RG using Proposition 7.3.1

RG = max ρep
G (k, k ),
x0 ,x1 ∈∂CN , k,k  =0,1···N, k =k 


where ρepG (k, k ) is the global radius of curvature at xk/N seen from xk  /N
 
( ρG (k, k ) = ρep
ep
G (k , k)!), given by

ep  max{0, N}/D if V, V   ≥ 0,
ρG (k, k ) = (4.72)
max{0, N} if V, V   ≤ 0,
with (Proposition 7.2.6)
⎧
⎪
⎪ X = ϕ(xk/N ), X  = ϕ(xk /N ),
⎨
v = V /V , v  = V  /V  ,
⎪
⎪ N = sgn(k  − k)X  − X, v  ,
⎩
D = (1 − v, v 2 )1/2 .
When RG > 0, all stability results of this chapter apply on a neighborhood of
size RG ≤ R (Remark 4.2.10).
4.8. APPLICATION TO EXAMPLE 2 WITH H 1 OBSERVATION 191

Checking for Finite Curvature (Question 4)

Constructive proofs of finite curvature for an inverse problem (4.1), which
provide an estimation of the radius of curvature R of the inverse problem,
are very rare, the author knows of two cases only: the 1D elliptic parameter
estimation problem of Sect. 1.4 in the case of an H 1 -observation, analyzed in
Sect. 4.8, and the 2D elliptic nonlinear source estimation problem of Sect. 1.5
in the case of an H 1 -observation, analyzed in Sect. 5.2.
Hence one has to resort in practice to numerical estimation if one wants
to assign a value to R. Given one coverage of points and directions of C as
in Fig. 4.2, one can estimate R by
Vk 2
R= min , (4.73)
x0 ,x1 ∈∂CN , k=1···N −1 Ak 
and, using Proposition 4.2.4, the deflection Θ by
N −1
Ak  x(k+1/2)/N − x(k−1/2)/N 
Θ= × , (4.74)
k=1
Vk  x1 − x0 
def
where x(k−1/2)/N = (x(k−1)/N + xk/N )/2, and Vk and Ak are the velocity and
acceleration at xk/N along the vector x1 − x0 . They can be determined ana-
lytically if the codes computing the first and second directional derivative are
available. If not, they can be approximated numerically, for example, using
formula (4.71) for Vk , and for Ak ,
Ak Vk+1/2 − Vk−1/2
= , k = 1 · · · N − 1.
x1 − x0  x(k+1/2)/N − x(k−1/2)/N 
With this approximation, the formula for Θ becomes
N −1
Vk+1/2 − Vk−1/2 
Θ=2 .
k=1
Vk+1/2 + Vk−1/2 

4.8 Application to Example 2: 1D Parameter

Estimation with H 1 Observation
The estimation of the diffusion coefficient a from temperature measurements
is a long-time classic of inverse problems (see, e.g.,[4, 49, 47, 46]). We dis-
cuss in this section the OLS-identifiability of the diffusion coefficient a in a
192 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

one-dimensional elliptic problem (1.38) and (1.39) with an H 1 observation

(1.50) and (1.51), that is, when a measurement z of the space derivative
u of its solution u is available, as described in Sect. 1.4. This material is
adapted from the original paper [25], with a simpler proof and sharper es-
timations. The case of an L2 observation of u will be considered in Chap. 5
using state-space regularization. The results of this section can be gener-
alized to problems with distributed sources, at the price of more technical
complexities. We refer the interested reader to Sect. 6 of reference [25].
As explained in Sect. 1.4, we shall search for b = a−1 rather than for a.
The state equation over the domain Ω = [0, 1] is then, with the notations
(1.38),
− (b−1 uξ )ξ = gj δ(ξ − ξj ), ξ ∈ Ω, (4.75)
j∈J

with the Dirichlet boundary conditions

u(0) = 0, u(1) = 0. (4.76)

We choose as parameter space E = L2 (Ω) for b, with the admissible

parameter set C defined in (1.47). In fact we shall not be able to achieve
L2 stability on this parameter set, but only on a smaller set D ⊂ C to be
defined later.
For any b ∈ C, (4.75) and (4.76) have a unique solution u, whose space
derivative is given by
uξ = −b qb ,
where the heat flux profile qb defined by (1.41)–(1.43) depend on b through
the b-weighted mean H. 
Following (1.50), we choose F = L2 (Ω) as data space, and the estimation
of b ∈ C from a measurement z ∈ L2 (Ω) of uξ corresponds to the inversion of
the forward map ϕ defined in (1.51), which is given here by the particularly
simple closed form formula (1.44):

ϕ(b) = −b qb . (4.77)

For any b0 , b1 ∈ C and t ∈ [0, 1], define b = (1 − t)b0 + tb1 ∈ C.

Derivation of the state equations (4.75) and (4.76) or the closed form
formula (4.77) with respect to t shows that the velocity V = duξ /dt =
dϕ(b)/dt is given either by
4.8. APPLICATION TO EXAMPLE 2 WITH H 1 OBSERVATION 193

V = ηξ , where η is the solution of

 b1 − b0 
−(b−1 ηξ )ξ = − qb ξ , ξ ∈ Ω, (4.78)
b
η(0) = 0, η(1) = 0. (4.79)

or by  1
b
V = 1 (b1 − b0 )qb − (b1 − b0 )qb . (4.80)
0
b 0

Derivation of (4.80) gives then the acceleration A = d2 ϕ(b)/dt2 :

 1  1 
b (b1 − b0 ) b1 − b0
A = 2 1 (b1 − b0 )qb 0
1 − , (4.81)
b 0 b b
0 0

or equivalently, using (4.80),

  
V b1 − b0  b 1
A=2 + qb 1 (b1 − b0 ) − (b1 − b0 ) . (4.82)
b b b 0
0

Formula (4.82) will be used to estimate the deflection Θ, and (4.81) will serve
to estimate the curvature 1/R.
We check first that the minimum set of hypothesis (4.2) is satisfied: the
only part that needs a proof is the last property. Multiplication of (4.78) by
ηξ and integration over Ω = [0, 1] shows that
b1 − b0
|V |L2 = |ηξ |L2 ≤ bM | qb |L2 ,
b
and (4.2) holds with
bM
αM = qM .
bm

4.8.1 Linear Stability

Linear stability on C means that we can find αm > 0 such that

αm |b1 − b0 |L2 ≤ |ηξ |L2 = |V |L2 ∀b0 , b1 ∈ C , ∀t ∈ [0, 1]. (4.83)

With the notation

d = (b1 − b0 )/b,
194 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

(4.78) and (4.79), which define η, rewrite

− b−1 ηξ = −d qb + unknown constant, ξ ∈ Ω, (4.84)

η(0) = 0, η(1) = 0.
The flux function qb is constant between consecutive source points. If one of
these constants is zero, parameters b0 and b1 that differ only on the corre-
sponding interval produce η = 0, that is, V = 0, and linear stability cannot
hold. It is hence necessary, if we want stability, to suppose that the flux
satisfies
0 < qm ≤ qb ≤ qM ∀b (4.85)
for some 0 < qm ≤ qM (notice that (4.85) will be automatically satisfied if,
e.g., a finite number of sources and sinks of the same amplitude are located
in an alternate way on the [0, 1] interval).
But now qb−1 is finite, and so it can happen that d = (b1 − b0 )/b becomes
proportional to qb−1 . In this case, d qb is a constant, and (4.78) and (4.79)
imply η = 0. Hence V = 0, and linear stability fails once again! So we have
to prevent d qb to become close to a constant function. To quantify this, we
decompose L2 (Ω) into the direct sum of two orthogonal subspaces

L2 (Ω) = L2 (Ω)/IR ⊕ IR, (4.86)

where L2 (Ω)/IR is the quotient space of L2 by the equivalence relation “v

is equivalent to w if and only if v − w is a constant,” and IR is the space
of constant functions. Let then γ ∈ [0, π/2] be the indetermination angle
between the direction of d qb and that of constant functions (see Fig. 4.3).
The angle γ measure the “distance” of d qb to constants. We proceed in
two steps:
– First, we find a linear stability constant that depends on the angle γ
– Second, we reduce the admissible parameter set to D ⊂ C by adding
constraints that ensure that γ ≥ γm > 0 all over D
Step 1: The decomposition (4.86) is orthogonal, hence for any function
v of L2 (Ω), one has
|v|2L2 = |v|2L2/IR + |v|2IR,
where
 
|v|L2 /IR = inf |v + c|L2 = |v − v |L2 ≤ |v|L2 , |v|IR = | v |. (4.87)
c∈IR Ω Ω
4.8. APPLICATION TO EXAMPLE 2 WITH H 1 OBSERVATION 195

IR
dqb
γ

L2/IR

Figure 4.3: The decomposition L2 (Ω) = L2 (Ω)/IR ⊕ IR and the

indetermination angle γ

Then (4.84) shows that b−1 ηξ = b−1 V and d qb are equivalent, so that

|d qb |2L2 /IR = |b−1 V |2L2 /IR ≤ |b−1 V |L2 ≤ b−1

m |V |L2 . (4.88)

But, by definition of γ, one has

|d qb |L2 /IR = sin γ|d qb |L2 ,

which gives the linear stability estimate depending on γ:

bm sin γ|d qb |L2 ≤ |V |L2 . (4.89)

Step 2: To keep the angle γ away from zero, we remark that qb−1 is
constant between two source points, and is discontinuous at each active source
point, where gi = 0. So if we require that the admissible coefficients b (and
hence the vector d = (b1 − b0 /b!)) are regular, say, for example, constant,
on some neighborhood of at least one active source, then d qb , which is also
discontinuous at active source points, cannot be constant, and we can expect
that γ remains larger than some γm > 0.
So we define a smaller admissible parameter set

D = {b ∈ C | ∀j ∈ J, b(x) = bj = unknown constant on Ij }, (4.90)

196 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

where

Ij =]ξj − ηj− , ξj + ηj+ [ ∀j ∈ J (with ηj− > 0 , ηj+ > 0) (4.91)

are intervals surrounding the sources ξj . Of course, the intervals are chosen
such that

Ij ⊂ Ω =]0, 1[ ∀j ∈ J, Ij ∩ Ik = ∅ ∀j, k ∈ J, j = k. (4.92)

Let v and e be unit vectors in the directions of d qb and of constant functions

v = ±d qb /|d qb|L2 , e(ξ) = 1 ∀ξ ∈ Ω,

where the sign is chosen such that e, v ≥ 0. With this convention, the angle
γ between the directions of v and e is given by the median theorem:
1
0 ≤ cos γ = e, v = 1 − |e − v|2L2 , γ ∈ [0, π/2]. (4.93)
2
To minor γ when b0 , b1 ∈ D and t ∈ [0, 1], we have to search for a lower
bound to |e − v|2L2 :
 1
|e − v|2L2 = |e − v|2
0 
≥ |1 − v|2
j∈J Ij
& − '
= ηj (1 − dj qb (ξj− ))2 + ηj+ (1 − dj qb (ξj+ ))2 , (4.94)
j∈J

where dj = dj /|d qb|L2 ∈ IR. We have used the fact that, on each interval Ij ,
e is equal to 1, and v takes constant values dj qb (ξj− ) left from ξj and dj qb (ξj+ )
right from ξj . Taking the infimum over dj ∈ IR for each interval gives

ηj− ηj+
|e − v|2L2 ≥ g2 (4.95)
j∈J
ηj− qb (ξj− )2 + ηj+ qb (ξj+ )2 j
1 ηj− ηj+
≥ g2 (4.96)
2
qM j∈J
ηj− + ηj+ j
4.8. APPLICATION TO EXAMPLE 2 WITH H 1 OBSERVATION 197

Combining (4.95) with (4.93) shows that

∀b ∈ D, 0 ≤ cos γ ≤ cos γm ≤ 1, (4.97)
where the minimum indetermination angle γm is defined by
1 ηj− ηj+
cos γm = 1 − g 2 ≥ 0, 0 ≤ γm ≤ π/2. (4.98)
2
2qM j∈J
ηj− + ηj+ j

The sum term in the right-hand side of (4.98) depends only on the
strength of the sources (last factor), and on the disposition of the inter-
vals Ij surrounding the sources (first factor). It is strictly positive if at least
one active source is interior to the corresponding interval, that is,
∃j ∈ J such that ηj− ηj+ gj = 0. (4.99)
Combining (4.89) (step 1) and (4.97) (step 2) give then the “flux-weighted
relative stability estimate”
b − b 
 1 0 
bm sin γm  qb  ≤ |V |L2 . (4.100)
b L2

Thus we have proved the

Proposition 4.8.1 Let hypothesis (4.85) and (4.99) hold. The estimation
of b ∈ L2 from uξ ∈ L2 is then linearly stable (inequality (4.83)) on the
admissible parameter set D defined by (4.90)–(4.92), with a stability constant
given by
bm
αm = qm sin γm > 0, with γm defined in (4.98). (4.101)
bM
Remark 4.8.2 It is possible to obtain numerically a better (i.e., larger) es-
timation of γm – and hence of αm . Equation (4.95) can be rewritten, using
the definition (1.41) of qb ,
|e − v|2L2 ≥ g(Hb ).
where g : IR  IR and Hb ∈ IR are defined by (see (1.43))
ηj− ηj+
g(h) = g2,
j∈J
ηj− (h − H(ξj− ))2 + ηj+ (h − H(ξj+ ))2 j
1
b H(ξ) dξ
Hb = 0  1 ∈ IR.
0
b dξ
198 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Let Ω+ (resp. Ω− ) be the subsets of Ω = [0, 1], where H (defined in (1.43))

is positive (respectively, negative). Then the b-weighted mean Hb satisfies

Hb,m ≤ Hb ≤ Hb,M ,

where Hb,m and Hb,M are defined by

+   ,%& '
Hb,m = bm H(ξ) dξ − bM H(ξ) dξ bm |Ω+ | + bM |Ω− | ,
Ω+ −
+  Ω ,%& '
Hb,M = bM H(ξ) dξ − bm H(ξ) dξ bM |Ω+ | + bm |Ω− | .
Ω+ Ω−

A better estimation 
γm of γm is then
1
cos γm = 1 − inf g(h) ≤ cos γm ,
2 Hb,m ≤h≤Hb,M

where the infimum can be determined numerically, for example, by plotting

the function g.

4.8.2 Deflection Estimate

We start from (4.82). It can be rewritten, with the notations c = b1 − b0 and
d = (b1 − b0 )/b,
V  + b − 1 b  1 
 1 ,

A=2 + d qb 1 0
c− c− c .
b b 0 0
0

This gives, using the stability estimate (4.100) and the property (4.87) of the
norm in L2 /IR,
 *
2  1  |b|L2 /IR
|A|L2 ≤ 1+ |V |L2 (bM − bm ) + |c|L2 /IR . (4.102)
bm sin γm bm

Let us denote by max ∈ J the index of the source for which |Ij | = ηj− + ηj+
is maximum, and by cmax and dmax the constant values of c and d on Imax .
Using (4.87) again gives, as c − cmax and d − dmax vanish over Imax ,

|c|L2 /IR ≤ |c − cmax |L2 ≤ (1 − |Imax |)1/2 2(bM − bm ),

|d|L2/IR ≤ |d − dmax |L2 ≤ (1 − |Imax |)1/2 (bM − bm ).
4.8. APPLICATION TO EXAMPLE 2 WITH H 1 OBSERVATION 199

Hence (4.102) becomes

2  1  + bM − bm ,
|A|L ≤
2 1+ |V |L (1 − |Imax |) (bM − bm ) 2 +
2
1/2
,
bm sin γm bm
 b 2  1 
M
|A|L2 ≤ 2 −1 1+ (1 − |Imax |)1/2 |V |L2 .
bm sin γm
Thus we have proved the
Proposition 4.8.3 Let hypothesis (4.85) and (4.99) hold. The deflection Θ
for the identification of b ∈ L2 ([0, 1]) in the admissible parameter set D,
defined in (4.90), from a measurement of uξ ∈ L2 is
 b 2  1 
M
Θ=2 −1 1+ (1 − |Imax |)1/2 . (4.103)
bm sin γm
When |Imax | → 1, that is, when the coefficients tend to become constant over
Ω, we see that
– Last factor goes to zero
– sin γ increases – likely to one –
so that Θ → 0, which corresponds to the fact that the problem becomes
“more linear.”

4.8.3 Curvature Estimate

Equation (4.82) we have used for the deflection estimate is no more suited
for the estimation of the curvature: we want to major |A|L2 by |V |2L2 , but the
right-hand side of (4.82) contains the term V (b1 − b0 ), and there is no hope
−1
of majorating |V (b1 − b0 )|L2 by |V |L2 |b1 − b0 |L2 ≤ αm |V |2L2 !
So we start instead from (4.81). The stability estimate (4.100) gives
 1 
  bM
 (b1 − b0 )qb  ≤ |(b1 − b0 )qb |L1 ≤ |(b1 − b0 )qb |L2 ≤ |V |L2 ,
0 bm sin γm
b − b 
 1 0 1
  2 ≤ |V |L2 ,
b L bm qm sin γm
which, combined with (4.81), gives
2  b 2 + b ,
M M
|A|L2 ≤ + 1 |V |2L2 .
bm qm sin2 γm bm bm
200 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Proposition 4.8.4 (Same hypotheses as Proposition 4.8.3). The estimation

of b ∈ L2 ([0, 1]) in the admissible parameter set D defined by (4.90)–(4.92)
from a measurement of uξ ∈ L2 is a FC problem. Its radius of curvature is
given by
1 (bm /bM )3
R = bm qm sin2 γm > 0. (4.104)
2 1 + bm /bM
The question of the behavior of the curvature of the problem when |Imax | → 1
is open. One could expect that it tends towards zero, but all attempts to prove
it have failed....

4.8.4 Conclusion: OLS-Identifiability

We can now combine the above results with Theorem 4.4.1: the parameter
b in (4.75) and (4.76) is OLS-Identifiable in L2 ([0, 1], on the admissible set
D defined in (4.90), from a measurement of uξ in L2 ([0, 1], as soon as the
deflection (4.103) satisfies Θ ≤ π/2. The size of the wellposedness neighbor-
hood of the attainable set is then R given by (4.104), and the linear stability
constant αm is given by (4.101).

4.9 Application to Example 4: 2D Parameter

Estimation, with H1 Observation
We discuss in this section the OLS-identifiability of the diffusion coefficient
a in the two-dimensional elliptic problem described in Sect. 1.6 in the case
of an H 1 observation, that is, when a measurement z of the gradient ∇u of
its solution u is available. This material is adapted from the original paper
[30]. The case of an L2 observation of u will be considered in Chap. 5 using
state-space regularization.
There are no distributed source terms inside Ω or on the Neumann bound-
ary ∂ΩN in this example. All sources or sinks are modeled by holes with
boundaries ∂Ωi , i = 1, . . . , N, equipped with a given injection or production
rate condition (see (1.64)).
When the diffusion coefficient is only constrained to stay within a lower
and an upper bound, as it was the case for the set C defined in (1.66), the
coefficient is allowed to oscillate wildly, and the homogeneization theory [71]
shows that the least squares function does not in general attain its minimum
over C.
4.9. APPLICATION TO EXAMPLE 4 WITH H 1 OBSERVATION 201

So we regularize the problem a first time by adding the information that

we are only seeking
• . . . smooth coefficients: This will be achieved by requiring that the dif-
fusion coefficient belongs to the space C 0,1 (Ω) of functions which are
bounded and uniformly Lipschitz continuous over Ω
• . . . with limited oscillations: this can be implemented by requiring that
the Lipschitz constants of all coefficients a of C are uniformly bounded
by bM ≥ 0
• . . . and which take (unknown but) constant values on the source or sink
boundaries ∂Ωi : this is the 2D generalization of the hypothesis that a is
constant on some neighborhood of each Dirac source term, which was
required in the 1D case to ensure OLS-identifiability (see Sect. 4.8 above
and [25]). It is also physically not too restrictive, as one can assume
that the sizes of the ∂Ωi ’s, which model the well boundaries, are small
compared to the size of Ω and to the scale at which the coefficient a is
expected to vary.
This leads us to define a vector space
E = {a ∈ C 0,1 (Ω) : a|∂Ωi = unknown constant ai = i = 1, · · · , N}, (4.105)
and to replace the admissible set C defined in (1.66) by the smaller one
C = {a∈E | am ≤ a(ξ) ≤ aM ∀ξ ∈ Ω, (4.106)
|a(ξ1) − a(ξ0 )| ≤ bM ξ1 − ξ0  ∀ξ0 , ξ1 ∈ Ω },
where am , aM , and bM are given numbers that satisfy
aM ≥ am > 0 and bM ≥ 0, (4.107)
(this corresponds to the technique of Regularization by size reduction
of C of Sect. 1.3.4).
To be able to infer the smoothness properties (4.109) below the solution
u of (1.64) from the smoothness (4.105) and (4.106) of the coefficient a, we
will also to suppose that the domain Ω itself is smooth, in the sense that
its boundary ∂Ω is smooth, and that the subparts ∂ΩD , ∂ΩN , and ∂Ωi ,
i = 1, · · · , N do not intersect,

Ω ⊂ IR2 has a C 1,1 boundary ∂Ω,
(4.108)
∂ΩN , ∂ΩD and ∂Ωi , i = 1, · · · , N are pairwise disjoint
202 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

With these hypotheses, the regularity theorem in [76], page 180, implies
that {|ua |W 2,p : a ∈ C} is bounded for any p > 2. Since W 2,p (Ω) is contin-
uously embedded into C 1 (Ω) for every p > 2, there exists uM and γM such
that
|ua |L∞ (Ω) ≤ uM , |∇ua |L∞ (Ω) ≤ γM for all a ∈ C. (4.109)
The results in this section are derived under the assumption (1.65) that
the Dirichlet condition u = 0 holds on a nonvoid part ∂ΩD of the boundary
∂Ω, but all results can be extended to the case where meas (∂ΩD ) = 0,
see [30].
We have now to choose the Banach parameter space E for which one
hopes to prove the stability of the inverse problem. As we will end up in this
section by further reducing the problem to finite dimension, all norms will
be equivalent, and we shall choose simply for E the space that makes the
a  ua mapping naturally regular:

E = C 0 (Ω) with the norm vC 0 = sup |u(x)|. (4.110)

x∈Ω

The admissible set C defined in (4.106) is clearly a closed and convex subset
of E – but with a void interior!
To any a ∈ C, we associate the solution ua of the state equation (1.64),
which we write here for convenience in its variational form. We incorporate
for this the boundary conditions of lower order in the state-space Y :

Y = {v ∈ H 1 (Ω) : v|∂ΩD = 0, v|∂Ωi = vi = const , i = 1, · · · , N}
vY = |∇v|IL2 ,

(compare to (1.67)), and define ua as the solution of

 N
find u ∈ Y such that a∇u∇v = Qi vi for all v ∈ Y, (4.111)
Ω i=1

where the production or injection rates Qi ∈ IR, i = 1, · · · , N, through the

source or sink boundaries ∂Ωi are given.
The observation operator corresponding to the H 1 observation considered
in this section is (cf. (1.70))

M : w ∈ Y  ∇w ∈ F, where:
F = IL2 (Ω) is equipped with the norm vF = |v|IL2 (Ω) .
4.9. APPLICATION TO EXAMPLE 4 WITH H 1 OBSERVATION 203

Hence we are concerned with the inversion of the mapping:

ϕ : a ∈ C ⊂ E = C 0 (Ω)  ∇ua ∈ F = IL2 (Ω) (4.112)
in the least-squares sense
â minimizes 12 |∇ua − z|2IL2 over C, (4.113)
where z ∈ IL2 (Ω) is a given observation.
We analyze now the Q-wellposedness of this problem using the approach
of Sect. 4.4 summarized in Sect. 4.6. We have to evaluate the velocity V (t)
and acceleration A(t) along the curve t ∈ [0, 1]  ∇ua(t) ∈ IL2 (Ω) associated
to couples a0 , a1 of parameters of C by (4.11) and (4.13). These curves stay
by construction in the range of the mapping a  ∇ua . One finds
V (t)F = |∇η(t)|IL2 , A(t)F = |∇ζ(t)|IL2 ,
where η(t) and ζ(t) denote the first and second derivatives of ua(t) with respect
to t. The equations for η(t) and ζ(t) are simply obtained by derivating one
and two times with respect to t the state equation (4.111) written at point
a = (1 − t)a0 + ta1 :
 
a∇η · ∇v = − (a1 − a0 )∇ua · ∇v, for all v ∈ Y, (4.114)
Ω Ω
 
a∇ζ · ∇v = −2 (a1 − a0 )∇η · ∇v, for all v ∈ Y. (4.115)
Ω Ω
We investigate first the linear identifiability of a (Definition 4.3.1).
Lemma 4.9.1 For a0 , a1 ∈ C and t ∈ [0, 1], define a = (1 − t)a0 + ta1 ∈ C,
h = a1 − a0 ∈ E, u = ua , and v = hu
a
. Then v ∈ Y and
  N
1 h2 h2i
h∇u · ∇v = |∇u|2 + ui Qi .
Ω 2 Ω a i=1
a2i

Proof. Since C ⊂ E ⊂ C 0,1 (Ω), one has v = hu

a
∈ H 1 (Ω). Moreover, v satisfies
the boundary conditions defining Y and hence v ∈ Y . It follows that
  
h2
h∇u · ∇v = a
|∇u| + 2 ua ∇u · ∇h2
2 1
Ω Ω Ω

h2 u
− a2
∇a · ∇u.
Ω
204 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

Integrating by parts the second term on the right hand side implies
   u  1 N
1 h2 1 u h2i
h∇u · ∇v = |∇u|2 − 2
h Δu + 2 ∇a · ∇u + ui Qi ,
Ω 2 Ω a 2 Ω a a 2 i=1
a2i

which, utilizing −aΔu − ∇a · ∇u = 0, gives the desired result.

Proposition 4.9.2 Let notations and hypothesis (1.65) and (4.105) through
(4.107) hold. Then a is linearly identifiable over C as soon as

Qi , i = 1, · · · , N are not all zero, (4.116)

and
|∂Ωi |, i = 1, · · · , N, are sufficiently small. (4.117)

Proof. Let a0 , a1 ∈ C and t ∈ [0, 1] be such that the velocity η defined by

(4.114) is zero. Linear identifiability will be proved if we show that h =
a1 − a0 = 0. By construction, h satisfies

h ∇ua · ∇v = 0 for all v ∈ Y,
Ω

where ua is the solution of (4.111) for a = (1−t)a0 +ta1 . Lemma 4.9.1 implies
then
 N
1 h2 h2i
|∇ua |2 + 2
uiQi = 0. (4.118)
2 Ωa i=1
ai

We argue first that the second term in 4.118 can be made positive us-
ing (4.118). Suppose that ∂Ωi surrounds for each i = 1, · · · , N, a fixed
source/sink location xi . If |∂Ωi | → 0, for all i = 1, · · · , N, the solution ua con-
verges towards the weak solution associated to a right-hand side with Dirac

N
source term Qi δ(x − xi ), which is singular at xi . Hence ua |∂Ωi = ua,i → ∞
i=1
if Qi > 0 and ua |∂Ωi → −∞ if Qi < 0. Since C is compact in E by the Ascoli
theorem, and a → ua,i is continuous on E, we conclude that, when |∂Ωi |
satisfies (4.117), the solution ua satisfies

ua,i Qi ≥ 0 for i = 1, · · · , N, and all a ∈ C,

so that (4.118) implies h∇u = 0 a.e. on Ω.

4.9. APPLICATION TO EXAMPLE 4 WITH H 1 OBSERVATION 205

We argue now that ∇u(a) cannot vanish on a set I of positive measure. Let
γ denote a curve in Ω connecting the inner boundaries ∂Ωi to ∂ΩD ∪∂ΩN , such
that Ω \ γ is simply connected and meas γ = 0. Then Iγ = (Ω \ γ) ∩ I satisfies
meas Iγ > 0. From [5], Theorem 2.1, and Remark, it follows that either ua is
constant on Ω \ γ and hence on Ω or ua has only isolated critical points, that
is, points z satisfying ∇u(z) = 0. But ua cannot equal a constant over Ω as
this violates the boundary conditions at the wells ∂Ωi at which Qi = 0. On
the other hand, the number of isolated critical points in Iγ can be at most
countable, and hence meas Iγ = 0. Consequently, meas{x : ∇ua (x) = 0} = 0.
Hence h = 0 a.e. in Ω, and a is linearly identifiable over C, which ends
the proof.
We turn now to the deflection condition (4.16):
Proposition 4.9.3 Let notations and hypotheses (1.65) and (4.105) through
(4.107) hold. Then the deflection condition Θ ≤ π/2 is satisfied for problem
(4.113) as soon as
π
aM − am ≤ am . (4.119)
4
Proof: Taking v = ζ in (4.115) gives, using (4.107), the Cauchy–Schwarz
inequality and (4.112):

am ∇ζF ≤ 2a1 − a0 C 0 ∇ηF (4.120)

≤ 2(aM − am )∇ηF ,

that is, using (4.119),

π
∇ζF ≤ ∇ηF .
2
This shows that (4.25) is satisfied, and Corollary 4.2.5 ensures that the de-
flection condition is satisfied.
The next step toward OLS-identifiability would be to prove a linear sta-
bility property (4.39) or (4.57) and a finite curvature condition (4.13) or
(4.59), as this would imply OLS-identifiability using Theorem 4.4.1. How-
ever, there are some hints that already linear stability does not hold for the
infinite dimensional set C defined in (4.106) (see Remark 5.4.1 in Sect. 5.4
below). Hence the problem needs once again to be regularized. We describe in
Sect. 5.4 how to regularize this problem in a way that is specifically adapted
to its nature. But we conclude the present section by the simplest regular-
ization, which is to add the information that we search for the parameter in
a finite dimensional subspace of E. So we define
206 CHAPTER 4. OLS-IDENTIFIABILITY AND Q-WELLPOSED

E = finite dimensional subspace of E,
(4.121)
C = C ∩E,

(one can, e.g., construct E using finite elements or splines). The result follows
then immediately from Theorem 4.5.1:

Theorem 4.9.4 Let notations and hypothesis (1.65) and (4.105) through
(4.110) hold, as well as (4.116), (4.117), and (4.121). Then

1. a is linearly stable over C :

αm a0 − a1 C 0 (Ω) ≤ |∇u0 − ∇u1 |IL2 (Ω) ∀a0 , a1 ∈ C ,

where the constant αm is given by

|∇u|IL2 (Ω)
αm = inf >0
a0 ,a1 ∈C , a0 =a1 , t∈[0,1] a1 − a0 C 0 (Ω)

2. The estimation of a ∈ C from ∇u ∈ IL2 (Ω) is a finite curvature

problem, with a curvature
1 2
= < +∞ (4.122)
R αm am

3. If moreover the admissible set C satisfies aM /am ≤ 1 + π/4 (condition

(4.119)), the diffusion coefficient a is OLS-identifiable in the finite
dimensional subset C of C from a measurement z ∈ ϑ of ∇u ∈ IL2 (Ω),
where ϑ is the neighborhood of the attainable set defined by

ϑ = {z ∈ IL2 (Ω) | inf |z − ∇ua |IL2 (Ω) < R},

a∈C

and the Lipschitz constant of the z  â mapping is given by (4.43) and

(4.44) with x replaced by a.

Proof. To apply Theorem 4.5.1, we define  = am /2 > 0, and, according to

(4.47) and (4.48), Cη by

Cη = { a ∈ E | am −  < a(x) < aM +  ∀x ∈ Ω,

|a(x1 ) − a(x0 )| < (bM + )x1 − x0  ∀x0 , x1 ∈ Ω }.
4.9. APPLICATION TO EXAMPLE 4 WITH H 1 OBSERVATION 207

The FD-minimum set of hypotheses (4.46) is then verified, the set C – and
hence C – is obviously bounded, and Proposition 4.9.2 applied to Cη instead
of C shows that a is linearly identifiable over Cη – and hence over C η . Then
points 1 and 2 of Theorem 4.9.4 follow immediately from points 1 and 2 of
Theorem 4.5.1, and point 3 follows from Proposition 4.9.3 and point 3 of
Theorem 4.5.1.

Remark 4.9.5 When E is one member of a family of embedded subspace

filling out E, the stability constant αm and , following (4.122), the radius
of curvature R decrease – most likely to zero – when the dimension of E
goes to infinity, that is, when the discretization of the diffusion coefficient is
refined. However, the size condition (4.119) which ensures that the estimation
problem is Q-wellposed remains unchanged.
To figure out the shape of the attainable set for the diffusion problem, one
can take a look at the attainable sets ϕ(C 0 ) and ϕ(C 1 ) at scale 0 and 1 for
the prototype “nicely nonlinear” example of Fig. 3.3, Sect. 3.6: they both have
the same size, are obviously s.q.c., with the second one having a stronger
curvature and hence a much smaller neighborhood ϑ on which the projection
is Q-wellposed.
This leads us to conjecture that the a  ∇u map under study in the
present section is also a “nicely nonlinear” problem in the sense of Def-
inition 3.6.1: directions of coarse perturbations should correspond to large
sensibilities (i.e., large velocities V ) and small nonlinearities (i.e., small
accelerations A), whereas directions of fine perturbations should correspond
to small sensibilities and large nonlinearities. This “nice nonlinearity” prop-
erty has been shown to hold in [57] for the estimation of a one-dimensional
diffusion parameter, but further research is needed for the present two dimen-
sional case.
Another hint in favor of this conjecture is the recognized ability of mul-
tiscale parameterizations to overcome stationary points while searching for
diffusion coefficients, which is shared with “nicely nonlinear” problem (see
paragraph 4 of Sect. 3.6.3 in Chap. 3).
Hence Fig. 3.2 (bottom) is a good mental representation of the attainable
set for the diffusion inverse problem .
Chapter 5

Regularization of Nonlinear
Least Squares Problems

We consider in this chapter various approaches for the regularization of the

general NLS problem (1.10), recalled here for convenience:
1
x̂ minimizes J(x) = ϕ(x) − z2F over C. (5.1)
2
and we suppose throughout the chapter that it satisfies the minimum set of
hypothesis (1.12) or (4.2).
We develop three of the five approaches described in Sect. 1.3.4 of the
introduction: Levenberg–Marquardt–Tychonov (LMT), state-space, and
adapted regularization. The two remaining approaches, regularization by
parameterization and regularization by size reduction of the admissible
parameter set, have been already addressed in Chaps. 3 and 4, respectively.

5.1 Levenberg–Marquardt–Tychonov (LMT)

Regularization
The LMT regularization of problem (5.1) is, as announced in (1.25),
2
x̂ minimizes J (x) = J(x) + x − x0 2E over C, (5.2)
2
where  > 0 denotes the regularization parameter and x0 represents an
a-priori estimate to a solution of (5.1). The paper by Levenberg [54] goes
G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 209
and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 5, c Springer Science+Business Media B.V. 2009
210 CHAPTER 5. REGULARIZATION

back to the forties, and that of Marquardt [62] to the sixties. The approach
was popularized in the seventies by Tikhonov and the Russian school [75, 63].
In practice, the available data are always corrupted by noise. So we retain
the letter z to denote the noise free data, and suppose that a sequence zn of
noisy measurement of increasing quality is available:

|zn − z| ≤ δn , with δn → 0. (5.3)

One associates to these data a sequence of regularization parameters:

n > 0, with n → 0, (5.4)

and the sequence of regularized problems:

1 2
x̂n minimizes Jn (x) = ϕ(x) − zn 2F + n x − x0 2E over C. (5.5)
2 2
• For the class of linear problems, where ϕ(x) = Ax with A ∈ L(E, F ),
the theory is rather well developed, see, for example, the monographs
by Baumeister, Groetsch, Louis, and Morozov [8, 43, 59, 63], and the
papers [65, 66, 27]. The main results are the following:

1. The regularized problem (5.2) is Q-wellposed as soon as  > 0,

hence problems (5.5) are Q-wellposed for all n
2. When the original problem (5.1) admits a solution, x̂n converges
to the x0 -minimum-norm solution x̂ when  → 0, provided the
regularization parameter goes to zero slower than the noise on the
data
3. When the x0 -minimum-norm solution x̂ is regular, convergence
rates for x̂n − x̂ and of Ax̂n − Ax̂ are available, provided the reg-
ularization parameter goes to zero as the square root of the noise
on the data

We give in Sect. 5.1.1 a direct hard analysis proof of these results, which
will serve as guideline for the nonlinear case.

• The results of the linear case are generalized completely in Sect. 5.1.2 to
the class of finite curvature/limited deflection (FC/LD) problems, pro-
vided that the true data z is close enough to the attainable set ϕ(C).
5.1. LMT REGULARIZATION 211

This class contains NLS problems that satisfy both the finite curva-
ture property of Definition 4.2.1 and the deflection condition (4.25) of
Corollary 4.2.5. We follow [28] for the proof, but use sharper estima-
tions that allow to obtain the convergence results even for unattainable
data and without the need for the a-priori guess x0 to be close to the
minimal norm solution x̂.
• For general nonlinear problems, however, where the parameter  out-
put mapping ϕ exhibits no interesting mathematical properties except
regularity,  > 0 does not necessarily ensure wellposedness of the regu-
larized problem (5.2) any more, in particular for the small values of .
We give first in Sect. 5.1.3 an estimation of a minimum value min > 0
of , which restores Q-wellposedness of (5.2) [18]. Then we study the
convergence of a sequence of (non-necessarily unique) minimizers x̂n of
(5.5) when the data z is attainable and the a-priori guess x0 is close
enough to a minimum-norm solution x̂ of (5.1) [27]. Convergence results
for unconstrained nonlinear problems are also available in [37, 67, 45].

5.1.1 Linear Problems

We follow in this section the presentation of [27]. We consider here the case
where
ϕ(x) = Ax, A ∈ L(E, F ), C ⊂ F closed, convex and z ∈ F, (5.6)
so that the minimum set of hypothesis (1.12) is satisfied. We define
ẑ = projection of z on A(C). (5.7)

The x0 -Minimum-Norm Solution

When the unregularized problem (5.1) admits solution(s), one has
ẑ ∈ A(C) (always true when A(C) is closed !), (5.8)
and the solution set of problem (5.1) is then
X = {x ∈ E : Ax = ẑ} ∩ C closed, convex . (5.9)
The choice of an a-priori guess
x0 ∈ E (5.10)
212 CHAPTER 5. REGULARIZATION

implies the selection of a specific solution x̂ in X. It is the element closest to

x0 , and is found by solving
1
x̂ ∈ X minimizes x − x0 2E over X. (5.11)
2
Clearly (5.11) has a unique solution, which will be referred to as the x0 -
minimum-norm solution of (5.1).
To exhibit some of its properties, we shall utilize the following notions.
For any convex set K ⊂ E and x ∈ K, the tangent cone and negative polar
cone to K at x are defined by

T (K, x) = {y ∈ E : ∃xn ∈ K, λn > 0 with λn (xn − x) → y}, (5.12)

T (K, x)− = {y ∈ E : y, ȳ ≤ 0 for all ȳ ∈ T (K, x)}. (5.13)

T (K, x) and T (K, x)− are closed convex cones. The tangent cone to X sat-
isfies

Lemma 5.1.1

∀x ∈ X, T (X, x) ⊂ Ker A ∩ T (C, x).

Proof. Let y ∈ T (X, x), and {xn } in X, {λn } with λn > 0 be sequences such
that y = lim λn (xn − x). It follows that y ∈ KerA and, since xn ∈ X ⊂ C,
we also have that y ∈ T (C, x).
Unluckily, the converse inclusion is not true in general: for example, if C
is a closed ball and X is reduced to one single point {x}, then T (X, x) = {0}
and Ker A ∩ T (C, x) = Ker A, which do not coincide as soon as Ker A is not
trivial. So we make the following definition:

Definition 5.1.2 An element x ∈ X is said to be qualified if

T (X, x) = Ker A ∩ T (C, x). (5.14)

Lemma 5.1.3 Let (5.1) admit a solution and x be identifiable over C (Def-
inition 1.3.1). Then the solution set X contains one single element x̂, and x̂
is qualified.

Proof. The identifiability property implies (1) that X = {x̂}, so that

T (X, x̂) = {0}, and (2) that Ker A = {0} and (5.14) is proved.
5.1. LMT REGULARIZATION 213

For the case of linear constraints, one has the following result:
Lemma 5.1.4 Let C be defined by a finite number NC of linear constraints:
& def '
C = x ∈ E : Mi x ≤ bi , i ∈ I = {1 . . . NC } ,
where Mi are bounded linear functionals on E and bi ∈ IR. Then all points x
of X are qualified.
Proof. Step 1: for any x̃ ∈ C, we prove that

K = {y ∈ E : ∃x ∈ C, λ > 0 such that y = λ(x − x̃}

= ∪λ>0 λ(C − x̃)

coincides with T (C, x̃). By definition of T (C, x̃), one has T (C, x̃) = K, and
hence it suffices to show that K is closed. Let yn ∈ K and y ∈ E be such that
yn → y. As yn ∈ K, there exist λn > 0 and xn ∈ C such that yn = λn (xn − x̃).
Let us denote by I(x̃) the set of active indices at x̃: I(x̃) = {i ∈ I : Mi x̃ =
bi }. Then we find Mi yn = λn (Mi xn − Mi x̃) = λn (Mi xn − bi ) ≤ 0 for all
i ∈ I(x̃) and n = 1, 2, . . . . Hence Mi y ≤ 0 for all i ∈ I(x̃). Next we choose
λ > 0 small enough so that Mi x̃ + λMi y ≤ bi for all i ∈ / I(x̃). It is simple to
check that x̃ + λy ∈ C and hence y ∈ K and K is closed.
Step 2: we prove that Ker A ∩ K ⊂ T (X, x̃), for any x̃ ∈ X. Let y ∈
Ker A ∩ K be given. Then y = λ(x − x̃), where λ > 0 and x ∈ C, and
A y = 0. Hence A x = A x̃, so that x ∈ X and y = λ(x − x̃) ∈ T (X, x̃).
The following property will be useful in the convergence study of the
regularized solutions xn to the x0 -minimum-norm solution x̂:
Lemma 5.1.5 Hypothesis and notations (5.6) through (5.11).
1. If the x0 -minimum-norm x̂ solution of (5.1) is qualified, then

x0 − x̂ ∈ Rg A∗ + T (C, x̂)− . (5.15)

2. Conversely, if (5.15) holds, then x̂ is the x0 -minimum-norm solution.

Remark that in the case of an unconstrained problem, where C = E =
T (C, x̂), one has T (C, x̂)− = {0}, and property (5.15) reduces to

x0 − x̂ ∈ Rg A∗ = Ker A⊥ ,

which expresses the fact that x0 − x̂ is orthogonal to Ker A.

214 CHAPTER 5. REGULARIZATION

Proof. The Euler condition applied to problem (5.11) shows that the x0 -
minimum-norm solution x̂ satisfies

x0 − x̂, x − x̂ ≤ 0 ∀x ∈ X,

and, using the definition (5.12) of tangent cones

x0 − x̂, y ≤ 0 ∀y ∈ T (X, x̂),

or in terms of the negative polar cone

x0 − x̂ ∈ T (X, x̂)− . (5.16)

The qualification hypothesis for x̂ implies that T (X, x̂) is given by (5.14),
and using a property of polar cones

T (X, x̂)− = Ker A− + T (C, x̂)− .

But Ker A− = Rg A∗ , and (5.16) becomes

x0 − x̂ ∈ Rg A∗ + T (C, x̂)− .

Let η > 0 be given. We can first find x1 ∈ T (C, x̂)− and x2 ∈ Rg A∗ such
that |x0 − x̂ − x1 − x2 | ≤ η/2. Then we can find x3 ∈ Rg A∗ such that
|x3 − x2 | ≤ η/2. Hence we obtain

|x0 − x̂ − x1 − x3 | ≤ η,

which proves (5.15).

The second part of the lemma is proved by reversing the order of the pre-
vious calculations, using lemma 5.1.1 instead of the qualification hypothesis.

The Regularity Condition

To obtain convergence rates for the regularized problems (5.5), we shall re-
quire the
Definition 5.1.6 The x0 -minimum-norm solution x̂ satisfies the regularity
condition if
x0 − x̂ ∈ Rg A∗ + T (C, x̂)− . (5.17)
5.1. LMT REGULARIZATION 215

The name given to condition (5.17) will be explained after Definition 5.1.14.
The regularity condition is equivalent to the existence of a Lagrange mul-
tiplier for the optimization problem (5.11), which can be rewritten as a con-
strained optimization problem:
1
x̂ ∈ C minimizes x − x0 2E over C under the constraint Ax = ẑ.
2
The Lagrangian for this problem is
1
L(x, w) = x − x0 2E + w, A x − ẑF ,
2
and ŵ is a Lagrange multiplier at x̂ if

L(x̂, ŵ) ≤ L(x, ŵ) for all x ∈ C

or equivalently, as L is convex in x
∂L
(x̂, ŵ)(x − x̂) ≥ 0 for all x ∈ C,
∂x
x̂ − x0 , x − x̂ + ŵ, A (x − x̂) ≥ 0 for all x ∈ C. (5.18)
If we define μ ∈ E by
x0 − x̂ = A∗ ŵ + μ,
we can rewrite (5.18) as

μ, x − x̂ ≤ 0 for all x ∈ C.

Hence μ ∈ T (C, x̂)− , and x̂ satisfies the regularity condition (5.17), and the
equivalence is proved.

Remark 5.1.7 An alternative formulation of the regularity condition

(5.17) is  
x̂ ∈ PC Rg A∗ + {x0 } ,
where PC denote the metric projection in E onto C. This formulation is used
in [65, 67] to obtain the convergence results of part 3 of Theorem 5.1.8 below.
216 CHAPTER 5. REGULARIZATION

Convergence of Regularized Problems

Theorem 5.1.8 Hypothesis and notations (5.3), (5.4), (5.6), (5.7), (5.9),
and (5.11).
1. Without further hypothesis on z, the unregularized problem (5.1) has
generally no solution, but all regularized problems (5.5) have a unique
solution x̂n , which satisfies, when n → 0 and δn → 0,

n x̂n → 0. (5.19)
A x̂n → ẑ. (5.20)

2. Let the data z satisfy (5.8), which means that (5.1) has solution(s).
Then if the x0 -minimum-norm solution x̂ satisfies

x0 − x̂ ∈ Rg A∗ + T (C, x̂)− , (5.21)

(e.g., if x̂ is qualified), one has, when n → 0 and δn /n → 0,

xn − x̂ → 0, (5.22)
A xn − ẑF = O(n ). (5.23)

3. If moreover x̂ satisfies the regularity condition

x0 − x̂ ∈ Rg A∗ + T (C, x̂)− , (5.24)

one has, when n → 0 and δn ∼ 2n ,

1
xn − x̂E = O(n ) = O(δn2 ), (5.25)
A xn − ẑF = O(2n ) = O(δn ). (5.26)

Proof.

Part 1 Let η > 0 be given. Using (5.7), we can find x̃ ∈ C such that

A x̃ − ẑF ≤ η/4. (5.27)

Then by definition of x̂n we have

A x̂n − zn 2F + 2n x̂n − x0 2E ≤ A x̃ − zn 2F + 2n x̃ − x0 2E .

5.1. LMT REGULARIZATION 217

To estimate A x̂n − A x̃ and x̂n − x̃, we rewrite the above inequality as

A(x̂n − x̃)2F + 2n x̂n − x̃2E ≤ A(x̂n − x̃)2F + 2n x̂n − x̃2E (5.28)
+A x̃ − zn 2F + 2n x̃ − x0 2E
−A x̂n − zn 2F − 2n x̂n − x0 2E .

Using identity a2 + b2 − (a + b)2 = −2ab, we obtain

A(x̂n − x̃)2F + 2n x̂n − x̃2E ≤ 2A(x̂n − x̃), zn − A x̃ (5.29)

+22n x̂n − x̃, x0 − x̃.

But z − ẑ is orthogonal to A(x̂n − x̃) ∈ Rg A, and so (5.29) can be rewritten as

A(x̂n − x̃)2F + 2n x̂n − x̃2E ≤ 2A(x̂n − x̃), zn − z + ẑ − A x̃ (5.30)

+22n x̂n − x̃, x0 − x̃,

and, using (5.3) and (5.27) and the Cauchy–Schwarz inequality

 η
A(x̂n − x̃)2F + 2n x̂n − x̃2E ≤ 2A(x̂n − x̃)F δn + (5.31)
4
+22n x̂n − x̃E x0 − x̃E ,

which is of the form, with obvious notations,

a2 + b2 ≤ 2aα + 2bβ.

Hence,
(a − α)2 + (b − β)2 ≤ α2 + β 2 ≤ (α + β)2
and finally 
a ≤ 2α + β,
(5.32)
b ≤ α + 2β.
Hence we deduce from (5.31) that
 η
A(x̂n − x̃)F ≤ 2 δn + + n x0 − x̃E , (5.33)
4
η
n x̂n − x̃E ≤ δn + + 2n x0 − x̃E . (5.34)
4
From the first inequality we obtain, using (5.27),
3η
Ax̂n − ẑF ≤ 2δn + + n x0 − x̃E ,
4
218 CHAPTER 5. REGULARIZATION

so that Ax̂n − ẑF ≤ η for n large enough, and (5.20) is proved. Then we
deduce from (5.34) that
η
n x̂n E ≤ n x̃E + δn + + 2n x0 − x̃E ,
4
so that n x̂n E ≤ η for n large enough, and (5.19) is proved.

Part 2 From now on we suppose that (5.1) has a solution, and hence an
x0 -minimum-norm solution x̂. Hence we can choose η = 0 and x̃ = x̂ in
(5.27), so that (5.30), (5.33), and (5.34) become

A x̂n − ẑ2F + 2n x̂n − x̂2E ≤ 2A x̂n − ẑ, zn − z (5.35)

+22n x̂n − x̂, x0 − x̂

and

A x̂n − ẑF ≤ 2δn + n x0 − x̂E , (5.36)

n x̂n − x̂E ≤ δn + 2n x0 − x̂E , (5.37)

where (5.36) proves (5.23) as now δn /n → 0. But (5.37) gives no information
on the convergence of x̂n to x̂, it is necessary for that to use hypothesis (5.21)
on x̂: let again η > 0 be given, and w ∈ F, μ ∈ T (C, x̂)− be such that

x0 − x̂ − (A∗ w + μ)F ≤ η/3.

Then we can rewrite (5.35) as

A x̂n − ẑ2F + 2n x̂n − x̂2E ≤ 2A x̂n − ẑ, zn − z

+22n x̂n − x̂, x0 − x̂ − (A∗ w + μ)
+22n x̂n − x̂, A∗ w
+22n x̂n − x̂, μ,

and, transposing A∗ in the right-hand side and using the fact that x̂n −
x̂, μ ≤ 0

A x̂n − ẑ2F + 2n x̂n − x̂2E ≤ 2A x̂n − ẑ, zn − z + 2n w

+22n x̂n − x̂, x0 − x̂ − (A∗ w + μ).
5.1. LMT REGULARIZATION 219

The Cauchy–Schwarz inequality and the formula (5.32) give now

A x̂n − ẑF ≤ 2(δn + 2n wF ), (5.38)

2η
n x̂n − x̂E ≤ δn + 2n wF + n . (5.39)
3
When δn /n → 0, the last inequality shows that x̂n − x̂E ≤ η for n large
enough, which proves (5.22).

Part 3 Now x̂ satisfies the regularity condition (5.24), and so we can choose
η = 0 in Part 2, and estimations (5.38), and (5.39) simplify to

A x̂n − ẑF ≤ 2(δn + 2n wF ), (5.40)

x̂n − x̂E ≤ δn /n + n wF , (5.41)

which prove (5.25) and (5.26).

5.1.2 Finite Curvature/Limited Deflection (FC/LD)

Problems
We consider here the application of LMT-regularization to the class of FC/LD
least squares problems introduced in Definition 4.2.2, which we summarize
here:

Definition 5.1.9 The NLS problem (5.1) is a FC/LD problem if it satisfies,

beside the minimum set of hypothesis (4.2), the finite curvature condition of
Definition 4.2.1:
⎧
⎪
⎪ there exists R > 0 such that
⎪
⎨ ∀x0 , x1 ∈ C, the curve P : t  ϕ((1 − x0 )t + tx1 ) satisfies
1 (5.42)
⎪
⎪ P ∈ W 2,∞ ([0, 1]; F ) and A(t)F ≤ V (t)2F for a.e. t ∈ [0, 1],
⎪
⎩ R
where V (t) = P (t), A(t) = P  (t),

and the deflection condition of Corollary 4.2.5

π
A(t)F ≤ V (t)F for a.e. t ∈ [0, 1] and all x0 , x1 ∈ C. (5.43)
2
220 CHAPTER 5. REGULARIZATION

A set of geometric attributes (Definition 4.2.3) of problem (5.1) is then made

of the radius of curvature R given by (5.42), the deflection Θ = π/2 given
by (5.43), and the (arc length) size L = αM diam C.
Combining (5.42) and (5.43) shows that condition (5.43) is met as soon
as L ≤ π2 R. Hence, for a FC problem, the deflection condition (5.43) can
always be satisfied by diminishing the size of the admissible set C.
FC/LD problems have two nice properties:

• The forward map ϕ is not necessarily injective, but the preimages

ϕ−1 (X) of any X ∈ ϕ(C) are always closed and convex (Proposi-
tion 4.2.8), despite the fact that ϕ itself is not linear. This will simplify
the handling of the x0 -minimum-norm solution.

• The attainable set ϕ(C) is not necessarily convex nor closed, but Propo-
sition 4.2.7 shows that the projection of z onto ϕ(C) is unique and
stable (when it exists) as soon as z belongs to the neighborhood:
+ ,
ϑ = z ∈ F | d(z, ϕ(C)) < R . (5.44)

This property will allow to handle the case of nonattainable data, pro-
vided the error level is smaller than R.

Let
ẑ = projection of z on ϕ(C). (5.45)
The unregularized problem (5.1) admits solution(s) as soon as

ẑ ∈ ϕ(C) (always true when ϕ(C) is closed !), (5.46)

and the solution set of problem (5.1) is then

X = ϕ(ẑ)−1 ∩ C,

which is closed and convex as shown in Proposition 4.2.8.

The x0 -Minimum-Norm Solution

Given an a-priori guess x0 ∈ E, the x0 -minimum-norm solution x̂ of (5.1) is
the element of the solution set X closest to x0 , that is, the unique solution of
5.1. LMT REGULARIZATION 221

the minimization problem (5.11). We shall suppose throughout this section

that
ϕ admits a Gâteaux derivative ϕ (x̂) at x̂. (5.47)
and we follow the presentation of Sect. 5.1.1 on linear problems, with the
necessary adaptations.
The situation concerning the tangent cone to X at the x0 -minimum-norm
solution x̂ is similar to that of the linear case:
Lemma 5.1.10
T (X, x̂) ⊂ Ker ϕ (x̂) ∩ T (C, x̂). (5.48)
Proof. Let y ∈ T (X, x̂), and {xn } in X, {λn } with λn > 0 be such that
y = lim λn (xn − x̂). For n given, the point xn (t) = (1 − t)x̂ + txn belongs to
X ⊂ ϕ(ẑ)−1 for 0 ≤ t ≤ 1 because of the convexity of X. The existence of a
Gâteaux derivative of ϕ at x̂ gives, for each n,

lim (ϕ(xn (t)) − ϕ(x̂)) /t = ϕ (x̂)(xn − x̂) = 0,

t→0 ! "# $
=0

and hence
0 = lim λn ϕ (x̂)(xn − x̂) = ϕ (x̂)y.
n→∞

It follows that y ∈ Ker ϕ (x̂) and, since xn ∈ X ⊂ C, we have also y ∈

T (C, x̂).
We make the following definition:
Definition 5.1.11 An element x ∈ X is said to be qualified if

T (X, x) = Ker ϕ (x) ∩ T (C, x). (5.49)

Lemma 5.1.3 generalizes to the nonlinear case as follows:

Lemma 5.1.12 Let (5.1) admit a solution, ϕ be defined over a convex open
neighborhood Cη of C (see Sect. 4.5 for an example), and x be linearly identi-
fiable over Cη (Definition 4.3.1). Then the solution set X contains one single
element x̂ and x̂ is qualified.
Proof. Let x̂ ∈ X denote the minimum-norm solution, and x denote an
arbitrary element of X. Then the curve P : t  (1 − t)x̂ + tx satisfies
P (t) = ẑ ∀t ∈ [0, 1], so that V (t) = P  (t) = 0 ∀t ∈ [0, 1], and x = x̂ using the
linear identifiability property. Hence X = {x̂}, and T (X, x̂) = {0}.
222 CHAPTER 5. REGULARIZATION

Let now y ∈ Ker ϕ (x̂) be given. One can always suppose y is small
enough so that y = x − x̂ for some x ∈ Cη . Then the velocity along the curve
P : t  (1 −t)x̂+ tx satisfies V (0) = P (0) = ϕ (0)y = 0, which again implies
x = x̂ using the linear identifiability property. Hence y = 0, which proves
that Ker ϕ (x̂) = {0}, and the lemma is proved.

Lemma 5.1.13 Hypothesis and notations (5.42), (5.43), (5.46), (5.47),

(5.10), and (5.11).

1. If the x0 -minimum-norm x̂ solution of (5.1) is qualified then

x0 − x̂ ∈ Rg ϕ (x̂)∗ + T (C, x̂)− , (5.50)

2. Conversely, if (5.50) holds, then x̂ is the x0 -minimum-norm solution.

The proof is identical to that of Lemma 5.1.5. In the case of an unconstrained

problem, where C = E = T (C, x̂), property (5.50) reduces to

x0 − x̂ ∈ Rg ϕ (x̂)∗ = (Ker ϕ (x̂))⊥ ,

which expresses the fact that x̂ is a local minimum of the distance to x0 over
X = ϕ−1 (ẑ).

The Regularity Condition

Definition 5.1.14 The x0 -minimum-norm solution x̂ is said to satisfy a
regularity condition if

x0 − x̂ ∈ Rg ϕ (x̂)∗ + T (C, x̂)− . (5.51)

The name given to condition (5.51) comes from the distributer parameter
case: when E is an infinite dimension function space, Rg ϕ (x̂)∗ can be a
dense subspace of E made of regular functions, in which case (5.51) can be
satisfied only if x0 − x̂, and hence necessarily the data z, are smooth. This is
illustrated by (5.129) in Sect. 5.2 on the LMT regularization of the nonlinear
2D source problem.
As in the linear case, we shall obtain convergence rates for the solutions
x̂n of regularized problems (5.5) when this condition is satisfied.
5.1. LMT REGULARIZATION 223

Convergence of Regularized Problems

Theorem 5.1.15 Let (5.1) be a FC/LD problem (hypothesis and notations
(4.2)) plus (5.42) through (5.45)), and let x0 ∈ E be a given a-priori guess.

1. When the data z satisfies only

+ ,
z ∈ ϑ = z ∈ F | d(z, ϕ(C)) < R , (5.52)

the unregularized problem (5.1) has generally no solution, but the regu-
larized problems (5.5) are Q-wellposed for n large enough when n → 0
and δn → 0, and

n x̂n → 0 (5.53)
ϕ(x̂n ) → ẑ. (5.54)

2. Let in addition ϕ satisfy condition (5.47), and suppose that the data z
satisfy (5.46), which means that (5.1) has a convex nonempty solution
set, and a unique x0 -minimum-norm solution x̂. If this latter satisfies

x0 − x̂ ∈ Rg ϕ (x̂)∗ + T (C, x̂)− , (5.55)

(e.g., if x̂ is qualified), one has, when n → 0 and δn /n → 0,

x̂n → x̂ , (5.56)
ϕ(x̂n ) − ẑF = O(n ). (5.57)

3. If moreover x̂ satisfies the regularity condition

x0 − x̂ ∈ Rg ϕ (x̂)∗ + T (C, x̂)− , (5.58)

one has, when n → 0 and δn ∼ 2n ,

1
x̂n − x̂E = O(n ) = O(δn2 ), (5.59)
ϕ(x̂n ) − ẑF = O(2n ) = O(δn ). (5.60)
224 CHAPTER 5. REGULARIZATION

Proof.

Part 1 The regularized problem(5.5) is equivalent to

1
x̂n minimizes  (ϕ(x), n x) −(zn , n x0 )2F ×E over C. (5.61)
2 ! "# $
def
= ϕn (x)

Along the curves Pn : t  ϕn ((1 − t)x0 + tx1 ), the velocity and acceleration

Vn (t) = (V (t), n (x1 − x0 )), An (t) = (A(t), 0),

satisfy, using (4.2),

Vn (t)F ×E ≥ (1 + 2n /αM ) V (t)F ,

2 1/2
An (t)F ×E = A(t)F ,

so that ϕn and C satisfies (5.42) and (5.43) with

π
Rn = R(1 + 2n /αM
2
) > R and Θn = Θ/(1 + 2n /αM ) <Θ≤
2 1/2
. (5.62)
2
Hence (5.61) is a finite curvature problem that satisfies the deflection condi-
tion Θ ≤ π/2 , as well as the linear stability property (Definition 4.3.4):

Vn (t)F ×E ≥ n x1 − x0  with n > 0.

Then Theorem 4.4.1 proves that (5.61) is a Q-wellposed problem as soon as

dF ×E ((zn , n x0 ), ϕn (C)) < Rn = R(1 + 2n /αM

2
). (5.63)

But

dF ×E ((zn , n x0 ), ϕn (C)) ≤ dF ×E ((z, n x0 ), ϕn (C)) + δn

& '
≤ inf (z − ϕ(x)2 + 2n x0 − x2 )1/2 + δn
x∈C
≤ inf {z − ϕ(x) + n x0 − x} + δn
x∈C
≤ inf {z − ϕ(x) + n Radx0 C} + δn
x∈C
≤ d(z, ϕ(C)) + n Radx0 C + δn .

Hence, we se that (5.63) will hold as soon as

d(z, ϕ(C)) + n Radx0 C + δn < Rn = R(1 + 2n /αM

2
),
5.1. LMT REGULARIZATION 225

which is true for n large enough as n → 0, δn → 0, and d(z, ϕ(C) < R be-
cause of (5.52). So the Q-wellposedness of the regularized problems is proved
for n large enough.
We turn now to the proof of (5.53) and (5.54). Let , d, and η be chosen
such that

 > 0, (5.64)
0 ≤ d(z, ϕ(C)) < d < R, (5.65)
0 < η ≤ d − d(z, ϕ(C)), (5.66)
1 1
η + 2η 2 d 2 ≤ , (5.67)

which is always possible because of (5.52). The points of ϑ which have a

projection on ϕ(C) are dense in ϑ, and so one can find z̃ ∈ ϑ (Fig. 5.1) such
that  η
z̃ − zF < ,
4 (5.68)
z̃ admits a projection ϕ(x̃) on ϕ(C).
Let L̃n ≥ 0 be the length of the curve P̃n : t → ϕ((1 − t)x̃ + tx̂n ), which
satisfies

ϕ(x̂n ) − ϕ(x̃)F ≤ L̃n .

Then by definition of x̂n we have

ϕ(x̂n ) − zn 2F + 2n x̂n − x0 2E ≤ ϕ(x̃) − zn 2F + 2n x̃ − x0 2E , (5.69)

z
z̃
zn

ϕ(x̂n )
ϕ(C) L̃n ϕ(x̃) ẑ ∈ ϕ(C)

Figure 5.1: Notations for the nonlinear case

226 CHAPTER 5. REGULARIZATION

and we proceed from here as in the linear case, replacing the estimation of
A(x̂n − x̃)F by that of the arc length distance L̃n of ϕ(x̃) to ϕ(x̂n ) in ϕ(C).
So we rewrite (5.69) as (compare to (5.28))
 d 2  d 2
1− L̃ + n x̂n − x̃E ≤ 1 −
2 2
L̃ + 2n x̂n − x̃2E (5.70)
R n R n
+ ϕ(x̃) − zn 2F + 2n x̃ − x0 2E
−ϕ(x̂n ) − zn 2F − 2n x̂n − x0 2E .
But the distance of z̃ to P̃n has a local minimum at t = 0 by definition
of ϕ(x̃) = projection of z̃ on ϕ(C), and we can apply the obtuse angle
lemma 6.2.9
(1 − k(z̃, P̃n ))L̃2n ≤ z̃ − ϕ(x̂n )2 − z̃ − ϕ(x̃)2 . (5.71)
But (5.68) and the triangular inequality give
z̃ − ϕ(x̂n ) ≤ z̃ − z + z − zn  + zn − ϕ(x̂n )
= z̃ − z + z − zn  + zn − ϕ(x̃) + n x̃ − x0 
≤ 2z̃ − z + 2z − zn  + z̃ − ϕ(x̃) + n x̃ − x0 
= 2z̃ − z + 2z − zn  + d(z̃, ϕ(C)) + n x̃ − x0 
≤ 3z̃ − z + 2z − zn  + d(z, ϕ(C)) + n x̃ − x0 
≤ d(z, ϕ(C)) + 3η/4 + 2δn + n x̃ − x0 
≤ d for n large enough,
z̃ − ϕ(x̃) = d(z̃, ϕ(C))
≤ z̃ − z + d(z, ϕ(C))
≤ d(z, ϕ(C)) + η/4 ≤ d.
This implies, as in the proof of Theorem 7.2.10, that k(z̃, P̃n ) ≤ d/R, so that
(5.71) gives
 d 2
1− L̃ ≤ z̃ − ϕ(x̂n )2 − z̃ − ϕ(x̃)2 for n large enough. (5.72)
R n
Substitution of (5.72) in the right-hand side of (5.70) and addition and sub-
straction of z̃ − zn 2F gives
 d 2
1− L̃ + 2n x̂n − x̃2E ≤ ϕ(x̂n ) − z̃2F − ϕ(x̃) − z̃2F
R n
+z̃ − zn 2F − z̃ − zn 2F
5.1. LMT REGULARIZATION 227

−ϕ(x̂n ) − zn 2F +  ϕ(x̃) − zn 2F

+ 2n x̂n − x̃2E
+ 2n x̃ − x0 2E
− 2n x̂n − x0 2E .

Using three times the identity a2 + b2 − (a + b)2 = −2ab gives

 d 2
1− L̃ + 2n x̂n − x̃2E ≤ 2ϕ(x̂n ) − z̃, zn − z̃F
R n
−2ϕ(x̃) − z̃, zn − z̃F
+22n x̂n − x̃, x0 − x̃E ,
and hence
 d 2
1− L̃ + 2n x̂n − x̃2E ≤ 2ϕ(x̂n ) − ϕ(x̃), zn − z̃F (5.73)
R n
+22n x̂n − x̃, x0 − x̃E .
Using the Cauchy–Schwarz inequality and the formula (5.32) gives
 d 1/2  d −1/2  η
1− L̃n ≤ 2 1 − δn + + n x0 − x̃E , (5.74)
R R 4
 d −1/2  η
n x̂n − x̃E ≤ 1 − δn + + 2n x0 − x̃E . (5.75)
R 4
Formula (5.74) gives an estimation of ϕ(x̂n ) − ϕ(x̃)F ≤ L̃n , and we need
to estimate ϕ(x̃) − ẑF to prove (5.54). We shall distinguish two cases:
• If d(z, ϕ(C)) = 0, one has z = ẑ ∈ ϕ(C), so one can choose z̃ = ϕ(x̃) ∈
ϕ(C). Then
 d  d  dη η
1− ϕ(x̃) − ẑF = 1 − z̃ − zF ≤ 1 − ≤ ,
R R R 4 4
which, combined with (5.74), gives
 d 3η  d 1/2
1− ϕ(x̂n ) − ẑF ≤ 2δn + + 1− n x0 − x̃E ,
R 4 R
3  d 1/2
≤ 2δn + + 1− n x0 − x̃E ,
4 R
≤  for n large enough,

and (5.54) is proved.

228 CHAPTER 5. REGULARIZATION

• If d(z, ϕ(C)) > 0, let us choose η such that

η
0< ≤ d(z, ϕ(C)).
4

By construction, ẑ ∈ ϕ(C), so there exists yp ∈ C, p = 1, 2 . . . such that

1
ϕ(yp) − ẑF ≤ .
p

The function dp : t ∈ [0, 1]  z̃ − ϕ((1 − t)x̃ + typ )F has a local minimum
at t = 0, as ϕ(x̃) is the projection of z̃ on ϕ(C), and satisfies moreover
η
dp (0) = d(z̃, ϕ(C)) ≤ d(z, ϕ(C)) + ≤ d,
4
1 η
dp (1) = z̃ − ϕ(yp )F ≤ + d(z, ϕ(C)) + ≤ d for n large enough.
p 4
So we can apply once again the obtuse angle lemma 6.2.9
 d
1− ϕ(x̃) − ϕ(yp )2F + z̃ − ϕ(x̃)2F ≤ z̃ − ϕ(yp )2F . (5.76)
R
But
η
z̃ − ϕ(x̃)F = d(z̃, ϕ(C)) ≥ d(z, ϕ(C)) − ≥ 0,
4
1 η
z̃ − ϕ(yp )F ≤ + d(z, ϕ(C)) + ,
p 4

which, combined to (5.76), give

 d 1 η 2  η 2
1− ϕ(x̃) − ϕ(yp )2F ≤ + d(z, ϕ(C)) + − d(z, ϕ(C)) −
R p 4 4
 1 η  1 
= + + 2d(z, ϕ(C)) ,
p 2 p
and, passing to the limit when p → ∞,
 d
1− ϕ(x̃) − ẑ2F ≤ η d(z, ϕ(C)) ≤ η d. (5.77)
R
5.1. LMT REGULARIZATION 229

Combining (5.77) with (5.74) gives

 d η  d  12
1− ϕ(x̂n ) − ẑF ≤ 2δn + + n 1 − x0 − x̃E
R 2 R
1
 d  12 1
+ η2 1 − d2
R
η 1 1
≤ + η 2 d 2 + 2δn + n x0 − x̃E
2

≤ + 2δn + n x0 − x̃E ,
2
so that (1 − d/R)ϕ(x̂n ) − ẑF ≤  for n large enough, and (5.20) is proved.
Last, (5.53) follows from (5.75) as in the linear case
η
n x̂n E ≤ n x̃E + δn + + 2n x0 − x̃E ≤ η for n large enough.
4

Part 2 We suppose now that (5.1) has a solution, and hence an x0 -

minimum-norm solution x̂. By definition of ẑ one has ϕ(x̂) = ẑ, and so we
can choose z̃ = z, η = 0, and x̃ = x̂ in (5.68). Then (5.73), (5.74), and (5.75)
become (with L̃n replaced by L̂n )
 d 2
1− L̂ + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − ϕ(x̂), zn − zF (5.78)
R n
+22n x̂n − x̂, x0 − x̂E

and
 d 1/2  d −1/2
1− L̂n ≤ 2 1 − δn + n x0 − x̂E , (5.79)
R R
 d −1/2
n x̂n − x̂E ≤ 1 − δn + 2n x0 − x̂E , (5.80)
R
where (5.79) proves (5.57) as now δn /n → 0. But (5.80) gives no information
on the convergence of x̂n to x̂, it is necessary for that to use hypothesis (5.50)
on x̂: let again η > 0 be given, and w ∈ F, μ ∈ T (C, x̂)− be such that

x0 − x̂ − (ϕ (x̂)∗ w + μ)F ≤ η/3.

230 CHAPTER 5. REGULARIZATION

Equation (5.78) becomes

 d 2
1− L̂n + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − ϕ(x̂), zn − z
R
+ 22n x̂n − x̂, x0 − x̂ − (ϕ (x̂)∗ w + μ)
+ 22n x̂n − x̂, ϕ (x̂)∗ w
+ 22n x̂n − x̂, μ,

and, transposing ϕ (x̂)∗ in the right-hand side and using the fact that x̂n −
x̂, μ ≤ 0
 d 2
1− L̂ + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − ϕ(x̂), zn − z
R n
+ 22n x̂n − x̂, x0 − x̂ − (ϕ (x̂)∗ w + μ)
+ 22n ϕ (x̂)(x̂n − x̂), w.

A second-order Taylor expansion for P̂n : t  ϕ((1 − t)x̂ + tx̂n ) gives

 1

ϕ(x̂n ) − ϕ(x̂) = ϕ (x̂)(x̂n − x̂) + P̂n (t)(1 − t) dt,
0

so that
 d 2
1− L̂ + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − ϕ(x̂), zn − z + 2n w (5.81)
R n
+ 22n x̂n − x̂, x0 − x̂ − (ϕ (x̂)∗ w + μ)
 1
− 2n 
2
P̂n (t)(1 − t) dt, w.
0

But the deflection condition (5.43) gives

 1  1 
  π 1  π
 P̂n (t)(1 − t) dtF ≤ P̂n (t)F dt ≤ P̂n (t)F dt = L̂n ,
0 0 2 0 2
and, using the Cauchy–Schwarz inequality, (5.81) becomes
 d 2
1− L̂ + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − ϕ(x̂) (δn + 2n w)
R n
2
+ 2n ηx̂n − x̂E
3
+ π2n L̂n wF ,
5.1. LMT REGULARIZATION 231

which in turn gives, using formula (5.32),

 d 1/2  d −1/2   π  η
1− L̂n ≤ 2 1 − δn + 2n 1 + wF + n , (5.82)
R R 2 3
 d −1/2   π   2η
n x̂n − x̂E ≤ 1 − δn + 2n 1 + wF + n . (5.83)
R 2 3
When δn /n → 0, the last inequality shows that x̂n − x̂E ≤ η for n large
enough, which proves (5.56).

Part 3 Now x̂ satisfies the regularity condition (5.58), and so we can choose
η = 0 in Part 2, and estimations (5.82) and (5.83) simplify to
 d 1/2  d −1/2   π 
1− L̂n ≤ 2 1 − δn + 2n 1 + wF ,
R R 2
 d −1/2   π  
n x̂n − x̂E ≤ 1 − δn + 2n 1 + wF ,
R 2
which prove (5.59) and (5.60).
The previous theorem reduces exactly to the Theorem 5.1.8 when the
forward map ϕ is linear.

5.1.3 General Nonlinear Problems

We investigate in this section the Q-wellposedness of the LMT-regularized
problem (5.2) when the unregularized problem (5.1) satisfies only the mini-
mum set of hypothesis (4.2) and ϕ has a bounded second derivative
⎧
⎪
⎪ there exists β ≥ 0 such that
⎨
∀x0 , x1 ∈ C, the curve P : t  ϕ((1 − x0 )t + tx1 ) satisfies
(5.84)
⎪
⎪ P ∈ W 2,∞ ([0, 1]; F ) and A(t)F ≤ βx1 − x0 2E ,
⎩
for a.e. t ∈ [0, 1], where A(t) = P (t).

Hypothesis (4.2) and (5.84) are verified as soon as the map ϕ to be inverted
is smooth, which is the case in most of the situations. Hence the results of
this section apply to all examples of Chap. 1 for which no stronger FC/LD
property can be proved:
232 CHAPTER 5. REGULARIZATION

• The Knott–Zoeppritz equations of Sect. 1.1, all types of observation,

• The 1D elliptic parameter estimation problem of section 1.4 with L2 or
partial or boundary observation (for H1 observation the problem has
been shown to be Q-wellposed in section 4.8),
• The 2D elliptic nonlinear source estimation problem of Sect. 1.5 for
boundary or partial observation (the case of H 1 or L2 observation leads
to a FC/LD problem, see Sect. 5.2 below),
• The 2D elliptic parameter estimation problem of Sect. 1.6 for L2 or
partial or boundary observation (for H 1 observation, Q-wellposedness
results after reduction to finite dimension are available in Sect. 4.9).
However, under these sole hypothesis, nothing is known concerning the
existence and uniqueness of solution(s) to the NLS problem (5.1) and its
LMT-regularized version (5.2). This is a main difference with the linear case
of Sect. 5.1.1 (which corresponds to β = 0) and the case of finite curvature
limited deflection (FC/LD) problems (Sect. 5.1.2), where (5.2) has a solution
as soon as  > 0, possibly small enough!
We use first the Q-wellposedness conditions of Chap. 4 to quantify the
natural intuition that “a minimum amount of LMT-regularization” should
be added to compensate for the nonlinearity of ϕ and restore Q-wellposedness
of (5.2) as in the linear case [18].
The size of the parameter set and the position of the a-priori guess in the
admissible parameter set will play a role through the following quantities:
diam C = sup x − yE , radx0 C = sup x − x0 E .
x,y∈C x∈C

The regularized problem(5.2) is equivalent to

1
x̂ minimizes  (ϕ(x), x) −(z, x0 )2F ×E over C.
2 ! "# $
def
= ϕ (x)
Along the curves P : t  ϕ ((1 − t)x0 + tx1 ), the velocity and acceleration
V (t) = (V (t), (x1 − x0 )), A (t) = (A(t), 0)
satisfy, using (4.2) (5.84),
x1 − x0 E ≤ V (t)F ×E ≤ (αM
2
+ 2 )1/2 x1 − x0 E ,
A (t)F ×E = A(t)F ≤ βx1 − x0 2E ,
5.1. LMT REGULARIZATION 233

and hence
β β
A (t)F ×E ≤ V  (t) 2
F ×E , A (t) F ×E ≤ diam CV (t)F ×E .
2 
This shows that the the regularized problem (5.2) is linearly stable (Definition
4.3.4), and has a curvature and a deflection bounded by (compare with (5.42)
and (5.43) for the case of FC/LD problems)

1 β β
= 2, Θ = diam C.
R  
As expected, this upper bound to the curvature blows up to infinity when the
regularization parameter goes to zero. Application of Theorem 4.4.1 shows
that (5.2) is Q-wellposed in E × F on the neighborhood
+ 2 ,
(z, x0 ) ∈ F × E | dF ×E ((z, x0 ), ϕ (C)) < R =
β

of ϕ (C) as soon as the deflection condition

Θ = (β/)diam C ≤ π/2

is satisfied. But one checks easily that

dF ×E ((z, x0 ), ϕ (C))2 ≤ d(z, ϕ(C))2 + 2 rad2x0 C,

which proves the

Proposition 5.1.16 Hypothesis (4.2) and (5.84). If the regularization pa-

rameter  satisfies
def & '
 > min = β max radx0 C, (2/π)diam C ,

then 2 /β = R >  radx0 C and (β/)diam C = Θ ≤ π/2, and the regularized

problem (5.2) is Q-wellposed in F on the neighborhood
+ ,
ϑ = z ∈ F | dF (z, ϕ(C)) < d , (5.85)

where d is defined by
d2 + 2 rad2x0 C = R2 . (5.86)
234 CHAPTER 5. REGULARIZATION

When the data z0 , z1 ∈ ϑ are close enough, one can choose d < d such that
  12
z0 − z1 F + max d(zj , ϕ(C))2 + 2 rad2x0 C (5.87)
j=0,1
 2 1/2
≤ d + 2 rad2x0 C < R ,

and the corresponding regularized solutions x̂,j , j = 0, 1 satisfy

 (d2 + 2 rad2x0 C) 2 −1
1

 x̂,0 − x̂,1 E ≤ 1 − z0 − z1 F . (5.88)

R
This proposition can be used, for example, to determine, given an estimation
of an upper bound dmax of the measurement and model errors, a minimum
amount of regularization min , which ensures Q-wellposedness of the regular-
ized NLS problem on a neighborhood of ϕ(C) large enough to contain the
expected data.

Remark 5.1.17 Formula (5.85) through (5.88) describe also the stability
property of the regularized problem (5.5) for a fixed n, both in the linear case
of Theorem 5.1.8 (take 1/R = Θ = 0, so that d = +∞) and in the case
of FC/LD problems of Theorem 5.1.15 (replace R , Θ by Rn , Θn defined in
(5.62)).

Next we study the convergence of solutions x̂n to (5.5) when n → ∞ [27].

Because of the possibly infinite curvature of the attainable set, we consider
only the case where

z ∈ ϕ(C) (attainable data), (5.89)

which eliminates the need of projecting z onto ϕ(C). The following additional
properties of ϕ will be needed:

ϕ is sequentially closed, (5.90)

ϕ has a Fréchet derivative ϕ (x̂) at x̂. (5.91)

Condition (5.90) ensures that the solution set X = ϕ(z)−1 of (5.1) is non-
void, and so we can define an x0 -minimum-norm solution x̂ of (5.1) as one
solution of the minimization problem (5.11) (there may be more than one
x0 -minimum-norm solution as X is not necessarily convex).
5.1. LMT REGULARIZATION 235

The characterization of x̂ follows the same line of arguments as in

Sect. 5.1.2, with some adaptations: the tangent cone to X at x̂ is now defined
by (compare with (5.12))

T (X, x̂) = {y ∈ E : ∃xn ∈ X, λn > 0, xn → x̂ s.t. λn (xn − x̂) → y},

and a short calculation using hypothesis (5.91) shows that Lemma 5.1.10 still
holds
T (X, x̂) ⊂ Kerϕ (x̂) ∩ T (C, x̂) ,
and, when x̂ is qualified in the sense of definition 5.1.11, that lemma 5.1.13
also holds:
x0 − x̂ ∈ Rg ϕ (x̂)∗ + T (C, x̂)− .
The LMT-regularized problems (5.5) are not in general Q-wellposed, but
they all have at least one solution because of hypothesis (5.90).

Theorem 5.1.18 Let hypothesis (4.2), (5.84), and (5.89) through (5.91)
hold, and let x̂ be an x0 -minimum-norm solution to (5.1) that satisfies

x0 − x̂ ∈ Rg ϕ (x̂)∗ + T (C, x̂)− (Regularity Condition), (5.92)

βwF ≤ 1, (5.93)

where β is defined in (5.84), and w ∈ F is an element satisfying

x0 − x̂ = ϕ (x̂)∗ w + μ where μ ∈ T (C, x̂)− .

Then any sequence {xn } of solutions to (5.5) satisfies, when n → 0 and

δn ∼ 2n ,
1
x̂n − x̂E = O(n ) = O(δn2 ),
ϕ(x̂n ) − zF = O(2n ) = O(δn ).

Proof. By definition, x̂n satisfies

ϕ(x̂n ) − zn 2F + 2n x̂n − x0 2E ≤ ϕ(x̂) − zn 2F + 2n x̂ − x0 2E .

Adding and subtracting ϕ(x̂n ) − z2F + 2n x̂n − x̂2E gives, as in the proof of
Theorems 5.1.8 or 5.1.15,

ϕ(x̂n ) − z2F + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − z, zn − zF

+22n x̂n − x̂, x0 − x̂E .
236 CHAPTER 5. REGULARIZATION

By (5.92) this implies

ϕ(x̂n ) − z2F + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − z, zn − z (5.94)


+2n ϕ (x̂)(x̂n − x̂), w.
2

A second order Taylor expansion for P : t  ϕ((1 − t)x̂ + tx̂n ) gives

 1

ϕ(x̂n ) − ϕ(x̂) = ϕ (x̂)(x̂n − x̂) + P  (t)(1 − t) dt,
0

and, using (5.84)

 1
1
 P  (t)(1 − t) dt F ≤ β x̂n − x̂2E . (5.95)
0 2

Hence (5.94) becomes

ϕ(x̂n ) − z2F + 2n x̂n − x̂2E ≤ 2ϕ(x̂n ) − z, zn − z + 2n w (5.96)

 1
−2n 
2
P  (t)(1 − t)dt, w.
0

Formula (5.95) and the Cauchy–Schwarz inequality give, rearranging terms,

ϕ(x̂n ) − z2F + 2n (1 − βwF )x̂n − x̂2E ≤ 2ϕ(x̂n ) − zF (δn + 2n w), (5.97)

and, using (5.32),

ϕ(x̂n ) − z2F ≤ 2(δn + 2n wF ),

1
(1 − βwF ) 2 n x̂n − x̂E ≤ (δn + 2n wF ),

and the desired result follows.

Remark 5.1.19 Using weak subsequential arguments, one can show that any
sequence {x̂n } of solutions to (5.5) contains a subsequence, which converges
strongly to an X0 -minimum-norm solution of (5.1), provided that n → 0 and
δn /n → 0.
5.2. APPLICATION: NONLINEAR 2-D SOURCE PROBLEM 237

5.2 Application to the Nonlinear 2D Source

Problem
We give in this section an example of nonlinear infinite dimensional FC/LD
problem to which Theorem 5.1.15 on regularization applies. It is the source
estimation problem in a nonlinear elliptic equation described in Sect. 1.5:
⎧
⎨ −Δu + k(u) = f in Ω,
⎪
u = 0 on a part ∂ΩD of ∂Ω, (5.98)
⎪
⎩ ∂u = g on ∂Ω = ∂Ω \ ∂Ω ,
N D
∂ν
where the objective is to estimate the right-hand sides f, g from a measure-
ment z of the solution u in L2 (Ω). With the notation of Sect. 1.5 and

x = (f, g) ∈ E = L2 (Ω) × L2 (∂ΩN ), (5.99)

the NLS problem (1.61) coincides with (5.1) and its regularized versions
(1.63) coincides with (5.5).
We show in this section that (1.61) is a finite curvature problem, and
estimate the size of C, which ensures a deflection smaller than π/2, making
thus (1.61) a FC/LD problem to which Theorem 5.1.15 can be applied. We
follow for the proof reference [28], where the result was proved for a slightly
more complex model involving a convection term.
Remark 5.2.1 When the nonlinearity is in the higher order term, as, for
example, in

− ∇.(a(u)∇u) = f + boundary conditions,

the finite curvature property is lost. However, the least square objective func-
tion still possesses minimizer(s) over a suitable admissible set, is deriv-
able, and the parameter a can be retrieved numerically over the range of u
(see [31]).
We begin with the FC/LD properties of (1.61). The variational formulation
of (5.98) is, with the definition (1.58) for Y ,
⎧
⎨ find u ∈ Y such that
∇u, ∇wL2(Ω) + k(u), wL2(Ω) = f, wL2 (Ω) + g, wL2(∂ΩN ) (5.100)
⎩
for all w ∈ Y.
238 CHAPTER 5. REGULARIZATION

We endow Y with the norm

wY = |∇w|L2(Ω) .
Since ∂ΩD is assumed to be nonempty, this norm is equivalent to the usual
H 1 norm using the Poincaré inequality:
|w|L2(Ω) ≤ CP |∇w|L2(Ω) , where CP is the Poincaré constant. (5.101)
The right-hand side of (5.100) is a continuous linear form L on Y :
 
L(w) = fw + gw (5.102)
 Ω ∂ΩN

≤ CP |f |L2 (Ω) + CN |g|L2(∂ΩN ) wY
1
≤ (CP2 + CN2 ) 2 (f, g)E wY ,
! "# $
M
where CN denotes the continuity constant of the trace operator τN from Y
to L2 (∂ΩN ):
|w|L2(∂ΩN ) ≤ CN wY .

Lemma 5.2.2 (Hypothesis and notations of Sect. 1.5). Then (5.100) has a
unique solution u ∈ Y , which satisfies the a-priori estimate
u1 − u0 Y ≤ M(f1 , g1 ) − (f0 , g0 )E , M defined in (5.102). (5.103)
Proof. The left-hand side of (5.100) defines an operator A : Y  Y  :

∀v ∈ Y , A(v) ∈ Y  is defined by
A(v), wY  Y = ∇v, ∇wL2(Ω) + k(v), wL2(Ω) ∀w ∈ Y.
Using the properties imposed on k, one can check that A maps bounded
sets to bounded sets, and that A is hemicontinuous (i.e., for all the function
λ  A(u + λv), wY  Y is continuous from IR to IR). For every v, w ∈ Y ,
moreover,
⎧
⎨ A(v) − A(w), v − wY  Y = |∇(v − w)|2 2
  1  L (Ω)
+ Ω 0 k ((1 − t)w + tv)(v − w)2
⎩
≥ v − w2Y ,
hence A is strictly monotone, and the lemma is proved [56, page 171].
5.2. APPLICATION: NONLINEAR 2-D SOURCE PROBLEM 239

Lemma 5.2.3 (Hypothesis and notations of Sect. 1.5). Let xj = (fj , gj ) ∈

C, j = 0, 1 be given. Then P : t ∈ [0, 1]  ϕ((1 − t)x0 + tx1 ) is in
W 2,∞ ([0, 1], Y ), and its first and second derivatives V (t) and A(t) are the
variational solutions in Y of
⎧
⎪ −ΔV (t) + k  (P (t))V (t) = f1 − f0 in Ω,
⎨
V (t) = 0 on ∂ΩD , (5.104)
⎪
⎩ ∂V (t)
= g1 − g0 on ∂ΩN ,
∂ν
⎧
⎪ −ΔA(t) + k  (P (t))A(t) = −k  (P (t))V (t)2 in Ω,
⎨
A(t) = 0 on ∂ΩD , (5.105)
⎪
⎩ ∂A(t)
= 0 on ∂ΩN .
∂ν
Proof. Because of (5.103), P is Lipschitz continuous and hence a.e. differen-
tiable with

V (t)Y ≤ M(f1 , g1 ) − (f0 , g0 )E a.e. on [0, 1], (5.106)

so that P ∈ W 1,∞ ([0, 1], Y ). By definition, P (t) is the solution of

⎧
⎨ ∇P (t), ∇wL2(Ω) + k(P (t)), wL2(Ω) =
(1 − t)f0 + tf1 , wL2 (Ω) + (1 − y)g0 + tg1 , wL2(∂ΩN ) (5.107)
⎩
for all w ∈ Y.

Differentiation of (5.107) with respect to t shows that V (t) satisfies

⎧
⎨ ∇V (t), ∇wL2 (Ω) + k  (P (t))V (t), wL2(Ω) =
f1 − f0 , wL2(Ω) + g1 − g0 , wL2 (∂ΩN ) (5.108)
⎩
for all w ∈ Y,

which is the variational formulation of (5.104). We next argue that t  V (t)

is Lipschitzian from [0, 1] to Y . For t, τ ∈ [0, 1] we have
⎧
⎨ |∇(V (t) − V (τ ))|2L2 (Ω) + k  (P (t)), (V (t) − V (τ ))2 L2 (Ω)
= −(k  (P (t) − k  (P (τ ))V (τ ), V (t) − V (τ )L2 (Ω)
⎩
for all w ∈ Y,

and hence

V (t) − V (τ )2Y ≤ k  (P (t) − k  (P (τ )L3 (Ω) V (τ )L3 (Ω) V (t) − V (τ )L3 (Ω) .
240 CHAPTER 5. REGULARIZATION

But L3 (Ω) imbeds continuously into H 1 (Ω) if the dimension m of Ω is smaller

than 6, which is the case as we have supposed m ≤ 3 in (1.57). Hence if we
denote by const the corresponding embedding constant we obtain,

V (t) − V (τ )Y ≤ constk  L∞ P (t) − P (τ )L3 (Ω) V (τ )L3 (Ω)

≤ const3 k  L∞ P (t) − P (τ )Y V (τ )Y
≤ const3 Mk  L∞ P (t) − P (τ )Y (f1 , g1 ) − (f0 , g0 )E ,

where we have used (5.106) to obtain the last inequality. Since t  P (t) is
Lipschitz continuous, it follows that t  A(t) is Lipschitz continuous as well.
Hence V is a.e. differentiable, and using (5.103),

A(t)Y ≤ const3 M 2 k  L∞ (f1 , g1 ) − (f0 , g0 )2E a.e. in [0, 1],

which proves that P ∈ W 2,∞ ([0, 1], Y ). Derivation of (5.108) with respect to
t gives then
⎧
⎨ ∇A(t), ∇wL2(Ω) + k  (P (t))A(t), wL2(Ω) =
−k  (P (t))V (t)2 , wL2(Ω) (5.109)
⎩
for all w ∈ Y,

which is the variational formulation of (5.105).

Proposition 5.2.4 (Hypothesis and notations of Sect. 1.5). There exists

constants αM > 0, CR > 0, and CΘ > 0 such that

V (t)F ≤ αM (f1 , g1 ) − (f0 , g0 )E (5.110)

A(t)F ≤ CR k  L∞ (IR) V (t)2F (5.111)
A(t)F ≤ CΘ k  L∞ (IR) diam C V F , (5.112)

where F is the data space L2 (Ω) defined in (1.60). Hence (1.61) is a FC

problem, and its curvature and deflection of (1.61) are bounded by

1/R = CR k  L∞ (IR) , Θ = CΘ k  L∞ (IR) diam C. (5.113)

It follows that (1.61) is also a FC/LD problem as soon as

π
Θ = CΘ k  L∞ (IR) diam C ≤ . (5.114)
2
5.2. APPLICATION: NONLINEAR 2-D SOURCE PROBLEM 241

Proof. From (5.106) and the Poincaré inequality (5.101), (5.110) follows with
αM = CP M = CP (CP2 + CN2 )1/2 .
To prove (5.111), we introduce, for fixed but arbitrary t ∈ [0, 1] and
(f0 , g0 ), (f1 , g1 ) ∈ C, an operator B : D(B)  L2 (Ω) defined by
& ∂ψ '
D(B) = ψ ∈ H 2 (Ω) : ψ = 0 on ∂ΩD and = 0 on ∂ΩN , (5.115)
∂ν
∀ψ ∈ D(B), Bψ = −Δψ + k  (P (t))ψ ∈ L2 (Ω), (5.116)
where D(B) is endowed with the norm of H 2 (Ω). Because of the assumption
made in (1.57) that ∂ΩD is both open and closed with respect to ∂Ω and the
C 1,1 -regularity of ∂Ω, the regularity results on elliptic equations imply that
B is an isomorphism from D(B) onto L2 (Ω):
1
∃CB > 0 such that ψH 2 ≤ BψL2 ≤ CB ψH 2 ∀ψ ∈ D(B).
CB
The same regularity results imply that the solution A(t) of the variational
problem (5.109) satisfies A(t) ∈ H 2 (Ω), and hence is a strong solution of
(5.105), so that A(t) ∈ D(B), and

B A(t) = −k  (P (t)) V (t)2 ∈ L1 (Ω) ⊂ L2 (Ω).

Multiplication by ψ ∈ D(B) and integration over Ω gives

B A(t), ψL2 (Ω) = −k  (P (t)) V (t)2 , ψL2 (Ω) ∀ψ ∈ D(B). (5.117)

A(t) and ψ belong to D(B) ⊂ H 2 (Ω), so one can integrate by part twice in
the left-hand side of (5.117) using the Green formula, which gives

A(t), B ψL2 (Ω) = −k  (P (t)) V (t)2 , ψL2 (Ω) ∀ψ ∈ D(B).

Since B is an isomorphism from D(B) onto L2 (Ω), one has

|A(t)|L2 (Ω) = sup A(t), B ψL2 (Ω)

ψ∈D(B) , |Bψ|L2 (Ω) =1

= sup k  (P (t)) V (t)2 , ψL2 (Ω)

ψ∈D(B) , |Bψ|L2 (Ω) =1

≤ k  L∞ (Ω) |V |2L2 (Ω) sup ψL∞ (Ω)

ψ∈D(B) , |Bψ|L2 (Ω) =1

≤ CB CH k  L∞ (Ω) |V |2L2 (Ω) ,

242 CHAPTER 5. REGULARIZATION

where CH denotes the embedding constant of H 2 (Ω) into L∞ (Ω) (here we

use the hypothesis in (1.57) that the dimension m of Ω is smaller than 3).
This proves (5.111) and (5.112) with

CR = CB CH , CΘ = CB CH CP (CP2 + CN2 )1/2

and the proposition is proved.

To this point we have proved that (1.61) is a FC/LD problem when (5.114)
is satisfied. We check now that the remaining hypothesis required to apply
Theorem 5.1.15 on the convergence of LMT-regularized solutions are satisfied.

Lemma 5.2.5 (Hypothesis and notations of Sect. 1.5). The mapping ϕ is

injective over C, and has a Gâteaux derivative ϕ (x) ∈ L(E, F ) at every
x ∈ C, which is also injective. Hence x is linearly identifiable over C, and
all points x of C are qualified.
Moreover, the range of ϕ (x)∗ ∈ L(F, E) is given by
& '
Rg ϕ (x)∗ = (f, g) ∈ E : f ∈ D(B) and g = τN f , (5.118)

where D(B) is defined in (5.115), and τN denotes the trace on ∂ΩN .

Proof. Let xi = (fi , gi) ∈ C, i = 0, 1, be given, and define ui = ϕ(xi ), i = 0, 1.

Then u0 = u1 implies f1 − f0 , wL2 (Ω) + g1 − g0 , wL2(∂ΩN ) = 0 for all w ∈ Y ,
and hence f1 − f0 = 0 (take w = φ ∈ D(Ω)) and g1 − g0 = 0 (take w ∈ Y and
use the density in L2 (∂ΩN ) of the traces of functions of Y ), which proves the
injectivity of ϕ.
It is then easy to check that the Gâteaux differential δu = ϕ (x)(δf, δg) ∈
F = L2 (Ω) of ϕ at x = (f, g) ∈ C in the direction δx = (δf, δg) ∈ E is the
solution of
⎧
⎨find δu ∈ Y such that
∇δu, ∇wL2(Ω) + k  (u)δu, wL2(Ω) = δf, wL2(Ω) + δg, wL2(∂ΩN )
⎩
for all w ∈ Y,
(5.119)
where u is the solution of (5.100). The proof of the injectivity for ϕ (x) is the
same as for ϕ(x).
Concerning the qualification of x ∈ C, the injectivity of ϕ(x) ensures that
the solution set X contains at most one element, so that T (X, x) = {0}, and
5.2. APPLICATION: NONLINEAR 2-D SOURCE PROBLEM 243

the injectivity of ϕ (x) ensure that Ker ϕ (x) = {0}, so that (5.49) is trivially
satisfied, and x is qualified.
We determine now ϕ (x)∗ to find its range. Let δv ∈ F = L2 (Ω) and
δx = (δf, δg) ∈ E = L2 (Ω) × L2 (∂ΩN ) be given, and define δu and δh by

δu ∈ Y solution of (5.119), (5.120)

δh ∈ D(B) solution of B δh = δv, (5.121)

where B is the operator defined in (5.116) with P (t) replaced by u = ϕ(x).

Then

ϕ (x)∗ δv, δxE = δv, ϕ(x) δxF

= δv, δuF
= B δh, δuF
= −Δδh + k  (u)δh, δuF =L2(Ω) .

Here δh ∈ D(B) ⊂ H 2 (Ω), and δu ∈ Y ⊂ H 1 (Ω), so we can use the Green

formula to integrate one time by part:

ϕ (x)∗ δv, δxE = ∇δh, ∇δuL2(Ω) + k  (u)δh, δuL2(Ω (5.122)

   
∂δh ∂δh
+ , δu + , δu .
∂ν L2 (∂ΩD ) ∂ν L2 (∂ΩN )

Because of the boundary conditions included in the spaces Y and D(B), the
boundary terms vanish in (5.122), which becomes, using (5.119),

ϕ (x)∗ δv, δxE = δf, δhL2 (Ω) + δg, δhL2(∂ΩN ) (5.123)
= (δh, δk), (δf, δg)E ,
! "# $
δx
where we have set
δk = τN δh (5.124)
The last equation in (5.123) shows that

∀δv ∈ F, ϕ (x)∗ , δv = (δh, δk) ∈ E, defined by (5.121) and (5.124),

(5.125)
which proves (5.118), and the lemma is proved.
244 CHAPTER 5. REGULARIZATION

The nonlinear source estimation problem (1.61) has hence the following
properties before any regularization is applied:
Theorem 5.2.6 Let hypothesis and notations of Sect. 1.5 hold. Then
1. The parameter x is identifiable on C, and (1.61) is a FC problem, with
curvature and deflection bounded by 1/R and Θ given in (5.113).
2. If moreover the deflection Θ satisfies (5.114), then (1.61) is a FC/LD
problem, and Proposition 4.2.7 applies: when the data z is in the neigh-
borhood ϑ of the attainable set defined in (5.44), there exists at most one
solution x̂, when a solution exists the objective function J is unimodal
over C, and the projection in the data space onto the attainable set is
stable – but there is no stability estimate for x in the L2 (Ω) × L2 (∂ΩN )
parameter norm.
3. If moreover C is bounded, the FC/LD problem (1.61) has a (unique)
solution for all z ∈ ϑ.
Proof. Points 1 follows from Lemma 5.2.5 and Proposition 5.2.4, point 2 from
Proposition 4.2.7. The proof of point 3 goes as follows: let xk = (fk , gk ) be a
minimizing sequence of J, and uk = ϕ(xk ) the associated solutions of (5.100).
The sequence xk is bounded in E and hence uk is bounded in Y ⊂ H 1 (Ω),
which embeds compactly in L2 (Ω). Hence there exists x̂ ∈ C, û ∈ Y ⊂ H 1 (Ω)
and subsequences, still noted xk , uk such that
xk  x̂ in E, uk  x̂ in Y and almost everywhere on Ω,
where  denotes weak convergence. It is then possible to pass to the limit
in the variational formulation (5.100) giving uk , which shows that û = ϕ(x̂).
Hence x̂ is a minimizer of J over C, which ends the proof.
We can now apply the LMT-regularization to the FC/LD nonlinear source
problem:
Theorem 5.2.7 Let hypothesis and notations of Sect. 1.5 as well as the de-
flection condition (5.114) hold, and suppose the data z belong to the neigh-
borhood ϑ defined in (5.44). Then
1. The regularized problems (1.63) are all Q-wellposed for n large enough,
and
n fˆn → 0, n ĝn → 0,
un → ẑ.
when n → 0 and δn → 0, where ẑ =projection of z onto ϕ(C).
5.2. APPLICATION: NONLINEAR 2-D SOURCE PROBLEM 245

2. If the unregularized problem (1.61) admits a (necessarily unique) solu-

tion (fˆ, ĝ) – for example, if C is bounded in E = L2 (Ω) × L2 (∂ΩN ) –
then ẑ = ϕ(f, ˆ ĝ), and one has, when n → 0 and δn /n → 0

fˆn → fˆ, ĝn → ĝ,

un − ẑL2 (Ω) = O(n ).

3. If moreover (fˆ, ĝ) satisfies the regularity condition:

ˆ ĝ) ∈ {(f ∈ D(B), g = τn f )} + T (C, x̂)− ,
(f0 , g0 ) − (f, (5.126)
where D(B) is defined in (5.115) and τN is the “ trace on ∂ΩN” opera-
tor, one has, when n → 0 and δn ∼ 2n
fˆn − fˆL2 (Ω) = O(n ), fˆn − ĝL2 (∂ΩN ) = O(n ),
un − ẑF = O(2n ).

Proof. Theorem 5.2.6 shows that the unregularized problem (1.61) is a FC/LD
problem, so we can apply Theorem 5.1.15, which together with Lemma 5.2.5
gives the desired results.
We finally interpret the regularity condition (5.126) in two specific cases
depending on the location of (fˆ, ĝ) within C:
ˆ ĝ) is in the interior of C, then T (C, x̂)− = {0}, and (5.126) reduces
• If (f,
to
(f0 , g0 ) − (fˆ, ĝ) ∈ {(f ∈ D(B), g = τn f )}. (5.127)
• If C is a closed ball centered at the origin, and (fˆ, ĝ) lies on the bound-
ary of C, then T (C, x̂)− = {(f, g) : (f, g, ) = λ(fˆ, ĝ), λ > 0}, and
(5.126) becomes
∃λ > 0 : (f0 , g0 ) − (1 + λ)(fˆ, ĝ) ∈ {(f ∈ D(B), g = τn f )}. (5.128)

If zero is used as a-priori guess, then both “regularity conditions” (5.127)

and (5.128) are equivalent to
⎧
⎨ ˆ ∂ fˆ
f ∈ H 2 (Ω), fˆ = 0 on ∂ΩD , and = 0 on ∂ΩN , (5.129)
⎩ ∂ν
ˆ
ĝ = τN f ∈ H (∂ΩN ).
3/2

This justifies the terminology “regularity condition” used for (5.51) or

(5.126).
246 CHAPTER 5. REGULARIZATION

Remark 5.2.8 Problem (1.61) is also a FC/LD problem for the stronger ob-
servation space F = H 1 (Ω)). In this case, it suffices to require in (1.57) that
m ≤ 4, and the condition that ∂ΩD is both open and closed is not needed. This
follows from Lemma 5.2.3 and its proof, (5.106), and the following estimate:
A(t)Y ≤ k  L∞ (IR) |V (t)|2L2 (Ω)
≤ k  L∞ (IR) |V (t)|2L4 (Ω)
≤ constk  L∞ (IR) |V (t)|2Y ,
where const is the embedding constant of Y into L4 (Ω) for m ≤ 4. The
proof of the existence of the Gâteaux derivative ϕ (x) is the same, and the
characterization of its adjoint follows the same arguments, with now B ∈
L(Y, Y  ) and D(B) = Y . Then
& '
Rg ϕ (x)∗ = (f, g) ∈ E : f ∈ Y and g = τN f ,
and (5.129) is replaced by

fˆ ∈ H 1 (Ω), fˆ = 0 on ∂ΩD ,
ĝ = τN fˆ ∈ H 1/2 (∂ΩN ).

5.3 State-Space Regularization

We have seen in Sect. 5.1.3 that LMT-regularization failed to produce Q-
wellposed problems for small ’s when applied to infinite curvature problems,
that is, to problems where no upper bound to the curvature of the curves
P of ϕ(C) could be proved. So we consider in this section a special class
of infinite curvature problem, where the origin of the ill-posedness resides
in a poor measurement of the state of the system. To make this precise, we
suppose that a state-space decomposition (2.1) (2.2) of the direct mapping
ϕ : x ∈ C  v ∈ F exists such that:
• The parameter-to-state map
φ : x ∈ C  y ∈ F solution of e(x, y) = 0 (5.130)
can be “inverted” in a sense to be made precise. Hence x would be
“identifiable” if a full measurement of the state y ∈ Y were available,
5.3. STATE-SPACE REGULARIZATION 247

• But the only observations available are measures of v ∈ F related to

y by a linear continuous observation operator, which incurs a loss of
information,
M : y ∈ Y  v = My ∈ F, M ∈ L(Y, F ). (5.131)

So the NLS problems (5.1) considered here are of the form

1
x̂ minimizes J(x) =  Mφ(x) −z2F over C, (5.132)
2 ! "# $
ϕ(x)
where the data z is supposed to be attainable:
∃ x̂ ∈ C, z = ẑ = M ŷ ∈ F, ŷ = φ(x̂) ∈ Y. (5.133)
For such problems, the state-space regularization [26, 29] consists in choos-
ing an a-priori guess y0 ∈ Y of the state y = φ(x), and to use this additional
information to build up the state space regularized problem:
2
x̂ minimizes J (x) = φ(x) − y0 2Y + J(x) over C, (5.134)
2
to be compared with the LMT-regularized problems (5.2).
State-space regularization corresponds to a widespread approach in the
engineering community, which consists in smoothing or interpolating first
the data before performing inversion. We analyze the behavior of problems
(5.134) for two levels of hypotheses:
• Strong hypotheses: we suppose that the NLS problem associated to
(C, φ) is Q-wellposed, and that the observation is dense, that is, the
observation operator M is injective, and so there is no irremediable
information loss. We shall prove in this case that the state-space reg-
ularized problems (5.134) are, after localization, Q-wellposed and con-
verging when  → 0. We shall use for this a geometric approach, along
the lines of Chap. 4.
• Weak hypotheses: we suppose only that φ is invertible on C, and con-
sider the case of a incomplete observation, where M is allowed to be
noninjective – in which case there can be a definite loss of information.
Here, Q-wellposedness of the state-space regularized problems will be
lost, and only weak convergence results to a “state-space minimum-
norm” solution will be available for subsequences of solutions. The tools
there will be soft analysis.
248 CHAPTER 5. REGULARIZATION

5.3.1 Dense Observation: Geometric Approach

A typical example of this situation is M = canonical injection from H 1 in L2 .
It corresponds to parameters x, which are known to be OLS-identifiable from
measurements of y in H 1 , but where the only data at hand are measurements of
y in L2 . For example, the one-dimensional elliptic inverse problem of Sect. 1.4
has been shown in Sect. 4.8 to be Q-wellposed when a strong H 1 observation was
available,andsostate-spaceregularizationisindicatedwhenonlyL2 observation
is available. A similar situation for L2 observation occurs with the diffusion
coefficient estimation problem in a 2D elliptic equation of Sect. 1.6, whose
Q-wellposedness for H 1 observation is studied in Sects. 4.9 and 5.4.

Properties of (C, φ) and M:

We suppose that C and φ satisfy
⎧
⎪
⎪ E = Banach space, Y = Hilbert space,
⎪
⎪
⎪
⎪ C = closed, convex subset of E,
⎪
⎪
⎪
⎪ there exists 0 < αm ≤ αM , R > 0 and Θ < π/2 such that
⎪
⎪
⎨ ∀x0 , x1 ∈ C, P : t  φ((1 − x0 )t + tx1 ) is in W 2,∞ ([0, 1]; F )
and, for a.e. t ∈ [0, 1], (5.135)
⎪
⎪
⎪
⎪ αm x1 − x0 E ≤ V (t)Y ≤ αM x1 − x0 E ,
⎪
⎪
⎪
⎪ A(t) Y ≤ (1/R)V (t)Y ,
2
⎪
⎪
⎪
⎪ A(t)Y ≤ ΘV (t)Y ,
⎩
where V (t) = P (t), A(t) = P  (t).

This ensures by Theorem 4.4.1 that the NLS problem

1
x̂ minimizes φ(x) − y2Y over C (5.136)
2
is Q-wellposed for y in a neighborhood of φ(C) of size R in Y . The strict
inequality Θ < π/2 is not required by Theorem 4.4.1, but it will simplify
(somehow...) the calculations as it will allow to use the simple deflection
condition Θn ≤ π/2 to ensure Q-wellposedness of the regularized problems.
Then we suppose that the observation operator satisfies

M ∈ L(Y, F ), with M injective and F = Hilbert space, (5.137)

so that the true state ŷ and parameter x̂ associated to z = ẑ by (5.133) are

uniquely defined.
5.3. STATE-SPACE REGULARIZATION 249

Properties of Data:
We suppose that a sequence zn ∈ F, n = 1, 2 . . . of noise corrupted measure-
ments is available, which converges to the attainable noise free data z = ẑ:
zn ∈ F, zn − ẑF ≤ δn , δn → 0 when n → ∞, (5.138)
and that an a-priori guess
y0 ∈ Y (5.139)
of the true state ŷ has been chosen. This a-priori guess can be enhanced by
application of the linear LMT-regularization theory of Sect. 5.1.1 to the esti-
mation of ŷ from zn , n = 1, 2 . . . : let n → 0 be a sequence of regularization
parameters, and define yn by the auxiliary problem
2n 1
yn ∈ Y minimizes y − y0 2Y + M(y) − zn 2F over Y. (5.140)
2 2
Suppose that ŷ satisfies the regularity condition (5.24) of Theorem 5.1.8 (iii):
ŷ − y0 = M ∗ w for some w ∈ F, (5.141)
and that the regularization parameters 2n goes to zero more slowly than the
error δn on the data:
∃λ > 0 such that ∀n ∈ IN : λ wF 2n ≥ δn . (5.142)
The error estimate (5.41) gives then
yn − ŷY ≤ n (λ + 1)wF , (5.143)
which leads us to use yn as a priori guess for the state-space in lieu of y0 in
(5.134). So we replace in a first step (5.134) by
2n 1
x̂n minimizes Jn (x) = φ(x) − yn 2Y + M φ(x) − zn 2F over C. (5.144)
2 2

Properties of State-Space Regularized Problem on C:

To study the Q-wellposedness of (5.144), we define
⎧
⎪ Fn = Y × F equipped with the norm: (y, z)2n = 2n y2Y + z2F
⎨ ϕ : x ∈ C  φ(x), M φ(x) ∈ F ,
⎪
n   n
(5.145)
⎪
⎪ Z = y , z (noisy data),
⎩ (  
n n n
Z = ŷ, ẑ = ϕn (x̂) (error free attainable data),
250 CHAPTER 5. REGULARIZATION

and rewrite it as
1
x̂n minimizes Jn (x) = ϕn (x) − Zn 2n over C. (5.146)
2
A simple calculation shows that the quantities αm,n , αM,n , Rn , Θn associated
to C and ϕn by (5.135) are
αm,n = αm n > 0, (5.147)
1
αM,n = αM (2n + M ) ,
2 2 (5.148)
2n
Rn = R > 0, Rn → 0 when n → ∞, (5.149)
(2n + M2 )1/2
(2n + M2 )1/2
Θn = Θ → ∞ when n → ∞, (5.150)
n
which satisfy, for all n,
2n+1 Rn+1 n+1
2
≤ ≤ . (5.151)
n Rn n
We see from (5.147)–(5.149) that (5.146) is a linearly stable FC-problem,
which is good, but (5.150) shows also that its deflection Θn will become
larger than π/2 for n large enough, which is bad! However, Θn decreases to
Θ < π/2 when n → +∞, so one chooses 0 such that
1  − 1
(20 + M2 ) 2 ≥ (1 − Θ2 /(π/2)2 2 M ⇔ Θ0 ≤ π/2, (5.152)
which ensures that ϕ0 (C) is s.q.c. in F0 , and guarantees at least that (5.146)
is Q-wellposed for n = 0!

Localizing the State-Space Problem to Cn ⊂ C:

Then for n = 1, 2 . . . , we decide to control the deflection by limiting the size
of the attainable set, and replace in a final step (5.146) by
1
x̂n minimizes Jn (x) = ϕn (x) − Zn 2n over Cn , (5.153)
2
where we have restricted the search to the subsets Cn of C defined by
C0 = C (5.154)
& '
( n ≤ χC Rn
Cn = x ∈ C : ϕn (x) − Z for n = 1, 2 . . . , (5.155)
where 0 < χC < 1 will be chosen later.
Of course, this localization constraint cannot be implemented in prac-
( which by definition
tice, as the center of the ball is the error-free data Z,
is unknown! But we shall see that the algorithm can be so tuned that this
constraint is never active during the course of the resolution.
5.3. STATE-SPACE REGULARIZATION 251

Lemma 5.3.1 Let 0 satisfy (5.152), n , n = 1, 2, . . . satisfy

∃ 0 < μ < 1 such that 2n > 2n+1 ≥ μ 2n , (5.156)

and 0 < χC < μ be chosen such that

2χC π
1 ≤ . (5.157)
(1 − χC /μ) 2 2

Then for all n = 0, 1, . . . one has

Cn ⊃ Cn+1 , (5.158)
Cn is closed and convex in E, (5.159)
ϕn (Cn ) is closed and s.q.c. in Fn , (5.160)

where ϕn (Cn ) is equipped with the family of paths image of the segments of
Cn by ϕn .

Proof. We proceed by induction. The properties are clearly satisfied for n = 0.

We suppose they hold for some n ∈ IN, and we prove that they hold for n+1:

• Cn ⊃ Cn+1 : Let x ∈ Cn+1 be given. One has

( n ≤ n ϕn+1 (x) − Z
ϕn+1 (x) − Z ( n+1 ≤ χC n Rn+1 ≤ χC Rn ,
n+1 n+1

where we have used (5.151), so that x ∈ Cn .

• Cn+1 is convex: Let x0 , x1 ∈ Cn+1 be given, and denote by P the curve

t ∈ [0, 1]  ϕn+1 ((1 − t)x0 + tx1 ) ∈ Fn+1 , by p its reparameterization
as function of the arc length ν, by Ln+1 the arc length of P in Fn+1 ,
and by f the function

f (ν) = Z( − p(ν)2n+1 for 0 ≤ ν ≤ Ln+1 . (5.161)

We want to show that [x0 , x1 ] ⊂ Cn+1 , that is,

f (ν) ≤ (χC Rn+1 )2 for 0 ≤ ν ≤ Ln+1 . (5.162)

252 CHAPTER 5. REGULARIZATION

Derivating twice f gives, as in the proof of the median Lemma 6.2.6

 
f  (ν) = 2 1 − Z( − p(ν), a(ν)n+1 (5.163)
 
≥ 2 1 − sup Z( − p(ν)n+1 /Rn+1 (5.164)
0≤ν≤Ln+1
 
≥ 2 1 − sup Z( − p(ν)n /Rn+1 (5.165)
0≤ν≤Ln+1
 
≥ 2 1 − χC Rn /Rn+1 (5.166)
 
≥ 2 1 − χc /μ . (5.167)

We have used in the fourth line the fact that Cn+1 is included in the
convex Cn , and inequalities (5.151) and (5.156) in the last line. The
convexity property gives then
 χ Ln+1 − ν ν
c
f (ν)+ν(Ln+1 −ν) 1− ≤ f (0)+ f (Ln+1 ) (5.168)
μ Ln+1 Ln+1
≤ (χC Rn+1 )2

for 0 ≤ ν ≤ Ln+1 . The term 1 − χC /μ is strictly positive because of

(5.157), and (5.162) is proved.

• ϕ(Cn ) is s.q.c. in Fn+1 : Choosing ν = Ln+1 /2 in (5.168) gives

L2n+1 χc
(1− ) ≤ (χC Rn+1 )2 . (5.169)
4 μ

Hence the deflection Θn+1 of ϕn+1 (Cn+1 ) in Fn+1 satisfies

Ln+1 2χC π
Θn+1 ≤ ≤ 1 ≤ , (5.170)
Rn+1 (1 − χC /μ) 2 2

by choice of χC , and ϕn+1 (Cn+1 ) is s.q.c. in Fn+1 .

• Cn and ϕn (Cn ) are closed: C is closed, so the linear stability property of

ϕn – see (5.147) – implies that ϕn (C) is closed. Hence ϕn (Cn ) is closed
as the intersection of two closed sets, and Cn is closed as the preimage
of a closed set by a continuous mapping.
5.3. STATE-SPACE REGULARIZATION 253

Main Result
We can now summarize the hypothesis we have made so far and state the
main result of this section:

Theorem 5.3.2 Let the following hypotheses and notations hold:

• C and ϕ = M ◦ φ satisfy (5.135) and (5.137)

• The true data ẑ is attainable: ẑ = M ŷ with ŷ = φ(x̂) (5.133)

• The noisy data zn ∈ F satisfy zn − ẑF ≤ δn with δn → 0 (5.138)

• An a-priori guess y0 of the true state ŷ is available such that

ŷ − y0 = M ∗ w for some w ∈ F (5.141)

• A sequence of regularization parameters n → 0 is chosen such that

1  − 1
(20 + M2 ) 2 ≥ (1 − Θ2 /(π/2)2 2 M, (5.152)
∃λ ≥ 0 : ∀n, λ wF 2n ≥ δn , (5.142)
∃μ > 0 : ∀n, 2n > 2n+1 ≥ μ 2n , (5.156)

• A sequence Cn of subsets Cn of C is defined by

C0 = C, (5.154)
& '
( n ≤ χC Rn
Cn = x ∈ C : ϕn (x) − Z n = 1, 2, . . . , (5.155)
where 0 < χC < μ is chosen such that
1
0 < 2χC /(1 − χC /μ) 2 ≤ π/2 (5.157)

• The a-priori guess y0 is close enough to ŷ so that

χZ /(1 − χZ )1/2 ≤ μ χC and χZ < 1 − χC , (5.171)

where χZ is defined by

M  (λ + 1)2 + λ2 1/2

χZ = wF (5.172)
R 1 − Θ2 /(π/2)2

Then, for all n = 0, 1, 2, . . . , one has

1. Cn is closed and convex in E

254 CHAPTER 5. REGULARIZATION

2. The state-space regularized problem (5.153)

2n 1
x̂n minimizes Jn (x) = φ(x)−yn 2Y + M φ(x)−zn 2F over Cn , (5.173)
2 2
where yn is computed from y0 by the auxiliary problem
2n 1
yn ∈ Y minimizes y −y0 2Y + M(y)−zn 2F over Y (5.140)
2 2
is Q-wellposed in Fn : it has a unique solution x̂n , and Jn has no para-
sitic local minimum

3. The localization constraint are not active at x̂n

( n < χC Rn .
ϕn (x̂n ) − Z (5.174)
4. x̂n ∈ Cn+1 , so it can be used as initial guess for the (n + 1)-th optimiza-
tion

5. Jn is strictly convex on Cn for n ≥ 1

Proof. The two first points follow from Lemma 5.3.1, which shows that Cn is
convex, and that (5.173) is a FC/LD problem with an enlargement neighbor-
hood of size Rn given by (5.149) and a linearly stable problem as one can see
in (5.147) – and hence a Q-wellposed problem according to Theorem 4.4.1.
It remains to check that the distance of the data Zn = (yn , zn ) to ϕn (Cn ) is
strictly smaller than Rn . Inequality (5.143) and (5.142) gives
( 2n = 2n yn − ŷ2Y + zn − ẑ2F
Zn − Z (5.175)
& '
≤ 4n (λ + 1)2 + λ2 w2F
 R 2 & '
n
≤ (2n + M2 ) (λ + 1)2 + λ2 w2F
R
 R 2 & '
n
≤ (20 + M2 ) (λ + 1)2 + λ2 w2F
R
 R 2 2 2
n 2 (λ + 1) + λ
≤ M w2F
R 1 − Θ/(π/2) 2
2
= (χZ Rn ) ,
where we have used (5.149) and (5.152) in the third and fifth lines. Hence
dn (Zn , ϕn (Cn )) ≤ χZ Rn < Rn , (5.176)
which ends the proof of the two first points.
5.3. STATE-SPACE REGULARIZATION 255

We turn now to points three and four. Define

Z(n = ϕn (x̂n )
Pn : t  ϕn ((1 − t)x̂n + tx̂)
Ln = arc length of Pn in Fn
dn (t) = Zn − Pn (t)n
dn = sup0≤t≤1 dn (t).

By construction, dn has a minimum at t = 0, so we can apply the obtuse

angle Lemma 6.2.9
 dn  2
Zn − Z(n 2n + 1 − ( 2.
L ≤ Zn − Z
Rn n n

( n ≤ χC Rn < Rn with ϕn (Cn ) s.q.c., and

But dn (Zn , ϕn (Cn )) ≤ Zn − Z
Proposition 7.2.12 shows that t  dn (t) is an s.q.c. function. Hence dn =
( n ≤ χZ Rn , which shows that
max{dn (0), dn (1)} = Zn − Z
χZ
Ln ≤ Rn ≤ μ χC Rn .
(1 − χZ )1/2

But μ < 1 gives

( n ≤ Ln < χC Rn ,
Z(n − Z
which is (5.174), and point three is proved. To prove point four, we notice
that
( n+1 ≤ Z(n − Z
Z(n − Z ( n ≤ Ln < χC μ Rn ≤ χC Rn+1 ,

where we have used (5.151), and point four is proved.

We prove now point five. Hypothesis (5.135) on C, φ and the properties
(5.149) and (5.147) of ϕn show that, for x0 , x1 ∈ Cn , the second Gâteaux
derivative of Jn at x0 in the direction x1 − x0 is, with the usual meaning for
Vn and An ,

Jn (x0 )(x1 − x0 )2 = Vn (0)2n + ϕn (x0 ) − Zn , An (0)n ,

 ϕn (x0 ) − Zn n 
≥ Vn (0)2n 1 − ,
Rn
 ϕn (x0 ) − Zn n 
≥ 2n αm
2
1− .
Rn
256 CHAPTER 5. REGULARIZATION

But

ϕn (x0 ) − Zn n ≤ ( n +Z( − Zn n ≤ (χC + χZ )Rn < Rn .

ϕ (x ) − Z
! n 0"# $
≤ χC Rn for n ≥ 1

Hence Jn is positive definite over Cn , which proves the last point.

5.3.2 Incomplete Observation: Soft Analysis

We suppose in this section that C, φ verify only
⎧
⎪
⎪ E, Y = reflexive Banach spaces,
⎪
⎪
⎨ C = closed, convex, bounded subset of E,
φ : C  Y is weakly sequentially closed, that is, (5.177)
⎪
⎪
⎪
⎪ x  x in E with xn ∈ C, and φ(xn )  φ̂ in Y imply
⎩ n
x ∈ C and φ̂ = φ(x),

φ is continuously invertible at x̂ ∈ C, that is,
(5.178)
if φ(xn ) → φ(x̂) in Y then xn → x̂ in E,
where the symbols  and → denote, respectively, weak and strong conver-
gence, and that the observation operator M satisfies

M ∈ L(Y, F ), with F = normed linear space. (5.179)

In applications, M may be an imbedding, a restriction, a point evaluation,

or a boundary observation.
We suppose that the true data z = ẑ is attainable

∃ x∗ ∈ C : ẑ = ϕ(x∗ ) ⇔ ẑ = My ∗ ∈ F with y ∗ = φ(x∗ ) ∈ Y, (5.180)

but now x∗ and y ∗ can be multiply defined as M, and possibly φ, is not

necessarily injective. We also denote by zn ∈ F a sequence of noisy data
zn ∈ F, zn − ẑF ≤ δn , δn → 0 when n → ∞, (5.138)
which converges to ẑ, and by
y0 ∈ F (5.139)
∗
an a-priori guess for a “true” state y . The main difference with the previous
section is that, due to the underdetermination for y ∗ , it is not possible to
deduce from y0 an upgraded a-priori guess yn , which converges to one specific
5.3. STATE-SPACE REGULARIZATION 257

“true” state y ∗ ! There are, however, often natural choices for y0 : for example,
if ẑ represent pointwise data in a finite dimensional space F and Y is a
function space, then y0 can be an interpolation in Y of the pointwise data. If
both Y and F are function spaces with Y strictly embedded in F and ẑ ∈ F
but ẑ ∈/ Y , then y0 would arise from ẑ by a smoothing process, for example,
by solving one auxiliary problem (5.140).
Hence the state-space regularized problems are here (compare with
(5.173))

2n 1
x̂n minimizes Jn (x) = φ(x) − y0 2Y + M φ(x)−zn 2F over C, (5.181)
2 2
where the regularization parameters n are chosen such that

n > 0, n → 0, δn /n bounded. (5.182)

Theorem 5.3.3 Hypothesis and notations (5.177), (5.179), (5.180), (5.138),

(5.139), and (5.182). Let x̂n be a solution of (5.181) for n = 1, 2, . . . Then there
exists a weakly convergent subsequence of {x̂n }, and every weak limit x̂ of such
a sequence {x̂nk } satisfies
1. x̂ is a solution to the unregularized problem (5.132)

ϕ(x̂) = M φ(x̂) = ẑ.

2. When k → ∞,
φ(xnk )  φ(x̂) weakly in Y,

M φ(x̂nk ) − ẑF = O(nk ).

3. If δn /n → 0,
φ(xnk ) → φ(x̂) strongly in Y,
and x̂ is a state-space y0 -minimum-norm solution:
+ ,
φ(x̂) − y0 Y ≤ min φ(x) − y0 Y : x ∈ C and M φ(x) = ẑ .

4. If in addition φ satisfies the invertibility hypothesis (5.178) on C, then

x̂nk → x̂ strongly in E.
258 CHAPTER 5. REGULARIZATION

Proof. We follow the proof of [26]. Because of (5.180), there exists x∗ ∈ C

such that M φ(x∗ ) = ẑ, so we have for all n

2n 1 2 1
φ(x̂n ) − y0 2Y + M φ(x̂n )−zn 2F ≤ n φ(x∗ ) − y0 2Y + ẑ−zn 2F (5.183)
2 2 2 2
2n δ 2
≤ φ(x∗ ) − y0 2Y + n .
2 2
Boundedness of δn /n implies that φ(xn ) is bounded in Y . Since C is bounded
as well, there exists a weakly convergent subsequence of xn , again denoted
by xn , and (x̂, φ̂) ∈ E × Y such that (xn , φ(xn ))  (x̂, φ(x̂)) weakly in E × Y .
Then (5.177) implies that x̂ ∈ C and φ̂ = φ(x̂). Since M φ(xn ) → ẑ in F , it
follows also that M φ(x̂) = ẑ. This proves point 1 and the first assertion of
point 2.
Since zn − ẑF ≤ δn and δn/ n is bounded, it follows that

M φ(xn ) − ẑF ≤ n φ(x∗ ) − y0 Y + 2δn = O(n ),

which ends the proof of Part 2. Next we assume that δn /n → 0. Choosing
x∗ = x̂ in (5.183) gives

δ2
φ(x̂n ) − y0 2Y ≤ φ(x̂) − y0 2Y + 2n ,
n
and consequently

lim φ(x̂n ) − y0 Y ≤ φ(x̂) − y0 Y ≤ lim φ(x̂n ) − y0 Y .

This implies that φ(x̂n ) → ϕ(x̂) strongly in Y . Then (5.183) gives, for an
arbitrary element x∗ satisfying M φ(x∗ ) = ẑ,

φ(x̂) − y0 2Y = lim φ(x̂n ) − y0 2Y , (5.184)

n→+∞
+ δ2 ,
≤ lim φ(x∗ ) − y0 2Y + 2n , (5.185)
n→+∞ n
∗
= φ(x ) − y0 Y ,
2
(5.186)

and Part 3 is proved. Finally Part 4 follows immediately from the hypothesis
(5.178).
Because of the much weaker hypothesis involved on C, φ, and M,
Theorem 5.3.3 is more widely applicable than Theorem 5.3.2: it applies of
5.4. ADAPTED REGULARIZATION FOR EXAMPLE 4 259

course to the same examples (one-dimensional elliptic inverse problem of

Sect. 1.4, diffusion coefficient estimation problem in a 2D elliptic equation
of Sect. 1.6) for point or boundary observation, but also for problems where
Q-wellposedness for a state-space observation cannot be proved. Examples
can be found in the original paper [26].

5.4 Adapted Regularization for Example 4:

2D Parameter Estimation with H 1
Observation
As we have seen in Sect. 1.3.4, adapted regularization tries to supply only the
information on the parameter that cannot be retrieved from the data. It is
expected to create less bias in the regularized estimate, and is the most desir-
able regularization. But its implementation is possible only after the missing
information has been identified, which is a difficult task. So we illustrate
this approach on the 2D elliptic parameter estimation problem described in
Sect. 1.6, where this happens to be possible. Q-well-posedness of this problem
was studied in Sect. 4.9, where linear stability and finite curvature could be
obtained only at the price of reduction to finite dimension. We show now
that adapted regularization permits to restore at least linear stability for
the infinite dimensional parameter set C defined in (4.106). The material is
adapted from the two references [30, 21].
Let notations and hypothesis (1.65) and (4.105) through (4.107) hold,
so that Propositions 4.9.2 (a is linearly identifiable) and 4.9.3 (deflection
condition) hold. Let x0 , x1 ∈ C be given, and for all t ∈ [0, 1] denote by η(t)
the associated velocity given by (4.114), which we rewrite as
 
(a1 − a0 )∇ua(t) · ∇v = − a(t)∇η(t) · ∇v, for all v ∈ Y, (5.187)
Ω Ω

where a(t) = (1 − t)a0 + ta1 .

To study the linear stability property (4.39), one has first to equip the
parameter space E with a norm for which this stability has some chance to
hold. As can be seen in (4.111) for u and (5.187) for η, the diffusion coefficient
a appears always as a coefficient of ∇u in the equations. Though ∇u = 0 a.e.
in Ω, it can vanish at stagnation points (see the proof of Proposition 4.9.2),
and so there is little hope to obtain stability for a norm involving uniquely a.
260 CHAPTER 5. REGULARIZATION

So we shall rather consider stability for a gradient weighted “distance” in the

parameter space:
 1
dgrad (a0 , a1 ) = |(a1 − a0 )∇ua(t) |IL2 (Ω) dt. (5.188)
0

This quantity satisfies the two first axioms of a distance, but we do not know
whether it satisfies the third one (triangular inequality). This is why we have
written the word distance between quotes.

5.4.1 Which Part of a is Constrained by the Data?

To implement adapted regularization, we have  1 to understand which part of
dgrad (a0 , a1 ) is controlled by the arc length 0 |∇η(t)|IL2 dt in the data space
of the curve image by ϕ of the [a0 , a1 ] segment. Once this will be done, it will
be possible to introduce the missing information through the addition of an
ad-hoc regularization term.
Equation (5.187) for η suggests to define an equivalence relation ∼ of
vector fields in IL2 (Ω) by

q ∼ q if (q, ∇v) = (q , ∇v) for all v ∈ Y, (5.189)

where (·, ·) denotes the scalar product in IL2 (Ω), and to decompose accord-
ingly IL2 (Ω) into the sum of two orthogonal subspaces

IL2 (Ω) = G ⊕ G⊥ ,

where

G = IL2 (Ω)/∼ the quotient space
(5.190)
G⊥ the orthogonal complement,

with orthogonal projections P and P ⊥ .

Equation (5.187) becomes then

(a1 − a0 )∇ua(t) ∼ −a(t)∇η(t) ∀t ∈ [0, 1],

or equivalently
   
P (a1 − a0 )∇ua(t) = P − a(t)∇η(t) ,
5.4. ADAPTED REGULARIZATION FOR EXAMPLE 4 261

and hence  
|P (a1 − a0 )∇ua(t) |IL2 (Ω) ≤ aM |∇η(t)|IL2 (Ω) . (5.191)
Hence we see that |∇η(t)|IL2 (Ω) constrains only the norm of the component
in G of (a1 − a0 )∇ua(t) . There is no direct information on the norm of its
component in G⊥ , it is exactly this information that has to be supplied by
the regularization term.

Remark 5.4.1 Before providing the missing information through a regular-

ization term, one should investigate whether, by chance, the component in
G⊥ is not controlled by its component in G through the constraints that de-
fine the set C in (4.106). This would mean the existence of some M > 0 such
that

∀a0 ,a1 ∈ C one has   (5.192)
|P ⊥ (a1 − a0 )∇ua(t) |IL2 (Ω) ≤ M |P (a1 − a0 )∇ua(t) |IL2 (Ω) .

A partial counter example to this property has been given in [30] for a simple
diffusion problem on the unit square with a diffusion coefficient a = 1, no
source or sink boundaries inside the domain, and flow lines parallel to the
x1 -axis. It was shown there that for any perturbation h orthogonal to the flow
line – and hence function of x2 only – and satisfying
 1
h ∈ C ([0, 1]), h(0) = h(1) = 0,
0,1
h = 0, (5.193)
0

one has, for some constant c,

 2 ⊥
 
3
|h| L2 ([0,1]) ≤ |P  h∇u  |IL2 (Ω)
a
|P h∇ua |IL2 (Ω) ≤ 2c|h|H −1/2 ([0,1]) .

But given  > 0, one can always construct a sequence hn , n ∈ IN of pertur-

bations satisfying (5.193) and such that

|hn |L2 ([0,1]) =  > 0, |hn |H −1/2 ([0,1]) → 0, (5.194)

so that (5.192) cannot be satisfied for large n when a1 − a0 is replaced by hn .

However, for this construction to be a complete counter example to (5.192)
would require the existence for each n of a0,n , a1,n ∈ C and tn ∈ [0, 1] such
that
hn = a1,n − a0,n , a = 1 = (1 − tn )a0,n + tn a1,n . (5.195)
262 CHAPTER 5. REGULARIZATION

This in turn would require that hn satisfies the additional constraints

hn C 0 [0,1] ≤ aM − am , Lip(hn ) ≤ 2bM ∀n ∈ IN, (5.196)

where Lip(hn ) denotes the Lipschitz constant of hn .

Let g be a function on the real axis defined by
⎧
⎨ g ∈ C 0,1 (IR) with g periodic
1
of period one,
g(0) = g(1) = 0, g = 0, (5.197)
⎩ 0
g not identically zero,

and define a sequence of perturbations hn by

hn (x2 ) = g(nx2 ) ∀x2 ∈ IR. (5.198)

1
This sequence satisfies (5.193) and (5.194) with  defined by 2 = 0 g 2. It
satisfies also the left part of the additional constraints (5.196), provided g is
chosen such that gC 0 [0,1] ≤ aM − am , which is always possible. However,
the Lipschitz constant of hn is Lip(hn ) = nLip(g), so that the right part
of (5.196) will necessarily be violated for n large enough. Hence a sequence
hn defined by (5.197) and (5.198) does not contradict (5.192), as it cannot
satisfy (5.196) and hence cannot be of the form (5.195).
It is not known whether there exists a sequence hn that satisfies (5.193),
(5.194), and (5.195), and hence would contradicts (5.192). On the other
side, there is for the time no proof that (5.192) holds true, and property
(5.192), which would imply the linear stability for the dgrad “distance” on the
infinite dimensional set C, remains undecided.

5.4.2 How to Control the Unconstrained Part?

To seewhich regularization
 term can be used to bring the missing information
on P ⊥ (a1 −a0 )∇ua(t) , we use the grad-rot decomposition of IL2 (Ω) = G⊕G⊥ .
We introduce for this the space

W = {ψ ∈ H 1 (Ω) : ψ|∂ΩN = 0},


(where the condition Ω ψ = 0 is added to the definition of W in the case
where the Neuman boundary ∂ΩN is empty), and recall the definition of
rotational
5.4. ADAPTED REGULARIZATION FOR EXAMPLE 4 263
⎛ ∂ϕ ⎞
∂x2
∀ϕ ∈ H 1 (Ω),  ϕ = ⎝
rot ⎠,
− ∂x
∂ϕ
1

 ∈ H 1 (Ω) × H 1 (Ω),
∀ψ  =
rot ψ ∂ψ2
− ∂ψ1
.
∂x1 ∂x2

Lemma 5.4.2 Let (1.65) and (4.105) through (4.107) and (5.189) and
(5.190) hold. Then

G = {∇ϕ : ϕ ∈ Y }, G⊥ = {rot
 ψ : ψ ∈ W }, (5.199)

and, for every q ∈ IL2 (Ω), one has

P q = ∇ϕ, P ⊥ q = rot
 ψ,

where ϕ ∈ V and ψ ∈ W are given by


(∇ϕ, ∇v) = (q, ∇v) for all v ∈ Y,
(5.200)
 ψ, rot
(rot  w) = (q, rot
 w) for all w ∈ W.

Proof. Except for the atypical boundary conditions, this decomposition is

rather standard. A outline of the proof, which is adapted from [39], Chap. 1,
can be found in [30].
Let then ∇φ, rot
 ψ be the grad-rot decomposition of (a1 −a0 )∇ua(t) given
 
by Lemma 5.4.2. We evaluate |P ⊥ (a1 − a0 )∇ua(t) |IL2 (Ω) = |rotψ|
 IL2 (Ω) :

|rotψ|
 IL2 (Ω) = sup 
(rotψ, s)
s∈IL2 (Ω),|s|IL2 (Ω) =1


= sup 
(rotψ, ∇v + rotw)

v∈Y,w∈W,|∇v|2 2 
+|rotw| 2 =1
IL (Ω) IL2 (Ω)

= sup 
(rotψ, 
rotw)
w∈W,|∇w|IL2 (Ω) =1

|rotψ|
 IL2 (Ω) = sup (a1 − a0 )∇ua(t) · rotw,
 (5.201)
w∈W,|∇w|IL2 (Ω) =1 Ω

where we have used the grad-rot decomposition ∇v + rot  w of s, the orthog-
⊥
onality of G and G , the fact that |rot w| = |∇v|, and the Definition (5.200)

264 CHAPTER 5. REGULARIZATION

of ψ. Because of the smoothness hypotheses (4.105) and (4.106) on the pa-

rameters, the gradient – and hence the rotational – of u are in IL∞ (Ω), and
we can use the Green’s formula in the right-hand side:
 
(a1 − a0 )∇ua(t) · rot w =
  1 − a0 ) · ∇ua(t) w (5.202)
rot(a
Ω Ω

∂ua(t)
− (a1 − a0 ) w.
∂Ω ∂τ
where ∂/∂τ denotes the tangential derivative along ∂Ω. Since ua = 0 on ∂ΩD
and ua = const on each ∂Ωi , we have ∂ua /∂τ = 0 on ∂ΩD and ∂Ωi , i =
1, · · · , N, and w = 0 on ∂ΩN , so that the boundary term vanishes in (5.202).
Combining (5.201) and (5.202) one obtains

⊥
|P ((a1 − a0 )∇ua(t) )|IL2 = sup  1 − a0 ) · ∇ua(t) w,
rot(a
w∈W, |∇w|IL2 (Ω) =1 Ω

which, by Poincarés inequality in W ,

|w|L2 (Ω) ≤ CW |∇w|IL2 (Ω) ∀w ∈ W (5.203)

shows that
 
|P ⊥ (a1 − a0 )∇ua(t) |IL2 (Ω) ≤ CW |rot(a
 1 − a0 ) · ∇ua(t) |L2 (Ω) . (5.204)

This shows that the part P ⊥ ((a1 − a0 )∇ua(t) ) of parameter perturbation,

 1 −a0 )·∇ua(t) ,
which is not controlled by the data ∇ua(t) , is controlled by rot(a
that is, by the variation of the diffusion coefficient orthogonal to the flow
lines. This observation is in phase with the intuition that the knowledge of
the pressure field ∇ua(t) gives little information on the diffusion parameter a
orthogonal to flow lines.

5.4.3 The Adapted-Regularized Problem

In absence of more specific a-priori information, one can then decide to search
for diffusion coefficients which, loosely speaking, are “smooth orthogonal to
the flow lines.” This can be done by replacing the forward map ϕ defined in
(4.112) by the adapted-regularized forward map:

ϕ : a  (∇ua(t) ,  rot
 a · ∇ua ) ∈ IL2 (Ω) × L2 (Ω), (5.205)
5.4. ADAPTED REGULARIZATION FOR EXAMPLE 4 265

where  > 0 is the regularization parameter. The corresponding adapted-

regularized problem is then (compare to (4.113)

1 2 
â minimizes |∇ua − z|2IL2 + |rota· ∇ua |2L2 over C. (5.206)
2 2

5.4.4 Infinite Dimensional Linear Stability

and Deflection Estimates
As expected, problem (5.206) is linearly stable:

Proposition 5.4.3 Let (1.65) and (4.105) through (4.107) and (5.188) hold.
Then the adapted-regularized forward map (5.205) satisfies the linear stability
estimate (compare with (4.39)

∀a0 , a1 ∈ C one has
1 (5.207)
αm () dgrad (a0 , a1 ) ≤ 0 V (t)IL2 ×L2 dt,

where dgrad is defined in (5.188), and where

+ 2 1 ,
αm ()2 = Min 2
, 2
, (5.208)
2CW (1 + 2CW )a2M

with CW equal to the Poincaré constant for the space W defined in (5.203).

Proof. Let a0 , a1 ∈ C. For t ∈ [0, 1], derivation with respect to t of ϕ evalu-

ated at a(t) = (1 − t)a0 + ta1 shows that the velocity V (t) of the adapted-
regularized problem is
  
V (t) = ∇η(t),  rot  a(t) · ∇η(t) + rot (a1 − a0 ) · ∇ua(t) , (5.209)

where η(t) is given by (5.187).

On the other hand, we obtain from the orthogonal decomposition G ⊕ G⊥
of IL2 (Ω) that

|(a1 − a0 )∇ua(t) |2IL2 = |P ((a1 − a0 )∇ua(t) )|2IL2 + P ⊥ ((a1 − a0 )∇ua(t) )|2IL2 ,

and, using (5.191) and (5.204),

|(a1 − a0 )∇ua(t) |IL2 ≤ |(aM ∇η(t), CW rot(a

 1 − a0 ) · ∇ua(t) )|IL2 ×L2 . (5.210)
266 CHAPTER 5. REGULARIZATION

We define now, for each t ∈ [0, 1], a linear mapping Gt from IL2 × L2 into
itself by ⎧
⎨ Gt (q, v) = (aM q , CW (v/ − rot  a(t) · q)),
which satisfies (5.211)
⎩
αm ()Gt (q, v)IL2 ×L2 ≤ (q, v)IL2 ×L2 ,
so that
G(V (t)) = (aM ∇η(t), CW rot(a
 1 − a0 ) · ∇ua(t) ). (5.212)
Combining then (5.210) with (5.212) and (5.211) gives

αm ()|(a1 − a0 )∇ua(t) |IL2 ≤ V (t)IL2 ×L2 , (5.213)

which gives (5.207) after integration between 0 and t.

So we see that adapted regularization ensures the stability of the diffusion
coefficient a in the infinite dimensional set C for the L2 norm weighted by
|∇ua |, which is what one could reasonably expect at best. As we have seen in
the proof of Proposition 4.9.2, |∇ua (x)| can only vanish on a set of zero mea-
sure. In this case, the weighted stability constrains a almost everywhere on
Ω, but with a strength that decreases when one approaches one (necessarily
isolated) stationary point where |∇ua (x)| = 0.

Remark 5.4.4 Unweighted L2 stability of a can only be obtained on subsets

of Ω where a uniform lower bound γ to |∇ua | exists. In the very particular
case where there is only one source or sink boundary ∂Ω1 with a flow rate
Q1 = 0, and where the remaining boundary is equipped with a Dirichlet con-
dition (in short ∂ΩN = ∅), such a γ ≥ 0 exists over the entire domain Ω, so
that
γ|a1 − a0 |L2 (Ω ) ≤ dgrad (a0 , a1 )
and the unweighted L2 (Ω)-stability holds for a in the infinite dimensional
set C.

The next step towards OLS-identifiability is to check whether the deflection

condition Θ ≤ π/2 can be satisfied by the adapted regularized problem.
The velocity V for this problem has been already given in (5.209), and the
acceleration is
  
A (t) = ∇ζ(t),  rot a(t) · ∇ζ(t) + 2rot (a1 − a0 ) · ∇η(t) ,
  (5.214)

where η(t) is given by (5.187).

5.4. ADAPTED REGULARIZATION FOR EXAMPLE 4 267

Proposition 5.4.5 Let (1.65) and (4.105) through (4.107) hold. The de-
flection condition Θ ≤ π/2 is satisfied for the adapted-regularized problem
(5.206) as soon as
 1/2 π
(aM − am )2 + 2 b2M (aM + am )2 ≤ am , (5.215)
4
where am , aM , and bM are the constants that define the admissible parameter
set C in (4.106).
Proof. We use (5.214) to evaluate the norm of A (t) to see whether it can
satisfy the deflection sufficient condition (4.25)

a2m A (t)2F ≤ a2m |∇ζ(t)|2IL2 + a2m 2 ∇a(t)IL∞ |∇ζ(t)|IL2 (5.216)
2
+2∇(a1 − a0 )IL∞ |∇η(t)|IL2
 
≤ 4a1 − a0 2C 0 + 2 2bM a1 − a0 C 0
2 
+2am ∇(a1 − a0 )IL ∞ |∇η(t)|2IL2 ,

wherewehaveusedthemajoration(4.120) of |∇ζ(t)|IL2 by2a1 −a0 C 0 |∇η(t)|IL2 .

Using the Definition (4.106) of C one obtains
 1/2
am A (t)F ≤ 2 (aM − am )2 + 2 b2M (aM + am )2 |∇η(t)|IL2 . (5.217)

But (5.209) shows that |∇η(t)|IL2 ≤ V F , so that (4.25) follows from (5.215)
and (5.217), which ends the proof.
This is as far as one can currently go for the problem (5.206) on the infinite
dimensional set C.

5.4.5 Finite Curvature Estimate

It is an open problem to know whether a finite curvature estimate (4.13)
exists for the infinite dimensional admissible parameter set C. So one has to
regularize one more time the problem by reduction to finite dimension:
Proposition 5.4.6 Let (1.65) and (4.105) through (4.107) hold, and let C
be the finite dimensional parameter set defined in (4.121). Then the adapted-
regularized problem (5.206) has a finite curvature over C , given by
1 2
= M(, C ), (5.218)
R am αm ()
268 CHAPTER 5. REGULARIZATION

where αm () given in (5.208) is independent of the dimension of C , and

where M(, C ) is defined by
M(, C ) = sup Ma0 ,a1 , t , (5.219)
a0 ,a1 ∈C , t∈[0,1]

where
a1 − a0 L∞  ∇(a1 − a0 )L∞ 2 1/2
2
Ma0 ,a1 , t = 1 +  (bM + am ) .
|(a1 − a0 )∇ua(t) |IL2 a1 − a0 L∞
When C is defined through a finite element approximation with mesh size
h > 0, the curvature 1/R blows up to infinity like 1/h2 .
Proof. From (5.216) and |∇η(t)|IL2 ≤ V F one obtains, with the notation
 · L∞ instead of  · C 0 ,

am A (t)F ≤ 2 a1 − a0 2L∞
 2 1/2
+2 bM a1 − a0 L∞ + am ∇(a1 − a0 )IL∞ V (t)F .

Combining with (5.213) gives

2
A (t)F ≤ Ma0 ,a1 , t V (t)2F ,
am αm ()
which proves (5.218) using the finite curvature estimate (4.13).
When C is defined via finite elements with mesh size h, the ratios
 · L∞ / · L2 and ∇ · L∞ / · L∞ blow up like 1/h when h → 0, which
proves the last part of the theorem.

5.4.6 OLS-Identifiability for the Adapted Regularized

Problem
One can now combine Propositions 5.4.3, 5.4.5, and 5.4.6 with Theorem 4.4.1
to obtain finite dimensional OLS-identifiability for the adapted regularized
problem. We shall need the diameter D of the attainable set for the adapted-
regularization term
√ N
D = sup |( rot
 a · ∇ua )|L2 ≤ bM 2 Ci |Qi |, (5.220)
a∈C
i=1

where Ci is the continuity constant of the linear mapping v ∈ Y  v|∂Ωi ∈ IR.

5.4. ADAPTED REGULARIZATION FOR EXAMPLE 4 269

Theorem 5.4.7 Let (1.65) and (4.105) through (4.107) hold, and suppose
that

1. The deflection condition (5.215) is satisfied

2. The admissible parameter set C is finite dimensional as in (4.121)

Let 1/R < +∞ be the curvature of the adapted-regularized problem given by

(5.218), D the diameter introduced in (5.220), and suppose that  is such
that
0 <  < ¯ , where  is defined by ¯D = R. (5.221)
Then a is OLS-identifiable via the adapted-regularized problem (5.206) on the
finite dimensional parameter set C for the dgrad “distance” in E:

1. Existence, uniqueness, and stability:

for any z0 , z1 in ϑ = {z ∈ IL2 (Ω) : dIL2 (z, ϕ(C )) < R −  D} (5.222)

such that
z0 − z1 IL2 + max dIL2 (zj , ϕ(C )) + D ≤ d < R, (5.223)
j=0,1

the least-squares problem (5.206) stated on C admits unique solutions

âj , j = 0, 1, which satisfy

d −1
αm () dgrad (â1 − â0 ) ≤ (1 − ) z1 − z0 IL2 , (5.224)
R
where αm () is given by (5.208) and dgrad by (5.188).

2. Optimizability:
∀z ∈ ϑ, the least squares problem (5.206) has no parasitic local mini-
mum over C .

Proof. Propositions 5.4.3, 5.4.5, and 5.4.6 ensure that the hypotheses of The-
orem 4.4.1 are satisfied for the adapted regularized problem (5.206). Hence
existence, uniqueness, stability, and optimizability will hold for this problem
as soon as the distance of the regularized data z = (z, 0) to the regularized
attainable set is strictly less than R:
 
dF (z, 0), ϕ (C ) < R. (5.225)
270 CHAPTER 5. REGULARIZATION

Using the Definition 5.220 of D we obtain

   
dF (z, 0), ϕ (C ) =  · ∇ua |L2 1/2
inf z − ϕ(a)2IL2 + 2 |rota
a∈C
 1/2
≤ inf z − ϕ(a)2IL2 + 2 D 2
a∈C
 1/2
≤ inf z − ϕ(a)2IL2 + 2 D 2
a∈C
≤ inf z − ϕ(a)IL2 + D
a∈C
 
≤ dIL2 z, ϕ(C ) + D.
 
Hence condition (5.225) is satisfied as soon as dIL2 z, ϕ(C ) ≤ R − D, and
the announced existence, uniqueness, stability, and optimizability properties
are simply a rewriting of (4.42)–(4.44) from Theorem 4.4.1.
 Part II

A Generalization of Convex
Sets

271
PART II: A GENERALIZATION OF CONVEX SETS 273

Chapters 6 and 7 give a comprehensive presentation of the theory of

quasi-convex (q.c.) and strictly quasi-convex (s.q.c.) sets, which retain the
desirable properties of convex sets listed at the beginning of Chap. 4.
Chapter 8 develops deflection-sufficient conditions for the strict quasi-
convexity of a set, which are then applied to the case of the attainable set of
a nonlinear least squares problem.
The presentation of s.q.c sets given there is different – and hopefully
more natural – than that in the original material [20, 19, 28]. These chapters
can be read independently of the rest of the book if one is interested in the
projection on nonconvex sets in Hilbert spaces.
Chapter 6

Quasi-Convex Sets

In this chapter, we define a new class of subsets of a Hilbert space, called

the quasi-convex sets to which properties (i) (uniqueness), (iii) (stability)
and (iv) (existence as soon as the set is closed) of Proposition 4.1.1 can be
generalized, provided they are required to hold only on some neighborhood.
Technically, the whole chapter will consist in adapting the classical proofs for
convex sets to the case where the segments are replaced by paths with finite
curvature.
We postpone to Chap. 7 the generalization of property (ii) on the absence
of parasitic stationary points.
There exists already many results concerning the projection on nonconvex
sets. For example, given η > 0, a point X is called an η-projection of some
point z ∈ F on D if it satisfies

X ∈ D, X − z ≤ d(z, D) + η. (6.1)

So if one defines the so-called Edelstein set of D by

+
E(D) = z ∈ F | ∀ε > 0, ∃η > 0 such that (6.2)
,
X0 , X1 = η − projections of z =⇒ X0 − X1  ≤ ε ,

then clearly any z ∈ E(D) has the property that any sequence that minimizes
the distance to z over D is a Cauchy sequence. This implies that, when D is
closed, all points of the Edelstein set E(D) have a unique projection on D.
The interesting result is (cf, e.g., Aubin 1979) that this set fills out almost
G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 275
and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 6, c Springer Science+Business Media B.V. 2009
276 CHAPTER 6. QUASI-CONVEX SETS

completely F , in the sense that it is a dense countable intersection of open

sets. This result alone does not allow to generalize Proposition 4.1.1 as desired
for at least two reasons: the Edelstein result does not guarantee that the “bad
points,” which have no or more than one projection, will stay outside the
neighborhood of D: the set E(D) will, in general, contain no neighborhood
of D.
The second reason is that there is no guarantee, for a point z ∈ E(D),
that the distance to z has no parasitic stationary points, which is one of the
properties of convex sets we want to generalize.
So our generalizations of convex sets, the quasi-convex sets of this chapter
and the strictly quasi-convex sets of Chap. 7, will be chosen in such a way that
their Edelstein set E(D) contains a neighborhood ϑ of D. This will ensure
an easy generalization of the properties (i) and (iv) of Proposition 4.1.1
(uniqueness and existence of the projection). Then (ii) and (iii) (stationary
points and stability) will be generalized by other arguments.

In the applications to inverse problems we have in mind, D is the attain-

able set
D = {ϕ(x)|x ∈ C}, (6.3)
image of the usually closed and convex admissible parameter set C by the
nonlinear mapping ϕ to be inverted. So it will be easy, in the applications,
to draw curves on D using the above parametric representation of D, for
example, the images by ϕ of the segments of C. We shall call such a curve
a “path,” which, under certain conditions to be made precise at the end of
Chap. 8, will have a finite curvature.
We take advantage of this at the abstract level, and suppose throughout
the present chapter and the next one that the set D of F is equipped with
a family P of finite curvature paths connecting any two points of D. These
paths will play the role of the segments for D in our proofs when D happens
to be convex. The family of paths P associated with the set D ⊂ F will be
defined in an axiomatic way, without reference to C and ϕ. So the results on
quasi-convex and strictly quasi-convex sets can be used for other applications
than for the analysis of nonlinear least squares problems.

The organization of the chapter is as follows:

• In Sect. 6.1, we define precisely the axioms that a family P of paths
equipping D has to satisfy, as well as the geometric attributes that are
naturally associated with a path of P (arc length, curvature, etc. . . ).
6.1. EQUIPPING THE SET D WITH PATHS 277

• In Sect. 6.2, we define the property, for a set D equipped with a family of
paths P – in short a set (D, P) – to be quasi-convex. Then we re-do, for
such a set, all classical proofs for the projection of a point on a convex
set, with the necessary adaptations. This leads to the conclusion that
the properties (i), (iii) and (iv) of Proposition 4.1.1 can be generalized
to neighborhoods of quasi-convex sets. However, parasitic stationary
points may still exist, and (ii) does not generalize to quasi-convex sets.

6.1 Equipping the Set D with Paths

The first step of our construction consists in choosing, in the possibly noncon-
vex set D, a family P of paths p, which will play for D the role the segments
play for a convex set.
In a convex set, the segments are “pieces of straight line,” and so have a
zero curvature. We shall relax this property by requiring only that a path p
of D is a “piece of a curve with finite curvature”:

Definition 6.1.1 Given  > 0, a curve p : [0, ] → F is a path of D if and

only if
p ∈ W 2,∞ ([0, ]; F ) (6.4)
p(ν) ∈ D ∀ν ∈ [0, ] (6.5)
p (ν)F = 1 for a.e. ν ∈ [0, ] (6.6)

In (6.4), W 2,∞ ([0, ]; F ) is the space of function from [0, ] into F , whose two
first distributional derivatives happen to be L∞ ([0, ]; F ) functions.
Condition (6.5) ensures that the path p stays in the set D. Then by definition
of the arc length along p, one has
 ν
arc length from p(0) to p(ν) = p (ν)F dν, (6.7)
0

which in the view of (6.6) can be rewritten simply as

arc length from p(0) to p(ν) = ν. (6.8)

Conversely, if ν is defined as the arc length along p, then necessarily (6.6) will
hold (the proof will be given in Proposition 8.2.1 below). So the hypothesis
6.6 simply means that we have chosen to parameterize our paths p by their
278 CHAPTER 6. QUASI-CONVEX SETS

arc length ν, which brings considerable simplification in the evaluation of the

geometric quantities associated with the path p:
Definition 6.1.2 Let a path p be given as in Definition 6.1.1. Then ν ∈ [0, ]
is the arc length along p, and
def
the (arc) length of p is L(p) = , (6.9)
the velocity v(ν) = p (ν) is the unit tangent vector to p at p(ν),
def
(6.10)
the acceleration a(ν) = p (ν) is the main normal to p at p(ν),
def
(6.11)
the radius of curvature of p at ν is ρ(ν) = a(ν)−1
def
F , (6.12)
where ρ(ν) can possibly be infinite.
So we see that, with the above choice of parametrization, the condition
(6.4) on p can be rewritten as
1/ρ ∈ L∞ ([0, ]),
which is equivalent to say that p is a path with finite curvature.
Now that we have relaxed the “zero-curvature condition” for the paths
of D, we want to equip D with a collection of paths, which retains the other
properties of the segments of a convex set, namely the following:
• There is always one segment connecting any two distinct points of a
convex
• any subsegment of a segment is a segment
So we are led to the following axiomatic definition of a family of paths for D:

Definition 6.1.3 (Family of paths). A set of curves P is a family of paths

on D if and only if
P is made of paths on D : (6.13)
all curves p of P satisfy Definition 6.1.1;
P is complete on D : (6.14)
∀X, Y ∈ D, X = Y,
∃p ∈ P such that p(0) = X and p(L(p)) = Y ;
P is stable with respect to restriction : (6.15)
   
∀p ∈ P, ∀ν , ν ∈ [0, L(p)], ν < ν , the path
p̃ : ν ∈ [0, ν  − ν  ] → p(ν  + ν) belongs to P.
6.1. EQUIPPING THE SET D WITH PATHS 279

From now on, we shall always consider that the set D ⊂ F is equipped with a
family of paths P, which we shall write as (D, P). The notions of quasi-convex
and strictly quasi-convex sets will be developed for such couples (D, P), and
hence depend on the choice made for the family of paths P, which equips D.
We discuss now possible choices for the family of paths P:

• When D is convex, one can always take P = {segments of D}, and

for this choice, all the results on quasi-convex sets (this chapter) and
strictly quasi-convex sets (next chapter) will reduce to their classical
convex counterparts.

• When D is nonconvex, a first natural idea, which is the direct gener-

alization of one possible definition of the segments in a convex, is to
define P as the collection of all minimum-length paths connecting any
two points of D. The difficulty with this choice will be to check that
the regularity properties of Definition 6.1.1 (in particular finite curva-
ture!) are satisfied by the minimum length paths. This choice, when
it is possible, is undoubtedly the most intrinsic one; the radii of cur-
vature of the paths give in that case a direct information on those of
the “manifold ” D (we use quotes because we shall carefully avoid any
rigorous developments involving D as a differential manifold), and so
one can expect that the size×curvature conditions to be derived later
in Chap. 8 will then give the most precise characterization of strict
quasi-convexity. However, the choice of the geodesics for P is not nec-
essarily the best, even when D is convex (e.g., in the case where one
searches for the best Lipschitz constant for the projection on a con-
vex set, and where one wants to take into account the curvature of
the boundary of D). The use of a geodesic or adapted collection of
paths P is a completely open problem, which we do not consider in
this book.

• In applications to nonlinear inverse problems, we will take advantage

of D being the attainable set ϕ(C) of the mapping ϕ to be inverted
on C (see (6.3)), and choose for P the image by ϕ of all segments
of C:
P = {ϕ([x, y]), x, y ∈ C}. (6.16)
280 CHAPTER 6. QUASI-CONVEX SETS

It will also be sometimes convenient to suppose the existence in P of a

generating family of paths PG , which mimics the family of segments connect-
ing two points of the boundary of a convex set:
Definition 6.1.4 (Generating family of paths). A subset PG of P is said to
be a generating family of paths for P if and only if
P = ∪p∈PG {p |p is a subpath of p}. (6.17)
In the applications to nonlinear inverse problems (Chaps. 4 and 5, where
D is defined by (6.3) and P by (6.16)), a generating family of path is given by
PG = {ϕ([x, y]), x, y ∈ ∂C}. (6.18)
We conclude this section by indicating how the choice of a collection of
paths P on D leads naturally to the definition of an “arc length distance”
on D:
Definition 6.1.5 Let (D, P) be given. Then for any X, Y ∈ D, we call arc
length distance in D of X and Y the quantity
δ(X, Y ) = sup L(p), (6.19)
p∈P
p:X →Y
with the convention that
δ(X, Y ) = 0 if there is no path from X to Y (6.20)
Because of (6.14), (6.20) can arise only at points X, Y such that Y = X. But
there may exists points X and paths p with nonzero length going from X to
X, so that one would have at such points
δ(X, X) > 0. (6.21)
Such paths will not occur if (D, P) is quasi-convex (Proposition 6.2.5 below),
and a-fortiori if it is in the smaller class of strictly quasi-convex (s.q.c.) sets
of Chap. 7. Notice that one always has
X − Y  ≤ δ(X, Y ), ∀X, Y ∈ D (6.22)
and that δ does not necessarily satisfy the axiom of a distance! The reason
for the sup in Definition 6.1.5 is that it will enable us to write in a simple
way (see Proposition 6.2.11) the stability results for the projection on quasi-
convex sets and s.q.c. sets, where the distance between the two projections
will be measured by the length L(p) of any path p of P connecting the two
projections.
6.2. DEFINITION AND MAIN PROPERTIES 281

6.2 Definition and Main Properties

of q.c. Sets
The word “quasi-convex” has been used with various meaning, especially
when applied to functions. We use it here to qualify sets that share some of
the properties of convex sets.
So let us now consider a set D of F equipped with a collection P of paths
according to Definition 6.1.3.
Given a point z ∈ F , and a path p ∈ P, we study the properties of the
“distance to z along p” defined by

dz,p (ν) = z − p(ν)F , ∀ν ∈ [0, L(p)] (6.23)

(when D is convex and p is a segment, d2z,p is a strictly convex function).

To that purpose, we introduce a quantity k(z, p; ν) ∈ IR, which will in-
dicate where the projection H of z on the main normal to p at M = p(ν)
is located with respect to the center of curvature C of p (see Fig. 6.1). If we
define
MH
k(z, p; ν) = ,
MC
where MH and MC denote the algebraic measures on the oriented normal,
the condition
k(z, p; ν) < 1 (6.24)
corresponds to the “good” situation where the point H is as follows:

• Either on the “convex side” of p (k(z, p; ν) ≤ 0)

• Or on the “concave side” of p, but at a distance smaller than the radius

of curvature (0 ≤ k(z, p; ν) < 1).

So one can expect that the situation deteriorates when k approaches 1, as

this corresponds to H approaching C, and quasi-convex sets will be obtained
by requiring that k stays uniformly away from 1. We give first a simple
expression for k(z, p; ν) in terms of z and p. With the notation of Fig. 6.1,

ρ(ν) = MC = radius of curvature of p at M,

γ = angle between z − p(ν) and MC,

282 CHAPTER 6. QUASI-CONVEX SETS

z dz,p(ν)
1111
0000
0000
1111
0000
1111
0000
1111 path p
0000
1111
0000
1111
C

H γ
ρ(ν)
M = P (ν)
main normal to p

Figure 6.1: Notations for formula (6.25)

k(z, p; ν) can be rewritten as

dz,p (ν)
k(z, p; ν) = cos γ, (6.25)
ρ(ν)
that is, using the fact that a(ν) = p (ν) is a normal vector with length 1/ρ(ν)
and pointing to the center of curvature C (see Definition 6.1.3):
k(z, p; ν) = z − p(ν), a(ν). (6.26)
First derivation of d2z,p with respect to ν gives
d
(d2 )(ν) = − 2z − p(ν), v(ν),
dν z,p
and second derivation gives, as v(ν) = 1,
d2  
(d 2
)(ν) = 2 1 − k(z, p; ν) . (6.27)
dν 2 z,p
Then, given z and p, we shall consider the worst case when ν varies in
[0, L(P )], which in view of (6.24) is obtained by setting
k(z, p) = sup k(z, p; ν). (6.28)
ν∈[0,L(p)]

Finally, given z and η > 0, we will consider the worst case for all paths p ∈ P,
which connect two η-projections of z on D, that is, which are in the subset
P(z, η) = {p ∈ P| p(j) − zF ≤ d(z, D) + η, j = 0, L(p)} (6.29)
6.2. DEFINITION AND MAIN PROPERTIES 283

of P. This leads to define

k(z, η) = sup k(z, p). (6.30)

p∈P(z,η)

The function η → k(z, η) is nondecreasing, and so it has a right limit at

η = 0, which we shall denote by k(z, 0):

k(z, 0) = lim k(z, η). (6.31)

η→0+

By construction, k(z, 0) satisfies

k(z, 0) ≤ k(z, η), ∀η > 0. (6.32)

Definition 6.2.1 (Quasi-convex set). Let D ⊂ F be equipped with a fam-

ily of path P. The set (D, P) is quasi-convex if and only if there exists
a neighborhood ϑ of D in F and a lower semi-continuous (l.s.c.) function
ηmax : ϑ →]0, +∞], such that

∀z ∈ ϑ, ∀η, such that 0 < η < ηmax (z), one has
(6.33)
k(z, η) < 1

or equivalently

∀z ∈ ϑ, ∀η, such that 0 < η < ηmax (z), one has
(6.34)
d2z,p is uniformly α − convex over P(z, η).

We shall call ϑ a regular (q.c.) neighborhood of (D, P).

The equivalence between the conditions (6.33) and (6.34) results from

d2  
2
(d2z,p)(ν) = 2 1 − k(z, p; ν) , (6.35)
dν
≥ 2(1 − k(z, p)), (6.36)
≥ 2(1 − k(z, η)), ∀p ∈ P(z, η). (6.37)

We shall see soon that, as its name suggests, the neighborhood ϑ is the
one on which the “projection on D” is well-behaved. But before getting into
284 CHAPTER 6. QUASI-CONVEX SETS

this, we check first that Definition 6.2.1 ensures the existence of a largest
open regular neighborhood ϑ:
Proposition 6.2.2 Let (D, P) be quasi-convex. Then there exists a largest
open regular neighborhood ϑ of D, and a largest l.s.c. function ηmax : ϑ →
]0, +∞] satisfying the definition 6.2.1 of quasi-convex sets.

Proof. Let us denote by ϑi , ηmax,i , i ∈ I all open neighborhoods and l.s.c.

functions satisfying Definition 6.2.1. Then

ϑ = ∪i∈I ϑi (6.38)

is an open neighborhood of D. If we denote by η̃max,i the extension of ηmax,i

to ϑ by zero outside ϑi , then η̃max,i is l.s.c. as ηmax,i is l.s.c. on ϑi and ϑi is
open. Define then

∀z ∈ ϑ : ηmax (z) = sup η̃max,i (z), (6.39)

i∈I

which is l.s.c. as a supremum of a family of l.s.c. functions. Hence ϑ, ηmax will

satisfy the definition of quasi-convexity as soon as they satisfy (6.33), which
we prove now. Let z ∈ ϑ and 0 < η < ηmax (z) be given, and set

α = (ηmax (z) − η)/2 > 0.

From the Definition (6.39) of ηmax (z), there exists i0 ∈ I such that

η̃max,i0 (z) ≥ ηmax (z) − α > η > 0.

This proves, as η̃max,i0 (z) > 0, that z ∈ ϑi0 . But ϑi0 and ηmax,i0 satisfy (6.33)
by hypothesis, so that
k(z, η) < 1,
which proves that ϑ and ηmax satisfy also (6.33).
We give now two very simple examples:
Example 6.2.3 If D is convex and P made of the family of all segments of
D, then (D, P) is quasi-convex, with

ϑ=F
ηmax (z) = +∞ ∀z ∈ F.
6.2. DEFINITION AND MAIN PROPERTIES 285

Example 6.2.4 Let D be an “arc of circle” of F of radius R and arc length

L (Fig. 6.2), equipped with the sole family of path P available, made of all
sub-arcs of circles included in D.
Then D is quasi-convex as soon as

L < 2πR,

and the largest associated q.c. regular neighborhood ϑ is shown in Fig. 6.2,
together with a graphical illustration of the way ηmax (z) is determined.

• L ≤ πR D
11111
00000
ηmax
00000
11111
00000
11111
ϑ = complementary
of gray area
ηmax(z) as shown
z

• πR ≤ L < 2πR D
ϑ = complementary
of thick half-line
1111111111
0000000000
ηmax (z) = M in{η1, η2} 0000000000
1111111111
η2
0000000000
1111111111
0000000000
1111111111
0000000000
1111111111
0000000000
1111111111
0000000000
1111111111
0000000000
1111111111
z
0000000000
1111111111
η1 0000000000
1111111111
0000000000
1111111111

Figure 6.2: Quasi-convex arcs of circle

286 CHAPTER 6. QUASI-CONVEX SETS

We investigate now the properties of quasi-convex sets. We begin with the

Proposition 6.2.5 Let (D, P) be quasi-convex, and ϑ, ηmax be associated

regular neighborhood and function. Then the paths p satisfy

ν → p(ν) is injective ∀p ∈ P,
or equivalently, in term of arc length distance

δ(X, X) = 0 ∀X ∈ D.

Proof. If ν → p(ν) were not injective, there would exist ν  , ν  ∈ [0, L(p)], ν  <
ν  , such that p(ν  ) = p(ν  ). Let us call this point of D as X

X = p(ν) = p(ν  ) ∈ D ν  < ν  ,

and p̃ the path

p̃ : ν ∈ [0, ν  − ν  ]  p(ν  + ν),
which is in P because of property (6.15) of the collection of paths P. The
path p̃ has both ends at X, and so p̃ ∈ P(X; η) for any 0 < η < ηmax (X).
This implies, using the Definition 6.2.1 of quasi-convex sets, that the function
˜ 2 = p̃(ν)−X2 is strictly convex, which is contradictory to the fact that
d(ν)
˜
d(ν) ˜ = d(ν
≥ 0 and d(0) ˜  − ν  ) = 0. This ends the proof of the injectivity
of ν → p(ν), which in view of the Definition 6.1.5 of the arc length distance
δ(X, Y ) is clearly equivalent to δ(X, X) = 0.
We give now a lemma that generalizes the median lemma to triangles
with one curvilinear side:
Lemma 6.2.6 (median lemma). Let D be equipped with a collection of paths
P, and let z ∈ F and p ∈ P be given. Then
L(p)2 1 1
d21/2 + (1 − k) ≤ d20 + d21 , (6.40)
4 2 2
where (see Fig. 6.3)

dt = z − p(tL(p))F ∀t ∈ [0, 1], (6.41)

k = k(z, p) defined by (6.28) (6.42)

6.2. DEFINITION AND MAIN PROPERTIES 287

L(p)/2
p(L(p))
L(p)/2
p(L(p)/2)
p(0)
d1/2 d1

d0

Figure 6.3: Notations for the median lemma

Proof. Define, for any ν ∈ [0, L(p)],

f (ν) = z − p(ν)2F = (dν/L(p) )2 ,
which satisfies
f  (ν) ≥ 2(1 − k(z, p)).
def
This proves that the function ν → g(ν) = f (ν) + ν(L(p) − ν)(1 − k(z, p))
is convex over [0, L(p)]. This implies that
 L(p)  1 1
g ≤ g(0) + g(L(p)),
2 2 2
which is (6.40).
Of course, under the sole hypothesis that z ∈ F and p ∈ P, it can very
well happen that k ≥ 1, in which case the formula (6.40) is not very useful!
So we shall use the median lemma in situations where one can ensure that
k < 1, as, for example, in the
Corollary 6.2.7 Let (D, P) be quasi-convex, ϑ, ηmax be an associated regular
neighborhood and function. Then for any z ∈ ϑ and any ε > 0, there exists
η = η(z, ε) ∈]0, ηmax (z)[ such that, for any η-projections X0 and X1 of z on
D, one has
X0 − X1 F ≤ δ(X0 , X1 ) ≤ ε. (6.43)
This implies in particular that ϑ is included in the Edelstein set E(D) of D.
288 CHAPTER 6. QUASI-CONVEX SETS

Proof. Let z ∈ ϑ and ε > 0 be given. For η ∈]0, ηmax (z)], let X0 and X1 be
two η-projections of z on D. Then
either X0 = X1 , in which case one has using Proposition 6.2.5

X0 − X1 F = δ(X0 , X1 ) = 0 < ε for any η ∈]0, ηmax (z)[. (6.44)

or X0 = X1 , in which case there exists necessarily a path p ∈ P connecting

X0 to X1 . As X0 and X1 are η-projections of z, the path p is in P(z, η),
so that k(z, p) ≤ k(z, η), with k(z, η) < 1 because of the quasi-convexity
of (D, P). Application of the median lemma 1 to z and p shows that
(6.40) holds with

d1/2 ≥ d(z, D),

1>k = k(z, η) ≥ k(z, p),
d0 = X0 − zF ≤ d(z, D) + η,
d1 = X1 − zF ≤ d(z, D) + η,

which gives

L(p)2
d(z, D)2 + (1 − k(z, η)) ≤ (d(z, D) + η)2 = d(z, D)2 + 2ηd(z, D) + η 2 ,
4

that is,
2d(z, D) + η
L(p)2 ≤ 4η . (6.45)
1 − k(z, η)

When η → 0+, k(z, η) → k(z, 0) < 1 and the right-hand side of (6.45)
goes to zero.
Hence there exists η(z, ε) ∈]0, ηmax (z)[ such that

L(p) ≤ ε as soon as η = η(z, ε),

which implies

X0 − X1 F ≤ L(p) ≤ ε as soon as η = η(z, ε).

Taking the supremum for all paths p, connecting X0 to X1 gives

X0 − X1 F ≤ δ(X0 , X1 ) ≤ ε as soon η = η(z, ε). (6.46)

6.2. DEFINITION AND MAIN PROPERTIES 289

Comparing (6.44) and (6.46) shows that (6.43) holds in all cases as soon as
η is taken equal to the η(z, ε) determined in the X0 = X1 case, which ends
the proof of the Corollary 6.2.7.
Notice that in Corollary 6.2.7, the estimation (6.43) on the proximity
of two η-projections X0 and X1 of z is obtained not only for X0 − X1 F
(which corresponds exactly to saying that ϑ ⊂ E(D)), but also for the arc
length distance δ(X0 , X1 ): this stronger result will be very important in the
applications we have in mind to nonlinear least-squares inversion where D =
ϕ(C), as in this case δ(X, Y ) can be made equivalent in a natural way to the
distance in the parameter set C, whereas X − Y  cannot.
We prove now that some of the properties of the projection on convex
sets recalled in Proposition 4.1.1 generalize to quasi-convex sets. We begin
with the
Proposition 6.2.8 Let (D, P) be quasi-convex, and ϑ, ηmax be a pair of as-
sociated regular neighborhood and function. Then properties (i) and (iv) of
Proposition 4.1.1 generalize as follows:
(i) Uniqueness: for any z ∈ ϑ, there exists at most one projection X( of z
on D
(ii) Existence: if z ∈ ϑ, any minimizing sequence Xn ∈ D of the “distance
to z” function over D is a Cauchy sequence in for both X − Y F and
δ(X, Y ). Hence Xn converges in F to the (unique) projection X( of z
on the closure D of D.
( ∈ D, and δ(Xn , X)
If D is closed, then X ( → 0 when n → 0.

Proof. We prove first (i). Let z ∈ ϑ be such that it admits two projections X(0
and X(1 on D. As X(0 and X (1 are η-projections of z for any η ∈]0, ηmax (z)[,
we see from Corollary 6.2.7 that
X (1 F ≤ δ(X
(0 − X (0 , X
(1 ) ≤ ε for any ε > 0,

which proves that X (0 = X

(1 . Hence z has at most one projection on D.
We prove now (ii). Let z ∈ ϑ and ε > 0 be given, and let η(z, ε) be the
associated value of η defined in Corollary 6.2.7. Let {Xn ∈ D, n ∈ IN } be a
minimizing sequence of the “distance to z function” over D, which satisfies
by definition
Xn − zF → d(z, D) = inf X − zF . (6.47)
X∈D
290 CHAPTER 6. QUASI-CONVEX SETS

Then there exists N(z, ε) ∈ IN such that

∀n ≥ N(z, ε), Xn − zF ≤ d(z, D) + η(z, ε), (6.48)
which proves that all Xn , n ≥ N(z, ε) are η(z, ε)-projections of z on D. Then
Corollary 6.2.7 implies that
∀p, q ≥ N(z, ε), Xp − Xq F ≤ δ(Xp , Xq ) ≤ ε,
which shows that {Xn } is a Cauchy sequence for X − Y F and δ(X, Y ).
As F is complete (Hilbert space), the Cauchy sequence {Xn } has a limit
( ∈ D, and using (6.47),
X
( − z = d(z, D) = inf X − zF ,
X
X∈D

which shows that X ( is the (necessarily unique as we have seen earlier) pro-
jection of z on D.
When ϕ(C) is closed, it remains to prove that Xn converges to X ( also in the
stronger arc length distance δ(X, Y ).
This results once again from the Corollary 6.2.7:
X( is an η-projection for any η ∈]0, ηmax (z)[, and Xn is an η(z, ε)-projection
for all n ≥ N(z, ε), as we have seen in (6.48), so one has
∀n ≥ N(z, ε), ( F ≤ δ(Xn , X)
Xn − X ( ≤ ε,
( → 0.
which proves that δ(Xn , X)
Notice that, if we choose for (D, P) a convex set D equipped with the
family P of its segments, then ϑ = F and X − Y F = δ(X, Y ), and the
Proposition 6.2.8 reduces exactly to the corresponding results of Proposi-
tion 4.1.1!
We turn now to the generalization of the stability property (iii) of Propo-
sition 4.1.1 to quasi-convex sets. We begin with two lemma:
Lemma 6.2.9 (obtuse angle lemma). Let D be equipped with a collection of
paths P, and let z ∈ F and p ∈ P be given.
If
def
t → dt = z − p(tL(p))F has a local minimum at t = 0,
then
d20 + (1 − k(z, p))L(p)2 ≤ d21 , (6.49)
(where k(z, p) is not necessarily smaller than one).
6.2. DEFINITION AND MAIN PROPERTIES 291

Proof. Define, as in the proof of Lemma 6.2.6, for any ν ∈ [0, L(p)]:

f (ν) = z − p(ν)2F = (dν/L(p) )2 .

Then f (ν) has also a local minimum at ν = 0, so that

f  (0) ≥ 0.

Derivating twice f (ν) gives, as in (6.36),

f  (ν) ≥ 2(1 − k(z, p)) ∀ν ∈ [0, L(p)].

Hence, the Taylor expansion


f (L(p)) = f (0) + f  (0).L(p) + 12 f  (ν0 ).L(p)2 ,
where ν0 ∈ [0, L(p)]

becomes
f (L(p)) ≥ f (0) + (1 − k(z, p))L(p)2 ,
which is (6.49).
We have illustrated in Fig. 6.4 the geometric interpretation of this lemma:
formula (6.49) is the analogous for the curvilinear triangle (z, p(0), p(L(p))) of
the property that, in a triangle, the sum of squared length of edges adjacent
to an obtuse angle is smaller than the squared length of the opposite edge.
Of course, this analogy holds only in situations where z and p are such that
k(z, p) < 1!

Lemma 6.2.10 (Continuity lemma). Let D be equipped with a collection of

paths P, and let z0 , z1 , ∈ F be two points of F admitting projections X0 = X1
on D.
Then, for any path p ∈ P from X0 to X1 , one has

(1 − k)L(p) ≤ z0 − z1 F , (6.50)

where
k(z0 , p) + k(z1 , p)
k= . (6.51)
2

Proof. Let z0 , z1 , X0 , X1 , and p be given as in the lemma. We define a function

(see Fig. 6.5)

t ∈ [0, 1]  dt = (1 − t)z0 + tz1 − p(t)F ,

292 CHAPTER 6. QUASI-CONVEX SETS

L(p) p(L(p))

p(0) obtuse angle

d1
d0

Figure 6.4: Illustration of the obtuse angle lemma

z1
z0
d0 d1
d0 d1/2 d1
p(0) = X0 X1 = p(L)

L/2 p(L/2) 2
L/

Figure 6.5: Notations for the continuity lemma

where we have used the notation

 = L(p)

for the length of the path p.

6.2. DEFINITION AND MAIN PROPERTIES 293

The second derivative of d2t with respect to t is then (with the usual
notation v = p and a = p )
(d2t ) = 2z1 − z0 − v(t)2F − 2(1 − t)z0 + tz1 − p(t), 2 a(t)F ,
which can be rewritten as
(d2t ) = 2z1 − z0 2F − 4z1 − z0 , v(t)F + 22 v(t)2F
−2(1 − t)2 z0 − p(t), a(t)F
−2t2 z1 − p(t), a(t)F .
Using the Cauchy–Schwarz inequality, the fact that v(t)F = 1, and the
inequality
zj − p(t), a(t)F = k(zj , p; t) ≤ k(zj , p) j = 0, 1,
one can minorate (d2t ) as follows:
+ z0 − z1 F z0 − z1 2F ,
2 
(dt ) ≥ 2 1 − 2
2
+ − (1 − t)k0 − tk1 ,
 2
where
kj = k(zj , p), j = 0, 1.
This implies the convexity of the function
+ 2 1 z0 − z1 F z0 − z1 2F ,
t  d2t +t(1−t)2 1− (k0 +k1 )+ ((1−t)k1 +tk0 )−2 +
3 3  2
over the [0, 1] interval. Hence
2 + 1 z0 − z1 F z0 − z1 2F , 1 2 1 2
d21 + 1 − (k0 + k1 ) − 2 + ≤ d0 + d1 . (6.52)
2 4 2  2 2 2
We now use the hypothesis that X0 is a projection of z0 on D. This implies
that the “distance to z0 ” function has a local minimum on p at ν = 0.
Hence the curvilinear triangle (z0 , X0 , p(/2)) has an “obtuse angle” at X0 .
Application of the Lemma 6.2.9 gives then, with the notation of Fig. 6.5,
2
d20 + (1 − k0 ) ≤ d2
0 .
4
In a similar way, the fact that X1 is a projection of z1 on D implies that
2
d21 + (1 − k1 ) ≤ d2
1.
4
294 CHAPTER 6. QUASI-CONVEX SETS

Combining the two gives

1 2 1 2 1 2 1 2 2 + 1 ,
d0 + d1 ≤ d0 + d1 − 1 − (k0 + k1 ) ,
2 2 2 2 4 2
which, combined with (6.52), shows that
2 + 1 ,  1 1 2 1
1 − (k0 + k1 ) − z0 − z1 F ≤ d2
0 + d1 − d 1 − z0 − z1  . (6.53)
2 2
2 2 2 2 2 2 4
But F is a Hilbert space, and from the median theorem in the triangle
(z0 , p(/2), z1 ), we see that the right-hand side of (6.53) is equal to zero.
Dividing then (6.53) by /2, we obtain
+ 1 ,
1 − (k0 + k1 )  ≤ z0 − z1 F ,
2
which is the announced result.
We can now prove the
Proposition 6.2.11 Let (D, P) be quasi-convex, and ϑ, ηmax be a pair of
associated regular neighborhood and function. Then property (iii) of Proposi-
tion 4.1.1 on the stability of the projection generalizes as follows:
If
z0 , z1 , ∈ ϑ admit projections X0 , X1 on D, (6.54)
and are close enough so that
⎧
⎨ ∃d ≥ 0 such that
z0 − z1 F + maxj=0,1 d(zj , D) (6.55)
⎩
≤ d < minj=0,1 {d(zj , D) + ηmax (zj )},
then, for any d chosen as above, one has

X0 − X1 F ≤ δ(X0 , X1 ) ≤ (1 − k)−1 z0 − z1 F , (6.56)

where k < 1 is defined by

k = (k(z0 , η0 ) + k(z1 , η1 ))/2, (6.57)

with ηj , j = 0, 1 defined by

0 ≤ ηj = d − d(zj , D) < ηmax (zj ). (6.58)

6.2. DEFINITION AND MAIN PROPERTIES 295

Proof. We check first that the hypothesis (6.55) is satisfied as soon as z0 −z1 
is small enough: if z0 ∈ ϑ and z1 → z0 , then

z0 − z1 F + max d(zj , D) → d(z0 , D)

j=0,1

because of the continuity of the norm and the “distance to D” function, and

lim inf min {d(zj , D) + ηmax (zj )} → d(z0 , D) + ηmax (z0 ) > d(z0 p)
ε→0 z0 −z1 ≤ε j=0,1

because of the lower semi continuity of the function f : z  d(z, D)+ηmax (z),
which implies that z  min{f (z), f (z0 )} is l.s.c. at z = z0 . This ensures the
existence of a d satisfying (6.55).
We check now that the functions η0 , η1 defined by (6.58) satisfy inequali-
ties announced there, namely

0 ≤ ηj < ηmax (zj ), j = 0, 1. (6.59)

As d satisfies (6.55), we see that

z0 − z1 F + d(zj , D) ≤ d < d(zj , D) + ηmax (zj ) j = 0, 1,

which proves (6.59) by subtracting d(zj , D) from all three terms and using
the definition
ηj = d − d(zj , D), j = 0, 1
of ηj . Majoration (6.59) implies, by definition of quasi-convex sets, that
k(zj , ηj ) < 1 for j = 0, 1, so that k defined by (6.57) satisfies also k < 1.
We check now that X0 is an η1 -projection of z1 on D:

X0 − z1 F ≤ X0 − z0 F + z0 − z1 F

= d(z0 , D) + z0 − z1 F
≤ d = d(z1 , D) + η1 .

A similar proof shows that X1 is an η0 -projection of z0 on D. Hence

X0 , X1 are ηj − projections of zj on D, j = 0, 1. (6.60)

We conclude with the proof of the stability result (6.56):

• If X0 = X1 , then the Proposition 6.2.5 ensures that δ(X0 , X1 ) = 0, so
that (6.56) holds trivially.
296 CHAPTER 6. QUASI-CONVEX SETS

• If X0 = X1 , then there exists at least one path p ∈ P going from X0 to

X1 , so that we can apply the continuity Lemma 6.2.10 to z0 , z1 , and p:
(1 − k(p))L(p) ≤ z0 − z1 F , (6.61)

where
k(p) = (k(z0 , p) + k(z1 , p))/2.

But from (6.60) we see that p ∈ P(zj , ηj ) for j = 0, 1 so that, by definition

of k(zj , ηj ), one has
k(zj , p) ≤ k(zj , ηj ),
which proves that
k(p) ≤ k < 1,
with k defined by (6.57). Hence (6.61) can be rewritten

L(p) ≤ (1 − k)−1 z0 − z1 F
for any path p ∈ P from X0 to X1 ,
which proves (6.56) by taking the supremum over all paths p from X0 to X1 .
This ends the proof of Proposition 6.2.11.

Corollary 6.2.12 Let (D, P) be quasi-convex and ϑ be one associated regu-

lar neighborhood. Then
(i) The projection on D is (whenever it exists) locally uniformly Lipschitz
continuous on ϑ

(ii) The injection of (D,  F ) in (D, δ) is continuous, that is,

*
Xn , X ∈ D
⇒ δ(Xn , X) → 0. (6.62)
Xn − XF → 0

Proof. We prove first (i). Let z0 ∈ ϑ be given, and suppose that z1 → z0 .

For z1 close enough to z0 , hypothesis (6.55) will hold, so that (6.56) will hold
(provided z0 and z1 admit projections of course), with the following upper
bound for k:
k0 + 1
k≤ < 1,
2
which is independent from z1 and hence proves the continuity of the
projection.
6.2. DEFINITION AND MAIN PROPERTIES 297

Then, (ii) results immediately from the application of Proposition 6.2.11

to the case z0 = X, z1 = Xn , in which case the projection of zj on D obviously
exists!
If we summarize now the situation, we see that quasi-convex sets pro-
vide a satisfactory generalization of the following properties of convex sets
listed in Proposition 4.1.1: uniqueness and existence (Proposition 6.2.8), and
stability (Proposition 6.2.11). However, the absence of parasitic stationary
points (property ii) of Proposition 4.1.1) is not guaranteed for quasi-convex
sets, as it appears clearly on Fig. 6.2: for the two quasi-convex arcs of circles
D, and for the chosen z ∈ ϑ, the “distance to z” function admits a global
minimum at one end of the arc, and one parasitic local minimum at the other
end. In both cases, one sees in Fig. 6.2 that the value of the parasitic local
minimum is larger than (top) or equal (bottom) to d(z, D) + ηmax (z). This is
in fact general:

Proposition 6.2.13 Let (D, P) be quasi-convex, and ϑ be an associated reg-

ular neighborhood. Then if, for a given z ∈ ϑ, the function
d2z (X) = z − X2

admits two distinct stationary points over D, then necessarily one of them
gives to dz a value larger than or equal to d(z, D) + ηmax (z).

Proof. Let z ∈ ϑ be given such that the “distance to z” function admits two
distinct stationary points at X0 , X1 ∈ D, with values strictly smaller than
d(z, D) + ηmax (z):

Xj − zF < d(z, D) + ηmax (z), j = 0, 1. (6.63)

As X0 and X1 , are distinct, there exists p ∈ P going from X0 to X1 , because
of (6.63) is in P(z, η) for some η ∈]0, ηmax (z)[. Hence from Definition 6.2.1,
we see that ν  d2z,p (ν) = p(ν) − z2F is strictly convex. But, as X0 and X1
are stationary points of d2z over D, necessarily the function d2z,p , which is its
trace on p, has two distinct stationary points at ν = 0 and ν = L(p), which
is impossible for a strictly convex (Definition 6.2.1). Hence the assumption
(6.63) is false, and the proposition is proved.
So we introduce in Chap. 7 for a stronger notion, which will eliminate the
parasitic stationary points.
Chapter 7

Strictly Quasi-Convex Sets

The quasi-convex sets introduced in Chap. 6 do a good job in generalizing the

properties of convex sets with respect to uniqueness, stability, and existence of
the projection. But they miss their point on the subject of parasitic stationary
points. So we shall start in this chapter from a complementary point of
view, and introduce in Sect. 7.1 another family of sets, called the strictly
quasi-convex sets (s.q.c. sets in short) which, almost by definition, will ensure
the absence of parasitic stationary points. Note that the name “s.q.c.” has
provisorily to be taken as a whole, as it will not be clear at all from the
definition that s.q.c. sets are quasi-convex!
In Sect. 7.2, we shall characterize s.q.c. sets (D, P) as those for which
RG (D) > 0, where RG is a new geometric attribute, called the global radius of
curvature. As a byproduct of this characterization, we shall prove that s.q.c.
sets necessarily satisfy R(D) ≥ RG (D) > 0, where R is the ordinary radius
of curvature, and that s.q.c. sets are quasi-convex in the sense of Chap. 6.
The end of Sect. 7.2 will be devoted to summarizing the nice generalization
provided by s.q.c. sets concerning uniqueness, absence of parasitic stationary
points, stability, and existence of the projection. Finally, we shall discuss in
Sect. 7.3 the computation of the global radius of curvature RG (D) of a set D
equipped with a family of paths P, both from an analytical and a numerical
point of view.

G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 299
and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 7, c Springer Science+Business Media B.V. 2009
300 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

7.1 Definition and Main Properties

of s.q.c. Sets
Let D ⊂ F be given, and equipped with a family of paths P in the sense of
the Definition 6.1.3. For any z ∈ F and p ∈ P, we denote as usual by

dz,p(ν) = z − p(ν)F (7.1)

the “distance to z along p” function.

When D is convex and P is made of the segments of D, d2z,p is a quadratic
and strictly convex function, which is hence 2-convex and has a unique sta-
tionary point.
According to the Definition 6.2.1, (D, P) is quasi-convex as soon as d2z,p
retains from the convex case the property that it is uniformly α-convex over
P(z, η) as soon as z ∈ ϑ and 0 < η < ηmax (z).
This property is very local, as paths p of P(z, η) connect two η-projections
of z, which become close when η → 0 (cf Corollary 6.2.7).
So we shall require, to define strictly quasi-convex sets, that (D, P) in-
herits from the convex case a different, more global property, namely that
d2z,p has a unique stationary point as soon as z ∈ ϑ and p connects any two
points of D through an η-projection of z, 0 ≤ η < ηmax (z).
It will be convenient to give a name to functions with a unique stationary
point:
Definition 7.1.1 A function f ∈ C 1 ([0, L]) is called strictly quasi-convex
(s.q.c. in short) if and only if the inequation

f  (ν)λ ≥ 0 ∀λ ∈ IR, ν + λ ∈ [0, L] (7.2)

has a unique solution ν in [0, L].

A function f ∈ C 0 ([0, L]) is usually said to be quasi-convex when it satisfies

∀ν0 , ν1 ∈ [0, L], ν0 < ν1 ,& ' (7.3)
∀ν ∈]ν0 , ν1 [: f (ν) ≤ Max f (ν0 ), f (ν1 ) .

A quasi-convex function has one or more global minima, and no parasitic sta-
tionary points. But it can have parasitic stationary points (in the zones where
f is constant), as well as inflexion points. Sometimes, the name “strictly
quasi-convex” is used for functions f ∈ C 0 ([0, L]), which satisfy (7.3) with
7.1. DEFINITION AND MAIN PROPERTIES 301

a strict inequality. Such functions eliminate parasitic local minima (strict

or not), but not, when f ∈ C 1 ([0, L]), parasitic stationary points where f 
vanish.
Elimination of all parasitic stationary point – and hence of all parasitic
local minima – is essential for our purpose, as it guaranties that local opti-
mization algorithms used to compute the projection on D will not stop at
such a stationary point. So we shall always use s.q.c. function in the sense of
Definition 7.1.1.
We define now our new family of sets:

Definition 7.1.2 (Strictly quasi-convex sets). A set (D, P) is strictly quasi-

convex (s.q.c. in short) if and only if there exists a neighborhood ϑ of D and
a function ηmax : ϑ →]0, +∞] such that

∀z ∈ ϑ, ∀η, 0 < η < ηmax (z) one has
(7.4)
d(z, p) ≤ d(z, D) + η ⇒ d2z,p is s.q.c. along p,

and (ηmax is continued by zero outside of ϑ):

+ ,
lim inf d(z, D) + ηmax (z) > 0. (7.5)
→0 d(z,D)≤

The neighborhood ϑ is called an (s.q.c.) regular neighborhood of D.

Condition (7.4) of course is meant to handle the problem of parasitic

stationary points, but condition (7.5) is difficult to interpret directly. As we
shall see in the proof of Theorem 7.2.5, it will have the effect to ensure that
the regular neighborhood ϑ associated to s.q.c. sets contains necessarily an
enlargement neighborhood of D, which is a desirable property (see (4.4) in
Chap. 4). Notice that (7.5) is automatically satisfied as soon as F is a finite
dimensional, D bounded, and ηmax an l.s.c. function.
We check first that, as for quasi-convex sets, there exists a largest neigh-
borhood ϑ and function ηmax :
Proposition 7.1.3 Let (D, P) be a s.q.c. set. Then there exists a largest
regular open neighborhood ϑ of D, and a largest function ηmax : ϑ →]0, +∞]
satisfying the Definition 7.1.2 of s.q.c. sets.

Proof. Let ϑi , ηmax,i , i ∈ I denote the collection of all open neighborhoods

and functions satisfying Definition 7.1.2, and let ϑ and ηmax be defined by
302 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

(6.38) and (6.39). Then ηmax satisfies obviously (7.5). We check now that ϑ
and η satisfy (7.4); let us choose z ∈ ϑ and p ∈ P such that

d(z, p) < d(z, D) + ηmax (z).

From the formula (6.39) defining ηmax , we see that there exists î ∈ I such
that
d(z, p) < d(z, D) + ηmax î (z). (7.6)
As d(z, p) ≥ d(z, D), (7.6) implies that ηmax î (z) > 0, which shows that
necessarily z ∈ ϑî . Then the strict quasi-convexity of d2z,p over p results from
the fact that (7.4) holds, by hypothesis, for all ϑ = ϑi and ηmax = ηmax i , i ∈ I,
and hence in particular for î.
The definition chosen for s.q.c. sets makes it very easy to handle the
problem of parasitic stationary points:

Proposition 7.1.4 Let (D, P) be s.q.c., and ϑ be an associated regular

neighborhood. Then property (ii) of Proposition 4.1.1 on stationary points
generalizes as follows:
If z ∈ ϑ admits a projection X̂ on D, the “distance to z” function has
no parasitic stationary point on D (its unique stationary point is X̂). In
particular, X̂ is necessarily unique.

Proof. Let z ∈ ϑ be given such that it admits a projection X on D, and sup-

pose that the “distance to z” function possesses, beside the global minimum
on D at X, a stationary point on D at a point Y = X. Let p ∈ P be a path
connecting X to Y . As X ∈ p and X is the projection of z onto D, one has

d(z, p) ≤ z − X = d(z, D) < d(z, D) + ηmax (z).

Using the Definition 7.1.2 of s.q.c. sets, this implies that the “distance to z”
function d2z,p is s.q.c. along the path p, which contradicts the fact that this
function has a global minimum at ν = 0 (because X is a global minimum on
D) and a stationary point at ν = L(p) (because of the hypothesis we have
made that Y is a stationary point of d2z on D).
Of course, convex sets, which were already quasi-convex sets as we have
seen in Chap. 6 earlier, are also s.q.c. sets with a regular neighborhood ϑ = F
and a function ηmax (z) = +∞. Arcs of circle of radius R and length L,
7.1. DEFINITION AND MAIN PROPERTIES 303

D
ηmax
L < πR
ϑ = complementary
of gray area
ηmax (z) as shown z

Figure 7.1: s.q.c. arc of circle

however, are s.q.c. only if L < πR, with a largest regular neighborhood ϑ
shown in Fig. 7.1. Comparison with Fig. 6.2 shows the diminution in size of
the neighborhood ϑ associated to the same arc of circle of length L < πR
when it is considered as s.q.c. instead of quasi-convex.
As we see in Fig. 7.1, the largest regular neighborhood ϑ associated to
the arc of circle D by the Definition 7.1.2 of s.q.c. sets catches exactly all
points z of the plan admitting a unique projection on D with no parasitic
stationary points on D of the distance to z. So the notion of s.q.c. provides
a sharp description of the sets D to which the result of Proposition 4.1.1 can
be generalized, when D is an arc of circle. In fact, as we shall see in the next
section, this remains true for all D made of one path p.
The definition of s.q.c. set is quite technical. We have been able to use
it directly and to determine the associated largest regular neighborhood ϑ
in the above examples only because they were extremely simple (D was a
convex set or an arc of circle). Using the definition itself to recognize s.q.c.
sets in the applications to nonlinear least-squares problems (see Chap. 1)
seems quite impractical, if not unfeasible. So we develop in the next section
a characterization of s.q.c. sets, which will be more easy to use, and will
provide the size RG of the largest open regular enlargement neighborhood
ϑ = {z ∈ F |d(z, D) < RG }
associated to the s.q.c. set (the reason for calling RG the size of the neigh-
borhood will become clear in the next section).
For the applications to nonlinear least-squares, the determination of such
a regular enlargement neighborhood is very important, as RG gives an upper
304 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

bound to the size of the noise level (measurement and model errors) on data
for which the least-squares problem remains Q-wellposed.

7.2 Characterization by the Global Radius

of Curvature
The idea in this section is to use the Definition 7.1.1 of s.q.c. function to
localize, for a given path p ∈ P and a given ν ∈ [0, L(p)], the points z ∈ F
such that d2z,p has one stationary point at ν. This leads to
Definition 7.2.1 (Affine normal cone)
Given a path p ∈ P and ν ∈ [0, L(p)], we define the affine normal cone
N(ν) by
+  ,
N(ν) = z ∈ F z − p(ν), λv(ν)F ≤ 0 ∀λ ∈ IR, ν + λ ∈ [0, L] ,

where as usual v(ν) = p (ν) is a unit tangent vector to p at p(ν).

Hence z ∈ N(ν) means exactly that d2z,p has a stationary point at ν. So
by construction, when ν = ν  , the intersection N(ν) ∩N(ν  ) is made of points
z such that d2z,p has two distinct stationary points at ν and ν  , so that d2z,p is
not s.q.c.
Figure 7.2 shows these affine normal cones N(ν) and N(ν  ) at two different
points ν and ν  of a given path p of IR2 , and their intersection (bold point,
bold line or darker grey area) in three different cases:
(a) The two points p(ν) and p(ν  ) are interior points of the path. Then the
intersection of N(ν) and N(ν  ) is a point. If we call z this point, one
sees clearly that the d2z,p function possesses, besides a global minimum
at ν, a parasitic stationary point at ν  .
(b) One of the two points (here p(ν)) is an endpoint of p. Then the intersec-
tion of N(ν) and N(ν  ) is a half-line. If we call z the projection of p(ν)
on this half line, one sees once again that the d2z,p possesses, as earlier,
a parasitic stationary point at ν  .
(c) The two points are the endpoints of p. Then the intersection of N(ν)
and N(ν  ) is the bold dashed area. If we call z the projection of p(ν)
on the intersection, once again d2z,p has a parasitic stationary point at
the endpoint ν  = L.
7.2. CHARACTERIZATION BY THE GLOBAL RADIUS 305

N (ν)
p
p(ν) p(ν )
Case a) N (ν )
z

p
p(ν )
N (ν) N (ν )
p(ν) = p(0)
Case b) p(L)
z

N (ν)
p(ν) = p(0) p(ν ) = p(L)
Case c)
z N (ν )

Figure 7.2: Intersection of affine normal cones at two points of a

given paths p

So we see that the distance of p(ν) to N(ν) ∩ N(ν  ) plays a particular

role, and we give it a name:
Definition 7.2.2 (Global radius of curvature)
Let p ∈ P be given. Then, for any ν, ν  ∈ [0, L(p)] with ν = ν  , we define
the global radius of curvature of p at ν seen from ν  by
 
ρG (ν, ν  ) = d p(ν), N(ν) ∩ N(ν  ) ∈ [0, +∞], (7.7)
with the natural convention that ρG (ν, ν  ) = +∞ if N(ν) and N(ν  ) do not
intersect.
Consideration of the worst case for ν ∈ [0, L(p)] then for all p of P leads
to the
306 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

Definition 7.2.3 (Global radius of curvature of a path and a set).

Let (D, P) be given. Then we define the global radius of curvature of a
path p ∈ P by
ρG (ν, ν  ),
def
RG (p) = 
inf 
(7.8)
ν,ν ∈[0,L(p)] , ν =ν

and that of the set (D, P) by

def
RG (D) = inf RG (p). (7.9)
p∈P

As we have seen in Fig. 7.2, the d2z,p function has parasitic stationary
points as soon as d(z, p) is equal to the global radius of curvature at the
projection p(ν) of z on p, seen from some other point p(ν  ) of p. It seems
reasonable to make the conjecture that d2z,p will have no parasitic stationary
points as soon as d(z, p) is strictly smaller than the infimum of all global
radii of curvature! This is confirmed by the

Proposition 7.2.4 Let (D, P) be given. Then

RG (D) = inf {d(z, p) | z ∈ F , p ∈ P and d2z,p not s.q.c.}, (7.10)

where d2z,p is defined in (7.1).

Proof. We show first that RG (D) ≤ inf{...}.

So let h ∈ {...} be given, and z ∈ F and p ∈ P be a couple of corresponding
point and path, which hence satisfy

d2z,p is not s.q.c.

Then Definition 7.1.1 of s.q.c. functions implies that d2z,p possesses at least,
beside a global minimum at some ν0 , a second stationary point at ν1 = ν0 .
Hence (7.2) with f replaced by d2z,p holds at both ν0 and ν1 . This can be
rewritten, using the Definition 7.2.1 of affine normal cones, as

z ∈ N(ν0 ) ∩ N(ν1 ),

which implies

d(p(ν0 ), N(ν0 ) ∩ N(ν1 )) ≤ p(ν0 ) − zF . (7.11)

7.2. CHARACTERIZATION BY THE GLOBAL RADIUS 307

But the left-hand side of (7.11) is ρG (ν0 , ν1 ) from the Definition 7.2.2 of the
global radius of curvature, and the right-hand side is d(z, p) by definition of
ν0 . Hence (7.11) becomes

ρG (ν0 , ν1 ) ≤ d(z, p), (7.12)

and, using the Definition 7.2.3 of RG (D) and the properties of z, and p

RG (D) ≤ ρG (ν0 , ν1 ) ≤ d(z, p), (7.13)

which ends the first part of the proof.

We check now that RG (D) ≥ inf{...}. Let  > 0 be given. From the
Definition (7.9) of RG (D), we see that there exists p ∈ P such that

RG (p) ≤ RG (D) + /2, (7.14)

and, from the definition (7.8) of RG (p) , that there exists ν0 , ν1 ∈ [0, L(p)],
ν0 = ν1 , such that
ρG (ν0 , ν1 ) ≤ RG (p) + /2. (7.15)
Let z ∈ F be the projection on N(ν0 ∩ N(ν1 )) of p(ν0 ). By construction, d2z,p
has two distinct stationary points at ν0 and ν1 , and so cannot be s.q.c.! On
the other hand, the definition (7.7) of ρG shows that

ρG (ν0 , ν1 ) = z − p(ν0 )F ≥ d(z, p). (7.16)

Combining (7.14), (7.15), and (7.16) gives

d(z, p) ≤ RG (D) + . (7.17)

Hence given  > 0, we have been able to find a z ∈ F and p ∈ P such that
(7.17) holds and d2z,p is not s.q.c., which proves that

inf{...} ≤ RG (D) + .

This holds for any  > 0, which proves that inf{...} ≤ RG (D).

Proposition 7.2.4 gives immediately a characterization of s.q.c. sets:

Theorem 7.2.5 A set D ⊂ F equipped with a family of paths P is s.q.c. if
and only if
RG (D) > 0. (7.18)
308 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

The largest associated open regular enlargement neighborhood ϑ is given by

+ ,
ϑ = z ∈ F |d(z, D) < RG (D) , (7.19)

and the corresponding ηmax function by

∀z ∈ ϑ, ηmax (z) = RG (D) − d(z, D) > 0. (7.20)

Proof. We prove first the sufficient condition. So let (7.18) hold, and define
ϑ and ηmax by (7.19) and (7.20). Then for any z, η, and p satisfying the
hypothesis of (7.4), one has, by definition of ηmax ,

h = d(z, p) < RG (D),

which by Proposition 7.2.4 implies that d2z,p is necessarily s.q.c. So (7.4) holds,
and (7.9) holds trivially as d(z, D) + ηmax (z) = RG (D) > 0. Hence (D, P) is
s.q.c.
We prove now the necessary condition. So let (D, P) be s.q.c., and suppose
that RG (D) = 0. Using Proposition 7.2.4, we can find zn ∈ F , pn ∈ P for all
n = 1, 2, ... such that
1
d(zn , pn ) = , (7.21)
n
d2zn ,pn not s.q.c. (7.22)
But (7.21) implies that d(zn , D) → 0, and, using (7.9), the existence of γ > 0
such that
d(zn , D) + ηmax (zn ) ≥ γ > 0 ∀n.
Hence, for n large enough, one has
2
d(zn , D) + ηmax (zn ) > . (7.23)
n
This implies, as d(zn , D) ≤ d(zn , pn ) = 1/n, that
1
ηmax (zn ) > .
n
So (7.23) can be rewritten as
1
d(zn , pn ) = ≤ d(zn , D) + ηn ,
n
7.2. CHARACTERIZATION BY THE GLOBAL RADIUS 309

1
where ηn = ηmax (zn ) − satisfies
n
0 < ηn < ηmax (zn ).
As (D, P) is s.q.c., the two last inequalities imply that d2zn ,pn is s.q.c. for n
large enough, which contradicts (7.22); hence necessarily RG (D) > 0, and
Proposition 7.2.4 implies that ϑ defined in (7.19) is the largest regular en-
largement neighborhood associated with the s.q.c. set (D, P).
Theorem 7.2.5 is illustrated graphically in Fig. 7.3 in the case of a set D
made of one s.q.c. arc of circle p: one sees that the enlargement neighbor-
hood of size RG (p) is the largest regular enlargement included in the largest
possible regular neighborhood, which is recalled from Fig. 7.1.

ϕ
0 ≤ θ ≤ π/2 p
RG(p) = R
R

0000000
1111111
1111111
0000000
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111
0000000
1111111

ϕ
π/2 ≤ θ < π
p
RG(p) = R sin θ R

11111111111111111111111
00000000000000000000000
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111
00000000000000000000000
11111111111111111111111

Figure 7.3: Largest regular enlargement s.q.c. neighborhood of size

RG (p) (in gray) for paths p made of an arc of circle of radius R and
length L = R θ (the overall largest s.q.c. neighborhoods correspond
to the complementary of the dashed areas)
310 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

Now that we have defined, and characterized, s.q.c. sets that allow for
the generalization of property (ii) of 4.1.1 (no parasitic stationary points for
the projection), we would like to see whether properties (i), (iii), and (iv)
(uniqueness, stability, and existence) can also be generalized to s.q.c. sets.
According to the results of Chap. 6, this would be true if s.q.c. sets happened
to be quasi-convex sets; to prove that this is indeed the case, we need to study
further the analytical properties of the global radius of curvature ρG (ν, ν  )
and its relation with the usual, local radius of curvature ρ(ν) = a(ν)−1
defined in Chap. 6.

Proposition 7.2.6 Let a path p ∈ P and ν, ν  ∈ [0, L(p)], ν = ν  be given,

and denote ⎧
⎪
⎪ X = p(ν), X  = p(ν  ),
⎪
⎨ v = v(ν), v  = v(ν  ),
⎪ N = sgn(ν  − ν)X  − X, v  , (7.24)
⎪
⎪  1/2
⎩ D = 1 − v, v 2 .
Then ρG (ν, ν  ) is given by the following formula:
NUM
ρG (ν, ν  ) = , (7.25)
DEN
where: 
|N| if ν is interior,
NUM = (7.26)
N + if ν is an end point,

D if ν is interior or Nv, v   ≥ 0,
DEN = (7.27)
1 if ν is an end point and Nv, v  ≤ 0,
where N + = Max {N, 0}, and “interior” and “end point” are relative to p.

The proof of this formula is elementary, with the basic ingredients being the
projection of a point on a hyperplane and the angle between two hyperplanes.
If the reader looks now at Fig. 7.2 and imagines what is going to happen when
ν  → ν, he will not be surprised by the

Proposition 7.2.7 Let a path p ∈ P be given. Then

(i) For any ν ∈ [0, L(p)], there exists an open neighborhood I(ν) in [0, L(p)]
such that, for any ν  ∈ I(ν) one has, with the notation (7.24),
- -
ρG (ν, ν  ) = N D, ρG (ν  , ν) = N  D, (7.28)
7.2. CHARACTERIZATION BY THE GLOBAL RADIUS 311

(ii) For almost every ν ∈ [0, L(p)], we have

ρG (ν, ν  ) → ρ(ν), ρG (ν  , ν) → ρ(ν), (7.29)

when ν  → ν in I(ν), where ρ(ν) = a(ν)−1 is the usual radius of

curvature of p at ν.

Proof. Property (i) follows immediately from Proposition 7.2.6 if we prove

that N and v, v   are positive when ν  is close enough to ν. But, with the
notation dν = ν  − ν, one has

N = sgn dν p(ν + pν) − p(ν), v(ν + dν),

that is, using a Taylor formula

N = |dν|  v(ν + θ dν), v(ν + dν)

for some 0 ≤ θ ≤ 1, and

v, v  = v(ν), v(ν + dν).

As the velocity v is a continuous function over [0, L(p)], we see that

⎧ -
⎨ N |δν| → 1
when δν → 0, (7.30)
⎩
ν, ν   → 1

which ends the proof of (i).

We compute now the limit of ρG when ν  → ν. Because of (i), this amounts
to searching for the limit of N/D. We have already found in (7.30) an equiva-
lent of N in term of δν = ν  − ν, and we give now an equivalent of D in terms
of δμ = v  − v. The theorem of the median applied to v and v  implies that
-
v, v  = 1 − δμ2 2.

Hence  - 1/2
D = δμ 1 − δμ2 4 ,
and, as δμ → 0 when δν → 0,

D/δμ → 1 when δν → 0. (7.31)

312 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

Using (7.30) and (7.31), we can

- replace the search for the limit of ρG (ν, ν+dν)
by that of the limit of |dν| δμ when dν → 0. As our guess for this limit is
ρ(ν) = a(ν)−1
F , let us choose one realization of a(ν) in L
∞
[0, L(p)] ; F –
which
- will still be denoted by a(ν) – so that a(ν) is well-defined, and compare
δμ |dν| to a(ν)

v  − v  1  dν 
δμ  
= =  a(ν + τ )dτ  .
|dν| |dν| dν 0 F

The triangular inequality implies then

 δμ   1  dν 
   
 − a(ν)F  ≤  a(ν + τ )dτ − a(ν) . (7.32)
|dν| dν 0 F

But the right-hand side of (7.32) tends to zero when dν → 0 each time
ν is a Lebesgue point for the chosen realization of a (see, e.g., Theorem 8.8
of [Rudin 1987]). As almost every point of [0, L(p)] is a Lebesgue point, we
have proven the left part of (7.29). To prove the right part, we remark that

ρG (ν, ν  ) − dβ ≤ ρG (ν  , ν) ≤ ρG (ν, ν  ) + dβ,

where
|X  − X, v − v  | |dν| |v(ν + θ dν, v − v  |
0 ≤ dβ = =
D D
for some 0 ≤ θ ≤ 1. The Cauchy–Schwarz inequality implies then

|dν| δμ
0 ≤ dβ ≤ ,
D
-
which, as D δμ → 1 (see (7.31)), shows that

δβ → 0 when dν → 0.

This ends the proof of the theorem.

We can now compare the global radius of curvature of a path and a set
to the usual ones:
7.2. CHARACTERIZATION BY THE GLOBAL RADIUS 313

Definition 7.2.8 (Radius of curvature of a path and a set)

Let (D, P) be given. Then we define the (local) radius of curvature of the
path p ∈ P by
def
R(p) = inf ess ρ(ν), (7.33)
ν∈[0,L(p)]

and that of the set (D, P) by

def
R(D) = inf R(p). (7.34)
p∈P

Proposition 7.2.9 Let (D, P) be given. Then

(i) For any p ∈ P one has

+ ∞ ≥ R(p) > 0, (7.35)

and
R(p) ≥ RG (p) ≥ 0, (7.36)

(ii) The local and global radius of curvatures of D satisfy

R(D) ≥ RG (D) ≥ 0. (7.37)

 
Proof. Inequality(7.35) holdsbecausep ∈ W 2,∞ [0, L(p)] (cf.Definition6.1.1),
and inequalities (7.36) and (7.37) follow from Proposition 7.2.7 (ii).
We can now state the second main result of this chapter:

Theorem 7.2.10 Let (D, P) be given. Then

(D, P) s.q.c. =⇒ (D, P) quasi − convex. (7.38)

Proof. Let (D, P) be s.q.c. Then we know from Theorem 7.2.5 that

RG (D) > 0, (7.39)

and that  & '

ϑ = z ∈ F | d(z, D) < RG (D) ,
(7.40)
ηmax (z) = RG (D) − d(z, D),
satisfy (7.4).
314 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

Then Proposition 7.2.9, together with (7.39), imply that

R(D) ≥ RG (D) > 0, (7.41)

so that the curvature of the paths of P is uniformly bounded.

We check now that ϑ and ηmax defined by (7.40) satisfy (6.33), which will
prove, as ηmax is continuous and hence l.s.c., that (D, P) is quasi-convex.
So let z ∈ ϑ and η, 0 < η < ηmax (z) be given. We want to prove that
k(z, η) < 1, that is, k(z, p; ν) stays uniformly away from 1 over all p ∈ P(z, η)
and ν ∈ [0, L(p)]. So let p ∈ P(z, η) and ν ∈ [0, L(p)] be given. By definition,
p satisfies
z − p(j)F ≤ d(z, D) + η j = 0, L(p), (7.42)
which implies that
d(z, p) ≤ d(z, D) + η,
and, as ϑ and ηmax satisfy (7.4)

d2z,p is s.q.c. (7.43)

Combining (7.42) and (7.43), we obtain

dz,p (ν) ≤ max dz,p (j) ≤ d(z, D) + η, ∀ν ∈ [0, L(p)]. (7.44)

j=0,L(p)

From the definition (6.26) of k(z, p; ν), we see that

k(z, p; ν) ≤ |k(z, p, ν)| ≤ z − p(ν)F a(ν)F ,

or, with the notation dz,p (ν) = z − p(ν)F and a(ν)F = 1/ρ(ν)

k(z, p, ν) ≤ dz,p (ν)/ρ(ν),

and, using (7.44) and the definition of R(D)

d(z, D) + η
k(z, p, ν) ≤ .
R(D)
This upper bound is clearly uniform in p, ν, and satisfies, as η < ηmax defined
in (7.40)
d(z, D) + η RG (D)
< ≤ 1
R(D) R(D)
because of (7.41). Hence (6.33) is satisfied, and (D, P) is quasi-convex.
7.2. CHARACTERIZATION BY THE GLOBAL RADIUS 315

We summarize in the next theorem the nice properties of s.q.c. sets with
respect to projection, which generalize those recalled for the convex case in
Proposition 4.1.1:

Theorem 7.2.11 Let (D, P) be s.q.c., and

+ ,
ϑ = z ∈ F | d(z, D) < RG (D) (7.45)

be the largest associated open regular enlargement neighborhood. Then:

(i) Uniqueness: for any z ∈ ϑ, there exists at most one projection of

z on D.

(ii) Unimodality: if z ∈ ϑ admits a projection X ( on D, the “distance to z”

(
function has no parasitic stationary point on D other that X.

(iii) Stability: if z0 , z1 ∈ ϑ admit projections X(0 , X

(1 on D and are close
enough so that there exists d ≥ 0 satisfying

z0 − z1 F + max d(zj , D) ≤ d < RG (D), (7.46)

j=0,1

(0 to X
then for any path p going from X (1
-
X(0 − X(1  ≤ L(p) ≤ (1 − d R(p))−1 z0 − z1 F , (7.47)
F

where R(p) ≥ R(D) ≥ RG (D) > 0.

(iv) existence: if z ∈ ϑ, any minimizing sequence Xn ∈ D of the “distance

to z” function over D is a Cauchy sequence for both distances X −Y F
( of z onto
and δ(X, Y ). Hence Xn converges to the (unique) projection X
the closure D of D.
If moreover D is closed, then X( ∈ D, and δ(Xn , X̂) → 0 when n → 0.

Proof. Theorem 7.2.11 is simply a compilation of Propositions 6.2.5, 6.2.8,

and 7.1.4 and Theorems 7.2.5 and 7.2.10, with some simplifications in the
stability formula brought by the simple form of the ηmax (z) function.
We summarize also the properties of the “squared distance to z” function
d2z,p along paths of P when (D, P) is s.q.c.:
316 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

Proposition 7.2.12 Let (D, P) be s.q.c. Then, for any z ∈ ϑ and any p ∈ P
such that
d(z, p) < RG (D),
the function d2z,p satisfies
(i) d2z,p – and hence dz,p – is s.q.c. over the whole path p
(ii) For any η such that 0 < η < RG (D) − d(z, D), d2z,p is α-convex between
any two η-projections of z, where
 d(z, D) + η   d(z, D) + η 
α=2 1− ≥2 1− > 0.
R(p) R(D)

7.3 Formula for the Global Radius

of Curvature
We establish in this section the formula which will be our starting point for
the evaluation of RG (p) and RG (D) both numerically and analytically.
We remark first that, when computing RG (p) for some path p, one does
not need to consider in the computation of ρG (ν, ν  ) all the cases indicated
in Proposition 7.2.6, according to whether ν and/or ν  are interior or end
points of the paths: it is enough to evaluate ρG (ν, ν  ) always as if ν and ν 
were endpoints. This is made precise in the next proposition.

Proposition 7.3.1 Let p ∈ P be given. Then


RG (p) = inf ρep
G (ν, ν ), (7.48)
ν,ν  ∈[0,L(p)]
ν=ν 

where ρep
G (“ep” stands for “end points”) is given by
 +
ep  N /D if v, v  ≥ 0,
ρG (ν, ν ) = (7.49)
N+ if v, v  ≤ 0,

where N + = max{N, 0} and v, v  , N, and D are defined in (7.24).

Moreover, the infimum in (7.48) is not changed if one eliminates from
the search all couples ν = ν  such that

ν is an interior point of p,
(7.50)
N > 0 and v, v  < 0.
7.3. FORMULA FOR THE GLOBAL RADIUS 317

Proof. From the definitions of ρG and ρep

G one has

ρG (ν, ν  ) ≥ ρep  
G (ν, ν ) ∀ν = ν ,

which shows that


inf G (ν, ν ) ≤ RG (p).
ρep
ν,ν  ∈[0,L(p)] , ν =ν 

So (7.48) will hold as soon as


∀ν, ν  ∈ [0, L(p)] , ν = ν  , ∃ν̄ ∈ [0, L(p)] such that
  (7.51)
G (ν, ν ) ≥ ρG (ν̄, ν ),
ρep
which we prove now. So let p ∈ P, and ν, ν  be given such that (a similar
proof holds if ν  < ν)
0 ≤ ν < ν  ≤ L(p).
Let X, X  , v, v  , N, and D be defined as in (7.24), and set, for any τ ∈
[0, L(p)],
N(τ, ν  ) = X  − p(τ ), v  ,
v(τ ) = p (τ ).

We treat separately the cases according to the signs of N and v, v :
Case 1: N = N(ν, ν  ) ≤ 0.
The mapping τ → N(τ, ν  ) is continuous, negative at τ = ν, and positive
when τ is inferior and close enough to ν  . Hence, there exists ν̄ ∈ [ν, ν  [ such
that N(ν̄, ν  ) = 0. Hence
 
G (ν, ν ) ≥ 0 = ρG (ν̄, ν ),
ρep
and (7.51) is satisfied.
Case 2: N(ν, ν  ) > 0.
Subcase 2.1 : v, v  ≥ 0.
Then the formula for ρG and ρep
G coincide, so that
 
ρep
G (ν, ν ) = N/D = ρG (ν, ν ),

and (7.51) holds with ν̄ = ν.

Subcase 2.2 : v, v  < 0.
318 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

Then the formula for ρep

G is


ρep
G (ν, ν ) = N, (7.52)

but that for ρG depends on the nature of ν:

Subsubcase 2.2.1 : ν = 0 (i.e., ν is an endpoint). Then

ρG (ν, ν  = N = ρep 
G (ν, ν ),

and (7.51) holds with ν̄ = ν.

Subsubcase 2.2.2 : ν > 0 (i.e., ν is an interior point). Then

ρG (ν, ν  ) = N/D > N = ρep 

G (ν, ν ), (7.53)

and we have to exhibit some ν̄ to satisfy (7.51). So let ν̄ be defined as

⎧ + 
⎨ ν̄ = inf ν̃ ∈ [0, ν[N(τ, ν  ) > 0 and v(τ ), v  < 0
, (7.54)
⎩ ∀τ, ν̃ < τ < ν .

As N(ν, ν  ) > 0 and v, v   < 0, one has necessarily ν̄ < ν, and one checks
easily that the function τ  N(τ, ν  )2 is strictly increasing over the [ν̄, ν]
interval. Hence
N(ν̄, ν)2 < N(ν, ν  )2 = N 2 . (7.55)
As the functions τ  N(τ, ν  ) and v(τ ), v  are continuous, the abscissa ν̄
defined by (7.54) satisfies necessarily one of the two following conditions:
• Either N(ν̄, ν  ) = 0 and v(ν̄), v   ≤ 0. Then
 
ρep
G (ν, ν ) = N > 0 = ρG (ν̄, ν ), (7.56)

• Or N(ν̄, ν  ) > 0 and v(ν̄), v   = 0. Then D̄ = (1 − v(ν̄), v  2 )1/2 = 1,

so Proposition 7.2.6 shows that

ρG (ν̄, ν  ) = N(ν̄, ν  ), (7.57)

no matter whether ν̄ is an endpoint or not. Combining (7.52), (7.55),

and (7.57) gives
 
ρep
G (ν, ν ) > ρG (ν̄, ν ), (7.58)
so that (7.51) holds with a strict inequality.
7.3. FORMULA FOR THE GLOBAL RADIUS 319

This ends the proof of formula (7.48). Then the possibility of skipping
in the couples ν = ν  that satisfy (7.50) follows from (7.56) and (7.58), and
from the remark that, in both cases,

ρG (ν̄, ν  ) = ρep 
G (ν̄, ν ). (7.59)

This ends the proof of the proposition.


A second remark is that, because ρep G (ν, ν ) given by (7.49) does not de-
pend on the nature of ν and ν  (interior or endpoints), one obviously has, for
any p ∈ P,
RG (p) = inf 
RG (p ). (7.60)
p ⊂p

Notice that this result cannot be seen immediately when RG (p) is computed
using ρG (ν, ν  ) as in its definition: if p is the sub-path of p going from ν to ν  ,
the normal cones at ν and ν  to p are necessarily larger than the normal cones
to p, so that one has in general only

ρG (ν, ν  ; p) ≥ ρG (ν, ν  ; p ).

When a generating family PG of P is available, as defined in Chap. 6, it

is possible to “limit” the search to paths of PG when computing the global
radius of curvature of (D, P):

Proposition 7.3.2 Let (D, P) be given, and PG be a generating family of P.

Then
RG (D) = inf RG (p) (7.61)
p∈PG

R(D) = inf R(p) (7.62)

p∈PG

where RG (p) can be computed by (7.48) and R(p) by (7.33).

From a numerical point of view, Proposition 7.3.2 allows, at least in prin-

ciple, to evaluate R(D) and RG (D). For D = ϕ(C), this evaluation requires
to discretize the boundary ∂C of C, and to choose a discretization tj of [0, 1].
Then R(D) and RG (D) can be approximated by

R(D)  inf inf ρep

G (i, j) (7.63)
x0 ,x1 ∈∂C i−j=1

RG (D)  inf inf ρep

G (i, j) (7.64)
x0 ,x1 ∈∂C i=j
320 CHAPTER 7. STRICTLY QUASI-CONVEX SETS

Of course, these computations become quickly unfeasible when the dimension

of the unknown parameter exceeds a few units, and a precise numerical eval-
uation of R(D) and RG (D) is in most cases out of reach. This is why we
develop in the next chapter analytical lower bounds to R(D) and RG (D).
They will provide us with sufficient conditions for a set to be s.q.c., which
can be checked analytically.
Chapter 8

Deflection Conditions for the

Strict Quasi-convexity of Sets

We develop in this chapter sufficient conditions for a set (D, P) to be s.q.c.

As we have seen in Chap. 7, an s.q.c. set, which is characterized by the fact
that RG (D) > 0, has necessarily a finite curvature, as R(D) ≥ RG (D) (see
Proposition 7.2.9). But the condition

R(D) > 0 (8.1)

is not sufficient to ensure that (D, P) is s.q.c.

This can be seen on the simple case where (D, P) is an arc of circle of
radius R and arc length L (Fig. 7.3). The deflection of this arc of circle, that
is, the largest angle between two of its tangents – obviously the ones at its
two ends in this case – is given by
-
Θcircle = L R. (8.2)

In the upper part of the figure, one sees that D is s.q.c., with the largest
regular enlargement neighborhood ϑ of size RG = R, as soon as it satisfies
the deflection condition
Θcircle ≤ π/2. (8.3)
But the lower part of the figure indicates that it is in fact enough that
Θcircle satisfies the extended deflection condition

Θcircle < π (8.4)

G. Chavent, Nonlinear Least Squares for Inverse Problems: Theoretical Foundations 321
and Step-by-Step Guide for Applications, Scientific Computation,
DOI 10.1007/978-90-481-2785-6 8, c Springer Science+Business Media B.V. 2009
322 CHAPTER 8. DEFLECTION CONDITIONS

for the set D to remain s.q.c., at the price of a possible reduction of the size
of the regular enlargement neighborhood ϑ to RG ≤ R given by

1 if 0 ≤ Θcircle ≤ π/2,
RG = R × (8.5)
sin Θ if π/2 < Θcircle < π.

Because of (8.2), conditions (8.3) and (8.4) are size×curvature conditions,

as they ensure strict quasi-convexity by limiting the product of the size L of
the arc of circle by its curvature 1/R.
To see how the above results can be generalized, we define the size and
the deflection associated to a path p ∈ P and to a set (D, P) equipped with
a family of paths (the global radius of curvature and the radius of curvature
associated to p and D have been defined in Chap. 7, Definitions 7.2.3 and
7.2.8):

Definition 8.0.3 (Size of paths and sets) Let (D, P) and p ∈ P be given.
The size of p is its arc length (Definition 6.1.2):

L(p) = , where p : [0, ]  F, (8.6)

and that of D is

L(D) = sup L(p). (8.7)

p∈P

Formula (8.6) for L(p) is duplicated from Definition 6.1.1, which gives the
geometrical quantities associated to a path. Then (8.7) defines L(D) as the
size of D measured along its paths.

Definition 8.0.4 (Deflection, see Fig. 8.1)

Let (D, P) and p ∈ P be given. For any ν, ν  ∈ [0, L(p)], the deflection
of p between ν and ν  is

θ(ν, ν  ) = Arg cos  v(ν), v(ν  ) F , (8.8)

which satisfies
⎧
⎨ θ(ν, ν  ) ∈ [0, π] ∀ν, ν  ∈ [0, L(p)],
θ(ν, ν) = 0 ∀ν ∈ [0, L(p)], (8.9)
⎩
θ(ν, ν  ) = θ(ν  , ν) ∀ν, ν  ∈ [0, L(p)].
CHAPTER 8. DEFLECTION CONDITIONS 323

v(ν) θ(ν, ν )

p(ν)
p(ν ) p

v(ν )

Figure 8.1: The deflection θ(ν, ν  ) between two points ν and ν  (ν > ν  )
of a path p

The deflection of the path p is

Θ(p) = sup θ(ν, ν  ), (8.10)
ν,ν  ∈[0,L(p)]

and the deflection of (D, P) is

Θ(D) = sup Θ(p). (8.11)
p∈P

In Sect. 8.1, we search sufficient conditions for a (D, P) to be s.q.c.

We study first the properties of the deflection, and prove in
Proposition 8.1.2 that
 L(p)
dν
∀p ∈ P, Θ(p) ≤ ≤ L(p)/R(p) ≤ L(D)/R(D). (8.12)
0 ρ(ν)
This shows that the deflection Θ(D) of the set D is always smaller than
the deflection of an arc of circle with radius R(D) and arc length L(D) (see
(8.2)). From the point of view of deflection, arcs of circles, which “steadily
turn in the same direction” are the worst sets!
In practice, it will be only possible in applications to have access to
lower/upper bounds to R(D), L(D), Θ(D), and RG (D), and so we give them
a name:
Definition 8.0.5 We call geometric attributes of the set (D, P) any set of
numbers R (radius of curvature), L (size), Θ (deflection), RG (global radius
of curvature), which are lower/upper bounds to
324 CHAPTER 8. DEFLECTION CONDITIONS

R(D) ≥ 0, L(D) > 0, Θ(D) ≤ L(D)/R(D), RG (D) ≤ R(D). (8.13)

and satisfy
R ≥ 0, L > 0, Θ ≤ L/R, RG ≤ R. (8.14)
Then we search a lower bound RG to RG (D) as a function of the
lower/upper bounds R, L, and Θ. We prove in Theorem 8.1.5 that
def
∀p ∈ P, RG (p) ≥ RG (D) ≥ RG = RG (R, L, Θ), (8.15)

where

r,   0 ≤ θ ≤ π/2,
RG (r, l, θ) = (8.16)
r sin θ + l − rθ cos θ, π/2 ≤ θ ≤ π.

This shows first, using Theorem 7.2.5, that a sufficient condition for a set
(D, P) with finite curvature 1/R to be s.q.c. with an enlargement neighbor-
hood ϑ of size RG = R > 0 is to satisfy the deflection condition

Θ ≤ π/2, (8.17)

which generalizes condition (8.3). If one accepts the possibility of an enlarge-

ment neighborhood ϑ of reduced size RG ≤ R, the same theorem shows that
a finite curvature set (D, P) is s.q.c. as soon as R, L, and Θ ≤ L/R satisfy
the extended deflection condition
 
RG R, L, Θ > 0. (8.18)

We show in Fig. 8.2 the set of values of the deflection Θ and the size × curva-
ture product L/R that satisfy (8.18), and ensure that the set (D, P) is s.q.c.,
with a regular enlargement neighborhood ϑ of size 0 < RG R, L, Θ ≤ R.
A simple calculation shows that (8.18) is equivalent to

def
Θ ≤ Θmax = L/R if 0 ≤ L/R < π, (8.19)
Θ < Θmax such that Θmax − tan Θmax = L/R if π ≤ L/R.

This justifies the name given to condition (8.18): the upper limit to the
deflection Θ is extended beyond π/2, up to a value θmax which depends on the
estimated size×curvature product L/R. This extended deflection condition
reduces to the condition (8.4) obtained for an arc of circle as soon as the
“worst” estimate Θ = L/R is used for the deflection.
CHAPTER 8. DEFLECTION CONDITIONS 325

L/R
size
×
curvature RG < 0

π RG > 0

Θ
0 π/2 π deflection

Figure 8.2: The domain of deflection Θ and size × curvature product

L/R, which ensure strict quasi-convexity of a finite curvature set
(D, P) (extended deflection conditions (8.18) or (8.19))

Finally, in Sect. 8.2, we consider the case where the set D is the attainable
set ϕ(C) of a nonlinear least squares problem set over a convex set C of
admissible parameters. It is then natural to try to equip D with the set of
paths P made of the images by ϕ of the segments of C: to any x0 , x1 ∈ C,
one can always associate a curve P drawn on D = ϕ(C) by
 
P : t ∈ [0, 1]  P (t) = ϕ (1 − t)x0 + tx1 . (8.20)

The first question is to know under which conditions the curve P – when
it is not reduced to a point – becomes, once reparameterized by its arc length
ν, a path p in the sense of Definition 6.1.1, that is, a W 2,∞ function of ν.
A necessary condition is of course to require that
 
P ∈ W 2,∞ [0, 1]; F , (8.21)
326 CHAPTER 8. DEFLECTION CONDITIONS

which allows to define the velocity V (t) and acceleration A(t) along the
curve by
dP d2 P
V (t) = (t), A(t) = (t). (8.22)
dt dt2
(We reserve the lower case notation p, v, and a to path, velocity, and accel-
eration with respect to arc length, as in Definition 6.1.2).
But under the sole hypothesis (8.21), the reparametrization
 p of P with
respect to arc length satisfies only p ∈ W 1,∞
[0, ]; F . So our first task will
be to show in Proposition 8.2.2 that the additional condition

∃R ∈]0, +∞] such that
   
A(t) ≤ 1 V (t)2 a.e. on ]0, 1[ (8.23)
F R F

ensures that, when P is not reduced to a point, the reparameterization p

of P by its arc
2,∞
 length has a finite curvature 1/R(p) ≤ 1/R, so that p ∈
W [0, ]; F , and hence is a path in the sense of Definition 6.1.1.
This will make it possible when (8.21) and (8.23) are satisfied for any
x0 , x1 ∈ C, to equip the attainable set ϕ(C) with the family of paths P made
of the reparameterizations p of the curves P , which are images of segments
of C by ϕ and are not reduced to a point.
To apply to (ϕ(C), P) the sufficient conditions for strict quasi-convexity
of Sect. 8.1, it is necessary to estimate the geometric attributes R, L, Θ of
(ϕ(C), P). This is addressed in the last Proposition 8.2.2, where it is shown
that, if there exists αM ≥ 0 and R > 0 satisfying
∀x0 , x1 ∈ C, V (t)F ≤ αM x1 − x0 E ∀t ∈ [0, 1], (8.24)
1
∀x0 , x1 ∈ C, A(t)F ≤ V (t)2F a.e. on ]0, 1[, (8.25)
R
then R is a lower bound to the radius of curvature of (ϕ(C), P):
def
inf R(p) = R(ϕ(C)) ≥ R > 0. (8.26)
p∈P

and any numbers L and Θ that satisfy

 1
∀x0 , x1 ∈ C, V (t)F dt ≤ L ≤ αM diam(C), (8.27)
0
⎧ 1
⎨ 0 θ(t) dt ≤ Θ ≤ L/R,
where (8.28)
⎩
A(t)F ≤ θ(t)V (t)F for a.e. t ∈ [0, 1] and all x0 , x1 ∈ C,
8.1. THE GENERAL CASE: D ⊂ F 327

are upper bounds to the arc length size and deflection of (ϕ(C), P). Hence
R, L, Θ are the sought geometric attributes of ϕ(C), P.
The sufficient condition (8.18) for strict quasi-convexity (Sect. 8.1) and
the estimations (8.24) through (8.28) of R, L, Θ (Sect. 8.2) are the basis for
all Q-wellposedness results of Chaps. 4 and 5. Examples of application of these
conditions can be found in Sect. 5.2, where it is proved that the attainable
set of the 2D elliptic nonlinear source estimation problem of Sect. 1.5 is s.q.c.,
and in Sects. 4.8 and 4.9, where the same result is proved for the 1D and 2D
parameter estimation problems of Sects. 1.4 and 1.6.

8.1 The General Case: D ⊂ F

We consider in this section a set D ⊂ F equipped with a family of paths
P in the sense of Definition 6.1.3, and we search for sufficient conditions for
(D, P) to be s.q.c.
Definition 8.1.1 Given a function g : [a, b]  IR, we denote by vara,b g or
varb,a g the total variation (when it exists!) of g over the [a, b] interval
⎧
⎪ + N
⎪
⎨ vara,b g = varb,a g = sup |g(ti) − g(ti−1 |, N ∈ IN,
(8.29)
⎪
⎪ & '
i=1
& ',
⎩ min a, b ≤ t0 ≤ t1 ≤ ... ≤ tN ≤ max a, b .

When g ∈ W 1,1 ([a, b]), the total variation is given by

 b
varab g = |g (t)| dt. (8.30)
a

We investigate now the regularity of the deflection ν  θ(ν, ν  ), and its

relation to arc length ν and radius of curvature ρ(ν).
Proposition 8.1.2 (deflection estimate) Let p be a path of P with length
L(p) and smallest radius of curvature R(p) > 0. Then
1. For any ν  ∈ [0, L(p)], the ν  θ(ν, ν  ) deflection function is abso-
lutely continuous and has a bounded variation over [0, L(p)]. Hence
∂θ/∂ν(., ν  ) ∈ L1 ([0, L(p)]), and the usual formula holds:
 ν
 ∂θ
θ(ν, ν ) = (t, ν  ) dt. (8.31)
ν  ∂ν
328 CHAPTER 8. DEFLECTION CONDITIONS

2. Moreover,
 ∂θ  1
 (ν, ν  ) ≤ a(ν)F def
= for a.e. ν ∈ [0, L(p)], (8.32)
∂ν ρ(ν)
so that ∂θ/∂ν(., ν  ) ∈ L∞ ([0, L(p)]).
3. The largest deflection Θ(p) along p satisfies
 L(p)
∂ν
Θ(p) ≤ aL1 (0,L(p);F ) = ≤ L(p)/R(p). (8.33)
0 ρ(ν)

Proof: Let p ∈ P and ν  ∈ [0, L(p)] be given, and choose 0 < θ < π/2. The
function θ(., .) is continuous over [0, L(p)] × [0, L(p)], and hence uniformly
continuous. So there exists Δν > 0 such that

νj ∈ [0, L(p)] j = 1, 2 and |ν1 − ν2 | ≤ Δν (8.34)

implies
θ(ν1 , ν2 ) ≤ θ. (8.35)
A Taylor–MacLaurin development of cos t at t = 0 gives
t2
cos t = 1 − cos αt for some α ∈ [0, 1],
2
and, as the cosine is a decreasing function over [0, π]
t2
cos t ≤ 1 − cos t for 0 ≤ t ≤ π.
2
Hence,  
t2 cos t ≤ 2 1 − cos t for 0 ≤ t ≤ π.
Choosing t = θ(ν1 , ν2 ), where ν1 , ν2 satisfy (8.34) gives, as then cos t ≥
cos θ > 0 and cos θ(ν1 , ν2 ) = v1 , v2 ,
 2 2(1 − v1 , v2 )
θ ν1 , ν2 ≤ .
cos θ
This can be rewritten as
v1 − v2 F
θ(ν1 , ν2 ) ≤ ,
(cos θ)1/2
8.1. THE GENERAL CASE: D ⊂ F 329

and, using the triangular inequality for the curvilinear triangle v1 , v2 , v  on

the unit sphere,
 
θ(ν1 , ν  ) − θ(ν2 , ν  ) ≤ v1 − v2 F . (8.36)
(cos θ)1/2

We prove first the absolute continuity of the ν  θ(ν, ν  ) function. Let  > 0
be given, and (αi , βi ), i = 1, 2...N, be disjoint segments of the interval [0, L(p)]
satisfying
βi − αi ≤ Δν i = 1, 2...N.
Then we get from (8.36) that
N N
  1
|θ(βi , ν ) − θ(αi , ν )| ≤ v(βi ) − v(αi )F ,
i=1
(cos θ)1/2 i=1

where, as p ∈ W 2,∞ ([0, L(p)]; F ):

 βi
v(βi ) − v(αi ) = a(t) dt
αi

so that
 βi  
v(βi ) − v(αi )F ≤ a(t)F dt ≤ βi − αi a∞ .
αi

Hence,
n
     a∞
n
 
θ βi , ν  − θ αi , ν   ≤ βi − αi , (8.37)
i=1
(cos θ)1/2 i=1

which can be made smaller than  by choosing the intervals (αi , βi ) such that
n
 + (cos θ)1/2 ,
βi − αi ) ≤ min Δν,  .
i=1
a∞

This proves that the function θ(., ν  ) is absolutely continuous over the [0, L(p)]
interval, which in turn implies that ∂θ/∂ν(., ν  ) is in L1 (0, L(p)), and that
formula (8.31) holds.
330 CHAPTER 8. DEFLECTION CONDITIONS

We prove now that θ(., ν  ) has a bounded variation over [0, L(p)]. Let

0 ≤ t0 < t1 < .... < tN ≤ L(p)

be given. One can always add a finite number of points to obtain a new
subdivision:
  
0 ≤ t0 < t1 < ..... < tN  ≤ L(p),
with N  ≥ N, such that
 
|ti − ti−1 | ≤ Δν i = 1, 2...N  . (8.38)

Then of course one has, because of the triangular inequality,

N N
 
|θ(ti , ν ) − θ(ti−1 , ν )| ≤ |θ(ti , ν  ) − θ(ti−1 , ν  )|,
i=1 i=1

and because of (8.38), one obtains, as in (8.37),

N N
a∞
|θ(ti , ν  ) − θ(ti−1 , ν  )| ≤ |ti − ti−1 |.
i=1
(cos θ)1/2 i=1

The two last inequalities imply immediately that

N
a∞
|θ(ti , ν  ) − θ(ti−1 , ν  )| ≤ L(p)
i=1
(cos θ)1/2

independently of the positions and number of the points ti , which proves that
the function θ(., ν  ) has a bounded total variation.
We prove now formula (8.32). Let ν ∈ [0, L(p)] be a Lebesgue point for
both a ∈ L∞ ([0, L(p)]; F ) and ∂/∂ν(., ν  ) ∈ L1 ([0, L(p)]) (almost every point
of [0, L(p)] has this property!), and dν = 0 such that ν + dν ∈ [0, L(p)] and
|dν| ≤ Δν defined at the beginning of the proof. Then we get from (8.36)
that -
 θ(ν + dν, ν  ) − θ(ν, ν  )  v(ν + dν) − v(ν) |dν|
  ≤ F
,
dν (cos θ)1/2
which, by definition of the Lebesgue points, converges when dν → 0 to
 ∂θ  a(ν)F
 (ν, ν  ) ≤ .
∂ν (cos θ)1/2
8.1. THE GENERAL CASE: D ⊂ F 331

But θ can be chosen arbitrarily in the ]0, π/2[ interval, which proves (8.32)
as cos θ can be made arbitrarily close to one.
Finally, (8.33) follows immediately from (8.31) and (8.32), and
Proposition 8.1.2 is proved.
We turn now to the estimation of a lower bound to the global radius of
curvature RG (p) of a path p. Because of Proposition 7.3.1, this amounts to
search for a lower bound to ρep
G
(ν, ν  ) independent of ν and ν  , where
 +
ep  N /D if v, v   ≥ 0,
ρG (ν, ν ) = (8.39)
N +
if v, v   ≤ 0,

and N and D are defined by (cf. formulas (7.24) in Proposition 7.2.6)

N = sgn (ν  − ν) X  − X, v  , (8.40)
 1/2
D = 1 − v, v  2 . (8.41)
We give first a lower bound on the numerator N.

Lemma 8.1.3 Let (D, P) and p ∈ P be given. Then, for any ν, ν  ∈ [0, L(p)]
one has  
N ≥ R sin θ + |ν  − ν| − R θ cos θ, (8.42)
where
& '
R = inf ess min(ν, ν  ) < t < max(νν  ) , (8.43)
ρ(t),
& '
θ = sup ess θ(t, ν  ), min(ν, ν  ) < t < max(νν  ) , (8.44)

Proof. We notice first that

 ν
 
X − X , v  = v(t), v   dt
ν
 ν
= cos θ(t, ν  ) dt.
ν

Hence, if we define

ν − = min(ν, ν  ), ν + = max(ν, ν  ), (8.45)

332 CHAPTER 8. DEFLECTION CONDITIONS

then we obtain for N the formula

 ν+
N = cos θ(t, ν  ) dt,
ν−

and, using θ defined by (8.44)

 ν+
   
N= cos θ(t, ν  ) − cos θ dt + ν + − ν − cos θ. (8.46)
ν−

To find a lower bound on N, we notice that, by definition of θ, one has

cos θ(t, ν  ) − cos θ ≥ 0 for all t ∈ [ν − , ν + ],
so that one can plug into the integral in (8.46) the inequality
 ∂θ 
1 ≥ ρ(t) (t, ν  ) a.e. on ]ν − , ν + [,
∂ν
which has been proven in Proposition 8.1.2. This leads, as ρ(t) ≥ R defined
in (8.43), to
 ν+
   ∂θ 
N ≥R cos θ(t, ν  ) − cos θ  (t, ν  )dt + (ν + − ν − ) cos θ,
ν− ∂ν
that is,
 ν+ ∂  
N ≥R  sin θ(t, ν  ) − θ(t, ν  ) cos θ dt + (ν + − ν − ) cos θ,
ν− ∂ν
that is,
   
N ≥ R varν − ,ν + sin θ(., ν ) − θ(., ν ) cos θ + ν + − ν − cos θ.
 
(8.47)

But θ(., ν  ) is continuous over the [ν − , ν + ] interval, and so there exists ν ∈

[ν − , ν + ] such that
θ(ν, ν  ) = θ.
The formula (8.29) defining the total variation, applied with N = 1, t0 = ν,
t1 = ν  , gives then, as θ(ν  , ν  ) = 0,
   
varν − ,ν + sin θ(., ν  ) − θ(., ν  ) cos θ ≥  sin θ − θ cos θ . (8.48)
But the function t  sin t − t cos t is positive over the [0, π] interval. Hence
we can drop the absolute value in (8.48), and substitute it into (8.47), which
gives the desired lower bound (8.42) on N.
8.1. THE GENERAL CASE: D ⊂ F 333

Remark 8.1.4: The lower bound (8.42) on N retains its maximum value
only from the shape of the deflection θ(., ν  ). However, the inequality (8.47)
shows that, among pieces of paths having the same size |ν  − ν| and curvature
1/R, the ones whose deflection has a large total variation are more likely
to have positive global radii of curvature, and hence to be s.q.c. (the θ →
sin θ − θ cos θ function is nondecreasing over the [0, θ] interval, and so a
large variation of θ(., ν  ) corresponds to a large variation of sin θ(., ν  ) −
θ(., ν  ) cos θ, and hence is more likely to produce through (8.47) a positive
lower bound to N).
But the total variation of the deflection is difficult to estimate in applica-
tions, and so we shall retain only the less sharp estimate (8.42) based on the
maximum deflection.

Proposition 8.1.5 (Global radius of curvature estimate). Let p be a path of

P, and denote by R(p) > 0 its smallest radius of curvature, L(p) > 0 its
length, and 0 ≤ Θ(p) ≤ L(p)/R(p) its largest deflection.
Then its smallest global radius of curvature RG (p) satisfies
 
RG (p) ≥ RG R(p), L(p), Θ(p) , (8.49)

where RG (r, l, θ), defined by (8.16), is a decreasing function of 1/r, l, and θ

over the domain 0 ≤ θ ≤ π, l − rθ ≥ 0.
In particular, if R, L, Θ are three first geometric attributes of (D, P)
(Definition 8.0.5), a fourth one is given by
def
RG = RG (R, L, Θ). (8.50)

Proof. The partial derivatives of RG are positive with respect to r, and neg-
ative with respect to l and θ over the domain of definition, which proves the
announced monotonicity property.
Let now p ∈ P be given. We shall drop in the rest of the proof the argu-
ment p in R(p), RG (p), L(p), Θ(p), and write simply instead R, RG , L, Θ.
Let then ν, ν  ∈ [0, ] be given. Formula (8.33) shows that
  
ν − ν  − R θ ≥ 0, (8.51)

where R and θ are defined in (8.43) and (8.44).

334 CHAPTER 8. DEFLECTION CONDITIONS

We consider first the case where 0 ≤ Θ ≤ π/2. By definition of θ and Θ

one has
θ(ν, ν  ) ≤ θ ≤ Θ ≤ π/2, (8.52)
and, as the cosine function is decreasing over the [0, π/2]

v, v  = cos θ(ν, ν  ) ≥ cos θ ≥ 0.

This implies using (8.39) that

 
ρep
G
ν, ν = N + /D ≥ N/D, (8.53)

and using (8.42) and (8.51) that

N ≥ R sin θ. (8.54)

The sine function is increasing over [0, π/2], and so we obtain from (8.41)
that  1/2
D = 1 − v, v  2 = sin θ(ν, ν  ) ≤ sin θ. (8.55)
Then (8.53), (8.54), and (8.55) imply

ρep
G
(ν, ν  ) ≥ R ,

which implies (8.49) as R ≥ R = RG (R, L, Θ).

We turn now to the case where π/2 < Θ ≤ π.
The maximum deflection θ̄ on p between ν and ν  satisfies 0 ≤ θ̄ ≤ Θ. Hence
two cases can happen:

• Either 0 ≤ θ ≤ π/2, and then one finds as above that

ρep
G
(ν, ν  ) ≥ R ≥ R. (8.56)

But 0 ≤ sin Θ ≤ 1, L/R − Θ ≥ 0 because of (8.33), and cos Θ ≤ 0,

and so
 
ρep
G
(ν, ν  ) ≥ R sin Θ ≥ R sin Θ + L − RΘ cos Θ = RG (R, L, Θ),

which implies (8.49).

8.1. THE GENERAL CASE: D ⊂ F 335

• Or π/2 < θ ≤ Θ, and then v, v   = cos θ(ν, ν  ) can be either positive
(if 0 ≤ θ(ν, ν  ) ≤ π/2) or negative (if π/2 ≤ θ(ν, ν  ) ≤ θ), and so the
only information we get from (8.39) is

ρep
G
(ν, ν  ) ≥ N + ≥ N.

Combined with lemma (8.31), this shows that

 
ρep
G
(ν, ν  ) ≥ R sin θ + |ν  − ν| − R θ cos θ = RG (R, |ν  − ν| , θ).

But
R≥R>0 |ν  − ν| ≤ L θ ≤ Θ,

0 ≤ θ ≤ |ν  − ν|/R 0 ≤ Θ ≤ L/R,
so the monotonicity property of RG shows again that

ρep
G
(ν, ν  ) ≥ RG (R, L, Θ),

which implies (8.49). Then (8.50) follows immediately from the mono-
tonicity properties of the r, l, θ  RG function.

We can now state the main result of this section:

Theorem 8.1.6 Let (D, P) be a set equipped with a family of paths, and
R, L, Θ be three first geometric attributes of (D, P) (Definition 8.0.5). Then

• (D, P) is s.q.c. as soon as the fourth geometric attribute given by (8.50)

satisfies RG > 0, that is,
def
RG = RG (R, L, Θ) > 0, (extended deflection condition) (8.57)

where the function RG is defined in (8.16)

• The global radius of curvature of (D, P) satisfies

RG (D) ≥ RG > 0 (8.58)

Proof. It follows immediately from (8.50) in Proposition 8.1.5 that (D, P)

has a strictly positive global radius of curvature. It is hence s.q.c. by virtue
of Theorem 7.2.5.
336 CHAPTER 8. DEFLECTION CONDITIONS

We have already illustrated in Fig. 8.2 the domain of deflection estimates

Θ and size × curvature estimates L/R, which satisfy (8.57). We show now in
Fig. 8.3 that these conditions are sharp: as soon as the deflection estimate Θ
becomes strictly larger than π/2, the upper bound L/R < Θ − tan Θ on the
size × curvature estimate is the best possible. The set D of Fig. 8.3 is made of
a path p, made itself of two parts: an arc of circle of radius R and deflection
Θ ∈]π/2, π[ and, tangent at one end of the arc of circle, a segment of length
| tan Θ| = − tan Θ. This set is not s.q.c.: let us choose for p the longest path
of D, that is, p = D, and for z the end of p located on the segment. Then
d(z, p) = 0, but the function ν  dz,p(ν)2 has, beside its global minimum
at z (with value zero !), a parasitic stationary point, with zero derivative, at
the other end of p. Such a function is not s.q.c., so that property (7.4) of the
Definition 7.2.2 of s.q.c. sets cannot hold true for D!
But the best curvature, deflection, and size estimates for this set D are
obviously R, Θ, and L = R(Θ − tanΘ), which satisfy exactly (8.57) – or
the second inequality of (8.19) – with the strict inequality replaced by an
equality. Hence the upper bound (8.19) on L/R is the smallest, which can
ensure the strict quasi-convexity of (D, P).

R tanΘ

RΘ z
p

Figure 8.3: Illustration of the sharpness of the extended deflection

condition (8.57)
8.2. THE CASE OF AN ATTAINABLE SET D = ϕ (C) 337

8.2 The Case of an Attainable Set D = ϕ (C)

We consider in this section a set D of F , which is the image, by some mapping
ϕ of a convex set C of E. In the context of inverse problem, E is the parameter
space, C the set of admissible parameters, F the data space, and ϕ the
forward map to be inverted. Strict quasi-convexity of the attainable set D =
ϕ(C) will be the key to Q-wellposedness of the inverse problem (see Chap. 4).
So we discuss first in this section the possibility of equipping D = ϕ(C) with
a family of path P, as required in the definition of s.q.c. sets, and we estimate
a set of geometric attributes R, L, and Θ, as required by Theorem 8.1.6, to
prove that (ϕ(C), P) is s.q.c.
We suppose now that
⎧
⎨ E = Banach space,
C ⊂ E convex, (8.59)
⎩
ϕ : C  F.
A natural way to equip D = ϕ(C) with a family of paths is to consider the
curves P : [0, 1] → D, which are image by ϕ of some segment of C:

to x0 , x1 ∈ C we associate
 P : [0, 1]→ D s.t. :
(8.60)
∀t ∈ [0, 1], P (t) = ϕ (1 − t)x0 + tx1 .
We suppose that the forward map ϕ is smooth enough so that
 
∀x0 , x1 ∈ C, P ∈ W 2,∞ [0, 1]; F , (8.61)
and we denote, as indicated in (8.22), by V (t) and A(t) the first and second
derivatives of P .
To see derivatives whether a curve P defined as above can be turned into
a path p in the sense of Definition 6.1.1, we have first to reparameterize it as
a function of its arc length ν, defined by
 t
 
∀t ∈ [0, 1], ν(t) = V (t) dt,
F
0

which satisfies
def
0 ≤ ν ≤  = ν(1),
and the arc length of the curve P is
 1  
L(P ) =  = ν(1) = V (t) dt. (8.62)
F
0
338 CHAPTER 8. DEFLECTION CONDITIONS

Definition 6.1.1 requires that paths of ϕ(C) have a strictly positive length.
Hence we shall consider only in the sequel the curves P such that L(P ) > 0 .
By construction, t  ν(t) is a nondecreasing mapping from [0, 1] onto
[0, ], which can be constant on some intervals of [0, 1]. The reparameterized
path p : [0, ] → D is hence unambiguously defined by
 
p ν(t) = P (t) ∀t ∈ [0, 1]. (8.63)

It will be convenient to denote the velocity and acceleration along p, that is,
the two first derivatives of p with respect to ν, when they exist, by the lower
case letters

v(ν) = p (ν), a(ν) = v  (ν) = p (ν) ∀ν ∈ [0, ].

Proposition 8.2.1 Let (8.59) and (8.61) hold, and x0 , x1 ∈ C be such that
the curve P defined by (8.60) has an arc length L(P ) > 0. Then its reparam-
eterization p by arc length defined in (8.63) satisfies
 
p ∈ W 1,∞ [0, ]; F , (8.64)
 
v(ν) = 1 a.e. on [0, ], (8.65)
F
and P has, at all points t ∈ [0, 1], where V is derivable and V (t) = 0, a finite
curvature 1/ρ(t) – that is, a radius of curvature ρ(t) > 0 – given by
5 2  26 12
1 AF A V AF
= a(ν(t))F = − , ≤ . (8.66)
ρ(t) V 2F V F V F
2 V 2F

 
Proof. The reparameterized path p is in L∞ [0, ]; F by construction, and
hence defines a distribution on ]0, [ with values in F. We compute the∞deriva-
 
tive v of this distribution p. For any ϕ ∈ D ]0, [ (the space of C ]0, [
functions with compact support in ]0, [) one has

v, ϕ = − p(ν) ϕ (ν) dν
0
 1
   
= − p ν(t) ϕ ν(t) ν  (t) dt
0 1
d  
= − P (t) ϕ ν(t) dt
0 dt
8.2. THE CASE OF AN ATTAINABLE SET D = ϕ (C) 339
 
But ν(.) belongs to W 1,∞ (]0, 1[) and so does ϕ ν(.) , with moreover zero

values at t = 0 and t = 1. So we can integrate by part, as P ∈ W 2,∞
]0, 1[; F ,
 1
 
v, ϕ = V (t) ϕ ν(t) dt. (8.67)
0

To express- v, ϕ, as an integral with respect to ν, one has to replace in (8.67)
dt by dν V (t)F , which is possible only if V (t) = 0. So we define
+ ,
I = t ∈]0, 1[ | V (t) = 0 .

As I is an open set, it is the reunion of a countable family of pair-wise disjoint

intervals:
7∞
I= Ii , where Ii =]αi , βi [ , i = 1, 2....
i=1

So we can rewrite (8.67) as


 
v, ϕ = V (t) ϕ ν(t) dt
I
∞  βi  
= V (t) ϕ ν(t) dt
i=1 αi
∞  βi
V (t)  
= ϕ ν(t) V (t)F dt
i=1 αi V (t)F

We define now
7
∞
J = Ji , where Ji = ν(Ii ) i = 1, 2....
i=1

The sets Ji i = 1, 2... are (also pair-wise disjoints) open intervals of ]0, [. So
we can associate to any ν ∈ J a number t(ν) ∈]0, 1[, which is the reciprocal
of the t  ν(t) function over the interval Ji containing ν. Hence we see that
∞ 
V (t(ν))
v, ϕ = ϕ(ν) dν,
i=1 Ji
V (t(ν))F

V (t(ν))
v, ϕ = ϕ(ν) dν, (8.68)
J V (t(ν))F
340 CHAPTER 8. DEFLECTION CONDITIONS

and similarly that

 1 
 = V (t)F dt = V (t)F dt
0 I
∞  βi ∞ 
 = V (t)F dt = dν
αi Ji
i=1 i=1

 = dν = meas J
J

Hence we see that the complementary of J in ]0, [ has zero measure. If we

continue by some t0 ∈ I the ν → t(ν) function on this zero-measure set, we
can rewrite (8.68) as

V (t(ν))
v, ϕ = ϕ(ν) dν,
0 V (t(ν))F

which shows that the distribution v = p is in fact a function

V (t(ν))
v(ν) = a.e. on ]0, [. (8.69)
(V t(ν))F
This proves (8.64) and (8.65). Given any ν ∈ J, we can differentiate (8.69)
with respect to ν. This gives
 
A V A V
a(ν) = − , ,
V 2F V F V 2F V F
and  2  2
AF A V
a(ν)F =
2
− , , (8.70)
V 2F V F V F
2

where right-hand sides are evaluated at t = t(ν). Hence for any t ∈ I, such
that V (t)F > 0, one has ν(t) ∈ J, and (8.70) shows that
A(t)F
a(ν(t)) ≤ < +∞,
V (t)2F
which is (8.66).

So we see that the hypothesis that P ∈ W 2,∞ [0, 1]; F ) is not enough to
ensure that its reparametrization p as a function of arc length is W 2,∞ ([0, ]; F ):
in general, the derivative v of p can have discontinuities at points ν ∈ J!
8.2. THE CASE OF AN ATTAINABLE SET D = ϕ (C) 341

Proposition 8.2.2 Let (8.59) and (8.61) hold, and x0 , x1 ∈ C be such that
the curve P associated by (8.60) satisfies
  1  
there exists RP > 0 s.t.: A(t)F ≤ V (t)2
F
a.e. on ]0, 1[. (8.71)
RP
Then one of the two following properties holds:
• Either
V (t) = 0 ∀t ∈ [0, 1] ⇐⇒ L(P ) = 0, (8.72)
where L(P ) is the length of P defined in (8.62), and the curve P is
reduced to one point of ϕ(C),
• or
V (t) = 0 ∀t ∈ [0, 1] =⇒ L(P ) > 0, (8.73)
and the reparameterization p of P by its arc length, defined by (8.63),
is a path in the sense of Definition 6.1.1. So we can define the radius
def
of curvature of the curve P by R(P ) = R(p) and its emphasize by
def
Θ(P ) = Θ(p), which satisfy the following:
(i) Curvature estimate:
R(P ) ≥ RP > 0. (8.74)

(ii) Deflection estimate:

 1
A(t)F L(P )
Θ(P ) ≤ dt ≤ . (8.75)
0 V (t)F RP

Proof. Let P ∈ W 2,∞ ([0, 1]; F ) satisfying (8.71) be given.

To prove (8.72) and (8.73), we suppose that V (t0 ) = 0 for some t0 ∈ [0, 1],
and we prove that V (t) = 0 ∀t ∈ [0, 1]. Suppose this is not true, and define
I =]α, β[⊂ [0, 1] as the largest interval containing t0 such that V (t) = 0 ∀t ∈ I.
Then necessarily α > 0 and/or β < 1, say β < 1, in which case V (β) = 0.
Then because of (8.71), the function g(t) = V (t)F satisfies
 
 dg 
 (t) ≤ A(t)F ≤ V (t)F ≤ g(t)
2 2

 dt  ∀t ∈ I,
RP R
   
d 1 
 (t) ≤ 1 ∀t ∈ I. (8.76)
 dt g  RP
342 CHAPTER 8. DEFLECTION CONDITIONS

Hence,
1 1 |t − t0 | 1 1
≤ + ≤ + ∀t ∈ I,
g(t) g(t0 ) RP g(t0 ) RP
so that  1
def 1 −1
g(t) ≥ c = + >0 ∀t ∈ I.
g(t0 ) RP
It follows that g(β) = V (β)F ≥ c > 0, which is a contradiction as we have
seen that V (β) = 0. Hence I =]0, 1[, and (8.72) and (8.73) are proved.
Let now (8.73) hold, and p be the reparameterization of P by arc length.
As V (t) = 0 ∀t ∈ [0, 1], formula (8.65) of Proposition 8.2.1 applies for all
t ∈ [0, 1], where V is derivable, that is, almost everywhere on [0, 1]. Hence
a(t)F ≤ 1/RP for a.e. t ∈ [0, 1], so that p ∈ W 2,∞ ([0, L(P )]; F ), and p is a
path of curvature smaller than 1/RP , which proves (8.74).
Then Proposition 8.1.2 part (iii) gives the following estimate for the de-
flection of p:
 L(P )
Θ(p) ≤ a(ν) dν. (8.77)
0
Changing for the variable t ∈ [0, 1] in the integral gives
 1
Θ(p) ≤ a(ν(t)) V (t) dt, (8.78)
0

and, because of (8.66)

a(ν(t)) V (t) ≤ A(t)/V (t),
which proves the first inequality in (8.75). Then (8.71) gives
 1  1  1
A(t) A(t) 1 L(P )
dt = V (t) dt ≤ V (t) dt = ,
0 V (t) 0 V (t)
2
0 RP RP
and the last inequality in (8.75) is proved.
We can now equip ϕ(C) with the family of curves P – or of paths p –
defined by
+ ,
P = P : t ∈ [0, 1]  ϕ((1 − t)x0 + tx1 ), x0 , x1 ∈ C | L(P ) > 0 . (8.79)

The next theorem gives a sufficient condition, which guarantees that P is a

family of path of ϕ(C) in the sense of Definition 6.1.3, and provides a set
of geometric attributes R, L, Θ of the attainable set (ϕ(C), P), given the
velocity V (t) and acceleration A(t) along the curves of P.
8.2. THE CASE OF AN ATTAINABLE SET D = ϕ (C) 343

Theorem 8.2.3 Let C and ϕ be given such that (8.59) and (8.61) hold.
(i) If there exists R > 0 such that
1
∀x0 , x1 ∈ C, A(t)F ≤ V (t)2F a.e. in [0, 1], (8.80)
R
then the family of curves P defined in (8.79) is, once reparameterized by
the arc length using (8.63), a family of paths of ϕ(C) in the sense of
Definition 6.1.3, and the attainable set (ϕ(C), P) has a finite curvature:

R(ϕ(C)) ≥ R > 0. (8.81)

(ii) If there exists αM ≥ 0 such that

∀x0 , x1 ∈ C, V (t)F ≤ αM x1 − x0 E a.e. in [0, 1], (8.82)

then any number L ≥ 0 that satisfies

 1
∀x0 , x1 ∈ C, V (t)F dt ≤ L ≤ αM diam(C) (8.83)
0

is an upper bound to the (arc length) size L(ϕ(C)) of (ϕ(C), P).

(iii) Any number Θ ≥ 0 that satisfies
⎧ 1
⎨ 0 θ(t) dt ≤ Θ ≤ L/R
where (8.84)
⎩
A(t)F ≤ θ(t)V (t)F for a.e. t ∈ [0, 1] and all x0 , x1 ∈ C

is an upper bound to the deflection Θ(ϕ(C)) of (ϕ(C), P).

Proof. The announced results follow immediately from Proposition 8.2.2 by

considering the worst case over all curves P of P.
Bibliography

[1] Aki, K., Richards, P.G., 1980, Quantitative seismology: Theory and
methods, W.H. Freeman, New York 6

[2] Al Khoury, Ph., 2005, Algorithmes géométriques de résolution des moin-

dres carrés non linéaires et problèmes inverses en spectroscopie des
flammes, PhD Thesis, University of Paris 10, March 4 88, 135, 148

[3] Al Khoury, Ph., Chavent, G., 2006, Global line search strategies for
nonlinear least squares problems based on curvature and projected cur-
vature, Inverse Probl. Sci. Eng. 14(5), 495–509 135

[4] Alessandrini, G., 1986, An identification problem for an elliptic equation

in two variables, Ann. Math. Pura Appl. 145, 265–296 191

[5] Alessandrini, G., Magnanini, R., 1994, Elliptic equations in divergence

form, geometric critical points of solutions, and stekloff eigenfunctions,
SIAM J. Math. Anal. 25(5), 1259–1268 205

[6] Anterion, F., Eymard, R., Karcher, B., 1989, Use of parameter gradients
for reservoir history matching, In SPE Symposium on Reservoir Simu-
lation, Society of Petroleum Engineers, Houston, Texas, SPE 18433 78

[7] Banks, H.T., Kunisch, K., 1989, Estimation techniques for distributed
parameter systems, Birkhäuser, Boston 29

[8] Baumeister, J., 1987, Stable solutions of inverse problems, Vieweg,

Braunschweig 17, 210

[9] Ben-Ameur, H., Kaltenbacher, B., 2002, Regularization of parameter

estimation by adaptive discretization using refinement and coarsening
indicators. J. Inverse Ill Posed Probl. 10(6), 561–583 116

345
346 BIBLIOGRAPHY

[10] Ben-Ameur, H., Chavent, G., Jaffré, J., 2002, Refinement and coarsening
indicators for adaptive parametrization: Application to the estimation
of the hydraulic transmissivities. Inverse Probl. 18, 775–794 113, 116,
120, 123

[11] Ben-Ameur, H., Clément, F., Chavent G., Weis P., 2008, The multi-
dimensional refinements indicators algorithm for optimal parameteriza-
tion, J. Inverse Ill-Posed Probl. 16(2), 107–126 116, 122

[12] Bjork, A., 1990, Least squares methods, In Ciarlet, P.G., and Lions, J.L.,
eds, Handbook of Numerical Analysis, North-Holland, Amsterdam 17

[13] Bonnans, J.F., Gilbert, J.C., Lemaréchal, C., Sagastizbal, C.A., 2003,
Numerical optimization: Theoretical and practical aspects, Springer,
Universitext Series XIV, p 423 32, 127, 148

[14] Borzi, A., 2003, Multigrid methods for optimality systems, Habilita-
tion Thesis, Institut fr Mathematik, Karl-Franzens-Universitt Graz,
Austria 31

[15] Chardaire, C., Chavent G,. Jaffré J., Liu J., 1990, Multiscale representa-
tion for simultaneous estimation of relative permeabilities and capillary
pressure, Paper SPE 20501, In Proceedings of the 65th SPE Annual
Technical Conference and Exhibition, New Orleans, Louisiana, pp 303–
312 96

[16] Chavent, G., 1979, Identification of distributed parameter systems:

About the output least squares method, its implementation and iden-
tifiability, In Proceedings of the IFAC Symposium on Identification,
Pergamon, pp 85–97 12

[17] Chavent, G., 1986, Identifiability of parameters in the output least

square formulation, In Walter, E., ed, Structural Identifiability of
Parametrics Model, chapter 6, Pergamon Press, pp 67–74 12

[18] Chavent, G., 1990, A new sufficient condition for the wellposedness of
nonlinear least-squares problems arising in identification and control,
In Bensoussan, A., and Lions, J.L., eds, Lecture Notes in Control and
Information Sciences 144, Springer, Berlin, pp 452–463 211, 232
BIBLIOGRAPHY 347

[19] Chavent, G., 1991, New size×curvature conditions for strict quasi-
convexity of sets, SIAM J. Contr. Optim. 29(6), 1348–1372 12, 273

[20] Chavent, G., 1991, Quasi-convex sets and size×curvature condition,

application to nonlinear inversion, J. Appl. Math. Optim. 24(1), 129–169
12, 273

[21] Chavent, G., 2002, Adapted regularization for the estimation of the dif-
fusion coefficient in an elliptic equation, In Proceedings of Picof 02,
Carthage, Tunisie 259

[22] Chavent, G., 2004, Curvature steps and geodesic moves for nonlin-
ear least squares descent algorithms, Inverse Probl. Sci. Eng. 12(2),
173–191 135

[23] Chavent, G., Bissel, R., 1998, Indicator for the refinement of
parametrization. In Tanaka, M., and Dulikravich, G.S., eds, Inverse
Problems in Engineering Mechanics, Elsevier, Amsterdam, pp 309–314
113, 116, 120

[24] Chavent, G., Clement, F., 2001, Migration-based traveltime waveform

inversion of 2-D simple structures: A synthetic example, Geophysics
66(3), 845–860 97

[25] Chavent, G., Kunisch, K., 1993, A geometric theory for the inverse prob-
lem in a one-dimensional elliptic equation from an H 1 -observation, Appl.
Math. Optim. 27, 231–260 192, 201

[26] Chavent, G., Kunisch, K., 1993, Regularization in state space, M2AN
27, 535–564 18, 247, 258, 259

[27] Chavent, G., Kunisch, K., 1994, Convergence of Tikhonov regularization

for constrained ill-posed inverse problems, Inverse Probl. 10, 63–76 210,
211, 234

[28] Chavent, G., Kunisch, K., 1996, On weakly nonlinear inverse problems,
SIAM J. Appl. Math. 56(2), 542–572 16, 166, 211, 237, 273

[29] Chavent, G., Kunisch, K., 1998, State space regularization: Geometric
theory, Appl. Math. Opt. 37, 243–267 18, 247
348 BIBLIOGRAPHY

[30] Chavent, G., Kunisch, K., 2002, The output least square identifiability
of the diffusion coefficient from an H 1 observation in a 2-D elliptic equa-
tion, ESAIM: Contr. Optim. Calculus Variations 8, 423 97, 200, 202,
259, 261, 263

[31] Chavent, G., Lemonnier, P., 1974, Identification de la non linéarité d’une
équation parabolique quasilinéaire, J. Appl. Math. Optim. 1(2), 121–162
237

[32] Chavent, G., Jaffré, J., Jégou, S., Liu, J., 1997, A symbolic code gener-
ator for parameter estimation. In Berz, M., Bischof, C., Corliss, G., and
Griewank, A., eds, Computational Differentiation, SIAM, 129–136 38

[33] Chavent, G., Jaffré, J., Jan-Jégou, S., 1999, Estimation of relative per-
meabilities in three-phase flow in porous media, Inverse Probl. 15, 33–39
116

[34] Chicone, C., Gerlach, J., 1987, A note on the identifiability of distributed
parameters in elliptic systems, SIAM J. Math. Anal. 18(5), 1378–1384
185

[35] Cominelli, A., Ferdinandi, F., De Montleau, P., Rossi, R., 2005, Using
gradients to refine parameterization in field-case history match projects,
In Proceedings of 2005 SPE Reservoir Simulation Symposium, paper
SPE 93599, Houston, Texas, January 31st–February 2nd 116

[36] Delprat-Jannaud, F., Laiily, P., 1992, What information on the earth
model do reflection travel times provide?, J. Geophys. Res. 97(B13),
19827–19844 92

[37] Engl, H.W., Kunisch, K., Neubauer, A., 1989, Convergence rates for
Tikhonov regularization of nonlinear ill-posed problems, Inverse Probl.
5, 523–540 18, 211

[38] Engl, H.W., Hanke, M., Neubauer, A., 1996, Regularization of inverse
problems, Kluwer, Dordrecht, p 321, (Mathematics and its applications,
375) ISBN 0-7923-4157-0 17, 117

[39] Girault, V., Raviart, P.A., 1979, Finite element methods for Navier-
Stokes equations, Springer, Berlin 263
BIBLIOGRAPHY 349

[40] Griewank, A., 1992, Achieving logarithmic growth of temporal and

spatial complexity in reverse automatic differentiation, Optim. Meth.
Software 1, 35–54 37, 38, 77

[41] Griewank, A., 2000, Evaluating derivatives: Principles and tech-

niques of algorithmic differentiation (Frontiers in Applied Mathematics
19), Society for Industrial and Applied Mathematics, p 369, ISBN:
0898714516 38

[42] Grimstad, A.A., Mannseth T., Nævdal G., Urkedal H., 2003, Adaptive
multiscale permeability estimation, Comput. Geosci. 7, 1–25 111

[43] Groetsch, C.W., 1984, The theory of Tykhonov regularization for

Fredholm equations of the first kind, Research Notes in Mathematics
105, Pitman, Boston 17, 210

[44] Hayek, M., Lehmann, F., Ackerer, Ph., 2007, Adaptive multiscale
parameterization for one-dimensional flow in unsaturated porous media,
Adv. Water Resour. (to appear) 116, 270, 343

[45] Hein, T., 2009, Regularization in Banach space – convergence rates by

approximative source conditions, J. Inverse Ill-Posed Probl. 17, 27–41
211

[46] Isakov, V., 1998, Inverse problems for partial differential equations,
Springer, Berlin, p 284 (Applied mathematical sciences, 127) ISBN
0-387-98256-6 11, 185, 191

[47] Ito, K., Kunisch, K., 1994, On the injectivity and linearization of the
coefficient to solution mapping for elliptic boundary value problems,
J. Math. Anal. Appl. 188(3), 1040–1066 11, 185, 191

[48] Jaffard, S., Meyer, Y., Ryan, R.D., 2001, Wavelets (Tools for science
and technology), Society for Industrial and Applied Mathematics, p 256,
ISBN 0-89871-448-6 104

[49] Kunisch, K., 1988, Inherent identifiability of parameters in elliptic dif-

ferential equations, J. Math. Anal. Appl. 132, 453–472 191

[50] Lavaud, B., Kabir, N., Chavent, G., 1999, Pushing AVO inversion be-
yond linearized approximation, J. Seismic Explor. 8, 279–302 6, 88, 92
350 BIBLIOGRAPHY

[51] Le Dimet, F.-X., Charpentier I., 1998, Méthodes de second ordre en

assimilation de données, Equations aux Dérivées Partielles et Applica-
tions (Articles dédiés Jacques-Louis Lions), Gauthiers-Villars, pp 623–
640

[52] Le Dimet, F.-X., Shutyaev, V., 2001, On Newton method in data assim-
ilation, Russ. J. Numer. Anal. Math. Model. 15(5), 419–434 31

[53] Le Dimet, F.-X., Navon I.M., Daescu, D.N., 2002, Second order infor-
mation in data assimilation, Mon. Weather Rev. 130(3), 629–648 31

[54] Levenberg, K., 1944, A method for the solution of certain nonlinear
problems in least squares, Appl. Math. 11, 164–168 17, 209

[55] Lines, L.R., Treitel, S., Tutorial: A review of least-squares inversion and
its application to geophysical problems, Geophys. Prospect. 39, 159–181
17, 270, 343

[56] Lions, J.L., 1969, Quelques Méthodes de Résolution des Problèmes aux
limites Non Linéaires, Dunod, Paris 25, 238

[57] Liu, J., 1993, A multiresolution method for distributed parameter esti-
mation, SIAM J. Sci. Comput. 14, 389 96, 97, 104, 207

[58] Liu, J., 1994, A sensitivity analysis for least-squares ill-posed problems
using the haar basis, SIAM J. Numer. Anal. 31, 1486 96, 97, 104

[59] Louis, A.K., 1989, Inverse und Schlecht Gestellte Probleme, Teubner,
Stuttgart 17, 210

[60] Mannseth, T., 2003, Adaptive multiscale identification of the fluid

conductivity function within prous-media flow. Conference on Applied
Inverse Problems: Theoritical and computational aspects, Lake Arrow-
head, May 18–23 111

[61] Marchand, E., Clément, F., Roberts, J.E., Pépin, G., 2008, Deterministic
sensitivityanalysisfor a modelfor flow in porousmedia, Adv.Water Resour.
31, 1025–1037, http://dx.doi.org/10.1016/j.advwatres.2008.04.004 38

[62] Marquardt, D.W., 1963, An algorithm for least squares estimation of

nonlinear parameters, J. Soc. Ind. Appl. Math. 11, 431–441 17, 210
BIBLIOGRAPHY 351

[63] Morozov, V.A., 1984, Methods for solving incorrectly posed problems,
Springer, New York 17, 210
[64] Næval, T., Mannseth, T., Brusdal, K., Nordtvedt, J.E., 2000, Multiscale
estimation with spline wavelets, with application to two-phase porous
media flow, Inverse Probl. 16, 315–332 96
[65] Neubauer, A., 1987, Finite dimensional approximation of constrained
Tikhonov-regularized solutions of ill-posed linear operator equations,
Math. Comput. 48, 565–583 210, 215
[66] Neubauer, A., 1988, Tikhonov reularization of ill-posed linear operator
equations on closed convex sets, J. Approx. Theor. 53, 304–320 210
[67] Neubauer, A., 1989, Tikhonov regularization for nonlinear ill-posed
problems: Optimal convergence rate and finite dimensional approxima-
tion, Inverse Probl. 5, 541–558 18, 211, 215
[68] Nocedal, J., Wright, S.J., 1999, Numerical optimization, Springer Series
in Operation Research, New York 32, 127
[69] Økland Lien, M., 2005, Adaptive methods for permeability estimation
and smart well management, Dr. Scient. Thesis in Applied Mathematics,
Department of Mathematics, University of Bergen, Norway 96, 116
[70] Richter, G.R., 1981, An inverse problem for the steady state diffusion
equation, SIAM J. Math. 4, 210–221 11
[71] Sanchez Palencia, E., 1983, Homogenization method for the study of
composite media, Lecture Notes in Mathematics 985, 192–214, Springer,
Berlin 200
[72] Schaaf, T., Mezghani, M., Chavent, G., 2002, Direct conditioning of fine-
scale facies models to dynamic data by combining gradual deformation
and numerical upscaling techniques, In Proceedings of the 8th European
Conference on Mathematics of Oil Recovery (ECMOR VIII), Sept 3–6,
Freiberg, Germany
[73] Schaaf, T., Mezghani, M., Chavent, G., 2003, In Search of an opti-
mal parameterization: An innovative approach to reservoir data inte-
gration, paper SPE 84273, In Proceedings of the SPE Annual Technical
Conference and Exhibition, Denver
352 BIBLIOGRAPHY

[74] Sen, A., Srivastava, M., 1990, Regression analysis: Theory, methods and
applications, Springer, Berlin 85

[75] Tikhonov, A.N., Arsenin, V., 1977, Solutions of ill-posed problems,

Wiley, New York 17, 210

[76] Troianiello, G.M., 1987, Elliptic differential equations and obstacle prob-
lems, Plenum Press, New York 202

[77] van Laarhoven, P., Aarts, E., 1987, Simulated annealing, theory and
practice, Kluwer, Dordrecht 31

[78] Vogel, C., 2002, Computational methods for inverse problems, Frontiers
in Applied Mathematics series 23, SIAM 135, 270, 343

[79] Zhang, J., Dupuy, A., Bissel, R., 1996, Use of optimal control technique
for history matching, 2nd International Conference on Inverse Problems
in Engineering: Theory and Practice, June 9–14, Le Croisic, France,
Engineering Foundation ed. 78, 135

[80] Zhang, J., Jaffré, J., Chavent, G., 2009, Estimating nonlinearities in
multiphase flow in porous media, INRIA Report 6892 88, 185
Index

α-convex, 283 2D elliptic problem

k(z, η), 283 L2 observation, 248
k(z, p), 282 continuous adjoint, 59
1-D elliptic problem, 21 2D nonlinear source problem, 24, 237
H 1 observation, 191–200 continuous adjoint, 59
curvature estimate, 199
deflection estimate, 198 Finite Curvature, see FC
linear stability, 193 Finite Dimension, see FD
OLS-identifiability, 200 Gauss-Newton, see GN
L2 observation, 248 Levenberg-Marquardt-Tychonov, see
partial observation, 232 LMT
2-D elliptic problem, 26 Limited Deflection, see LD
H 1 observation, 200–207 Maximum Projected Curvature, see
adapted regularization, MPC
259–270 Nonlinear Least Squares, see NLS
deflection condition, 205 Output Least Squares, see OLS
linear identifiability, 203 Quadratically-, see Q-
regularization by discretization, quasi-convex, see q.c.
205 strictly quasi-convex, see s.q.c.
partial observation, 232 acceleration, 137, 325
continuous adjoint, 65 w.r.t. arc length, 338
gradient calculation, 46–59 adapted multiscale basis, 102–103
2-D nonlinear source problem adaptive parameterization, 88,
H 1 observation, 246 108–126
L2 observation adaptive time steps, 72
FC/LD properties, 237–242 adjoint approach, 33
LMT regularization, 242–246 implementation, 38
partial observation, 232 adjoint state equation, 36, 40
continuous adjoint, 65 continuous, 62, 66
gradient calculation, 46–59 discretized, 68

353
354 INDEX

examples, 41, 44, 45, 71, 76, 133 multiscale

for variational problems, 54–59 adapted, 102–103
arc length, 137, 277 example, 105–108
distance, 171, 280, 286 Haar basis, 107
reparameterization by, 325, 341 orthogonal versus oblique, 103
size
of a curve, 170 calibration, 80, 84
of a path, 322 data, 83
of a set, 322 parameters, 80
of an attainable set, 343 changing parameters, 35, 89, 126,
of an inverse problem, 170 129
upper bounds to, 327 from coarse to fine mesh, 130
Armijo condition, 136, 142 polar coordinates, 132
attainable set, 9 classical solution, 62
(arc length) size of, 343 coarse mesh, 94, 95, 101
modeling & measurement errors, coarsening indicators, 109, 122–126
163 code
arc length distance on, 164, 171 constraints, 127–129
closedness of, 187 gradient with respect to
curvature of, 343 optimization parameters,
deflection of, 343 129–134
family of paths on, 326 inversion, 126, 128
for a diffusion problem, 207 optimization, 89, 128
geodesics of, 138 regularizing functional, 129
geometric attributes of, 337–343 simulation, 88, 128
of a FC problem, 167 conditioning, 89, 90
of a FC/LD problem, 168, 220 cone
projection on, 172 affine normal, 304, 306
property of preimage, 173 negative polar, 212
of a nicely nonlinear problem, 98 tangent
auxiliary problem, 249, 254 for convex sets, 212
for non-convex sets, 235
back-tracking, 135, 145 continuity lemma, 291
background space, 102 convergence of solutions
basis functions for parameters for LMT regularization
adaptive, 108–126 FC/LD problems, 223
Lagrange, 49 general nonlinear problems, 234
local, 93, 100–101 linear problems, 216
INDEX 355

for state space regularization Curry, 137

geometric theory, 253 MPC, 135
soft analysis, 257 detail space, 102
curvature, see radius of curvature differential equations, 65
curve, 166, 337 discrete integration by part, 69,
cuts, 119, 125 76
discrete marching problems, 73
data adjoint gradient calculation, 75
calibrated, 83 sensitivity functions, 74
space, 9, 23, 25, 38, 84 sensitivity versus adjoint, 77
deflection, 322 discretized versus discrete, 65, 72
condition, 168, 324 distance
extended, 173, 190, 324, 335 arc length, 164, 171, 280,
sufficient condition for, 170 286
of a curve, 168, 341 gradient weighted, 260
of a path, 323
of a set, 323 Edelstein set, 275, 287
of an attainable set, 343 enlargement neighborhood, 163,
of an inverse problem, 168, 170 171
numerical determination, 188
family
upper bounds to, 170, 327, 328
of curves, 342
derivatives, 29
of paths, 276–279, 342
by adjoint method, 33–41
choice of, 279
by finite difference, 32
definition, 278
by sensitivity functions, 33
continuous, 64 geometric attributes
discrete, 30, 47, 65, 67, 68 of a FC problem, 170
discretized, 30, 65, 67, 68 of a set, 323
sensitivity versus adjoint, 77 of an attainable set, 342
w.r.t. nonlinearity, 58 Godstein condition, 136
descent algorithm, 135 grad-rot decomposition, 262
Gauss-Newton, 144 gradient equations, 36, 40
Levenberg–Marquardt, 144 continuous, 64, 66
quasi-Newton, 144 discretized, 68
steepest descent, 144 examples, 45, 58, 72, 76, 134
descent direction, 135 Green formula, 62
descent step guaranteed decrease property, 140,
acceptable, 135, 136 142
356 INDEX

Haar basis, 107 Jacobian transposed, 130

handling constraints, 126
heterogeneity, 93 Knott–Zoeppritz, 6, 92,
231
identifiability, 11 gradient calculation, 41
linearized, 174
Lagrange finite element, 48
for bilinear problems, 174 Lagrangian, 35, 39
for FC/LD problems, 175
continuous, 61, 66
local OLS-, 183
examples, 42, 55, 69, 75,
OLS-, 13, 162
133
for FC/LD problems, 176
for variational problems, 54
for FD problems, 180
linear case stationary
image segmentation, 121
arc length, 141
implementation issues, 80
residual, 141
interpolation linesearch, 136
data, 247, 257
Lipschitz stability, 13, 163,
parameters, 95, 108,
177
131
local minima, see stationary points
inverse problems, 9, 31, 161
localization constraint, 171,
bilinear, 174, 186, 188 250
FC (Finite Curvature), 13,
166 map
geometric attributes, 170 direct, see forward
FC/LD, 13, 165–174 forward, 9, 38,
Q-wellposed, 176 88
FD (Finite Dimension), 179 curvature, 96
Q-wellposed, 180 sensitivity, 96
LD (Limited Deflection), 168 input-output, see forward
sufficient conditions for, 170 maximum likelihood, 83
linear, 163, 211 median lemma, 286
nicely nonlinear, 3, 96–103, 109, minimum-norm solution
118, 207 x0 -, 17, 211, 220,
weakly nonlinear, see FC 235
state-space y0 -, 257
Jacobian, 33, 34, 85 MPC descent step, 135–149
computational cost, 33, 37, convergence properties, 143
46 definition, 138, 140
transposed, 34 implementation, 144
INDEX 357

performance, 148 parameter space, 9

properties, 142 parameterization
multiscale decomposition, 102 matrix, 95, 108
orthogonal versus oblique, 103 routine, 89, 128
adjoint, 89, 128
nicely nonlinear problems, 3, 96–103,
parameterization strategy
109, 118, 207
adaptive, 88, 108–126
NLS problems, 9, 161, 209
by closed form formula, 90
noise level, 80, 84, 87, 89, 92, 109, 163,
multiscale, 93–108
304
on a singular basis, 91
nominal value, 84, 91
parameters
nonlinearity, 24, 47, 93
calibrated, 80
gradient w.r.t., 58
discretized, 49, 88, 93
in the higher order, 237
optimization, 79, 89, 126
number of retrievable parameters,
on a singular basis, 92
87
adaptive, 109–126
observation operator, 18, 21, at scale k, 95, 100–104
31 closed form formula, 91
continuous, 47, 60, 66 retrievable, 79, 84–88
examples, 8, 23, 41, 53, 67, simulation, 79, 84, 88
75 distributed, 93
obtuse angle lemma, 290 vector valued, 116
use of, 226, 228, 255, 293 paths, 276, 277
optimizable, 12, 162 (arc) length of, 322
optimization arc length along, 277
adaptive, 124 curvature of
algorithm, 31, 128 global radius of, 305
MPC descent step, 135–149 deflection of, 168, 323
multiscale, 104 family of, 276–279, 342
scale by scale, 94–104 choice of, 279
optimization parameters, 89, definition, 278
126 generating family of, 280
adaptive, 109–126 penalization, 128
at scale k, 95, 100–104 piecewise constant parameters, 50,
closed form formula, 91 81, 93, 100
on a singular basis, 92 piecewise linear parameters, 50, 82,
overparameterization, 3, 80, 88, 89, 93, 101
108, 109, 117, 118 polar coordinates, 132
358 INDEX

projected curvature, 140 of a curve, 167, 341

projection of a path, 313
η-, 275 of a set, 313
on non-convex sets, 275 of an attainable set, 167, 343
on q.c. sets of an inverse problem, 167,
stability, 294 170
uniqueness & existence, 289 local versus global, 311–313
on s.q.c. sets, 172, 315 projected, 140
stability, 315 reference value, 81, 82
unimodality, 315 refinement indicators, 109–126
uniqueness & existence, 315 algorithm, 124
first order, 111
Q-wellposed, 13, 162, 182
as Lagrange multipliers, 112
available tools, 185–191
for image segmentation, 121
for finite dimensional parameters,
multiscale, 116–120
183
first order versus GN, 113
for infinite dimensional parame-
GN, 110
ters, 184
nonlinear, 110
q.c. sets, 275–297
refinement mesh, 124
definition, 283
regular neighborhood
quadrature formula, 49
q.c.sets, 283
qualified element
for an arc of circle, 285
FC/LD problems, 221
largest, 284
linear problems, 212
s.q.c. sets, 301
radius of curvature for an arc of circle, 303
global, 173, 299 for an arc of circle, largest, 310
characterization, 305 largest, 301
definition, 305 largest enlargement, 303, 308
formula for, 316–319 regularity condition
lower bound to, 333 for FC/LD problems, 222
numerical determination, 190, for linear problems, 214
319 interpretation, 245
of a path, 305 regularization, 14
of a set, 305 adapted, 14, 259–270
local (usual), 138, 167, 338 deflection condition, 267
lower bound to, 326 FD OLS-identifiability, 268
numerical determination, 191, linear stability, 265
319 missing information, 260
INDEX 359

by parameterization, 15, 79, simulation mesh, 48, 81, 93

89–126 simulation parameters, 79, 84, 88
by size reduction, 15, 171, 201 for a diffusion coefficient, 88
LMT, 17, 209–236 for a distributed parameter, 93
FC/LD problems, 210, 219–231 singular value
general nonlinear problems, analysis, 84–88
211, 231–236 decomposition, 85
linear problems, 210–219 size, see arc length size
state-space, 18, 246–259 size×curvature conditions, 171,
geometric approach, 247–256 322
soft analysis, 247, 256–259 stability, 11, 177
residual, 137 linearized, 176, 177
reverse mode, 38 state equation, 18, 21, 31, 36, 39
s.q.c. functions, 300 continuous, 47, 60, 65
s.q.c. sets, 165, 168, 299–320 examples, 7, 23, 42, 53, 67, 73
are q.c. sets, 313 state-space
characterization, 307 continuous, 60
definition, 301 examples, 41, 53, 54, 67, 75
projection on, 315 state-space decomposition, 20, 31
scalar product 2D elliptic eq., continuous, 60
in the Lagrangian, 39, 55, 69 2D elliptic eq., discrete, 53
on parameter space, 81–83, 94, bilinear equations, 174, 186,
109, 117 188
scale (coarse, fine), 94 Knott-Zoeppritz, 41
scale by scale optimization, 94–104 boundary conditions, 53, 55, 58
search curve for state-space regularization, 246
geodesic, 136, 138 in the adjoint approach, 39
straight, 136 polar coordinates, 133
straight versus geodesic, 142 stationary points in . . .
sensitivity analysis, 84–88 inverse problems, 32
sensitivity functions, 33 definition, 162
examples, 74 optimizability, 162
sensitivity matrix, 33 adaptive parameterization, 109
sets scale by scale optimization, 98
(arc length) size of, 322 projection on q.c. sets, 297
curvature of, 167, 313, 343 projection on s.q.c. sets, 301,
deflection of, 323 302
global radius of curvature of, 305 sub-mesh, 94, 117
360 INDEX

tentative degrees of freedom, 110 weak solution, 62

total variation, 327 weakly nonlinear problem, 166
triangulation, 48 wellposed, 10, 162
Q-, 13, 162
uncertainty
FC/LD problems, 176–178
on data, 83, 85
FD, 179
on parameter
Wolfe condition, 136
absolute, 85
worst case stationary
relative, 86
arc length, 141
uncertainty analysis, 85
residual, 141
unimodal, 12, 13, 162,
164 zonation, 117
variational formulation, 48, “optimal”, 117
237 aggregating zones, 123
finite dimensional, 51 made of single connected compo-
velocity, 137, 325 nents, 126
w.r.t. arc length, 338 splitting zones, 118