Get the full ebook with Bonus Features for a Better Reading Experience on ebookmeta.

com

Sparse Estimation with Math and Python: 100

Exercises for Building Logic 1st Edition Joe
Suzuki

https://ebookmeta.com/product/sparse-estimation-with-math-
and-python-100-exercises-for-building-logic-1st-edition-joe-
suzuki-2/

OR CLICK HERE

DOWLOAD NOW

Download more ebook instantly today at https://ebookmeta.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Sparse Estimation with Math and Python: 100 Exercises for

Building Logic 1st Edition Joe Suzuki

https://ebookmeta.com/product/sparse-estimation-with-math-and-
python-100-exercises-for-building-logic-1st-edition-joe-suzuki-2/

ebookmeta.com

Statistical Learning with Math and Python: 100 Exercises

for Building Logic Joe Suzuki

https://ebookmeta.com/product/statistical-learning-with-math-and-
python-100-exercises-for-building-logic-joe-suzuki/

ebookmeta.com

Kernel Methods for Machine Learning with Math and Python:

100 Exercises for Building Logic Joe Suzuki

https://ebookmeta.com/product/kernel-methods-for-machine-learning-
with-math-and-python-100-exercises-for-building-logic-joe-suzuki/

ebookmeta.com

The Rough Guide to Portugal Travel Guide eBook Stuart

Butler

https://ebookmeta.com/product/the-rough-guide-to-portugal-travel-
guide-ebook-stuart-butler/

ebookmeta.com
A Toolbox for the Application of the Rules of Targeting
1st Edition Tetyana (Tanya) Krupiy

https://ebookmeta.com/product/a-toolbox-for-the-application-of-the-
rules-of-targeting-1st-edition-tetyana-tanya-krupiy/

ebookmeta.com

A Sampsons Quarry Mystery Box Set Books 1 3 Sophie Tucker

https://ebookmeta.com/product/a-sampsons-quarry-mystery-box-set-
books-1-3-sophie-tucker/

ebookmeta.com

Urban China 1st Edition Xuefei Ren

https://ebookmeta.com/product/urban-china-1st-edition-xuefei-ren/

ebookmeta.com

Towards a Sustainable Economy The Case of Bangladesh 1st

Edition Moazzem Hossain

https://ebookmeta.com/product/towards-a-sustainable-economy-the-case-
of-bangladesh-1st-edition-moazzem-hossain/

ebookmeta.com

Constituent Power and the Legitimacy of International

Organizations The Constitution of Supranationalism 1st
Edition John G. Oates
https://ebookmeta.com/product/constituent-power-and-the-legitimacy-of-
international-organizations-the-constitution-of-supranationalism-1st-
edition-john-g-oates/
ebookmeta.com
Decision Makers Guide to Road Tolling in CAREC Countries
1st Edition Asian Development Bank

https://ebookmeta.com/product/decision-makers-guide-to-road-tolling-
in-carec-countries-1st-edition-asian-development-bank/

ebookmeta.com
Joe Suzuki

Sparse
Estimation
with Math
and Python
100 Exercises for Building Logic
Sparse Estimation with Math and Python
Joe Suzuki

Sparse Estimation with Math

and Python
100 Exercises for Building Logic
Joe Suzuki
Graduate School of Engineering Science
Osaka University
Toyonaka, Osaka, Japan

ISBN 978-981-16-1437-8 ISBN 978-981-16-1438-5 (eBook)

https://doi.org/10.1007/978-981-16-1438-5

© The Editor(s) (if applicable) and The Author(s), under exclusive license to Springer Nature
Singapore Pte Ltd. 2021
This work is subject to copyright. All rights are solely and exclusively licensed by the Publisher, whether
the whole or part of the material is concerned, specifically the rights of reprinting, reuse of illustrations,
recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or
information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar
methodology now known or hereafter developed.
The use of general descriptive names, registered names, trademarks, service marks, etc. in this publication
does not imply, even in the absence of a specific statement, that such names are exempt from the relevant
protective laws and regulations and therefore free for general use.
The publisher, the authors and the editors are safe to assume that the advice and information in this book
are believed to be true and accurate at the date of publication. Neither the publisher nor the authors or
the editors give a warranty, expressed or implied, with respect to the material contained herein or for any
errors or omissions that may have been made. The publisher remains neutral with regard to jurisdictional
claims in published maps and institutional affiliations.

This Springer imprint is published by the registered company Springer Nature Singapore Pte Ltd.
The registered company address is: 152 Beach Road, #21-01/04 Gateway East, Singapore 189721,
Singapore
Preface

I started considering the sparse estimation problems around 2017 when I moved
from the mathematics department to statistics in Osaka University, Japan. I have
been studying information theory and graphical models for over thirty years.
The first book I found is “Statistical Learning with Sparsity” by T. Hastie, R.
Tibshirani, and M. Wainwright. I thought it was a monograph rather than a textbook
and that it would be tough for a non-expert to read it through. However, I downloaded
more than fifty papers that were cited in the book and read them all. In fact, the book
does not instruct anything but only suggests how to study sparsity. The contrast
between statistics and convex optimization gradually attracted me as I understand
the material.
On the other hand, it seems that the core results on sparsity have come out around
2010–2015 for research. However, I still think further possibilities and expansions
are there. This book contains all the mathematical derivations and source programs,
so graduate students can construct any procedure from scratch by getting help from
this book.
Recently, I published books “Statistical Learning with Math and R” (SLMR),
“Statistical Learning with Math and Python” (SLMP), and “Sparse Estimation with
Math and R” (SEMR). The common idea is behind the books (XXMR/XXMP). They
not only give knowledge on statistical learning and sparse estimation but also help
build logic in your brain by following each step of the derivations and each line of
the source programs. I often meet data scientists engaged in machine learning and
statistical analyses for research collaborations and introduce my students to them.
I recently found out that almost all of them think that (mathematical) logic rather
than knowledge and experience is the most crucial ability for grasping the essence
in their jobs. Our necessary knowledge is changing every day and can be obtained
when needed. However, logic allows us to examine whether each item on the Internet
is correct and follow any changes; we might miss even chances without it.

v
vi Preface

What makes SEMP unique?

I have summarized the features of this book as follows.

1. Developing logic
To grasp the essence of the subject, we mathematically formulate and solve each
ML problem and build those programs. The SEMP instills “logic” in the minds
of the readers. The reader will acquire both the knowledge and ideas of ML, so
that even if new technology emerges, they will be able to follow the changes
smoothly. After solving the 100 problems, most of the students would say “I
learned a lot”.
2. Not just a story
If programming codes are available, you can immediately take action. It is
unfortunate when an ML book does not offer the source codes. Even if a package
is available, if we cannot see the inner workings of the programs, all we can do
is input data into those programs. In SEMP, the program codes are available for
most of the procedures. In cases where the reader does not understand the math,
the codes will help them understand what it means.
3. Not just a how to book: an academic book written by a university professor.
This book explains how to use the package and provides examples of executions
for those who are not familiar with them. Still, because only the inputs and
outputs are visible, we can only see the procedure as a black box. In this sense,
the reader will have limited satisfaction because they will not be able to obtain
the essence of the subject. SEMP intends to show the reader the heart of ML
and is more of a full-fledged academic book.
4. Solve 100 exercises: problems are improved with feedback from university
students
The exercises in this book have been used in university lectures and have been
refined based on feedback from students. The best 100 problems were selected.
Each chapter (except the exercises) explains the solutions, and you can solve
all of the exercises by reading the book.
5. Self-contained
All of us have been discouraged by phrases such as “for the details, please refer
to the literature XX.” Unless you are an enthusiastic reader or researcher, nobody
will seek out those references. In this book, we have presented the material in
such a way that consulting external references is not required. Additionally, the
proofs are simple derivations, and the complicated proofs are given in the appen-
dices at the end of each chapter. SEMP completes all discussions, including the
appendices.
Preface vii

6. Readers’ pages: questions, discussion, and program files

The reader can ask any question on the book via https://bayesnet.org/books.

Osaka, Japan Joe Suzuki

September 2021
Acknowledgments The author wishes to thank Tianle Yang, Ryosuke Shinmura, Tomohiro Kamei
and Daichi Kashiwara for checking the manuscript in Japanese. The author thanks Professors Shu-
ichi Kawano, Hidetoshi Matsui, and Kei Hirose for their helpful advices for three years before the
current book has been published. This English book is largely based on the Japanese book published
by Kyoritsu Shuppan Co., Ltd. in 2020. The author would like to thank Kyoritsu Shuppan Co., Ltd.,
in particular its editorial members Mr. Tetsuya Ishii and Ms. Saki Otani. The author also appreciates
Ms. Mio Sugino, Springer, for preparing the publication and providing advice on the manuscript.
Contents

1 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.2 Subderivative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3 Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.4 Ridge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
1.5 A Comparison Between Lasso and Ridge . . . . . . . . . . . . . . . . . . . . . . 17
1.6 Elastic Net . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.7 About How to Set the Value of λ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
Exercises 1–20 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2 Generalized Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
2.1 Generalization of Lasso in Linear Regression . . . . . . . . . . . . . . . . . . . 39
2.2 Logistic Regression for Binary Values . . . . . . . . . . . . . . . . . . . . . . . . . 41
2.3 Logistic Regression for Multiple Values . . . . . . . . . . . . . . . . . . . . . . . 50
2.4 Poisson Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.5 Survival Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
Appendix Proof of Proposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
Exercises 21–33 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3 Group Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.1 When One Group Exists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
3.2 Proxy Gradient Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
3.3 Group Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.4 Sparse Group Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
3.5 Overlap Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.6 Group Lasso with Multiple Responses . . . . . . . . . . . . . . . . . . . . . . . . . 95
3.7 Group Lasso Via Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . 98
3.8 Group Lasso for the Generalized Additive Models . . . . . . . . . . . . . . . 102
Appendix Proof of Proposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
Exercises 34–46 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106

ix
x Contents

4 Fused Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

4.1 Applications of Fused Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
4.2 Solving Fused Lasso Via Dynamic Programming . . . . . . . . . . . . . . . 120
4.3 LARS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
4.4 Dual Lasso Problem and Generalized Lasso . . . . . . . . . . . . . . . . . . . . 126
4.5 ADMM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
Appendix Proof of Proposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
Exercises 47–61 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5 Graphical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.1 Graphical Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
5.2 Graphical Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
5.3 Estimation of the Graphical Model Based
on the Quasi-Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
5.4 Joint Graphical Lasso . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
Appendix Proof of Propositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
Exercises 62–75 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
6 Matrix Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
6.1 Singular Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
6.2 Eckart-Young’s Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6.3 Norm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
6.4 Sparse Estimation for Low-Rank Estimations . . . . . . . . . . . . . . . . . . . 202
Appendix Proof of Propositions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
Exercises 76–87 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
7 Multivariate Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
7.1 Principal Component Analysis (1): SCoTLASS . . . . . . . . . . . . . . . . . 213
7.2 Principle Component Analysis (2): SPCA . . . . . . . . . . . . . . . . . . . . . . 218
7.3 K -Means Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
7.4 Convex Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Appendix Proof of Proposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
Exercises 88–100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
Chapter 1
Linear Regression

In general statistics, we often assume that the number of samples N is greater than
the number of variables p. If this is not the case, it may not be possible to solve
for the best-fitting regression coefficients using the least squares method, or it is
too computationally costly to compare a total of 2 p models using some information
criterion.
When p is greater than N (also known as the sparse situation), even for linear
regression, it is more common to minimize, instead of the usual squared error, the
modified objective function to which a term is added to prevent the coefficients from
being too large (the so-called regularization term). If the regularization term is a
constant λ times the L1-norm (resp. L2-norm) of the coefficient vector, it is called
Lasso (resp. Ridge). In the case of Lasso, if the value of λ increases, there will be
more coefficients that go to 0, and when λ reaches a certain value, all the coefficients
will eventually become 0. In that sense, we can say that Lasso also plays a role in
model selection.
In this chapter, we examine the properties of Lasso in comparison to those of
Ridge. After that, we investigate the elastic net, a regularized regression method that
combines the advantages of both Ridge and Lasso. Finally, we consider how to select
an appropriate value of λ.

1.1 Linear Regression

Throughout this chapter, let N ≥ 1 and p ≥ 1 be integers, and let the (i, j) element
of the matrix X ∈ R N × p and the kth element of the vector y ∈ R N be denoted by
xi, j and yk , respectively. Using these X, y, we find the intercept β0 ∈ R and the
slope β = [β1 , . . . , β p ]T that minimize
y − β0 − X β . Here, the L2-norm of z =
2

N
[z 1 , . . . , z N ]T is denoted by z := 2
i=1 z i .

© The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd. 2021 1
J. Suzuki, Sparse Estimation with Math and Python,
https://doi.org/10.1007/978-981-16-1438-5_1
2 1 Linear Regression

First, for the sake of simplicity, we assume that the jth column of X , ( j =
1, . . . , p)and y have already been centered. That is, for each j = 1, . . . , p, define
N
x̄ j := N1 i=1 xi, j , and assume that x̄ j has already been subtracted from each xi, j so
N
that x̄ j = 0 is satisfied. Similarly, defining ȳ := N1 i=1 yi , we assume that ȳ was
subtracted in advance from each yi so that ȳ = 0 holds. Under this condition, one of
the parameters (β̂0 , β̂) for which we need to solve, say, β̂0 , is always 0. In particular,

∂   
N p p
0= (yi − β0 − xi, j β j )2 = −2N ( ȳ − β0 − x̄ j β j ) = 2N β0
∂β0 i=1 j=1 j=1

holds. Thus, from now, without loss of generality, we may assume that the intercept
β0 is zero and use this in our further calculations.
We begin by first observing the following equality:
⎡ ⎤ ⎡ ⎤
∂   
N p p

⎢ (yi − βk xi,k ) ⎥
2
⎡ ⎤ ⎢ y1 − βk x1,k ⎥
⎢ ∂β1 i=1 ⎥ x1,1 · · · x N ,1 ⎢ ⎥
⎢ k=1 ⎥ ⎢ k=1 ⎥
⎢ .. ⎥ ⎢ .. . . .
. ⎥ ⎢ .
. ⎥
⎢ . ⎥ = −2 ⎣ . . . ⎦ ⎢ . ⎥
⎢ ⎥ ⎢ ⎥
⎢ ⎥ x1, p · · · x N , p ⎣⎢  ⎥
⎣ ∂  
N p p
2⎦ yN − βk x N ,k
⎦
(yi − βk xi,k )
∂β p i=1 k=1 k=1

= −2X T (y − X β). (1.1)

In particular, the jth element of each side can be rewritten as follows:


N 
p
−2 xi, j (yi − βk xi,k ).
i=1 k=1

Thus, when we set the right-hand side of (1.1) equal to 0, and if X T X is invertible,
then β becomes
β̂ = (X T X )−1 X T y. (1.2)

For the case where p = 1, write each column of X as x1 , . . . , x N ; we see that

N
xi yi
β̂ = i=1
N
. (1.3)
i=1 xi2

If we had not performed the data centering, we would still obtain the same slope
β̂, though the intercept β̂0 would be


p
β̂0 = ȳ − x̄ j β̂ j . (1.4)
j=1
 1.1 Linear Regression 3

Here, x̄ j ( j = 1, . . . , p) and ȳ are the means before data centering.

We can implement the above using the Python as follows:

1 def linear (X , y ):
2 p = X . shape [1]
3 x_bar = np . zeros ( p )
4 for j in range ( p ):
5 x_bar [ j ] = np . mean ( X [: , j ])
6 for j in range ( p ):
7 X [: , j ] = X [: , j ] - x_bar [ j ] # Centralize X
8 y_bar = np . mean ( y )
9 y = y - y_bar # Centralize y
10 beta = np . dot (
11 np . linalg . inv ( np . dot ( X .T , X )) ,
12 np . dot ( X .T , y )
13 )
14 beta_0 = y_bar - np . dot ( x_bar , beta )
15 return beta , beta_0

In this book, we focus more on the sparse case, i.e., when p is larger than N . In
this case, a problem arises. When N < p, the matrix X T X does not have an inverse.
In fact, since
rank(X T X ) ≤ rank(X ) ≤ min{N , p} = N < p,

X T X is singular. Moreover, when X has precisely the same two columns, rank(X ) <
p, and the inverse matrix does not exist.
On the other hand, if p is rather large, we have p independent variables to choose
from for the predictors of the variable when carrying out model selection. Thus, the
combinations are
{}, {1}, {2}, . . . , {1, . . . , p}

(that is, whether we choose each of the variables or not). This means we have to
compare a total of 2 p models. Then, to extract the proper model combination by using
an information criterion or cross-validation, the computational resources required
will grow exponentially with the number of variables p.
To deal with this kind of problem, let us consider the following. Let λ ≥ 0 be a
constant. We add a term to y − X β2 that penalizes β for being too large in size.
Specifically, we define

1
L := y − X β2 + λβ1 (1.5)
2N
or
1
L := y − X β2 + λβ22 . (1.6)
N

Our work now is to solve for the minimizer β

p
∈ R p to one of the above quanti-
ties. Here, for β = [β1 , . . . , β p ] , β1 := j=1 |β j | is the L1-norm and β2 :=
T

p
j=1 β j is the L2-norm of β. To be more specific, we plug mean-centered
2

X ∈ R N × p and y ∈ R N into either (1.5) or (1.6), then minimize it with respect to the
4 1 Linear Regression

slope β̂ ∈ R p (this is called Lasso or Ridge, respectively), and finally, we use (1.4)
to compute β̂0 .

1.2 Subderivative

To address the minimization problem for Lasso, we need a method for optimizing
functions that are not differentiable. When we want to find the points x of the max-
ima or minima of a single-variable polynomial, say, f (x) = x 3 − 2x + 1, we can
differentiate it and find the solution to f  (x) = 0. However, what should we do when
we encounter functions such as f (x) = x 2 + x + 2|x|, which contain an absolute
value? To address this, we need to extend our concept of differentiation to a more
general one.
Throughout the following claims, let us assume that f is convex [4, 6]. In general,
we say that f is convex (downward)1 if, for any 0 < α < 1 and x, y ∈ R,

f (αx + (1 − α)y) ≤ α f (x) + (1 − α) f (y)

holds. For instance, f (x) = |x| is convex (Fig. 1.1, left) because

|αx + (1 − α)y| ≤ α|x| + (1 − α)|y|

is satisfied. To check this, since both sides are nonnegative, the RHS squared minus
the LHS squared gives 2α(1 − α)(|x y| − x y) ≥ 0. As another example, consider

1, x = 0
f (x) = (1.7)
0, x = 0

This function satisfies the following:

f (α · 0 + (1 − α) · 1) = 1 > 1 − α = α f (0) + (1 − α) f (1).

Therefore, it is not convex (Fig. 1.1, right). If functions f, g are convex, then for any
β, γ ≥ 0, the function β f (x) + γg(x) has to be convex since the following holds:

β{ f (αx + (1 − α)y)} + γ{g(αx + (1 − α)y)}

≤ αβ f (x) + (1 − α)β f (y) + αγg(x) + (1 − α)γg(y)
= α{β f (x) + γg(x)} + (1 − α){β f (y) + γg(y)}.

Next, for any convex function f : R → R, fix x0 ∈ R arbitrarily. For all x ∈ R,

we say that the set of all z ∈ R that satisfies

1 Throughout this book, when we refer to convex, we mean convex downward.

1.2 Subderivative 5


1, x = 0
f (x) = |x| f (x) =
0, x=0
1
 
f (x) = −1 f (x) = 1 Not Convex

0 x
Convex

Fig. 1.1 Left: f (x) = |x| is convex. However, at the origin, the derivatives from each side differ;
thus, it is not differentiable. Right: We cannot simply judge from its shape, but this function is not
convex

f (x) ≥ f (x0 ) + z(x − x0 ) (1.8)

is a subderivative of f at x0 .
If f is differentiable at x0 , then the subderivative will be a set that contains only
1 element, say, f  (x0 ).2 We prove this as follows.
First, the convex function f is differentiable at x0 ; thus, it satisfies f (x) ≥
f (x0 ) + f  (x0 )(x − x0 ). To see this, since f is convex,

f (αx + (1 − α)x0 ) ≤ α f (x) + (1 − α) f (x0 ).

This can be rewritten as

f (x0 + α(x − x0 )) − f (x0 )
f (x) ≥ f (x0 ) + (x − x0 ).
α(x − x0 )

In fact, whether x < x0 or x > x0 , we have that

f (x0 + α(x − x0 )) − f (x0 )

lim = f  (x0 ).
α 0 α(x − x0 )

holds. Thus, the above inequation is true.

Next, when the convex function f is differentiable at x0 , we can show that f  (x0 )
is the one and only value of z that satisfies (1.8). In particular, when x > x0 , for (1.8)
f (x) − f (x0 )
to be satisfied, we need ≥ z. Similarly, when x < x0 , for (1.8) to be
x − x0
f (x) − f (x0 )
satisfied, we need ≤ z. Thus, z needs to be larger than or equal to the
x − x0
derivative on the left and, at the same time, be less than or equal to the derivative
on the right at x0 . Since f is differentiable at x0 , those 2 derivatives are equal; this
completes the proof.

2 In this case, we would not write it as a set { f  (x0 )} but as an element f  (x0 ).
 6 1 Linear Regression

The main interest of this book is specifically the case where f (x) = |x| and
x0 = 0. Hence, by (1.8), its subderivative is the set of z such that for any x ∈ R,
|x| ≥ zx. These values of z lie in the interval greater than or equal to −1 and less
than or equal to 1, and

for any arbitrary x, |x| ≥ zx ⇐⇒ |z| ≤ 1

is true. Let us confirm this. If for any x, |x| ≥ zx holds, then for x > 0 and x < 0,
z ≤ 1 and z ≥ −1, respectively, need to be true. Conversely, if −1 ≤ z ≤ 1, then
zx ≤ |zx| ≤ |x| is true for any arbitrary x ∈ R.

Example 1 By dividing into 3 cases x < 0, x = 0, and x > 0, find the values x that
attain the minimum of x 2 − 3x + |x| and x 2 + x + 2|x|. Note that for x = 0, we
can find their usual derivatives, but for f (x) = |x| at x = 0, its subderivative is the
interval [−1, 1].

x 2 − 3x + x, x ≥ 0 x 2 − 2x, x ≥ 0
x 2 − 3x + |x| = =
x 2 − 3x − x, x < 0 x 2 − 4x, x < 0
⎧
⎨ 2x − 2, x >0
(x 2 − 3x + |x|) = 2x − 3 + [−1, 1] = −3 + [−1, 1] = [−4, −2]  0, x = 0
⎩
2x − 4 < 0, x < 0.

Therefore, x 2 − 3x + |x| has a minimum at x = 1 (Fig. 1.2, left).

x 2 + x + 2x, x ≥ 0 x 2 + 3x, x ≥ 0
x 2 + x + 2|x| = =
x 2 + x − 2x, x < 0 x 2 − x, x < 0
⎧
⎨ 2x + 3 > 0, x >0
(x 2 + x + 2|x|) = 2x + 1 + 2[−1, 1] = 1 + 2[−1, 1] = [−1, 3] 0, x = 0
⎩
2x − 1 < 0, x < 0.

Therefore, x 2 + x + 2|x| has a minimum at x = 0 (Fig. 1.2, right). We use the fol-
lowing code to draw the figures.

1 x = np . arange ( -2 , 2 , 0.1)
2 y = x **2 - 3 * x + np . abs ( x )
3 plt . plot (x , y )
4 plt . scatter (1 , -1 , c = " red " )
5 plt . title ( " $y  =  x 2  - 3 x  +  | x | $ " )
6
7 x = np . arange ( -2 , 2 , 0.1)
8 y = x **2 + x + 2 * np . abs ( x )
9 plt . plot (x , y )
10 plt . scatter (0 , 0 , c = " red " )
11 plt . title ( " $y  =  x 2  +  x  +  2| x | $ " )

The subderivative of f (x) = |x| at x = 0 is the interval [−1,1]. This fact sum-
marizes this chapter.
Another Random Scribd Document
with Unrelated Content
“It was Ed Stimson that come to me—that’s it. Ed bought the Eagle
House from old Bill Williams’s widder, and Ed and me was pretty
close pardners in them days. ‘Ebner,’ says he, ‘Doc Rand claims
number nine’s got the pneumonia. She’s been out of her head since
daylight. She’s been askin’ for you. Guess you’re elected, Eb.’”
He rambled on, unconscious that every word he uttered was far from
welcome to his listener, who sat before him helpless, dazed, and
indignant, unable to stem the tide of his worldly narrative. He
enlightened her to the fact that he and Little Nell had had supper
together only two days before in an oyster-parlor of a friend of his.
He insisted that she had taken a shine to him from the first, and that
now that she was ill and penniless in the Eagle House, the only
decent thing he could do was to pay the doctor and her board bill,
dilating on the detail that he was human and incapable of seeing any
woman in distress, without coming to her aid like a gentleman, and
ended this remarkable résumé by flinging himself back on the sofa
with a satisfied smile, stretching his lean jaws in a yawn, as if the
incident was only one of many in his wide experience.
“Warm, ain’t it—for March?” he declared, breaking the awkward
silence that ensued.
Miss Ann agreed that it was, the needles slowing down to their
normal speed.
“It ain’t a mite too warm for me,” he remarked, displaying a thick and
drooping sock above his cracked patent-leather shoes. “Warm
weather means plenty of business in the laundry line, Miss Moulton.
A feller can get along all right in cold weather, but take it in collar-
meltin’ time and clean shirts are a necessity. Ever stop to think how
many percales and fancy madrases are spoiled by cheap wringers?
Chewed to holes ’fore the iron touches ’em.”
Miss Ann laid her knitting in her lap in forced attention. Something far
graver than his visit had worried her to-day, a question of money, a
discouraging letter from her brother, which she had kept from her
sister, not having the heart to tell her that some property she had
counted on to relieve their present modest income had turned out a
failure.
“I don’t mind tellin’ you, Miss Moulton, a little secret,” continued Ford,
“seein’ we’re old friends and neighbors. It’s sort of lettin’ the cat out
of the bag,” he added thoughtfully, “but I’ve been thinkin’ it over,
neighbor; besides, I don’t know anybody I’d rather help than you,” he
declared, as he fished in his pocket and drew out a square chunk of
dark rubber.
“That’s pure Para,” he announced gravely, holding it up for her
inspection. “Take a good look at it, Miss Moulton; you don’t often see
it. It ain’t worth its weight in gold, but it’s close to it when it comes to
wringers. It’s them cheap rollers that does the dirty work. If you was
to know what they’re made of, I presume likely you wouldn’t care to
wear the clothes they come through. It’s the sulphur in ’em that does
the stainin’.”
Again his long hand fumbled in his pocket; this time it drew out a
folded paper with a mechanical drawing, a model of a clothes-
wringer, which he spread out flat on his knees.
“There she is,” he declared with conviction. “Looks pretty neat, don’t
it? That there layer of pure Para on the rollers does the trick, and
them two extra cog-wheels on the speed-accelerator keeps her
movin’, I kin tell you. Saves time! One turn of that crank’s worth ten
of any other household wringer on the market. Can’t jam, can’t
squeeze, can’t rust, every nut, screw, and rivet in it galvanized. Even
pressure on anything from a lady’s handkerchief to a baby’s bib. Got
any idea, friend, what it costs delivered to sufferin’ humanity? Four
dollars. Got any idea what it makes?”
“I haven’t an idea,” confessed Miss Ann, looking up, relieved at the
sudden and cleanly change in the conversation, and, despite herself,
becoming more and more interested.
“’Course you haven’t, Miss Moulton. Be a little surprised, wouldn’t
you, if I was to tell you that old Mrs. Miggs, one of our stockholders,
doubled her income; that she’s got already a couple of thousand
dollars laid aside for a rainy day that she’d never had if I hadn’t come
to her in a friendly way. I don’t know as if I’ve ever seen a woman
happier. Her mortgage on her house in Yonkers all paid up, nice little
new home for herself and niece, and a tidy little sum in the bank—a
sum that’s growin’ daily, friend, without so much as liftin’ her little
finger. As our head canvasser on the road wrote me yesterday, a
man of over twenty years’ experience sellin’ wringers—‘You needn’t
worry no more,’ he writes, ‘about the Household Gem holdin’ her
own; I’m averagin’ two gross a week right here in Elmira. I could sell
three if I had ’em.’ Hold on. I’ve got it, if I ain’t mistaken.” He whipped
out the letter and read it aloud, including its postscript.
“You should see the pleased faces on Mondays—women
who have never had an easy wash-day before in their
lives. The new ad: ‘Let baby do the work,’ catches ’em.
Hoping your folks are well,
“Yours successfully,
“E. P. Redmond,
“Managing Salesman of The United Family
Laundry Association, Limited.”
He thrust the letter back in his pocket and waited for its effect,
beating a tattoo on the arm of the sofa, and though Miss Ann did not
reply, the nervous way she dropped her stitches assured him he had
made an impression.
“Anybody, my friend, with a little ready money, can double it,” he
resumed persuasively. “Just as sure as two and two makes four.
Take Mrs. Miggs, for instance. Six months ago she was skimpin’
along as usual—always ailin’, too—worry done that, as I told her,
worry; not knowin’ how she was goin’ to end one month and begin
another. Lookin’ sallower’n a peck of mustard—no appetite—worry—
and what for? Kept what little money she had in her bank, afraid to
invest a dollar of it in anything. Let it lay there in cold storage without
givin’ her a cent of interest. Spendin’ little by little her capital without
a dollar of it free to make another. ’Twa’n’t right, and I told her so
plainly. It’s all she had, she told me. It’ll be all you’ll ever get, I told
her, if you keep on leaving it in jail. Any dollar, my dear friend, that
ain’t worth more than a dollar, that can’t make a cent for itself, is a
pretty shiftless greenback, and ought to be ashamed to look its
owner in the face. Give every dollar a show. That’s common sense,
ain’t it?”
He shot out a frayed cuff and slapped his knee soundly.
“I ain’t the kind to believe in speculatin’, ’specially for women. They
wa’n’t never made to handle the heavy risks that men are. They ain’t
capable of shoulderin’ the enormous responsibilities that we have to.
How many women have come to me, beggin’ me to invest their
money in speculations that I’ve refused. Funny, ain’t it, how some
women like to gamble? That’s all speculatin’ is—gamblin’. Gamblin’s
agin my principles, friend, and always was. There ain’t no
righteousness in gamblin’. It’s an ungodly sin, worse vice’n the liquor
habit. Our gains, says the Bible, is to be measured by the sweat of
our brows. Honest business means hard toil and sound judgment.
Why, I’ve seen times when if it hadn’t been for my sound judgment—
business acumen, they call it—I’d been a ruined man. Sellin’ honest
goods ain’t got nothin’ to do with gamblin’. Sellin’ somethin’ that folks
need—honestly made and honestly sold; that folks who have paid for
it and used it swear by. An article that enters the home circle as a
helpin’ hand; that makes the home happier, and keeps the doctor
from the door. No more backaches for mother; a child can turn the
handle of the Gem. The accelerator tends to that. Easy as a fish-
reel, friction down to the minimum. Any wonder that it sells? As our
Southern agent wrote us the other day: ‘It wrings out the dollars, as
easy as it does a heavy day’s wash.’”
He laughed softly.
“Yes; it’s given the wringer trade a tough blow—patents all covered.
There ain’t an inch of it they kin imitate. When men like Hiram
Sudwell, president of the National Mangle Company, come sniffin’
round to buy,” he chuckled. “‘Sudwell,’ I says to him, ‘you ain’t got
money enough if you was to pile it as high as the ceilin’ to buy the
Gem.’ He sorter laughed. He knowed there wa’n’t no use.
“‘Couldn’t you let me in a little on the ground floor?’ says he. ‘How
about lettin’ me have ten thousand shares of your preferred? If it’s a
go here’s my check for it,’ says he. I let him talk. I see he was lookin’
kind er down in the mouth. Bimeby he begun to coax an’ whine. ‘See
here,’ says he, ‘there ain’t no use ’n our hemmin’ and hawin’ round
the bush. I’m plain-spoken. The Gem’s a gold mine, and you know it.
Tell you what I’ll do,’ says he; ‘if you’ll let me have ten thousand spot
cash, I’ll throw in five hundred of the Mangle’s preferred just to show
there’s no hard feelin’.’ ‘Sudwell,’ says I, ‘we ain’t sellin’ stock to rival
companies. First thing you know you’d want more. Next thing we’d
know you’d have us out in the cold....’”
Miss Ann had risen. She laid her knitting with a trembling hand in her
work-basket, went over to the window and stood there gazing out,
struggling with herself over a decision so stupendous to that
conservative little woman, that every quivering nerve in her was
strung to its utmost. As she stood by the window she seemed to be
praying.
Suddenly she turned to him, her hands clasped behind her, her eyes
downcast, one small foot slightly advanced toward a step that even
then made her tremble, her mind filled with doubt, that forerunner of
hasty decision.
“I’m going to speak to you very frankly,” she said, in a voice whose
strange weakness belied its courage. “My sister, as you know, is ill.
She has been ill nearly all her life, Mr. Ford. We are neither of us
young; what little money is ours I have always tried to manage for
the best. It is I who have always taken the responsibility of this, and it
is I who must continue to do it. I have no one to come to, either for
counsel or advice, neither for protection. I tell you this frankly, for I
want you to feel it and understand it. Had my sister and I all that is
rightly due us, we should be in far different circumstances.”
She raised her eyes bravely.
“My sister needs comforts, I mean real comforts, Mr. Ford, comforts I
have not dared risk the giving. A purer air than New York, long
summers in some pleasant country place, more luxuries than I feel
we can afford and live within our means, and people around her who
would take her mind from herself. You may not realize it, but far from
growing better, she is growing worse. I, who am constantly with her,
see it only too plainly. Her extreme weakness at times frightens me.
Now what I feel is this——”
Ford started, his shrewd eyes alert to her slightest word or gesture.
“If it were possible to invest safely, as you say, even the small
amount that I could dare give you—it is so serious, Mr. Ford, you
must understand just how I feel. If I were to give you this—and
anything should happen to it——”
Ebner Ford sprang to his feet.
“Can you doubt it,” he exclaimed earnestly, “in the face of plain
figgers? You don’t suppose, my dear friend, I’d lead you into a risk,
do you?”
“I don’t believe you would, sir,” said she. “That would be too cruel.”
He drove his thumbs into his armholes, and for a moment stood in
thought, tapping his fancy waistcoat with his long, bony fingers.
“Suppose I let you have a thousand shares?” he said with a benign
smile. “Think what it would mean to you. No more worryin’ over little
things; you’ll have money enough then to have some peace of mind.”
“I’ve had so little,” she said with a saddened smile, “that it would be
most welcome, I assure you. How much are the shares?” she asked
timidly. “I know so little about such matters.”
“Preferred?” he questioned briskly, elevating his eyebrows. “They
pay you considerable more, you know, than the common stock.”
“I’d like the best;” said she, “that is, if I can afford it.”
“That’s right,” said he. “It always pays to git the best. The best
always pays in the end. There wa’n’t never yit a couple of cheap
things worth one good one. I’d like to see yer git the best—somethin’
you’d be proud of ownin’, like our gilt-edged preferred.” He rammed
his long hands in his trousers pockets, and for some seconds paced
slowly before her, lost in thought. “Let’s see—let’s see,” he muttered.
“I’ll tell you what I’ll do,” he said suddenly. “Let us say fifteen hundred
shares preferred. I’ll waive what they’re worth to-day. I’ll let you have
’em at par, my friend, at ten dollars a share, cash. That’ll make it an
even fifteen thousand dollars. You deserve it, Miss Moulton, if ever
any woman did,” he cried magnanimously. “I’d give a good deal to
see old Hiram Sudwell in your shoes right now.”
“But fifteen thousand dollars,” gasped the little spinster, “is half of all
we’ve got in the world, Mr. Ford!”
“I see,” said he gravely.
She started to speak, but he waved his hand.
“Hold on,” he resumed cheerfully, “We’ll do better than that,” and
again he paced before her. “I’m the last man in the world to ask
anybody to put all their eggs in the same basket. Suppose we say
half that amount?” He saw her hesitate, nervously fingering the long,
thin gold chain that circled her neck, and which all her life had served
her as guardian of her mother’s watch.
“I say half,” said he, breaking the silence. “Why, you’ll think nothin’ of
buyin’ the rest of that fifteen hundred with what you’ll make on that
half.”
“And you advise it?” she ventured. He assured her without speaking,
his expression one of kindly approval, unvarnished, without a vestige
of a doubt. “That would be seven thousand, five hundred dollars,
wouldn’t it?” she inquired, still struggling with herself.
“There ain’t no use of my advisin’ less to you,” he declared. “It
wouldn’t be worth your botherin’ about. I’d like to see you happy—
real happy. You needn’t thank me now, but you’ll thank me some
day, my friend. You won’t never regret it.”
“I—I feel so alone—so helpless,” she returned, “as if I really ought to
think it all seriously over; would you mind letting me do that? I’d feel
better, I think.”
“That’s just what Mrs. Miggs said to me. Now look at her. Do you
suppose Mrs. Miggs has ever regretted it? Her little nest-egg
beginnin’ from the very day she bought her shares; woke up the next
mornin’ knowin’ her troubles were over. Took her little niece straight
down to Stewart’s and bought her a new outfit from head to toe.
Suppose she’d er waited? I want to see you happy, friend. I want that
there happiness to begin now—to-day.” He put forth his hand to her,
forcing her own small hand into its grasp, where it lay as frightened
as a wren with a broken wing.
“Perhaps, then, I’d better decide,” she breathed, with a beating heart,
gazing at the floor.
“That’s right!” he cried. “That’s the right kind of talk. I know sich
matters are hard to think over, and decide. But we’ve done the
thinkin’ and we’ve done the decidin’, ain’t we? And all them gnawin’
little doubts is over.”
“Yes,” she said, looking up at him quickly, and withdrawing her hand,
a strange new courage in her eyes. “I have decided, Mr. Ford. I will
take the seven thousand five hundred dollars’ worth of shares.”
In precisely seven minutes by Ebner Ford’s watch Miss Ann Moulton
became the sole possessor of seven hundred and fifty shares of the
Household Gem, preferred, and its receipt, and before the ink was
fairly dry on her check it was tucked in Ford’s portfolio next to a five-
dollar bill that his stepdaughter had loaned him that morning. He had
feared the sister’s return. He had had experience with two women
deciding together. It was while he was engaged in exploiting the
millions contained in a vast hen industry in the Far West destined to
supply half the eggs to the world—at bottom prices—the army of A
No. 1 Leghorn layers being fed on imitation corn made by a secret
process, producing the best cold-storage egg on the market.
He had hardly reached his room before Miss Jane’s key opened the
front door. He stood screened back of his own ajar, listening to her
as she wearily climbed the stairs, her purple parasol aiding her,
stopping on the landings for breath. It still lacked twenty minutes
before his bank in Union Square closed at three. In less than fifteen
he had handed over to its silent but astonished receiving teller, for
deposit, a check for more money than he had ever had to his credit
in his life.
This done, he walked briskly over to the Everitt House, and through
a swing-door smelling of lemons and old Bourbon sours, feeling a
good deal richer than Hiram Sudwell, and of much more importance
in the world than the President of the United States. The bartender
noticed the change in him at a glance. He seemed younger, more at
his ease. There was already a certain indescribable air of geniality
and prosperity about his customer that sent the bartender’s quick
hand over the bottle of “ordinary” and on to the “special,” hesitated,
and settled over the neck of the decanter of “private stock,” which he
produced with a clean doily and a smile of welcome.
“Warm for March—ain’t it?” remarked Ford, pouring out for himself a
stiff drink.
“It sure is a grand day,” returned the bartender. “Ain’t seen you
around lately, mister—er—busy, I suppose, as usual—well, that’s the
way to be.”
“Busy,” declared Ford. “Ain’t had time to eat.”
Then he paid for his drink, recounted the fifty dollars in new bills he
had drawn, called a cab and went off to Koster & Bial’s, where he
managed to secure, late as it was for the matinée, his favorite seat at
a front-row table.
It was only when Miss Jane reached her room and learned the story
from her sister’s lips that she realized their great good fortune. For
some moments Miss Ann held her in her arms, petting her like a
child.
“I felt it was for the best, dear,” she kept repeating. They both wept a
little; all the worry was over now, her sister assured her. Miss Jane
seemed dazed. She could not fully realize it. She sat on the edge of
her bed, smiling through the tears, smoothing Miss Ann’s hand. Then
they set about making plans for the summer. They decided on Lake
Mohonk. Finally, exhausted as she was, Miss Jane went to bed, Miss
Ann waiting until she fell asleep before straightening out their
meagre accounts of the week before, some of whose items had
frightened her, especially the druggist’s bill which had come in the
morning’s mail with that hopeless letter from her brother. They were
nothing now—new hope, new courage had entered her heart.
 CHAPTER XI
Now it happened that Sue had come in fresh and rosy from a walk,
glowing with health this fine April afternoon, and had brought Pierre
Lamont home with her. There is no secret about where she found
him, nothing could have been more public or more innocent than
their chance meeting on Fifth Avenue before the Reservoir, that solid
and dignified monument with its wavy covering of ivy, which Joe
considered the most impressive mass of stone in the city, with Bryant
Park as its back yard, and enough Croton water soundly held within
its four solemn Egyptian walls to have satisfied the most rabid of
teetotalers, and before which Lamont’s patent-leather shoes and
English buff-colored spats shone resplendently almost every
afternoon between four and five. Indeed, he was so familiar a figure
on Fifth Avenue, that his absence was noticed by many whose daily
habit it was to see and be seen along the city’s most fashionable
highway. More than one man noted in passing the cut and pattern of
Lamont’s clothes before ordering his own. And though, unlike Beau
Brummel, he did not actually set the fashion, they could rest assured
that everything he wore was of the latest. The newest derby was his
the day after it appeared in the window of the best hatter. He was a
connoisseur as well in gloves and walking-sticks. He was said to pay
a formidable price for his clothes, and they were conspicuous in
return for their smartness and good taste. At least he dressed like a
thoroughbred and a gentleman, and his ease and good looks carried
him along triumphantly through many an escapade.
Like Bompard, that idle Norman of Maupassant’s, Lamont “was born
with an unbelievable aptitude to do nothing, and an immoderate
desire never to disturb that vocation.” This, however, did not prevent
him from amusing himself, or of taking a flier on rising stocks, or the
races now and then, with his wife’s money. It is safe to say, he
worked harder in amusing himself than any other New Yorker of his
time, and since there is no more strenuous existence than the daily
pursuit of pleasure, no wonder that the silver touch to his temples
was whiter for his years than most men’s, though even at thirty-five
he had the clean-cut, bronzed complexion of a boy and the hands of
a nobleman. Had Jean Valjean encountered him, he would have
given him some sound advice; he would have said to him, as he did
to Montparnasse: “Some day you will see others afar off working in
the fields, and they will seem to you to be resting.” A counsel that
clever footpad and criminal jeered at while the old ex-convict held
him by the collar—quite as Lamont would have jeered—for every
gentleman’s ways are his own, are they not?—and of no one else’s
business.
Lamont knew Fifth Avenue as well as any man could know it, and as
there is always one popular side to every thoroughfare, he chose
that flanking the Reservoir, his promenade carrying him as far up as
the Fifth Avenue Church, and as far down as the Hotel Brunswick,
which he invariably crossed over to for a cocktail and a look over the
coach horses, and where often several people from London of his
acquaintance were stopping.
Any one with half an eye could have seen how frequently society
women whom he knew stopped to greet him. He made a tall,
handsome figure as he bent over them, chatting about the dinner of
the night before, or the cotillon, or the play, or the new lot of
débutantes. They thought him fascinating—and he was. When a
woman spoke to him, she spoke directly into his brilliant black eyes.
In her presence he was always in a state of irrepressible good-
humor, agreeing with her in everything, and skilful enough, you may
be sure, never to criticise her rival. That he forced a would-be
friendly smile from others, in passing, of no acquaintance
whatsoever, was purely his own affair—and theirs. He always knew
what to say instantly, no matter who she was, or where he imagined
they had last met. No Italian could have been more gallant, and no
Frenchman more courteous or experienced.
He had seen Sue’s trim, slender little figure ahead of him step from
the overcrowded stage, gain the sidewalk, and turn rapidly down
Fifth Avenue. Instantly he quickened his pace, drawing up to her,
Sue unconscious that he was following her, until he smilingly lifted
his hat.
“Hello, little playmate!” he laughed. “And where are you going, pray
tell?” Sue started and turned.
“Why, Mr. Lamont! Why, I’m going home,” said she. “Isn’t it a glorious
day! The stage was so noisy and stuffy I couldn’t stand it any longer.
I just had to get out and walk.”
“Home,” he ventured, with the vestige of a sigh. “May I come?”
“Why—why, yes, of course you may,” she laughed back, “if you’d
really like to,” swept off her feet by their sudden meeting and his
quick proposal.
“Like to!” he smiled. “If you only knew how good you are to ask me.
I’m so wretchedly lonely to-day.”
“Now, Mr. Lamont, that’s a fib and you know it. You don’t mean to tell
me you’re lonely on a day like this? It’s too glorious. Did you ever
see such a sky?”
“I hadn’t noticed it,” he confessed, slipping deftly to her left side.
“Wonderful!” he exclaimed, looking up. “Marvellous! It’s blue, isn’t it?”
“You didn’t think it was green, did you, like the moon? They say it’s
really made of green cheese,” she laughed mischievously. “Isn’t it
just the most adorable blue? Don’t you think New York skies are
wonderful? Didn’t you ever wish you were a swallow, and could go
skimming about in that exquisite space? Think of it.”
“But I don’t want to be a swallow,” said he, swinging his stick. “I
cannot imagine anything more deadly dull than being a swallow. I
enjoy my flights of imagination much more, I assure you. How well
you look.”
She glanced up at him with an embarrassed little smile, her pretty
teeth gleaming whiter than the single small pearl at her throat.
“It’s wonderful how New York agrees with you,” he declared, as they
strode on past the white marble balustrade of the Stewart mansion,
his eyes taking in at their ease the dimples in her rosy cheeks, and
the full color of her lips. “Do you know there’re lots of girls here
who’d give anything for your color. They’re faded out, poor little
dears, with too much rich food and dancing; never get to bed until
morning, and seldom out of it until noon. I never give a débutante
more than six months to look as old as her chaperon.”
“I think we’d better cross here,” she said, as they reached Madison
Square; “it’s shorter.”
“Careful,” said he.
His hand grasped her soft arm tenderly. She felt his strength as he
guided her firmly between the passing carriages, his grip relaxing
again to a gentle pressure that was almost a caress as they reached
the opposite curbstone in safety.
“Thank you,” said she, a little flushed. His lighter prattle had
subsided. On their way through the square they fell quickly into their
bond of common sympathy—music—of which he knew and talked as
fluently as a professional—a wider knowledge sadly lacking in Joe,
whose limitations were confined to the tunes he could whistle. He
filled her eager ears with a host of interesting remarks about the true
value of the diminished seventh, explaining to her how it was often
overdone meaninglessly, like many pyrotechnic displays in chromatic
scales meant to épater the audience, and which no sane composer
would think of letting run riot in his orchestration. “Meaningless
pads,” he called them, and Sue clearly understood. By the time they
had cut through Fourth Avenue and Union Square, he had explained
to her the difference between the weird, cold harmonies of Grieg and
the subtler passion of Chopin, carrying her on to the orchestral
effects of Tschaikowsky, and how he produced them. Then in lighter
vein he spoke of Planquette and his merry “Chimes of Normandy,”
and of Planquette’s snug little villa among his pines and flower-beds
on the Norman French coast, which he had been to and had had
many a good day’s shooting from Planquette’s snipe-blind close by
on the dune, in ear-shot of his piano—of what a genial host he was.
Sue strolled on by his side, absorbed as a child in the midst of a
fairy-tale. By the time they reached Waverly Place, she had had the
most delightful walk of her life. “How could he ever be lonely,” she
thought, “with all those memories? Why had he not told her more of
them before?” She began to feel sorry for her treatment of him that
brilliant tragic evening at the Van Cortlandts’, and almost confessed
it to him as they went up the stoop together and she opened the
dingy black walnut door with its ground-glass panels, one of which
depicted Fortune hugging a dusty sheaf of wheat, and the other,
Mercury in full flight through a firmament of sand-blasted clouds. He
followed her up the stairs. Nothing escaped him, neither the mat
which Ebner Ford had placed himself in front of his threshold, with a
deep “Welcome” branded on it in red letters, or the Rogers group
which Mrs. Ford had generously given to the niche in the hallway,
and which portrayed a putty-colored father reading the evening
paper to the spellbound delight of his wife and five putty-colored
children.
Mrs. Ford, who had just put her hat on and caught sight of them as
they came up the stoop, rushed instantly to the piano; she flew at the
most difficult part of her chef-d’œuvre, “The Storming of Sebastopol,”
with a will, as if nothing had happened, as if Mr. Pierre Lamont was
not only then actually ascending the stairs to her door, if he had not
already reached it; whereas the delighted expectancy of that lady
was so intense, that she mistook the loud pedal for the soft, opening
a broadside from the English fleet at precisely the moment Sue
opened the door. Her surprise as her small, pudgy hands left the
keyboard in the position her “Manual for Beginners” decreed, can be
imagined!
“Why, Mr. Lamont!” she exclaimed effusively, forgetting she had
never met him, oblivious to her daughter’s hasty introduction. “How
good of you to come.”
“We met at the Reservoir,” declared Sue frankly, laying aside her hat
and jacket, and patting her fair hair neatly in place before the mirror
over the mantel.
“By chance, I assure you, Mrs. Ford,” explained Lamont, his Parisian
code of delicacy in such matters tactfully coming to the rescue.
“Well, I’m glad you did,” beamed the mother. “Don’t you think she
looks splendidly, Mr. Lamont?”
She slipped an arm lovingly about her daughter’s neck.
“I’ve already complimented Miss Preston upon that,” he returned
graciously.
“Now, Mr. Lamont, you know how I hate compliments,” protested
Sue.
“But when they’re true,” he laughed, seating himself upon the new
gilt chair Mrs. Ford had offered him.
“Mr. Lamont, I tell her she is much too modest, with all her talents,”
the mother declared, framing the rosy cheeks in her hands, much to
Sue’s embarrassment.
“After all, Mrs. Ford,” returned Lamont, “is there anything more
charming than modesty in a young girl? Isn’t that a talent in itself?
Most girls are so ridiculously conceited nowadays—often over
nothing, I assure you.” He sat gracefully at his ease, his ringed
hands still gloved, still holding his stick and hat, much to the mother’s
surprise and anxiety—another Parisian method—a formality he
carefully observed in calling upon young girls in the presence of their
mothers. Had she been his fiancée he would have done the same in
France. Had she been alone, married or widowed, with the door
liable to open at any instant by husband or friend, at least they would
have found his presence correct and above suspicion, since it can
be logically argued by the French that a gentleman whose hands are
enslaved with his gloves, hat, and stick cannot possibly make love
any more than the ostrich can pursue his mate with his head in the
sand.
Mrs. Ford’s anxiety was noticeable.
“Do let me take your stick and hat,” she ventured, unable longer to
repress her fears of his possible sudden departure. He seemed to
give them to her almost unwillingly, peeling off his dogskin gloves
and expressing himself as deeply touched by her welcome, and
adding that he feared he was “very much de trop,” as he noticed that
she was about to go out.
“You must be frank with me, Mrs. Ford; I fear I am keeping you,” he
declared, rising briskly.
“You see, darling,” she explained to Sue, “I was just going around to
see the little Jones girl; she’s been desperately ill, you know. You
mustn’t think of going, Mr. Lamont. You’ll excuse me, won’t you?—
and you’ll make yourself at home, won’t you? You’ll stay to tea, of
course. Just one moment while I tell the maid.”
“Won’t you please go on telling me more of the wonderful things of
your life, Mr. Lamont?” pleaded Sue, as her mother returned. “Oh,
mother, I have had such a glorious walk. If you could only have
heard all the interesting things Mr. Lamont has been telling me. Do
tell me more about Planquette. Think of it, mother—Mr. Lamont
actually knew him.”
“Oh, do!” exclaimed Mrs. Ford. “How interesting—oh, dear! I wish I
could stay, but I must see the Jones girl. They’ll be hurt if I don’t, you
know, deary,” she smiled, nodding to Sue. “But you’re coming again,
aren’t you, Mr. Lamont?” she insisted, grasping his hand warmly.
“I should be charmed to,” said he, and bowed over her hand; in fact,
he lifted it to his lips, a gesture Mrs. Ford had read about in novels
and seen on the stage, but had never experienced. Her startled,
embarrassed delight did not escape him.
“Then you can tell me all about Planquette,” said she, beaming over
the honor he had bestowed upon her finger-tips. “Planquette! What a
wonderful man he was, wasn’t he? Of course, we’ve all read his
books, his ‘Miserables’ was one of my father’s favorites. Grand, isn’t
it, Mr. Lamont? So full of quaint pathos and humor. I’ve simply
shrieked over it when I was a girl.”
“But, mother dear,” exclaimed Sue, “we were speaking of Planquette,
the composer—not Victor Hugo!”
“Why, of course—how stupid of me.”
“I was just telling your daughter,” he explained, “that I happened to
know Planquette, you see, because my mother and I used to rent a
little villa in Cabourg for the summer, not far from his on the
Normandy coast. We lived in France several years, Mrs. Ford, long
after my schoolboy days there.”
“Think of it! Well, I never; and you really lived in France. Of course
you speak the French language fluently. They say the French are so
excitable. Cora Spink ought to know. She lived a whole month right
in Paris, among the French. She said they pull and haul you about
so.”
The smile he had been able to repress for the last few minutes got
the better of him. He grinned.
“I never found them so,” he confessed quietly. “They’re the kindest
and calmest people in the world.”
“S’pose you’ve seen everything,” she affirmed, edging, to Lamont’s
intense relief, toward the door. “The guillotine, and the Opera House,
and where Napoleon is buried.”
Her small, pudgy hand hesitated on the big, white-china knob, while
she added:
“How well I remember my father’s engravings of these. They hung in
the hall of our ancestral mansion in North Carolina. Mr. Snyder, an
artist neighbor of ours, told my father—I remember so well—it was
just after he became judge—that they were quite valuable. Father
was a great admirer of the French. I recall him now going down into
the cellar himself to decanter some old French brandy we had, the
finest, they used to say, in the State of North Carolina, Mr. Lamont—
as they always said,” she declared proudly, “what the judge didn’t
have under his roof, no other North Carolinian did. Now I must be
going. That little girl’s ears are tingling, I know, to hear more about
your wonderful discoveries. Good-by—or, rather, au revoir I should
say, shouldn’t I?”
She waved her hand lightly toward them both.
“Au revoir, madame,” he returned, with a low bow.
The door with the china knob closed. She was gone, her step
growing fainter down the stairs, and when at last she opened that
half of the front door bearing Fortune hugging her sheaf of wheat,
closed it with a click, and had stepped over the whirling dust and two
circulars of a dentist celebrated for his cheap prices, and had made
her way safely down the stoop, and Sue, with her back to her
precious Chippendale table, started to break the awkward silence
that had followed her mother’s departure, Lamont stretched out both
hands to her pleadingly.
“Come!” he exclaimed, softly. “Let us have a good talk. I have so
much to say to you. Won’t you sit there?” he entreated, nodding to
the sofa.
He saw, with sudden delight, that her lips were quivering, and felt
half the battle won.
“What’s the matter?” he asked, tenderly, his hand hovering
temptingly over her smooth shoulder, the pink flesh veiled by the
thin, dark-blue sheen of her blouse.
“Nothing,” she returned faintly, her voice trembling. “Oh! Mr. Lamont,
please don’t ask me.”
“Are you lonely, too?” he asked. “Something has happened—
something I’ve said, perhaps——”
He bent over her.
“Tell me. Have I hurt you? Tell me, dear—have I?”
She did not lift her eyes. Two big, hot tears blurred them, and went
their own way down her burning cheeks. His word, “dear,” had had
its effect.
“I can’t tell you,” she protested painfully.
“But you must,” he insisted. “I’ve seen a lot, little girl. There’s nothing
that you could ever tell me that I wouldn’t understand.”
She made a brave effort to meet his eyes candidly.
“It wouldn’t be right,” she declared. “That is—it wouldn’t be loyal of
me. Oh! can’t you understand? I should hate myself—afterward.”
“Ah!” he exclaimed. “Then it is more serious than I supposed.”
“You couldn’t help me, if I did tell you,” she managed to say at
length. “No one can help me. I’ve just got to go on and bear it, I
suppose.”
“But I wouldn’t tell a soul,” he insisted, his lips close to her cheek.
“And perhaps I could help you. Little girl—whatever it is I’ll never tell
a soul. There—do you believe me? Ah! my poor little playmate—you
were so happy this afternoon when we met.”
“I’m never really happy,” he heard her murmur. “I’ve never been
really happy for a whole day in my life,” she continued, twisting her
handkerchief nervously into a hard moist knot. “Oh, can’t you
understand?”
“And who has?” he argued cheerily. “Happy for a whole day! Ah, no,
my dear! One is never happy for a whole day. Happiness is never
more than a question of seconds, and even they are rare. Happy for
a whole day! Parbleu! you do not ask much, do you, little
gourmande.”
“So many people are happy,” she faltered.
“You’re not ill?” he ventured. “Bah! Not with that splendid health of
yours. Then what? Tell me, are you in love?”
She started.
“If you are, you’d better get out of it—love’s a terrible game. It
doesn’t pay. It’s about as stupid a pastime as being jealous. Your
eyes are too blue to be jealous. Come, be frank with me—am I
right?”
“Your life’s so different,” she weakened to explain.
“My life? Ah! my poor little playmate, and so you consider my life’s a
happy one—married to a woman who never loved me from the first.”
“Oh, please!” she protested.
“Whose indifference,” he continued, “has taken the heart out of me at
last, whose entire interest lies in her club and her women friends. I
did love her; I loved her madly—madly, do you understand?—but,
what’s the use? Ah, non, mon Dieu!” he cried. “Real happiness in life
lies in a good comrade,” and would have gone on further to explain,
but checked himself. “I see,” he said after a moment. “It’s this,” he
ventured, sweeping his black eyes dramatically over the ugly little
room.
She gave him a startled look in protest.
“I don’t blame you, my dear.”
She feared he would continue. He had guessed the truth, and to her
relief ceased speaking, not daring for the moment to touch even as
skilfully as he could upon her impossible mother, or her stepfather,
whom he could imagine by hearsay but had never seen. Nothing, in
fact, escaped him; neither the sordid commonness of the apartment,
with its hodgepodge of bad taste, its dingy semblance of comfort, or
the mother’s effusive ignorance. He had reached that period in his
suit when he felt that he was wasting time, when he longed to take
this little rose that had tumbled into all this common débris of the
boresome and the ordinary into his arms.
She was again on the verge of confessing to him, innocently enough,
at least how much pretty things appealed to her. Deep down in her
young heart (though she was too loyal to confess it) she saw clearly
her mother’s ignorance and her failings; still deeper down she
abhorred Ebner Ford. Even her respect for him had vanished shortly
after her mother’s marriage. He had even lied to her about the little
money she had earned and had given him. And yet she ended by
saying simply:
“Mother is so silly at times.” Even this she softened by the fact that
she loved her dearly.
“You seem so out of place in all this,” he declared tensely, and so
suddenly that before she knew it he had seized her swiftly in his
arms. “Sue—listen to me!”
“Don’t!” she gasped faintly, every nerve in her quivering in a helpless
effort to free herself.
“Sue! Listen to me—you poor darling!” She strained away from him,
covering her lips with her clenched hands while he sought her fresh
young mouth.
“Don’t!” she pleaded. “Oh, please! Please! Plea——”
He stifled her words with his lips, in a kiss that left her trembling and
dazed. Only when he saw the fear in her eyes, did he open his arms