Explore the full ebook collection and download it now at textbookfull.

com

Introduction to Machine Learning with Applications

in Information Security 1st Edition Mark Stamp

https://textbookfull.com/product/introduction-to-machine-
learning-with-applications-in-information-security-1st-
edition-mark-stamp/

OR CLICK HERE

DOWLOAD EBOOK

Browse and Get More Ebook Downloads Instantly at https://textbookfull.com

Click here to visit textbookfull.com and download textbook now
 Your digital treasures (PDF, ePub, MOBI) await
Download instantly and pick your perfect format...

Read anywhere, anytime, on any device!

Introduction to machine learning with R rigorous

mathematical analysis First Edition Burger

https://textbookfull.com/product/introduction-to-machine-learning-
with-r-rigorous-mathematical-analysis-first-edition-burger/

textbookfull.com

An Introduction To Statistical Learning With Applications

In R Gareth James

https://textbookfull.com/product/an-introduction-to-statistical-
learning-with-applications-in-r-gareth-james/

textbookfull.com

Fundamentals of optimization theory with applications to

machine learning Gallier J.

https://textbookfull.com/product/fundamentals-of-optimization-theory-
with-applications-to-machine-learning-gallier-j/

textbookfull.com

Fundamentals of optimization theory with applications to

machine learning Gallier J.

https://textbookfull.com/product/fundamentals-of-optimization-theory-
with-applications-to-machine-learning-gallier-j-2/

textbookfull.com
Artificial Intelligence With an Introduction to Machine
Learning 2nd Edition Richard E. Neapolitan

https://textbookfull.com/product/artificial-intelligence-with-an-
introduction-to-machine-learning-2nd-edition-richard-e-neapolitan/

textbookfull.com

Introduction to Machine Learning with Python A Guide for

Data Scientists 1st Edition Andreas C. Müller

https://textbookfull.com/product/introduction-to-machine-learning-
with-python-a-guide-for-data-scientists-1st-edition-andreas-c-muller/

textbookfull.com

A first course in machine learning Second Edition Mark

Girolami

https://textbookfull.com/product/a-first-course-in-machine-learning-
second-edition-mark-girolami/

textbookfull.com

Introduction to Machine Learning with Python A Guide for

Data Scientists Andreas C. Müller

https://textbookfull.com/product/introduction-to-machine-learning-
with-python-a-guide-for-data-scientists-andreas-c-muller/

textbookfull.com

Machine learning and security protecting systems with data

and algorithms First Edition Chio

https://textbookfull.com/product/machine-learning-and-security-
protecting-systems-with-data-and-algorithms-first-edition-chio/

textbookfull.com
 INTRODUCTION TO

MACHINE
LEARNING with
APPLICATIONS
in INFORMATION
SECURITY
 Chapman & Hall/CRC
Machine Learning & Pattern Recognition Series

SERIES EDITORS

Ralf Herbrich Thore Graepel

Amazon Development Center Microsoft Research Ltd.
Berlin, Germany Cambridge, UK

AIMS AND SCOPE

This series reflects the latest advances and applications in machine learning and pattern rec-
ognition through the publication of a broad range of reference works, textbooks, and hand-
books. The inclusion of concrete examples, applications, and methods is highly encouraged.
The scope of the series includes, but is not limited to, titles in the areas of machine learning,
pattern recognition, computational intelligence, robotics, computational/statistical learning
theory, natural language processing, computer vision, game AI, game theory, neural networks,
computational neuroscience, and other relevant topics, such as machine learning applied to
bioinformatics or cognitive science, which might be proposed by potential contributors.

PUBLISHED TITLES

BAYESIAN PROGRAMMING
Pierre Bessière, Emmanuel Mazer, Juan-Manuel Ahuactzin, and Kamel Mekhnacha
UTILITY-BASED LEARNING FROM DATA
Craig Friedman and Sven Sandow
HANDBOOK OF NATURAL LANGUAGE PROCESSING, SECOND EDITION
Nitin Indurkhya and Fred J. Damerau
COST-SENSITIVE MACHINE LEARNING
Balaji Krishnapuram, Shipeng Yu, and Bharat Rao
COMPUTATIONAL TRUST MODELS AND MACHINE LEARNING
Xin Liu, Anwitaman Datta, and Ee-Peng Lim
MULTILINEAR SUBSPACE LEARNING: DIMENSIONALITY REDUCTION OF
MULTIDIMENSIONAL DATA
Haiping Lu, Konstantinos N. Plataniotis, and Anastasios N. Venetsanopoulos
MACHINE LEARNING: An Algorithmic Perspective, Second Edition
Stephen Marsland
SPARSE MODELING: THEORY, ALGORITHMS, AND APPLICATIONS
Irina Rish and Genady Ya. Grabarnik
A FIRST COURSE IN MACHINE LEARNING, SECOND EDITION
Simon Rogers and Mark Girolami
INTRODUCTION TO MACHINE LEARNING WITH APPLICATIONS IN
INFORMATION SECURITY
Mark Stamp
 Chapman & Hall/CRC
Machine Learning & Pattern Recognition Series

INTRODUCTION TO

MACHINE
LEARNING with
APPLICATIONS
in INFORMATION
SECURITY

Mark Stamp
San Jose State University
California
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

© 2018 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Printed on acid-free paper

International Standard Book Number-13: 978-1-138-62678-2 (Hardback)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize
to copyright holders if permission to publish in this form has not been obtained. If any copyright material
has not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information storage
or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access
www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc.
(CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization
that provides licenses and registration for a variety of users. For organizations that have been granted a
photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are
used only for identification and explanation without intent to infringe.

Visit the Taylor & Francis Web site at

http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com
To Melody, Austin, and Miles.
Contents

Preface xiii

About the Author xv

Acknowledgments xvii

1 Introduction 1
1.1 What Is Machine Learning? . . . . . . . . . . . . . . . . . . . 1
1.2 About This Book . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Necessary Background . . . . . . . . . . . . . . . . . . . . . . 4
1.4 A Few Too Many Notes . . . . . . . . . . . . . . . . . . . . . 4

I Tools of the Trade 5

2 A Revealing Introduction to Hidden Markov Models 7

2.1 Introduction and Background . . . . . . . . . . . . . . . . . . 7
2.2 A Simple Example . . . . . . . . . . . . . . . . . . . . . . . . 9
2.3 Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.4 The Three Problems . . . . . . . . . . . . . . . . . . . . . . . 14
2.4.1 HMM Problem 1 . . . . . . . . . . . . . . . . . . . . . 14
2.4.2 HMM Problem 2 . . . . . . . . . . . . . . . . . . . . . 14
2.4.3 HMM Problem 3 . . . . . . . . . . . . . . . . . . . . . 14
2.4.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.5 The Three Solutions . . . . . . . . . . . . . . . . . . . . . . . 15
2.5.1 Solution to HMM Problem 1 . . . . . . . . . . . . . . 15
2.5.2 Solution to HMM Problem 2 . . . . . . . . . . . . . . 16
2.5.3 Solution to HMM Problem 3 . . . . . . . . . . . . . . 17
2.6 Dynamic Programming . . . . . . . . . . . . . . . . . . . . . 20
2.7 Scaling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.8 All Together Now . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.9 The Bottom Line . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.10 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

vii
viii CONTENTS

3 A Full Frontal View of Profile Hidden Markov Models 37

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.2 Overview and Notation . . . . . . . . . . . . . . . . . . . . . 39
3.3 Pairwise Alignment . . . . . . . . . . . . . . . . . . . . . . . . 42
3.4 Multiple Sequence Alignment . . . . . . . . . . . . . . . . . . 46
3.5 PHMM from MSA . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6 Scoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
3.7 The Bottom Line . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4 Principal Components of Principal Component Analysis 63

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
4.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.2.1 A Brief Review of Linear Algebra . . . . . . . . . . . . 64
4.2.2 Geometric View of Eigenvectors . . . . . . . . . . . . 68
4.2.3 Covariance Matrix . . . . . . . . . . . . . . . . . . . . 70
4.3 Principal Component Analysis . . . . . . . . . . . . . . . . . 73
4.4 SVD Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.5 All Together Now . . . . . . . . . . . . . . . . . . . . . . . . . 79
4.5.1 Training Phase . . . . . . . . . . . . . . . . . . . . . . 80
4.5.2 Scoring Phase . . . . . . . . . . . . . . . . . . . . . . . 82
4.6 A Numerical Example . . . . . . . . . . . . . . . . . . . . . . 83
4.7 The Bottom Line . . . . . . . . . . . . . . . . . . . . . . . . . 86
4.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

5 A Reassuring Introduction to Support Vector Machines 95

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
5.2 Constrained Optimization . . . . . . . . . . . . . . . . . . . . 102
5.2.1 Lagrange Multipliers . . . . . . . . . . . . . . . . . . . 104
5.2.2 Lagrangian Duality . . . . . . . . . . . . . . . . . . . . 108
5.3 A Closer Look at SVM . . . . . . . . . . . . . . . . . . . . . . 110
5.3.1 Training and Scoring . . . . . . . . . . . . . . . . . . . 112
5.3.2 Scoring Revisited . . . . . . . . . . . . . . . . . . . . . 114
5.3.3 Support Vectors . . . . . . . . . . . . . . . . . . . . . 115
5.3.4 Training and Scoring Re-revisited . . . . . . . . . . . . 116
5.3.5 The Kernel Trick . . . . . . . . . . . . . . . . . . . . . 117
5.4 All Together Now . . . . . . . . . . . . . . . . . . . . . . . . . 120
5.5 A Note on Quadratic Programming . . . . . . . . . . . . . . . 121
5.6 The Bottom Line . . . . . . . . . . . . . . . . . . . . . . . . . 124
5.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
CONTENTS ix

6 A Comprehensible Collection of Clustering Concepts 133

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
6.2 Overview and Background . . . . . . . . . . . . . . . . . . . . 133
6.3 �-Means . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
6.4 Measuring Cluster Quality . . . . . . . . . . . . . . . . . . . . 141
6.4.1 Internal Validation . . . . . . . . . . . . . . . . . . . . 143
6.4.2 External Validation . . . . . . . . . . . . . . . . . . . 148
6.4.3 Visualizing Clusters . . . . . . . . . . . . . . . . . . . 150
6.5 EM Clustering . . . . . . . . . . . . . . . . . . . . . . . . . . 151
6.5.1 Maximum Likelihood Estimator . . . . . . . . . . . . 154
6.5.2 An Easy EM Example . . . . . . . . . . . . . . . . . . 155
6.5.3 EM Algorithm . . . . . . . . . . . . . . . . . . . . . . 159
6.5.4 Gaussian Mixture Example . . . . . . . . . . . . . . . 163
6.6 The Bottom Line . . . . . . . . . . . . . . . . . . . . . . . . . 170
6.7 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

7 Many Mini Topics 177

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
7.2 �-Nearest Neighbors . . . . . . . . . . . . . . . . . . . . . . . 177
7.3 Neural Networks . . . . . . . . . . . . . . . . . . . . . . . . . 179
7.4 Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
7.4.1 Football Analogy . . . . . . . . . . . . . . . . . . . . . 182
7.4.2 AdaBoost . . . . . . . . . . . . . . . . . . . . . . . . . 183
7.5 Random Forest . . . . . . . . . . . . . . . . . . . . . . . . . . 186
7.6 Linear Discriminant Analysis . . . . . . . . . . . . . . . . . . 192
7.7 Vector Quantization . . . . . . . . . . . . . . . . . . . . . . . 202
7.8 Naı̈ve Bayes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
7.9 Regression Analysis . . . . . . . . . . . . . . . . . . . . . . . 205
7.10 Conditional Random Fields . . . . . . . . . . . . . . . . . . . 208
7.10.1 Linear Chain CRF . . . . . . . . . . . . . . . . . . . . 209
7.10.2 Generative vs Discriminative Models . . . . . . . . . . 210
7.10.3 The Bottom Line on CRFs . . . . . . . . . . . . . . . 213
7.11 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213

8 Data Analysis 219

8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
8.2 Experimental Design . . . . . . . . . . . . . . . . . . . . . . . 220
8.3 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
8.4 ROC Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
8.5 Imbalance Problem . . . . . . . . . . . . . . . . . . . . . . . . 228
8.6 PR Curves . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
8.7 The Bottom Line . . . . . . . . . . . . . . . . . . . . . . . . . 231
8.8 Problems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
x CONTENTS

II Applications 235

9 HMM Applications 237

9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
9.2 English Text Analysis . . . . . . . . . . . . . . . . . . . . . . 237
9.3 Detecting Undetectable Malware . . . . . . . . . . . . . . . . 240
9.3.1 Background . . . . . . . . . . . . . . . . . . . . . . . . 240
9.3.2 Signature-Proof Metamorphic Generator . . . . . . . . 242
9.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . 243
9.4 Classic Cryptanalysis . . . . . . . . . . . . . . . . . . . . . . . 245
9.4.1 Jakobsen’s Algorithm . . . . . . . . . . . . . . . . . . 245
9.4.2 HMM with Random Restarts . . . . . . . . . . . . . . 251

10 PHMM Applications 261

10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
10.2 Masquerade Detection . . . . . . . . . . . . . . . . . . . . . . 261
10.2.1 Experiments with Schonlau Dataset . . . . . . . . . . 262
10.2.2 Simulated Data with Positional Information . . . . . . 265
10.3 Malware Detection . . . . . . . . . . . . . . . . . . . . . . . . 269
10.3.1 Background . . . . . . . . . . . . . . . . . . . . . . . . 270
10.3.2 Datasets and Results . . . . . . . . . . . . . . . . . . . 271

11 PCA Applications 277

11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
11.2 Eigenfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
11.3 Eigenviruses . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
11.3.1 Malware Detection Results . . . . . . . . . . . . . . . 280
11.3.2 Compiler Experiments . . . . . . . . . . . . . . . . . . 282
11.4 Eigenspam . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
11.4.1 PCA for Image Spam Detection . . . . . . . . . . . . . 285
11.4.2 Detection Results . . . . . . . . . . . . . . . . . . . . . 285

12 SVM Applications 289

12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
12.2 Malware Detection . . . . . . . . . . . . . . . . . . . . . . . . 289
12.2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . 290
12.2.2 Experimental Results . . . . . . . . . . . . . . . . . . 293
12.3 Image Spam Revisited . . . . . . . . . . . . . . . . . . . . . . 296
12.3.1 SVM for Image Spam Detection . . . . . . . . . . . . 298
12.3.2 SVM Experiments . . . . . . . . . . . . . . . . . . . . 300
12.3.3 Improved Dataset . . . . . . . . . . . . . . . . . . . . 304
CONTENTS xi

13 Clustering Applications 307

13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
13.2 �-Means for Malware Classification . . . . . . . . . . . . . . 307
13.2.1 Background . . . . . . . . . . . . . . . . . . . . . . . . 308
13.2.2 Experiments and Results . . . . . . . . . . . . . . . . 309
13.2.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 313
13.3 EM vs �-Means for Malware Analysis . . . . . . . . . . . . . 314
13.3.1 Experiments and Results . . . . . . . . . . . . . . . . 314
13.3.2 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 317

Annotated Bibliography 319

Index 338
Preface

“Perhaps it hasn’t one,” Alice ventured to remark.

“Tut, tut, child!” said the Duchess.
“Everything’s got a moral, if only you can find it.”
— Lewis Carroll, Alice in Wonderland

For the past several years, I’ve been teaching a class on “Topics in Information
Security.” Each time I taught this course, I’d sneak in a few more machine
learning topics. For the past couple of years, the class has been turned on
its head, with machine learning being the focus, and information security
only making its appearance in the applications. Unable to find a suitable
textbook, I wrote a manuscript, which slowly evolved into this book.
In my machine learning class, we spend about two weeks on each of the
major topics in this book (HMM, PHMM, PCA, SVM, and clustering). For
each of these topics, about one week is devoted to the technical details in
Part I, and another lecture or two is spent on the corresponding applica-
tions in Part II. The material in Part I is not easy—by including relevant
applications, the material is reinforced, and the pace is more reasonable.
I also spend a week covering the data analysis topics in Chapter 8 and
several of the mini topics in Chapter 7 are covered, based on time constraints
and student interest.1
Machine learning is an ideal subject for substantive projects. In topics
classes, I always require projects, which are usually completed by pairs of stu-
dents, although individual projects are allowed. At least one week is allocated
to student presentations of their project results.
A suggested syllabus is given in Table 1. This syllabus should leave time
for tests, project presentations, and selected special topics. Note that the
applications material in Part II is intermixed with the material in Part I.
Also note that the data analysis chapter is covered early, since it’s relevant
to all of the applications in Part II.
1
Who am I kidding? Topics are selected based on my interests, not student interest.

xiii
xiv PREFACE

Table 1: Suggested syllabus

Chapter Hours Coverage

1. Introduction 1 All
2. Hidden Markov Models 3 All
9. HMM Applications 2 All
8. Data Analysis 3 All
3. Profile Hidden Markov Models 3 All
10. PHMM Applications 2 All
4. Principal Component Analysis 3 All
11. PCA Applications 2 All
5. Support Vector Machines 3 All
12. SVM Applications 3 All
6. Clustering 3 All
13. Clustering Applications 2 All
7. Mini-topics 6 LDA and selected topics
Total 36

My machine learning class is taught at the beginning graduate level. For

an undergraduate class, it might be advisable to slow the pace slightly. Re-
gardless of the level, labs would likely be helpful. However, it’s important to
treat labs as supplemental to—as opposed to a substitute for—lectures.
Learning challenging technical material requires studying it multiple times
in multiple different ways, and I’d say that the magic number is three. It’s no
accident that students who read the book, attend the lectures, and conscien-
tiously work on homework problems learn this material well. If you are trying
to learn this subject on your own, the author has posted his lecture videos
online, and these might serve as a (very poor) substitute for live lectures.2
I’m also a big believer in learning by programming—the more code that you
write, the better you will learn machine learning.

Mark Stamp
Los Gatos, California
April, 2017

2
In my experience, in-person lectures are infinitely more valuable than any recorded or
online format. Something happens in live classes that will never be fully duplicated in any
dead (or even semi-dead) format.
About the Author

My work experience includes more than seven years at the National Security
Agency (NSA), which was followed by two years at a small Silicon Valley
startup company. Since 2002, I have been a card-carrying member of the
Computer Science faculty at San Jose State University (SJSU).
My love affair with machine learning began during the early 1990s, when
I was working at the NSA. In my current job at SJSU, I’ve supervised vast
numbers of master’s student projects, most of which involve some combination
of information security and machine learning. In recent years, students have
become even more eager to work on machine learning projects, which I would
like to ascribe to the quality of the book that you have before you and my
magnetic personality, but instead, it’s almost certainly a reflection of trends
in the job market.
I do have a life outside of work.3 Recently, kayak fishing and sailing my
Hobie kayak in the Monterey Bay have occupied most of my free time. I also
ride my mountain bike through the local hills and forests whenever possible.
In case you are a masochist, a more complete autobiography can be found at

http://www.sjsu.edu/people/mark.stamp/

If you have any comments or questions about this book (or anything else)
you can contact me via email at [email protected]. And if you happen
to be local, don’t hesitate to stop by my office to chat.

3
Of course, here I am assuming that what I do for a living could reasonably be classified
as work. My wife (among others) has been known to dispute that assumption.

xv
Acknowledgments

The first draft of this book was written while I was on sabbatical during the
spring 2014 semester. I first taught most of this material in the fall semester
of 2014, then again in fall 2015, and yet again in fall 2016. After the third
iteration, I was finally satisfied that the manuscript had the potential to be
book-worthy.
All of the students in these three classes deserve credit for helping to
improve the book to the point where it can now be displayed in public without
excessive fear of ridicule. Here, I’d like to single out the following students
for their contributions to the applications in Part II.

Topic Students
HMM Sujan Venkatachalam, Rohit Vobbilisetty
PHMM Lin Huang, Swapna Vemparala
PCA Ranjith Jidigam, Sayali Deshpande, Annapurna Annadatha
SVM Tanuvir Singh, Annapurna Annadatha
Clustering Chinmayee Annachhatre, Swathi Pai, Usha Narra

Extra special thanks go to Annapurna Annadatha and Fabio Di Troia.

In addition to her major contributions to two of the applications chapters,
Annapurna helped to improve the end-of-chapter exercises. Fabio assisted
with most of my recent students’ projects and he is a co-author on almost
all of my recent papers. I also want to thank Eric Filiol, who suggested
broadening the range of applications. This was excellent advice that greatly
improved the book.
Finally, I want to thank Randi Cohen and Veronica Rodriguez at the
Taylor & Francis Group. Without their help, encouragement, and patience,
this book would never have been published.
A textbook is like a large software project, in that it must contain bugs.
All errors in this book are solely the responsibility of your humble scribe.
Please send me any errors that you find, and I will keep an updated errata
list on the textbook website.

xvii
Chapter 1

Introduction

I took a speed reading course and read War and Peace in twenty minutes.
It involves Russia.
— Woody Allen

1.1 What Is Machine Learning?

For our purposes, we’ll view machine learning as a form of statistical discrim-
ination, where the “machine” does the heavy lifting. That is, the computer
“learns” important information, saving us humans from the hard work of
trying to extract useful information from seemingly inscrutable data.
For the applications considered in this book, we typically train a model,
then use the resulting model to score samples. If the score is sufficiently high,
we classify the sample as being of the same type as was used to train the
model. And thanks to the miracle of machine learning, we don’t have to
work too hard to perform such classification. Since the model parameters are
(more-or-less) automatically extracted from training data, machine learning
algorithms are sometimes said to be data driven.
Machine learning techniques can be successfully applied to a wide range
of important problems, including speech recognition, natural language pro-
cessing, bioinformatics, stock market analysis, information security, and the
homework problems in this book. Additional useful applications of machine
learning seem to be found on a daily basis—the set of potential applications
is virtually unlimited.
It’s possible to treat any machine learning algorithm as a black box and, in
fact, this is a major selling points of the field. Many successful machine learn-
ers simply feed data into their favorite machine learning black box, which,
surprisingly often, spits out useful results. While such an approach can work,

1
2 INTRODUCTION

the primary goal of this book is to provide the reader with a deeper un-
derstanding of what is actually happening inside those mysterious machine
learning black boxes.
Why should anyone care about the inner workings of machine learning al-
gorithms when a simple black box approach can—and often does—suffice? If
you are like your curious author, you hate black boxes, and you want to know
how and why things work as they do. But there are also practical reasons
for exploring the inner sanctum of machine learning. As with any technical
field, the cookbook approach to machine learning is inherently limited. When
applying machine learning to new and novel problems, it is often essential to
have an understanding of what is actually happening “under the covers.” In
addition to being the most interesting cases, such applications are also likely
to be the most lucrative.
By way of analogy, consider a medical doctor (MD) in comparison to a
nurse practitioner (NP).1 It is often claimed that an NP can do about 80%
to 90% of the work that an MD typically does. And the NP requires less
training, so when possible, it is cheaper to have NPs treat people. But, for
challenging or unusual or non-standard cases, the higher level of training of
an MD may be essential. So, the MD deals with the most challenging and
interesting cases, and earns significantly more for doing so. The aim of this
book is to enable the reader to earn the equivalent of an MD in machine
learning.
The bottom line is that the reader who masters the material in this book
will be well positioned to apply machine learning techniques to challenging
and cutting-edge applications. Most such applications would likely be beyond
the reach of anyone with a mere black box level of understanding.

1.2 About This Book

The focus of this book is on providing a reasonable level of detail for a reason-
ably wide variety of machine learning algorithms, while constantly reinforcing
the material with realistic applications. But, what constitutes a reasonable
level of detail? I’m glad you asked.
While the goal here is for the reader to obtain a deep understanding of
the inner workings of the algorithms, there are limits.2 This is not a math
book, so we don’t prove theorems or otherwise dwell on mathematical theory.
Although much of the underlying math is elegant and interesting, we don’t
spend any more time on the math than is absolutely necessary. And, we’ll
1
A physician assistant (PA) is another medical professional that is roughly comparable
to a nurse practitioner.
2
However, these limits are definitely not of the kind that one typically finds in a calculus
book.
1.2 ABOUT THIS BOOK 3

sometimes skip a few details, and on occasion, we might even be a little bit
sloppy with respect to mathematical niceties. The goal here is to present
topics at a fairly intuitive level, with (hopefully) just enough detail to clarify
the underlying concepts, but not so much detail as to become overwhelming
and bog down the presentation.3
In this book, the following machine learning topics are covered in chapter-
length detail.

Topic Where
Hidden Markov Models (HMM) Chapter 2
Profile Hidden Markov Models (PHMM) Chapter 3
Principal Component Analysis (PCA) Chapter 4
Support Vector Machines (SVM) Chapter 5
Clustering (�-Means and EM) Chapter 6

Several additional topics are discussed in a more abbreviated (section-length)

format. These mini-topics include the following.

Topic Where
�-Nearest Neighbors (�-NN) Section 7.2
Neural Networks Section 7.3
Boosting and AdaBoost Section 7.4
Random Forest Section 7.5
Linear Discriminant Analysis (LDA) Section 7.6
Vector Quantization (VQ) Section 7.7
Naı̈ve Bayes Section 7.8
Regression Analysis Section 7.9
Conditional Random Fields (CRF) Section 7.10

Data analysis is critically important when evaluating machine learning ap-

plications, yet this topic is often relegated to an afterthought. But that’s
not the case here, as we have an entire chapter devoted to data analysis and
related issues.
To access the textbook website, point your browser to

http://www.cs.sjsu.edu/~stamp/ML/

where you’ll find links to PowerPoint slides, lecture videos, and other relevant
material. An updated errata list is also available. And for the reader’s benefit,
all of the figures in this book are available in electronic form, and in color.
3
Admittedly, this is a delicate balance, and your unbalanced author is sure that he didn’t
always achieve an ideal compromise. But you can rest assured that it was not for lack of
trying.
4 INTRODUCTION

In addition, extensive malware and image spam datasets can be found on

the textbook website. These or similar datasets were used in many of the
applications discussed in Part II of this book.

1.3 Necessary Background

Given the title of this weighty tome, it should be no surprise that most of
the examples are drawn from the field of information security. For a solid
introduction to information security, your humble author is partial to the
book [137]. Many of the machine learning applications in this book are
specifically focused on malware. For a thorough—and thoroughly enjoyable—
introduction to malware, Aycock’s book [12] is the clear choice. However,
enough background is provided so that no outside resources should be neces-
sary to understand the applications considered here.
Many of the exercises in this book require some programming, and basic
computing concepts are assumed in a few of the application sections. But
anyone with a modest amount of programming experience should have no
trouble with this aspect of the book.
Most machine learning techniques do ultimately rest on some fancy math.
For example, hidden Markov models (HMM) build on a foundation of dis-
crete probability, principal component analysis (PCA) is based on sophisti-
cated linear algebra, Lagrange multipliers (and calculus) are used to show
how and why a support vector machine (SVM) really works, and statistical
concepts abound. We’ll review the necessary linear algebra, and generally
cover relevant math and statistics topics as needed. However, we do assume
some knowledge of differential calculus—specifically, finding the maximum
and minimum of “nice” functions.

1.4 A Few Too Many Notes

Note that the applications presented in this book are largely drawn from your
author’s industrious students’ research projects. Note also that the applica-
tions considered here were selected because they illustrate various machine
learning techniques in relatively straightforward scenarios. In particular, it is
important to note that applications were not selected because they necessarily
represent the greatest academic research in the history of academic research.
It’s a noteworthy (and unfortunate) fact of life that the primary function of
much academic research is to impress the researcher’s (few) friends with his
or her extreme cleverness, while eschewing practicality, utility, and clarity.
In contrast, the applications presented here are supposed to help demystify
machine learning techniques.
 Part I

Tools of the Trade

5
Chapter 2

A Revealing Introduction to
Hidden Markov Models

The cause is hidden. The effect is visible to all.

— Ovid

2.1 Introduction and Background

Not surprisingly, a hidden Markov model (HMM) includes a Markov pro-
cess that is “hidden,” in the sense that we cannot directly observe the state
of the process. But we do have access to a series of observations that are
probabilistically related to the underlying Markov model.
While the formulation of HMMs might initially seem somewhat contrived,
there exist a virtually unlimited number of problems where the technique
can be applied. Best of all, there are efficient algorithms, making HMMs
extremely practical. Another very nice property of an HMM is that structure
within the data can often be deduced from the model itself.
In this chapter, we first consider a simple example to motivate the HMM
formulation. Then we dive into a detailed discussion of the HMM algorithms.
Realistic applications—mostly from the information security domain—can be
found in Chapter 9.
This is one of the most detailed chapters in the book. A reason for going
into so much depth is that once we have a solid understanding of this partic-
ular machine learning technique, we can then compare and contrast it to the
other techniques that we’ll consider. In addition, HMMs are relatively easy
to understand—although the notation can seem intimidating, once you have
the intuition, the process is actually fairly straightforward.1
1
To be more accurate, your dictatorial author wants to start with HMMs, and that’s all
that really matters.

7
8 HIDDEN MARKOV MODELS

The bottom line is that this chapter is the linchpin for much of the remain-
der of the book. Consequently, if you learn the material in this chapter well,
it will pay large dividends in most subsequent chapters. On the other hand,
if you fail to fully grasp the details of HMMs, then much of the remaining
material will almost certainly be more difficult than is necessary.
HMMs are based on discrete probability. In particular, we’ll need some
basic facts about conditional probability, so in the remainder of this section,
we provide a quick overview of this crucial topic.
The notation “|” denotes “given” information, so that � (� | �) is read as
“the probability of �, given �.” For any two events � and �, we have

� (� and �) = � (�) � (� | �). (2.1)

For example, suppose that we draw two cards without replacement from a
standard 52-card deck. Let � = {1st card is ace} and � = {2nd card is ace}.
Then
� (� and �) = � (�) � (� | �) = 4/52 · 3/51 = 1/221.
In this example, � (�) depends on what happens in the first event �, so we
say that � and � are dependent events. On the other hand, suppose we flip
a fair coin twice. Then the probability that the second flip comes up heads
is 1/2, regardless of the outcome of the first coin flip, so these events are
independent. For dependent events, the “given” information is relevant when
determining the sample space. Consequently, in such cases we can view the
information to the right of the “given” sign as defining the space over which
probabilities will be computed.
We can rewrite equation (2.1) as

� (� and �)
� (� | �) = .
� (�)

This expression can be viewed as the definition of conditional probability.

For an important application of conditional probability, see the discussion of
naı̈ve Bayes in Section 7.8 of Chapter 7.
We’ll often use the shorthand “�, �” for the joint probability which, in
reality is the same as “� and �.” Also, in discrete probability, “� and �” is
equivalent to the intersection of the sets � and � and sometimes we’ll want
to emphasize this set intersection. Consequently, throughout this section

� (� and �) = � (�, �) = � (� ∩ �).

Finally, matrix notation is used frequently in this chapter. A review of

matrices and basic linear algebra can be found in Section 4.2.1 of Chapter 4,
although no linear algebra is required in this chapter.
2.2 A SIMPLE EXAMPLE 9

2.2 A Simple Example

Suppose we want to determine the average annual temperature at a particular
location on earth over a series of years. To make it more interesting, suppose
the years we are focused on lie in the distant past, before thermometers were
invented. Since we can’t go back in time, we instead look for indirect evidence
of the temperature.
To simplify the problem, we only consider “hot” and “cold” for the av-
erage annual temperature. Suppose that modern evidence indicates that the
probability of a hot year followed by another hot year is 0.7 and the proba-
bility that a cold year is followed by another cold year is 0.6. We’ll assume
that these probabilities also held in the distant past. This information can
be summarized as
� �
︂ ︂ (2.2)
� 0.7 0.3
� 0.4 0.6
where � is “hot” and � is “cold.”
Next, suppose that current research indicates a correlation between the
size of tree growth rings and temperature. For simplicity, we only consider
three different tree ring sizes, small, medium, and large, denoted �, � , and �,
respectively. Furthermore, suppose that based on currently available evi-
dence, the probabilistic relationship between annual temperature and tree
ring sizes is given by
� � �
︂ ︂ (2.3)
� 0.1 0.4 0.5
.
� 0.7 0.2 0.1
For this system, we’ll say that the state is the average annual tempera-
ture, either � or �. The transition from one state to the next is a Markov
process,2 since the next state depends only on the current state and the fixed
probabilities in (2.2). However, the actual states are “hidden” since we can’t
directly observe the temperature in the past.
Although we can’t observe the state (temperature) in the past, we can
observe the size of tree rings. From (2.3), tree rings provide us with prob-
abilistic information regarding the temperature. Since the underlying states
are hidden, this type of system is known as a hidden Markov model (HMM).
Our goal is to make effective and efficient use of the observable information,
so as to gain insight into various aspects of the Markov process.
2
A Markov process where the current state only depends on the previous state is said
to be of order one. In a Markov process of order n, the current state depends on the n
consecutive preceding states. In any case, the “memory” is finite—much like your absent-
minded author’s memory, which seems to become more and more finite all the time. Let’s
see, now where was I?
10 HIDDEN MARKOV MODELS

For this HMM example, the state transition matrix is

︂ ︂
0.7 0.3
�= , (2.4)
0.4 0.6

which comes from (2.2), and the observation matrix is

︂ ︂
0.1 0.4 0.5
�= , (2.5)
0.7 0.2 0.1

which comes from (2.3). For this example, suppose that the initial state
distribution, denoted by �, is
︀ ︀
� = 0.6 0.4 , (2.6)

that is, the chance that we start in the � state is 0.6 and the chance that
we start in the � state is 0.4. The matrices �, �, and � are row stochastic,
which is just a fancy way of saying that each row satisfies the requirements
of a discrete probability distribution (i.e., each element is between 0 and 1,
and the elements of each row sum to 1).
Now, suppose that we consider a particular four-year period of interest
from the distant past. For this particular four-year period, we observe the
series of tree ring sizes �, �, �, �. Letting 0 represent �, 1 represent � , and 2
represent �, this observation sequence is denoted as
︀ ︀
� = 0, 1, 0, 2 . (2.7)

We might want to determine the most likely state sequence of the Markov
process given the observations (2.7). That is, we might want to know the most
likely average annual temperatures over this four-year period of interest. This
is not quite as clear-cut as it seems, since there are different possible inter-
pretations of “most likely.” On the one hand, we could define “most likely”
as the state sequence with the highest probability from among all possible
state sequences of length four. Dynamic programming (DP) can be used to
efficiently solve this problem. On the other hand, we might reasonably define
“most likely” as the state sequence that maximizes the expected number of
correct states. An HMM can be used to find the most likely hidden state
sequence in this latter sense.
It’s important to realize that the DP and HMM solutions to this problem
are not necessarily the same. For example, the DP solution must, by defini-
tion, include valid state transitions, while this is not the case for the HMM.
And even if all state transitions are valid, the HMM solution can still differ
from the DP solution, as we’ll illustrate in an example below.
Before going into more detail, we need to deal with the most challenging
aspect of HMMs—the notation. Once we have the notation, we’ll discuss the
Exploring the Variety of Random
Documents with Different Content
 Addressing the Answer.

The answer to an invitation should always be addressed to the

person in whose name it is sent. If “Mr. and Mrs. Richard Roe
request the pleasure,” etc., address the answer to “Mr. and Mrs.
Richard Roe.” If “Mrs. Richard Roe is At Home” on a certain date,
address the reply to her alone. In case of wedding invitations,
address all answers to the parents of the bride, in whose name they
are sent out, never to the bride, although she may be your only
personal acquaintance in the family, the civility being due to the
issuers of the invitation. This is customary in the case of all
invitations.

Wedding Invitations.

Wedding invitations are usually thought to require no answer

unless it be to a sit-down wedding breakfast. In this case the same
exactness in reply and the same form is demanded as for a dinner
invitation. If the invitation is extended to friends at a distance and
pre-supposes an intention to entertain the recipients for any length
of time, the obligation for speedy reply is equally necessary.
If the invitation is given by an informal note, as is the case with
some very quiet weddings, an answer must always be returned and
in the same note form. This attention is demanded by courtesy.
To a large crush wedding a regret, accompanied or not by a gift,
may be sent if desired; an acceptance is not necessary. Where the
invitations are to the church only, they are amply answered by
sending or leaving cards at the house. To receive a card stating that
the wedded pair will be “At Home” on certain dates, means that they
desire to continue their acquaintance with the parties thus invited,
who should either call in person or send cards promptly.

Wedding Anniversaries.
 Anniversary invitations require an answer, thus giving a very
pleasant opportunity for congratulating the happy couple. The
following forms are suitable:

Mr. and Mrs. Arthur Cummings accept with pleasure the kind
invitation of Mr. and Mrs. Kennet Wade for Thursday evening,
October tenth, and present their warmest congratulations on
their Silver Wedding Anniversary. 45 Church Street. Thursday.

For a refusal:

Mr. and Mrs. Arthur Cummings sincerely regret that, owing to

an unexpected absence from town, they are unable to accept
the very kind invitation of Mr. and Mrs. Kennet Wade for
Thursday evening, October tenth, but beg to present to them
their warmest congratulations on this occasion of their Silver
Wedding Anniversary. 24 Church Street. Wednesday.

The same formulæ in answering will apply to any of the

anniversary festivities.

Theater and Opera Parties.

These parties are frequently made up on rather short notice and

the invitations are then sent to the house by special messenger who
awaits the reply, which must be written at once, that the lady or
gentleman giving the entertainment may be sure of a certain
number to fill the box or stalls, engaged for the evening.
Occasionally, when the party is given by a gentleman, he takes a
carriage and gives out the invitations in person when a verbal
answer is returned.

Luncheons and Suppers.

Invitations for these are written in the same form as for a dinner,
merely substituting the word “luncheon” or “supper” for “dinner,”
and should be accepted or refused in precisely the same style.
Answers also should be sent with the same promptness that the
hostess may be certain of arranging her table satisfactorily.

Other Invitations.

Other invitations, aside from those already specified in this

department, scarcely demand an answer, except they bear the
words: “The favor of an answer is requested,” or the initials, “R. S. V.
P.” Simple “at home” affairs never need an answer, though cards
must always be sent, or left in person, immediately afterward.
Garden parties, where they are held at any distance from the city
and carriages are to be sent to convey the guests thither, always
require an answer; this, however, is usually indicated upon the card.

Refusing After Acceptance.

Should it unfortunately occur, after accepting an invitation, that,

by one of the sorrowful happenings so often marring our best laid
plans, we are prevented from fulfilling our promise, let the regret
sent be prompt, that your hostess, especially if the entertainment be
a dinner or luncheon, may possibly, even at the eleventh hour, be
able to supply the vacancy. Make it explanatory as well, that she
may feel positive that no mere whim has caused the disarrangement
of her plans.

What Not to Do.

Never write the word “accepts,” “regrets” or “declines” upon your

visiting card and send in lieu of a written note. To do so is not only
an insult to your hostess but a mark as well of your own ill-breeding.
An invitation, which is always an honor and implies the best that
your host is able to offer, should always receive the courtesy of a
civil reply.
 Etiquette of Courtship and Marriage

“C
OURTSHIP,”
according to
Sterne,
“consists in a
number of quiet
attentions, not so
pointed as to alarm,
nor so vague as not
to be understood.”
In this little quotation lies the spirit and the letter of all etiquette
regarding courtship. The passion of love generally appearing to
everyone save the man who feels it, so entirely disproportionate to
the value of the object, so impossible to be entered into by any
outside individual, that any strong expressions of it appear ridiculous
to a third person. For this reason it is that all extravagance of feeling
should be carefully repressed as an offense against good breeding.
Man was made for woman, and woman equally for man. How shall
they treat each other? How shall they come to understand their
mutual relations and duties? It is lofty work to write upon this
subject what ought to be written. Mistakes, fatal blunders, hearts
and lives wrecked, homes turned into bear-gardens, tears, miseries,
blasted hopes, awful tragedies—can you name the one most prolific
cause of all these?
 If our young people were taught what they ought to know—if it
were told them from infancy up—if it were drilled into them and they
were made to understand what now is all a mystery to them—a
dark, vague, unriddled mystery—hearts would be happier, homes
would be brighter, lives would be worth living and the world would
be better.
“Good Night! Good Night! Parting is such sweet sorrow, That I shall say
good night till it be morrow.”
A POLITE ESCORT.
 This is now the matter—matter grave and serious enough—which
we have in hand. There are gems of wisdom founded on health,
morality, happiness, which should be put within reach of every
household in our whole broad land. It is a most important, yet
neglected subject. People are squeamish, cursed with mock
modesty, ashamed to speak with their lips what their Creator spoke
through their own minds and bodies when he formed them. It is
time such nonsense—nonsense shall we say?—rather say it is time
such fatal folly were withered and cursed by the sober common
sense and moral duty of universal society.
Courtship! Its theme, how delightful! Its memories and
associations, how charming! Its luxuries the most luxurious proffered
to mortals! Its results how far reaching, and momentous! No mere
lover’s fleeting bauble, but life’s very greatest work! None are
equally portentous, for good and evil.

Errors of Love-Making.

God’s provisions for man’s happiness are boundless and endless.

How great are the pleasures of sight, motion, breathing! How much
greater those of mind! Yet a right love surpasses them all; and can
render us all happier than our utmost imaginations can depict; and a
wrong more miserable.
Right love-making is more important than right selection; because
it affects conjugal life for the most. Men and women need
knowledge concerning it more than touching anything else. Their
fatal errors show their almost universal ignorance concerning it. That
most married discords originate in wrong love-making instead of
selection, is proved by love usually declining; while adaptation
remains the same.
Right courtship will harmonize natural discordants, much more
concordants, still more those already in love; which only some
serious causes can rupture. The whole power of this love element is
enlisted in its perpetuity, as are all the self-interests of both. As
nature’s health provisions are so perfect that only its great and long-
continued outrage can break it; so her conjugal are so numerous
and perfect that but for outrageous violation of her love laws all who
once begin can and will grow more and more affectionate and happy
every day.
Any man who can begin to elicit any woman’s love, can perfectly
infatuate her more and more, solely by courting her right; and all
women who once start a man’s love—no very difficult achievement—
can get out of him, and do with him, anything possible she pleases.
The charming and fascinating power of serpents over birds is as
nothing compared with that a woman can wield over a man, and he
over her. Ladies, recall your love hey-day. You had your lover
perfectly spell-bound. He literally knew not what he did or would do.
With what alacrity he sprang to indulge your every wish, at whatever
cost, and do exactly as you desired! If you had only courted him just
right, he would have continued to grow still more so till now. This is
equally true of a man’s power over every woman who once begins to
love him. What would you give to again wield that same bewitching
wand?

How to Carry on Courtship.

Intuition, our own selfhood, is nature’s highest teacher, and

infallible; and tells all, by her “still, small voice within,” whether and
just wherein they are making love right or wrong. Every false step
forewarns all against itself; and great is their fall who stumble.
Courtship has its own inherent consciousness, which must be kept
inviolate.
Then throw yourself, O courting youth, upon your own interior
sense of propriety and right, as to both the beginning and
conducting of courtship, after learning all you can from these pages,
and have no fears as to results, but quietly bide them, in the most
perfect assurance of their happy eventuality!
 “What can I do or omit to advance my suit? prevent dismissal?
make my very best impression? guarantee acceptance? touch my
idol’s heart? court just right?” This is what all true courters say.
Cultivate and manifest whatever qualities you would awaken. You
inspire in the one you court the precise feeling and traits you
yourself experience. This law effects this result. Every faculty in
either awakens itself in the other. This is just as sure as gravity itself.
Hence your success must come from within, depends upon yourself,
not the one courted.
Study the specialties, likes and dislikes in particular, of the one
courted, and humor and adapt yourself to them.
Be extra careful not to prejudice him or her against you by
awakening any faculty in reverse. Thus whatever rouses the other’s
resistance against you, antagonizes all the other faculties, and
proportionally turns love for you into hatred. Whatever wounds
ambition reverses all the other feelings, to your injury; what delights
it, turns them in your favor. All the faculties create, and their action
constitutes human nature; which lovers will do right well to study. To
give an illustration:

A Case to the Point.

An elderly man with points in his favor, having selected a woman

eighteen years younger, but most intelligent and feminine, had two
young rivals, each having more points in their favor, and came to his
final test. She thought much of having plenty of money. They saw
they could “cut him out” by showing her that he was poor; she till
then thinking his means ample. All four met around her table, and
proved his poverty. His rivals retired, sure that they had made “his
cake dough,” leaving him with her. It was his turning-point. He
addressed himself right to her affections, saying little about money
matters, but protesting an amount of devotion for her to which she
knew they were strangers; and left his suit right on this one point;
adding:
“You know I can make money; know how intensely I esteem,
admire, idolize, and love you. Will not my admitted greater affection,
with my earnings, do more for you than they with more money, but
less love?”
Her clear head saw the point. Her heart melted into his. She said
“yes.” He triumphed by this affectional spirit alone over their much
greater availability.
Manifesting the domestic affections and virtues, a warm, gushing
friendly nature, fondness for children and home, inspires a man’s
love most of all, while evincing talents by a man peculiarly enamors
woman.
Relations, you shall not interfere, where even parents may not.
Make your own matches, and let others make theirs; especially if
you have bungled your own. One such bungle is one too many.
The parties are betrothed. Their marriage is “fore-ordained” by
themselves, its only rightful umpires, which all right-minded
outsiders will try to promote, not prevent. How despicable to
separate husbands and wives! Yet is not parting those married by a
love-spirit, equally so? Its mere legal form can but increase its
validity, not create it. Marriage is a divine institution, and consists in
their own personal betrothal. Hence breaking up a true love-union
before its legal consummation, is just as bad as parting loving
husband and wife; which is monstrous. All lovers who allow it are its
wicked partakers.

Choice of Associates.

The first point to be considered on this subject is a careful choice

of associates, which will often, in the end, save future unhappiness
and discomfort, since, as Goldsmith so truthfully puts it, “Love is
often an involuntary passion placed upon our companions without
our consent, and frequently conferred without even our previous
esteem.”
This last most unhappy state of affairs may, to a great extent, be
avoided by this careful choosing of companions. Especially is this
true on the part of the lady, since, from the nature and constitution
of society, an unsuitable acquaintance, friendship, or alliance, is
more embarrassing and more painful for the woman than the man.
As in single life an undesirable acquaintance is more derogatory to a
woman than to a man, so in married life, the woman it is who
ventures most, “for,” as Jeremy Taylor writes, “she hath no sanctuary
in which to retire from an evil husband; she may complain to God as
do the subjects of tyrants and princes, but otherwise she hath no
appeal in the causes of unkindness.”

First Steps.

To a man who has become fascinated with some womanly ideal,

we would say, if the acquaintanceship be very recent, and he, as
yet, a stranger to her relatives, that he should first consider in detail
his position and prospects in life, and judge whether or not they are
such as would justify him in striving to win the lady’s affections, and
later on her hand in marriage. Assured upon this point, and let no
young man think that a fortune is necessary for the wooing of any
woman worth the winning, let him then gain the needful
introductions through some mutual friend to her parents or
guardians.
If, on the other hand, it is a long acquaintance that has ripened
into admiration, this latter formality will be unnecessary.
As to the lady, her position is negative to a great extent. Yet it is
to be presumed that her preferences, though unexpressed, are
decided, and, if the attentions of a gentleman are agreeable, her
manners will be apt to indicate, in some degree, the state of her
mind.
Prudence, however, does, or should, warn her not to accept too
marked attentions from a man of whose past life she knows nothing,
and of whose present circumstances she is equally ignorant.

Character.

There is one paramount consideration too often overlooked and

too late bewailed in many a ruined home, and that is the character
of the man who seeks to win a woman’s hand. Parents and
guardians cannot be too careful in this regard, and young women
themselves should, by refusing such associates, avoid all danger of
contracting such ties. Wealth, nor family rank, nor genius, availeth
aught if the character of the man be flawed.
Let parents teach their daughters and let girls understand for
themselves that happiness, or peace, in married life is impossible
where a man is, in any wise, dissipated, or liable to be overcome by
any of the fashionable vices of the day. Better go down to your
grave a “forlorn spinster” than marry such a man.

Disposition.

As to temper or disposition, the man or woman can easily gain

some insight into the respective peculiarities of another’s
temperament by a little quiet observation. If the gentleman be
courteous and careful in his attentions to his mother and sisters, and
behave with ease and consideration toward all women, irrespective
of age, rank, or present condition, she may feel that her first
estimate was a correct one. On the other hand, should he show
disrespect toward women as a class, sneer at sacred things, evince
an inclination for expensive pleasures in advance of his means, or for
low amusements or companionship; be cruel to the horse he drives,
or display an absence of all energy in his business pursuits, then is it
time to gently, but firmly, repel all nearer advances on his part.
As to the gentleman, it will be well for him also to watch carefully
as to the disposition of the lady and her conduct in her own family.
If she be attentive and respectful to her parents, kind and
affectionate toward her brothers and sisters, not easily ruffled in
temper and with inclination to enjoy the pleasures of home;
cheerful, hopeful and charitable in disposition, then may he feel,
indeed, that he has a prize before him well worth the winning.
If, however, she should display a strong inclination towards
affectation and flirtation; be extremely showy or else careless in her
attire, frivolous in her tastes and eager for admiration, he may
rightly conclude that very little home happiness is to be expected
from her companionship.

Trifling.

A true gentleman will never confine his attentions exclusively to

one lady unless he has an intention of marriage. To do so exposes
her to all manner of conjecture, lays an embargo on the formation of
other acquaintances, may very seriously compromise her happiness,
and by after withdrawal frequently causes her the severest
mortification. Hence a gentleman with no thought of marriage is in
honor bound to make his attentions to ladies as general as possible.
Still more reprehensible is the conduct of the man who insinuates
himself into the affections of a young girl by every protestation and
avowal possible, save that which would be binding upon himself, and
then withdraws his attentions with the boastful consciousness that
he has not committed himself.
Again, the young lady who willfully, knowingly, deliberately, draws
on a man to place hand and heart at her disposal simply for the
pleasure of refusing him and thus adding one more name to her list
of rejected proposals, is utterly unworthy the name of woman.
 Etiquette of Making and Receiving Gifts.

On the question of gifts there is a point of etiquette to be

observed. Gentlemen, as a rule, do not offer ladies presents, save of
fruits, flowers, or confections; which gifts, notwithstanding that a
small fortune may be lavished upon their purchase, are supposed, in
all probability from their perishable character, to leave no obligation
resting upon the lady.
Should the conversation, however, turn upon some new book or
musical composition, which the lady has not seen, the gentleman
may, with perfect propriety, say, “I wish that you could see such or
such a work and, if you will permit, I should be pleased to send you
a copy.” It is then optional with the lady to accept or refuse.
Should a gentleman persist in offering other gifts there must be
no secrecy about it. She should take early opportunity of saying, in
the presence of her father and mother, “I am very much obliged to
you for that ring, pin (or other gift) which you were so kind as to
offer me the other day, and I shall be happy to accept it if Papa or
Mamma does not object.” If the lady is positive in her objections to
receiving gifts, it is easy to say, “I thank you for the kindness but I
never take expensive presents;” or, “Mamma never permits me to
accept expensive presents.” These refusals are always to be taken by
the gentleman in good part. Where a present has been unadvisedly
accepted, it is perfectly proper for the mother to return it with
thanks, saying, “I think my daughter rather young to accept such
expensive gifts.”
After an engagement is formally made the etiquette of gifts is
somewhat altered, though even then expensive presents, unless it
be the engagement ring, are not in the best taste. These should be
reserved for the marriage gifts.

Proposals of Marriage.
 The proposal itself is a subject so closely personal in its nature
that each man must be a law unto himself in the matter, and time
and opportunity will be his only guides to success, unless, mayhap,
his lady-love be the braver of the two and help him gently over the
hardest part, for there be men and men; some who brook not “no”
for an answer, and some that a moment’s hesitation on the part of
the one sought would seal their lips forever.
A woman must always remember that a proposal of marriage is
the highest honor that a man can pay her, and, if she must refuse it,
to do so in such fashion as to spare his feelings as much as possible.
If she be a true and well-bred woman, both proposal and refusal will
be kept a profound secret from every one save her parents. It is the
least balm she can offer to the wounded pride of the man who has
chosen her from out all women to bear his name and to reign in his
home. A wise woman can almost always prevent matters from
coming to the point of a declaration, and, by her actions and her
prompt acceptance of the attentions of others, should strive to show
the true state of her feelings.
A gentleman should usually take “no” for an answer unless he be
of so persevering a disposition as to be determined to take the fort
by siege; or unless the “no” was so undecided in its tone as to give
some hope of finding true the poet’s words:
“He gave them but one tongue to say us, ‘Nay,’
And two fond eyes to grant.”
On the gentleman’s part, a decided refusal should be received as
calmly as possible, and his resolve should be in no way to annoy the
cause of all his pain. If mere indifference be or seem to be the origin
of the refusal, he may, after a suitable length of time, press his suit
once more; but if an avowed or evident preference for another be
the reason, it becomes imperative that he should at once withdraw
from the field. Any reason that the lady may, in her compassion, see
fit to give him as cause for her refusal, should ever remain his
inviolable secret.
SOCIAL PASTIME ON RETURN VOYAGE
DECLINED WITH REGRETS.
 As whatever grows has its natural period for maturing, so has
love. At engagement you have merely selected, so that your
familiarity should be only intellectual, not affectional. You are yet
more acquaintances than companions. As sun changes from
midnight darkness into noonday brilliancy, and heats, lights up, and
warms gradually, and as summer “lingers in the lap of spring;” so
marriage should dally in the lap of courtship. Nature’s adolescence of
love should never be crowded into a premature marriage. The more
personal, the more impatient it is; yet to establish its Platonic aspect
takes more time than is usually given it; so that undue haste puts it
upon the carnal plane, which soon cloys, then disgusts.

Unbecoming Haste.

Coyness and modesty always accompany female love, which

involuntarily shrink from close masculine contact until its mental
phase is sufficiently developed to overrule the antagonistic intimacies
of marriage.
Besides, why curtail the luxuries of courtship? Should haste to
enjoy the lusciousness of summer engulf the delights of spring? The
pleasures of courtship are unsurpassed throughout life, and quite
too great to be curtailed by hurrying marriage. And enhancing or
diminishing them redoubles or curtails those of marriage a
hundredfold more. A happy courtship promotes conjugal felicity
more than anything else whatever. A lady, asked why she didn’t
marry, since she had so many making love to her, replied: “Because
being courted is too great a luxury to be spoilt by marrying.”
No man should wait to make his pile. Two must acquire a
competence conjointly, in order fully to really enjoy it together. This
alone can give full zest to whatever pleasures it produces.
A formal proffer of marriage naturally follows a man’s selection
and decision as to whom he will marry. Consent to canvass their
mutual adaptations implies consent to marry, if all is found
satisfactory; yet a final test and consummation now become
necessary, both to bring this whole matter to a focus, and allow both
to state, and obviate or waive, those objections which must needs
exist on both sides; including any improvements possible in either.

How to Deal with Objections.

The best time to state and waive or remove all objections,

seeming and real, not already adjusted, is at his proposal, and her
acceptance. A verbal will do, but a written is much better, by
facilitating future reference. A long future awaits their marriage;
hence committing this its initial point to writing, so that both can
look back to it, is most desirable. And he can propose, and she
accept, much better when alone, and they have all their faculties
under full control, than verbally, perhaps, when excited. Those same
primal reasons for reducing all other contracts to writing obtain
doubly in reference to marriage.
You who fear awkwardness on paper, remember that true human
nature always appears well, even when poorly dressed. A diamond is
no less brilliant because set in clay. Mode is nothing, reality
everything. All needed to appear well is to feel right, and express
naturally what is felt. Saying plainly what you have to say, is all
required.
The acceptance or rejection should also be unequivocal, or any
contingencies stated, and waived if minor, but if they can neither be
obviated nor compromised, should terminate their relations, that
both may look elsewhere. If any bones of contention exist, now is
the time to inter them finally, and to take the initiatory steps for
perfecting both in each other’s eyes. Bear in mind that as yet your
relations are still those of business merely, because neither has
acquired or conceded any right to love or be loved. Without
pretending to give model letters of proposal, acceptance, or
rejection, because varying circumstances will vary each ad infinitum,
the following may serve as samples from which to work:
 “Much Esteemed Friend: As we have agreed to canvass our
mutual adaptations for marriage, and my own mind is fully
made up, a final decision now becomes necessary.
“What I have learned of and from you confirms that high
opinion of you which prompted my selection of you, and inspires
a desire to consummate it. Your pleasing manner and mode of
saying and doing things; your intelligence, taste, prudence,
kindness, and many other excellencies, inspire my highest
admiration.
“Will you let me love what I so much admire?
“But my affections are sacred. I can bestow them only on one
who reciprocates them; will bestow them upon you, if you will
bestow yours on me; not otherwise; for only mutual love can
render either happy. I can and will love you alone, with all my
heart, provided you can and will love only me, with all of yours.
Do you accord me this privilege, on this condition, for life,
forever? I crave to make you my wife; to live with and for you,
and proffer you my whole being, with honest, assiduous toil,
fidelity to business, what talents I possess, and all I can do to
contribute to your creature comforts. Do you accord me this
privilege, on this condition? May I enshrine you as queen of my
life?
“Say wherein you find me faulty, or capable of improvement
in your eyes, and I will do my utmost, consistently with my
conscience, to render myself worthy and acceptable to you.
“I wish some things were different in you—that you had
better health, arose earlier, were less impulsive, knew more
about keeping house, etc.; yet these minor matters sink into
insignificance in comparison with your many excellences, and
especially that whole-souled affection obviously inherent in you.
“Deliberate fully, for this is a life affair, and if, in order to
decide judiciously, you require to know more of me, ask me, or
 —— and ——. Please reply as soon as you can well decide.
“Decline unless you accept cordially, and can love me truly
and wholly; but if you can and will reciprocate my proffered
affection, say yes, and indicate your own time and mode of our
marriage. Meanwhile, with the highest regards, I am, and hope
ever to remain, Yours truly,
A. B.”

A true woman could give a better answer than the following,

which does not claim to be a model. It is hardly time yet for a
gushing love-letter, or we would not profane this sacred subject by
making the attempt; yet should like to receive one in spirit
somewhat as follows:

“Dear Sir: Your proffer of your hand and heart in marriage has
been duly received, and its important contents fully considered.
“I accept your offer: and on its only condition, that I
reciprocate your love, which I do completely; and hereby both
offer my own hand and heart in return, and consecrate my
entire being, soul and body, all I am and can become, to you
alone; both according you the ‘privilege’ you crave of loving me,
and ‘craving’ a like one in return.
“Thank Heaven that this matter is settled; that you are in very
deed mine, while I am yours, to love and be loved by, live and
be lived with and for; and that my gushing affections have a
final resting-place on one every way so worthy of the fullest
reciprocal sympathy and trust.
“The preliminaries of our marriage we will arrange whenever
we meet, which I hope may be soon. But whether sooner or
later, or you are present or absent, I now consider myself as
wholly yours, and you all mine; and both give and take the
fullest privilege of cherishing and expressing for you that whole-
souled love I find even now gushing up and calling for
 expression. Fondly hoping to hear from and see you soon and
often, I remain wholly yours forever,
C. D.”

Sealing the Vow.

The vow and its tangible witnesses come next. All agreements
require to be attested; and this as much more than others as it is
the most obligatory. Both need its unequivocal and mutual
mementos, to be cherished for all time to come as its perpetual
witnesses. This vow of each to the other can neither be made too
strong, nor held too sacred. If calling God to witness will strengthen
your mutual adjuration, swear by Him and His throne, or by
whatever else will render it inviolable, and commit it to writing, each
transcribing a copy for the other as your most sacred relics, to be
enshrined in your “holy of holies.”
Two witnesses are required, one for each. A ring for her and locket
for him, containing the likeness of both, as always showing how they
now look, or any keepsake both may select, more or less valuable,
to be handed down to their posterity, will answer.
Your mode of conducting your future affairs should now be
arranged. Though implied in selection, yet it must be specified in
detail. Both should arrange your marriage relations; say what each
desires to do, and have done; and draw out a definite outline plan of
the various positions you desire to maintain towards each other. Your
future home must be discussed: whether you will board, or live in
your own house, rented, or owned, or built, and after what pattern;
or with either or which of your parents. And it is vastly important
that wives determine most as to their domiciles; their internal
arrangements, rooms, furniture, management; respecting which
they are consulted quite too little, yet cannot well be too much.
Family rules, as well as national, state, corporate, financial, must
be established. They are most needed, yet least practiced in
marriage. Without them, all must be chaotic. Ignoring them is a
great but common marital error. The Friends wisely make family
method cardinal.

A Full Understanding.

Your general treatment of each other now especially requires to be

mutually agreed upon. Each should say, “I should like to treat and
be treated by you thus, but not so; and let you do this but not that;”
and both mutually agree on a thousand like minor points, better
definitely arranged at first than left for future contention; each
making requisitions, conceding privileges, and stipulating for any
fancies, idols, or “reserved rights.”
Differences must needs arise, which cannot be adjusted too soon.
Those constitutionally inherent in each should be adjusted in love’s
early stages; it matters less how, than whether to your mutual
satisfaction. Or if this is impossible, “agree to disagree;” but settle
on something.
A concessionary spirit is indispensable, and inheres in love. Neither
should insist, but both concede, in all things; each making, not
demanding sacrifices. The one who loves most will yield to oblige
most. What course will make both happiest should overrule all your
mutual relations.
Write down and file all. Your present decisions, subject to mutual
changes and amendments, will become more and more important
for future reference, as time rolls on, by enabling each to correct
both for our own changes make us think others have changed. A
mutual diary is desirable; for incidents now seemingly trivial, may
yet become important.

Important Trifles.
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

textbookfull.com