Nonlinear Digital Filtering with Python An Introduction 1st

Edition

Visit the link below to download the full version of this book:

https://medipdf.com/product/nonlinear-digital-filtering-with-python-an-introduct
ion-1st-edition/

Click Download Now

CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

© 2016 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Version Date: 20150501

International Standard Book Number-13: 978-1-4987-1413-6 (eBook - PDF)

This book contains information obtained from authentic and highly regarded sources. Reasonable efforts
have been made to publish reliable data and information, but the author and publisher cannot assume
responsibility for the validity of all materials or the consequences of their use. The authors and publishers
have attempted to trace the copyright holders of all material reproduced in this publication and apologize to
copyright holders if permission to publish in this form has not been obtained. If any copyright material has
not been acknowledged please write and let us know so we may rectify in any future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced, transmit-
ted, or utilized in any form by any electronic, mechanical, or other means, now known or hereafter invented,
including photocopying, microfilming, and recording, or in any information storage or retrieval system,
without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access www.copyright.
com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc. (CCC), 222 Rosewood
Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization that provides licenses and
registration for a variety of users. For organizations that have been granted a photocopy license by the CCC,
a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used
only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com

and the CRC Press Web site at

http://www.crcpress.com

© 2016 by Taylor & Francis Group, LLC

 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page v — #7

i i

Contents

Preface ix

Authors xiii

1 Introduction 1
1.1 Linear versus nonlinear filters: an example . . . . . . . . . . . . . 2
1.2 Why nonlinearity? Data cleaning filters . . . . . . . . . . . . . . 10
1.3 The many forms of nonlinearity . . . . . . . . . . . . . . . . . . . 14
1.3.1 A simple nonlinear filter taxonomy . . . . . . . . . . . . . 15
1.3.2 Recursive filters: instability and chaos . . . . . . . . . . . 17
1.4 Python and reproducible research . . . . . . . . . . . . . . . . . . 19
1.4.1 A very brief introduction to Python . . . . . . . . . . . . 20
1.4.2 Reproducible research . . . . . . . . . . . . . . . . . . . . 21
1.4.3 How this book was developed . . . . . . . . . . . . . . . . 24
1.5 Organization of this book . . . . . . . . . . . . . . . . . . . . . . 25

2 Python 27
2.1 A high-level overview of the language . . . . . . . . . . . . . . . . 27
2.1.1 How Python programs work . . . . . . . . . . . . . . . . . 28
2.1.2 Using Python interactively . . . . . . . . . . . . . . . . . 28
2.1.3 Using Python from the command prompt . . . . . . . . . 32
2.2 Key language elements . . . . . . . . . . . . . . . . . . . . . . . . 33
2.2.1 The general structure of Python . . . . . . . . . . . . . . 34
2.2.2 Object types . . . . . . . . . . . . . . . . . . . . . . . . . 35
2.2.3 Data types . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.2.4 Lists: a key data type . . . . . . . . . . . . . . . . . . . . 39
2.2.5 Tuples: immutable lists . . . . . . . . . . . . . . . . . . . 41
2.2.6 Character strings . . . . . . . . . . . . . . . . . . . . . . . 42
2.2.7 Dictionaries: another key data type . . . . . . . . . . . . 45
2.2.8 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.2.9 Control structures . . . . . . . . . . . . . . . . . . . . . . 49
2.2.10 Input and output . . . . . . . . . . . . . . . . . . . . . . . 53
2.3 Caveat emptor: a few Python quirks . . . . . . . . . . . . . . . . 58
2.3.1 In-place operations, side-effects, and None returns . . . . . 58

v
© 2016 by Taylor & Francis Group, LLC
i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page vi — #8

i i

vi CONTENTS

2.3.2 Deep versus shallow copies . . . . . . . . . . . . . . . . . 59

2.3.3 Default values for function parameters . . . . . . . . . . . 61
2.4 A few filtering examples . . . . . . . . . . . . . . . . . . . . . . . 62
2.4.1 The SymmetricFilter function . . . . . . . . . . . . . . . 62
2.4.2 Moving window end effects and the Extend function . . . 64
2.4.3 The SMfilter function . . . . . . . . . . . . . . . . . . . 66
2.4.4 The LinearFIRfilter function . . . . . . . . . . . . . . . 69
2.5 Learning more about Python . . . . . . . . . . . . . . . . . . . . 71

3 Linear and Volterra Filters 77

3.1 Linear digital filters . . . . . . . . . . . . . . . . . . . . . . . . . 78
3.1.1 Linear filter representations . . . . . . . . . . . . . . . . . 78
3.1.2 Ideal filters and idempotence . . . . . . . . . . . . . . . . 82
3.1.3 Positive-linear filters . . . . . . . . . . . . . . . . . . . . . 84
3.2 Linearity, smoothness, and harmonics . . . . . . . . . . . . . . . 86
3.3 Volterra filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.3.1 The Volterra filter structure . . . . . . . . . . . . . . . . . 94
3.3.2 Some important special cases . . . . . . . . . . . . . . . . 94
3.3.3 General block-oriented structures . . . . . . . . . . . . . . 101
3.4 Universal approximations . . . . . . . . . . . . . . . . . . . . . . 102
3.4.1 Specific results . . . . . . . . . . . . . . . . . . . . . . . . 102
3.4.2 Limitations of universal approximation . . . . . . . . . . . 104

4 Median Filters and Some Extensions 105

4.1 The standard median filter . . . . . . . . . . . . . . . . . . . . . 105
4.1.1 The problem of edge jitter . . . . . . . . . . . . . . . . . . 106
4.1.2 Median filter root sequences . . . . . . . . . . . . . . . . . 109
4.2 Median filter cascades . . . . . . . . . . . . . . . . . . . . . . . . 113
4.2.1 Median filter convergence . . . . . . . . . . . . . . . . . . 113
4.2.2 The data sieve . . . . . . . . . . . . . . . . . . . . . . . . 114
4.3 Order statistic filters . . . . . . . . . . . . . . . . . . . . . . . . . 116
4.4 The recursive median filter . . . . . . . . . . . . . . . . . . . . . 117
4.4.1 Definition of the recursive median filter . . . . . . . . . . 117
4.4.2 Root sequences and idempotence . . . . . . . . . . . . . . 118
4.4.3 A simple example . . . . . . . . . . . . . . . . . . . . . . 119
4.5 Weighted median filters . . . . . . . . . . . . . . . . . . . . . . . 119
4.5.1 Weighted medians . . . . . . . . . . . . . . . . . . . . . . 121
4.5.2 The weighted median filter . . . . . . . . . . . . . . . . . 124
4.5.3 A weighted median filter example . . . . . . . . . . . . . . 128
4.5.4 Weighted median filter root sequences . . . . . . . . . . . 129
4.5.5 Center-weighted median filters . . . . . . . . . . . . . . . 130
4.5.6 Recursive weighted median filters . . . . . . . . . . . . . . 131
4.5.7 Weighted median filter cascades . . . . . . . . . . . . . . . 133
4.6 Threshold decompositions and stack filters . . . . . . . . . . . . . 134
4.7 The Hampel filter . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
4.7.1 Decision-theoretic filters . . . . . . . . . . . . . . . . . . . 137

© 2016 by Taylor & Francis Group, LLC

i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page vii — #9

i i

CONTENTS vii

4.7.2 Relation to the median filter . . . . . . . . . . . . . . . . 139

4.7.3 MADM implosion: a cautionary note . . . . . . . . . . . . 139
4.7.4 A Hampel filter example . . . . . . . . . . . . . . . . . . . 140
4.8 Python implementations . . . . . . . . . . . . . . . . . . . . . . . 142
4.9 Chapter summary . . . . . . . . . . . . . . . . . . . . . . . . . . 148

5 Forms of Nonlinear Behavior 149

5.1 Linearity versus additivity . . . . . . . . . . . . . . . . . . . . . . 150
5.2 Homogeneity and positive homogeneity . . . . . . . . . . . . . . . 152
5.2.1 Homogeneity . . . . . . . . . . . . . . . . . . . . . . . . . 152
5.2.2 Positive homogeneity . . . . . . . . . . . . . . . . . . . . . 154
5.2.3 L-filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
5.3 Generalized homogeneity . . . . . . . . . . . . . . . . . . . . . . . 157
5.3.1 Homogeneity of order zero . . . . . . . . . . . . . . . . . . 159
5.3.2 Ranks are positive homogeneous of order zero . . . . . . . 161
5.3.3 Generalized combination filters . . . . . . . . . . . . . . . 162
5.4 Location-invariance . . . . . . . . . . . . . . . . . . . . . . . . . . 163
5.4.1 The Gastwirth filter . . . . . . . . . . . . . . . . . . . . . 164
5.4.2 Mallows’ class of nonlinear smoothers . . . . . . . . . . . 165
5.5 Restricted linearity . . . . . . . . . . . . . . . . . . . . . . . . . . 166
5.6 Nonlinear structure versus behavior . . . . . . . . . . . . . . . . . 168

6 Composite Structures: Bottom-up Design 169

6.1 A practical overview . . . . . . . . . . . . . . . . . . . . . . . . . 169
6.1.1 Cascades, categories, and LULU filters . . . . . . . . . . . 169
6.1.2 Parallel combinations, groups, and MMD filters . . . . . . 170
6.1.3 Algebraic clones and the FMH filter . . . . . . . . . . . . 172
6.2 Cascade interconnections and categories . . . . . . . . . . . . . . 173
6.2.1 A brief introduction to category theory . . . . . . . . . . 173
6.2.2 Filter categories . . . . . . . . . . . . . . . . . . . . . . . 176
6.2.3 The LULU filter category . . . . . . . . . . . . . . . . . . 185
6.3 Parallel interconnections and groupoids . . . . . . . . . . . . . . 189
6.3.1 Two illustrative examples . . . . . . . . . . . . . . . . . . 189
6.3.2 Groups, monoids, semigroups, and groupoids . . . . . . . 190
6.3.3 Parallel combinations of K-linear filters . . . . . . . . . . 192
6.3.4 Associative binary operators . . . . . . . . . . . . . . . . 194
6.3.5 Positive homogeneous associative binary operators . . . . 195
6.4 Clones: more general interconnections . . . . . . . . . . . . . . . 196
6.4.1 Basic concepts of clones . . . . . . . . . . . . . . . . . . . 196
6.4.2 General filter clone characteristics . . . . . . . . . . . . . 198
6.4.3 Clone representation of the Hampel filter . . . . . . . . . 199
6.4.4 The L-filter clone . . . . . . . . . . . . . . . . . . . . . . . 200
6.5 Python implementations . . . . . . . . . . . . . . . . . . . . . . . 204
6.6 Extensions to more general settings . . . . . . . . . . . . . . . . . 219

© 2016 by Taylor & Francis Group, LLC

i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page viii — #10

i i

viii CONTENTS

7 Recursive Structures and Stability 221

7.1 What is different about recursive filters? . . . . . . . . . . . . . . 222
7.2 Recursive filter classes . . . . . . . . . . . . . . . . . . . . . . . . 224
7.3 Initializing recursive filters . . . . . . . . . . . . . . . . . . . . . . 226
7.4 BIBO stability . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228
7.4.1 The concept of BIBO stability . . . . . . . . . . . . . . . 228
7.4.2 BIBO stability of non-recursive filters . . . . . . . . . . . 229
7.4.3 A rational filter counterexample . . . . . . . . . . . . . . 230
7.4.4 BIBO stability of recursive filters . . . . . . . . . . . . . . 230
7.4.5 Finite stability . . . . . . . . . . . . . . . . . . . . . . . . 231
7.5 Steady-state responses . . . . . . . . . . . . . . . . . . . . . . . . 232
7.5.1 Steady-state characterizations . . . . . . . . . . . . . . . . 232
7.5.2 Consequences of ouput multiplicity . . . . . . . . . . . . . 234
7.6 Asymptotic stability . . . . . . . . . . . . . . . . . . . . . . . . . 236
7.6.1 General formulation of the problem . . . . . . . . . . . . . 236
7.6.2 Asymptotic stability of linear recursive filters . . . . . . . 238
7.6.3 Asymptotic and BIBO stability are different . . . . . . . . 242
7.6.4 Input-dependent stability . . . . . . . . . . . . . . . . . . 245
7.6.5 PAC for nonrecursive filters . . . . . . . . . . . . . . . . . 246
7.7 Inherently nonlinear behavior . . . . . . . . . . . . . . . . . . . . 247
7.7.1 Subharmonic generation . . . . . . . . . . . . . . . . . . . 248
7.7.2 Chaos . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
7.8 Fading memory filters . . . . . . . . . . . . . . . . . . . . . . . . 251
7.9 Structured Lipschitz filters . . . . . . . . . . . . . . . . . . . . . . 253
7.9.1 Basic definitions . . . . . . . . . . . . . . . . . . . . . . . 254
7.9.2 Stability results . . . . . . . . . . . . . . . . . . . . . . . . 255
7.9.3 Examples of structured Lipschitz filters . . . . . . . . . . 256
7.10 Behavior of key nonlinear filter classes . . . . . . . . . . . . . . . 257
7.10.1 Linear recursive filters . . . . . . . . . . . . . . . . . . . . 257
7.10.2 NFIR filters . . . . . . . . . . . . . . . . . . . . . . . . . . 258
7.10.3 The recursive median filter . . . . . . . . . . . . . . . . . 259
7.10.4 Block-oriented filters . . . . . . . . . . . . . . . . . . . . . 260
7.10.5 Lur’e filters . . . . . . . . . . . . . . . . . . . . . . . . . . 261
7.10.6 Bilinear filters . . . . . . . . . . . . . . . . . . . . . . . . . 262
7.10.7 Output-affine filters . . . . . . . . . . . . . . . . . . . . . 263
7.10.8 Polynomial recursive filters . . . . . . . . . . . . . . . . . 265
7.11 Stability of interconnected systems . . . . . . . . . . . . . . . . . 265
7.12 Challenges and potential of recursive filters . . . . . . . . . . . . 270
7.12.1 Block-oriented designs . . . . . . . . . . . . . . . . . . . . 271
7.12.2 Output affine filters . . . . . . . . . . . . . . . . . . . . . 271
7.12.3 ZPSL filters . . . . . . . . . . . . . . . . . . . . . . . . . . 272

Bibliography 273

Index 283

© 2016 by Taylor & Francis Group, LLC

i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page ix — #11

i i

Preface

Linear filters that separate signals into components based on their frequency
content play an essential role in the theory and practice of electrical engineer-
ing. Historically, the linearity of these filters has been exploited heavily in the
development of theoretical characterizations and practical design rules. Unfor-
tunately, not all practical signal separation problems are amenable to linear
solutions: the data cleaning problem that provides the central motivation for
this book is a case in point. There, localized “spikes,” “glitches,” or “impul-
sive noise” cannot be effectively removed from other signals using linear filters.
In fact, this inability is a direct consequence of linearity, as demonstrated in
Chapter 1.
Fortunately, a number of nonlinear digital filters have been developed to
address these problems, and research in this area remains active. A classic
example is the standard median filter, proposed by John Tukey in a confer-
ence paper in 1974. This filter turns out to be extremely effective in removing
impulsive “spikes” from signals, but in many applications it also introduces un-
acceptable distortion in the underlying signal components we wish to preserve.
These two characteristics of the median filter have led to the development of
many extensions that attempt to retain the median filter’s effectiveness against
impulsive noise, while reducing its tendency to distort signal components of
interest. Because of the central role the class of median-based filters plays in
practial applications, this book devotes an entire chapter to this class, including
such extensions of the standard median filter as weighted median filters, recur-
sive median filters, recursive weighted median filters, and further generalizations
like the class of stack filters.
A key practical challenge in the analysis of these filters and the design of new
ones is their nonlinearity. Specifically, linearity represents an extremely pow-
erful mathematical condition that has led to remarkably complete and useful
characterizations of linear filters. Removing this mathematical constraint gives
us tremendous additional flexibility in how the resulting filters can behave, but
it introduces equally tremendous obstacles to their analysis and design. In par-
ticular, while the class of linear, time-invariant digital filters exhibits essentially
complete and essentially equivalent behavioral and structural characterizations,
this is no longer true for the class of nonlinear digital filters. That is, the prin-
ciple of superposition describes how linear filters behave, and this behavioral
description leads directly to the convolution representation of these filters in

ix
© 2016 by Taylor & Francis Group, LLC
i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page x — #12

i i

x PREFACE

terms of their impulse response. This structural characterization and various

other, essentially equivalent representations (e.g., the filter’s frequency response)
provide a practical basis for linear filter design.
Conversely, the term “nonlinear digital filter” leads to neither complete be-
havioral nor complete structural descriptions. This essential characteristic of
nonlinear digital filters forces us to focus on specific sub-classes of these filters
for which one or the other characterization is available. Since structural charac-
terizations provide a basis for practical filter design, most books on the subject
adopt a structural approach, focusing on one or more structurally-defined filter
classes (e.g., Volterra filters based on polynomial nonlinearities). Conversely, it
is possible to define nonlinear filter classes behaviorally by replacing the prin-
cipal of superposition with some other mathematical condition. One example
is homogeneity, or invariance under input scaling: multiplying the filter’s input
sequence by some constant λ causes the response to be multiplied by the same
constant. This alternative condition—which may be viewed as a “relaxation of
linearity”—is satisfied by all linear filters, but it is also satisfied by the median
filter and a number of other practically important nonlinear filters. Furthermore,
homogeneity is an extremely desirable characteristic in data cleaning filters, be-
cause it means that the operations of filtering and re-scaling the data commute.
For example, applying a homogeneous filter to a sequence of velocities expressed
in kilometers per hour and then converting the results to miles per hour gives
exactly the same results as first converting the squence to miles per hour and
then filtering the converted sequence.
This book adopts both of these approaches—structural and behavioral—in
characterizing and designing nonlinear digital filters. Important structural filter
classes discussed here include the median filter and a number of its extensions
(e.g., weighted and recursive median filters) and Volterra filters based on poly-
nomial nonlinearities. An important focus of this book is the use of results
from algebra and the theory of functional equations to construct and charac-
terize behaviorally-defined nonlinear filter classes (e.g., the class of positive-
homogeneous filters that includes all linear filters, the median filter and most
of its extensions, and a number of other classes like the LULU filters described
by Rohwer, but does not include Volterra filters). This approach allows us to
analyze the impact of a range of useful interconnection strategies on filter behav-
ior. These ideas are discussed in some detail in Chapter 6 and they allow us to
develop practical “bottom-up” design strategies for combining relatively simple
component filters into more complex—and thus more flexible—composite struc-
tures, while retaining key behavioral characteristics of the component filters.
An important difference between the nonlinear digital filters described in
this book and their linear counterparts is that it is much harder to design use-
ful recursive nonlinear filters than to design useful recursive linear filters. This
difference—and the fundamental reasons behind it—motivates Chapter 7, which
discusses and illustrates the behavioral consequences of allowing recursive (i.e.,
feedback) interconnections in nonlinear digital filters. In particular, the stability
of nonlinear digital filters is a much more complicated topic than the stability of
linear filters: concepts that are equivalent or nearly so in the linear case become

© 2016 by Taylor & Francis Group, LLC

i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page xi — #13

i i

xi

both distinct and much harder to characterize in the nonlinear case. In addi-
tion, many nonlinear recursive filters can exhibit highly undesirable behavior like
chaotic responses to simple input changes (e.g., steps or impulses), something
nonrecursive nonlinear digital filters are immune to. Beyond explaining why
most nonlinear digital filters now in use are nonrecursive—the recursive median
filter is a notable exception, discussed in both Chapters 4 and 7—a key motiva-
tion for including a chapter on recursive nonlinear filters here is to highlight an
extremely challenging but potentially equally promising research frontier. That
is, recursive linear filters are known to exhibit a number of practial advantages
over their nonrecursive counterparts. If a class of recursive nonlinear filters
could be identified that was immune to the worst forms of pathological behavior
described in Chapter 7, it might form the basis for novel new nonlinear filter
designs with characteristics superior to anything known today.
Without question, the most popular software environment for digital filter
research and design is MATLAB. Why, then, does this book advocate Python?
First, while excellent, MATLAB is a commercial product with a non-negligible
cost, while Python is a free, open-source computing environment. Although
the level of specialized support for digital filter design is much more extensive
in MATLAB, there is some support in Python (e.g., the scipy package sup-
ports linear digital filter design), and this support is growing (indeed, this book
represents a small contribution). Second, MATLAB was developed to support
mathematical computations based on linear algebra and related notions, while
Python is a general-purpose programming language like Java or C, with an
enormous variety of support packages available to extend the language’s ba-
sic capabilities: as of 29-March-2015, the Python package index (PyPI) listed
57,266 add-on packages, a number that grows daily. These packages support
everything from machine learning tools (e.g., in the scikit-learn package) and
natural language processing (e.g., in the nltk package), to web scraping (e.g.,
with the scrapy package) and HTML parsing (e.g., with the BeautifulSoup
package). In addition, Python has extensive capabilities as an operating system-
independent scripting language (e.g., file management, scheduling tasks, etc.),
with extensive support for website development. Finally, Python also provides
substantial and growing support for reproducible research or dynamic document
development, creating documents that contain text, computational results, and
the code used to obtain those results. This last capability was used in the prepa-
ration of this book and it offers enormous time savings when minor revisions
are required, a point discussed further in Chapter 1. Python implementations
of the most important digital filters discussed in this book are included, and
Chapter 2 presents an essential introduction to Python programming for those
with little or no prior Python experience.
This book is intended as either a textbook or a reference for self-study. As
a textbook, the material presented here would be appropriate for an introduc-
tory course on nonlinear digital filters, at either the advanced undergraduate or
the beginning graduate level. As a reference for self-study, this book should be
useful to those interested in implementing, developing, and using data cleaning
filters for dynamic data analysis or time-series modeling. Specific application

© 2016 by Taylor & Francis Group, LLC

i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page xii — #14

i i

xii PREFACE

areas include physics, analytical chemistry, engineering disciplines (especially

applications like industrial process modeling and control), or econometrics. Lit-
tle background is assumed beyond a certain level of comfort with mathematical
ideas and notation. In particular, no prior exposure to algebra, functional equa-
tions, or Python programming is assumed: where specific concepts or details
are required, they are introduced here.

© 2016 by Taylor & Francis Group, LLC

i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page xiii — #15

i i

Authors

Ronald K. Pearson holds the position of Data Scientist with DataRobot in

Boston, a software company whose products support large-scale predictive mod-
eling for a wide range of business applications and are based on Python and R.
Previously, he held a variety of industrial, business, and academic positions,
including both the DuPont Company and the Swiss Federal Institute of Tech-
nology (ETH Zürich), where he was an active researcher in the area of nonlin-
ear dynamic modeling for industrial process control, the Tampere University of
Technology where he was a visiting professor involved in teaching and research
in nonlinear digital filters, and the Travelers Companies, where he was involved
in predictive modeling for insurance applications. He holds a PhD in Elec-
trical Engineering and Computer Science from the Massachussetts Institute of
Technology and has published conference and journal papers on topics ranging
from nonlinear dynamic model structure selection to the problems of disguised
missing data in predictive modeling. Dr. Pearson has authored or co-authored
four previous books, the most recent being Exploring Data in Engineering, the
Sciences, and Medicine (Oxford University Press, 2011).
Moncef Gabbouj is an Academy of Finland Professor of Signal Process-
ing in the Department of Signal Processing at Tampere University of Technol-
ogy, Tampere, Finland. He received his BS degree in Electrical Engineering in
1985 from Oklahoma State University, Stillwater, and his MS and PhD degrees
in Electrical Engineering from Purdue University, West Lafayette, Indiana, in
1986 and 1989, respectively. Dr. Gabbouj is internationally recognized for his
research in the areas of nonlinear signal and image processing and analysis. In
addition, his research interests include multimedia analysis, indexing and re-
trieval, machine learning, voice conversion, and video processing and coding.
Dr. Gabbouj held several visiting professorships at different universities,
including The Hong Kong University of Science and Technology, Hong Kong;
Purdue University; the University of Southern California; and the American
University of Sharjah, United Arab Emirates. He was Head of the Tampere
University of Technology Department of Signal Processing from 2002-2007, and
served as Senior Research Fellow of the Academy of Finland from 1997–1998
and from 2007–2008.
Dr. Gabbouj is a Fellow of the Institute of Electrical and Electronics Engi-
neers (IEEE), a member of the Academia Europaea, and of the Finnish Academy
of Science and Letters. He is a member of the IEEE Fourier Award, and past

xiii
© 2016 by Taylor & Francis Group, LLC
i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page xiv — #16

i i

xiv AUTHORS

Chairman of the DSP Technical Committee of the IEEE Circuits and Systems
Society. He was Honorary Guest Professor of Jilin University, China (2005–
2010). He served as Distinguished Lecturer for the IEEE Circuits and Systems
Society in 2004–2005, and is a Past-Chairman of the IEEE-EURASIP Nonlinear
Signal and Image Processing Board. He was Chairman of the Algorithm Group
of the Eurpean Commisson COST 211quat, served as Associate Editor of the
IEEE Transactions on Image Processing, and was Guest Editor of Multimedia
Tools and Applications and the European Journal Applied Signal Processing.
He is the past Chairman of the IEEE Finland Section and the IEEE SP/CAS
Finland Chapter. He was also Chairman and Technical Program Chair of many
national and international conferences and workshops. He is a member of the
IEEE Signal Processing (SP) and Circuits and Systems (CAS) societies.
Dr. Gabbouj was recipient of the 2012 Nokia Foundation Visiting Professor
Award, the 2005 Nokia Foundation Recognition Award, and many best-paper
awards. He has co-authored over 600 publications and supervised 40 PhD theses.

© 2016 by Taylor & Francis Group, LLC

i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page 1 — #17

i i

Chapter 1

Introduction

The title of this book immediately raises the following three questions:
1. Why digital filters?
Signal processing is one of the core areas of electrical engineer-
ing, emerging from advances central to the development of radio,
television, and other forms of electronic communication. One of
the key subject areas within this field is the design and appli-
cation of filters, devices that separate signal components (e.g.,
extracting the signal transmitted by the radio station you want
to listen to from others around it). With the advent and sub-
sequent miniaturization of computers, digital filter implementa-
tions became increasingly practical and widely applied.
2. Why nonlinear digital filters?
The theory of linear filters is well-developed and provides an
extremely powerful foundation for the analysis and design of
practical filters, both digital and analog. Not all filtering prob-
lems have adequate linear solutions, however: a case in point is
the data cleaning problem introduced in this chapter that mo-
tivates much of the work presented in this book. Because the
class of “nonlinear filters” is vastly more heterogeneous than the
class of “linear filters,” there is no general theory analogous to
that for linear filters. The primary objective of this book is to
present a collection of ideas and techniques that are useful in
the design and application of nonlinear digital filters.
3. Why Python?
Without question, the primary software environment used in
signal processing research is MATLAB, a commercial product

1
© 2016 by Taylor & Francis Group, LLC
i i

i i
 i i

“knitrMainBook” — 2015/8/20 — 17:01 — page 2 — #18

i i

2 CHAPTER 1. INTRODUCTION

that provides excellent support for computational linear alge-

bra, with a vast range of add-on packages that support many
signal processing tasks. Why, then, are we advocating Python
instead of MATLAB? A more detailed discussion is given in Sec.
1.4, but the short answer is that Python is an open-source lan-
guage, freely available to anyone, while MATLAB is a commer-
cial product that must be purchased. As a consequence, there is
significant interest in the scientific and engineering communities
in taking advantage of—and contributing to—the growing sup-
port for numerical computations in Python [94]. Second, unlike
MATLAB, which evolved from a linear algebra-centered start-
ing point, Python is a general purpose programming language,
useful in a much broader range of applications. Finally, because
of this broader focus, the Python language offers greater support
for reproducible research, an important idea discussed further in
Sec. 1.4.

1.1 Linear versus nonlinear filters: an example

Many of the central ideas of this book are illustrated with the following simple
example. Fig. 1.1 shows a sequence of 2048 successive measurements of an elec-
trocardiogram (ECG), an electrical signal measured in millivolts and sampled
180 times per second. This example illustrates the kinds of large-magnitude
“spikes” that can appear in a real data sequence, potentially obscuring other
important details. This particular data sequence is available as part of the ade4
add-on package [29] for the R software environment [99]. Like Python, R is an
open-source software package, developed to support a wide variety of statistical
and data analysis procedures and discussed further in Sec. 1.4. The ECG dataset
considered here is available as the object ecg, and the description available from
the R documentation notes that the signal exhibits a variety of biologically sig-
nificant components on different time scales, including a relatively long-term
baseline drift due to breathing artifacts, movement artifacts, and identifiable
components of the heart rhythm, including an occasional arrhythmia. The doc-
umentation also notes that this data sequence was provided by Gust Bardy and
Per Reinall of the University of Washington, and it cites the book by Percival
and Walden for further discussion [93].
Mathematically, the ECG signal shown in Fig. 1.1 corresponds to a finite
sequence of real numbers {xk }, where the sequence index k runs from a minimum
value of k = 1 to a maximum value of k = N where N = 2048. All of the digital
filters considered in this book correspond to a mapping of one sequence of length
N , say {xk }, into another sequence, say {yk }, also of length N . Most—but not
all—of these filters may be expressed as:

yk = F{xk } = Φ(xk−K , . . . , xk−1 , xk , xk+1 , . . . , xk+K ), (1.1)

for some nonnegative integer K and some function Φ(·) that maps 2K + 1

© 2016 by Taylor & Francis Group, LLC

i i

i i