Department of Computer Science

Series of Publications A
Report A-2009-4

Matrix Decomposition Methods

for Data Mining:
Computational Complexity
and Algorithms

Pauli Miettinen

Academic Dissertation

To be presented, with the permission of the Faculty of

Science of the University of Helsinki, for public criticism
in Auditorium XII, University Main Building, on 20
May 2009 at twelve o’clock noon.

University of Helsinki
Finland
Copyright © 2009 Pauli Miettinen

ISSN 1238-8645
ISBN 978-952-10-5497-6 (paperback)
ISBN 978-952-10-5498-3 (PDF)
http://ethesis.helsinki.fi/

Computing Reviews (1998) Classification: H.2.8, F.2.1, F.2.2

Helsinki University Print

Helsinki, May 2009 (164+6 pages)
Matrix Decomposition Methods for Data
Mining: Computational Complexity and
Algorithms
Pauli Miettinen
Department of Computer Science
P.O. Box 68, FI-00014 University of Helsinki, Finland
[email protected]
http://www.cs.helsinki.fi/pauli.miettinen/

Abstract
Matrix decompositions, where a given matrix is represented as a
product of two other matrices, are regularly used in data mining.
Most matrix decompositions have their roots in linear algebra, but
the needs of data mining are not always those of linear algebra. In
data mining one needs to have results that are interpretable – and
what is considered interpretable in data mining can be very different
to what is considered interpretable in linear algebra.
The purpose of this thesis is to study matrix decompositions
that directly address the issue of interpretability. An example is
a decomposition of binary matrices where the factor matrices are
assumed to be binary and the matrix multiplication is Boolean.
The restriction to binary factor matrices increases interpretability –
factor matrices are of the same type as the original matrix – and
allows the use of Boolean matrix multiplication, which is often more
intuitive than normal matrix multiplication with binary matrices.
Also several other decomposition methods are described, and
the computational complexity of computing them is studied to-
gether with the hardness of approximating the related optimization
problems. Based on these studies, algorithms for constructing the
decompositions are proposed.
Constructing the decompositions turns out to be computationally
hard, and the proposed algorithms are mostly based on various
heuristics. Nevertheless, the algorithms are shown to be capable of
finding good results in empirical experiments conducted with both
synthetic and real-world data.

iii
iv

Computing Reviews (1998) Categories and Subject Descriptors:

H.2.8 [Database Management]: Database
Applications—data mining
F.2.1 [Analysis of Algorithms and Problem Complexity]:
Numerical Algorithms and Problems—computations
on matrices
F.2.2 [Analysis of Algorithms and Problem Complexity]:
Nonnumerical Algorithms and
Problems—computations on discrete structures

General Terms: Algorithm, Experimentation, Theory

Additional Key Words and Phrases: Matrix Decompositions, Set

Cover, Binary Matrices, Column Decompositions,
Column–Row Decompositions
Acknowledgements

Behind every PhD thesis there is a student, and behind every student
there is a supervisor. I am grateful to my supervisor, Professor Heikki
Mannila, who managed to find such a good balance between the
seemingly contradicting requirements of a supervisor: he guided me
firmly through the process while still providing me with the liberty
to make my own decisions and mistakes.
The research in this thesis was done at hiit and at the Department
of Computer Science, University of Helsinki. I gratefully acknowledge
the financial support from them, as well as from Helsinki Graduate
School in Computer Science and Engineering.
But the place itself, buildings, infrastructure, means nothing
without the people. And I have been lucky to have such great
people around me. I thank all my co-authors for letting me work
with you and learn from you. Especially I shall never forget the
deep influence Aristides Gionis and Taneli Mielikäinen had on me
when I was starting my PhD studies. And I am forever grateful
to Professor Stefano Leonardi for giving me the possibility to visit
Rome and for his hospitality during that visit.
I am grateful to many friends and colleagues of mine for their
readiness to exchange ideas and dwell upon random topics: Esa,
with whom I have shared an office all these years, Niina, with whom
I have had many a pleasant discussion, Jussi, Jarkko, Jaana, and
Matti, I thank you all.
I owe much to my parents who taught me the importance of
education and who have always supported me.
But above all I must thank my beloved wife Anu. Her patience
was needed when I was writing this thesis; her impatience helped
me to finish it.

v
Contents

1 Introduction 1

2 Notation and Related Work 7

2.1 Notation and Initial Definitions . . . . . . . . . . . . 7
2.1.1 Notation . . . . . . . . . . . . . . . . . . . . . 7
2.1.2 A Word About Terminology . . . . . . . . . . 10
2.1.3 Decision and Optimization Problems . . . . . 10
2.1.4 Parameterized Complexity Theory . . . . . . 11
2.2 Related Work . . . . . . . . . . . . . . . . . . . . . . 15
2.2.1 Real-Valued Matrix Decompositions . . . . . 15
2.2.2 Decompositions for Binary-Valued Matrices . 16
2.2.3 Column and Column-Row Decompositions . . 18

3 The Positive–Negative Partial Set Cover and Basis

Usage Problems 21
3.1 Introduction and Problem Definitions . . . . . . . . 21
3.2 Connections to Data Mining . . . . . . . . . . . . . . 24
3.3 Previous Results . . . . . . . . . . . . . . . . . . . . 25
3.4 Computational Complexity of the ±psc Problem . . 26
3.4.1 Results . . . . . . . . . . . . . . . . . . . . . 26
3.4.2 The Exact-rbsc and Exact-±psc Problems . 28
3.4.3 From rbsc to ±psc . . . . . . . . . . . . . . 29
3.4.4 From ±psc to rbsc . . . . . . . . . . . . . . 30
3.4.5 Parameterized Complexity . . . . . . . . . . . 30
3.5 The ±psc and bu Problems Are Equivalent . . . . . 32
3.6 Algorithms for the Problems . . . . . . . . . . . . . 34
3.6.1 Peleg’s Algorithm . . . . . . . . . . . . . . . 34

vii
viii Contents

3.6.2 Iterative Algorithm . . . . . . . . . . . . . . . 35

3.6.3 An Integer-Programming Formulation . . . . 36
3.7 Experimental Evaluation . . . . . . . . . . . . . . . . 38
3.7.1 The Data Generation Process . . . . . . . . . 38
3.7.2 Results . . . . . . . . . . . . . . . . . . . . . 39
3.7.3 Conclusions . . . . . . . . . . . . . . . . . . . 41

4 The Boolean Matrix Factorization and Partition Prob-

lems 43
4.1 Problem Definitions . . . . . . . . . . . . . . . . . . 43
4.2 Boolean Decompositions as Data Mining Methods . 45
4.3 Matrix Ranks . . . . . . . . . . . . . . . . . . . . . . 48
4.3.1 Matrix Decompositions and Matrix Ranks . . 48
4.3.2 Relations Between Ranks . . . . . . . . . . . 50
4.4 Computational Complexity of the bmf Problem . . . 52
4.5 Computational Complexity of the bmp Problem . . . 55
4.6 Algorithms for bmf . . . . . . . . . . . . . . . . . . . 60
4.6.1 Finding the Basis . . . . . . . . . . . . . . . . 60
4.6.2 Variations on Finding the Basis . . . . . . . . 64
4.6.3 Using the Basis . . . . . . . . . . . . . . . . . 66
4.7 Algorithms for bmp . . . . . . . . . . . . . . . . . . . 66
4.8 Experimental Evaluation . . . . . . . . . . . . . . . . 68
4.8.1 Algorithms Used . . . . . . . . . . . . . . . . 68
4.8.2 Measuring the Error . . . . . . . . . . . . . . 69
4.8.3 Synthetic Data . . . . . . . . . . . . . . . . . 70
4.8.4 Real-World Data Sets . . . . . . . . . . . . . 72
4.8.5 Randomization of the Data . . . . . . . . . . 75
4.8.6 Results on Real-World Data . . . . . . . . . . 77

5 The Nonnegative Column and Column-Row Decom-

position Problems 87
5.1 Problem Definitions . . . . . . . . . . . . . . . . . . 87
5.1.1 Alternative Interpretation of cx and ncx . . 88
5.2 Motivation as Data-Mining Techniques . . . . . . . . 90
5.3 Properties of ncx and ncur Decompositions . . . . 91
5.4 Nonnegative Column Subset Selection Algorithms . . 92
5.4.1 Solving X and C̃ . . . . . . . . . . . . . . . . 95
5.4.2 Convergence of the Algorithms . . . . . . . . 96
5.4.3 Time Complexity . . . . . . . . . . . . . . . . 96
Contents ix

5.5 Algorithms for ncur Decompositions . . . . . . . . . 96

5.6 Experimental Evaluation . . . . . . . . . . . . . . . . 97
5.6.1 Algorithms Used . . . . . . . . . . . . . . . . 97
5.6.2 Synthetic ncx Data . . . . . . . . . . . . . . 99
5.6.3 Synthetic ncur Data . . . . . . . . . . . . . 102
5.6.4 Solving the cx Decomposition . . . . . . . . 104
5.6.5 Real-World Data . . . . . . . . . . . . . . . . 107
5.6.6 Conclusion of Experiments . . . . . . . . . . 114

6 The Boolean Column and Column-Row Decomposi-

tion Problems 115
6.1 Problem Definitions . . . . . . . . . . . . . . . . . . 115
6.2 Applications to Data Mining . . . . . . . . . . . . . 116
6.3 Computational Complexity of the bcx Problem . . . 116
6.4 The Mixing Matrix Problem . . . . . . . . . . . . . . 120
6.5 Algorithms for the bcx, bcur, and mm Problems . . 123
6.5.1 The LocBCX Algorithm . . . . . . . . . . . . . 124
6.5.2 The LocBCUR Algorithm and Solving the mm
Problem . . . . . . . . . . . . . . . . . . . . . 125
6.5.3 Time Complexity . . . . . . . . . . . . . . . . 126
6.6 Experimental Evaluation . . . . . . . . . . . . . . . . 128
6.6.1 Algorithms Used . . . . . . . . . . . . . . . . 128
6.6.2 Synthetic bcx Experiments . . . . . . . . . . 128
6.6.3 Synthetic bcur Experiments . . . . . . . . . 131
6.6.4 Experiments with Real-World Data . . . . . . 135

7 Conclusions 149

References 153

Appendix A Proof of Lemma 3.4 165

Appendix B Proof of Theorem 4.4 169

CHAPTER 1

Introduction

Data mining is about finding new and interesting information from

data [HK01, pp. 5ff.]. The underlying assumption is that there
is too much data for a human to interpret, and thus one needs
an automated method to find that new and interesting informa-
tion. What constitutes a new and interesting information is – if
not completely ill-defined – at least very subjective and data- and
application-dependent. This, of course, has not stopped data miners
from defining various metrics for the quality of the obtained in-
formation, if for no other reason, to facilitate the comparison of
various methods and their outputs. (See, e.g. [HK01, p. 27] for a
characterization of what makes a data mining result interesting, and
e.g. [GH06] for a survey of some metrics developed for measuring
the interestingness.)
The type of information sought can be coarsely divided into two
classes: local and global [HMS01, p. 9]. Local information is about
a specific (not necessarily small) area of the data, and does not
take the other parts of the data into account. Global information,
on the other hand, is about the data on the whole. It should be
noted that it might not be possible to distinguish between local and
global information by just examining the results of some data mining
methods; it is the process how the information was obtained that
defines the type, i.e. whether the whole data was taken into account
when the (parts of the) results were constructed. In the parlance of
data mining, it is typical to speak about local patterns and global
models (see e.g. [HMS01]) to further distinguish these two types.
This thesis considers only global information (i.e. models).

1
2 1 Introduction

Especially when global information is sought, the aspect of the

size of the results, i.e. the amount of information returned to the
user, becomes important (whereas, for local patterns, the optimum
result would contain exactly the interesting patterns – for whatever
measure of interestingness is used [HK01, p. 28]). If the volume of the
information returned by the data mining method used exceeds the
volume of the original information it is very unlikely that the results
ease the user’s burden on interpreting the data. Thus, a condensed
representation of the original data is sought. But while constructing
a global information about the input data the method used should
take into account the whole data, the resulting representation almost
never represents the whole data. Indeed, condensed representations
of the whole data are called (lossless) compressions of the data, and
that is not what data mining is about. Data mining is about finding
the most important aspects of the data, and reporting them to the
user.
A typical, and indeed natural, requirement for data mining
method’s results is that the user should be able to interpret them
in the context of the input data [HK01, p. 27] (indeed, Hand et al.
include the requirement of understandability in their very defini-
tion of data mining [HMS01, p. 1]). For example, a typical data
compression method is not a good data mining method, as the user
is probably unable to make any sense of the resulting bit string.
Another, less extreme example would be a user inputting student
information together with the courses the students have taken, and
getting as an output a set of high-dimensional real vectors. Espe-
cially if the possible values in these vectors are not restricted, the
results can be very hard to interpret. The interpretability of results
is an ongoing topic of this thesis and the principalis causa of the
methods presented.

Matrices are basic concepts of elementary linear algebra. In linear

algebra, an n-by-m matrix is usually interpreted as a linear map from
n-dimensional space to m-dimensional (see e.g. [GVL96, Chapter 2]).
But in data mining, and in this thesis, matrices are a convenient
way to store and manipulate data (e.g. storing text documents as
term frequency matrices [HK01, p. 430]). Matrix decompositions
also originate to linear algebra. In short, when performing a matrix
decomposition on some matrix, it is represented as a product of two
 3

or more factor matrices. Matrix decompositions have been developed

for many purposes. The purposes, stemming from problems in
linear algebra (e.g. solving linear systems [GVL96, p. 94]), are not
necessarily well-suited for a data mining task. Specifically, the
interpretability of the factor matrices is, at best, a by-product of
the definition of the decomposition and the application in hand.
This has motivated many new matrix decomposition definitions and
methods that aim at higher interpretability of the results.
This thesis is about new, interpretable matrix decompositions.
How is the interpretability ensured? All matrix decomposition
formulations presented in this thesis have certain characteristics in
common. First, it is assumed that the factor matrices are smaller
than the original matrix, although defining the actual size of the
factor matrices is left to the user. Second, the values of the factor
matrices are restricted to come from the same set of values where
the original data came from. That is to say, if the original matrix
was nonnegative, then so are the factor matrices; and if the original
matrix was binary-valued, then so are the factor matrices.
Other tools to increase interpretability apply only to some of the
proposed decompositions. With binary data, the matrix multiplica-
tion used is Boolean, that is, the addition is defined as 1 + 1 = 1.
This captures the intuition that when reconstructing the original
matrix by multiplying the factor matrices, the resulting matrix
should still be binary-valued. With normal matrix multiplication
this would be much harder to achieve.
Another idea is to restrict one factor matrix to a subset of
columns of the original matrix. Such decompositions are called
column decompositions, and it is easy to see that they preserve
some interpretability: if the columns of the original matrix were
interpretable, then so should be their subsets.

None of the above ideas are new, but many of their combinations are.
The contribution of this thesis is to formulate the new decomposi-
tions, analyse the computational complexity of constructing them,
and provide algorithms for them. The analysis of the problems
concentrates on the NP-hardness and on the hardness of approxim-
ability. Usually, the problems related to the decompositions are hard
even to approximate, and thus the proposed algorithms are based on
various heuristics. Algorithms’ performance and the interpretability
4 1 Introduction

of the results are studied in empirical experiments conducted with

synthetic and real-world data.
Experimental evaluation of the interpretability of results is not an
easy task – after all, interpretability is in the eye of the beholder. The
approach taken in this thesis is to take some real-world data sets that
present information accessible to non-experts, apply the proposed
algorithms to these data sets, and study the results. This approach
cannot guarantee that the algorithms will produce interpretable
results on other data sets – a goal that, without formal definition of
‘interpretability’, seems hard to obtain.
Scalability is one important characteristic of good data mining
algorithms. In this thesis scalability is addressed by examining the
asymptotic time complexity of the algorithms. This approach can
yield somewhat pessimistic results compared to what one could
achieve with real-world data, but it has the benefit of being immune
to implementation details.
The algorithms presented in this thesis are not optimized for
speed and scalability. Instead, the focus is on succinct and clear
representation of the ideas underlying the algorithms. Also, the
algorithms used in the experiments present a heterogeneous set of
proprietary third-party algorithms (such as those that come with
Matlab), algorithms implemented by their inventors, and algorithms
implemented by the author, all with varying implementation meth-
ods ranging from high-level languages such as Matlab to low-level
languages such as C. Hence, the empirical time complexity of the
proposed algorithms is not usually studied separately, although
if an algorithm was considerably slow (or exceptionally fast) it is
mentioned in the experiments.

Overview of the thesis. The first new matrix decomposition

formulations are given in Chapter 4, and the following two chapters
continue to introduce new formulations. The first chapter to contain
novel research, Chapter 3, does not discuss matrix decomposition
formulations, but its contents are vital background information for
Chapters 4 and 6. Chapter 2 contains preliminaries: notation, initial
definitions and related work. Last chapter, Chapter 7, contains
concluding remarks.

Relation to the author’s prior work. This thesis is mainly

based on four articles by the author, listed below. The notation and
 5

terminology have been unified, the connections between the articles

have been made more clear, and some missing details have been
supplied. Also all experiments with synthetic data, and most exper-
iments with real-world data, are new, and replace those presented
in the original publications.
The four articles on which this thesis is based are:
[HMT08] Saara Hyvönen, Pauli Miettinen, and Evimaria
Terzi. Interpretable nonnegative matrix decom-
positions. In Proc. 14th ACM SIGKDD Interna-
tional Conference on Knowledge Discovery and
Data Mining (KDD ’08), pages 345–353, 2008.
[Mie08a] Pauli Miettinen. The Boolean column and column-
row matrix decompositions. Data Mining and
Knowledge Discovery, 17:39–56, 2008.
[Mie08b] Pauli Miettinen. On the positive–negative partial
set cover problem. Information Processing Letters,
108:219–221, 2008.
[MMG+ 08] Pauli Miettinen, Taneli Mielikäinen, Aristides
Gionis, Gautam Das, and Heikki Mannila. The
discrete basis problem. IEEE Transactions on
Knowledge and Data Engineering, 20(10):1348–
1362, 2008.

The ±psc problem, discussed in Chapter 3, and results concern-

ing it, were presented in [Mie08b]. Most of the discussion about
the bu problem is new, although the problem itself was introduced
in [Mie06]. Also the proof of Lemma 3.4 is new. The new al-
gorithm presented in that chapter, IterX, was mentioned informally
in [MMG+ 08] and more formally explained in [Mie08a].
The contents of Chapter 4 are mostly from [MMG+ 08], but The-
orems 4.3 and 4.4 are new. Chapter 5 is based on [HMT08] with the
addition of LocNCUR algorithm. Chapter 6 is based on [Mie08a] with
the addition of Proposition 6.2, its proof, and LocBCUR algorithm.
CHAPTER 2

Notation and Related Work

Where we learn the notation used, and about the work related to this
thesis.

2.1 Notation and Initial Definitions

This section covers the basic notation used throughout this thesis.
More specific notation is introduced when it is encountered the
first time. This section contains also definitions for some high-level
concepts that are used in this thesis.

2.1.1 Notation

Matrices are used throughout this thesis, and they are set with
capital bold type letters, as in M = (mij ). Vectors are set with
lower-case bold type, v. The ith row vector of M is written as mi ,
while the jth column vector is written as mj . The (i, j)-element
of M is mij . Sometimes also (M)ij is used, especially when M is
a result of an operation, as in (N + P)ij . If I is a set of positive
integers, then MI is the submatrix of M containing the rows mi
for i ∈ I, and similarly MI is the submatrix of M containing the
columns mi for i ∈ I.
If S is a set and n and m are positive integers, then the set
of all n-by-m matrices taking values from S is S n×m . The sets
S commonly used for this purpose are the binary set {0, 1}, non-
negative integers Z≥0 , non-negative real numbers R≥0 , and the set

7
8 2 Notation and Related Work

of real numbers R. The notations Z≥0 and R≥0 are used instead
of the (more common) N and R+ to clear the ambiguity with the
latter notation, namely, whether 0 is in N and in R+ . Two special
matrices are used throughout this thesis: In , the n-dimensional
identity matrix and Jn×m , the n-by-m matrix full of 1s.
Let A and B be binary matrices of sizes n-by-k and k-by-m,
respectively (i.e. A ∈ {0, 1}n×k and B ∈ {0, 1}k×m ). The Boolean
matrix product of A and B is defined as normal matrix product, but
with the addition defined as 1 + 1 = 1. In other words, the algebra
is done over the Boolean semi-ring (0, 1, ∨, ∧). The Boolean matrix
product of A and B is denoted as A ◦ B. In the context of matrix
multiplication (either Boolean or normal), the matrices A and B
are referred to as factor matrices.
Sets of elements are set with normal capital letters while the ele-
ments are set with normal lower-case letters, as in S = {s1 , . . . , sn }.
A collection of sets is denoted by S = {S1 , . . . , Sm }. The cardinality
of a set (or collection) is denoted by |S|. If S = {S1 , . . . Sm } is a
collection, then [
∪S = S.
S∈S

The power set of a set S is written as P (S). A set system is a

pair (U, S), where U is called a universe or basis set and S is a
collection of subsets of U . Throughout this thesis the elements of
sets (and sets in collections) are supposed to have some arbitrary
ordering, and both sets and collections are assumed to be finite,
unless otherwise stated or clear from the context. In general, sets
are not assumed to contain multiple copies of same element (i.e.
they are not multi-sets), but collections may contain same sets many
times (or, as the sets are ordered, two sets containing exactly the
same elements can still be considered to be different).
Some special sets will have their own notation, some of which are
already defined (e.g. R and Z≥0 ). In general, the set {0, 1, 2, . . . , n}
for any n ∈ Z≥0 will be denoted by [n]. Two shorthands, poly(n)
and polylog(n), can also be considered as sets: the former is the set
of all polynomials of n and the latter is the set of all polylogarithms
of n (i.e. p(log n) for any polynomial p(x)). Alternatively, they can
be interpreted as anonymous functions from the respective sets.
Set operator arg max assumes a function and a set of elements
from the domain of the function and it returns the element that
2.1 Notation and Initial Definitions 9

maximizes the function over all elements in the input set. That
is, if f is a function of nonnegative integers, arg maxn∈[N ] f (n) =
arg max{f (n) : n ∈ [N ]} is the integer n ∈ {0, . . . , N } that max-
imizes f (n) over all nonnegative integers less than N + 1. The
counterpart to arg max is arg min that returns the element that
minimizes the function.
Sets and set systems are closely related to binary vectors and
matrices. Let (U, S) be a set system with U = {u1 , . . . , un }. The
incidence vector of a set S ∈ S is a binary n-dimensional column
vector v such that vi = 1 if and only if ui ∈ S. The incidence
matrix of (U, S) is the n-by-m matrix S with sj being the incidence
vector of Sj for 1 ≤ j ≤ m. If universe U is clear from context, it is
omitted and thus S is said to be the incidence matrix of S.
In addition to matrices and sets, also some functions are fre-
quently used. The indicator function 1(·) is one of them. If P is a
statement that is either true or false, then 1(P ) = 1 if P is true,
and 1(P ) = 0 if P is false.
The cost functions of optimization problems are a specific class
of functions. If Π is an optimization problem, I is an instance of Π,
and S is a feasible solution of I, then costΠ (I, S) denotes the cost
S incurs for I; the exact definition of cost is, of course, problem-
specific, but in general, cost functions are always nonnegative. When
instance I is clear from the context, it can be omitted. The optimum
cost of Π’s instance I is denoted by cost∗Π (I).
Two functions common in cost functions involved with matrices
are the Frobenius norm and sum of absolute values. If A = (aij ) is
an n-by-m matrix, the Frobenius norm dF (A) and sum of absolute
values d1 (A) are defined as

v
m
u n X m
n X
uX X
dF (A) = kAkF = t a2ij and d1 (A) = |aij | .
i=1 j=1 i=1 j=1

Neither of the functions places any restrictions over the values of A.

Note, however, that when A ∈ {0, 1}n×m , then dF (A)2 = d1 (A).
Thus, d1 is common with binary matrices, while dF is mostly used
with other types of matrices.
10 2 Notation and Related Work

2.1.2 A Word About Terminology

In data mining parlance, Boolean matrices are often referred to as

transaction databases, their columns being objects and rows being
transactions (or vice versa). Such terminology is not used in this
thesis, but this is only a matter of convention: there is nothing that
prevents one from considering the Boolean matrices of this thesis as
transaction databases.
Every matrix decomposition has three concepts related to it.
First of these is the formulation of the decomposition, that is, to
what kind of matrices the decomposition applies (e.g. only to non-
negative matrices or only to binary matrices), and what kind of
factor matrices are feasible for the decomposition (e.g. nonnegative
matrices or orthogonal matrices). Second concept is the concrete
decomposition of some matrix A. Third concept is the problem of
finding a decomposition that admits the formulation, given some
matrix A. Which of these concepts is meant is usually clear from
context, and thus for, say, Singular Value Decomposition (svd),
instead of always repeating ‘svd formulation’ (or decomposition,
or problem) simply ‘svd’ is used. Yet, to distinguish between a
decomposition and an algorithm for finding it, the names of al-
gorithms are set in monospaced typewriter font, as in SVD. Finally,
the term ‘decomposition’ (of a matrix) usually refers to an exact
decomposition (of a matrix). The focus in this thesis, however, is
on approximate decompositions, and therefore the term ‘decomposi-
tion’ usually refers to approximate decompositions; when an exact
decomposition is meant, it is explicitly mentioned.

2.1.3 Decision and Optimization Problems

The problems considered in this thesis are optimization problems,

specifically minimization problems. That is, the problems are of the
form ‘Given an instance I of Π, find a solution S of I such that S
minimizes costΠ (I, S).’ To study the computational complexity of
an optimization problem Π we need to consider the decision version
of Π. If Π is a minimization problem, its decision version, denoted
by d-Π, contains an extra variable t in its instance, and the problem
is of the form ‘Given an instance (I, t) of d-Π, is there a solution
S of I such that costΠ (I, S) ≤ t?’ Unless otherwise specified, all
2.1 Notation and Initial Definitions 11

decision versions of minimization problems studied in this thesis are

formed similarly.
The decision versions of almost every problem presented in this
thesis are NP-hard. Thus the approximation factor of the problem
is an important feature of its complexity. Let Π be a minimization
problem, and let A be an algorithm for it so that if I is an instance
of Π, then A(I) is the solution of I produced by A. Define the
approximation factor of A for problem Π to be a function f such
that for all I
costΠ (I, A(I))
≤ f (I). (2.1)
cost∗Π (I)
It is possible that cost∗Π (I) = 0. For such instances, define
x/0 = ∞ for x > 0 and 0 · ∞ = 0, and thus 0/0 = 0.
While the approximation factor is the standard measure for the
quality of an approximation algorithm, it is by no means the only one.
One important measure is the absolute approximation guarantee: an
algorithm A for problem Π has an absolute approximation guarantee
of c (for some constant c) if

|costΠ (I, A(I)) − cost∗Π (I)| ≤ c (2.2)

holds for all instances I of Π. The absolute approximation guarantee

is usually defined to require a constant bound (as is done above),
but we will also see a bound depending on the instance size.
The following terminology is used when the hardness of approx-
imation is studied. When an optimization problem is NP-hard to
approximate to within a factor of f (n), it means that unless P = NP,
there exists no polynomial-time approximation algorithm achieving
approximation factor of f (n); and when the problem is quasi-NP-
hard to approximate to within a factor of f (n), it means that unless
NP ⊆ DTIME(npolylog(n) ), there exists no polynomial-time approx-
imation algorithm achieving approximation factor of f (n). Similar
terminology is used in connection with absolute approximation
guarantees.

2.1.4 Parameterized Complexity Theory

Parameterized complexity theory, originating to the work of Downey

and Fellows [DF99], gives a framework to study computationally
12 2 Notation and Related Work

efficient ways of solving certain types of NP-hard problems. As the

name suggests, the problems studied in parameterized complexity
theory involve some parameter, the value of which can be inde-
pendent of the instance size. A parameterized problem instance is
a pair (I, k), where I is the original problem instance and k ∈ Z≥0
is the parameter. Parameterized problems are not optimization
problems, but decision problems: the task is to determine whether
certain condition involving I and k holds or not. However, many
optimization problems have natural parameterizations, and thus
parameterized complexity theory can give us useful information
about the computational complexity of the problem.
The theory classifies problems based on whether they can be
solved computationally efficiently if the parameter is assumed to be
fixed. This is captured in the notion of fixed-parameter tractability
(fpt for short).

Definition 2.1 ([FG06, Definition 1.4]). Let Π be a parameterized

problem, i.e. its instances are of the form (I, k).

1. An algorithm is an fpt-algorithm for a parameterized problem

Π if there is a computable function f : Z≥0 → Z≥0 and a
polynomial p such that for every parameterized instance (I, k)
of Π, the running time of the algorithm is at most

f (k)p(|I|).

2. A parameterized problem Π is fixed-parameter tractable if there

is an fpt-algorithm that decides it.

The class of all fixed-parameter tractable problems is denoted by

FPT.

The definition of fpt-algorithms does not restrict the function

f (k). This allows practically unfeasible problems to belong in class
FPT. An example of such a problem is the Set Basis problem
(sb) [GJ79, problem SP7], defined as follows.

Problem 2.1 (Set Basis, sb). Given a set system (U, C) and a
positive integer k, is there a collection B ⊆ P (U ) of at most k sets
(|B| ≤ k) such that for every set C ∈ C there is a subcollection
S
BC ⊆ B with B∈BC B = C?
2.1 Notation and Initial Definitions 13

The proof of the following proposition, that sb is in FPT, simul-

taneously illustrates some techniques to obtain a fixed-parameter
tractable algorithms and the possible infeasibility of such algorithms.

Proposition 2.1 ([DF99, Exercise 3.2.3]). The sb problem is in

class FPT.

Proof. Let n = |U | and m = |C|. The idea is to show that we can

bound n and m with a function depending only on k. We can then
use an exhaustive search to find B. Throughout the proof we must
assume that neither U nor C contains duplicates, and that no two
elements of U are contained in exactly the same sets C ∈ C. This
can be done without the loss of generality.
Notice first, that in a solution of sb, each set C ∈ C must be a
union of sets in B. There are at most 2k such subcollections for any
B with |B| ≤ k, and hence m ≤ 2k .
But there are no more than 2|C| different configurations on which
sets C an element u ∈ U belongs to. And because we assumed that
each element has a different configuration, there can be no more
k
than 2|C| ≤ 22 elements in U .
k
Therefore, if n > 22 or m > 2k , we know that there is no solution
of the instance, and otherwise, there are at most
!
2n 2k
≤ (2n )k ≤ 2k2
k

different collections B to try, and for each B, at most

m2k ≤ 2k 2k = 22k

different subcollections BC need to be tested. Thus, the function f

is  
k
2k k 22 +k
f (k) = 2k2 22k = 2 .

The above time complexity is, in fact, pessimistic, as, for example,
the subcollections BC can be constructed in a polynomial time (see
Lemma 3.5), yielding
2k
f (k) = 2k2 p(k),

where p(k) is a polynomial depending on k.

14 2 Notation and Related Work

While the above example unarguably shows the limitations of

the theory of fixed-parameter tractability, some celebrated results
– for example the O(1.2738k + kn)-time algorithm for the Vertex
Cover problem [CKX06] – have shown the strength of it.
As with standard complexity theory, reductions play an important
role in parameterized complexity theory. The reductions used in
parameterized complexity theory are called fpt-reductions.
Definition 2.2 ([FG06, Definition 2.1]). Let Π and Π′ be para-
meterized problems with instances (I, k) and (I ′ , k ′ ), respectively.
An fpt-reduction from Π to Π′ is pair of mappings (RI , Rk ) from
instances of Π to instances of Π′ such that:
1. For all (I, k), the pair (RI (I), Rk (k)) is a valid instance of Π′ .
2. RI and Rk are computable by an fpt-algorithm.
3. There is a computable function g : Z≥0 → Z≥0 such that
Rk (k) = k ′ ≤ g(k) for all valid parameterized instances (I, k)
of Π.
As could be expected, the class FPT is closed under fpt-reduc-
tions. The final piece of parameterized complexity theory we need
is the notion of W-hierarchy (see [DF99, FG06]), a hierarchy of
classes of problems not believed to be fixed-parameter tractable.
The classes in W-hierarchy are denoted by W[1], W[2], and so on,
the hierarchy being
FPT ⊆ W[1] ⊆ W[2] ⊆ · · · .
We will not need an explicit definition of the classes W[t] (which can
be found e.g. in [FG06]), but it will be sufficient to know problems
complete for each class W[t] under fpt-reductions. To define such
problems, we need the notion of t-normalized formulae, defined as
follows.
Definition 2.3 ([DF95]). A propositional formula is t-normalized
if it is of the form product-of-sums-of-products-of-. . . of literals with
V W V
t alternations (i.e. i j k · · · xijk··· ).
Hence, 2-normalized formula is in the conjunctive normal form.
The problems we need are the Weighted t-Normalized Satisfiabil-
ity problems. For t ≥ 2, Weighted t-Normalized Satisfiability is
complete for W[t] [DF95].
2.2 Related Work 15

Problem 2.2 (Weighted t-Normalized Satisfiability (wnst )). Given

a propositional formula ϕ in a t-normalized form and a nonnegative
integer k, does ϕ have a satisfying truth assignment of weight k,
where the weight of a truth assignment is the number of variables
set true?

Finally, we will need Monotone Weighted t-Normalized Satisfiab-

ility problems, defined as above, but requiring ϕ to be monotone,
that is, it cannot have negated variables. For even t, Monotone
wnst is complete for W[t] [DF95], and as Monotone wnst is clearly
fpt-reducible to Monotone wnst+1 , we have the following lemma.

Lemma 2.2. For even t, Monotone wnst+1 is W[t]-hard.

2.2 Related Work

This section introduces the prior work related to the themes discussed
in this thesis. The discussion in this section is more general, and some
of the more specific related works are discussed later in appropriate
places.

2.2.1 Real-Valued Matrix Decompositions

Probably the best-known method to decompose a matrix is the

Singular Value Decomposition (svd) [GVL96, p. 70]. It decomposes
a matrix A into the form UΣVT , where U and V are orthogonal
matrices, that is, UT U = VT V = I, and Σ is a diagonal matrix
with nonnegative entries—the singular values of A. The Singular
Value Decomposition gives the optimal rank-k approximation of the
matrix A with respect to the Frobenius norm.1 Matrix B is the
optimal rank-k approximation of A (with respect to the Frobenius
norm) if the rank of B is k, and for any other rank-k matrix C,
dF (A − C) ≥ dF (A − B). The optimal rank-k approximation of
A can be obtained from its Singular Value Decomposition simply
by setting all but the k largest singular values to 0. This result

1
In fact, the optimality of svd is not limited to the Frobenius norm, but
holds for any unitarily invariant norms (for definition of unitarily invariant norm
and proof of the claim, see [Mir60]).
16 2 Notation and Related Work

is sometimes referred to as the Eckart–Young Theorem after two

mathematicians who proposed it in 1936 [EY36], although the result
was established some twenty years earlier (see [Ste93]).
Computing the svd is also relatively fast: it can be done in time
O(min{nm2 , n2 m}) for n-by-m matrices [GVL96, p. 254]. The meth-
ods often employed in practice, such as Lanczos methods [GVL96,
pp. 470–507], are usually even faster. Nevertheless, for extremely
large matrices this can still be too much. This has motivated the
study of fast, approximate decompositions based on sampling the
matrix. Work done in this field include the results of Frieze et
al. [FKV04], Drineas et al. [DKM06a], and Achlioptas and McSh-
erry [AM07].
If matrix A is nonnegative (e.g. because it is a result of meas-
urements that can only yield nonnegative results), interpreting the
results of svd can be problematic, as the matrices U and V can
contain negative values. This problem is addressed by Nonnegative
Matrix Factorization (nmf), where the factor matrices are required
to have only nonnegative values. Early formulations of the nmf
problem include Paatero and Tapper’s [PT94], where they call it
‘positive matrix factorization’, and Cohen and Rothblum’s [CR93].
However, probably the most famous formulation is due to Lee and
Seung [LS99]. Since their article, the problem has attained a lot
of research and many algorithms are developed for it; for a recent
survey, see [BBL+ 07].
In addition to svd and nmf, many other matrix decomposition
methods have been proposed, most of which are based on probabil-
istic models. Such methods include multinomial Principal Compon-
ent Analysis (mpca) [Bun02], probabilistic Latent Semantic Index-
ing (plsi) [Hof99], and Latent Dirichlet Allocation (lda) [BNJ03].
There has been some research on expressing these decompositions
in a unified way (see Buntine and Jakulin [BJ06] and Singh and
Gordon [SG08]).

2.2.2 Decompositions for Binary-Valued Matrices

All of the aforementioned methods are designed for (possibly non-

negative) real- or integer-valued matrices. But the main focus of this
thesis is in decomposing binary-valued matrices, and some previously
proposed methods for that are given below.
2.2 Related Work 17

Some decomposition formulations for binary-valued matrices can

be found in Kim’s classical text-book [Kim82]. But as Kim’s focus
is on the existence of the exact decompositions, the results are not
always very useful in the context of this thesis.
Many decompositions designed for approximating a binary matrix
are probabilistic. They include, for example, aspect Bernoulli mod-
els [BKF09], topic models [SBM03], and logistic pca (lpca) [SSU03].
Logistic pca is perhaps the one that is most similar to the work
done in this thesis. It works as follows. Given an n-by-m matrix
(aij ), and an integer k, lpca finds n-by-k and k-by-m matrices U
and V. These matrices, however, are not binary. Instead, they
are real-valued, and their product defines an element-wise prob-
ability distribution for binary n-by-m matrices: The probability
Pr(aij = 1 | Θij ) that aij = 1 given the product Θ = UV is defined
to be σ(Θij ) = (1 + e−Θij )−1 , that is, the sigmoid (or logistic)
function of Θij .
Semi-discrete decomposition [OP83] lies between svd and the
Boolean decompositions studied here. Like svd, semi-discrete de-
composition decomposes a given matrix into three matrices, U, Σ,
and V, but the values in U and V are restricted to −1, 0, and
1 only. Semi-discrete decomposition was originally designed for
image compression [OP83], but has since been used, for example, in
information retrieval [KO98].
Also hierarchical descriptions of binary data have been studied:
the Proximus framework constructs a hierarchical clustering of
rows of a given binary matrix [KGR05] and hierarchical tiles are
probabilistic models hierarchically decomposing a binary matrix
into almost monochromatic binary submatrices [GMS04].
Tiling transaction databases (i.e. binary matrices) is another
line of related research [GGM04]. A tiling covers a given binary
matrix with a small number of submatrices full of 1s. The main
difference to Boolean decompositions is that no 0s can be covered in
a feasible tiling. Methods have been developed for finding also large
approximate tiles, for example fault-tolerant patterns [BPRB06] and
conjunctive clusters [MRS04], but obtaining an accurate description
of the whole dataset with a small number of approximate tiles has
not been studied previously explicitly.
In co-clustering (or bi-clustering) the goal is to cluster simul-
taneously both dimensions of a matrix [Har72]. A co-cluster is
18 2 Notation and Related Work

thus a tuple (R, C), R giving the indices of rows and C giving the
indices of columns. Decomposing a binary matrix into two matrices
can be seen as a co-clustering of binary data where the clusters
can overlap. The idea of co-clustering was originally proposed
by Hartigan in 1972 [Har72], but it has gained a lot of attention
recently, and many new methods have been proposed; see, for ex-
ample, [BDG+ 04, MO04, RF01].
The type of Boolean matrix factorization studied in this thesis
have gained very little attention hitherto. Nevertheless, Bělohlávek
and Vychodil [BV06] studied some properties of Boolean factoriza-
tion via formal concepts. Genetic algorithms and neural networks
have also been proposed to decompose a binary matrix using Boolean
decomposition. Snášel et al. [SPK+ 08, SKPH08] proposed and stud-
ied a genetic algorithm and a neural network concluding that, with
very simple data, the latter performed better than the former, but
with the cost of increased computational complexity.

2.2.3 Column and Column-Row Decompositions

The column and column-row matrix decompositions (respectively cx

and cur decompositions, for short) have been studied in computer
science and in numerical linear algebra, although in the latter field,
the terminology has often been somewhat different. The work
in numerical linear algebra is usually related to the qr matrix
decomposition [DMM07], where an n-by-m matrix A is represented
as A = QR, with Q ∈ Rn×n being orthogonal and R ∈ Rn×m being
upper triangular (see [GVL96] for more about qr decompositions).
For example, Stewart [Ste99, Ste05] provides algorithms to obtain a
sparse low-rank approximation of a large, sparse matrix. In practice,
these approximations are obtained via cx and cur decompositions.
In their seminal work, Gu and Eisenstat [GE96] proposed a
factorization called strong rank-revealing qr factorization, and gave
two algorithms to compute it. Their algorithms are efficient, taking
time O(mn2 ) (with m ≥ n) – the same time computing an svd takes.
For more information about the related work done in numerical
linear algebra, see [DMM07] and references therein.
The relative error of the algorithms by Stewart [Ste99, Ste05] (or
by Berry et al. [BPS05]) is not bounded. Gu and Eisenstat bound
the smallest singular value of C. In theoretical computer science,
2.2 Related Work 19

Frieze et al. [FKV04] proposed a probabilistic algorithm that makes

only two passes over matrix A and provides a bound

kA − CXkF ≤ kA − A∗k kF + ε kAkF ,

with A∗k being the best rank-k approximation of A. The number of

columns of C in Frieze et al.’s algorithm is polynomial to k, 1/ε, and
1/δ, with 1−δ being the success probability of the algorithm. Similar
results followed, see, for example, [DKM06a], reducing the number
of columns in C and analysing also the spectral norm. Related
results were also obtained for cur decompositions by Drineas et
al. [DKM06b], giving bound

kA − CURkξ ≤ kA − Ak kξ + ε kAkF

for ξ = 2, F , and with the number of columns in C and number of

rows in R depending polynomially on k, 1/ε, and 1/δ.
The additive error was removed by Drineas et al. [DMM07]
(see also [DMM06a] and [DMM06b]), by providing probabilistic
algorithms that, for A′ = CX and A′ = CUR, obtain an error
bounded by

A − A′ ≤ (1 + ε) kA − Ak kF .
F

Here Ak is again the best rank-k approximation of A, and 0 < ε ≤ 1.

The number of columns in C and rows in R depend on k, log(1/δ)
and 1/ε.
The best algorithm for cx decompositions where the number
of columns in C is constant (and user-defined) was recently given
by Boutsidis et al. [BMD08, BMD09], who provided a probabilistic
algorithm that returns a k-column matrix C such that
p
kA − CXkF ≤ O(k log k) kA − Ak kF

with probability at least 0.7.

CHAPTER 3

The Positive–Negative Partial

Set Cover and Basis Usage
Problems

Where we study a variant of the Set Cover problem and algorithms

for it and see how it all relates to the rest of the thesis.

3.1 Introduction and Problem Definitions

For a thesis about matrix decompositions, starting with a special
variant of the well-known Set Cover problem might seem strange.
The reason is that what is said here about sets and collections applies
to Boolean matrices and their factorizations via incidence matrices.
Particularly, the results – namely, upper and lower bounds for its
approximability – obtained here for the Positive–Negative Partial
Set Cover problem yield results about the Basis Usage problem,
which, in turn, appears frequently in subsequent chapters. Moreover,
the results about the Basis Usage problem provide a stepping-stone
for many other results.
Why not start with binary matrices (i.e. with the Basis Usage
problem), if that is what we are going to use? The answer is
twofold: it is easier to obtain the results via the intermediate step of
studying Positive–Negative Partial Set Cover, and even if they are
not much used in this thesis, these intermediate results have value
and applications of their own.

21
22 3 The ±PSC and BU Problems

That said, the main problem studied in this chapter, the Positive–
Negative Partial Set Cover problem is defined as follows.

Positive–Negative Partial Set Cover Problem (±psc). Given

a triple (N, P, Q), where N and P are disjoint sets and Q ⊆
P (N ∪ P ), find a subcollection D ⊆ Q that minimizes
cost±psc (N, P, D) = |P \ (∪D)| + |N ∩ (∪D)| . (3.1)

The elements in sets P and N are called positive and negative

elements, respectively. Hence, the goal of the problem is to cover
as much of the positive elements as possible and avoid covering the
negative elements; the quantity (3.1) is the number of uncovered
positive elements plus the number of covered negative ones. Denote
the cardinalities of the sets of the instance as follows:

|P | = π, |N | = ν, and |Q| = k.

Remark. The greedy algorithm achieves an O(log n) approximation

factor for the classical Set Cover problem [Joh74, Lov75, Chv79],
and this factor is optimal [LY94, RS97, Fei98, AMS06]. With ±psc
an analogous greedy algorithm would select the set Q ∈ Q with
the highest ratio |P ∩ Q| / |N ∩ Q| over the positive and negative
elements that are not yet covered by some of the sets selected earlier.
But this algorithm does not yield any useful approximation result.
To see that, consider the following instance of ±psc. There are 2π
positive elements p1 , . . . , pπ , p′1 , . . . , p′π and π + 2 negative elements
n1 , . . . , nπ , m, and m′ . Collection Q has two sets for each pair (pi , p′i )
of positive elements: Si and Si′ . Set Si contains pi , p′i and ni , while
set Si′ = {pi , p′i , m, m′ }. Clearly the optimal solution would be to
select all sets Si′ yielding cost 2, but the greedy algorithm, selecting
always the best-ratio set, selects all sets Si , yielding cost π.
A problem related to ±psc, and a crucial problem to study in
this chapter, is the Red–Blue Set Cover problem, due to Carr et
al. [CDKM00].

Problem 3.1 (Red–Blue Set Cover, rbsc). Given a triple (R, B, S),
where R and B are disjoint sets and S ⊆ P (R ∪ B), find a subcol-
lection C ⊆ S such that C covers B (i.e. B \ ∪C = ∅) and minimizes

costrbsc (R, B, C) = |R ∩ (∪C)| . (3.2)

3.1 Introduction and Problem Definitions 23

Analogously to ±psc, elements of R are called red elements and

elements of B are called blue elements. In rbsc the goal is to
cover all blue elements while covering as little of the red elements
as possible. The cardinalities of the sets are denoted as follows:

|R| = ρ, |B| = β, and |S| = σ.

Remark. The rbsc problem is clearly related to the ±psc problem,

the former being more restricted case of the latter. In the subsequent
sections we will see that these two problems are even more related
to each other than what is prima facie obvious. The relation
between rbsc and ±psc is not an atypical one – indeed, one can say
that ±psc is a prize-collecting version of rbsc. In prize-collecting
problems, one is asked to return a partial solution, instead of a
complete one, and the problem instance is endowed with prizes for
elements; the cost of the solution is the cost of the partial solution
to the original problem plus the sum of the prizes of the elements
not included in the partial solution. Examples of prize-collecting
problems include prize-collecting travelling salesman [Bal89] – where
the salesman does not have to visit all vertices, and the cost of the
solution is the length of the path plus the sum of the prizes of the
unvisited vertices – and prize-collecting Steiner tree [GW95] (defined
in a natural way). These problems enrich their original versions,
but this is not always true. For example, the prize-collecting set
cover problem (defined in a natural way) is redundant in the sense
that we can always reformulate an instance of it as an instance of
normal weighted set cover problem by identifying the prize of an
element as a singleton set with the weight of the prize.
The above problems, ±psc and rbsc, are the two main problems
considered in this chapter. However, outside this chapter they are
not used directly. Instead, in Section 3.5 we will see how the Basis
Usage problem, defined below, is linked to ±psc. In the subsequent
chapters, this chapter’s results are applied via it.
Basis Usage Problem (bu). Given matrices A ∈ {0, 1}n×m and
B ∈ {0, 1}n×k , find a matrix X ∈ {0, 1}k×m such that X minimizes

costbu (A, B, X) = d1 (A − B ◦ X). (3.3)

The Basis Usage problem asks for a half of a matrix decomposi-

tion: given the original matrix and the left factor matrix, what is the
24 3 The ±PSC and BU Problems

best possible right factor matrix. The matrix decomposition is not

a typical one – factor matrices are binary-valued, as is the original
one, and matrix multiplication is Boolean. This decomposition, the
Boolean Matrix Factorization, is studied in Chapter 4, where we
will also see some applications of this chapter’s results.

3.2 Connections to Data Mining

While the main purpose of this chapter is to build the stepping-
stones for the subsequent chapters, the problems studied here have
connections to and applications in data mining on their own.
The original motivation to rbsc was, according to Carr et
al. [CDKM00], a data mining question. The rbsc problem can
also be seen as a more machine-learning oriented question of ag-
gregating classifiers. Suppose we have a collection of partial (or
incomplete) classifiers over a training data, that is, a set of positive
(blue) and negative (red) elements. The classifiers are partial (or
incomplete) in the sense that they do not classify all elements in
the training data, but only a subset of them. They can also make
errors, and we view the results of such classifier as a set of elements
the classifier classifies as positive. This transforms naturally to an
instance of rbsc. Our goal is to aggregate these partial classifiers
to a single classifier that is complete, that is, it classifies all positive
elements as positive, and makes minimal error on classifying the
negative elements as positive. In other words, all blue points must be
covered, and the number of red points covered should be minimized.
The ±psc problem has a similar analogy to classifier aggregation.
With the same input, the goal is to minimize the number of false
negatives (positive points classified as negative) plus the number of
false positives (negative points classified as positive).
In addition to classifier aggregation, ±psc has other, perhaps
more indirect applications in data mining. For example, Afrati
et al. [AGM04] mentioned the negated version of ±psc, that is, a
problem with a goal to maximize the number of covered positive
elements minus the number of covered negative elements. Afrati et
al. noted the arbitrarily bad performance of the greedy algorithm for
solving the problem (cf. the remark above). They also mentioned
that being able to solve that problem could improve their algorithm;
3.3 Previous Results 25

in the light of this chapter’s results it seems improbable to obtain

improvements from that direction.

3.3 Previous Results

As mentioned before, rbsc is the problem upon which the results of

this chapter are built. It was introduced by Carr et al. [CDKM00]
who also gave two results about its hardness of approximation:

1. it is quasi-NP-hard  to approximate rbsc to within a factor of

(4 log σ) 1−ε
Ω 2 for any ε > 0; and

2. it is NP-hard  to approximate rbsc to within a factor of

1−δ
Ω 2 log β , with δ = (log log β)−c , for any constant c < 1/2.

The first result was independently proved by Elkin and Peleg [EP00],
and the second result was based upon a result by Dinur and
Safra [DS04].

Remark. The above results are somewhat unintuitive. How tight

are the bounds? The lower bounds are, essentially, of the form
1−ε
f (x) = 2log x , with the first result having a fixed ε > 0 and
the second result having ε = ε(x) → 0 as x → ∞. The func-
tion f (x), with fixed ε, is superpolylogarithmic and subpolynomial,
that is, f (x) grows asymptotically faster than any polylogarithm
of x, but slower than any polynomial of x (with standard notation
f (x) = ω(polylog(x)) and f (x) = o(poly(x))). Hence, the approx-
imation factor of any polynomial-time algorithm cannot have only
a polylogarithmic dependency on both σ and β.

The best upper bound for rbsc is due to Peleg [Pel07], who re-
√
cently presented a 2 σ log β-approximation algorithm for it. Peleg’s
upper bound is polylogarithmic with respect to β, but superpoly-
logarithmic with respect to σ. Also, notice that when σ = log β,
√
Peleg’s algorithm achieves 8 log β guarantee. Thus, in the family
of instances of rbsc where σ = log β, Peleg’s algorithm achieves
rather good approximation guarantees.
26 3 The ±PSC and BU Problems

3.4 Computational Complexity of the ±PSC

Problem
The results about the computational complexity of the ±psc problem
are given in the next subsection. The proofs of the results are mostly
postponed to Sections 3.4.2, 3.4.3, and 3.4.4. The parameterized
complexity of ±psc is studied in Section 3.4.5.

3.4.1 Results

Before heading to the main result of this chapter, the approximability

of ±psc, let us start by considering special cases of rbsc and ±psc.
In an earlier remark we saw that the greedy algorithm does not
work well for the ±psc problem, and it is easy to see that the same
example holds also in the case of rbsc. In the light of that remark,
it seems that much of the hardness of rbsc (and ±psc) lies in the
fact that red (or negative) elements can belong to multiple sets.
And indeed, this intuition is correct. For if no red element can be
contained in more than one set, we can identify the number of red
elements in each set as the weight of the set to obtain an instance
of the familiar weighted set cover problem. Thus, in such special
cases, we know that rbsc is no harder than the weighted set cover
(whereas, as we know, it is much harder in general cases). From
another remark we know that ±psc can be seen as prize-collecting
rbsc, and that prize-collecting set cover is as hard as weighted set
cover. Hence, we can conclude that in the special case where every
negative element belongs in exactly one set, the ±psc problem is as
hard as the weighted set cover problem.
When negative elements are allowed to belong to multiple sets, the
±psc problem becomes much harder. The main result of this chapter
relates the upper and lower bounds for the ±psc’s approximability
to the respective bounds for rbsc.

Theorem 3.1. ( i) rbsc can be approximated to within a factor

of f (ρ, β, σ) if ±psc can be approximated to within a factor of
f (ρ, β/ρmax , σ), where ρmax is the maximum number of red elements
in any set of the rbsc instance. ( ii) ±psc can be approximated to
within a factor of g(ν + π, π, k + π) if rbsc can be approximated to
within a factor of g(ν, π, k).
3.4 Computational Complexity of the ±PSC Problem 27

What this result shows is that we can sandwich the hardness

of approximating ±psc between the hardness of approximating
rbsc with different instances. Hence, whatever results we have
with respect to rbsc, the same results can be applied to ±psc.
Theorem 3.1 and the previous results provide the following two
corollaries.
Corollary 3.2. For any family of instances of ±psc where ν, π,
and k are polynomially related, and for any ε > 0,
1. it is quasi-NP-hard to approximate the ±psc problem to within
1−ε
a factor of Ω(2(4 log k) ); and

2. it is NP-hard to approximate the ±psc problem to within a

1−ε
factor of Ω(2log π ).
Corollary 3.3. There exists a polynomial-time approximation al-
gorithm
p
for the ±psc problem that achieves an approximation factor
of 2 (k + π) log π.
The first part of Corollary 3.2 follows from Theorem 3.1 by
the result of Carr et al. [CDKM00], while the algorithm in Corol-
lary 3.3 is the Peleg’s algorithm [Pel07]. These are straightforward
consequences of Theorem 3.1.
The second part of Corollary 3.2 requires more careful study. It
follows from a result by Dinur and Safra [DS04] applied to rbsc:
there exists a family of instances of rbsc where
 −c′

log1−(log log β) β c′
ρmax = O 2 (log log β)

for some constant c′ < 1/2, and unless P = NP there are no

polynomial-time approximation algorithms for it with an approxim-
1−(log log β)−c
ation factor of 2log β for any constant c < 1/2. Thus, if
1−(log log x)−c
we let gc (x) =2log x
for all c < 1/2, then assuming that
P 6= NP, there exists no polynomial-time approximation algorithm
to ±psc achieving an approximation factor of
 
π
gc . (3.4)
O(gc′ (π)(log log π)c′ )
It remains to bound the growth of (3.4) from below. This is done
in the following lemma, proof of which is postponed to Appendix A.
28 3 The ±PSC and BU Problems

Lemma 3.4. Let gc be defined as above. Then

 
π 1−ε
gc = Ω(2log π
) (3.5)
O(gc′ (π)(log log π)c′ )

for all c, c′ < 1/2 and ε > 0.

Theorem 3.1 is proved in Sections 3.4.3 and 3.4.4, while Sec-

tion 3.4.5 studies the parameterized complexity of ±psc. But before
going further, let us study two special cases of rbsc and ±psc in
Section 3.4.2.

3.4.2 The Exact-RBSC and Exact-±PSC Problems

In this section we will study simple special cases of rbsc and ±psc
called exact-rbsc and exact-±psc. These problems, as the name
suggest, ask for a collection C (resp. D) such that no red element is
covered (resp. all positive and no negative elements are covered), or
answer ‘no’ if no such collection exists.
Why are these problems interesting? Assume that it would be
NP-hard to decide whether the answer for, say, exact-±psc is ‘no’
or not. It would then follow, that no polynomial-time algorithm
could achieve any polynomially computable approximation factor
for ±psc (unless P = NP, of course). To formalize this idea,
consider the problem exact-Π asking, for an instance I, to return
a solution S such that costΠ (I, S) = 0 or ‘no’ if no such S exists;
problem Π is defined in an obvious way. Let the (decision version
of the) exact-Π problem be NP-hard. The claim is, that unless
P = NP, no polynomial-time algorithm can approximate Π to within
any polynomially computable function f (|I|). For a contradiction,
assume that it is possible, and that A achieves an approximation
factor of r(|I|). Let I be an instance of Π for which the solution
of exact-Π is other than ‘no’, and let A(I) be a solution of it.
Thus, costΠ (I, A(I))/cost∗Π (I) ≤ r(|I|). But because the solution of
exact-Π was not ‘no’, it must be that cost∗Π (I) = 0 and hence also
costΠ (I, A(I)) must be 0, meaning that A must always find an exact
solution if one exists. A contradiction, as exact-Π was assumed to
be NP-hard.
Luckily both exact-rbsc and exact-±psc have simple polynomial-
time algorithms.
3.4 Computational Complexity of the ±PSC Problem 29

Lemma 3.5. There exist polynomial-time algorithms for exact-rbsc

and exact-±psc.
Proof. The algorithms are similar: select all sets containing no red
(resp. negative) elements, and if all blue (resp. positive) elements
are covered, then – and only then – an optimal solution with zero
cost is found. Otherwise return ‘no’.

It is to be understood that henceforth all instances of rbsc and

±psc are such that the cost of their optimal solution is at least 1.

3.4.3 From RBSC to ±PSC

Consider an instance (R, B, S) of rbsc. We will map this instance
to an instance of ±psc. Let each negative element ni correspond
to exactly one red element ri . For each blue element bi , create
ρmax = maxS∈S |R ∩ S| positive elements pij , for j = 1, . . . , ρmax .
For each set S ∈ S create a set Q ∈ Q such that ni ∈ Q if ri ∈ S
and that, for j = 1, . . . , ρmax , pij ∈ Q if bi ∈ S.
Let D be a solution of this instance of ±psc. If D covers all
positive elements, then the corresponding subsets also cover all
blue elements in rbsc, and D is a feasible solution of (R, B, S).
Moreover, cost±psc (D) = costrbsc (D), i.e. D induces same costs in
both problems. If, on the other hand, there exists a positive element
pij not covered by D, then there must be at least ρmax positive
elements pij not covered by D. Thus we can add any set S with
pij ∈ S to D without increasing the cost of the solution, as we cannot
cover more than ρmax negative elements with any S ∈ S. If C is
the (possibly extended) solution to rbsc induced by D, we see that
cost±psc (D) ≥ costrbsc (C).
Finally, it is clear that the optimal solution of a ±psc instance
will cover exactly the negative elements corresponding to the red
elements covered by the optimal solution of rbsc, i.e. the costs of
the optimal solutions are equal. Hence,
cost±psc (P, N, Q, D) costrbsc (R, B, S, C)
∗ ≥ .
cost±psc (P, N, Q) cost∗rbsc (R, B, S)
Therefore, if we can approximate ±psc to within a factor of
f (ρ, β/ρmax , σ), then we can approximate rbsc to within a factor of
f (ρ, β, σ). This concludes the proof of the first part of Theorem 3.1.
30 3 The ±PSC and BU Problems

3.4.4 From ±PSC to RBSC

Consider an instance (N, P, Q) of ±psc. For each ni ∈ N , let there

be a red element ri− ∈ R, and for each pi ∈ P , let there be a blue
element bi ∈ B and a red element ri+ ∈ R. For each set Qj ∈ Q,
let there be a set Sj+ ∈ S and for each positive element pi ∈ P , let
there be a set Si− ∈ S. Define these sets as

Sj+ = {rk− | nk ∈ Qj } ∪ {bk | pk ∈ Qj } and

Si− = {ri+ , bi }.

Let C be a solution of the thus created rbsc instance. Create

D, a solution of the ±psc instance, by adding each Qj to D if the
corresponding set Sj+ is in C.
To show that this reduction preserves the approximability, we
start by considering the cost of D. First, let nk be a negative element
in ∪D. That is, there is a set Qj so that nk ∈ Qj and Qj ∈ D.
But this means that the corresponding set Sj+ must be in C, and
therefore the red element rk− corresponding to nk is in ∪C.
Second, let pk be a positive element that is not in ∪D, so none of
the sets Qj that contain pk are in D. This means that none of the sets
Sj+ that contain bk are in C. But as bk must be covered by C, it follows
that Sk− is in C, and so rk+ is in ∪C. Hence cost±psc (D) ≤ costrbsc (C).
Consider then D∗ , the optimal solution of (N, P, Q). We show
that the cost of the optimal solution of the rbsc instance created
from the ±psc instance is at most that of D∗ . Create C so that Sj+
is in C if Qj ∈ D∗ . For all blue elements bi not yet covered by C, add
Si− to C. It is straightforward to see that cost∗±psc = cost±psc (D∗ ) =
costrbsc (C) ≥ cost∗rbsc . Therefore

costrbsc (C) cost±psc (D)

≥ ,
∗
costrbsc cost∗±psc

so that if we can approximate rbsc to within a factor of g(ν, π, k),

then we can approximate ±psc to within a factor of g(ν +π, π, k +π).

3.4.5 Parameterized Complexity

Denote the parameterized versions of ±psc and rbsc by p-±psc

and p-rbsc. For both problems, the parameter we consider here is
3.4 Computational Complexity of the ±PSC Problem 31

the cost of the solution. First we need to know the parameterized

complexity of p-rbsc.
Lemma 3.6. The p-rbsc problem is W[2]-hard.
Proof. Carr et al. [CDKM00] showed a trivial reduction of rbsc
from and to Monotone wns3 (Problem 2.2, which they called Min-
imum Monotone Satisfying Assignment of level 3): Each red element
corresponds to a variable. The innermost conjunctions correspond
to sets in S; a variable appears in a conjunction if the element
corresponding to it belongs to the set corresponding the conjunction.
There is one disjunction of conjunctions for each blue element, and
a conjunction (set) appears in a disjunction if the blue element
appears in the set. Finally, the disjunctions of conjunctions are
joined with conjunctions. An example of this reduction is given
below. The reduction is clearly an fpt-reduction, and thus, together
with Lemma 2.2, proves the claim.

Example 3.1. Consider the following instance of rbsc: There are

two blue elements, b1 and b2 , five red elements, r1 , r2 , r3 , r4 , and r5 ,
and four sets S1 = {r1 , b1 }, S2 = {r2 , r3 , b1 }, S3 = {r3 , r4 , b2 }, and
S4 = {r5 , b2 }. An optimal solution uses sets S1 and S4 with cost of
2.
To build a monotone 3-normalized formula we start by identifying
the red elements as variables; thus we have 5 variables x1 , . . . , x5 .
Next we build 4 conjunctions of variables, corresponding to the
sets: a variable xi appears in conjunction Cj if ri ∈ Sj . Hence
the conjunctions are: C1 = x1 , C2 = x2 ∧ x3 , C3 = x3 ∧ x4 , and
C4 = x5 . Then we build disjunctions of Ci s, corresponding to blue
elements: Ci is in disjunction Dj if bj ∈ Si . The disjunctions are:
D1 = C1 ∨ C2 and D2 = C3 ∨ C4 . Finally, the formula ϕ is the
conjunction of D1 and D2 , that is,
ϕ = D1 ∧ D2 = (C1 ∨ C2 ) ∧ (C3 ∨ C4 )



= (x1 ) ∨ (x2 ∧ x3 ) ∧ (x3 ∧ x4 ) ∨ (x5 ) .
To satisfy ϕ, we need to satisfy both D1 and D2 . To satisfy D1 ,
we need to satisfy either C1 or C2 , and similarly for D2 . We can
satisfy C1 by assigning x1 to true, and satisfy C4 (to satisfy D2 )
by assigning x5 to true. This satisfies ϕ. Thus satisfying D1 by
satisfying C1 corresponds to covering b1 by S1 ; assigning x1 to true
is the cost induced by this selection. 3
32 3 The ±PSC and BU Problems

The reductions from rbsc to ±psc (Section 3.4.3) and from

±psc to rbsc (Section 3.4.4) can be used as the RI part of the
respective fpt-reductions. For Rk we can use identity function in
both cases. For function g from part 3 of Definition 2.2 we can
also use the identity function. It remains to show that there is a
solution to the original instance of rbsc (resp. ±psc) with cost at
most t if and only if there is a solution to the reduced instance of
±psc (resp. rbsc) with a cost at most t. But in the reduction from
rbsc to ±psc the costs of the optimal solutions are equal, and in
the reduction from ±psc to rbsc the cost of the optimal solution
to rbsc is at most the cost of the optimal solution to ±psc. This
proves that both reductions are indeed fpt-reductions, and gives rise
to the following proposition.

Proposition 3.7. When the parameter is the cost of the solution,

the p-±psc problem is equivalent to the p-rbsc problem under fpt-
reductions; especially, the p-±psc problem is W[2]-hard.

The cost of the solution is a natural parameterization for the

±psc problem. However, it is not the only possible parameterization.
Another alternative would be to take k, the number of sets in Q, as
the parameter (yet another alternatives include π and ν). It may
not look very intuitive to parameterize with respect to the instance
size, but notice that the cardinality of |P ∪ N | must also be taken
into account when computing the instance size for ±psc. Indeed,
we will shortly see that k can be a very intuitive parameterization
when the problem is viewed from a different perspective.

Lemma 3.8. The p-±psc problem is in FPT when the parameter

is |Q| = k.

Proof. There are only 2k possible subcollections D, and evaluating

the cost of each of them takes only polynomial time.

3.5 The ±PSC and BU Problems Are

Equivalent
We can now prove the above-mentioned connection between ±psc
and bu. Consider first a special case of bu where the instance
3.5 The ±PSC and BU Problems Are Equivalent 33

consists of an n-dimensional binary column vector a and a n-by-k

Boolean matrix B (i.e. matrix A has only one column). Thus the
task is to find a Boolean k-dimensional column vector x minimizing

d1 (a − B ◦ x) = ka − B ◦ xk1 , (3.6)

where k·k1 is the Hamming distance. It is easy to see that this

problem is exactly ±psc rephrased using matrices: identify the
matrix B as the incidence matrix of Q and set ai = 1 if the ith
element is positive, and ai = 0 if it is negative (hence, n = π + ν).
Vector x defines the collection D: the jth set of Q is in D if and only
if xj = 1. But this special case does not limit the generality of the bu
problem too much. To see that, notice that in bu each column of X
can be considered independently, and thus solving bu is equivalent to
solving the above problem once for each column of A. We can (and
hereafter will) assume that n is polynomially related to m, and this
repeated computing will only cause a polynomial slowdown. As for
the approximation, approximating multiple independent instances
is as hard or easy as approximating a single instance. Thus we have
the following proposition.

Proposition 3.9. The results about the computational complexity

of ±psc hold for bu up to polynomial factors; the same is also
true for parameterized complexity given that the parameters are
corresponding. The results about the approximability of ±psc hold
for bu.

Care must be taken when one transforms the upper and lower
bounds for the approximability of ±psc to the respective bounds
for bu, as some of the parameters that are natural for the former
problem are not natural for the latter one. Specifically, the size of
the collection S transforms to the number of columns in B, and the
number of positive elements in the instance, |P |, corresponds to the
number of 1s in A’s columns, and therefore, to the density of A.
Those are the two characteristics that determine the complexity of
bu; there are no known results bounding the approximability of it
using directly the number of rows in A.
Notice that this result also shows a clear difference between
normal linear algebra and Boolean arithmetic. If one replaces
the Boolean matrix product in (3.6) with normal matrix product,
34 3 The ±PSC and BU Problems

removes the requirement that x is binary, and optimizes with respect

to the L2 -norm, then the problem is solvable in polynomial time
using standard least-squares methods, for example, via svd [GVL96].
The problem stays in P even if we place linear constraints on the
feasible set of xs (e.g. we require x to be nonnegative), as it can be
solved using convex quadratic programming [KTH79].
Finally, note that we can also solve bu exactly in time O(2k knm)
using Lemma 3.8 and the independence of columns of X: there are
2k different column vectors x, it takes O(nk) time to compute (3.6)
for one column a when x is fixed, and there are m columns in total.

3.6 Algorithms for the Problems

Any algorithm solving the rbsc problem can be used to solve the
±psc problem (and vice versa), and any algorithm solving the ±psc
problem can be used to solve the bu problem, given the results we
have obtained so far. But the bu problem is their main framework
in this thesis, and hence the following discussion concentrates on
that problem.

3.6.1 Peleg’s Algorithm

Recall that given the reduction from Section 3.4.4 we can use Peleg’s
√
2pσ log β-approximation algorithm for rbsc [Pel07] to obtain a
2 (k + π) log π-approximation algorithm for ±psc (Corollary 3.3).
Peleg’s algorithm, PelegRB, works as follows: it discards sets that
have too many red elements, lets the weights of the remaining sets
to be the number of red elements each set contain, removes the
red elements from the sets, and solves the resulting weighted set
cover problem using the standard greedy algorithm. Peleg [Pel07]
does not give any method to compute the correct threshold for the
number of red elements used to discard the sets; instead, all values
from 1 to ρ are tried.
Peleg’s algorithm must be restarted to compute each column of
X. The parameter ρ corresponds to the number of 0s in the column
of A corresponding to the column of X currently constructed. Thus,
the time complexity of PelegRB is O(mzkSC(n, k)), where z is the
number of 0s in A and SC(n, k) is the time it takes to solve any
3.6 Algorithms for the Problems 35

instance of weighted Set Cover with n elements and k sets in the

input.
The approximation factor of Peleg’s algorithm, when used to
solve an instance of ±psc, depends on two characteristics of the
instance: k, the number of sets in Q, and π, the number of positive
elements. As mentioned above, π corresponds to the number of 1s
in A’s columns in the framework of the bu problem. Hence, the
denser the matrix, the worse the algorithm is expected to perform.

3.6.2 Iterative Algorithm

The next algorithm is based on the idea of iteratively updating
a greedy solution. It is explained as an algorithm for bu – as
that is its main usage in this thesis – but is equivalently a ±psc
algorithm. The algorithm, called IterX, does not have any proved
approximation guarantees but, at least in the experiments done for
this thesis (Section 3.7.2), it usually outperforms PelegRB.
The algorithm can be divided into two parts: greedy selection
and iterative update using the same greedy selection. For each
column aj of A, the greedy selection starts by considering the first
column of B, b1 , and if b1 covers more 1s in aj than it covers 0s, b1
is used to cover aj (i.e. x1j is set to 1). Then second column of B is
considered, but this time only those elements of aj are considered
that were not covered by b1 (given that b1 was used to cover aj , that
is). Again, if b2 covers more uncovered 1s than it covers uncovered
0s, x2j is set to 1. This procedure is repeated for each column of B,
always considering only those 1s and 0s of aj that were not covered
by any of the previously-considered columns of B.
The above procedure can be expressed using the following func-
tion cover:
cover(A, B, X, w) = w |{(i, j) : aij = 1 ∧ (B ◦ X)ij = 1}|
(3.7)
− |{(i, j) : aij = 0 ∧ (B ◦ X)ij = 1}| .

The procedure starts with X full of 0s and proceeds to update X

row-by-row. Notice that, as the columns of A are independent,
ith row of X can be computed at once (i.e. the algorithm can first
compute x1 , then x2 , and so on).
The purpose of the variable w is to weight the covering of 1s
against the covering of 0s. This can be used to achieve more intuitive
36 3 The ±PSC and BU Problems

Algorithm 1 The IterX algorithm for bu and ±psc.

Input: Matrices A ∈ {0, 1}n×m and B ∈ {0, 1}n×k , weight para-
meter w ∈ R≥0 .
Output: Matrix X ∈ {0, 1}k×m approximately minimizing d1 (A −
B ◦ X).
1: function IterX(A, B, w)
2: X ← (xij ) with xij = 0 for 1 ≤ i ≤ k, 1 ≤ j ≤ m
3: repeat
4: for i = 1, . . . , k do
5: xi ← arg maxxi ∈{0,1}m cover(A, B, X, w)
6: end for
7: until d1 (A − B ◦ X) does not decrease
8: return X
9: end function

results with very sparse datasets, as we will see in Section 4.8.6, but
for now we can assume that w = 1.
In the iterative part the algorithm proceeds as above, but this
time the matrix X is not empty, but contains the initial version of X.
When considering row i of X, it is first set to all-zero and then a xi
maximizing cover(A, B, X, w) given all other rows of X is computed
as above. The iteration is repeated until the reconstruction error
does not decrease anymore.
The algorithm IterX is given in whole as Algorithm 1.

3.6.3 An Integer-Programming Formulation

Lu et al. [LVA08] present an integer-programming (IP) formulation

for bu. Thus, any general-purpose IP-solver can be used to solve bu.
Unfortunately, the IP formulation presented by Lu et al. contains
an error (Haibing Lu, personal communication). The error, however,
is not hard to fix (idem), and the correct version is presented below.
The problem instance contains matrices A = (aij ) and B. Index i
runs from 1 to n, index j runs from 1 to m, and index t runs from
1 to k (number of columns in B). The value of M is to be defined
3.6 Algorithms for the Problems 37

later. The correct IP formulation is the following:

n X
X m
minimize uij
i=1 j=1

subject to
(BX)ij + uij ≥ 1 if aij = 1
(BX)ij − u′ij = 0 if aij = 0 (IP1)
M uij − u′ij ≥0
uij − u′ij ≤ 0
u′ij ∈ Z≥0
uij , xtj ∈ {0, 1}.

The formulation has three sets of variables, xij , uij , and u′ij . The
first two are binary, and the last is nonnegative. Variables uij (and,
indirectly, u′ij ) are used to count the induced error.
The intuition of the IP formulation is the following. The first
inequality makes sure that (BX)ij ≥ 1 for those i and j for which
aij = 1. If this is not the case, error (i.e. uij ) is increased by 1.
Because (B ◦ X)ij = 0 if and only if (BX)ij = 0, this inequality
counts the number of uncovered 1s.
The next equation is for covered 0s. Again the aim is to distin-
guish between (BX)ij = 0 and (BX)ij = 6 0, but this time the error
is not directly related to the value of (BX)ij . If (BX)ij ≥ 1 and
aij = 0, then u′ij is positive. If u′ij is positive, the actual error uij is
increased by 1, and if u′ij = 0, then no error is induced and uij = 0.
This relation between uij s and u′ij s is described in third and fourth
inequalities.
The third inequality guarantees that uij = 1 if u′ij ≥ 1. For that,
the value of M needs to be at least the maximum value of u′ij . As
the maximum value of (BX)ij = k, letting M ≥ k is sufficient. The
purpose of the fourth inequality is to guarantee that when u′ij = 0,
then also uij = 0.
To be able to solve (IP1) efficiently would be very advantageous
as that would mean one could solve bu optimally. Alas, it does not
seem probable, in general, to be able to do that (given the results
of this chapter), and a small experiment in Section 3.7.2 further
strengthens this assumption.
38 3 The ±PSC and BU Problems

3.7 Experimental Evaluation

The purpose of the experimental evaluation is to study the two
algorithms, IterX and PelegRB. Recall that while the latter has
a provable approximation accuracy, the former can, in principle,
perform arbitrarily badly. The aim is also to study the effects that
different characteristics of the data have on the performance of the
algorithms.

3.7.1 The Data Generation Process

To reliably achieve the stated goals, the experiments were performed

using synthetic data, enabling total control over the parameters.
The setting of the experiments was that of bu, that is, the problem
instances contained two matrices, A and B, and the task was to
find X to minimize costbu (A, B, X). This setting is more valid for
the subsequent chapters, and was thus used instead of the ±psc
setting.
The studied parameters were (1 ) the number of columns in B;
(2 ) the noise level; (3 ) the density of the columns in B; and (4 ) the
mean number of columns of B used to create each of the columns
of the data. The first parameter controls the complexity of the
data: the more columns in B, the more variety there will be in A’s
columns. It is also a parameter in PelegRB’s approximation factor.
The third and fourth parameters both change the overall density of
the data, which was the other parameter in the said approximation
factor.
All matrices A were of size 150-by-80. They were created as
follows. First, a random matrix B was created, using the above
parameters. Then, a random matrix X was created. The number
of 1s in X’s columns is, on expectation, the value of the fourth
parameter above. A preliminary version of matrix A was constructed
as A = B ◦ X. Finally, noise was introduced by changing the values
of A’s elements randomly, the probability of change being the defined
noise level.
Each of the aforementioned parameters was varied in turn and
20 matrices were created for each parameter combination. The
reported results are mean values over these 20 matrices. The default
values for the parameters, used when they were not varied, were:
3.7 Experimental Evaluation 39

IterX IterX
3500 PelegRB 5000 PelegRB

3000 4000
d1(A−B°X)

d1(A−B°X)
2500 3000

2000 2000

1000
1500
0
8 10 12 14 16 18 20 22 24 26 28 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
k noise level
(a) (b)
Figure 3.1: Reconstruction errors of bu decompositions when (a)
k, the number of columns in B, varies; and (b) noise level varies.
Markers are mean values over twenty instances, and the width of
error bars is two times the standard deviation.

(1 ) 16 columns in B; (2 ) 10% of the bits were reverted due to noise;

(3 ) the density of the columns of B was 10%; and (4 ) approximately
4 columns of B were combined to create a column of A.

3.7.2 Results

Number of columns in B. The number of columns in B, k,

varied from 8 to 28 with steps of size 2. The total density of the
resulting matrices was approximately 0.35.
The results of this experiment can be seen in Figure 3.1(a). The
IterX algorithm is clearly superior to PelegRB, both in terms of
error and deviation over different instances. Somewhat surprisingly,
the overall trend of PelegRB seems to be towards better results
when k increases, although the approximation factor of PelegRB
depends on k. The trend is, however, very weak.
Noise level. The level of noise varied from 0 to 0.4 with steps of
size 0.05. The results are presented in Figure 3.1(b). The results are
intuitive as the cost induced by both algorithms increases together
with the noise. Again, however, IterX is constantly better than
PelegRB. When no noise is present, both algorithms return the exact
solution. This was assumed, as when there is no noise, the problem
is equivalent to exact-±psc (see Section 3.4.2).
Introducing 5% of noise has dramatic effects to PelegRB’s per-
formance, whereas IterX’s performance degenerates more smoothly.
40 3 The ±PSC and BU Problems

IterX
7000 PelegRB IterX
4500 PelegRB

6000 4000
d1(A−B°X)

d1(A−B°X)
5000 3500

4000 3000
2500
3000
2000
2000
1500
0.05 0.1 0.15 0.2 0.25 0.3 4 6 8
density mean number of base vectors per column of A

(a) (b)
Figure 3.2: Reconstruction errors of bu decompositions when (a)
density of B’s columns varies; and (b) the mean of B’s columns used
to create a column of A (i.e. 1s in X’s columns) varies. Markers are
mean values over twenty instances, and the width of error bars is
two times the standard deviation.

For both algorithms, the standard deviation is very small, being

often negligible.
Density of B’s columns. The mean density (number of 1s
divided by the total number of elements) of B’s columns varied from
0.05 to 0.3. The total density of the resulting matrices varied from
approximately 0.2 to approximately 0.7. The results can be seen in
Figure 3.2(a).
Density is the other variable in PelegRB’s approximation factor,
and it indeed has effects to the algorithm’s performance. IterX, on
the other hand, shows virtually no decrease of performance when
the density increases.
Mean number of columns of B used for each column of A.
The mean number of columns of B involved in the combination
to create A had three possible values, 4, 6, and 8. The resulting
matrices had an approximate total density between 0.35 and 0.55.
The increased total density has, again, clear and adverse effects
to PelegRB’s results, as can be seen in Figure 3.2(b). And, again,
IterX’s performance is about the same in all experiments.
Solving BU optimally. There are two ways to solve bu optim-
ally. One could either use the exhaustive search (see Section 3.5) or
the IP formulation by Lu et al. [LVA08] (Section 3.6.3). To decide
which method should be used, their empirical time complexity was
studied. It is clear that the exhaustive search will become imprac-
3.7 Experimental Evaluation 41

tical for any larger values of k, but how about the IP formulation?
The exhaustive search was implemented using C programming lan-
guage and the IP solver used was the GNU Linear Programming
Kit1 . The data used was the same data that was used to study
the effects of noise. The programs were executed on a dual-core
2.66GHz PC running Linux. All reported times are wall clock times.
The first test was executed with a data having no noise. Thus,
the value of k was 16. The exhaustive search performed very well.
With no noise, the time it took was 3.4 seconds, and with 5% of
noise, it took 11.5 seconds. Solving the IP was considerably slower:
with no noise, it took 118.6 seconds (almost 2 minutes), and with
5% of noise, it did not even finish after more than 4 days. The
test was repeated with another matrix having same characteristics,
and the results were same. The huge difference is explained by the
fact that when there is an exact decomposition, the variables uij
of (IP1) are all 0 even in the linear relaxation of (IP1), and one
optimal solution of the relaxation is achieved when xij s are binary.
Hence, an optimal solution to the linear relaxation is also an optimal
solution to (IP1). This, of course, is not necessarily true when there
is no exact decomposition present.
While this small experiment is by no means conclusive, it makes
a strong point in favour of the exhaustive search – at least with
moderate values of k. Thus, in the rest of this thesis, when the bu
problem needs to be solved exactly, the exhaustive search is used.

3.7.3 Conclusions

Overall, the IterX algorithm seems to be superior to PelegRB,

showing small variation over different instances and robustness to
many characteristics of the data that have an adverse effect to
PelegRB. Why cannot PelegRB do better? It is hard to identify any
single reason, but one major factor seems to be the density of the
matrix: PelegRB’s approximation guarantee depends on it, according
to the experiments, for a good reason. The IterX algorithm will be
the method of choice for the future experiments, although the lack
of approximation guarantees for IterX means that also PelegRB’s
performance will be studied in the same settings.

1
http://www.gnu.org/software/glpk/
CHAPTER 4

The Boolean Matrix

Factorization and Partition
Problems

Where we shall study the bmf and bmp problems, and see why they
are interesting and how they relate to matrix ranks. The connections
of bmp to well-known problems are revealed and algorithms for bmf
are proposed. The algorithms are studied in empirical experiments.

4.1 Problem Definitions

The Boolean Matrix Factorization problem complements the Basis
Usage problem to a complete matrix decomposition: when bu
asks for one factor matrix, given the other, the Boolean Matrix
Factorization problem asks for both factor matrices. The problem
is defined as follows.
Boolean Matrix Factorization Problem (bmf). Given a binary
matrix A ∈ {0, 1}n×m and a positive integer k, find binary matrices
B ∈ {0, 1}n×k and X ∈ {0, 1}k×m such that they minimize
costbmf (A, B, X) = d1 (A − B ◦ X). (4.1)
The columns of the matrix B are called basis vectors. This is,
of course, only a matter of convention, as there is nothing special
about the columns of B compared to the rows of X, and the roles
of these two matrices could as well be exchanged.

43
44 4 The BMF and BMP Problems

As mentioned, the definition of bmf contains, in some sense, that

of bu. Indeed, the bmf problem can be split into two subproblems:
to find the left factor matrix, B, and to find the right factor matrix,
X, given the left factor matrix. The latter is the bu problem, the
former has been referred to as the Discrete Basis Problem [MMG+ 06,
Mie06]1 .

Discrete Basis Problem (db). Given a binary matrix A ∈

{0, 1}n×m and a positive integer k, find a binary matrix B ∈
{0, 1}n×k such that there exists a binary matrix X ∈ {0, 1}k×m
that, together with B, minimizes

costdb (A, B, X) = d1 (A − B ◦ X). (4.2)

That is, a feasible solution of an instance of db does not contain

the matrix X. Solving db without solving bu (to find X) is not
very meaningful, and the focus in this thesis is on their combination,
bmf. The bu problem is, as said, a subproblem of bmf, but it is not
immediately clear that bmf is harder than – or even as hard as – bu.
The complexity of bu is based on the fact that, for a given A and
B, it is hard to find a good X; however, that does not imply that
for a given A it would be hard to find B and X. While not being
a direct implication of the definitions, the bmf problem, indeed, is
harder to approximate than the bu problem (see Section 4.4).
It should be noted here that the abbreviation bmf is also used for
a related decomposition: the Binary Matrix Factorization [ZLDZ07].
The difference is that Binary Matrix Factorization uses normal mat-
rix multiplication, not the Boolean one. These two decompositions
should not be confused with each other.
We also study another version of bmf, called the Boolean Matrix
Partitioning problem:

Boolean Matrix Partition Problem (bmp). Given a binary

matrix A ∈ {0, 1}n×m and a positive integer k, find binary matrices
P and X such that there is exactly one 1 in each row of P and
matrices P and X minimize

costbmp (A, P, X) = d1 (A − P ◦ X). (4.3)

1
[MMG+ 08], on the other hand, refers to the bmf problem as the db problem.
4.2 Boolean Decompositions as Data Mining Methods 45

The only difference between bmf and bmp is the extra constraint
imposed on P: there must be exactly one 1 in each row of P.
Matrices fulfilling this constraint are called partition matrices as
they are incidence matrices of set systems that are partitions (see
example below). Hence the name of the problem.

Example 4.1. Suppose we have a set of four elements, S =

{s1 , . . . , s4 }, and we partition it into partition P of two sets P1 =
{s1 , s3 } and P2 = {s2 , s4 }. The corresponding partition matrix P
is then  
1 0
0 1
 
P= .
1 0
0 1
Similarly, given P we can easily construct the partition P. 3

Why is bmp an interesting problem? Its decompositions do not

have smaller reconstruction error (indeed, it is easy to see that
cost∗bmf (A) ≤ cost∗bmp (A) for all matrices A), nor is there any direct
reason why it should increase the interpretability of the results (but
neither is there any direct reason why it should not). The bmp
problem is introduced and studied because it turns out to be a much
easier problem than bmf, and because it has an interesting relation
to another well known problem, and this relation hopefully helps us
better understand the computational aspects of bmf.

4.2 Boolean Decompositions as Data Mining

Methods
Matrix decompositions, in general, are regularly applied in data
mining, but why use Boolean decomposition? Why would the results
of bmf be more appealing than, say, the results of svd? For a simple
motivation consider the following example.

Example 4.2. Consider three CS students, say, X, Y, and Z. Sup-

pose that X is interested in the Systems specialization area, and that
everybody in that area needs to study courses Operating Systems
and Programming Languages. Student Z is interested in the Soft-
ware specialization area, requiring Z to take courses Programming
46 4 The BMF and BMP Problems

Languages and Compilers. Student Y is interested in combining

both areas, and thus needs to take all three courses (among others).
These three students and their mandatory courses can be expressed
as a binary matrix as follows:
 
1 1 0
 
A = 1 1 1 students .
0 1 1
courses

Solving bmf on A with k = 2 yields

 
1 0 !
  1 1 0
B = 1 1 and X = .
0 1 1
0 1

This is a natural decomposition: columns of B represent the spe-

cialization areas and the rows of X tell which student has chosen
which area. It is also exact, that is, A = B ◦ X. 3
The crux of the Boolean decomposition lies in the following
fact: using normal matrix multiplication we cannot achieve the
previous example’s result (see the next section). Then what makes
the Boolean decomposition so natural in the above example? A
set of courses taken by a student is indeed a set: it cannot contain
duplicates. Thus a natural addition operation to the sets of courses
is the set union operation, and this is exactly what the Boolean
matrix multiplication does. Hence, when the data represents true
presence/absence information, a Boolean decomposition is well-
motivated. As discussed in the next section, it can also achieve better
results than the methods based on normal matrix multiplication.
The relatively general framework of bmf decompositions has
already been shown to apply to a more specific data-mining problem.
After the initial publication of the problem [MMG+ 06], Vaidya et
al. [VAG07] showed how a version of the role mining problem can
be reformulated as the bmf problem.
Finally, the idea of saving space deserves some attention. One
motivation that is sometimes given for matrix decompositions is
that when the inner dimension k is much smaller than n or m, then
the factor matrices can be considered as a compressed form of the
original matrix and storing only the factor matrices (either in disk,
4.2 Boolean Decompositions as Data Mining Methods 47

or in RAM) should take less space than storing the original matrix
(see e.g. [LS99, BBL+ 07, DKM06a, DKM06b]). This motivation,
however, raises two questions: what are we going to do with the
approximate representation of the matrix and whether the factor
matrices indeed take less space than the original matrix. The first
question is rarely addressed, although it seems to be very important.
The decomposition of a matrix saves only some aspects of the
data, namely those expressed by the factor matrices, and to yield
any savings in the space, it must discard some other aspects. An
accurate approximation, say, with respect to the Frobenius norm,
is sometimes all we want, but there is no general reason why this
should be true for all uses of the data. Therefore, the question of
whether the aspects we have saved are those we are going to need is
absolutely crucial – for we do not want to throw the baby out with
the bath water!
That said, let us turn our attention to the second question,
that is, whether the factor matrices actually take less space than
the original one. This question is usually studied in the special
case of sparse matrices: if the original matrix is sparse, can one
guarantee that also the factor matrices will be sparse. For if the
factor matrices of a sparse matrix can be dense (as is the case with
svd), it is questionable how much space one can save by keeping
only the factor matrices. But sparsity is not the only characteristic
of the data matrix one should consider. For if we know that the
original matrix assumes values from some finite, fixed set, we can
exploit this knowledge in order to store the matrix in smaller space
(see [KO98, KO00] for similar ideas).
This brings us back to the binary matrices and Boolean decom-
positions. If our original matrix is binary and the factor matrices
assume real values, it does not seem probable that we can save any
space by using the decomposition instead of the original matrix.
Thus, assuming we can settle the first question, it seems reasonable
to require that binary matrices are decomposed into binary factor
matrices, and better still, sparse binary matrices are decomposed
into sparse binary factor matrices. These goals are achieved by a
(good) bmf decomposition.
The factor matrices of bmf are binary by definition, but to see
that they are sparse for sparse matrices requires some thinking.
Consider a binary matrix A and its factor matrices B and X. The
48 4 The BMF and BMP Problems
W
product B ◦ X can be re-written as ki=1 bi xi , where ∨ denotes the
Boolean sum. Now, if any of the matrices bi xi in the sum are dense,
so is the sum. And the number of 1s in bi xi (for binary vectors
bi and xi ) is the number of 1s in bi times the number of 1s in xi .
Hence, if the matrices B and X are dense, then so is their product,
and if A was sparse, that product cannot be a good approximation
of A (nothing, of course, prevents bad approximations of sparse
matrices being dense).
Apart from previous paragraphs, the issue of saving storage
space by storing only the factor matrices is not discussed in this
thesis. Further discussion would require a specific use for which the
decomposition could be used instead of the original matrix, more
formal treatment of the vague terms like ‘sparse’ and ‘dense’, and
an evaluation of different methods for storing the matrices. All this
is considered to be out of the scope of this thesis.

4.3 Matrix Ranks

Matrix rank is a basic concept in linear algebra. It is usually defined

to be the number of linearly independent columns (or rows) of the
matrix, or, equivalently, as the dimension of the image of the linear
map defined by the matrix. These definitions are not very intuitive
in the context of this thesis: the matrices considered here are not well
interpret as linear maps. Nevertheless, there are other definitions
of matrix rank, equivalent in the case of real-valued matrices and
better suited in our purposes.

4.3.1 Matrix Decompositions and Matrix Ranks

Recall the setting of Example 4.2. That is, A is a 3-by-3 binary

matrix, where the rows represent the students X, Y , and Z, the
columns represent the courses Operating Systems, Programming
Languages, and Compilers, and aij = 1 denotes that student i has
taken course j.
 
1 1 0
 
A = 1 1 1 .
0 1 1
4.3 Matrix Ranks 49

Using Boolean matrix decomposition, the matrix A can be ex-

pressed exactly as
   
1 1 0 1 0 !
    1 1 0
A = 1 1 1 = 1 1 ◦ = B ◦ X.
0 1 1
0 1 1 0 1

The same is not possible with any method based on normal matrix
multiplication. Consider, for example, the rank-2 singular value
decomposition of A:
   
0.50 0.71 ! 0.50 0.71
  2.41 0  
U = 0.71 0 Σ = V = 0.71 0 .
0 1
0.50 −0.71 0.50 −0.71

To gain some intuition

√ about the values in U and V, one can notice
that 0.71 ≈ 1/ 2.
Compare the column vectors of B and U: the former present a
clear intuition about the data while the latter are arguably hard to
interpret. The Boolean decomposition was exact, but the rank of
matrix A is 3, and the approximation of A produced by svd with
rank-2 decomposition is
 
1.10 0.85 0.10
UΣVT = 0.85
 
1.21 0.85 .
0.10 0.85 1.10

By the optimality of svd, this is the best that can be achieved by

looking at real matrices of rank 2 and squared error.
Optimality for arbitrary matrices is not the whole story, however.
For binary matrices, one can study different types of ranks. The real
rank rankR (A) of a binary matrix A is simply the smallest value
of k such that A = BX with an n-by-k matrix B, a k-by-m matrix
X, and using normal matrix multiplication. The nonnegative rank
rankR≥0 (A) of A is similar to the real rank, but the factor matrices
are restricted to have only nonnegative values. The nonnegative
integer rank rankZ≥0 (A) of A further restricts the factor matrices
B and X to only contain nonnegative integers. Finally, the Boolean
rank rankB (A) of A is the smallest k such that A = B ◦ X, where
B is an n-by-k binary matrix, X is a k-by-m binary matrix, and
the matrix multiplication is Boolean.
50 4 The BMF and BMP Problems

The Boolean rank is thus the smallest k such that there exists a
bmf decomposition with zero error, the real rank is the smallest k
such that there exists an svd decomposition with zero error, and
the nonnegative rank is the smallest k such that there exists an
nmf decomposition with zero error. For bmp, it follows from the
requirement of the matrix X being a partition, that the normal
arithmetic could be used in the matrix multiplication: all of the
summations involved in the matrix multiplication can have at most
one non-zero term. Thus, the nonnegative integer rank is a lower
bound for k such that we can solve bmp with zero error. It is,
however, only a lower bound: the requirement that X is a partition
is a sufficient, yet not necessary, condition to make sure that the
product BX is a binary matrix.
The concepts of real and Boolean ranks discuss the exact repres-
entation of the matrix A, but often only an approximate representa-
tion is sought. One could define the e-ranks rankeR (A), rankeR≥0 (A),
rankeZ≥0 (A), and rankeB (A) to be the least integer k such that there
exists a rank-k matrix B for which kA − Bk ≤ e. For example, the
rankeB (A) of an n-by-m binary matrix A is the smallest k such that
d1 (A − B ◦ X) ≤ e with B ∈ {0, 1}n×k and X ∈ {0, 1}k×m .

4.3.2 Relations Between Ranks

There have been some studies with respect to the relations of the
aforementioned ranks (see [MPR95, GP83]). In particular, the
following inequalities hold for the nonnegative rank of a binary
matrix A [GP83]:

rankR (A) ≤ rankR≥0 (A) and (4.4)

rankB (A) ≤ rankR≥0 (A). (4.5)

Similar inequalities hold also for the nonnegative integer rank of

a binary matrix A [GP83]:

rankR (A) ≤ rankZ≥0 (A) and (4.6)

rankB (A) ≤ rankZ≥0 (A). (4.7)

Inequality (4.6) follows because both ranks use the same arithmetic,
and (4.7) follows because the factor matrices are binary in both
cases.
4.3 Matrix Ranks 51

Between the real and Boolean ranks there are no clear relations. It
can be shown that there are binary matrices A for which rankR (A) <
rankB (A) and vice versa [MPR95]. The complement of the identity
matrix of size n-by-n is an example where rankB (A) = O(log n),
but rankR (A) = n [MPR95]. This shows that while svd can use the
space of reals, bmf can, at least in some cases, take advantage of
the properties of Boolean operations to achieve much smaller rank
than svd. Thus it is not a priori obvious that svd will produce
more concise representations than the Boolean methods.
Computing the real rank is easy (excluding the precision issues),
and can be done, for example, using svd: the real rank of a matrix is
the number of its non-zero singular values [GVL96, p. 71]. Comput-
ing the Boolean rank, on the other hand, is NP-complete: identifying
a binary matrix as an adjacency matrix of some bipartite graph G,
the Boolean rank of that matrix is exactly the number of complete
bipartite subgraphs needed to cover all edges of G [MPR95]. This
problem, covering by complete bipartite subgraphs, is well known
to be NP-complete [GJ79, problem GT18].
Approximating Boolean rank is also hard. It was shown by
Simon [Sim90] to be as hard to approximate as the problem of
partitioning a graph into cliques [GJ79, problem GT15] (equivalently,
as hard to approximate as the minimum chromatic number). Yet,
there exist some upper and lower bounds for the Boolean rank,
and the relation between the Boolean and real ranks is known in
some special cases; see, for example, [MPR95] and references therein.
Finally, the problem “Given A and k, is rankB (A) ≤ k?” is fixed-
parameter tractable with parameter k (see Section 2.1.4) [FG06].
In the case of e-ranks, inequality (4.5) does not hold, but inequal-
ity (4.4) does hold. For nonnegative integer rank, the upper bound
property of course carries over in both cases, i.e.,
rankeR (A) ≤ rankeZ≥0 (A) and (4.8)
rankeB (A) ≤ rankeZ≥0 (A). (4.9)
In other words, knowing that one can solve bmp with parameter
k and error e, one knows that the same error is attainable in bmf
with some parameter k ′ ≤ k, or with the same parameter, the error
e′ ≤ e can be achieved. Similar results also hold for the real-valued
decompositions. Otherwise, even less seems to be known about
e-ranks than about exact ranks.
52 4 The BMF and BMP Problems

4.4 Computational Complexity of the BMF

Problem
As the bmf problem contains the bu problem, we might, at first
sight, think that bmf must be at least as hard as bu. This turns
out to be right, but the hardness of bmf does not seem to be a
direct consequence of the hardness of bu. Recall that proving the
hardness of bu requires selecting the matrix B adversely. This is not
possible with bmf, because it is the algorithm that gets to select the
matrix B. It is, however, probably possible to construct a reduction
from bu to bmf proving some level of relation between the two (cf.
Section 6.3).
Instead of having a reduction between bmf and bu, we will see a
reduction between bmf and the Set Basis problem (sb), defined as
Problem 2.1 in Section 2.1.4.
The bmf problem is an optimization problem: find the matrix
decomposition into k basis vectors that minimizes the representation
error. We formulate the decision version of it, the d-bmf problem,
defined as explained in Section 2.1.3. The proof that d-bmf is
NP-complete proceeds by showing that sb is a special case of d-bmf,
and that d-bmf is in NP.

Theorem 4.1. The d-bmf problem is NP-complete.

Proof. Recall that an instance of sb is a set system (U, C) and an

integer k, and the question is, is there a collection B of k subsets of
U such that each set of C can be represented as the union of some
sets of B. An instance of d-bmf contains a binary matrix A and
nonnegative integers k and t. The question of d-bmf is, is there a
bmf decomposition of A of size k that has error at most t.
We shall prove that for any instance of sb there is an equivalent
instance of d-bmf with t = 0. Let (U, C) be an instance of sb, and
let C be the incidence matrix of C. It is straight forward to see
that sb now reduces to the problem asking ‘Is there a binary matrix
B such that each column of C can be represented as a Boolean
combination of B’s columns?’ That is, are there binary matrices B
and X such that C = B ◦ X. This shows that d-bmf is NP-hard.
Finally, it immediately follows that d-bmf is in NP, because the
sizes of B and X are polynomial in the size of C.
4.4 Computational Complexity of the BMF Problem 53

Notice that in the above proof the hardness of d-bmf was solely
due to the problem of finding at least one matrix of the decomposition
(i.e. B in the above proof). Finding the other factor matrix when
the other one is given and no error is allowed, that is, solving
the exact-bu problem, is easy (straight forward consequence of
Proposition 3.9, showing exact-bu equivalent to exact-±psc, and
Lemma 3.5).
The reduction from sb to bmf with t = 0 shows that (the decision
version of) exact-bmf is NP-hard and hence implies the following
inapproximability result (see Section 3.4.2).

Theorem 4.2. The bmf problem cannot be approximated to within

any polynomially-computable factor in polynomial time, unless P =
NP.

Note that solving sb is equivalent to computing the Boolean rank:

the result of sb with parameter k is ‘yes’ if, and only if, the Boolean
rank of the matrix is at most k.
The non-existence of an approximation factor to bmf seems to be
– at least partly – due to its definition and the fact that its optimal
solution can have zero error. Thus, asking how many times we need
to multiply the optimal error to get our approximative answer might
not be meaningful in the first place. An alternative would be to
ask whether we can approximate bmf to within some absolute error.
Unfortunately, that is also a hard problem. Nevertheless, before
formulating the theorem, notice that we certainly can approximate
bmf to within an absolute error of n2 (if A ∈ {0, 1}n×n ): no
approximation, no matter how bad, can cause more error than that.

Theorem 4.3. The bmf problem cannot be approximated to within

any constant absolute approximation error in polynomial time, un-
less P = NP.

Proof. For a contradiction, let us assume that A is a polynomial-

time algorithm for bmf and c is a constant such that for every n and
m, for every A ∈ {0, 1}n×m , and for every k ∈ Z≥0 , k < min{n, m},

costbmf (A, A(A, k)) ≤ cost∗bmf (A, k) + c. (4.10)

We construct
 a new instance
 of bmf, A′ , by copying A c + 1
times, A = A A · · · A . Hence, A′ has n rows and (c + 1)m
′
54 4 The BMF and BMP Problems

columns. The parameter k is kept the same. The optimal solution

of (A′ , k) decomposes A optimally, and copies this decomposition
c + 1 times, yielding

cost∗ (A′ , k) = (c + 1)cost∗ (A, k). (4.11)

To see that (4.11) indeed holds, notice first that copying the
optimal decomposition of A gives the right-hand side of (4.11). On
the other hand, if the optimal decomposition yields smaller error
than that, then the average cost of approximating a copy of A must
be smaller than cost∗ (A, k) – but this is a contradiction, and hence
we have established (4.11).
Together (4.11) and (4.10) give us

cost(A′ , A(A′ , k)) ≤ (c + 1)cost∗ (A, k) + c. (4.12)

As the value of k is intact, A must, essentially, solve the original

instance (A, k) c + 1 times. To attain the bound (4.12), the average
error caused by the repeated instances (A, k) cannot be more than
the right-hand side of (4.12) divided by c + 1. Hence, there must
be at least one copy of A, denoted by A0 , such that
 
(c + 1)cost∗ (A, k) + c
cost(A0 , A(A0 , k)) ≤
c+1
 
c (4.13)
= cost∗ (A, k) +
c+1
∗
= cost (A, k),
which is a contradiction, as solving bmf optimally is an NP-hard
problem (Theorem 4.1).

It is, in fact, possible to prove a stronger version of Theorem 4.3

where c is no more a constant, but is allowed to depend on m, n or k.
When c depends only on k and c(k) is a polynomial on k, the above
construction works as it is – the value of k is not changed. Notice,
that c(k) is polynomially bounded by n and m, as we can assume
that k < min{n, m}. This bound is important as the construction
increases the instance size by c(k) + 1.
When c depends on n (or m), that is, c = c(n), we have the
aforementioned upper bound for the growth rate of c(n): c(n) =
o(n2 ). The claim we can establish using the previous construction
√
bounds c(n) even more, bounding it to c(n) = 4 n.
4.5 Computational Complexity of the BMP Problem 55

Theorem 4.4. There exist instances (A ∈ {0, 1}n×m , k) of the bmf

problem that cannot be approximated to within an additive error of
√ √
c(n, m) = max{ 4 n, 4 m}.
The proof of this theorem is an easy modification of the above
proof, and is postponed to Appendix B.

4.5 Computational Complexity of the BMP

Problem
Two basis vectors are said to overlap if they share a row having 1
(i.e. bi and bj overlap if there is a row k such that bik = bjk = 1).
The overlapping basis vectors seem to be a main source of difficulties
in bmf. Hence, it is natural to ask whether the problem becomes
easier if the basis vectors do not overlap. Such a variant of bmf
is the Boolean Matrix Partitioning problem (bmp). To see that
bmp can indeed be easier than bmf, consider the case where no
error is allowed (i.e. exact-bmf and exact-bmp). The exact-bmf
problem is NP-complete but exact-bmp can be solved in time linear
in the number of ones in the input matrix since the basis vectors
are exactly the maximal sets of identical rows.
We need to make a detour to study the computational complexity
of the bmp problem. First, we see how to express bmp as a clustering
problem: the Binary k-Median Problem (bkm). To that end, we
show that bmp and bkm are equivalent problems, in the sense that all
results regarding the computational complexity and approximation
of one of these problems carry over to the other. Second, we study
the computational complexity and hardness of approximation of
bkm, obtaining simultaneously the same results for bmp.
Problem 4.1 (Binary k-Median Problem). Given a set C of bin-
ary m-dimensional vectors with |C| = n and a positive integer k,
find a set M = {µ1 , . . . , µk } of binary m-dimensional vectors (the
medians), and a partition D = {D1 , . . . , Dk } of C such that M and
D minimize
k X
X

costbkm (C, M, D) = µj − c , (4.14)
1
j=1 c∈Dj

where k·k1 is the Hamming norm.

56 4 The BMF and BMP Problems

Note that in the above definition, the vectors µj are allowed

to be arbitrary Boolean vectors, that is, M does not have to be a
subset of C. If this is not the case (i.e. M ⊆ C is required), we
shall call the problem the (Binary) Graph k-Median problem (the
intuition is that in the latter case we can consider the input as a
weighted graph instead of a set of binary vectors). Both versions
are studied here: Binary k-Median is easier to relate to bmp, while
the algorithms are for (Binary) Graph k-Median. In what follows,
we will first see that we can concentrate on bkm instead of bmp,
and then prove results on bkm, yielding same results to bmp. Then
we will see how to use algorithms for Graph k-Median to solve bkm
(and bmp, also).
Remark. There exists unfortunate ambiguity concerning the problem
name k-Median. For example, Arya et al. [AGK+ 04] use k-Median
to denote a problem where M ⊆ C is required (Graph k-Median
above), whereas for example Ackermann al. [ABS08] and Megiddo
and Supowit [MS84] use k-Median (or p-Median) to denote the
version that does not require the medians to be a subset of the input
set. To complicate things even more, Chen [Che06], for example,
uses the term Geometric k-Median to refer to a version of k-Median
where the medians can be selected arbitrarily, but the metric must be
Euclidean. The terminology in this thesis follows that of Ackermann
et al.
The first task is to show that any computational complexity
result for bkm is also a result for bmp (and vice versa). This is done
by showing that the problems are equivalent in a sense described in
the following theorem.
Theorem 4.5. The Boolean Matrix Partitioning problem and the
Binary k-Median Problem are equivalent problems, that is, there is a
polynomially computable bijection (A, k, B, X) 7→ (C, k ′ , M, D) such
that costbmp (A, B, X) = costbkm (C, M, D).
Proof. The idea of the proof is to show that, with proper interpret-
ation, the partition matrix P of bmp corresponds to the partition
D of bkm, and that the medians in M correspond to the columns
of X.
For inputs, we can consider the rows of A (in bmp) to be the
vectors in C and vice versa. The parameter k does not change. This
is clearly one-to-one and onto.
4.5 Computational Complexity of the BMP Problem 57

Let (C, k) be an instance of bkm and let (M, D) be a solution of

it. Set M contains k m-dimensional vectors, which can be written
as the rows of X. Define the matrix P = (pij ) from partition
D = {D1 , . . . , Dk } by setting

1 if ci ∈ Dj
pij =
0 otherwise.

Because D is a partition of C, matrix P has exactly one non-zero

element in each row, and thus fulfils the definition of bmp. Using a
similar approach we can create M and D from X and P, respectively.
Thus this mapping is also one-to-one and onto.
It remains to show that costbmp = costbkm . It holds that
n X
X k
costbmp (A, P, X) = kai − xj k 1(pij = 1) ,
i=1 j=1

due to the structure of P. As this is exactly costbkm , the claim

follows.

Remark. The above results builds also a connection between bmf

and bkm, the latter being a restricted version of the former. The
connection is not unique: for many clustering problems there is a
matrix decomposition problem that generalizes them. This type
of connections have only recently gained more attention in the
literature (see [Li08] and references therein).
A polynomially computable, approximation-preserving reduction
allows us to study the NP-hardness and approximability of bkm.
The above theorem clearly furnishes us with such reduction for
both directions (to and from bkm). The first result to prove with
the reductions in hand is the NP-completeness of bkm(or, more
precisely, that of d-(bkm)). The decision version of bkm, d-bkm, is
defined as usual.

Lemma 4.6. The decision version of the Binary k-Median Problem

is NP-complete.

For the proof, we need another problem, defined below.

58 4 The BMF and BMP Problems

Problem 4.2 ([MS84]). Let k and t be positive integers and let A

be a set of n points from R2 such that if (a1 , a2 ), (a′1 , a′2 ) ∈ A then
 
{ai − a′i  : i = 1, 2} ⊆ [N ], (4.15)

where N ≤ 4n4 . The question is: is there a set B = {b1 , . . . , bk },

with bi ∈ R2 and a partition E = {E1 , . . . , Ek } of A such that

k X
X
kbj − ak1 ≤ t (4.16)
j=1 a∈Ej

holds?

Problem 4.2 is a variation of the k-Median problem, where the

set of instances is restricted by (4.15). The problem was used by
Megiddo and Supowit [MS84] to prove the NP-hardness of the k-
median problem in R2 . Even though Megiddo and Supowit do not
explicitly state Problem 4.2, they prove that it is NP-complete.

Proof of Lemma 4.6. First notice that d-bkm is in NP.

To prove the NP-hardness, we reduce an instance (A, k, t) of
Problem 4.2 to an instance (C, k, t) of the decision version of bkm
in a way that the answer to (C, k, t) is ‘yes’ if and only if the answer
to (A, k, t) is ‘yes’.
Consider an instance of Problem 4.2, that is, a triplet (A, k, t).
Without loss of generality, we may assume that the points in A are
in the non-negative quadrant of the real plane. Furthermore, due to
the restriction (4.15), we may assume that

A ⊆ [N ] × [N ],

for N = 4 |A|4 (and thus, A ⊂ Z2≥0 ).

Notice that the sum of the Hamming distances is minimized
by the coordinate-wise median of the points in the sum. That
is, for any solution (B, E) minimizing (4.16), we may assume that
B ⊆ [N ] × [N ].
For the reduction, we need to embed the points of A into a
Boolean space. To obtain such an embedding, notice that

{ a − a′ 1 : a, a′ ∈ A} ⊆ [2N ].
4.5 Computational Complexity of the BMP Problem 59

We can use the standard technique of writing the coordinates of

points of A in unary in order to embed A into a 2N -dimensional
Boolean space. This is done as follows. Let a = (x, y) be an element
of A. We have assumed that x, y ∈ [N ]. For the embedding, consider
a 2N -dimensional binary vector b ∈ {0, 1}2N . Let the first x bits
of b to be 1, followed by N − x 0s. Then let the following y bits
be again 1, and the remaining N − y bits be 0. Repeating this
procedure to all a ∈ A gives the desired embedding.
It is straight forward to see that this embedding does not change
the Hamming distance between the points in A. Furthermore, since
N is polynomially bounded in n, it can be computed in polynomial
time with respect to n.
Finally, consider a solution (M, D) of instance (C, k) so that
(4.14) evaluates to at most t. Because points in C represent two
integers written in unary, and because we can assume that the points
in M are coordinate-wise medians of subsets of points in C, we see
that also points in M represent two integers written in unary. Thus,
using the reverse of the above reduction we can have a solution
(B, E) of the original instance (A, k) so that (4.16) evaluates to at
most t.

The following two corollaries are immediate consequences of

Theorem 4.5 and Lemma 4.6.
Corollary 4.7. The Boolean Matrix Partitioning problem is NP-
hard.
Corollary 4.8. If it is possible to approximate bkm to within a
constant factor in polynomial time, then it is possible to approximate
bmp to within the same factor in polynomial time, and vice versa.
The bkm problem can be approximated to within a factor of at
most 10. To see this, recall that in the (Binary) Graph k-Median
problem, set M is required to be a subset of set C. An answer to
the Graph k-Median problem is clearly a valid answer to bkm, too,
and – due to the triangle inequality – the error of an optimal answer
to it is at most twice as large as the error of the optimal answer to
bkm.
Thus, an approximation algorithm for the Graph k-Median prob-
lem with an approximation factor r is an approximation algorithm
for bkm with an approximation factor 2r. The claim now follows,
60 4 The BMF and BMP Problems

because for example Arya et al. [AGK+ 04] have proposed an ap-
proximation algorithm for the Graph k-Median problem with an
approximation ratio of at most 5 when applied to binary instances
(see Section 4.7).

4.6 Algorithms for BMF

It seems natural, given the information we have, to approach the

problem of solving bmf by dividing it into two subproblems: first,
find the basis B, and second, find a way to use it in the form of
matrix X. The same information tells us that this approach can
lead to highly suboptimal behaviour as the matrix B selected can
turn out to be a very hard instance for the bu problem.
Naturally we cannot, and will not, solve these two subproblems in
complete isolation: to know which matrix B is good we need to have
at least some level of information about how to use it. Nevertheless,
this division is helpful in understanding the algorithms.

4.6.1 Finding the Basis

The bmf problem can be divided into two steps: first find the basis
and then find a way to use it. Similarly, we can also divide the
problem of finding the basis into two steps: first step is to find a
set of candidate vectors from which the actual columns of B are
selected in the second step.
Finding the candidates. The main method to construct the set
of candidate vectors is based on the association accuracies between
A’s rows. The accuracy of an association between the ith and jth
row of matrix A is defined as in association rule mining [AIS93],
that is, c(i ⇒ j, A) = hai , aj i / hai , ai i, where h·, ·i is the vector
inner product operation. An association between rows i and j is
τ -strong if c(i ⇒ j, A) ≥ τ .
The association accuracies are used in creating the candidate
vectors via an association accuracy matrix. The association accur-
acy matrix C contains columns ci having 1s in rows j such that
c(i ⇒ j, A) ≥ τ . Each column of C is considered as a candidate
for becoming a column of B. The threshold τ controls the level
4.6 Algorithms for BMF 61

of accuracy required to include an attribute to the basis vector

candidate, and it is assumed that τ is a parameter of the algorithm.
Why use association accuracies, that is, why consider the columns
of the matrix C as candidate basis vectors? To see the intuition,
consider a Boolean matrix A that has a decomposition B ◦ X. Let
bp be the pth column of B with bip = 1 and bjp = 1 for some i and
j. Let q be such that xpq = 1. Then we know that aiq = ajq = 1 for
all such q. If no other column of B has 1 in positions i and j, then
c(i ⇒ j, A) = c(j ⇒ i, A) = 1. Then, consider a more complex case
when there is another column of B, say r, that has bjr = 1, but still
bir = 0. In that case the rule j ⇒ i no longer has accuracy 1 if there
is a column q of B with spq = 0 and srq = 1. But c(i ⇒ j, A) is still
1. Thus, in a case with no noise, the column bp of B is a row of the
matrix A, given that there is i so that bip = 1 and biq = 0 for all
q 6= p.

Example 4.3. Consider the following matrices B, X, and B ◦ X =

A:
   
1 0 0   1 1 0 1
0 0 1 1 1 0 1 0 1 1 0
   
     
B = 1 0 0 X = 1 0 1 0 A = 1 1 0 1 .
   
1 1 1 0 1 1 0 1 1 1 1
0 1 0 1 0 1 0

Now b1,1 = b3,1 = 1, but b1,i = b3,i = 0 for i = 2, 3. Hence,

c(1 ⇒ 3, A) = c(3 ⇒ 1, A) = 1. The more complex case is presented
in fourth row with b4,1 = b4,2 = b4,3 = 1. The accuracies for rule
4 ⇒ i, where i = 1, 2, 3, 5, are 3/4, 1/2, 3/4, and 1/2, respectively.
The accuracy for 1 ⇒ 4 is, however, 1, as is the accuracy for 5 ⇒ 4.
In this case, selecting τ = 1 gives the accuracy matrix
 
1 0 1 0 0
0 1 0 0 0
 
 
C = 1 0 1 0 0 ,
 
1 1 1 1 1
0 0 0 0 1

and selecting the first, second, and last column gives the correct B.
3
62 4 The BMF and BMP Problems

Unfortunately the proposed technique does not handle more

complex cases: if for all i such that bip = 1 there is a q such that
also biq = 1, then we cannot find column bp from C. A concrete
example of this is given by the complement of the n-by-n identity
matrix, Īn . The Boolean rank of Īn is O(log n) and real rank is n
[MPR95] (see also Section 4.3). However, with k < n, no value of
τ will give a perfect decomposition with that input, as the actual
association accuracies in matrix C′ are
 
1 f (n) f (n)

f (n) 1 f (n) · · · n−2
C′ = 
f (n) f (n) 1
, with f (n) = .


 n−1
.. ..
. .

The basis vector candidates in C are constructed from C′ by

rounding it from τ (i.e. cij = 1(c′ij ≥ τ )) and thus C will either be
full of 1s or an identity matrix, depending on the value of τ .
Constructing the matrix B. With the matrix C of candidate
basis vectors in hand, we now need to select k of them to the columns
of B. A simple idea is to base the selection on a greedy procedure:
always select the column of C that reduces the reconstruction error
most (given all the columns selected in previous steps). The problem
is, of course, how to compute the reconstruction error, especially,
how to decide how a certain column of C should be used to minimize
the reconstruction error. Knowing this would require us to solve
the bu problem exactly. The solution is to use the cover function
from Section 3.6.2 to produce an approximation of proper use of
the columns.
The columns of B (i.e. the basis vectors) are selected from the
matrix C and the rows of the matrix X (some version of which we
need to know) are fixed in a greedy manner as follows. Initially B
and X are empty matrices. The matrix B is updated in the iteration
l by adding the lth column bl , and matrix X is updated by adding
the lth row xl . The column bl is a column ci from C and the row xl
is an arbitrary m-dimensional binary row vector. The selection of bl
and xl is done so as to maximize cover(A, B, X, w) using the cover
function defined in (3.7) (page 35). The value of function cover can
be considered as the ‘profit’ of describing A using the product of
matrices B and X.
4.6 Algorithms for BMF 63

Algorithm 2 An algorithm for the bmf using association rules

(Asso)
Input: A matrix A ∈ {0, 1}n×m for data, a positive integer k, a threshold
value τ ∈ ]0, 1], and a real-valued weight w.
Output: Matrices B ∈ {0, 1}n×k and X ∈ {0, 1}k×m such that B ◦ X
approximates A.
1: function Asso(A, k, τ, w)
2: for i = 1, . . . , n do ⊲ Construct the association matrix C column
by column.
n
3: ci ← 1(c(i ⇒ j, A) ≥ τ ) j=1
4: end for
5: B ← [ ], X ← [ ] ⊲ B and X are empty matrices.
6: for l = 1, . . . , k do ⊲ Select the k basis vectors
 from
 C.
7: (i, x) ← arg maxi∈{1,...,n},x∈{0,1}1×m cover(A, [ B ci ], X
x , w)
 
8: B ← [ B ci ] , X ← X x
9: end for
10: return B and X
11: end function

The whole algorithm, referred to as Asso and combining the

computation of association accuracies and the selection procedure,
is presented in Algorithm 2.
The association matrix can be constructed in time O(n2 m): com-
puting the association accuracy c(i ⇒ j, A) for fixed i and j can be
done in time O(m), and there are n2 pairs of rows (ai , aj ) that need
to be considered. Given the matrix C, finding one column of B and
one row of X can be done in time O(n2 m): there are O(n) columns
of C to try, and for each column c of C, we can compute the row
x of X maximizing cover in time O(nm) by setting xi = 1 if and
only if c covers more uncovered 1s than uncovered 0s in column i
of A (cf. Section 3.6.2). Thus, Algorithm 2 has time complexity
O(kn2 m).
The algorithm has two parameters that control the quality of
results: the threshold τ , and weight w. The straight forward way to
set the parameters is to try several different possibilities and take
the one that minimizes the reconstruction error. Alternatively the
weight w can be used to express different valuations for covering 1s
and 0s.
64 4 The BMF and BMP Problems

4.6.2 Variations on Finding the Basis

Though the Asso algorithm is very simple, it gives results that are
comparable to those of more complex methods, as we will see in
the next section. In certain occasions we know, however, that our
data contains some structure we wish to take into account when
computing the decomposition. This section presents a variation of
the Asso algorithm for such situations.

The proposed variation of the Asso algorithm stems from the

observation that for data that are easily clustered, the basis vectors
should be different among the different clusters. To take advantage
of this fact, one can first cluster the data and then solve bmf within
each cluster. The twofold goal of this clustered decomposition is,
on one hand, to find the features specific for each cluster, and on
the other hand, to find the features specific for a group of clusters
or the entire data.

The following approach achieves these two goals simultaneously.

First, cluster the data and compose the set of candidate basis
vectors from association rules within each cluster. This is different
from computing the association rules over the whole data, as the
association strengths can vary considerably. Second, select the final
basis vectors from these candidates one by one. To select a basis
vector, first select the cluster that has the most uncovered 1s, and
consider the candidate basis vectors created for this cluster. Second,
select the basis vector which maximizes the function cover within
the selected cluster. Finally, use that basis vector to cover the rest
of the data.

In general, the clustering method Cluster used in line 2 of

Algorithm 3 is not fixed. It should be selected by the user according
to her needs. Algorithm 3 finds the basis vector candidates for the
clusters in the same way as in Asso.

Other variations are, of course, possible. One very simple idea

is to transpose the input matrix. The optimal decomposition does
not change, but for a heuristic such as Asso this can make some
difference.
4.6 Algorithms for BMF 65

Algorithm 3 A bmf algorithm using clustering (Asso+clust).

Input: A matrix A ∈ {0, 1}n×m for data, a positive integer p for the
number of clusters, a positive integer k ≥ p, a threshold value τ ∈ ]0, 1],
and real-valued weight w.
Output: Matrices B ∈ {0, 1}n×k and X ∈ {0, 1}k×m such that B ◦ X
approximates A, and row indices for each cluster.
1: function AssoClust(A, p, k, τ, w)
2: (I1 , . . . , Ip ) ←Cluster(A, p) ⊲ Ih s contain indices for A’s
columns.
3: A ← [ ], X ← [ ], B ← [ ]
4: for h = 1, . . . , p do
5: for i = 1, . . . , n do ⊲ Construct the basis vector candidate
matrix C(h) for each cluster h.


n
6: Ci(h) ← 1 c(i ⇒ j, AIh ) ≥ τ j=1
7: end for
8: C ← [ C C(h) ]
9: end for
10: for l = 1, . . . , k do  
11: h ← arg max {(i, j) : (AIh )ij = 1, (B ◦ XIh )ij = 0}
h=1,...,p
 I 

12: (ci , x) ← arg max cover [ B ci ], Xxh , AIh , w
ci ∈C(h) ,x∈{0,1}m
 I 
13: B ← [ B c ], XIh ← Xxh
i

14: J ← {1, . . . , n} \ Ih  J
15: x ← arg maxx cover(B, Xx , AJ , w)
 J

16: XJ ← Xx
17: end for
18: return B, X, and (I1 , . . . , Ip )
19: end function
66 4 The BMF and BMP Problems

4.6.3 Using the Basis

The algorithms from the previous sections, Asso and Asso+clust,

already return some way to use the basis B (i.e. some X). While
this is one possible solution, it is often advantageous (cf. Section 4.8)
to try to improve that by using the methods for the bu problem
from the previous chapter. The three methods considered here are
to use the optimal, exponential-time search; to use the iterative
update; and to use the Peleg’s algorithm.
The first method is based on the exhaustive search. Recall from
Section 3.5 that the bu problem can be solved exactly in time
O(2k knm). Thus, if k is small – as it often is – and n and m are
not too big, either, it could be plausible to solve X exactly. This
method is henceforth called Asso+opt.
The second method is to update the matrix X returned by Asso
using the IterX algorithm (Algorithm 1, page 36). The algorithm
is called Asso+IterX.
Finally, it is possible to compute X using the PelegRB algorithm
via the reductions form the previous chapter. But PelegRB cannot
be seeded with a preliminary X, and thus it has to be computed from
scratch. Neither does PelegRB understand the weighting parameter
w, which is why it must be ignored when calling the algorithm.
Following the established naming convention, this algorithm is called
Asso+PelegRB.
Naturally, any other algorithm to solve the bu problem could
be used as well. Indeed, one of the advantages of splitting the bmf
problem into subproblems is that the independent improvements in
each subproblem have immediate effect on the bu problem. It must
also be noticed that the variations from Section 4.6.2 could be paired
with these methods, so that one could, for example, first compute
Asso+IterX and then take the transpose of the data matrix and
compute new Asso+IterX selecting the better of these two. For the
sake of clearness that is not done in this thesis, though.

4.7 Algorithms for BMP

Recall that, due to Theorem 4.5, we can use any existing algorithm
for bkm to solve bmp. Moreover, we can use any algorithm A that
4.7 Algorithms for BMP 67

approximates Graph k-Median, where the set of possible medians

is restricted to the set of input points, to approximate bkm with
approximation ratio at most twice that of A. Graph k-Median has
been the subject of many papers in the course of years, and new
results have been obtained recently.
Charikar and Guha [CG99] presented a 4-approximation al-
gorithm for the Graph k-Median problem. This was improved
by Arya et al. [AGK+ 04], who proposed a local-search heuristic for
the problem, obtaining a locality gap of 5 with single local swap,
and more generally, 3 + 2/p + ε approximation ratio for polynomial
number of local steps, each of which can be performed in time nO(p) .
The locality gap of a local-search heuristic is the approximation
ratio guaranteed for the algorithm when the maximum number of
local improvements is not limited. The maximum number of local
improvements made by the algorithm is upper bounded by the ratio
M/ε, where M is the maximum error possible for the input and ε is
the smallest possible non-zero change in the problem’s cost function.
In general, this ratio can be exponential with respect to the instance
size, but for example with bmp it is upper bounded by nm, the
size of the matrix: the maximum error M cannot be more than
that, and the minimum improvement ε is 1. It follows that when
the algorithm of Arya et al. [AGK+ 04] is applied to bmp, it has
guaranteed approximation ratio of 10 that it achieves in polynomial
time, as claimed above. The number of local swaps defines the size
of the neighbourhood for the local search, and usually, as is the case
with the algorithm of Arya et al., the size of the neighbourhood
increases exponentially with the number of local swaps.
Building upon the results of Charikar and Guha, Chen [Che06]
proposed a method based on core-sets that achieves (10 + ε) ap-
proximation ratio with time O(nk + k 7 ε−4 log5 n), which is faster
than the algorithm by Arya et al. [AGK+ 04]. Most recently, Acker-
mann, Blömer, and Sohler [ABS08] provided a universal k-median
clustering algorithm that works with those (not necessarily met-
ric) dissimilarity measures that satisfy certain properties. Their
algorithm does not restrict the medians to the set of input points,
applying directly to bmp. The  algorithm provides (1 + ε) approxim-
(k/ε) O(1)
ation in time O n2 . Notice, however, that their algorithm
assumes fixed-dimensional space, and thus its applicability to bmp
is questionable from that point of view.
68 4 The BMF and BMP Problems

Other recent results include Meyerson et al.’s algorithm [MOP04]

with running time independent of the number of input points, and
Chrobak et al.’s [CKY06] simple reverse greedy algorithm with
O(log n) approximation.

4.8 Experimental Evaluation

The main goal for the experimental evaluation was to verify that
the algorithms work as expected, that is, they find accurate and
meaningful decompositions. Synthetic data was used to study the
accuracy of the algorithms, and the effects various characteristics of
the data have on them. Accuracy with real-world data was studied,
but there also the meaningfulness and interpretability of the results
was considered. Real-world data was also used to confirm that the
behaviour of the algorithms observed with synthetic data was also
present with real-world data, instead of being a result of the data
generation process.

4.8.1 Algorithms Used

The seven algorithms used in these experiments contained four vari-

ations of the Asso algorithm, a bmp algorithm and two benchmark
algorithms. It should be noted that, to the best of the author’s
knowledge, this is the first work to address the algorithmic issues for
finding approximate Boolean decompositions, and the algorithms
from Section 4.6 are the only known algorithms for the problem.
Thus, they cannot be directly compared to other algorithms. Instead,
results from svd and nmf algorithms are presented as benchmarks,
those being two popular matrix factorization methods. Their results
are not directly comparable to the results of Asso and its variants,
but it could be argued that not being much worse than an svd or
nmf algorithm is already an achievement from a method that is
bounded to binary-valued factor matrices. The svd implementation
used was Matlab’s build-in command, and the nmf algorithm was
based on the alternating least squares approach (see [BBL+ 07]).
The algorithm for svd is denoted by SVD, and the algorithm for nmf
is denoted by NMF.
4.8 Experimental Evaluation 69

The variations of Asso differed in the way they selected the matrix
X, and the purpose of the experiments was to study the effects
different bu algorithms had on the overall performance. In addition
to plain Asso (with no post-processing to improve X), Asso+IterX,
Asso+PelegRB, and Asso+opt were used, the last method using the
optimal, O(2k knm)-time exhaustive search.
The bmp problem is not a central problem of this thesis, and given
the previous results, solving it is equivalent to solving a k-Medians
clustering with binary input. Nevertheless, some experiments were
performed to compare the results to the results of bmf algorithms.
For the algorithm, the local-search heuristic of Arya et al. [AGK+ 04]
with single local swap was selected.

4.8.2 Measuring the Error

Selecting a correct error measure for comparisons between real-

valued and Boolean methods is not a straight forward task. The
real-valued methods, for example, SVD and NMF, try to minimize
the squared reconstruction error (i.e. the Frobenius norm), which
squeezes the element-wise errors less than 1, but increases those
above 1. With the sum-of-absolute-differences distance, the con-
tribution of the small errors is increased, if at the same time that
of big errors is decreased. As an example, if SVD approximates an
element of a binary matrix by saying it is 0.5, the contribution of
this to the (squared) Frobenius error is only 0.25, while with the
sum-of-absolute-errors the error is 0.5. Furthermore, in the context
of approximating a binary matrix, the value 0.5 is equivalent to
saying ‘I don’t know’, and is, arguably, almost as bad as an error of
1.
To facilitate the comparison, two approaches are used in this
thesis. The first approach is to use the Frobenius norm. This
is the natural approach to all real-valued methods, but slightly
less natural to binary-valued ones. While minimizing the sum-of-
absolute-differences, in case of binary matrices, is equivalent to
minimizing the Frobenius, the fact that the smallest possible error
is 1 prevents the binary-valued methods on benefiting from the
aforementioned properties of the Frobenius norm.
The second approach uses the sum-of-absolute-differences dis-
tance. To overcome the problem with real-valued methods, the
70 4 The BMF and BMP Problems

rounded results of SVD and NMF are computed as follows: first, the
real-valued factor matrices are multiplied together using normal
matrix multiplication. The resulting matrix (aij ) is then rounded
to matrix (a′ij ) by setting a′ij = 0 if aij < 0.5 and a′ij = 1 otherwise.
The svd and nmf algorithms with rounding are referred to as SVD0/1
and NMF0/1 , respectively.
Both of these approaches can be seen to benefit the real-valued
methods, albeit in different situations. While a totally fair method
would definitely be better, the lack of such methods justifies using
the other comparison methods which are fair in the sense that at
least they are not presenting too optimal view of the proposed
algorithms.
Notice that bmf is a different problem than svd or nmf. Thus,
a straight forward comparison between these methods and their
results is not possible, and all conclusions based on the comparisons
should be taken with a grain of salt.

4.8.3 Synthetic Data

The data and parameters used were the same as in Section 3.7,
except this time the algorithms were only given the matrix A, and
they had to find both matrices B and X. The four parameters
whose effects were studied were the same, too: (1 ) the number of
columns in B; (2 ) the noise level; (3 ) the density of the columns in
B; and (4 ) the mean number of columns of B used to create each of
the columns of the data. For details of the data generation process,
see Section 3.7.1.
Only algorithms for bmf and benchmark algorithms were used
(i.e. no bmp algorithms were used). The results are presented below.
Number of columns in B. The number k of columns in B
varied from 8 to 28 with steps of size 2. The total density of the
resulting matrices was approximately 0.35.
The results are presented in Figure 4.1. Notice first that SVD
and NMF have reconstruction errors below the errors of the other
methods. This means that Asso and its variations were not fully
able to take advantage of the Boolean algebra. Increasing k slightly
increases the error of Asso and its variations; on the other hand,
SVD (and SVD0/1 ) seems to improve with larger values of k (this was
actually expected, as the matrices had full real rank).
4.8 Experimental Evaluation 71

Asso Asso+opt 80 Asso Asso+opt

5000 Asso+IterX
NMF0/1 Asso+IterX NMF
Asso+PelegRB SVD0/1 70 Asso+PelegRB SVD

4000
60
d1(A−B°X)

dF(A−B°X)
3000
50
2000 40

1000 30

0 20
8 10 12 14 16 18 20 22 24 26 28 8 10 12 14 16 18 20 22 24 26 28
k k
(a) (b)
Figure 4.1: Reconstruction errors of bmf decomposition when k,
the number of columns in B, varies with (a) d1 and (b) dF distances.
Markers are mean values over twenty instances, and the width of
error bars is two times the standard deviation. Matrix multiplication
is normal for SVD and NMF, and Boolean for the other methods.

For the proposed methods, Asso+IterX is always as good as

Asso+opt, that is, optimal with the given B, and even the plain
Asso is very close to those other two. Asso+PelegRB is clearly the
worst in both cases (recall that PelegRB does not use the already-
computed X from Asso).
The difference between sum-of-absolute-differences (Figure 4.1(a))
and Frobenius distance (Figure 4.1(b)) is very small: only the relative
performance of SVD improves considerably against the other methods
when Frobenius norm is used.
Noise level. The level of noise varied from 0 to 0.4 with steps
of size 0.05. The results are presented in Figure 4.2. All algorithms
follow essentially the same, intuitive trend. Again, SVD (or SVD0/1 )
is the best, but with zero noise, all bmf algorithms are better than
NMF with Frobenius distance (Figure 4.2(b)). Also, with zero noise,
Asso+PelegRB is better than plain Asso, but after that it is worst
of all methods with both error measures.
Similar to previous experiments, Asso+IterX is always as good
as Asso+opt, and – except in the zero noise case – plain Asso is
very comparable to these two. The variance of all methods was very
small with all values of noise greater than 0.05.
Density of columns of B. The mean density (number of 1s
divided by the total number of elements) of B’s columns varied from
0.05 to 0.3. The total density of the resulting matrices varied from
72 4 The BMF and BMP Problems

6000 Asso Asso+opt Asso Asso+opt

NMF0/1 Asso+IterX NMF
Asso+IterX 80
SVD0/1 Asso+PelegRB SVD
5000 Asso+PelegRB

60
d1(A−B°X)

dF(A−B°X)
4000

3000
40
2000
20
1000

0 0
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
noise level noise level
(a) (b)
Figure 4.2: Reconstruction errors of bmf decomposition when the
level of noise varies with (a) d1 and (b) dF distances. Markers are
mean values over twenty instances, and the width of error bars is
two times the standard deviation. Matrix multiplication is normal
for SVD and NMF, and Boolean for the other methods.

approximately 0.2 to approximately 0.7. The results can be seen in

Figure 4.3.
Density does not seem to have any effects on SVD or NMF. The
results of Asso, Asso+IterX, and Asso+opt decline slightly as dens-
ity increases to 0.2, but then they seem to improve slightly. As
was expected, the results of Asso+PelegRB get worse as density
increases. The variance is negligible in almost all cases.
Mean number of columns of B used for each column of A.
The mean number of columns of B involved in the Boolean combin-
ation to create columns of A (i.e. mean number of 1s in columns of
X) had three possible values, 4, 6, and 8. The resulting matrices had
an approximate total density between 0.35 and 0.55. The results,
seen in Figure 4.4, follow those of IterX and PelegRB in a similar
experiment (see Figure 3.2(b)): the quality of Asso+PelegRB’s res-
ults decreases, while other methods are almost immune to it having
virtually no variance.

4.8.4 Real-World Data Sets

Several real-world data sets were used to study the performance of

the algorithms. A detailed description of each data set, its source,
and possible preprocessing steps taken is given below and some of
the characteristics of the data sets are summarized in Table 4.1.
4.8 Experimental Evaluation 73

100 Asso Asso+opt

8000 Asso Asso+opt
NMF0/1 Asso+IterX NMF
Asso+IterX Asso+PelegRB SVD
Asso+PelegRB SVD0/1 80
6000

dF(A−B°X)
d1(A−B°X)

60

4000 40

20
2000

0
0.05 0.1 0.15 0.2 0.25 0.3 0.05 0.1 0.15 0.2 0.25 0.3
density density
(a) (b)
Figure 4.3: Reconstruction errors of bmf decomposition when the
density varies with (a) d1 and (b) dF distances. Markers are mean
values over twenty instances, and the width of error bars is two
times the standard deviation. Matrix multiplication is normal for
SVD and NMF, and Boolean for the other methods.

Asso Asso+opt
Asso Asso+opt
NMF0/1 Asso+IterX NMF
6000 Asso+IterX 80 Asso+PelegRB SVD
Asso+PelegRB SVD0/1

5000
dF(A−B°X)
d1(A−B°X)

60
4000
40
3000

2000 20

1000 0
4 6 8 4 6 8
mean number of base vectors per column of A mean number of base vectors per column of A

(a) (b)
Figure 4.4: Reconstruction errors of bmf decomposition when the
mean number of B’s columns used for each column of A varies with
(a) d1 and (b) dF distances. Markers are mean values over twenty
instances, and the width of error bars is two times the standard
deviation. Matrix multiplication is normal for SVD and NMF, and
Boolean for the other methods.
74 4 The BMF and BMP Problems

Table 4.1: Some characteristics of the real-world data sets. Average

numbers of 1s are rounded to the nearest integer.

Data Rows Columns Density (%) Average # of 1s

per row per column
Abstracts 4 894 12 841 0.9 115 44
DBLP 19 6 980 12.95 904 2
Dialect 1 334 506 16.14 82 215
20Newsgroups 5 163 19 997 0.89 177 46
NOW 124 139 11.48 16 14

The Abstracts data. This data2 is a collection of project ab-

stracts submitted for funding by National Science Foundation of
the USA. The data is represented in a bag-of-words format, and a
sample of the data, instead of the full data, is considered here. The
data was preprocessed by first stemming the words using Porter
stemmer3 [Por80]. Then, all words occurring in less than 10 or in
more than 999 sampled abstracts were removed. Only the pres-
ence/absence information was saved, omitting the word counts, to
achieve a binary matrix. The resulting data has 12, 841 abstracts
and 4894 words.
The DBLP data. This data is collected from the DBLP article
database4 , and it contains conference–author information. More
precisely, it contains 19 hand-picked conferences and 6980 authors.
The conferences, namely www, sigmod, vldb, icde, kdd, sdm,
pkdd, icdm, edbt, pods, soda, focs, stoc, stacs, icml, ecml,
colt, uai, and icdt, were selected to present various, yet slightly
overlapping fields of computer science, and the authors are all of
the authors with in total at least 2 publications in the selected
conferences. The resulting matrix is very flat and wide.
The Dialect data. This data contains 1334 features of spoken
dialect collected from 506 Finnish municipalities [EW97, EW00].
Each municipality is additionally characterized by two spatial co-
ordinates, but these were only used in plotting the results. The
suggested division of Finnish dialects is given in Figure 4.5.

2
http://kdd.ics.uci.edu/databases/nsfabs/nsfawards.html
3
http://tartarus.org/~martin/PorterStemmer/perl.txt
4
http://www.informatik.uni-trier.de/~ley/db/
4.8 Experimental Evaluation 75

south-west

central S.W.

Tavastia

S. Ostrobothnia

C. & N. Ostrob.

far north

Savonia

south-east

Figure 4.5: The division of Finnish dialects [EW97, EW00].

The 20Newsgroups data. This data5 is a bag-of-words repres-

entation of almost 20, 000 Usenet news posts spread (almost) evenly
in 20 newsgroups. As with the Abstracts data, the data was first
stemmed using Porter Stemmer, and all words appearing in less
than 36 or in more than 999 posts were removed. This data set
was only used in this chapter; for subsequent chapters, a different
preprocessing was applied to yield a smaller data set.
The NOW data. This data6 contains information about spe-
cies’ fossils found in specific palaeontological sites in Europe [F+ 03].
The data is preprocessed following Fortelius et al. [FGJM06].

4.8.5 Randomization of the Data

A simple reconstruction error, even if it is small, does not necessary

guarantee that the data contains the latent structure we were trying
to find. It is possible that the results were obtained by chance.
The standard method to distinguish these cases is to measure the
statistical significance of the results against the null hypothesis,
which – broadly speaking – is ‘results were obtained by chance’.

5
http://people.csail.mit.edu/jrennie/20Newsgroups/
6
NOW public release 030717, available from http://www.helsinki.fi/
science/now/.
76 4 The BMF and BMP Problems

The general (randomization) framework for testing the signific-

ance is the following. First, generate multiple instances of (random)
data that satisfy the null hypothesis. Then, apply the algorithm
whose results are studied to the generated data and measure the
error. If, say, 99% of the results obtained with the random data are
worse than the results with the original data, the null hypothesis can
be abandoned and the original results are considered statistically
significant (at significance level 99%) (see, e.g. [GMMT07]).
There are two problems on applying this approach: how to define
the null hypothesis and how to generate the data agreeing with it.
In general, one would like the null hypothesis to be complex in the
sense that it rules out any simpler explanation of the data than
the one sought. Naturally, also the original data must agree with
the null hypothesis for the test to be meaningful. An example of
a typical null hypothesis is ‘the results are due to the density of
the data matrix’. But density is a very low-level characteristic, and
being able to rule out that null hypothesis does not guarantee that
there is not some more complex, yet uninteresting, feature of the
data that explains the good results.
On the other hand, the data-generation process is not usually
known, complex null hypotheses are hard to define, or the testing
method is required to be applicable to different kinds of data. All
these limit the possible complexity of the null hypothesis. Moreover,
there is very important practical limitation: generating truly random
data satisfying complex null hypotheses can be very hard. A null
hypothesis which cannot be tested is not very useful. Simpler null
hypotheses do not usually have this problem, for example generating
random binary matrices with (approximately) same density is easy.
Thus, a compromise between the complexity of the null hypothesis
and the hardness of generating random data satisfying it is sought,
and here the compromise comes in the form of marginal sums of
the binary matrix. The marginal sums of an n-by-m binary matrix
A = (aij ) are given by a pair of vectors, (c, r). Vector c contains
P
the sums over A’s columns, that is, cj = ni=1 aij , and vector
P
r contains the sums over A’s rows (i.e. ri = m j=1 aij ). Given
a pair (c, r), define M(c, r) to be the set of all n-by-m binary
matrices such that (c, r) gives their marginal sums. The task of
generating a uniform sample from this set can be done by performing
swap randomizations (see, e.g. [CC03]). To generate the swap
4.8 Experimental Evaluation 77

randomizations, an MCMC method by Gionis et al. [GMMT07] was

used. As with any Markov chain, it requires enough iterations to
be fully mixed. In all experiments the number of iterations was
10000nm for n-by-m data matrix. The chain was re-initialized for
each random sample.
Hence the procedure of checking the statistical significance of
the results is as follows. First, a seed matrix is created. Then, a
predefined number of random matrices are generated, initializing
the Markov chain every time with the seed matrix. The purpose of
this back-and-forth process is to make sure that the original matrix
is reachable from the same seed matrix as the random matrices with
the same number of iterations. Then the Asso algorithm (or some
other algorithm) is executed on the original and each of the random
matrices. Finally, the reconstruction errors are compared, and if the
reconstruction error of the original matrix is smaller than the error
of 99% of the random matrices, the result is considered significant.
The results for randomizations are presented below, after the
reconstruction errors.

4.8.6 Results on Real-World Data

The purpose of the experiments with the real-world data was, first,
to verify that the algorithms can achieve comparable results also
with real-world data, and second, to verify that the results returned
by the algorithms are intuitive and reveal interesting information
about the data.
Due to its poor performance with synthetic data, Asso+PelegRB
was not used with real-world data. Also Asso+opt was excluded
from the list of algorithms – not because of its poor performance,
but because of its practically infeasible running time with larger
data sets.
The results are reported as follows. First, the reconstruction
errors for bmf algorithms and benchmark algorithms (i.e. NMF and
SVD and their variations) are given. Then, these results are compared
against the results obtained from randomized data sets, following
experiments and discussion about the interpretability of the results
by bmf algorithms. Finally, some examples about the quality and
interpretability of the results obtained using the local-search heuristic
of Arya et al. [AGK+ 04] for the bmp problem are studied.
78 4 The BMF and BMP Problems

Table 4.2: Reconstruction errors from real-world data sets, d1 dis-

tance.

Data k Asso Asso+IterX NMF0/1 SVD0/1

Abstracts 5 559 436 559 436 563 248 563 336
10 554 569 554 569 561 476 561 503
15 549 780 549 780 558 703 559 011
DBLP 5 8 676 8 676 8 623 8 384
10 4 503 4 503 4 121 3 863
15 1 519 1 519 1 087 988
Dialect 5 58 683 55 057 43 194 40 846
10 48 967 41 649 32 409 28 136
15 45 231 35 825 26 343 22 379
20Newsgroups 5 902 585 902 582 905 003 904 293
10 896 389 896 388 897 896 896 984
15 891 564 891 540 894 237 892 147
NOW 5 1 573 1 569 1 421 1 379
10 1 407 1 396 1 168 1 020
15 1 278 1 272 971 760

Reconstruction errors. The reconstruction errors for real world

data are given in Tables 4.2 (for d1 distance) and 4.3 (for dF dis-
tance).
With dF distance, SVD is definitely the best method with NMF
being close to it. Asso and Asso+IterX are much worse. One must
remember, however, that with Boolean decomposition the smallest
error one can make in each element is 1. This has the tendency
of increasing the squared error, especially when compared to those
methods that can take advantage of the properties of squaring, that
is, that errors less than 1 are decreased even further.
When d1 distance is used, results change slightly. Again, SVD0/1
and NMF0/1 are often better than Asso or Asso+IterX, although
this time the difference is not so striking. But with Abstracts and
20Newsgroups, Asso and Asso+IterX are better than NMF0/1 or even
SVD0/1 (in fact, with Abstracts and k = 15, NMF0/1 is better than
SVD0/1 ). As the results change dramatically with dF distance, this
suggests that NMF and SVD err in many elements of Abstracts and
20Newsgroups by slightly more than 0.5, yielding larger rounded d1
error, but still keeping the Frobenius error small.
Comparing Asso and Asso+IterX to real-valued decompositions
arguably has its problems, but comparing them to each other is
4.8 Experimental Evaluation 79

Table 4.3: Reconstruction errors from real-world data sets, dF

distance.

Data k Asso Asso+IterX NMF SVD

Abstracts 5 747.95 747.95 727.48 726.60
10 744.69 744.69 721.25 719.28
15 741.47 741.47 717.23 713.95
DBLP 5 93.15 93.15 81.18 81.01
10 67.10 67.10 58.69 58.27
15 38.97 38.97 33.67 33.11
Dialect 5 242.25 234.64 179.93 177.72
10 221.28 204.08 157.60 152.09
15 212.68 189.27 146.16 137.53
20Newsgroups 5 950.04 950.04 920.27 919.12
10 946.78 946.78 913.13 910.62
15 944.23 944.21 908.25 904.37
NOW 5 39.66 39.61 32.23 31.93
10 37.51 37.36 29.65 28.46
15 35.75 35.67 27.90 25.84

straight forward. We know that Asso+IterX will always do at

least as well as Asso, but whether it will give any added accuracy
to the decompositions is still an open question. The synthetic
experiments suggested that it should. And so it seems to do, but
only in remarkably few cases, and even then usually very little.
From Table 4.2 one can see that IterX aids Asso considerably
only with Dialect data. There are some improvements also with
20Newsgroups and NOW, but they are basically insignificant, and
with Abstracts and DBLP the Asso algorithm has already reached
the local optimum. It thus seems that improving X with additional
methods (at least, with IterX) is not usually necessary, although
with some cases it can provide considerable improvements.
Randomization results. Randomized versions of data sets were
used to assess the significance of the results, as described in Sec-
tion 4.8.5. In total, 100 random versions of each data set, ex-
cluding 20Newsgroups, were created. The 20Newsgroups data was
excluded due to its size: running experiments on 100 copies of it
would have taken too much time. The algorithms used were Asso,
Asso+IterX, NMF0/1 , and SVD0/1 , and the measure of error was the
d1 distance. For Abstracts data, only Asso and SVD0/1 were used:
the results of Asso+IterX were exactly the same as those of Asso
80 4 The BMF and BMP Problems

Table 4.4: Percentage of random data sets that gave smaller d1 error
than the original data. 100 random data sets were used. Missing
results are denoted by —.

Data k Asso Asso+IterX NMF0/1 SVD0/1

Abstracts 5 2 — — 0
10 1 — — 0
15 0 — — 0
DBLP 5 2 2 2 2
10 1 1 1 1
15 0 0 0 0
Dialect 5 2 2 2 2
10 1 1 1 1
15 0 0 0 0
NOW 5 1 0 1 1
10 0 0 0 0
15 0 0 0 0

with that data, and NMF0/1 was unable to finish in reasonable time
(roughly one week). The results are presented in Table 4.4.
In short, all results can be deemed significant with confidence
level 95%, and all results with k ≥ 10 are significant with confidence
level 99%. In other words, all four algorithms, Asso, Asso+IterX,
NMF0/1 , and SVD0/1 , were able to find such structures in the data
sets that are highly unlikely to be a consequence of the marginal
sums of the data matrices.
Intuitiveness of the results. The Abstracts, 20Newsgroups, and
Dialect data sets were used to examine the interpretability Asso’s
results. As the terms in the first two data sets were stemmed, so
were the words in the respective results. To increase the readability,
the words are returned to their original singular form whenever that
form is clear.
Intuitiveness and accuracy do not come hand in hand, and the
parameters giving lowest reconstruction error are not necessary those
giving the most intuitive results. The parameter w, used in function
cover to weight the different types of errors, is a good example of
this. To achieve a good reconstruction error, setting w = 1 is usually
required (and this is what is done above), but especially with very
sparse datasets some other value, typically w > 1, can give more
intuitive results.
4.8 Experimental Evaluation 81

For Abstracts, the parameters used were τ = 0.3 and w = 6. The

rows of the data represent words, and thus the columns of B represent
sets of words. Studying the intuitiveness and interpretability of these
results is done by studying the columns of B, and seeing if the words
in sets form a meaningful concept. Examples of these sets of words
are given below. For some sets, only a subset of words is presented,
and this is indicated by an ellipsis. The sets are:

1. fund, NSF, year

2. cell, gene, molecular, protein, . . .

3. gopher, Internet, network, world, wide, web, . . .

4. behaviour, effect, estim, impact, measure, model, overestimate,

predict, test, . . .

5. course, education, enrol, faculty, institute, school, student,

undergraduate, . . .

6. abstract, don, set.

Many of the sets found consisted of words typical to a specific

field of science; examples 2 and 3 are of this type. Some of them
were of more general, for example, words used in science in general
(example 4), or words referring to education (example 5). Due to
the nature of the data, words related to money and funding were
also found (example 1). Despite the fact that most of the resulting
sets of words were very natural, also less natural ones were found;
example 6 illustrates this behaviour.
For 20Newsgroups, no specific pair of parameters was better than
others, as most of the results were very intuitive. The examples
given here are obtained setting τ = 0.4 and w = 4, leading to
somewhat large sets of words. Examples are:

1. disk, FAQ, FTP, newsgroup, server, Sun, UNIX, Usenet, . . .

2. Bible, Christ, church, faith, Jesus, Lord, Paul, scripture, . . .

3. playoff, team, win

4. agent, BATF, cult, FBI, fire, Koresh, Waco, . . .

82 4 The BMF and BMP Problems

5. Arab, Israel, Jew, land, Palestinian, war, . . .

6. American, announce, campaign, crime, federal, fund, office,

policy, president, . . .

7. Armenia, Azerbaijan, border, defend, military, Soviet, war, . . .

8. earth, orbit, space.

First example is clearly computer-related, second contains re-

ligious words, specific for Christianity, and third is about sports.
Finding such basis vectors could hardly be regarded as a surprise, as
the 20Newsgroups data contains many newsgroups in each of these
subjects.
To understand the fourth example, one needs to have some
knowledge of the recent history of the United States. All words are
related to the so-called Waco siege in 1993, where FBI and BATF
raided a religious group led by David Koresh in Waco, Texas.
Examples 5 and 7 are partly overlapping; word ‘war’ appears in
both of them. Yet, the wars are different ones. Example 5 clearly
points to Israel–Palestinian conflicts, while example 7 is about
Armenia and Azerbaijan, neighbours and former Soviet republics.
Example 6 includes words usually seen in political contexts while
the last example is from a space-related newsgroup.
In summary, Asso was able to find decompositions that were intu-
itive and informative. For example, there were no prior information
telling that the Waco siege was discussed in some newsgroup. Yet,
it came out very strongly in the results, indicating that it had been
a ‘hot topic’ for some time, at least. Among the informative and
intuitive results there were also less intuitive and less informative
ones. Careful selection of the algorithm’s parameters helps reducing
the number of unintuitive results, but they cannot be totally avoided.
This should not be a problem, as there are always meaningful and
interesting basis vectors present in the answers, too.
For Dialect data, the balancing variable w was set to 1. The
linguistic studies divide Finnish into 8 main dialects, namely south-
western dialects, central south-western dialects, Tavastian dialects,
South Ostrobothnian dialects, Central and North Ostrobothnian
dialects, the dialects of the far north, Savonian dialects, and south-
eastern dialects [Itk65]. While this division has been rather stable,
4.8 Experimental Evaluation 83

(a) (b) (c)

Figure 4.6: (a) The division of municipalities reported by
Asso+IterX with k = 8. (b) The eight clusters used in Asso+clust.
(c) The division of municipalities reported by Asso+clust with
k = 16. Black dots represent municipalities that are not reported to
belong to any dialect.

some minor changes have taken place during the past half a century
(cf. [Itk65, p. 31] and [SYL94, p. 19]). The dialect regions, following
Itkonen [Itk65], are presented in Figure 4.5.
When decomposing this data with Asso+IterX, the columns
represented the municipalities and k was set to 8. That is, the goal
was to find eight sets of dialectical features in B such that these sets
would reveal which dialect was spoken in which municipality. One
could infer that dialect i was spoken in municipality j if xij = 1.
The division of dialects obtained by this method is presented in
Figure 4.6(a). There are certain differences between the results of
Asso+IterX and the proposed division (e.g. splitting the Savonian
dialects into two), meaning that Asso+IterX was not able to fully
agree with the linguistics’ model. However, its results certainly
follow the main trends of the known dialect borders, and it would
be interesting to further study the algorithm’s results to see if they
contain some interesting linguistic results.
The division between dialects is not a strict one, and the exact
borders between two dialects are usually more or less arbitrary.
Nevertheless, the Asso+clust algorithm was used with this data.
84 4 The BMF and BMP Problems

Recall that Asso+clust works in two phases: at first phase it clusters

the data, and at the second phase it construct the basis vectors.
For the first phase, the k-Means algorithm from som Toolbox7 was
used. Its results for eight clusters are presented in Figure 4.6(b).
We can see that the south-western dialects and Tavastian dialect
are all combined into one dialect, while Savonian dialect is divided
into three. With such a starting point, it is not surprising that
Asso+clust cannot produce the original ground truth. Asso+clust
was run with k = 16, that is, on average 2 basis vectors per cluster.
It does not have to divide the basis vectors equally, though – and
indeed, that was not done. The dialect boundaries returned by
Asso+clust can be seen in Figure 4.6(c): no set of linguistic features
(i.e. basis vector) contains the features on Savonian dialect. Instead,
the huge cluster of south-western and Tavastian dialects is split into
many small, partly overlapping, regions. This splitting is somewhat
intuitive, as the big cluster contains very heterogeneous dialects
that presumably share very few features, and while the results in
Figure 4.6(c) do not resemble those in Figure 4.5, they do show
that the Asso+clust algorithm can provide interesting information
about the data, and something that cannot be obtained with pure
clustering methods.
Results for BMP. The algorithm used for the bmp problem was
Arya et al.’s local-search algorithm [AGK+ 04] (henceforth referred
to as AryaLocal). The results presented here serve as examples
of the differences between bmf and bmp, and do not provide a
throughout comparison between the respective algorithms.
Only two domains are considered, namely, 20Newsgroups and
Dialect, and the reconstruction errors obtained by AryaLocal are
compared against those obtained by Asso+IterX. Results are presen-
ted in Table 4.5.
We can see that, with k = 5, the AryaLocal algorithm finds a
slightly better decomposition for the Dialect data than Asso+IterX.
Yet, with the Dialect data and higher values of k, Asso+IterX
performs considerably better than AryaLocal. This was expected,
for example, on the basis of inequality (4.9). Moreover, with the
20Newsgroups data, the AryaLocal algorithm is clearly inferior to
the Asso+IterX algorithm.

7
http://www.cis.hut.fi/projects/somtoolbox/
4.8 Experimental Evaluation 85

Table 4.5: Reconstruction errors from real-world data sets, d1 dis-

tance.

Data k AryaLocal Asso+IterX

Dialect 5 55 029 55 057
10 42 840 41 649
15 36 060 35 825
20Newsgroups 5 1 053 752 902 582
10 1 042 428 896 388
15 1 035 561 891 540

Also the intuitiveness of AryaLocal’s results with 20Newsgroups

data left a lot to hope for: most of the words in a single partition
did not have any meaningful relation. An example of partition
with some meaning in it was a partition containing words ‘George’,
‘Bush’, ‘Jimmy’, ‘Carter’, ‘president’, ‘Arab’, ‘Jew’, and ‘Israel’, but
this was the best the algorithm could do.
The poor performance of AryaLocal with 20Newsgroups data was
expected: after all, the idea of partitioning all terms in documents
does not fit well with the fact that many terms are used in many
documents, and even other terms are homonyms, meaning that they
have a different meaning in different contexts.
A data set that is seemingly better suited for the context of
bmp is the Dialect data; this idea is emphasized by the relatively
good performance of AryaLocal with Dialect data in Table 4.5. The
dialects can be considered to be partly overlapping, but traditionally
the boundaries have been strict and some dialect has been used in
every municipality. Thus, the municipalities could be partitioned
according to which dialect is used in which municipality. This
was done using AryaLocal with k = 7 and k = 8, and results are
presented in Figure 4.7.
The results show that, indeed, AryaLocal is able to find mean-
ingful partition of the municipalities. Comparing to the ground
truth (Figure 4.5), one can see that the major differences are the
introduction of a new partition in central South Finland, combina-
tion of central south-western dialect with the Tavastian one, and, in
case of k = 7, combination of far north and Northern Ostrobothnian
dialects. With k = 8, the results are otherwise similar, except the
division of the northern partition into two.
86 4 The BMF and BMP Problems

(a) (b)
Figure 4.7: The partition of municipalities to different dialect regions
by AryaLocal with (a) k = 7 and (b) k = 8.

In short, AryaLocal’s results were mostly what was expected: in

reconstruction errors, it was not as good as Asso+IterX, given the
greater freedom the latter has in constructing the decomposition,
and from the intuitiveness point of view, it performed well only with
data that assumed some partitioned structure.
CHAPTER 5

The Nonnegative Column and

Column-Row Decomposition
Problems

Where the idea of cx and cur decompositions is motivated (with and

without the nonnegativity constraint) and the convex cone interpret-
ation of the nonnegative cx decomposition is discussed. Algorithms
for both problems are proposed and compared against prior algorithms
for cx and cur.

5.1 Problem Definitions

The problems considered in this chapter are no longer restricted to
the Boolean world, but use the space of nonnegative real numbers,
R≥0 . Nevertheless, the goal is still to create an interpretable matrix
decomposition. In the decompositions studied in this chapter, the
n×m
input matrix is assumed to be in R≥0 , the inner dimension of
factor matrices, k, is given as an input, and the factor matrices
are also required to be nonnegative. As an additional requirement,
the left-hand factor matrix is required to be a subset of the input
matrix’s columns. Also, in the other decomposition discussed in this
chapter where three factor matrices are used, the rightmost factor
matrix is required to be a subset of the input matrix’s rows. These
decompositions are called ncx and ncur, respectively, and they are
defined below.

87
88 5 The NCX and NCUR Problems

Nonnegative Column Decomposition Problem (ncx). Given

a nonnegative matrix A ∈ Rn×m≥0 and a positive integer k, find k
columns of A, constituting the matrix C, and a nonnegative matrix
X ∈ Rk×m
≥0 such that C and X minimize
costncx (A, C, X) = dF (A − CX). (5.1)
The ncur problem, using three factor matrices, is defined as
follows.
Nonnegative Column-Row Decompositiom Problem (ncur).
Given a nonnegative matrix A ∈ Rn×m≥0 and positive integers kc and
kr , find kc columns of A, constituting the matrix C, kr rows of A,
kc ×kr
constituting the matrix R, and a nonnegative matrix U ∈ R≥0
such that C, U, and R minimize
costncur (A, C, U, R) = dF (A − CUR). (5.2)
In the general counterparts of ncx and ncur, called cx and cur
problems, the input matrix can contain arbitrary real values, as can
matrices X and U, too.
The ncx problem can be divided into two subproblems, first
asking for the matrix C minimizing the reconstruction error, and
the second asking for the matrix X given some C. The latter
problem is well-known constrained least-squares fitting problem
(see Section 5.4.1; however, cf. this to connections between the
bu problem and Boolean cx decompositions in Chapter 6). The
former problem is known as the Column Subset Selection (css)
problem [BMD09], or, if the nonnegativity constraint is applied as
in ncx, as Nonnegative Column Subset Selection (ncss) problem.
Notice that in ncss only matrix C is asked, and if A is nonnegative,
so is any submatrix of it; thus, the nonnegativity constraint is only
implicitly assumed when considering how good a matrix C is.
The computational complexity of css is unknown [BMD09], the
same being the case with ncss. Indeed, it is not known whether the
nonnegativity constraint changes anything, but it seems possible it
does not.

5.1.1 Alternative Interpretation of CX and NCX

The cx and ncx problems have alternative interpretations using
subspaces and projections, much akin to the interpretations of svd
5.1 Problem Definitions 89

and pca. Considering the columns of matrices as points in n-

dimensional space, the goal of the cx problem is to select k of these
columns such that when we project the matrix A to the subspace
spanned by these columns, the projected matrix is as close to the
original one as possible (in terms of the Frobenius distance, that is).
The nonnegativity requirement of X in ncx means that we
cannot consider the whole subspace spanned by C. Instead, we
consider the convex cone generated by the columns selected in C.
A (pointed) convex cone is a set C of vectors such that if v, w ∈ C,
then αv + βw ∈ C for all α, β ∈ R≥0 . A convex cone C generated
by the k columns of a matrix C is the smallest convex cone such
that ci ∈ C for 1 ≤ i ≤ k. The goal of the ncx problem is to find C
such that, when A is projected to the convex cone generated by C,
the projected matrix is as close to the original as possible (again, in
the terms of the Frobenius distance).
In both problems, the ith column of A is projected to Cxi which
is either a point of a subspace (cx) or of a convex cone (ncx).

Example 5.1. Consider a matrix A defined as

 
0.6 0.9 0.6 0.4 0.7
 
A = 1.0 0.7 0.9 1.0 0.9 .
0.6 0.5 0.2 0.4 1.0

For an ncx decomposition with k = 3 we could select the first three

columns. These three columns, interpreted as points in 3D space,
generate a convex cone C = {αa1 + βa2 + γa3 : α, β, γ ∈ R≥0 }. The
squared reconstruction error dF (A − CX)2 equals to the sum of the
squared distances from the columns of A not in C to the nearest
points in this cone, that is,
X 2
dF (A − CX)2 = min ai − v 2 .
v∈C
ai ∈C
/

For example, the point of C closest to the fourth column of A is

0.6a1 + 0.3a3 .
The convex cone generated by the first three columns of A is
presented in Figure 5.1. Red circles are the generating vectors, and
the blue circles are the remaining two columns. The black lines
show where in the cone the blue points are projected to. 3
90 5 The NCX and NCUR Problems

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1
1.5
0 1
1.5
1 0.5
0.5
0 0

Figure 5.1: An example of interpreting ncx as a convex cone. Red

points are the generating vectors, blue points are the other vectors
of the matrix, and black lines show where in the cone the blue points
are projected to.

5.2 Motivation as Data-Mining Techniques

Interpretability is the key term when ncx and ncur decompositions

are considered as data-mining techniques. The standard cx and
cur decompositions can be used as data mining techniques, and
the interpretability of the results is one of the main reasons to do
so. The usual problem with matrix decompositions in data mining
is that the factor matrices can turn out to be hard to interpret.
Boolean Matrix Factorization addressed this problem by requiring
the factor matrices to be binary, much in the same lines as nmf
requires the factor matrices to be nonnegative. The cx and cur
decompositions do not restrict the values in the factor matrices as
such. Instead, the interpretability is achieved via the requirement of
using the actual columns and, in cur, rows of the original matrix. If
the columns (and rows) of the data matrix had some interpretation,
and they usually had, then so have the columns of C (and rows
of R). In a sense, the cx decomposition can be seen as a feature
selection method.
The ncx and ncur decompositions combine the approaches taken
by nmf and cx decompositions by assuming nonnegative input and
requiring the matrix X to be nonnegative. The matrix C is, of
course, nonnegative if A is.
5.3 Properties of NCX and NCUR Decompositions 91

Finally, it should be noted that a variation of the cx decomposi-

tion has been used in bioinformatics to reduce the redundancy in a
set of genetic markers [PDL+ 08].

5.3 Properties of NCX and NCUR

Decompositions
It is easy to see that the following basic properties hold for ncx
and ncur. An ncx decomposition of A with parameter k is at-
most-rank-k approximation of A, and cost∗ncx (A, k) ≥ cost∗svd (A, k);
the same holds for ncur with k replaced by min{kc , kr }. The
error of ncx decomposition is also bounded below by a nonnegative
decomposition of A, i.e. cost∗ncx (A, k) ≥ cost∗nmf (A, k), and the
same holds for ncur, again by replacing k with min{kc , kr } (but
notice that it is cost∗nmf ; the lower-bound property does not have
to hold for any specific nmf algorithm, unless it is optimal). Yet
another lower bound is given by cx (and cur) decompositions, and
again in the form that the optimal cx (or cur) decomposition costs
at most the cost of optimal ncx (or ncur) algorithm. The optimal
cost of ncx decreases when the number of columns in C increases,
but again, this does not need to hold for specific algorithms for ncx
(although we would certainly like it to hold).
These properties are simple and straight forward. A more inter-
esting question is the computational complexity of the problems.
Unfortunately, virtually nothing is known in this respect. The re-
search has hitherto concentrated on the cx problem (or on the css
problem; these two are essentially the same from computational
complexity’s point of view), but not even an NP-hardness result
for it is known [BMD09]. There is, however, some evidence to-
wards the hardness of the problem, as a recent result shows that
finding a submatrix C of k columns of A such that the volume of
the parallelepiped spanned by the columns of C is maximized is
NP-hard [ÇMI07].
One remarkable feature of the cx problem is that it does not
have any algorithm that has provable approximation guarantee with
respect to the optimal cx decomposition, but several algorithms
have been proposed that have guarantees with respect to the svd
of the same matrix (e.g. [BMD09]).
92 5 The NCX and NCUR Problems

Whether a hardness result for cx would imply anything about the

hardness of ncx is another open issue. Indeed, there is no obvious
relation between these two problems (other than the lower-bound
property), and finding one could perhaps help in understanding
both of the problems.
What we do know is the hardness of the problem where, given
a matrix A and a matrix C, a (possibly nonnegative) matrix X
is sought, and the cost to minimize is kA − CXkF . This prob-
lem, which is a clear analogy of the bu problem, is in P with and
without the nonnegativity constraint. Without the nonnegativity
constraint it can be solved optimally using the Moore–Penrose mat-
rix pseudoinverse of C, C + [GVL96, p. 257]: X = C+ A. With the
nonnegativity constraint, the problem is still solvable in polynomial
time as a convex quadratic program (CQP) [KTH79], and also gen-
eral convex optimization methods or quasi-Newton methods could be
used. As a result, we can find optimal cx and ncx decompositions
in time O(mk poly(n, m)) by enumerating all possible k-subsets of
the columns of A. Note, however, that this does not mean that the
cx (ncx) problem is in FPT: for that, the function exponential in
k must not depend on n or m.

5.4 Nonnegative Column Subset Selection

Algorithms
The possible intractability of the ncx problem lies in the ncss
problem, and thus the algorithms given here concentrate on solving
it. To that end, one could take any existing algorithm for the css
problem and then apply the nonnegativity constraint to matrix
X (for existing algorithms, see, e.g. [BPS05, DMM06b, BMD09])
Instead of taking this approach, two new algorithms are proposed:
LocNCX and ALS.
The structure of the ncss problem (and the css problem) lends
itself readily to a local-search heuristic, which LocNCX is. In general,
a local-search heuristic (or greedy local-search heuristic) works by
studying the neighbourhood of the current solution and moving to the
neighbouring solution improving the solution the most. The defin-
ition of the neighbourhood is problem- and algorithm-dependent,
but it usually consists of all solutions achievable from the current
5.4 Nonnegative Column Subset Selection Algorithms 93

Algorithm 4 A generic local-search heuristic.

Input: An instance I of a minimization problem Π.
Output: Solution S of I.
1: function Local(I)
2: S ← random solution of I
3: repeat
4: N ← neighbourhood of S
5: Sbest ← S
6: S ← arg minS∈N costΠ (I, S)
7: until costΠ (I, S) ≥ costΠ (I, Sbest )
8: return Sbest
9: end function

one with only a small change; in LocNCX the neighbourhood of C

consists of all matrices C′ with exactly one column being changed
to a column of A not in C. The algorithm continues until it reaches
a local optimum, that is, until no neighbouring solution is better
than the current solution.
Local-search algorithms need some initial solution to start with.
Usually, and particularly in this thesis, the initial solution is ran-
domly selected, but in some cases careful combination of randomness
and determinism can yield better results (cf. [AV07]).
Local-search algorithms can make exponentially many iterations
in the worst case if they approach a local optimum with exponentially
small steps, but in practice that seldom happens. Nevertheless, it is
possible to avoid this by requiring that the algorithm stops after the
best possible update is less than a certain threshold. Naturally, this
reduces the quality of the answer, and the trade-off is controlled
by a parameter that, together with the input size, determines the
threshold (for a concrete example, see [AGK+ 04]). Another option
would be to use a constraint, preferably a user-defined one, for
the maximum number of iterations allowed. A generic local-search
algorithm for minimization problems is given as Algorithm 4.
The LocNCX algorithm is a local-search heuristic with the neigh-
bourhood structure, as mentioned, being those matrices C′ having
exactly one column different from C. Thus, there are at most
k(n − k) neighbours for each matrix C to consider. An important
detail on LocNCX is the computation of the cost induced by a matrix
C – computing it requires computing the corresponding matrix X.
94 5 The NCX and NCUR Problems

Algorithm 5 The ALS algorithm for ncx.

Input: An n-by-m nonnegative matrix A and an integer k.
Output: Matrices C and X that approximate A.
1: J = k random column indices from {1, . . . , m}
2: C̃ = AJ
3: while err decreases and max iterations are not reached do
4: (X, err) = SolveX(A, C̃)
5: (C̃, err) = SolveC̃(A, X)
6: end while
7: C = MatchColumns(C̃, A)
8: (X, err) = SolveX(A, C)

For subsequent references, the subroutine to compute X for a given

C is called SolveX. For discussion about SolveX, see Section 5.4.1.
Notice also that the LocNCX algorithm is not bounded to the ncx
problem, but it can also be used as an alternative solution to the
standard cx problem simply by not requiring X to be nonnegative.
As we will see in Section 5.6.4, using LocNCX for the standard
cx problem can yield better results than the previously-proposed
algorithms.
The second algorithm for the ncx problem comes with the name
ALS and is inspired by the alternating least squares method used in
the nmf algorithms [BBL+ 07]. A sketch of the algorithm is given
in Algorithm 5. The idea is the following: first pick a random set of
k columns from the original matrix to construct a matrix C̃. Then
use this C̃ to compute the nonnegative matrix X using the routine
SolveX. In turn, compute a new nonnegative matrix C̃ using the
current X in an analogous way of computing X given C̃. The new
C̃ does not have to contain columns of A (hence C̃, not C), and
is computed by routine SolveC̃. Continue by recomputing X given
the new C̃, and so on.
The loop terminates either when the alternating process gives no
more improvement in the error function, or when a certain number of
iterations has been reached. Alternatively, stopping criteria can be
changed so that the while loop terminates when the improvement
in the error goes below some threshold. The user can supply the
algorithm with stopping criteria best suited to her needs.
After the while loop, matrix C̃ might not – and in general does
not – contain columns of A. Thus, an additional post-processing
5.4 Nonnegative Column Subset Selection Algorithms 95

step is needed, and this is what happens on line 7 of Algorithm 5.

The task is to find k distinct columns of A that are as close as
possible to the columns of C̃. This task is handled by the routine
MatchColumns as follows: A weighted complete bipartite graph is
constructed, the columns of A being the vertices in the left-hand side
and the columns of C̃ being the vertices in the right-hand side. The
weight of the edge between columns ai and c̃j is the cosine similarity


between them, that is, ai , c̃j /(kai kkc̃j k), but also other similarity
measures could be used. The columns of C are those matched with
the columns of C̃ in the minimum-weight matching of the bipartite
graph. An optimal matching can be found in polynomial time using
the Hungarian method [PS82, pp. 221–224]. Finally, the last row
recomputes X for the final C.

5.4.1 Solving X and C̃

The methods SolveX and SolveC̃ require solving the constrained

least-squares fitting problem: given A and C, minimize kA − CXk2F
such that xij ≥ 0. As mentioned above, this is doable in polynomial
time using various methods.
Unfortunately, these methods can be too slow to be used in a
local-search algorithm such as LocNCX, and thus in this thesis a very
simple approach is taken instead. First, an approximation of X,
called X′ , is computed using the pseudoinverse: X′ = C+ A. Second,
the final matrix X is constructed from X′ by projecting all negative
elements of the latter to 0. That is,

x′ if x′ij ≥ 0
ij
xij =
0 if x′ij < 0.

This projection-based approach is in fact a common practice em-

ployed in many nmf algorithms [BBL+ 07], where similar constrained
least-squares fitting problems are encountered.
The projection approach is very simple, and can be improved
with some simple ideas. For example, one could project all values of
X′ that are smaller than some positive ε to 0. A careful selection of
ε can indeed yield a smaller reconstruction error, but not without
introducing a new parameter. Thus, aiming to simplicity, this idea
is not used in the algorithms.
96 5 The NCX and NCUR Problems

It should be noted that LocNCX, equipped with the aforementioned

projection method, is not strictly a local-search heuristic as described
in Algorithm 4, as the selected neighbour is not necessary optimal
over all neighbours. The same is, of course, true for all similar
algorithms using possibly suboptimal neighbour selection.

5.4.2 Convergence of the Algorithms

The LocNCX algorithm will eventually converge to some stationary

point since there is only limited, albeit exponential, number of
different matrices C. Yet, this stationary point is not guaranteed to
be a local optimum. The convergence of ALS is a more complicated
issue. Would there not be the requirement for the columns to be
from the original matrix, the results reported in [BBL+ 07] would
directly apply here. That is, the while loop in line 3 of the ALS
algorithm converges to a local optimum provided that the optimal
solution is computed in every step. But the last step of mapping
the columns of C̃ to the closest columns of the original matrix A
(line 7) prevents any claims about the optimality of the obtained
solution.

5.4.3 Time Complexity

The time complexity of the algorithms is dictated by the time

complexity of SolveX and SolveC̃, both requiring the computation
of a Moore–Penrose pseudoinverse for an n-by-k (or k-by-m) matrix,
and a multiplication of two matrices of sizes n-by-k and k-by-m.
Using svd for the pseudoinverse, this is doable in time O(nk 2 )
(assuming n = m and k < n) [GVL96, p. 254]. That is also the time
needed for a single iteration of the ALS algorithm. A single iteration
of LocNCX takes time O((n − k)nk 3 ). Assuming that the number
of iterations is constant, these are the resulting time complexities,
albeit with possibly large hidden constants.

5.5 Algorithms for NCUR Decompositions

The simplest approach to find cur (and ncur) decompositions is
to first find the matrix C using any algorithm for the cx (resp.
5.6 Experimental Evaluation 97

ncx) decomposition, and then find the matrix R by applying the

same algorithm on the transpose of the input matrix. Finally, the
mixing matrix U needs to be solved. For cur this is done using the
pseudoinverse as U = C+ AR+ ; for ncur the projection approach
can be used.
The transpose-based approach is actually used for solving cur
(e.g. [BPS05]), but an alternative approach, explained here, is to
apply the local-search heuristic to the cur (or ncur) decomposition.
The algorithm resembles LocNCX, the difference being in the neigh-
bourhood structure. In this algorithm, LocNCUR, the neighbourhood
of (C, R) is the set of all pairs (C′ , R′ ) where either C′ = C or
R′ = R and in the other matrix exactly one column (in C′ ) or row
(in R′ ) is different to C or R. Again, the projection method is used
to compute U and thus the cost of pair (C, R).

5.6 Experimental Evaluation

The purposes of the experiments in this section are the same as in
the other experimental evaluations in this thesis, namely, to study
the effects of different data characteristics; to compare the proposed
methods to other ones; to validate the numerical results obtained
with synthetic data with real-world data; and to study the inter-
pretability of the results obtained from real-world data. But unlike
the other chapters, this chapter deals with nonnegative, real-valued
matrices, not binary ones, and this makes the experiments slightly
different. Additionally, a small experiment is done in order to study
the usability of LocNCX as an algorithm for the cx decomposition.

5.6.1 Algorithms Used

In total six different ncx and cx algorithms (and two variations)

were used in these experiments, plus eight additional ncur and
cur algorithms. The ncx (and ncur) decompositions have not
been studied previously, and thus the proposed algorithms cannot
be compared against other ncx (and ncur) algorithms. Instead,
two algorithms solving the standard cx decomposition are used,
namely 2Step and SPQR. The former, by Boutsidis, Mahoney, and
Drineas [BMD08, BMD09], is a new algorithm for cx decomposition
98 5 The NCX and NCUR Problems

that, with a high probability, approximates the best rank-k Singular

√
Value Decomposition to within a factor of O(k log k) with respect
to the Frobenius norm. Notice that this is a very good result,
as a cx decomposition of k columns is a very restricted type of
rank-k decomposition, whereas svd provides the optimal rank-k
decomposition. The latter algorithm, SPQR, is due to Stewart et
al. [Ste99, Ste05, BPS05]. Unlike 2Step, it does not have any
guaranteed approximation properties.
These algorithms take very different approaches to the problem:
SPQR selects columns in C in a greedy way, such that with the same
data its results for k and k − 1 differ only in the last column of
C. The 2Step algorithm, on the other hand, works in two steps:
first, it samples some number c > k columns from A based on
carefully selected probabilities. In the second step, it employs some
deterministic algorithm to select the final k columns from the c
sampled ones. Thus, its results with k and k − 1 columns (and,
indeed, between two runs with same value of k) can be completely
different.
A variation of both 2Step and SPQR were also used. They are
referred to as 2Step≥0 and SPQR≥0 , and they differ from the original
versions in that their decompositions are forced to be nonnegative.
Specifically, the matrices X returned by 2Step and SPQR were pro-
jected to the nonnegative orthant (i.e. all coordinates were projected
to nonnegative values) in 2Step≥0 and SPQR≥0 .
The other two algorithms against which the proposed algorithms
were compared were NMF and SVD. Notice that both present, at least
in theory, a lower bound for ncx (and ncur) decompositions.
The details for NMF and SVD are same as in Section 4.8.1. For
2Step and SPQR, the implementations were provided by the respect-
ive authors. The deterministic subroutine in 2Step was SPQR, it
selected c = Θ(k log k) columns in the first step, and the algorithm
was re-started 40 times, as suggested in [BMD09].
When studying the applicability of LocNCX to solve the cx prob-
lem, the algorithm is denoted by LocCX; it differs from LocNCX only
by not projecting the computed X to the nonnegative orthant.
The LocNCX, LocCX, and ALS algorithms were re-started 13 times
to overcome the adverse effects of a bad initial selection of C.
5.6 Experimental Evaluation 99

5.6.2 Synthetic NCX Data

The purposes of studying the synthetic ncx data were same as the
purposes of on synthetic bmf data, that is, to study the effects of
(1 ) the number of columns in C; (2 ) the noise level; (3 ) the density
of the columns in C; and (4 ) the mean number of columns of C
used to create each of the columns of the data.
Data generation process. The data was generated by varying
each of the four parameters one-by-one. Twenty matrices were
created for each combination of parameters. The default values for
parameters used when the parameter was not varied were k = 16,
10% of noise, density of 0.1 and 4 columns of C per each column of
the data.
The data generation process was different from that of previous
chapters. First, a nonnegative random matrix C was generated with
each element being sampled uniformly at random from ]0, 1[. To
model the sparsity of real data, d% of the elements were then set to
0, selecting the values at random.
A random matrix X was created again by sampling the elements
from ]0, 1[. Random elements in X were set to 0 in a way that,
on expectation, there were a prescribed number of non-zero entries
in each column of X. A matrix à was created as à = CY with
Y = (Ik X), so that the columns of C are the first k columns of Ã.
Noise was added by selecting, uniformly at random, a required
percentage of elements of à and perturbing these elements’ values
with Gaussian noise with mean 0 and standard deviation 0.5. If this
yielded negative values in the resulting matrix, they were projected
to 0 to obtain the final matrix A.
Number of columns in C. The number of columns in C, k,
varied from 8 to 28 with steps of size 2. The total density of the
resulting matrices varied between 0.34 and 0.4.
The results of these experiments are presented in Figure 5.2. All
ncx (and cx) methods are very close to each other at all points.
They also follow the trend of NMF and SVD closely, although the
latter two are clearly the best methods. Forcing the results of 2Step
and SPQR to the nonnegative orthant does not seem to have strong
effects on their results’ quality, hinting that they already find such
columns in C that are best used as nonnegative combinations to
present the remaining columns.
100 5 The NCX and NCUR Problems

LocalNCX SPQR≥ 0 LocalNCX SPQR

50 ALSNCX ALSNCX NMF
2Step≥ 0
NMF 48
SVD 2Step SVD
46
dF(A−CX)

dF(A−CX)
45 44
42

40 40
38
36
35
8 10 12 14 16 18 20 22 24 26 28 8 10 12 14 16 18 20 22 24 26 28
k k
(a) (b)
Figure 5.2: Reconstruction errors of ncx decomposition when k,
the number of columns in C, varies. The matrix X returned by
2Step and SPQR is projected to nonnegative orthant in (a) and left
as it in (b). Markers are mean values over twenty instances, and
the width of error bars is two times the standard deviation.

On the other hand, the good performance of ALS and LocNCX

show that the ability of using negative real values does not add any
extra power to the other methods.
Noise level. The level of noise varied from 0 to 0.4 with steps
of size 0.05. The resulting matrices’ density varied between 0.35
and 0.45. The results are presented in Figure 5.3.
All algorithms’ error increase with increasing noise, as is expected,
and the difference between the best (SVD) and the worst stays
constant and is very small. Nevertheless, 2Step≥0 and SPQR≥0 show
strange behaviour with data having no noise.
Density of columns of C. The mean density (number of non-
zero elements divided by the total number of elements) of columns
of C varied from 0.05 to 0.3. The total density of the resulting
matrices varied from approximately 0.2 to approximately 0.75. The
results can be seen in Figure 5.4.
The increasing density seems to increase the error with all al-
gorithms. Otherwise the results follow the previous ones: SVD is the
best, NMF the second, and all other algorithms are basically indis-
tinguishable above it. The small variations between the ordering
of these algorithms are all well within the standard deviation, and
thus are not very significant.
Mean number of columns of C used for each column of A.
The mean number of columns of C involved in the nonnegative
5.6 Experimental Evaluation 101

LocalNCX SPQR≥ 0 LocalNCX SPQR

80 ALSNCX NMF
80 ALSNCX NMF
2Step≥ 0 2Step SVD
SVD

60 60
dF(A−CX)

dF(A−CX)
40 40

20 20

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
noise level noise level
(a) (b)
Figure 5.3: Reconstruction errors of ncx decomposition when the
level of noise varies. The matrix X returned by 2Step and SPQR is
projected to the nonnegative orthant in (a) and left as it is in (b).
Markers are mean values over twenty instances, and the width of
error bars is two times the standard deviation.

LocalNCX SPQR≥ 0 LocalNCX SPQR

52 50 ALSNCX NMF
ALSNCX NMF
2Step≥ 0 2Step SVD
SVD
50 48
dF(A−CX)

dF(A−CX)

48 46
46
44
44
42
42
40 40
0.05 0.1 0.15 0.2 0.25 0.3 0.05 0.1 0.15 0.2 0.25 0.3
density density
(a) (b)
Figure 5.4: Reconstruction errors of ncx decomposition when the
density of C’s columns varies. The matrix X returned by 2Step
and SPQR is projected to the nonnegative orthant in (a) and left as
it is in (b). Markers are mean values over twenty instances, and the
width of error bars is two times the standard deviation.
102 5 The NCX and NCUR Problems

50 LocalNCX SPQR≥ 0 LocalNCX SPQR

48
ALSNCX ALSNCX NMF
NMF
2Step≥ 0 2Step SVD
48 SVD
46
dF(A−CX)

dF(A−CX)
46
44
44

42
42

40 40
4 6 8 4 6 8
mean number of base vectors per column of A mean number of base vectors per column of A

(a) (b)
Figure 5.5: Reconstruction errors of ncx decomposition when the
mean number of C’s columns used for each column of A varies.
The matrix X returned by 2Step and SPQR is projected to the
nonnegative orthant in (a) and left as it is in (b). Markers are mean
values over twenty instances, and the width of error bars is two
times the standard deviation.

linear combination used to create each column of A (i.e. the number

of positive coefficients) had three possible values, 4, 6, and 8. The
resulting matrices had an approximate total density between 0.4
and 0.55. The results, from Figure 5.5, do not bring anything new
to the discussion: they follow trends similar to the previous ones.

5.6.3 Synthetic NCUR Data

The purposes of studying the synthetic ncur data were similar to

those of synthetic ncx data (and hence, to those of synthetic bmf
data): they were used to study the effects of (1 ) the number of
columns in C and the number of rows in R; (2 ) the noise level; and
(3 ) the density of the columns of C and rows of R. The fourth
characteristic in the synthetic ncx and bmf data, the mean number
of columns of C used to create each of the columns of the data, does
not have a counterpart in the ncur data generation process.
Data generation process. The data was generated by varying
each of the three parameters one-by-one. Twenty matrices were cre-
ated for each combination of parameters, using the following default
values for parameters when they were not varied: = 16, 10% of noise,
and density of 0.1. To ensure that the generated matrices, before
the introduction of noise, do have an exact ncur decomposition,
5.6 Experimental Evaluation 103

only a special type of decomposition was used. Parameters n and

m were set to 150 and 80, respectively. The number of columns in
C was equal to the number of rows in matrix R, both denoted by
k, and the structure of the matrices was
!
Ik  
C= and R = Ik X′ k×m ,
Xn×k

where Xn×k and X′ k×m are n-by-k and k-by-m random matrices
with independently and identically distributed entries from ]0, 1[. To
model the sparsity, predetermined number of the entries (selected
uniformly at random) were set to 0. In the data generation process,
the mixing matrix U was a k-by-k identity matrix Ik , and the
generated matrix was A = CR. The first k rows of A were those
of R, and the first k columns of A were those of C. Noise was
introduced by perturbing a required percentage of the elements of
A with Gaussian noise with mean 0 and standard deviation 0.5; if
the perturbation resulted in negative-valued elements, they were
projected to 0.
Number of columns in C and rows in R. The first parameter,
k, varied from 8 to 28 with steps of size 2. The total density of the
resulting matrices varied between 0.06 and 0.2.
The results of this experiment are presented in Figure 5.6. SVD
and NMF are the best. Over all methods returning an ncur de-
composition (Figure 5.6(a)), LocNCUR is the best. Also ALST and
LocNCXT perform better than 2StepT≥0 or SPQRT≥0 .
When the nonnegativity constraint is removed from 2StepT and
SPQRT , they start to perform better, being slightly better than
LocNCUR with higher values of k (Figure 5.6(b)). In general, the
results of all cur and ncur algorithms are comparable to each
other, all following the same trends. None of the algorithms is
able to attain a reconstruction error close to that of nmf or svd,
hinting an inherent complexity in finding a good ncur (or cur)
decomposition.
Noise level. The level of noise varied from 0 to 0.4 with steps
of size 0.05. The resulting matrices’ density varied between 0.1 and
0.16. The results, presented in Figure 5.7 are similar to those in
Figure 5.6: SVD and NMF are the best and, with 2StepT and SPQRT
forced to nonnegative U, LocNCUR, ALST , and LocNCXT are slightly
104 5 The NCX and NCUR Problems

SPQR≥ 0 LocalCUR SPQR

LocalCUR
60 ALSTNCUR NMF
60 ALSTNCUR
NMF
SVD
LocalTNCUR SVD LocalTNCUR
2StepT≥ 0 2StepT
55 55

dF(A−CX)
dF(A−CX)

50 50

45 45

40 40
8 10 12 14 16 18 20 22 24 26 28 8 10 12 14 16 18 20 22 24 26 28
k k
(a) (b)
Figure 5.6: Reconstruction errors of ncur decomposition when the
number of columns in C and the number of rows in R, k, varies.
The matrix U returned by 2StepT and SPQRT is projected to the
nonnegative orthant in (a), and left as it is in (b). Markers are
mean values over twenty instances, and the width of error bars is
two times the standard deviation.

better. With 2StepT and SPQR allowed to return arbitrary-valued

U, the differences between the methods is indistinguishable.
Density of columns in C and rows in R. The mean density
of columns of C and rows of R varied from 0.05 to 0.3, yielding
matrices with total density between 0.03 and 0.6. This experi-
ment presents the biggest difference between the methods designed
for ncur decompositions (namely, LocNCUR, LocNCXT , and ALST )
and methods designed for general cur decomposition (2StepT and
SPQRT ). When the results of 2StepT and SPQRT are forced to non-
negative values, their results are notably inferior to those of LocNCUR,
LocNCXT , and ALST with higher levels of density, as can be seen
from Figure 5.8(a). On the other hand, when they are allowed
to return an arbitrary-valued matrix U, only LocNCUR can present
comparable results (Figure 5.8(b)).

5.6.4 Solving the CX Decomposition

A small experiment was conducted in order to verify whether the

local search is applicable also to the standard cx problem, and how
well LocNCX would do without the nonnegativity constraint. To this
end, two set of matrices admitting cx decomposition were created
in a process described below.
5.6 Experimental Evaluation 105

120
LocalCUR SPQR≥ 0 LocalCUR SPQR
100 NMF
100 ALSTNCUR NMF ALSTNCUR
SVD
LocalTNCUR SVD T
Local NCUR
2StepT≥ 0 80 2StepT
80
dF(A−CX)

dF(A−CX)
60
60

40 40

20 20

0 0
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
noise level noise level
(a) (b)
Figure 5.7: Reconstruction errors of ncur decomposition when the
level of noise varies. The matrix U returned by 2StepT and SPQRT
is projected to the nonnegative orthant in (a), and left as it is in
(b). Markers are mean values over twenty instances, and the width
of error bars is two times the standard deviation.

LocalCUR LocalCUR
ALSTNCUR 80 ALSTNCUR
120
LocalTNCUR LocalTNCUR
2StepT≥ 0 2StepT
70 SPQR
dF(A−CX)
dF(A−CX)

100 SPQR≥ 0
NMF
NMF SVD
SVD
80 60

60 50

0.05 0.1 0.15 0.2 0.25 0.3 0.05 0.1 0.15 0.2 0.25 0.3
density density
(a) (b)
Figure 5.8: Reconstruction errors of ncur decomposition when
the density parameter varies. The values of x-axes refer to the
parameter used in generating the data, not to the actual density of
the data. The matrix U returned by 2StepT and SPQRT is projected
to the nonnegative orthant in (a), and left as it is in (b). Markers
are mean values over twenty instances, and the width of error bars
is two times the standard deviation.
106 5 The NCX and NCUR Problems

Data generation process. All matrices had 150 rows and 92

columns, 80 of which are random linear combinations of the remain-
ing 12.
First set of matrices is based on a uniform distribution. First,
a random 100-by-12 matrix C was created by sampling its values
uniformly at random from ]0, 1[. Then, a random 12-by-92 matrix
X was created by setting the leftmost 12-by-12 submatrix as an
identity matrix, and sampling the values for the rest uniformly at
random from ]0, 1[. These two matrices were multiplied together to
obtain a matrix Ã.
The second set of matrices was constructed as the first one, but
when in the first one the sampling was done from the uniform distri-
bution, in the second one it was done from the normal distribution
with mean 0 and variance 1.
Noise was added to the obtained matrices as follows. A predefined
portion of matrix elements were selected uniformly at random. These
elements were perturbed using Gaussian noise with 0 mean and
standard deviation 0.5. The portion of elements perturbed varied
from 0 to 0.3 with steps of 0.05. Forty matrices were created in
both sets for each level of noise, resulting in a total of 560 matrices.
Experiments and results. The LocCX algorithm (i.e. LocNCX
without the nonnegativity constraint) was compared against 2Step
and against a result by a third algorithm that had the matrix C used
in creating the data as an input and solved X optimally. Notice,
however, that this algorithm is not necessarily optimal: with the
introduction of the noise, it is not clear that the columns from the
original matrix C still constitute the optimal factor matrix for the
data.
Figure 5.9 presents the results. In the figure, the dotted lines
represent maxima and minima of reconstruction error and the solid
line represents the mean. With data generated using uniform distri-
bution (Figure 5.9(a)), all methods perform similarly. Upon close
examination, one can see that with higher levels of noise LocCX and
2Step perform better than the method of selecting the original C
(i.e. there are better columns to be selected than those originally
used), and that between LocCX and 2Step the former is constantly,
albeit narrowly, better than the latter.
The difference between LocCX and 2Step is more visible in Fig-
ure 5.9(b), where the data was created using normal distribution:
5.6 Experimental Evaluation 107

120
150
100
dF(A−CX)

dF(A−CX)
80
100
60

40 50
20 LocCX LocCX
2Step 2Step
Original C Original C

0 0.05 0.1 0.15 0.2 0.25 0.3 0 0.05 0.1 0.15 0.2 0.25 0.3
noise level noise level
(a) (b)
Figure 5.9: Reconstruction errors of cx decomposition. Dotted
lines represent maxima and minima of reconstruction error and
solid line the mean over 40 random matrices created with identical
parameters. (a) Matrices created using uniform distribution. (b)
Matrices created using normal distribution.

the worst result of LocCX is always better than the average result of
2Step. In contrast to the uniform distribution, selecting the original
matrix C proves to be very advantageous, being clearly superior to
LocCX and 2Step. This shows that neither of the algorithms is able
to find optimal cx decomposition. Yet, the difference between LocCX
and the method using the original C stays approximately constant
after the noise level has reached 0.1, the reconstruction error of the
former being roughly 6/5 of the reconstruction error of the latter.
Thus, this experiment does not rule out the possibility that LocCX
could have a provable, constant approximation guarantee.

5.6.5 Real-World Data

The purposes of experiments done with the real-world data were

twofold. On one hand, to verify that the good results obtained with
synthetic data carry over to real-world data sets, and on the other
hand, to make sure that the interpretability of the results, a main
motivation behind ncx and ncur decompositions, is indeed good.
Data sets used. The data sets used were similar to those in
Section 4.8.4. Nevertheless, there are a few differences. Abstracts
data was not used in these experiments; DBLP data contained also
information about how many times an author has published in a
conference, not just the published/not published information used
108 5 The NCX and NCUR Problems

with bmf; Dialect and NOW data sets were exactly the same. The
20Newsgroups data set was replaced with a smaller subset, called
4News, having 87 news from 4 Usenet groups, namely, sci.crypt,
sci.med, sci.space, and soc.religion.christian. The data
was preprocessed1 using Andrew McCallum’s Bow Toolkit2 with
stemming and stop word removal, and taking the 100 most frequent
words. Thus the data matrix is 100-by-348, and it contains the
frequencies of the terms per document.
Reconstruction errors. The results for ncx and cx decompos-
itions with real-world data are given in Table 5.1. Perhaps the most
important result is the good performance of LocNCX: it consistently
presents results that are equivalent, or even better, than the results
of 2Step or SPQR. That is, LocNCX is able to find a nonnegative
cx (ncx) decomposition that is better than the best cx decom-
position found by the other algorithms. This is important for two
reasons. On one hand, it shows that LocNCX can find good ncx
decompositions, and on the other hand, it shows the concept of ncx
decompositions being meaningful. LocNCX and ALS are almost on
par, with the notable exception of DBLP data with k = 15 where
ALS is considerably worse.
In Table 5.1, 2Step and SPQR were allowed to use negative values
in matrix X. It is not surprising that when their results are forced to
be nonnegative, via the same projection method used in LocNCX and
ALS, the error is increased. Results for such experiments are reported
in Table 5.2. Comparing to Table 5.1, we see that the change in
error can be negligible (as with 4News and k = 5) or tremendous
(DBLP and k = 15). Yet, LocNCX is always better than 2Step≥0
and SPQR≥0 , giving it best overall performance. The big change in
2Step≥0 ’s results, compared to 2Step’s results, with Dialect data
hints that the cx and ncx decompositions of the data are very
different; on the other hand, the cx and ncx decompositions of
4News data are probably very similar, given the good performance
of 2Step≥0 and SPQR≥0 .
Similar experiments were conducted with ncur and cur de-
compositions, but the DBLP data was left out due to the shape

1
The author is grateful to Ata Kabán for preprocessing the data, and to Ella
Bingham for providing the preprocessed data.
2
http://www.cs.cmu.edu/~mccallum/bow/
5.6 Experimental Evaluation 109

Table 5.1: Reconstruction errors of ncx and cx decompositions

with real-world data.

Data k LocNCX ALS 2Step SPQR NMF SVD

4News 5 134.96 135.62 134.87 134.87 128.16 127.64
10 104.89 105.39 103.48 103.93 99.45 98.19
15 93.37 92.91 87.70 89.16 85.97 83.00
DBLP 5 258.22 259.14 258.68 291.59 253.75 251.92
10 165.92 173.00 168.44 170.58 158.97 157.30
15 72.99 131.95 76.22 95.33 71.16 70.65
Dialect 5 201.40 201.46 205.67 228.46 179.93 177.72
10 175.53 175.91 181.32 202.03 157.60 152.09
15 159.88 160.88 165.46 184.30 146.16 137.53
NOW 5 35.07 35.08 36.11 36.12 32.23 31.93
10 32.39 33.10 32.32 32.60 29.65 28.46
15 30.50 31.19 30.03 29.90 27.91 25.84

Table 5.2: Reconstruction errors of ncx decompositions with real-

world data. Results of 2Step≥0 and SPQR≥0 are forced to be non-
negative.

Data k LocNCX ALS 2Step≥0 SPQR≥0 NMF SVD

4News 5 134.96 135.62 134.96 134.96 128.16 127.64
10 104.89 105.39 105.29 106.86 99.45 98.19
15 93.37 92.91 96.43 96.24 85.97 83.00
DBLP 5 258.22 259.14 314.65 293.98 253.75 251.92
10 165.92 173.00 222.98 240.33 158.97 157.30
15 72.99 131.95 316.64 427.55 71.16 70.65
Dialect 5 201.68 201.73 205.84 228.54 179.93 177.72
10 176.43 176.95 182.45 203.08 157.60 152.09
15 162.83 163.71 168.22 186.50 146.16 137.53
NOW 5 35.07 35.08 36.29 36.58 32.23 31.93
10 32.39 33.10 32.89 34.23 29.65 28.46
15 30.50 31.19 31.97 33.46 27.91 25.84
110 5 The NCX and NCUR Problems

of the matrix (having only 19 rows). The results are reported in

Table 5.3. Over the proposed ncur algorithms, LocNCUR gives the
best overall performance, being next to LocNCXT only once, and
always better than ALST . LocNCXT is better than ALST , although the
differences are almost negligible. The cur decompositions reported
by 2StepT are better than those of SPQRT , and also better than
ncur decompositions by LocNCUR in 4News data. In Dialect and
NOW, excluding cases with k = 15, LocNCUR’s ncur decomposition
yield smaller error than 2StepT ’s cur decomposition, hinting that
the ncur decompositions seems natural to these data sets.
When only ncur decompositions were asked, the results (in
Table 5.4) followed those with ncx decompositions (in Table 5.2):
with 4News, 2StepT≥0 and SPQRT≥0 perform almost as well as 2StepT
and SPQRT , but with Dialect, the error of the former methods in-
creases heavily. Overall, the best method for ncur decompositions
seems to be LocNCUR. This is not surprising, as it is the only method
that is designed specifically for this problem.
Interpretability of the results. The ncx decomposition was
used with the DBLP dataset (recall that it has only 19 rows). The
results, with k = 6, give an example of intuitive column selection.
The columns correspond to authors, and a priori, one would hope to
have authors that represent the various fields of Computer Science
covered by the selected conferences. As can be seen from Table 5.5,
this is indeed the case with both LocNCX and ALS, although their
results are different.
In ALS’s results Umesh V. Vazirani is selected to represent the-
oretical computer scientists that publish in focs and stoc. In
LocNCX’s results, Vazirani’s role is given to Leonid A. Levin. Equi-
valently, John Shawe-Taylor (in LocNCX’s results) represents a group
of machine learners, and Hector Garcia-Molina (in ALS) and Divesh
Srivastava (in LocNCX) a group of data management researchers.
Another example, this time for ncur decomposition, is given by
the Dialect data. The experiment is somewhat similar to that done
with bmf algorithms, except that in case of ncur decompositions,
the results contain actual municipalities (all sharing some feature)
and features (all sharing some municipality). The first experiments
are conducted with ALST . In Figure 5.10(a), C has 7 columns and
R has 7 rows; in Figure 5.10(b) they have 8 columns and rows,
respectively. In both cases, the spread of selected features (rows
 5.6 Experimental Evaluation
Table 5.3: Reconstruction errors of ncur and cur decompositions with real-world data.

Data k LocNCUR LocNCXT ALST 2StepT SPQRT NMF SVD

4News 5 142.48 140.68 146.83 140.11 151.30 128.16 127.64
10 117.56 118.69 119.42 113.56 116.76 99.45 98.19
15 109.63 115.67 116.99 93.96 99.06 85.97 83.00
Dialect 5 221.77 221.84 225.29 225.52 247.09 179.93 177.72
10 199.29 197.64 207.05 201.63 226.12 157.60 152.09
15 191.94 189.58 201.17 186.48 209.24 146.16 137.53
NOW 5 37.43 37.43 37.95 38.28 38.97 32.23 31.93
10 35.18 35.22 36.39 35.85 36.59 29.65 28.46
15 33.95 34.41 35.32 33.66 34.37 27.91 25.84

111
 112
Table 5.4: Reconstruction errors of ncur decompositions with real-world data. The results of 2StepT≥0 and SPQRT≥0
are forced to be nonnegative.

Data k LocNCUR LocNCXT ALST 2StepT≥0 SPQRT≥0 NMF SVD

4News 5 142.48 140.68 146.83 140.68 152.14 128.16 127.64
10 117.56 118.69 119.42 126.30 131.10 99.45 98.19

5 The NCX and NCUR Problems

15 109.63 115.67 116.99 159.65 156.70 85.97 83.00
Dialect 5 221.77 221.84 225.29 294.54 278.07 179.93 177.72
10 199.29 197.64 207.05 388.28 401.95 157.60 152.09
15 191.94 189.58 201.17 543.55 568.72 146.16 137.53
NOW 5 37.43 37.43 37.95 38.28 39.27 32.23 31.93
10 35.18 35.22 36.39 37.26 39.79 29.65 28.46
15 33.95 34.41 35.32 41.88 46.75 27.91 25.84
5.6 Experimental Evaluation 113

Table 5.5: Results for DBLP data using ALS and LocNCX with k = 6.

ALS LocNCX
Naoki Abe Divesh Srivastava
Craig Boutilier Leonid A. Levin
Uli Wagner Souhila Kaci
Umesh V. Vazirani Xintao Wu
Hector Garcia-Molina Uli Wagner
Dirk Van Gucht John Shawe-Taylor

(a) (b) (c)

Figure 5.10: Spread of selected features (rows of R) and the selected
municipalities (columns of C) as reported by (a) ALST with k = 7,
(b) ALST with k = 8, and (c) LocNCUR with k = 7.

of R) is represented by open-faced (non-solid) symbols, while the

selected municipalities (columns of C) are plotted as solid dots.
With k = 7, ALST ’s results are a good approximation of the
ground truth (Figure 4.5 on page 75): as there are only 7 features,
central south-western dialects have been merged to neighbouring
dialects. There is also some overlapping, as few Ostrobothnian mu-
nicipalities have both Tavastian and South Ostrobothnian dialects,
and some northern municipalities have dialects of far north and
North Ostrobothnia. Given that the overlaps happen in the borders
of neighbouring dialects, the phenomenon can be considered natural.
When the number of features and municipalities is increased to
8 (Figure 5.10(b)), some changes occur. As ALS adjusts its selection
114 5 The NCX and NCUR Problems

of columns (and rows, in ALST ) based on the parameter k, there are

differences in more than one dialect.
For both values of k, the municipalities selected (columns of C)
are mostly close to the geographical midpoint of their dialect regions.
This is intuitive, as the ‘purest’ dialect can be assumed to be found
in the middle of its spread.
If the results of ALST contained many non-overlapping regions, the
same cannot be said about the results of LocNCUR (Figure 5.10(c)),
where the overlapping is more a rule than an exception: only south-
western and south-eastern dialects are recognizable.

5.6.6 Conclusion of Experiments

Overall, the proposed algorithms performed very well when com-

pared against other algorithms. LocNCX, in particular, succeeded
in finding ncx decompositions that were even better than the best
cx decompositions found. When applied to finding cx decomposi-
tions, LocNCX outperformed 2Step constantly, if also narrowly. ALS
followed LocNCX with slightly higher reconstruction errors.
With cur and ncur decompositions, the results were similar.
That LocNCXT performed so well against LocNCUR in terms of re-
construction errors is somewhat surprising. The problematic inter-
pretation of the results reported by LocNCUR seem to indicate that
transposing an ncx algorithm, no matter how naïve it sounds, is
the preferred approach for ncur algorithms.
CHAPTER 6

The Boolean Column and

Column-Row Decomposition
Problems

Where the problems are motived as a combination of bmf and

ncx/ ncur, and new algorithms are proposed and compared against
algorithms for bmf and for cx and cur.

6.1 Problem Definitions

In this chapter we return to the Boolean world, but without for-
getting the ideas from the previous chapter. Indeed, this chapter
presents a combination of ideas from the previous chapters: Boolean
decomposition of Chapter 4 is combined with cx and cur decom-
positions of Chapter 5 resulting to the Boolean column and Boolean
column-row decomposition problems (bcx and bcur for short).
These are the two main problems studied in this chapter. They are
defined in a natural way.

Boolean Column Decomposition Problem (bcx). Given a

binary matrix A ∈ {0, 1}n×m and a positive integer k, find k columns
of A, constituting the matrix C, and a binary matrix X ∈ {0, 1}k×m
such that C and X minimize

costbcx (A, C, X) = d1 (A − C ◦ X). (6.1)

115
116 6 The BCX and BCUR Problems

Boolean Column-Row Decomposition Problem (bcur). Giv-

en a binary matrix A ∈ {0, 1}n×m and positive integers kc and kr ,
find kc columns of A, constituting the matrix C, kr rows of A,
constituting the matrix R, and a binary matrix U ∈ {0, 1}kc ×kr
such that C, U, and R minimize

costbcur (A, C, U, R) = d1 (A − C ◦ U ◦ R). (6.2)

6.2 Applications to Data Mining

The ideas behind the bcx and bcur decompositions were taken
from the previous chapters, and, indeed, also their applications
are similar. Naturally, these decompositions are suitable only for
binary-valued data, and the usage of Boolean matrix decomposition
makes the inherent assumption that the set union is a meaningful
way of combining two columns in the data. Assuming this, the
results are supposedly easier to interpret as they are just binary
matrices, that is, they represent collections of sets and their unions.
On the other hand, the general idea of cx and cur decomposi-
tions also supports interpretability by selecting actual columns (and
rows) of the data. These ‘prototypical representatives’ could give
valuable information, for example, when selecting the objects for
further studies.

6.3 Computational Complexity of the BCX

Problem
The connection between the bcx and bu problems is of course
evident, and this connection seems to suggest that bcx is hard even
to approximate. Such a strong result, however, is something we do
not know how to prove (or even if it is true), but we can only prove a
slightly modest result of the NP-completeness of the decision version
of the bcx problem (d-bcx). Notice, that this is still more than
what is known about the counterparts of bcx, the (nonnegative)
Column Subset Selection problem (see Section 5.3).
To study the complexity of the d-bcx problem, first notice that
although the columns of C are known to be from A, this does not
6.3 Computational Complexity of the BCX Problem 117

make the bu problem any easier, not at least as long as k < n (i.e.
as long as not all columns of A appear in C). The columns of A
present in C are of course easy to cover, but recall that bu is hard
already when A has only one column. In other words, covering the
columns of A not in C, even if there is only one of them, is a hard
problem even to approximate.
Then why is this relation not enough to prove a strong inap-
proximability result to bcx as well? Because the hardness of bu is
based on the adversary selection of the columns in C, something
we can decide when solving the bcx problem. Yet, it is possible
to construct an instance of bcx where finding the optimal solution
requires solving a hard instance of bu, and this is exactly what is
done to prove Theorem 6.1. Alas, this reduction does not preserve
the approximability.
Theorem 6.1. The d-bcx problem is NP-complete.
Proof. It is obvious that d-bcx belongs to NP. To show its NP-
hardness, we need to reduce d-bu to it. To this end, let a be an
n-dimensional binary vector, C a binary matrix of size n-by-k, and
t < n a nonnegative integer such that (a, C, t) is an instance of the
d-bu problem with A having only one column. We will see how to
reduce this to an instance of d-bcx.
Create a binary matrix A′ with n + 2k rows and k(n + k) + 1
columns. Let the first n rows of the first column of A′ be equal to a,
the next k rows be full of 1s, and the last k rows be full of 0s. Copy
C at the top of the next k columns of A′ and below that place two
k-by-k identity matrices. Copy these k columns (i.e. not the first
column) n + k − 1 times. Thus the final matrix A′ contains n + k
copies of C, that is,
 
a C C
′  
A = 1k Ik · · · Ik  ,
0k Ik Ik

where 1k is a k-dimensional column vector full of 1s, 0k is a k-

dimensional column vector full of 0s, and Ik is a k-by-k identity
matrix. Let k ′ = k and t′ = t + k to complete the construction of
the instance (A′ , k ′ , t′ ).
We need to show that the d-bu instance (a, C, t) has an answer
‘yes’ if and only if the constructed d-bcx instance has an answer
118 6 The BCX and BCUR Problems

‘yes’. Let us start by assuming that x is such that d1 (A − C ◦ x) ≤ t.

We can construct C′ by taking one column from each of the n + k
identical columns that correspond to the columns of C. The first
column of X′ is x. The remaining columns of X′ can map the
columns of C′ to their identical columns introducing no cost. The
first column of X′ defines a cover of the first n rows of A′ which is
exactly the cover defined by x to a. No matter what columns of C′
are used to cover the first column of A′ , the last 2k rows introduce
error of k proving this side of the claim.
For the other direction, let C′ and X′ be such that d1 (A′ − C′ ◦
X′ ) ≤ t + k. First we show that C′ contains columns corresponding
to each column of C. For a contradiction, assume that this is not the
case. It is easy to see that to cover the n + k columns corresponding
to the missing column, one must have at least a unit error in each
column, yielding total cost of at least n + k > t + k. Thus, a column
corresponding to each column of C must be present in C′ and,
moreover, the first column of A′ cannot be in C′ . Hence, no matter
which columns of C′ are used to cover the first column of A′ , the
last 2k rows will introduce an error of k and therefore, to cover A′
with cost at most t + k, the first n rows of the first column can
introduce error of at most t.

The above proof does not preserve approximability, and hence

tells us nothing about the approximability of the bcx problem. But
while we do not know much about the approximability of bcx in
general, we do know something about a special case of it, namely, we
can show that the exact-bcx problem can be solved in polynomial
time.
Proposition 6.2. The exact-bcx problem can be solved in polyno-
mial time with respect to the instance size.
We need some new terminology before we can proceed to the
proof. A binary vector a ∈ {0, 1}n is dominated by binary vector
b ∈ {0, 1}n if ai = 0 whenever bi = 0, and we write a ≤ b. This
defines a partial ordering on {0, 1}n . If I is a subset of [n], we say
that a is dominated by b in I (and write aI ≤ bI ) if ai = 0 when
bi = 0 for all i ∈ I. If J ⊂ I, then aI ≤ bI implies aJ ≤ bJ ; in
particular, if a ≤ b, then aI ≤ bI for all I ⊆ [n]. If c ∈ {0, 1}n is
a binary vector, we say that a is dominated by b in c (ac ≤ bc ) if
aI ≤ bI for I = {i : ci = 0} (notice that it is not ci = 1).
6.3 Computational Complexity of the BCX Problem 119

Proof of Proposition 6.2. We are given a pair (A ∈ {0, 1}n×m , k)

and we need to decide whether there exists a subset of k columns of
A (denoted C) and a k-by-m binary matrix X such that A = C ◦ X.
For this, we construct the matrix C column-by-column together
with matrix X. The idea is to go through the columns of A in the
(partial) order defined by the relation ≤, defined above. We will see
that this will either give the desired matrix C or show that no such
matrix exists.
We start by selecting the column a of A that has the least
1s (assuming all columns have at least one 1 and breaking ties
arbitrarily). Vector a must be in C because if one wants to express
a as a Boolean combination of other binary vectors, a = d◦e◦· · ·◦z,
it must be that d, e, . . . , z ≤ a, and clearly there are no such vectors
in A. We continue by removing this column from A and considering
the column that has the least 1s and that does not dominate a. By
the same argument as above, this matrix must also be in C, and it is
removed from A. The next column we consider is the one with the
least 1s and that does not dominate either of the selected vectors.
This process is continued as long as there are columns that do not
dominate any of the selected columns.
Call the matrix containing the columns selected above C(1) , and
construct the respective matrix X(1) by setting (X(1) )ij = 1 if
cj(1) ≤ ai and xij = 0 otherwise. Let A′(1) = C(1) ◦ X(1) . Following
the terminology of previous chapters, we say that an element aij of
A is covered by A′ if a′ij = 1. A column of A is covered completely
if all of its 1s are covered. By the construction we know that no aij
that is 0 can be covered by A′ . In what follows we only consider
the columns of A that are not covered completely.
Among the remaining columns of A select the one that has the
least number of uncovered 1s (again breaking ties arbitrarily). Call
the selected vector a. Vector a must be in C: to be able to cover
a there must be at least one column in A, call it b, that is not in
C(1) and that is dominated by a, but such b cannot exist as by the
above construction we must have some columns d, e, . . . , z of C(1)
such that d ◦ e ◦ · · · ◦ z ≤ b ≤ a, implying that b should have less
uncovered 1s than a.
We continue by selecting the next column as the column with the
least number of 1s not covered by A′(1) and that do not dominate
a. By the same argument as above, that column must also be in
120 6 The BCX and BCUR Problems

the final matrix C. This process is continued until all remaining

columns of A dominate at least one of the recently-selected columns
(or are covered completely by A(1) ). Add the columns selected in
this phase to the columns of C(1) to obtain the matrix C(2) , and
compute the matrix X(2) as above, that is, (X(2) )ij = 1 if cj(1) ≤ ai .
We continue as above. In phase l we have matrix C(l) of columns
we know must be in the final matrix C, and matrix X(l) telling
us which 1s of A we can cover with C(l) without covering any 0s.
Hence, when we consider a column of A that has the least number
of 1s not covered by A′(l) we know, by the above argument, that it
must be in C.
The above process is followed until one of the following conditions
hold: (1) we have an exact decomposition, that is, A = C(l) ◦ X(l)
and C(l) has at most k columns; or (2) matrix C(l) has more than
k columns.
This process can be done in polynomial time. At each phase we
can compute the approximation A′(l) in polynomial time, and hence
also the number of uncovered 1s for each column of A. Because also
the relation ‘is dominated by’ is polynomially computable, we can
select each column of C in polynomial time. Selecting a column of C
is irreversible, and we cannot select more than n columns, showing
that the overall process can be done in polynomial time.

Proposition 6.2 shows that we can achieve at least some sort of

approximation guarantee for bcx, namely, the trivial nm for n-by-m
matrices A. What the best possible approximation guarantee is
remains an open problem.

6.4 The Mixing Matrix Problem

What the bu problem was to the bcx problem, the Mixing Matrix
(mm) problem is to the bcur problem. But when in bu we can, in
general, assume that the columns of C are arbitrary, in mm we must
assume that they are columns of A.
Mixing Matrix Problem (mm). Given an n-by-m binary matrix
A, an n-by-kc binary matrix C containing kc columns of A, and a
kr -by-m binary matrix R containing kr rows of A, find a kc -by-kr
binary matrix U that minimizes (6.2).
6.4 The Mixing Matrix Problem 121

In the bu problem, the columns of X are independent, and it

was this feature that made the problem applicable to both bmf and
bcx. On the other hand, neither the columns nor the rows of U are
independent in the mm problem. It is this why the columns of C
and rows of R are required to be subsets of columns and rows of
A in the definition of mm, and it is this why the naïve algorithm
for mm must try all 2kc kr different matrices U, whereas a similar
algorithm for bu can construct X column-by-column.
Assume here and henceforth that kc = kr = k. The role of U is
best clarified when looking a single element of the matrix C ◦ U ◦ R:
k
_ k _
_ k
(C ◦ U ◦ R)ij = cih ∧ (U ◦ R)hj = cih ∧ uhl ∧ rlj . (6.3)
h=1 h=1 l=1

Thus, unlike with the cx decomposition, each element of U can

potentially affect each element of C ◦ U ◦ R. Nevertheless, we have
a relation between mm and ±psc, analogous to the relation between
bu and ±psc.

Theorem 6.3. The mm problem can be reduced to the ±psc problem

in an approximation preserving way.

Proof. From (6.3) we see that the value of uhl can affect the value
of (C ◦ U ◦ R)ij – and thus the accuracy of the decomposition –
only when cih = rlj = 1. Denoting the set of interesting index pairs
by Iij = {(h, l) : cih = rlj = 1}, we may write (C ◦ U ◦ R)ij =
W
(h,l)∈Iij uhl . Now, consider an instance of the mm problem, that
is, a triplet (A, C, R). To map this to an instance of ±psc let
P = {aij : aij = 1}, N = {aij : aij = 0}, and S = {S1,1 , . . . , Sk,k }
with Shl = {aij : (h, l) ∈ Iij }. We also need to be able to recover U
from the ±psc solution, that is, from the collection C, and this is
done by letting uhl = 1 if and only if Shl ∈ C.
To prove that this reduction preserves the approximability, it is
enough to show that the cost d1 (A − C ◦ U ◦ R) is equal to the cost
induced by P , N , and C in ±psc. To this end, select an arbitrary
element aij . Consider first the case that aij = 1. We know that
(C ◦ U ◦ R)ij = 1 if and only if uhl = 1 for some (h, l) ∈ Iij , in which
case there is also a set Shl ∈ C such that aij ∈ Shl . Otherwise the
error in both mm and ±psc is increased by 1. Second, assume that
aij = 0. By construction (C ◦ U ◦ R)ij = 0 if and only if uhl = 0
122 6 The BCX and BCUR Problems

for all (h, l) ∈ Iij . This is only possible if for all Shl ∈ C it holds
that aij ∈/ Shl and neither solution introduces any error. Otherwise
both solutions must introduce a unit error. The claim follows as the
errors introduced by the solutions are equivalent for each aij .

The above reduction shows that we can use the algorithm for the
±psc problem to solve mm. If α denotes the numberpof 1s in A, the
algorithm achieves the approximation factor of 2 (k 2 + α) log α.
The reduction also gives us a way to check if the given mm instance
has a solution with zero cost, as solving that question is easy for
±psc. Henceforth we assume that the optimal solution to any mm
instance has cost at least 1.
The reduction from ±psc to mm is not that straight forward.
An additional constant factor will be lost, but that does not matter
in our asymptotic considerations. Also, the number of positive
elements |P | has no logical counterpart in mm in this reduction.
Thus, only the quasi-NP lower bound is applicable.

Theorem 6.4. The ±psc problem can be reduced to the mm problem

preserving the approximability up to constant factors.

Proof. Let (P, N, S) be an instance of ±psc with |P | + |N | = n and

|S| = k. Construct the matrices A, C, and R as follows. First make
an n-dimensional binary row vector v with vi = 1 if the ith element
is in P and vi = 0 otherwise, and create a binary nk(k + 1)-by-n
matrix V by setting each of its rows to v. Second, let a k-by-n
binary matrix T be the transpose of the incidence matrix of S. To
create the matrix A, place T below V and add an n-dimensional
row vector full of 0’s below them. Finally, add an additional column
full of 1s as the (n + 1)st column of the matrix A, yielding an
(nk(k + 1) + k + 1)-by-(n + 1) matrix
 
V Jnk(k+1)×1
 
A= T Jk×1  .
01×n 1

Let the matrix C be the last column of A, that is, the column
full of 1s, and the matrix
 R to be the last k + 1 rows of A, that
T Jk×1
is, R = 01×n 1 . Notice that as C has only one column, the
matrix U can have only one row. Therefore we can identify U as
6.5 Algorithms for the BCX, BCUR, and MM Problems 123

a row vector. To create the collection C from U, let Si ∈ C if and

only if ui = 1.
mm (U) cost±psc (C)
We need to show that if costcost∗mm ≤ r, then cost∗±psc ≤ 2r. For
that, start by considering cost∗mm . Notice first that the optimal
solution can cover the last column of A simply by setting uk+1 = 1.
It remains to cover the submatrix A = ( VT TT 0k×1 )T . Because
V has nk(k + 1) identical rows, making a single error in one of its
rows will induce an error of nk(k + 1), which is k times more than
making an error in each of the other elements. Therefore any optimal
solution must cover V optimally. But to cover V it is enough to
find a way to cover a single row v, and covering v optimally is equal
to optimally solving the ±psc instance. The proof of this is similar
to the one above.
Any error done in covering v is multiplied nk(k + 1) times in V,
and covering the rest of the matrix can add at most (k + 1)n error.
Thus we have that cost∗mm ≤ nk(k+1)cost∗±psc +(k+1)n. Analogously
to the optimal costs, the error U induces in covering V is nk(k + 1)
times the error C induces in total. Thus nk(k + 1)cost±psc (C) ≤
costmm (U). Plugging in these two inequalities, we get
costmm (U) nk(k + 1)cost±psc (C) cost±psc (C)
≥ ≥ .
cost∗mm nk(k + 1)cost∗±psc + (k + 1)n cost∗±psc + 1/k
The claim follows as the optimal cost and k are both at least 1.

The most intuitive counterpart for |S| in this reduction is the

number kr of rows in R. By the construction we have kr = |S| + 1 ≤
2 |S|. Transposing the above construction shows that the same
holds for kc . Thus the lower bound result for mm is: for any
ε > 0 it is quasi-NP-hard to approximate mm to within a factor of
1−ε 4
Ω(2log (max{kr ,kc }/2) ).

6.5 Algorithms for the BCX, BCUR, and MM

Problems
The previous results show that there is little or no hope of finding
the best possible bcx and bcur decompositions within a feasible
time, and thus the best we can hope for would be an algorithm
with some approximation guarantee. Unfortunately, the algorithms
124 6 The BCX and BCUR Problems

presented here do not come with any approximation guarantees,

notwithstanding the lack of any lower bounds for approximability.
This increases the importance of careful empirical evaluation of the
algorithms, and such evaluation will be carried out in forthcoming
sections.

6.5.1 The LocBCX Algorithm

A simple local-search algorithm would work as follows: the starting

solution would be a random sample of columns of A constituting
the matrix C, and the neighbourhood would be the set of matrices
obtained by changing exactly one of C’s columns. The LocBCX
algorithm, however, implements a certain level of greediness in it.
In LocBCX, columns of C are considered one-by-one. When column
i is considered, the local search is applied to the neighbourhood
consisting of all matrices obtained from C by changing column i to
some other column, not currently in C (as opposed to considering
changing any column of C to any other column). When all columns
are considered, the process is restarted if any of the columns was
changed, and stopped if no upgrades were possible. This type of
local search was chosen after initial tests, where it was noticed to
converge faster and – somewhat surprisingly – to better solutions
than the more straight forward implementation.
Akin to the LocNCX algorithm (see Section 5.4), also the LocBCX
algorithm needs to solve the cost induced by a specific matrix C
multiple times during one local search. But unlike with LocNCX,
computing the cost of a specific matrix C exactly is hard even in
theory, requiring to solve the bu problem exactly. The solution is,
as with the Asso algorithm (Algorithm 2), to use the function cover
to give a rough estimate of the matrix X.
The LocBCX algorithm returns matrices C and X, but as with the
Asso algorithm, the quality of X can be improved after the columns
of C are fixed. As this is exactly the bu problem, the algorithms
used to solve it are those presented in Section 3.6, namely PelegRB
and IterX. The combinations of these two algorithms with LocBCX
are referred to as LocBCX+PelegRB and LocBCX+IterX.
6.5 Algorithms for the BCX, BCUR, and MM Problems 125

6.5.2 The LocBCUR Algorithm and Solving the MM Problem

As the ncur and bcur problems are very much alike, similar ideas
can be applied to both of them. Thus the simplest idea to solve the
bcur problem would be to first solve C using LocBCX, then solve R
using LocBCX again, now to a transposed matrix, and finally solve
U using PelegRB via the reduction used to prove Theorem 6.3.
This approach has certain drawbacks. First of them is that the
reduction from Theorem 6.3 increases the instance size easily beyond
what is practically feasible. This, combined with the generally slow
performance of PelegRB (see Section 3.6.1), makes the final step
of the algorithm a bottleneck. The approach used in IterX does
not immediately apply here, as the columns (or the rows) of U are
not independent, and there is no counterpart to the cover function.
Instead, a simpler iterative approach is possible. Starting with a
matrix U full of 0s, each element uij is considered, and the value of
uij is changed (from 0 to 1 and vice versa in later iterations) if it
reduces the reconstruction error. The process is restarted from u1,1
until it converges. This algorithm is referred to as IterU.
Compared to IterX, IterU requires more elements to be updated
in each iteration, but given the small size of U (kc -by-kr ), this is
usually still feasible.
An alternative way to solve the mm problem is to take advantage
of the structure of the Boolean cur decomposition. Recall from (6.3),
that element uhl can determine the value of (C ◦ U ◦ R)ij only if
cih = rhj = 1. As in Section 6.4, denote the set of index pairs (h, l)
such that uhl can change the value of (C ◦ U ◦ R)ij by Iij , that is,
Iij = {(h, l) : cih = rlj = 1}.
The algorithm starts with matrix Ũ full of 0s and iterates over
aij s. If aij = 0 it decreases the value of ũhl by 1 for each (h, l) ∈ Iij ,
and if aij = 1 it increments the value of ũhl by some fixed b ∈ ]0, 1] in
the same positions. When this is done, U is constructed by setting
uij = 1 if ũij > 0, and uij = 0 otherwise. The balancing variable
b is needed because if aij = 0, then for all (h, l) ∈ Iij , uhl must
be 0, but if aij = 1 it is enough that uhl = 1 for one (h, l) ∈ Iij .
Unfortunately there does not seem to be any other way to select the
balancing variable b than trying out different values. This algorithm
is known as Maj, referring to its idea of selecting uhl s based on a
(scaled) majority voting.
126 6 The BCX and BCUR Problems

Another problem with the naïve way of solving bcur is that

columns of C and rows of R are selected independently while in the
resulting decomposition they depend on each other heavily. The
solution to this is, of course, a local-search heuristic similar to
LocNCUR, where the neighbourhood contains neighbours of C and
neighbours of R.
The problem with this approach is in updating the matrix U to
compute the reconstruction error for each neighbour. In LocNCUR a
new U was always computed, using SolveNU, but here using either
IterU or Maj to compute U from scratch can be time-consuming.
The answer is to exploit the local structure of possible changes,
similar to what was done in Maj. The algorithm, again, starts with
random C and R. It then computes U for these matrices using Maj,
and keeps also Ũ used in Maj. It then works as follows. Assume
a column ch is changed to c̃h . We only need to consider rows i
where cih 6= c̃ih . Assume i is such a row and j is an arbitrary
row. All elements of Ũ that need to be considered for this pair
(i, j) are those from row h and column l such that (h, l) is either
in Iij = {(h, l) : cih = rlj = 1} or in I˜ij = {(h, l) : c̃ih = rlj = 1}.
Assuming (h, l) is such, the update rules are as follows. If cih = 0
and c̃ih = 1, then let

ũ +b if aij = 1
hl
ũhl =
ũhl − 1 if aij = 0,

and if cih = 1 and c̃ih = 0, then let


ũ
hl −b if aij = 1
ũhl =
ũhl + 1 if aij = 0.

The same procedure applies mutatis mutandis when a row of R is

changed instead of a column of C.
The algorithm implementing the local search for bcur with the
above updates to U is called LocBCUR.

6.5.3 Time Complexity

Analysing the time complexity of the above algorithms requires

solving two questions: what is the time a single local improvement
6.5 Algorithms for the BCX, BCUR, and MM Problems 127

step takes and what is the maximum number of steps that can be
taken. The latter question is easy to answer: if the input matrix
is n-by-m, then the maximum error any approximation can cause
is nm, and as each local improvement is guaranteed to improve
the result by at least 1, the maximum number of improvements is
nm (cf. Section 4.7). This is, of course, a very coarse upper bound,
and results in rather pessimistic upper bounds. Tighter analysis is,
however, hard to achieve. As the maximum number of steps is the
same for all algorithms, the time each algorithm needs to compute
a single step becomes even more interesting.
For LocBCX, a single iteration consist of trying all columns of
C one-by-one and trying to change them with some other column.
Assume the input matrix is n-by-m. There are k columns in C,
and m − k other columns to try. For each possible change, we
need to compute the matrix X, taking time O(kmn), which also
encompasses the time needed to compute the error and other book-
keeping required. Thus the time complexity for one iteration is


O(k(m − k)kmn) = O(k 2 mn(m − k)) yielding O (knm)2 (m − k)
total time complexity.
One iteration of IterU requires going through the matrix U
(kc kr steps), and computing the product C ◦ U ◦ R in every step
(O(nkc kr m) time), though in practice this can be improved by
considering only those elements of the product that will change.
With nm as the maximum number of iterations, the total time


complexity is O (nmkc kr )2 .
The Maj algorithm is different from other algorithms in that it
does not need any iterations, and thus its time complexity is at most
O(nkc kr m), and in practice we can assume |Iij | ≪ kc kr for most
Iij s.
For the LocBCUR, the neighbourhood structure is bigger: each of
the kc columns of C can be changed to (m − kc ) other columns and
each of the kr rows of R can be changed to (n − kr ) other rows,
totalling kc (m − kc ) + kr (n − kr ) neighbours to try. The algorithm
was devised to utilize the fact that usually |Iij | ≪ kc kr , but the
worst-case bound must still assume |Iij | = kc kr , yielding O(nkc kr m)


time for each neighbour, and O (kc (m − kc ) + kr (n − kr ))kc kr (nm)2
in total.
128 6 The BCX and BCUR Problems

6.6 Experimental Evaluation

As the Boolean cx and cur decompositions are natural combinations
of the problems in the previous two chapters, the experiments
are similar to the previous ones. Their purpose and settings are
familiar: synthetic data is used to study the effects of various data
characteristics in well-controlled settings, and real data is used to
verify these results and provide results about the interpretability of
the algorithms’ results.

6.6.1 Algorithms Used

Except for the algorithms presented in this chapter, all other al-
gorithms used in the experiments are familiar from the previous
chapters; indeed, to the best of the author’s knowledge, there are
no algorithms for the bcx (and bcur) decompositions except those
presented here.
The bcx decomposition is a Boolean column decomposition, and
thus algorithms for it were compared against algorithms for Boolean
decomposition (i.e. bmf) and for column decomposition (cx). For
the former, Asso+IterX was used; for the latter, 2Step and SPQR
were used. In addition, svd was used to provide a benchmark.
As with bmf, there are certain problems when comparing results
from real-valued methods such as 2Step or SPQR to results from
Boolean methods (see Section 4.8.2). Also following the experiments
with bmf, two versions of real-valued methods were used, where
the results of the modified versions, SVD0/1 , 2Step0/1 , and SPQR0/1 ,
were rounded to binary matrices. Once again, it should be noted
that these algorithms are still solving a different problem, and thus
they do not tell much about how good an optimal bcx (or bcur)
decomposition would be.

6.6.2 Synthetic BCX Experiments

The synthetic bcx data and the parameter values were the same that
were used in Sections 3.7 and 4.8.3. Yet there was one difference. In
the aforementioned sections the input data for the algorithms was of
the form A′ = B ◦ X, but to satisfy the setting of bcx, the columns
of B (which, for the sake of consistency, is henceforth referred to
6.6 Experimental Evaluation 129

5000 2Step0/1 2Step0/1

Local+IterX Local+IterX
Local+PelegRB SPQR0/1 70 Local+PelegRB SPQR0/1
4000 Asso+IterX SVD0/1 Asso+IterX SVD0/1
60
d1(A−C°X)

dF(A−C°X)
3000
50
2000
40
1000
30
0
8 10 12 14 16 18 20 22 24 26 28 8 10 12 14 16 18 20 22 24 26 28
k k
(a) (b)
Figure 6.1: Reconstruction errors of bcx decomposition when k, the
number of columns in C, varies with (a) d1 and (b) dF distances.
Markers are mean values over twenty instances, and the width of
error bars is two times the standard deviation. Matrix multiplication
is Boolean for LocBCX+IterX, LocBCX+PelegRB, and Asso+IterX,
and normal for the other methods.

as C) were appended as the first k columns of A′ yielding the final

A used in the experiments. In other words, when matrices C and
X were generated as in Section 3.7.1, matrix X′ was set to (Ik X),
and A was constructed as C ◦ X′ . The varied parameters and their
values were the same as in the aforementioned sections.
Number of columns in C. The number of columns in C, k,
varied from 8 to 28 with steps of size 2. The total density of
the resulting matrices was approximately 0.35. The results are
presented in Figure 6.1. One can see that, first, increasing k does
not, in general, seem to have adversarial effects to the proposed
algorithms, and second, that LocBCX+IterX performs very well:
with d1 distance (Figure 6.1(a)) it is second to none but SVD0/1 , and
even with Frobenius distance it is the second-best up until k = 26
when 2Step and SPQR reach its level.
Notice that LocBCX+IterX is constantly better than Asso+IterX,
even though the bmf decomposition presents a theoretical lower
bound for the bcx decomposition. These results seem to suggest
that when the data has the inherent bcx structure LocBCX+IterX,
knowing what to look for, is able to find it while Asso+IterX fails to
find a comparable general bmf decomposition. The LocBCX+PelegRB
is the worst of all algorithms, and this is in line with the previous
results for PelegRB. (cf. Sections 3.7 and 4.8.3).
130 6 The BCX and BCUR Problems

SPQR0/1 SPQR0/1
6000 Local+IterX
80
Local+IterX
Local+PelegRB SVD0/1 Local+PelegRB SVD0/1
Asso+IterX Asso+IterX
5000 2Step0/1 2Step0/1
60
d1(A−C°X)

dF(A−C°X)
4000

3000 40
2000
20
1000

0 0
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
noise level noise level
(a) (b)
Figure 6.2: Reconstruction errors of bcx decomposition when the
level of noise varies with (a) d1 and (b) dF distances. Markers are
mean values over twenty instances, and the width of error bars is
two times the standard deviation. Matrix multiplication is Boolean
for LocBCX+IterX, LocBCX+PelegRB, and Asso+IterX, and normal
for the other methods.

Noise level. The level of noise varied from 0 to 0.4 with steps
of size 0.05. The results are presented in Figure 6.2. Increasing noise
has the expected effect to the error of all methods – it increases. But
again LocBCX+IterX performs very well, especially with d1 distance
(Figure 6.2(a)), where it is the second-best method (next to only
SVD0/1 ) up until 30% of noise; with dF distance (Figure 6.2(b)) it
is second-best until 15% of noise, but constantly the best Boolean
method, being always better than Asso+IterX.
Density of columns of B. The mean density (number of 1s
divided by the total number of elements) of columns of B varied
from 0.05 to 0.3. The total density of the resulting matrices varied
from approximately 0.2 to approximately 0.7. The results can be
seen in Figure 4.3.
Increasing the density decreases other algorithms’ errors some-
what, but LocBCX+IterX seems to be immune to it, being second
only to SVD (and SVD0/1 ) in all evaluation points and with both error
measures.
Mean number of columns of B used for each column of A.
The mean number of columns of B involved in the Boolean com-
binations used to create columns of A had three possible values, 4,
6, and 8. The resulting matrices had an approximate total density
between 0.35 and 0.55. As above, LocBCX+IterX is again the second
6.6 Experimental Evaluation 131

Local+IterX SPQR0/1 Local+IterX SPQR0/1

7000 Local+PelegRB SVD0/1 80 Local+PelegRB SVD0/1
Asso+IterX Asso+IterX
6000 2Step0/1 2Step0/1
70
d1(A−C°X)

dF(A−C°X)
5000
60
4000
50
3000
40
2000
1000 30
0.05 0.1 0.15 0.2 0.25 0.3 0.05 0.1 0.15 0.2 0.25 0.3
density density
(a) (b)
Figure 6.3: Reconstruction errors of bcx decomposition when the
density varies with (a) d1 and (b) dF distances. Markers are mean
values over twenty instances, and the width of error bars is two
times the standard deviation. Matrix multiplication is Boolean for
LocBCX+IterX, LocBCX+PelegRB, and Asso+IterX, and normal for
the other methods.

best method with both distance measures (Figure 6.4), and totally
immune to this parameter. Most of the other methods also per-
form with constant quality, but Asso+IterX’s error increase slightly
and LocBCX+PelegRB’s error increases linearly, as was expected (cf.
Section 3.7).

6.6.3 Synthetic BCUR Experiments

The synthetic bcur data was similar to synthetic ncur data (see
Section 5.6.3), except that it was binary-valued and Boolean matrix
multiplication was used in the generation process; the exact genera-
tion process is described below. The phenomena studied with these
data were analogous to those in previous experiments, namely the
effects of (1 ) the number of columns in C and the number of rows
in R; (2 ) the noise level; and (3 ) the density of C and R.
Data generation process. For a fair experiment, the synthetic
data was generated so that it had an exact bcur decomposition
before noise was added. To achieve this, the data generation process
used a special type of bcur decomposition, similar to that used
in Section 5.6.3. First, the parameters n and m were set to 150
and 80, respectively. The final matrices had n + k rows and m + k
columns, where k is the number of columns in C, which was equal
132 6 The BCX and BCUR Problems

80 2Step0/1
2Step0/1 Local+IterX
Local+IterX
SPQR0/1 Local+PelegRB SPQR0/1
5000 Local+PelegRB
70 SVD0/1
Asso+IterX SVD0/1 Asso+IterX

dF(A−C°X)
d1(A−C°X)

4000 60

3000 50

2000 40

1000 30
4 6 8 4 6 8
mean number of base vectors per column of A mean number of base vectors per column of A

(a) (b)
Figure 6.4: Reconstruction errors of bcx decomposition when the
mean number of C’s columns used for each column of A varies
with (a) d1 and (b) dF distances. Markers are mean values over
twenty instances, and the width of error bars is two times the stand-
ard deviation. Matrix multiplication is Boolean for LocBCX+IterX,
LocBCX+PelegRB, and Asso+IterX, and normal for the other meth-
ods.

to the number of rows in R. The matrices C and R had a special

structure, too:
!
Ik  
C= and R = Ik X′ k×m .
Xn×k

Matrices Xn×k and X′ k×m are n-by-k and k-by-m random matrices
with elements sampled independently from the Bernoulli distribution.
The mixing matrix was a k-by-k identity matrix Ik , and the (noise-
free) data matrix was constructed as A = C◦R. The first k columns
of A are the columns of C and the first k rows the rows of R. Noise
was added by flipping the values of randomly selected elements.
Twenty matrices were made for each configuration of parameters,
and the default values for parameters were k = 16, 10% of noise,
and density of 0.1.
Number of columns in C and rows in R. Number of columns
in C (as well as the number of rows in R) varied from k = 8 to
k = 28 with steps of size 2. The density of the resulting matrices
varied between 0.14 and 0.26.
The results, presented in Figure 6.5, show that this parameter
has small, yet recognizable effect to LocBCUR and to LocBCXT +Maj.
These two algorithms report exactly the same results in each data
6.6 Experimental Evaluation 133

10000 LocalBCUR 100 LocalBCUR

LocalT+IterU LocalT+IterU
T
Local +Maj
90 LocalT+Maj
8000 Asso+IterX Asso+IterX
2StepT0/1 80
d1(A−C°X)

dF(A−C°X)
2StepT
SPQRT0/1 SPQRT
6000 70 SVD
SVD0/1
60
4000
50
2000 40

8 10 12 14 16 18 20 22 24 26 28 8 10 12 14 16 18 20 22 24 26 28
k k
(a) (b)
Figure 6.5: Reconstruction errors of bcur decomposition when the
number of columns in C and rows in R, k, varies with (a) d1 and
(b) dF distances. Markers are mean values over twenty instances,
and the width of error bars is two times the standard deviation.
Matrix multiplication is Boolean for LocBCUR, LocBCXT +IterU,
LocBCXT +Maj, and Asso+IterX, and normal for the other meth-
ods.

points. The third algorithm, LocBCXT +IterU, faces considerable

problems when k increases over 24; no intuitive explanation was
found for this sudden and sharp increase in error.
All these three methods lose to the methods they were compared
to. The other methods, of course, are not solving bcur, and
thus these results can be considered to point out the hardness of
finding a good bcur decomposition. The hardness of finding good
bcur (or cur) decompositions is further emphasized by the good
performance of Asso+IterX. It is worth noting, however, that the
relative performances of the methods are almost identical with d1
and dF distances.
Noise level. The level of noise varied from 0 to 0.4 with steps
of size 0.05. The resulting matrices’ density varied between 0.11 and
0.43. The results are presented in Figure 6.6.
The results are similar to those with changing k: LocBCUR and
LocBCXT +Maj are (almost) indistinguishable, but LocBCXT +IterU’s
error increases faster with high values of noise. Again, the proposed
methods lose to the other methods, yet this time all algorithms
(except, perhaps, SVD) follow a very similar trend, and the difference
between, say, LocBCUR and SPQRT0/1 is relatively small, at least with
d1 distance.
134 6 The BCX and BCUR Problems

LocalBCUR LocalBCUR SPQRT

8000 LocalT+IterU LocalT+IterU SVD
T 80
Local +Maj LocalT+Maj
Asso+IterX Asso+IterX
2StepT0/1
d1(A−C°X)

dF(A−C°X)
6000 60 2StepT
SPQRT0/1
SVD0/1
4000 40

2000 20

0 0
0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4
noise level noise level
(a) (b)
Figure 6.6: Reconstruction errors of bcur decomposition when the
level of noise varies with (a) d1 and (b) dF distances. Markers are
mean values over twenty instances, and the width of error bars is
two times the standard deviation. Matrix multiplication is Boolean
for LocBCUR, LocBCXT +IterU, LocBCXT +Maj, and Asso+IterX, and
normal for the other methods.

Density of C and R. The density of the random parts of C and

R varied from 0.05 to 0.3; the density of the resulting matrices varied
between 0.12 and 0.65. The results, presented in Figure 6.7, show
that density’s effects to the proposed algorithms are not straight
forward.
The results also rule out one explanation for LocBCXT +IterU’s be-
haviour in previous experiments: denser matrices cause problems to
LocBCXT +IterU. Unlike in previous experiments, here LocBCXT +IterU
is the best of the proposed methods, and its results even improve
with increased density. The growth in the error caused by LocBCUR
starts to flatten when the error of LocBCXT +IterU starts to de-
crease, but the error caused by LocBCXT +Maj peaks at that very
point. The results of 2StepT0/1 and SPQRT0/1 follow the same trend
as LocBCXT +IterU (Figure 6.7(a)), as do the results of 2StepT and
SPQRT when the dF distance is used (Figure 6.7(b)).
Conclusions. The experiments with synthetic data showed that
the proposed algorithms behave much as was expected: increasing
noise, as well as increasing the number of columns and rows in
the factor matrices, increases the error. Density’s effects to the
proposed algorithms were not as straight forward, but it is possible
that the most problematic case for the algorithms is when the data
has density about 1/2.
6.6 Experimental Evaluation 135

SPQRT0/1 80
LocalBCUR LocalBCUR SPQRT
6000 SVD
LocalT+IterU SVD0/1 LocalT+IterU
LocalT+Maj 70 LocalT+Maj
5000 Asso+IterX Asso+IterX
d1(A−C°X)

dF(A−C°X)
2StepT0/1 2StepT
60
4000

3000 50

2000 40

0.05 0.1 0.15 0.2 0.25 0.3 0.05 0.1 0.15 0.2 0.25 0.3
density density
(a) (b)
Figure 6.7: Reconstruction errors of bcur decomposition when
the density of C and R varies with (a) d1 and (b) dF distances.
The values of x-axes refer to the success probability of Bernoulli
distribution used to create the matrices, not to the actual density of
the data. Markers are mean values over twenty instances, and the
width of error bars is two times the standard deviation. Matrix mul-
tiplication is Boolean for LocBCUR, LocBCXT +IterU, LocBCXT +Maj,
and Asso+IterX, and normal for the other methods.

The proposed methods were inferior to the other methods in all

experiments. As stated before, one must remember that the other
methods are solving different problems, and thus straight forward
conclusions about the quality of the proposed algorithms cannot be
made. It seems, however, that finding a good bcur decomposition
is a very challenging task, although it is possible that even with
the optimal bcur decomposition one could not attain much smaller
reconstruction errors than those of the proposed algorithms.

6.6.4 Experiments with Real-World Data

The experiments done with the real-world data sets are analogous
to those in previous chapters. The goals were to verify that the
algorithms’ behaviour with real-world data follows that observed
with synthetic data, to study the significance of the results using
randomization, and to study the interpretability of the results. The
four real-world data sets used were 4News, DBLP, Dialect, and NOW.
The last three were identical to those explained in Section 4.8.4.
The first data set was explained in Section 5.6.5 but for the purposes
of this chapter, all values greater than 0 were set to 1.
136 6 The BCX and BCUR Problems

The results are reported in the following order. First, the re-
construction errors for different decomposition sizes are listed. The
significance of these results is studied next, and the last part contains
the discussion about the interpretability of the results.
Reconstruction errors. The reconstruction errors with respect
to d1 distance are given in Table 6.1, and with respect to dF distance
in Table 6.2.
Of these two tables, Table 6.1 is naturally more interesting. The
first thing to note there is that LocBCX’s results do not improve
when it is combined with IterX, except with Dialect data, where
LocBCX+IterX performs much better than plain LocBCX. As could
be expected from the previous results, LocBCX+PelegRB’s results
are inferior to both LocBCX and LocBCX+IterX.
Interestingly, LocBCX’s (and LocBCX+IterX’s) results with 4News
are better than the results of Asso+IterX when k is 5 or 10, albeit
the latter should present a lower bound for the former. This suggests
that 4News has a strong latent bcx structure which the LocBCX
algorithm was able to find. In other data sets, Asso+IterX is better
than (or at least as good as) any bcx algorithm, but one could expect
an even larger marginal between LocBCX+IterX and Asso+IterX.
The rounded 2Step algorithm, 2Step0/1 , shows good performance
over all datasets. In the first four rows of Table 6.1 its reconstruction
error is bigger than the error caused by LocBCX+IterX, but in the
remaining rows it is smaller. The rounded SPQR, SPQR0/1 , on the
other hand, has more variations in its results against LocBCX+IterX:
with 4News it is constantly worse, with DBLP and k = 5 it causes
enormous error, but with the same data and k = 15, it has very
small reconstruction error, being more than ten times better than
LocBCX+IterX. The rounded SVD is superb in its reconstruction
error when compared against all other methods; a sole exception to
this is DBLP data with k = 15.
The case of DBLP data set with k = 15 is interesting for other
reasons, as well. Namely, all studied methods obtain major decrease
in the reconstruction errors of the data when k increases from 10
to 15. For LocBCX the error with k = 10 is almost 4 times the error
with k = 15, while for SPQR0/1 the error with k = 10 is 22.5 times
the error with k = 15.
When the dF distance is used instead of d1 distance, the results
change only in that the continuous methods, namely, 2Step, SPQR,
 6.6 Experimental Evaluation
Table 6.1: Reconstruction errors of bcx and cx decompositions with real-world data and d1 distance.

Data k LocBCX LocBCX LocBCX Asso 2Step0/1 SPQR0/1 SVD0/1

+IterX +PelegRB +IterX
4News 5 1807 1807 1876 1814 1916 1972 1701
10 1605 1602 1731 1613 1737 1780 1348
15 1439 1439 1585 1431 1449 1539 1075
DBLP 5 9250 9250 9537 8676 9827 14420 8384
10 5423 5423 5602 4503 5367 2727 3863
15 1519 1519 1519 1519 774 120 988
Dialect 5 59192 55759 107858 55057 53967 67677 40846
10 54018 43952 106560 41649 42339 52457 28136
15 49606 36850 105557 35825 35318 42972 22379
NOW 5 1625 1625 1881 1569 1664 1701 1379
10 1450 1446 1791 1396 1427 1384 1020
15 1332 1310 1717 1272 1208 1173 760

137
 138
Table 6.2: Reconstruction errors of bcx and cx decompositions with real-world data and dF distance.

Data k LocBCX LocBCX LocBCX Asso 2Step SPQR SVD

+IterX +PelegRB +IterX
4News 5 42.51 42.51 43.31 42.59 39.86 39.32 36.25
10 40.06 40.02 41.61 40.16 36.69 36.47 33.04
15 37.93 37.93 39.81 37.83 33.94 34.16 30.38
DBLP 5 96.18 96.18 97.66 93.15 87.49 101.44 81.01

6 The BCX and BCUR Problems

10 73.64 73.64 74.85 67.10 62.85 73.48 58.27
15 38.97 38.97 38.97 38.97 37.70 49.10 33.11
Dialect 5 243.29 236.13 328.42 234.64 205.73 228.46 177.72
10 232.42 209.65 326.44 204.08 181.82 202.03 152.09
15 222.72 191.96 324.90 189.27 165.67 184.30 137.53
NOW 5 40.31 40.31 43.37 39.61 36.05 36.12 31.93
10 38.08 38.03 42.32 37.36 32.76 32.60 28.46
15 36.50 36.19 41.44 35.67 29.92 29.90 25.84
6.6 Experimental Evaluation 139

and svd, perform consistently better than the Boolean ones. Never-
theless, LocBCX+IterX performs relatively well even when compared
to svd.
The reconstruction errors of bcur and cur decompositions are
reported in Tables 6.3 (for d1 ) and 6.4 (for dF ). No bcur (or cur)
decomposition was done to the DBLP data as it has only 19 rows.
The LocBCUR algorithm was considered to take too much time with
the Dialect data, and it was omitted. Overall, it was clearly the
slowest method.
With 4News data and when k was 5 or 10, the LocBCUR algorithm
was slightly worse than LocBCXT +IterU; when k = 15, it was slightly
better. With NOW data, LocBCUR gave the best bcur decompos-
ition. Between LocBCXT +IterU and LocBCXT +Maj the former was
constantly better than the latter, albeit the difference was sometimes
very small.
On the contrary to the bcx results, the 2StepT0/1 algorithm was
not constantly better than the best of bcur algorithms. Indeed, it
presented smaller error only in three cases: with Dialect and k = 15,
and with NOW and k being 10 and 15. The results for SPQRT0/1 were
similar, it being better than the best of bcur algorithms only with
NOW and k = 15. What was said also holds if we do not consider
LocBCUR at all, that is, if we compare 2StepT0/1 and SPQRT0/1 to
LocBCXT +IterU only.
Perhaps the most interesting result with the dF distance is that
both LocBCXT +IterU and LocBCXT +Maj are better than SPQRT with
Dialect data and k being 5 or 10.
Randomization results. The data was randomized using the
swap randomization method, as described in Section 4.8.5. Only d1
distance was considered, as it is the distance measure used by the
proposed algorithms. The results of the randomization experiments
tell in how many random data sets the algorithm was able to perform
better than it performed on the original data.
The results for bcx and rounded cx decompositions are given
in Table 6.5. The first thing to notice is LocBCX+PelegRB’s column:
the randomized data yields smaller reconstruction error in almost
every case. But this is not as surprising as it might look at first
sight. Recall that the approximation guarantee of PelegRB depends
on the number of 1s in the columns of the data matrix, and as was
shown in Section 3.7, the algorithm’s performance actually depend
 140
Table 6.3: Reconstruction errors of bcur and cur decompositions with real-world data and d1 distance. Missing
results are denoted by —.

Data k LocBCUR LocBCXT LocBCXT Asso 2StepT0/1 SPQRT0/1 SVD0/1

+IterU +Maj +IterX
4News 5 1912 1896 1908 1814 2082 2148 1701
10 1794 1755 1773 1613 1865 2001 1348

6 The BCX and BCUR Problems

15 1642 1651 1652 1431 1660 1792 1075
Dialect 5 — 58799 59034 55057 64944 80452 40846
10 — 48438 50817 41649 49773 65746 28136
15 — 45160 52527 35825 42009 53722 22379
NOW 5 1743 1874 1880 1569 1882 1920 1379
10 1658 1740 1804 1396 1633 1738 1020
15 1623 1731 1753 1272 1503 1545 760
 6.6 Experimental Evaluation
Table 6.4: Reconstruction errors of bcur and cur decompositions with real-world data and dF distance. Missing
results are denoted by —.

Data k LocBCUR LocBCXT LocBCXT Asso 2StepT SPQRT SVD

+IterU +Maj +IterX
4News 5 43.73 43.54 43.68 42.59 42.03 42.86 36.25
10 42.36 41.89 42.11 40.16 39.90 40.54 33.04
15 40.52 40.63 40.64 37.83 37.44 38.71 30.38
Dialect 5 — 242.49 242.97 234.64 225.64 247.09 177.72
10 — 220.09 225.43 204.08 201.44 226.12 152.09
15 — 212.51 229.19 189.27 186.04 209.24 137.53
NOW 5 41.75 43.29 43.36 39.61 38.60 38.97 31.93
10 40.72 41.71 42.47 37.36 35.67 36.59 28.46
15 40.29 41.61 41.87 35.67 33.72 34.37 25.84

141
142 6 The BCX and BCUR Problems

on this. The randomization technique applied keeps the marginal

sums constant, in other words, the number of 1s in data matrix’s
columns stays constant over the random samples. But what is
surprising, is that in many cases, the LocBCX+PelegRB algorithm
performs worse in the original data than it performs in any of the
random samples.
The LocBCX+PelegRB algorithm is not the only one performing
better in the randomized versions of the DBLP data. For k equals to
5 and 10, the results of LocBCX, LocBCX+IterX, and 2Step0/1 can
be considered to be an artifact of the marginal sums more than the
other structures of the data (or, at least, such hypothesis cannot be
rejected). But, interestingly enough, with k = 15, all these methods,
LocBCX+PelegRB included, have the reconstruction error with the
original data smaller than with any randomized version of the data.
This is perhaps explained by the huge drop in reconstruction errors
when k increases from 10 to 15 (cf. above). For other data sets we
can safely assume that the algorithms, LocBCX+PelegRB excluded,
did not found their results only because the marginal sums of the
data.
The results for the randomization tests with bcur and rounded
cur decompositions are given in Table 6.6. There, the results
for which we cannot reject the null hypothesis are those obtained
for 4News data using LocBCXT +IterU, LocBCXT +Maj, and 2StepT0/1 ,
albeit for 2StepT0/1 this is only true for k = 5, and – depending on
the level of confidence we want to have – perhaps also for k = 10.
No straight forward reasons were found for why these algorithms
work so well with the randomized versions of 4News.
Interpretability. Not all data sets, nor all algorithms, were used
for interpretability tests. For example, interpreting the results from
the NOW data would require knowledge on palaeontology, and the
results from LocBCX+PelegRB are inferior in reconstruction error
and, according to the randomization tests, of questionable statistical
significance.
The first results are from 4News data, using LocBCX. As the data
is a terms-by-documents matrix, the bcx decomposition of it would
select some documents in C. The actual documents selected are
not the only thing that is interesting here, but also the newsgroups
to which the selected documents belong. Assuming all newsgroups
have their own special vocabulary, and the news have approximately
 6.6 Experimental Evaluation
Table 6.5: Percentage of random data sets having smaller bcx or cx reconstruction error than the original one, d1
distance. Missing results are denoted by —.

Data k LocBCX LocBCX LocBCX Asso 2Step0/1 SPQR0/1 SVD0/1

+IterX +PelegRB +IterX
4News 5 0 0 65 0 0 0 0
10 0 0 85 0 0 0 0
15 0 0 79 0 0 0 0
DBLP 5 100 100 100 0 49 0 0
10 100 100 100 0 82 0 0
15 0 0 0 0 0 0 0
Dialect 5 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0
NOW 5 0 0 100 0 0 0 0
10 0 0 100 0 0 0 0
15 0 0 100 0 0 0 0

143
 144
Table 6.6: Percentage of random data sets having smaller bcur or cur reconstruction error than the original one,
d1 distance. Missing results are denoted by —.

Data k LocBCUR LocBCXT LocBCXT Asso 2StepT0/1 SPQRT0/1 SVD0/1

+IterU +Maj +IterX
4News 5 1 12 38 0 83 0 0
10 0 17 33 0 1 0 0

6 The BCX and BCUR Problems

15 0 17 15 0 0 0 0
Dialect 5 — 0 0 0 0 0 0
10 — 0 0 0 0 0 0
15 — 0 0 0 0 0 0
NOW 5 0 0 0 0 0 0 0
10 0 0 0 0 0 0 0
15 0 0 0 0 0 0 0
6.6 Experimental Evaluation 145

Table 6.7: Newsgroups of posts in C of bcx decomposition of 4News

data. The LocBCX algorithm.

k=4 k=6 k=8

christian christian christian
med crypt crypt
crypt christian christian
crypt crypt crypt
space space
med christian
med
space

the same frequency of terms over the newsgroups, with k = 4 we

could assume the algorithm to select one post from each group. The
results for various values of k are presented in Table 6.7.
As can be seen from Table 6.7, the assumption fails: for k = 4,
the algorithm selects two news posts from sci.crypt and none from
sci.space. Newsgroups sci.crypt and soc.religion.christian
tend to appear more often than sci.med and sci.space when k is
increased. Why is this? The assumption of equidense submatrices
is false. The submatrix containing the soc.religion.christian
newsgroup has density of approximately 0.08 and the submatrix
containing the sci.crypt newsgroup has density of approximately
0.07, but the submatrices containing sci.med and sci.space have
densities of approximately 0.05. Thus the number of posts selec-
ted from different newsgroups is not equal, but rather reflects the
underlying density.
One could infer the topics of two or three of the newsgroups
given the documents selected by LocBCX with k = 8: an article
with words ‘encrypt’, ‘public’, and ‘chip’ suggests the cryptography
quite clearly; an article with words ‘faith’, ‘Christian’, and ‘God’
the Christian religion; and an article that could suggest space as
the topic of one of the newsgroups was a laconic single-word article:
‘space’. Cryptography and Christian religion were also clear topics
of two other posts, whereas the remaining three posts contained
more general words. Although one of the posts was from sci.med,
the terms present in it did not suggest specifically medical content.
146 6 The BCX and BCUR Problems

space man
zoo
peopl chip
space
effect public
christian effect
christian
system escrow
system
escrow peopl
crypt med space christian crypt med space christian
(a) (b)
Figure 6.8: Terms selected, and their appearance, by LocBCX in bcx
decomposition of a transpose of 4News. Labels on y axis represent
the selected terms, and labels on x axis the newsgroups. (a) k = 6;
(b) k = 10.

Lack of clearly medical terminology was also recognized below when

the terms were selected instead of documents.
The LocBCX algorithm was executed with transposed 4News as
input, this time to select some terms. The submatrices of the
selected terms for k = 6 and k = 10 are presented in Figure 6.8. The
algorithm clearly selects both terms specific to a single newsgroup
and terms appearing frequently in all newsgroups. For example,
from Figure 6.8(a) we can see that the term ‘space’ – not surprisingly
– appears almost solely in newsgroup sci.space, whereas the term
‘peopl[e]’ appears in all newsgroups.
One should be able to identify the topics of two newsgroups
given Figure 6.8(a) (without x labels, of course); the terms ‘space’
and ‘christian’ are rather suggestive, and define the borders of the
newsgroups well. The border between sci.crypt and sci.med can
be decided using the term ‘escrow’, but to define the topics of the
newsgroups is not that easy without other information.
A bcur decomposition of the Dialect data was obtained using
the LocBCUR and LocBCXT +IterU algorithms for kc = kr = 7 and 8.
The results are presented in Figure 6.9.
From Figure 6.9(a) we can see that LocBCUR with k = 7 recreates
the proposed ground truth (see Figure 4.5 at page 83) up to some
accuracy: the far north dialects are combined with the Central and
North Ostrobothnian dialects, and this dialect is also present in a
narrow area between Tavastian and Savonian dialects; the small
6.6 Experimental Evaluation 147

(a) (b) (c)

Figure 6.9: Spread of selected features (rows of R) and the selected
municipalities (columns of C) as reported by (a) LocBCUR with
k = 7, (b) LocBCXT +IterU with k = 7, and (c) LocBCXT +IterU
with k = 8.

central south-western dialects are migrated to Tavastian dialects;

and a new dialect is found in central Southern Finland.
When LocBCXT +IterU is used, the results change somewhat.
First notable thing in Figures 6.9(b) and 6.9(c) is that the selected
municipalities do not correspond with the spreads of the selected
features, unlike with, say, LocBCUR. This could be expected, as
with LocBCXT +IterU the two are selected independently of each
other, whereas with LocBCUR they are selected concurrently. With
k = 7, the spread of the features does not present any big anomalies:
the central south-western dialects were, once again, combined with
south-western dialects, but all other dialects are present, if with
somewhat different borders. With k = 8 the new dialect does not
appear in place of central south-western dialects, but instead splits
the Savonian dialects, and the dialect of far north spreads in some
new municipalities.
Conclusions. The experiments with the real-world data agreed
with results obtained in synthetic settings, confirming the results of
the previous section. The proposed algorithms performed relatively
well against other methods, showing also the meaningfulness of
the bcx and bcur concepts. Randomization tests revealed some
148 6 The BCX and BCUR Problems

results that seemed to be more a consequence of the marginal

sums of the data than any deeper latent structure. The proposed
algorithms were not the only ones to get such results, and the case
emphasizes how one should be careful when one presents claims
about latent structures supposedly found by one’s algorithms. The
interpretability of the results was somewhat twofold: the results were
mostly intuitive, but – perhaps due to the very restrictive nature
of bcx (and bcur) decompositions – they were not able to fully
encompass the most important aspects of the data. Nevertheless, the
proposed methods provide an enhancement to data miners’ toolbox
for binary data.
CHAPTER 7

Conclusions

This thesis studied six matrix decompositions, namely, bmf, bmp,

ncx, ncur, bcx, and bcur. The last four of them are novel formula-
tions, bmp is semi-novel – it is essentially a well-known problem, but
rephrased as a matrix decomposition – and bmf is not new, but its
use in data mining is. The key motivation behind all decompositions
was the interpretability of the results; restrictions were placed for
factor matrices to increase the interpretability. This does not mean
that the results would always be interpretable, though – different
applications and data call for different methods. New matrix decom-
positions will likely be introduced in the future, addressing other
issues and emphasizing different aspects of interpretability.
The main issue with the studied decompositions is their com-
putational complexity (or lack of knowledge thereof). Most of the
known results concerning decompositions using Boolean algebra can
be traced back to the properties of ±psc, and for bcur, for which
nothing is known, it is certainly plausible that similar results exist.
The computational complexity, and possible hardness of approxim-
ation, of ncx and ncur (or cx and cur) is an interesting open
problem, and is hopefully solved in the near future.
The hardness of constructing the proposed decompositions limits
the worst-case performance of any algorithm for the task – often the
best that could be hoped for is not much better than what any naïve
algorithm could produce. For this reason, the focus in this thesis
was on applying heuristics to find decompositions, and measuring
their empirical performance in constructed as well as real-world
instances.

149
150 7 Conclusions

The experiments demonstrated the algorithms’ capabilities on

producing meaningful results and compared their reconstruction
errors to other methods. No proposed algorithm was able to outper-
form SVD or NMF, notwithstanding the fact that bmf decomposition
could, in some cases, yield smaller reconstruction errors. To some
extent the somewhat poor performance of the Boolean methods can
be considered to be a consequence of the hardness of comparing
real-valued and Boolean methods, but it also shows the hardness
of finding such decompositions. Nevertheless, if one is willing to
sacrifice some reconstruction accuracy for higher interpretability,
then the proposed methods should present a viable alternative.
For the real-valued algorithms for ncx and ncur the situation
was somewhat different. These algorithms were, in some cases,
even better than the algorithms for cx and cur, and close to the
theoretical lower bound presented by nmf and svd.
Throughout this thesis the size of the decomposition, k, has been
defined by the user as a part of the problem instance. This approach
is desirable when, say, the user has expert knowledge of the data
or of the proper size of the decomposition. But on the other cases,
no such knowledge exists or it cannot be used. Then a method of
automatically selecting the size of the decomposition is required.
These model-selection methods are not covered in this thesis, but
they definitely form an interesting and important topic for future
research.
The main focus of this thesis was not on the scalability of the
algorithms. Many heuristics applied, for example local search, are
regularly employed in practical applications. Yet their scalability
to huge data sets is somewhat questionable. The purpose of the
algorithms was to provide first examples of methods that could
construct the proposed decompositions, and to (empirically) verify
that it can be done despite the theoretical complexity of the problems
concerned. For the sake of clear and succinct representation the
algorithms, or the implementations thereof, did not employ many
standard techniques to speed up the computation such as different
pruning methods for local search.
Another topic not discussed in this thesis is parallel computation,
while many of the proposed methods are, in fact, suitable for at least
some level of parallelization. Developing faster, and more accurate,
algorithms for the tasks is an important topic for future research.
 151

The decompositions presented in this thesis are intuitive and

have many properties that make them attractive methods for data
mining. They also have proved to have applications outside the
field of data mining. It is therefore to be hoped that future research
will provide the missing results, close the current gaps, and present
more efficient algorithms, hopefully with provable approximation
guarantees. But the most important goal, and a prerequisite for the
other goals, is to increase the awareness of the Boolean and column
decompositions in all fields of computer science.
References

[ABS08] Marcel R. Ackermann, Johannes Blömer, and Christian

Sohler. Clustering for metric and non-metric distance
measures. In Proc. 19th Annual ACM-SIAM Symposium
on Discrete Algorithms (SODA ’08), pages 799–808,
2008.

[AGK+ 04] Vijay Arya, Naveen Garg, Rohit Kjandekar, Adam

Meyerson, Kamesh Munagala, and Vinayaka Pandit.
Local search heuristics for k-median and facility location
problems. SIAM Journal on Computing, 33(3):544–562,
2004.

[AGM04] Foto Afrati, Aristides Gionis, and Heikki Mannila. Ap-

proximating a collection of frequent sets. In Proc. 10th
ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD ’04), pages 12–19,
2004.

[AIS93] Rakesh Agrawal, Tomasz Imielinski, and Arun Swami.

Mining association rules between sets of items in large
databases. In Proc. ACM SIGMOD International Con-
ference on Management of Data, pages 207–216, 1993.

[AM07] Dimitris Achlioptas and Frank McSherry. Fast compu-

tation of low-rank matrix approximations. Journal of
the ACM, 54(2), 2007. Article 9.

[AMS06] Noga Alon, Dana Moshkovitz, and Shmuel Safra. Al-

gorithmic construction of sets for k-restrictions. ACM
Transactions on Algorithms, 2(2):153–177, 2006.

153
154 References

[AV07] David Arthur and Sergei Vassilvitskii. k-means++: The

advantages of careful seeding. In Proc. 18th Annual
ACM-SIAM Symposium on Discrete Algorithms (SODA
’07), pages 1027–1035, 2007.

[Bal89] Egon Balas. The prize collecting traveling salesman

problem. Networks, 19(6):621–636, 1989.

[BBL+ 07] Michael W. Berry, Murray Browne, Amy N. Langville,

V. Paul Pauca, and Robert J. Plemmons. Algorithms
and applications for approximate nonnegative matrix
factorization. Computational Statistics & Data Analysis,
52(1):155–173, 2007.

[BDG+ 04] Arindam Banerjee, Inderjit Dhillon, Joydeep Ghosh,

Srujana Merugu, and Dharmendra S. Modha. A gen-
eralized maximum entropy approach to Bregman co-
clustering and matrix approximations. In Proc. 10th
ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining (KDD ’04), pages 509–514,
2004.

[BJ06] Wray Buntine and Aleks Jakulin. Discrete component

analysis. In Subspace, Latent Structure and Feature
Selection, volume 3940 of Lecture Notes in Computer
Science, pages 1–33, 2006.

[BKF09] Ella Bingham, Ata Kabán, and Mikael Fortelius. The

aspect Bernoulli model: Multiple causes of presences
and absences. Pattern Analysis & Applications, 12(1):55–
78, 2009.

[BMD08] Christos Boutsidis, Michael W. Mahoney, and Petros

Drineas. Unsupervised feature selection for principal
components analysis. In Proc. 14th ACM SIGKDD
International Conference on Knowledge Discovery and
Data Mining (KDD ’08), pages 61–69, 2008.

[BMD09] Christos Boutsidis, Michael W. Mahoney, and Petros

Drineas. An improved approximation algorithm for the
column subset selection problem. In Proc. 19th Annual
References 155

ACM-SIAM Symposium on Discrete Algorithms (SODA

’09), pages 968–977, 2009.

[BNJ03] David M. Blei, Andrew Y. Ng, and Michael I. Jordan.

Latent Dirichlet allocation. The Journal of Machine
Learning Research, 3:993–1022, 2003.

[BPRB06] Jérémy Besson, Ruggero G. Pensa, Céline Robardet,

and Jean-François Boulicaut. Constraint-based min-
ing of fault-tolerant patterns from Boolean data. In
Knowledge Discovery in Inductive Databases: 4th In-
ternational Workshop, KDID, volume 3933 of Lecture
Notes in Computer Science, pages 55–71, 2006.

[BPS05] Michael W. Berry, Shakhina A. Pulatova, and G. W.

Stewart. Algorithm 844: Computing sparce reduced-
rank approximations to sparce matrices. ACM Transac-
tions on Mathematical Software, 31(2):252–269, 2005.

[Bun02] Wray Buntine. Variational extensions to EM and mul-

tinomial PCA. In Proc. 13th European Conference on
Machine Learning (ECML ’02), volume 2430 of Lecture
Notes in Computer Science, pages 23–34, 2002.

[BV06] Radim Bělohlávek and Vilém Vychodil. On Boolean

factor analysis with formal concepts as factors. In Joint
Proc. International Conference on Soft Computing and
intelligent systems and International Symposium on
Advanced Intelligent systems (SCIS & ISIS ’06), pages
1054–1059, 2006.

[CC03] George W. Cobb and Yung-Pin Chen. An application

of Markov chain Monte Carlo to community ecology.
The American Mathematical Monthly, 110(4):265–288,
2003.

[CDKM00] Robert D. Carr, Srinivas Doddi, Goran Konjevod, and

Madhav Marathe. On the red-blue set cover problem. In
Proc. 11th Annual ACM-SIAM Symposium on Discrete
Algorithms (SODA ’00), pages 345–353, 2000.
156 References

[CG99] Moses Charikar and Sudipto Guha. Improved combin-

atorial algorithms for the facility location and k-median
problems. In Proc. 40th Annual Symposium on Founda-
tions of Computer Science (FOCS ’99), pages 378–388,
1999.

[Che06] Ke Chen. On k-median clustering in high dimensions. In

Proc. 17th Annual ACM-SIAM Symposium on Discrete
Algorithms (SODA ’06), pages 1177–1185, 2006.

[Chv79] V. Chvatal. A greedy heuristic for the set-covering

problem. Mathematics of Operations Research, 4(3):233–
235, 1979.

[CKX06] Jianer Chen, Iyad A. Kanj, and Ge Xia. Improved para-

meterized upper bounds for vertex cover. In Proc. Math-
ematical Foundations of Computer Science (MFCS),
volume 4162 of Lecture Notes in Computer Science,
pages 238–249, 2006.

[CKY06] Marek Chrobak, Claire Kenyon, and Neal Young. The

reverse greedy algorithm for the metric k-median prob-
lem. Information Processing Letters, 97:68–72, 2006.

[ÇMI07] Ali Çivril and Malik Magdon-Ismail. Finding maximum

volume sub-matrices of a matrix. Technical Report
07-08, Department of Computer Science, Rensselaer
Polytechnic Institute, 2007.

[CR93] Joel E. Cohen and Uriel G. Rothblum. Nonnegative

ranks, decompositions, and factorizations of nonnegative
matrices. Linear Algebra and Its Applications, 190:149–
168, 1993.

[DF95] Rod G. Downey and Michael R. Fellows. Fixed-

parameter tractability and completeness I: Basic results.
SIAM Journal on Computing, 24(4):873–921, 1995.

[DF99] R. G. Downey and M. R. Fellows. Parameterized Com-

plexity. Monographs in computer science. Springer, New
York, 1999.
References 157

[DKM06a] Petros Drineas, Ravi Kannan, and Michael W. Mahoney.

Fast Monte Carlo algorithms for matrices II: Computing
a low-rank approximation to a matrix. SIAM Journal
on Computing, 36(1):158–183, 2006.

[DKM06b] Petros Drineas, Ravi Kannan, and Michael W. Mahoney.

Fast Monte Carlo algorithms for matrices III: Comput-
ing a compressed approximate matrix decomposition.
SIAM Journal on Computing, 36(1):184–206, 2006.

[DMM06a] Petros Drineas, Michael W. Mahoney, and

S. Muthukrishnan. Subspace sampling and relative-
error matrix approximation: Column-based methods.
In Proc. 9th International Workshop on Approximation
Algorithms for Combinatorial Optimization Problems
and 10th International Workshop on Randomization
and Computation (APPROX + RANDOM), volume
4110 of Lecture Notes in Computer Science, pages
316–326, 2006.

[DMM06b] Petros Drineas, Michael W. Mahoney, and

S. Muthukrishnan. Subspace sampling and relative-
error matrix approximation: Column-row-based
methods. In Proc. 14th European Symposium on
Algorithms (ESA ’06), volume 4168 of Lecture Notes in
Computer Science, pages 304–314, 2006.

[DMM07] Petros Drineas, Michael W. Mahoney, and

S. Muthukrishnan. Relative-error CUR matrix
decompositions. Technical report, August 2007.
arXiv:0708.3696v1 [cs.DS].

[DS04] Irit Dinur and Shmuel Safra. On the hardness of ap-

proximating label-cover. Information Processing Letters,
89(5):247–254, 2004.

[EP00] Michael Elkin and David Peleg. The hardness of approx-

imating spanner problems. In Proc. 17th Symposium
on Theoretical Aspects of Computer Science (STACS
’00), volume 1770 of Lecture Notes in Computer Science,
pages 370–381, 2000.
158 References

[EW97] Sheila Embleton and Eric S. Wheeler. Finnish dialect

atlas for quantitative studies. Journal of Quantitative
Linguistics, 4(1–3):99–102, 1997.

[EW00] Sheila M. Embleton and Eric S. Wheeler. Computerized

dialect atlas of Finnish: Dealing with ambiguity. Journal
of Quantitative Linguistics, 7(3):227–231, 2000.

[EY36] Carl Eckart and Gale Young. The approximation of

one matrix by another of lower rank. Psychometrika,
1(3):211–218, 1936.

[F+ 03] M. Fortelius et al. Neogene of the old world database

of fossil mammals (NOW). Online, 2003. http://
www.helsinki.fi/science/now/, Accessed 8 Decem-
ber 2008.

[Fei98] Uriel Feige. A threshold of ln n for approximating set

cover. Journal of the ACM, 45(4):634–652, 1998.

[FG06] Jörg Flum and Martin Grohe. Parameterized Complex-

ity Theory. Texts in Theoretical Computer Science. An
EATCS Series. Springer, Berlin, 2006.

[FGJM06] Mikael Fortelius, Aristides Gionis, Jukka Jernvall, and

Heikki Mannila. Spectral ordering and biochronology of
European fossil mammals. Paleobiology, 32(2):206–214,
2006.

[FKV04] Alan Frieze, Ravi Kannan, and Santosh Vempala. Fast

Monte-Carlo algorithms for finding low-rank approxim-
ations. Journal of the ACM, 51(6):1025–1041, 2004.

[GE96] Ming Gu and Stanley C. Eisenstat. Efficient algorithms

for computing a strong rank-revealing QR factorization.
SIAM Journal on Scientific Computing, 17(4):848–869,
1996.

[GGM04] Floris Geerts, Bart Goethals, and Taneli Mielikäinen.

Tiling databases. In Proc. Discovery Science: 7th In-
ternational Conference, volume 3245 of Lecture Notes
in Computer Science, pages 278–289, 2004.
References 159

[GH06] Liqiang Geng and Howard J. Hamilton. Interestingness

measures for data mining: A survey. ACM Computing
Surveys, 38(3):Article 9, 2006.

[GJ79] Michael R. Garey and David S. Johnson. Computers and

intractability: A guide to the theory of NP-Completeness.
W. H. Freeman and Company, New York, 1979.

[GKP94] Ronald L. Graham, Donald E. Knuth, and Oren Pata-

shnik. Concrete Mathematics. Addison–Wesley, Upper
Saddle River, NJ, second edition, 1994.

[GMMT07] Aristides Gionis, Heikki Mannila, Taneli Mielikäinen,

and Panayiotis Tsaparas. Assessing data mining res-
ults via swap randomization. ACM Transactions on
Knowledge Discovery from Data, 1(3), 2007.

[GMS04] Aristides Gionis, Heikki Mannila, and Jouni K. Sep-

pänen. Geometric and combinatorial tiles in 0–1 data.
In Proc. Principles and Practice of Knowledge Discov-
ery in Databases (PKDD ’04), volume 3202 of Lecture
Notes in Computer Science, pages 173–184, 2004.

[GP83] D. A. Gregory and N. J. Pullman. Semiring rank:

Boolean rank and nonnegative rank factorizations.
Journal of Combinatorics, Information & System Sci-
ences, 8(3):223–233, 1983.

[GVL96] Gene H. Golub and Charles F. Van Loan. Matrix Com-

putations. The Johns Hopkins University Press, Bal-
timore, third edition, 1996.

[GW95] Michel X. Goemans and David P. Williamson. A general

approximation technique for constrained forest prob-
lems. SIAM Journal on Computing, 24:296–317, 1995.

[Har72] J. A. Hartigan. Direct clustering of a data mat-

rix. Journal of the American Statistical Association,
67(337):123–129, 1972.

[HK01] Jiawei Han and Micheline Kamber. Data Mining: Con-

cepts and Techniques. Morgan Kaufmann Publishers,
San Francisco, 2001.
160 References

[HMS01] David Hand, Heikki Mannila, and Padhraic Smyth.

Principles of Data Mining. The MIT Press, Cambridge,
Massachusetts, 2001.

[HMT08] Saara Hyvönen, Pauli Miettinen, and Evimaria Terzi.

Interpretable nonnegative matrix decompositions. In
Proc. 14th ACM SIGKDD International Conference
on Knowledge Discovery and Data Mining (KDD ’08),
pages 345–353, 2008.

[Hof99] Thomas Hofmann. Probabilistic latent semantic index-

ing. In Proc. 22nd ACM SIGIR Conference on Research
and Development in Information Retrieval, pages 50–57,
1999.

[Itk65] Terho Itkonen. Proto-Finnic Final Consonants: Their

history in the Finnic languages with particular reference
to the Finnish dialects, part I: 1, Introduction and The
History of -k in Finnish, volume 138(1) of Mémoires de
la Société Finno-Ougrienne. Suomalaisen Kirjallisuuden
Kirjapaino Oy, Helsinki, 1965.

[Joh74] David S. Johnson. Approximation algorithms for com-

binatiorial problems. Journal of Computer and System
Sciences, 9:256–278, 1974.

[KGR05] Mehmet Koyutürk, Ananth Grama, and Naren Ra-

makrishnan. Compression, clustering, and pattern dis-
covery in very-high-dimensional discrete-attribute data
sets. IEEE Transactions on Knowledge and Data En-
gineering, 17(4):447–461, 2005.

[Kim82] Ki Hang Kim. Boolean matrix theory and applications.

Marcel Dekker, Inc., New York, 1982.

[KO98] Tamara G. Kolda and Dianne P. O’Leary. A semidiscrete

matrix decomposition for latent semantic indexing in in-
formation retrieval. ACM Transactions on Information
Systems, 16(4):322–346, 1998.

[KO00] Tamara G. Kolda and Dianne P. O’Leary. Algortihm

805: Computation and uses of the semidiscrete matrix
References 161

decomposition. ACM Transactions on Mathematical

Software, 26(3):415–435, 2000.

[KTH79] M. K. Kozlov, S. P. Tarasov, and L. G. Hačijan. Poly-

nomial solvability of convex quadratic programming.
Soviet Mathematics. Doklady, 20(5):1108–1111, 1979.

[Li08] Tao Li. Clustering based on matrix approximation: a

unifying view. Knowledge and Information Systems,
17(1):1–15, 2008.

[Lov75] L. Lovász. On the ratio of optimal integral and fractional

covers. Discrete Mathematics, 13:383–390, 1975.

[LS99] Daniel D. Lee and H. Sebastian Seung. Learning the

parts of objects by non-negative matrix factorization.
Nature, 401:788–791, 1999.

[LVA08] Haibing Lu, Jaideep Vaidya, and Vijayalakshmi Atluri.

Optimal Boolean matrix decomposition: Application
to role engineering. In Proc. 24th IEEE International
Conference on Data Engineering (ICDE ’08), pages
297–306, 2008.

[LY94] Carsten Lund and Mihalis Yannakakis. On the hardness

of approximating minimization problems. Journal of
the ACM, 41(5):960–981, 1994.

[Mie06] Pauli Miettinen. The discrete basis problem. Master’s

thesis, Report C-2006-010, Department of Computer
Science, University of Helsinki, 2006.

[Mie08a] Pauli Miettinen. The Boolean column and column-row

matrix decompositions. Data Mining and Knowledge
Discovery, 17(1):39–56, 2008.

[Mie08b] Pauli Miettinen. On the positive–negative partial

set cover problem. Information Processing Letters,
108(4):219–221, 2008.

[Mir60] L. Mirsky. Symmetric gauge functions and unitarily

invariant norms. The Quarterly Journal of Mathematics.
Oxford. Second series, 11:50–59, 1960.
162 References

[MMG+ 06] Pauli Miettinen, Taneli Mielikäinen, Aristides Gionis,

Gautam Das, and Heikki Mannila. The discrete basis
problem. In Proc. 10th conference on Principles and
Practice on Knowldedge Discovery in Databases (PKDD
’06), volume 4213 of Lecture Notes in Computer Science,
pages 335–346, 2006.

[MMG+ 08] Pauli Miettinen, Taneli Mielikäinen, Aristides Gionis,

Gautam Das, and Heikki Mannila. The discrete basis
problem. IEEE Transactions on Knowledge and Data
Engineering, 20(10):1348–1362, 2008.

[MO04] Sara C. Madeira and Arlindo L. Oliveira. Bicluster-

ing algorithms for biological data analysis: A survey.
IEEE/ACM Transactions on Computational Biology
and Bioinformatics, 1(1):24–45, 2004.

[MOP04] Adam Meyerson, Liadan O’Callaghan, and Serge

Plotkin. A k-median algorithm with running time in-
dependent of data size. Machine Learning, 56:61–87,
2004.

[MPR95] Sylvia D. Monson, Norman J. Pullman, and Rolf Rees. A

survey of clique and biclique coverings and factorizations
of (0, 1)-matrices. Bulletin of the ICA, 14:17–86, 1995.

[MRS04] Nina Mishra, Dana Ron, and Ram Swaminathan. A new

conceptual clustering framework. Machine Learning,
56:115–151, 2004.

[MS84] Nimrod Megiddo and Kenneth J. Supowit. On the com-

plexity of some common geometric location problems.
SIAM Journal on Computing, 13(1):182–196, 1984.

[OP83] Dianne P. O’Leary and Shmuel Peleg. Digital image

compression by outer product expansion. IEEE Trans-
actions on Communications, 31(3):441–444, 1983.

[PDL+ 08] Peristera Paschou, Petros Drineas, Jamey Lewis, et al.

Tracing sub-structure in the European American popu-
lation with PCA-informative markers. PLoS Genetics,
4(7), 2008.
References 163

[Pel07] David Peleg. Approximation algorithms for the label-

coverMAX and red-blue set cover problems. Journal of
Discrete Algorithms, 5(1):55–64, 2007.
[Por80] M. F. Porter. An algorithm for suffix stripping. Program,
14(3):130–137, 1980.
[PS82] Christos H. Papadimitriou and Kenneth Steiglitz. Com-
binatorial Optimization Algorithms and Complexity.
Prentice-Hall, Inc., Englewood Cliffs, NJ, 1982.
[PT94] Pentti Paatero and Unto Tapper. Positive matrix fac-
torization: A non-negative factor model with optimal
utilization of error estimates of data values. Environ-
metrics, 5:111–126, 1994.
[RF01] Céline Robardet and Fabien Feschet. Efficient local
search in conceptual clustering. In Proc. Discovery
Science, volume 2226 of Lecture Notes in Computer
Science, pages 323–335, 2001.
[RS97] Ran Raz and Shmuel Safra. A sub-constant error-
probability low-degree test, and a sub-constant error-
probability PCP characterization of NP. In Proc.
29th annual ACM symposium on Theory of comput-
ing (STOC ’97), pages 475–484, 1997.
[SBM03] Jouni K. Seppänen, Ella Bingham, and Heikki Mannila.
A simple algorithm for topic identification in 0–1 data.
In Proc. Principles and Practice on Knowledge Discov-
ery in Databases (PKDD ’03), volume 2838 of Lecture
Notes in Computer Science, pages 423–434, 2003.
[SG08] Ajit P. Singh and Geoffrey J. Gordon. A unified view of
matrix factorization models. In Proc. European Confer-
ence on Machine Learning and Principles and Practice
of Knowledge Discovery in Databases (ECML PKDD
’08), volume 5212 of Lecture Notes in Computer Science,
pages 358–373, 2008.
[Sim90] Hans Ulrich Simon. On approximate solutions for com-
binatorial optimization problems. SIAM Journal on
Discrete Mathematics, 3(2):294–310, 1990.
164 References

[SKPH08] Václav Snášel, Pavel Krömer, Jan Platoš, and Dušan

Húsek. On the implementation of Boolean matrix fac-
torization. In Proc. 19th International Conference on
Database and Expert Systems Application, pages 554–
558, 2008.
[SPK+ 08] Václav Snášel, Jan Platoš, Pavel Krömer, Dušan Húsek,
Roman Neruda, and Alexander A. Frolov. Investigating
Boolean matrix factorization. In Proc. Workshop on
Data Mining using Matrices and Tensors, 2008.
[SSU03] Andrew I. Schein, Lawrence K. Saul, and Lyle H. Ungar.
A generalized linear model for principal component ana-
lysis of binary data. In Proc. 9th Internation Workshop
on Artificial Intelligence and Statistics, 2003.
[Ste93] G. W. Stewart. On the early history of the singular value
decomposition. SIAM Review, 35(4):551–566, 1993.
[Ste99] G. W. Stewart. Four algorithms for the efficient com-
putation of truncated pivoted QR approximations to
a sparse matrix. Numerische Mathematik, 83:313–323,
1999.
[Ste05] G. W. Stewart. Error analysis of the quasi-Gram–
Schmidt algorithm. SIAM Journal on Matrix Analysis
and Applications, 27(2):493–506, 2005.
[SYL94] Ilkka Savijärvi and Eeva Yli-Luukko. Jämsän äijän
murrekirja, volume 618 of Suomalaisen Kirjallisuuden
Seuran Toimituksia. Suomalaisen kirjallisuuden seura,
Helsinki, 1994.
[VAG07] Jaideep Vaidya, Vijayalakshmi Atluri, and Qi Guo. The
role mining problem: Finding a minimal descriptive
set of roles. In Proc. 12th ACM Symposium on Access
Control Models and Technologies (SACMAT), pages
175–184, 2007.
[ZLDZ07] Zhongyuan Zhang, Tao Li, Chris Ding, and Xiangsun
Zhang. Binary matrix factorization with applications.
In Proc. 7th IEEE International Conference on Data
Mining (ICDM ’07), pages 391–400, 2007.
APPENDIX A

Proof of Lemma 3.4

Recall that the claim of Lemma 3.4 was the following.

Lemma A.1 (Lemma 3.4). Let gc : R≥0 → R≥0 be defined by

1−(log log x)−c
gc (x) = 2(log x)

for constant c < 1/2. Then

 
x  1−ε

gc = Ω 2log x
(A.1)
O(gc′ (x)(log log x)c′ )

for any c, c′ < 1/2 and ε > 0.

Before proceeding to the actual proof, let us consider some simple

examples giving intuition why, at first place, the lemma might be
true and whether it is a special property of just these functions or a
consequence of a moregeneral property.
 First important thing to
1−ε
note is that gc (x) = Ω 2log x . This is relatively easy to see by
considering the logarithms of the functions:
−c
log(gc (x)) = log1−(log log x) x (A.2a)
 1−ε

log 2log x
= log1−ε x. (A.2b)

For x large enough, 1/(log log x)c < ε, and from that point on the
exponent of (A.2a) is bigger than the exponent of (A.2b), and thus
the former grows faster.
The problem, however, comes from the argument of gc in (A.1): it
is (essentially) x/gc′ (x). We aim at proving a lower bound, and thus

165
166 A Proof of Lemma 3.4

we hope a fast-enough growing algorithm. But at the same time,

if gc grows too fast, x/gc′ (x) does not grow fast enough. Indeed,
a superlinear polynomial x 7→ xp , p > 1, provides us a simple
2
example of too-fast-growing algorithm, as (x/xp )p = xp−p → 0 as
x → ∞. On the other hand, a polylogarithm x 7→ logp x grows
slow enough: (log(x/ logp x))p = (log x − p log log x)p = Θ(logp x).
As the function gc is superpolylogarithmic and subpolynomial (see
Section 3.3), neither of these observations furnishes us the desired
result.
To start the actual proof, we shall first remove the big-O nota-
tion from the denominator of gc ’s argument. The function gc is an
increasing function, and thus the smaller its argument, the smaller
its value. To prove a lower bound for it we can safely underes-
′
timate its argument’s value by replacing O(gc′ (x)(log log x)c ) with
M0 g1/2 (x)(log log x)1/2 for some constant M0 and considering only
large enough x. The constant M0 comes from the definition of big-O,
′
and it is easy to see that, for a fixed x, gc′ (x)(log log x)c increases
with c′ , attaining its maximum at c′ = 1/2. Moreover, we shall
omit M0 for the sake of clearer presentation. Hence for our task, to
prove (A.1), it is enough to prove that
!
x
gc
g1/2 (x)(log log x)1/2


−c
1− log log x/(g1/2 (x)(log log x)1/2 )
= 2(log(x/(g1/2 (x)(log log x)1/2 ) ))

1−ε
is bounded below by 2log x .
To simplify more, we can take logarithms of these functions,
yielding claim


−c
  1− log log x/(g1/2 (x)(log log x)1/2 )
1/2
log x/(g1/2 (x)(log log x) )
 
= Ω log1−ε x . (A.3)

To bound the left-hand side of (A.3) from below, we shall find a

lower bound for the base and for the exponent. Assume that f1 (x) is
an asymptotic lower bound for the base and f2 (x) is an asymptotic
lower bound for the exponent. Then there exists constants m1 and
m2 such that the left-hand side of (A.3) is at least (m1 f1 (x))m2 f2 (x)
 167

for x large enough. Our goal is, of course, to find functions f1 and
f2 that are easier to analyse than those in (A.3).


We start with the base log x/(g1/2 (x)(log log x)1/2 ) . We have
that


log x/(g1/2 (x)(log log x)1/2 )
= log x − log(g1/2 (x)) − 1/2 log log log x
−1/2
= log x − (log x)1−(log log x) − o(log x).

To analyse the middle term’s relation to log x, we can notice that,

although 1 − (log log x)−1/2 → 1 as x → ∞, it is still less than 1
−1/2
for all (finite) values of x, and thus (log x)1−(log log x) is less than
log x. This is formalized in the following lemma.
Lemma A.2. We have
log x
lim −1/2
= ∞.
x→∞ (log x)1−(log log x)
Proof. The proof is by l’Hôpital’s rule. The first derivative of the
nominator is
1
D log x = , (A.4)
x ln 2
and that of the denominator is
−1/2
D(log x)1−(log log x) =
p −1
 1− ln( ln x
ln 2 )
(ln 2)−1  
ln x ln x
1/2 ln
ln 2 ln 2
   −3/2
ln x
ln (ln 2)−1 x−1 (ln x)−1 (ln 2)−1
ln 2
 s !−1  !
 
ln x
+ 1 − ln (ln 2)−1  x−1 (ln x)−1
ln 2

which simplifies to
 √ √ 
− 1/2 ln 2 − 2 ln 2 − ln ln 2 + ln ln x
√ √ √
−3 ln
√2 − ln ln 2+ln ln x+2 ln 2
1/2 √ − √− ln lnln2+ln
2
(ln 2) ln 2 − ln ln 2+ln ln x (ln x) ln x

√ −1
x−1 − ln ln 2 + ln ln x (A.5)
168 A Proof of Lemma 3.4

The ln 2 (and ln ln 2) terms in (A.5) do not affect to our asymptotic

√
considerations. Thus the first row of (A.5) simplifies
√
to O( ln ln x),
the first factor of the second row to ln(2) 1/Θ( ln ln x) = O(1), and
√
the second factor of the second row to ln(x) −1/O( ln ln x) . The last
 √ −1
row simplifies to x O( ln ln x) .
Combining (A.4) and the asymptotic version of (A.5) gives
p !−1
D log x 1 O( ln(ln(x))) O(1)
−1/2
= √ √
D(log x)1−(log log x) x ln 2 x(ln x)1/O( ln ln x) O( ln ln x)
√ √
(ln x)1/O( ln ln x) O( ln ln x)
= √
O( ln ln x) O(1)
√
= ln(x)1/O( ln ln x)
/O(1) → ∞,
(A.6)

as x → ∞, and the claim follows from l’Hôpital’s rule.

Thus we have that

log x/(g1/2 (x)(log log x)1/2 ) = log x − o(log x) ≥ m1 log x

for some constant m1 and when x is big enough. Analysing the

exponent goes similarly:


−c
1 − log log x/(g1/2 (x)(log log x)1/2 )
−1/2

= 1 − log log x − (log x)1−(log log x) − 1/2 log log log x )−c


−c
= 1 − log log x − o(log x)
≥ 1 − (log(m2 log x))−c .

The second equation follows from Lemma A.2, and the inequality
from the fact that log x − o(log x) = Θ(log x) [GKP94, eq. 9.20].
The final part of the proof is to show that
−c
(m1 log x)1−(log(m2 log x)) = Ω(log1−ε x). (A.7)

But the left-hand side is, omitting m2 , m1 log gc (x), and even with
m2 in the exponent, we can follow our earlier reasoning by taking
logarithms and noticing that, with x big enough, (log(m2 log x))−c <
ε, from which the claim follows.
APPENDIX B

Proof of Theorem 4.4

Recall Theorem 4.4.

Theorem 4.4. There exists instances (A ∈ {0, 1}n×m , k) of the
bmf problem that cannot be approximated with an additive error of
√ √
c(n, m) = max{ 4 n, 4 m}.
We shall prove the theorem for case n ≤ m; the other case is
analogous. The proof resembles that of Theorem 4.3. Hence, for a
contradiction, let us assume that A is a polynomial-time algorithm
for bmf such that for every A ∈ {0, 1}n×m , and for every k < n,
√
costbmf (A, A(A, k)) ≤ cost∗bmf (A, k) + 4 m. (B.1)
√
Construct a new instance (A′ , k) by taking f (c(m)) = f ( 4 m)
copies of A, with f being a polynomial to be defined later. Thus
√
we have an n-by-f ( 4 m) matrix


A′ = A
| A {z· · · A} .
√
f ( 4 m) copies

The optimal solution of (A′ , k) decomposes A optimally, and

√
repeats that decomposition f ( 4 m) times, with
√
cost∗ (A′ , k) = f ( 4 m)cost∗ (A, k).
√
Matrix A′ has m′ = f ( 4 m)m columns, and thus we assume A’s
results to admit bound
√
cost(A′ , A(A′ , k)) ≤ f ( 4 m)cost∗ (A, k) + c(m′ )
√ q √ (B.2)
= f ( 4 m)cost∗ (A, k) + 4 f ( 4 m)m.

169
170 B Proof of Theorem 4.4

To attain this bound, the average error caused by each copy of

A must be at most
 √ q √ 
 f ( 4 m)cost∗ (A, k) + f ( 4 m)m 
 4 
 √ , (B.3)
f ( 4 m)

and there must be at least one copy of A, denoted by A0 , for which

the error indeed is below (B.3). But this means that the additional
error allowed is at most
q √ 
 4 
 f ( 4 m)m 
 √ . (B.4)
f ( 4 m)

Selecting f such that (B.4) is 0 gives us a contradiction, as with

Theorem 4.3. Letting f (x) = x2 is one possibility, as
q √ q √
4
f ( 4 m)m 4
( 4 m)2 m
√ = √
f ( 4 m) ( 4 m)2
m3/8
=
m1/2
= m−1/8 < 1,

proving the claim.

TIETOJENKÄSITTELYTIETEEN LAITOS DEPARTMENT OF COMPUTER SCIENCE
PL 68 (Gustaf Hällströmin katu 2 b) P.O. Box 68 (Gustaf Hällströmin katu 2 b)
00014 Helsingin yliopisto FIN-00014 University of Helsinki, Finland

JULKAISUSARJA A SERIES OF PUBLICATIONS A

Reports may be ordered from: Kumpula Science Library, P.O. Box 64, FIN-00014 University of
Helsinki, Finland.
A-2001-1 J. Rousu: Efficient range partitioning in classification learning. 68+74 pp. (Ph.D.
Thesis)
A-2001-2 M. Salmenkivi: Computational methods for intensity models. 145 pp. (Ph.D. Thesis)
A-2001-3 K. Fredriksson: Rotation invariant template matching. 138 pp. (Ph.D. Thesis)
A-2002-1 A.-P. Tuovinen: Object-oriented engineering of visual languages. 185 pp. (Ph.D. Thesis)
A-2002-2 V. Ollikainen: Simulation techniques for disease gene localization in isolated populations.
149+5 pp. (Ph.D. Thesis)
A-2002-3 J. Vilo: Discovery from biosequences. 149 pp. (Ph.D. Thesis)
A-2003-1 J. Lindström: Optimistic concurrency control methods for real-time database systems.
111 pp. (Ph.D. Thesis)
A-2003-2 H. Helin: Supporting nomadic agent-based applications in the FIPA agent architecture.
200+17 pp. (Ph.D. Thesis)
A-2003-3 S. Campadello: Middleware infrastructure for distributed mobile applications. 164 pp.
(Ph.D. Thesis)
A-2003-4 J. Taina: Design and analysis of a distributed database architecture for IN/GSM data.
130 pp. (Ph.D. Thesis)
A-2003-5 J. Kurhila: Considering individual differences in computer-supported special and ele-
mentary education. 135 pp. (Ph.D. Thesis)
A-2003-6 V. Mäkinen: Parameterized approximate string matching and local-similarity-based
point-pattern matching. 144 pp. (Ph.D. Thesis)
A-2003-7 M. Luukkainen: A process algebraic reduction strategy for automata theoretic verifica-
tion of untimed and timed concurrent systems. 141 pp. (Ph.D. Thesis)
A-2003-8 J. Manner: Provision of quality of service in IP-based mobile access networks. 191 pp.
(Ph.D. Thesis)
A-2004-1 M. Koivisto: Sum-product algorithms for the analysis of genetic risks. 155 pp. (Ph.D.
Thesis)
A-2004-2 A. Gurtov: Efficient data transport in wireless overlay networks. 141 pp. (Ph.D. Thesis)
A-2004-3 K. Vasko: Computational methods and models for paleoecology. 176 pp. (Ph.D. Thesis)
A-2004-4 P. Sevon: Algorithms for Association-Based Gene Mapping. 101 pp. (Ph.D. Thesis)
A-2004-5 J. Viljamaa: Applying Formal Concept Analysis to Extract Framework Reuse Interface
Specifications from Source Code. 206 pp. (Ph.D. Thesis)
A-2004-6 J. Ravantti: Computational Methods for Reconstructing Macromolecular Complexes
from Cryo-Electron Microscopy Images. 100 pp. (Ph.D. Thesis)
A-2004-7 M. Kääriäinen: Learning Small Trees and Graphs that Generalize. 45+49 pp. (Ph.D.
Thesis)
A-2004-8 T. Kivioja: Computational Tools for a Novel Transcriptional Profiling Method. 98 pp.
(Ph.D. Thesis)
A-2004-9 H. Tamm: On Minimality and Size Reduction of One-Tape and Multitape Finite Au-
tomata. 80 pp. (Ph.D. Thesis)
A-2005-1 T. Mielikäinen: Summarization Techniques for Pattern Collections in Data Mining.
201 pp. (Ph.D. Thesis)
A-2005-2 A. Doucet: Advanced Document Description, a Sequential Approach. 161 pp. (Ph.D.
Thesis)
A-2006-1 A. Viljamaa: Specifying Reuse Interfaces for Task-Oriented Framework Specialization.
285 pp. (Ph.D. Thesis)
A-2006-2 S. Tarkoma: Efficient Content-based Routing, Mobility-aware Topologies, and Temporal
Subspace Matching. 198 pp. (Ph.D. Thesis)
A-2006-3 M. Lehtonen: Indexing Heterogeneous XML for Full-Text Search. 185+3 pp. (Ph.D.
Thesis)
13
A-2006-4 A. Rantanen: Algorithms for C Metabolic Flux Analysis. 92+73 pp. (Ph.D. Thesis)
A-2006-5 E. Terzi: Problems and Algorithms for Sequence Segmentations. 141 pp. (Ph.D. Thesis)
A-2007-1 P. Sarolahti: TCP Performance in Heterogeneous Wireless Networks. (Ph.D. Thesis)
A-2007-2 M. Raento: Exploring privacy for ubiquitous computing: Tools, methods and experi-
ments. (Ph.D. Thesis)
A-2007-3 L. Aunimo: Methods for Answer Extraction in Textual Question Answering. 127+18
pp. (Ph.D. Thesis)
A-2007-4 T. Roos: Statistical and Information-Theoretic Methods for Data Analysis. 82+75 pp.
(Ph.D. Thesis)
A-2007-5 S. Leggio: A Decentralized Session Management Framework for Heterogeneous Ad-Hoc
and Fixed Networks. 230 pp. (Ph.D. Thesis)
A-2007-6 O. Riva: Middleware for Mobile Sensing Applications in Urban Environments. 195 pp.
(Ph.D. Thesis)
A-2007-7 K. Palin: Computational Methods for Locating and Analyzing Conserved Gene Regula-
tory DNA Elements. 130 pp. (Ph.D. Thesis)
A-2008-1 I. Autio: Modeling Efficient Classification as a Process of Confidence Assessment and
Delegation. 212 pp. (Ph.D. Thesis)
A-2008-2 J. Kangasharju: XML Messaging for Mobile Devices. 24+255 pp. (Ph.D. Thesis).
A-2008-3 N. Haiminen: Mining Sequential Data – in Search of Segmental Structures. 60+78 pp.
(Ph.D. Thesis)
A-2008-4 J. Korhonen: IP Mobility in Wireless Operator Networks. (Ph.D. Thesis)
A-2008-5 J.T. Lindgren: Learning nonlinear visual processing from natural images. 100+64 pp.
(Ph.D. Thesis)
A-2009-1 K. Hätönen: Data mining for telecommunications network log analysis. 153 pp. (Ph.D.
Thesis)
A-2009-2 T. Silander: The Most Probable Bayesian Network and Beyond. (Ph.D. Thesis)
A-2009-3 K. Laasonen: Mining Cell Transition Data. 148 pp. (Ph.D. Thesis)
A-2009-4 P. Miettinen: Matrix Decomposition Methods for Data Mining: Computational Com-
plexity and Algorithms. 164+6 pp. (Ph.D. Thesis)