Download full Introduction to Algorithms for Data Mining and Machine Learning Xin-She Yang ebook all chapters
Download full Introduction to Algorithms for Data Mining and Machine Learning Xin-She Yang ebook all chapters
com
https://ebookmass.com/product/introduction-to-algorithms-
for-data-mining-and-machine-learning-xin-she-yang/
OR CLICK BUTTON
DOWNLOAD NOW
https://ebookmass.com/product/introduction-to-algorithms-for-data-
mining-and-machine-learning-yang/
ebookmass.com
https://ebookmass.com/product/fundamentals-of-machine-learning-for-
predictive-data-analytics-algorithms/
ebookmass.com
https://ebookmass.com/product/machine-learning-for-signal-processing-
data-science-algorithms-and-computational-statistics-max-a-little/
ebookmass.com
https://ebookmass.com/product/big-data-analytics-introduction-to-
hadoop-spark-and-machine-learning-raj-kamal/
ebookmass.com
Machine Learning for Biometrics: Concepts, Algorithms and
Applications (Cognitive Data Science in Sustainable
Computing) Partha Pratim Sarangi
https://ebookmass.com/product/machine-learning-for-biometrics-
concepts-algorithms-and-applications-cognitive-data-science-in-
sustainable-computing-partha-pratim-sarangi/
ebookmass.com
https://ebookmass.com/product/machine-learning-algorithms-for-signal-
and-image-processing-suman-lata-tripathi/
ebookmass.com
Introduction to
Algorithms for Data Mining
and Machine Learning
Introduction to Algorithms for Data Mining and
Machine Learning
This page intentionally left blank
Introduction to
Algorithms for Data
Mining and Machine
Learning
Xin-She Yang
Middlesex University
School of Science and Technology
London, United Kingdom
Academic Press is an imprint of Elsevier
125 London Wall, London EC2Y 5AS, United Kingdom
525 B Street, Suite 1650, San Diego, CA 92101, United States
50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States
The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom
Copyright © 2019 Elsevier Inc. All rights reserved.
No part of this publication may be reproduced or transmitted in any form or by any means, electronic or
mechanical, including photocopying, recording, or any information storage and retrieval system, without
permission in writing from the publisher. Details on how to seek permission, further information about the
Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center
and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.
This book and the individual contributions contained in it are protected under copyright by the Publisher (other
than as may be noted herein).
Notices
Knowledge and best practice in this field are constantly changing. As new research and experience broaden our
understanding, changes in research methods, professional practices, or medical treatment may become necessary.
Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using
any information, methods, compounds, or experiments described herein. In using such information or methods
they should be mindful of their own safety and the safety of others, including parties for whom they have a
professional responsibility.
To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability
for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or
from any use or operation of any methods, products, instructions, or ideas contained in the material herein.
ISBN: 978-0-12-817216-2
1 Introduction to optimization 1
1.1 Algorithms 1
1.1.1 Essence of an algorithm 1
1.1.2 Issues with algorithms 3
1.1.3 Types of algorithms 3
1.2 Optimization 4
1.2.1 A simple example 4
1.2.2 General formulation of optimization 7
1.2.3 Feasible solution 9
1.2.4 Optimality criteria 10
1.3 Unconstrained optimization 10
1.3.1 Univariate functions 11
1.3.2 Multivariate functions 12
1.4 Nonlinear constrained optimization 14
1.4.1 Penalty method 15
1.4.2 Lagrange multipliers 16
1.4.3 Karush–Kuhn–Tucker conditions 17
1.5 Notes on software 18
2 Mathematical foundations 19
2.1 Convexity 20
2.1.1 Linear and affine functions 20
2.1.2 Convex functions 21
2.1.3 Mathematical operations on convex functions 22
2.2 Computational complexity 22
2.2.1 Time and space complexity 24
2.2.2 Complexity of algorithms 25
2.3 Norms and regularization 26
2.3.1 Norms 26
2.3.2 Regularization 28
2.4 Probability distributions 29
2.4.1 Random variables 29
2.4.2 Probability distributions 30
vi Contents
3 Optimization algorithms 45
3.1 Gradient-based methods 45
3.1.1 Newton’s method 45
3.1.2 Newton’s method for multivariate functions 47
3.1.3 Line search 48
3.2 Variants of gradient-based methods 49
3.2.1 Stochastic gradient descent 50
3.2.2 Subgradient method 51
3.2.3 Conjugate gradient method 52
3.3 Optimizers in deep learning 53
3.4 Gradient-free methods 56
3.5 Evolutionary algorithms and swarm intelligence 58
3.5.1 Genetic algorithm 58
3.5.2 Differential evolution 60
3.5.3 Particle swarm optimization 61
3.5.4 Bat algorithm 61
3.5.5 Firefly algorithm 62
3.5.6 Cuckoo search 62
3.5.7 Flower pollination algorithm 63
3.6 Notes on software 64
Bibliography 163
Index 171
About the author
Xin-She Yang obtained his PhD in Applied Mathematics from the University of Ox-
ford. He then worked at Cambridge University and National Physical Laboratory (UK)
as a Senior Research Scientist. Now he is Reader at Middlesex University London, and
an elected Bye-Fellow at Cambridge University.
He is also the IEEE Computer Intelligence Society (CIS) Chair for the Task Force
on Business Intelligence and Knowledge Management, Director of the International
Consortium for Optimization and Modelling in Science and Industry (iCOMSI), and
an Editor of Springer’s Book Series Springer Tracts in Nature-Inspired Computing
(STNIC).
With more than 20 years of research and teaching experience, he has authored
10 books and edited more than 15 books. He published more than 200 research pa-
pers in international peer-reviewed journals and conference proceedings with more
than 36 800 citations. He has been on the prestigious lists of Clarivate Analytics and
Web of Science highly cited researchers in 2016, 2017, and 2018. He serves on the
Editorial Boards of many international journals including International Journal of
Bio-Inspired Computation, Elsevier’s Journal of Computational Science (JoCS), In-
ternational Journal of Parallel, Emergent and Distributed Systems, and International
Journal of Computer Mathematics. He is also the Editor-in-Chief of the International
Journal of Mathematical Modelling and Numerical Optimisation.
This page intentionally left blank
Preface
Both data mining and machine learning are becoming popular subjects for university
courses and industrial applications. This popularity is partly driven by the Internet and
social media because they generate a huge amount of data every day, and the under-
standing of such big data requires sophisticated data mining techniques. In addition,
many applications such as facial recognition and robotics have extensively used ma-
chine learning algorithms, leading to the increasing popularity of artificial intelligence.
From a more general perspective, both data mining and machine learning are closely
related to optimization. After all, in many applications, we have to minimize costs,
errors, energy consumption, and environment impact and to maximize sustainabil-
ity, productivity, and efficiency. Many problems in data mining and machine learning
are usually formulated as optimization problems so that they can be solved by opti-
mization algorithms. Therefore, optimization techniques are closely related to many
techniques in data mining and machine learning.
Courses on data mining, machine learning, and optimization are often compulsory
for students, studying computer science, management science, engineering design, op-
erations research, data science, finance, and economics. All students have to develop
a certain level of data modeling skills so that they can process and interpret data for
classification, clustering, curve-fitting, and predictions. They should also be familiar
with machine learning techniques that are closely related to data mining so as to carry
out problem solving in many real-world applications. This book provides an introduc-
tion to all the major topics for such courses, covering the essential ideas of all key
algorithms and techniques for data mining, machine learning, and optimization.
Though there are over a dozen good books on such topics, most of these books are
either too specialized with specific readership or too lengthy (often over 500 pages).
This book fills in the gap with a compact and concise approach by focusing on the key
concepts, algorithms, and techniques at an introductory level. The main approach of
this book is informal, theorem-free, and practical. By using an informal approach all
fundamental topics required for data mining and machine learning are covered, and
the readers can gain such basic knowledge of all important algorithms with a focus
on their key ideas, without worrying about any tedious, rigorous mathematical proofs.
In addition, the practical approach provides about 30 worked examples in this book
so that the readers can see how each step of the algorithms and techniques works.
Thus, the readers can build their understanding and confidence gradually and in a
step-by-step manner. Furthermore, with the minimal requirements of basic high school
mathematics and some basic calculus, such an informal and practical style can also
enable the readers to learn the contents by self-study and at their own pace.
This book is suitable for undergraduates and graduates to rapidly develop all the
fundamental knowledge of data mining, machine learning, and optimization. It can
xii Preface
also be used by students and researchers as a reference to review and refresh their
knowledge in data mining, machine learning, optimization, computer science, and data
science.
Xin-She Yang
January 2019 in London
Acknowledgments
I would like to thank all my students and colleagues who have given valuable feedback
and comments on some of the contents and examples of this book. I also would like to
thank my editors, J. Scott Bentley and Michael Lutz, and the staff at Elsevier for their
professionalism. Last but not least, I thank my family for all the help and support.
Xin-She Yang
January 2019
This page intentionally left blank
Discovering Diverse Content Through
Random Scribd Documents
he blew and vexed the little brown pipe with rapid runs and nervous
fioriture, until great drops of sweat dripped from its round open
mouth. Sometimes, when he could not play fast enough to satisfy his
eagerness, he ran his finger up and down the vents; then, suddenly
lowering his instrument, he would scream, in a strong peasant voice,
verse after verse of the novena, to the accompaniment of the
zampogna (bagpipe). One was like a slow old Italian vettura, all
lumbered with luggage and held back by its drag; the other panting
and nervous at his work as an American locomotive, and as
constantly running off the rails. As they stood there playing, a little
group gathered round. A scamp of a boy left his sport to come and
beat time with a stick on the stone step before them; several children
clustered near; and one or two women, with black-eyed infants in
their arms, also paused to listen and sympathise.”
Every one who has been in Rome during Advent has seen this
group, or one mighty like it and equally characteristic. Turn to the
book (chapter on “Street Music in Rome”) for the little scene that
follows, for the music of the pifferari song, and for Mr Story’s
conversation with the enthusiastic piper, whom, with his
companions, he invited up into his house, where they agreeably
stunned him with their noisy music, to the delight of his children and
the astonishment of his servants, for whom piffero and zampogna
had long since lost all charm, and who doubtless looked upon their
introduction with somewhat of the same feeling of disgust with
which London flunkies would behold that of a couple of organ-
grinders and a cage of white mice into a Grosvenor Square drawing-
room. However, Mr Story took down the words of their quaint song,
which we find printed, probably for the first time, in his book, and he
also got from them some curious particulars of their wanderings. The
man who blew the little brown pipe was quite a character. He and his
companion had played together for three-and-thirty years, and their
sons, who presently came up, were to play together with them. “For
thirty-three years more, let us hope,” said Mr Story.
“‘Eh! Speriamo’ (let us hope so), was the answer of the pifferaro, as he showed
all his teeth in the broadest of smiles. Then, with a motion of his hand, he set both
the young men going, he himself joining in, straining out his cheeks, blowing all
the breath of his body into the little pipe, and running up and down the vents with
a sliding finger, until finally he brought up against a high, shrill note, to which he
gave the full force of his lungs, and after holding it in loud blast for a moment,
startled us by breaking off, without gradation, into a silence as sudden as if the
music had snapped short off like a pipe-stem.”
“My friend Count Cignale is a painter—he has a wonderful eye for colour and an
exquisite taste. He was making me a visit the other day, and in strolling about the
neighbourhood we were charmed with an old stone wall of as many colours as
Joseph’s coat: tender greys, dashed with creamy yellows and golden greens and
rich subdued reds, were mingled together in its plastered stonework; above
towered a row of glowing oleanders covered with clusters of roseate blossoms.
Nothing would do but that he must paint it, and so secure it at once for his
portfolio; for who knows, said he, that the owner will not take it into his head to
whitewash it next week, and ruin it? So he painted it, and a beautiful picture it
made. Within a week the owner made a call on us. He had seen Cignale painting
his wall with surprise, and deemed an apology necessary. ‘I am truly sorry,’ he said,
‘that the wall is left in such a condition. It ought to be painted all over with a
uniform tint, and I will do it at once. I have long had this intention, and I will no
longer omit to carry it into effect.’
“‘Let us beseech you,’ we both cried at once, ‘caro conte mio, to do no such thing,
for you will ruin your wall. What! whitewash it over!—it is profanation, sacrilege,
murder, and arson.’
“He opened his eyes. ‘Ah! I did not mean to whitewash it, but to wash it over
with a pearl colour,’ he answered.
“‘Whatever you do to it you will spoil it. Pray let it alone. It is beautiful now.’
“‘Is it, indeed?’ he cried. ‘Well, I hadn’t the least idea of that. But if you say so, I
will let it alone.’
“And thus we saved a wall.”
The preceding scrap reminds us of a passage from Alphonse Karr,
one of the most quietly-humorous of living French writers, who
relates, in one of his quaint, dreamy, desultory books, how a
neighbour of his, who lived in a poor thatched cottage on the fringe
of a wood, embowered in flowers, shaded by venerable trees,
refreshed by the balmiest of breezes, and enlivened by the songs of
countless birds, suddenly disappeared from the countryside. Karr,
who had long admired the sylvan retreat, and almost envied its
occupant, inquired his fate. He had become rich, he was told; a
legacy had enabled him to go and live in the town. He could afford to
rent two rooms with new furniture and a gaudy paper, and he looked
out upon a dirty street, along which omnibuses continually rolled.
“Poor rich man!” Karr pitying exclaims. He had whitewashed his
wall.
The Roman Ghetto furnishes the theme of one of Mr Story’s
longest and most lively chapters; Fountains and Aqueducts, Saints
and Superstitions, the Evil Eye, are the titles of three others. He
begins his second volume with a vivid and characteristic sketch of the
Markets of Rome, which are well worth the attention of foreign
visitors, especially of Englishmen, who will find their arrangements,
and much of what is there sold, to contrast strikingly with what they
are accustomed to in their own country. Carcasses of pigs and goats
adorned with scraps of gold-leaf and tinsel, blood puddings of a
brilliant crimson, poultry sold by retail—that is to say, piecemeal, so
that you may buy a wing, a leg, or even the head or gizzard of a fowl,
if so it please you. There is game of all sorts, and queer beasts and
fowls of many kinds are also there; the wild boar rough and snarling
—the slender tawny deer—porcupines (commonly eaten in Rome)—
most of our English game-birds—ortolans, beccaficoes, and a great
variety of singing-birds. Passing into the fruit and vegetable market,
one comes upon mushrooms of many colours, and some of them of
enormous size, most of which would in England be looked upon as
sudden death to the consumer, although in Italy they are found both
savoury and harmless. “Here are the grey porcini, the foliated
alberetti, and the orange-hued ovole; some of the latter of enormous
size, big enough to shelter a thousand fairies under their smooth and
painted domes. In each of these is a cleft stick, bearing a card from
the inspector of the market, granting permission to sell; for
mushrooms have proved fatal to so many cardinals, to say nothing of
popes and people, that they are naturally looked upon with
suspicion, and must all be officially examined to prevent accidents.”
Besides the fruits common in England, figs are very abundant, and of
many kinds; and when the good ones come in, in September, the
Romans of the lower classes assemble in the evenings, in the Piazza
Navona, for great feeds upon them. Five or six persons surround a
great basket and eat it empty, correcting possible evil results by a
glass of strong waters or a flask of red wine. But figs are a wholesome
fruit—much more so than one which at Rome, and in many parts of
Southern Europe, is the most popular of all—namely, the water-
melon. What millions of people, from the Danube’s banks to the
Portuguese coast, are daily refreshed the summer through by those
huge green gourds, hard and unpromising in outward aspect, but
revealing, at stroke of knife, rich store of rosy pulp, dotted with sable
seeds! Pesth is a great place for them; and daily, when morning
breaks, so long as they are in season, they are to be seen piled, all
along the river-side, in heaps like those of shot and shell in an
arsenal, only much broader and higher. All through the hot months,
in Hungary’s pleasant and interesting capital, few persons think of
dining without associating with the more heating viands a moiety or
enormous segment of one of those great cold fruits—a strange
digestive, as we Northerners should consider it, but found to answer
well in sultry climes. At Rome they are equally appreciated, and are
set above the choicest grapes. People make parties to go out of the
city and eat them; and this was especially the case some years ago,
when the authorities forbade their entrance on account of the
cholera, but were unable to prevent their extramural consumption.
In ordinary times you find heaps of them in the streets, especially in
the Piazza Navona, that great mart of fruit and frippery, vegetables,
old books, brilliant handkerchiefs, and other finery for the market-
women—old iron, old bottles, and rubbish of all kinds—amongst
which miscellany the patient investigator may sometimes discover
valuable copies of the classic authors and precious antique intagli, to
be purchased for a mere song. Here, as the story goes, a poor priest
once bought, for a few baiocchi, a large cut-glass bead which took his
fancy, and which a friend, more knowing than himself, afterwards
discovered to be a diamond of great value, now belonging, we are
told, to the Emperor of Russia. The priest disappeared, which leaves
any ingenious and inventive writer full liberty to build a romantic
tale upon the incident. The natural finale of the affair, Mr Story
opines, would have been for the priest to have married the Emperor’s
daughter, but his being in orders was an impediment; and so we are
justified in presuming that some less agreeable means was found of
easing him of his jewel, which, when he first possessed it, he took to
be a drop from a chandelier, but to which he of course clung with
desperate tenacity when enlightened as to the quality of the gem.
Rome ought to be a good preserve for fiction-writers, there are so
many family histories, traditions, and anecdotes current there, which
would serve the novelist’s turn. Edmund About availed himself of
one such in his tale of ‘Tolla;’ and another over-true tale was
interwoven, not very long since, in a pleasant novelet of Roman life
in the pages of this Magazine. Mr Story’s volumes abound in
suggestive passages of the kind. If Rome be an admirable residence
for an artist (and for some of the reasons why it is so, see the ‘Roba,’
i. p. 66, 67), it ought also to be an excellent one for a writer, were it
not that it is found by many unfavourable to mental exertion. This is
said to be particularly exemplified in the case of diplomatists, many
of whom, after a certain time passed in the Papal capital, are apt to
conceive an intense dislike to despatch-writing, and to keep their
Governments extremely uninformed concerning the state of the Holy
City and the prospects of Pontifical politics. We remember to have
been told, when in Rome, the names of more than one foreign
minister who had been recalled, it was asserted, for no other reason
but that nothing could induce him to write despatches. Rome is
certainly one of the places where there is most temptation, at least
for one half of the year, to neglect business for pleasure; but there is
possibly also something in the climate which disinclines many people
to headwork. It is much the fashion to abuse the Roman climate; and
this has been done, especially of late, by persons desirous to show
that Rome is an undesirable, because a highly insalubrious, capital
for united Italy. It is to be feared the grapes are sour, and that the
yellow flag now hoisted would be struck at the same time with the
French tricolour. Our own experience and observations induce us
very much to concur with those passages of Mr Story’s book which
relate to this question. “Rome has, with strangers, the reputation of
being unhealthy; but this opinion I cannot think well founded—to the
extent, at least, of the common belief.” Many maladies, virulent and
dangerous elsewhere, are very light in Rome; and for lung
complaints it is well known that people repair thither. The “Roman
fever,” as it is commonly called (intermittent and perniciosa), is
seldom suffered from by the better classes of Romans; and Mr Story
(who speaks with authority after his many years’ residence in Rome)
believes that, with a little prudence, it may easily be avoided. The
peasants of the Campagna are, it is well known, those who chiefly
suffer from it, and why? “Their food is poor, their habits careless,
their labour exhausting and performed in the sun, and they sleep
often on the bare ground or a little straw. And yet, despite the life
they lead and their various exposures, they are, for the most part, a
very strong and sturdy class.” Mr Story gives it as a fact that the
French soldiers who besieged Rome in ‘48, during the summer
months, suffered very little from fever, although sleeping out on the
Campagna; but they were better clothed and fed, and altogether
more careful of themselves, than the native peasants. Generally
speaking, the foreigners who visit Rome are less attentive than the
Romans to certain common rules for the preservation of health. They
eat and drink too much, and of the wrong things. They get hot, and
then plunge into cold churches or galleries; whereas an Italian flies
from a chill or current of air as from infection. Mr Story gives a few
simple rules, by following which he declares you may live twenty
years in Rome without a fever. He cautions Englishmen against
copious dinners, sherry and brandy, and his own countrymen against
the morning-dinner which they call a breakfast; and supplies other
useful hints and practical remarks. The subject is one which interests
many, and such are referred to the ‘Roba,’ i. p. 156–161, and to the
chapter on the Campagna, in which high authorities and ingenious
arguments are brought to prove that in old times it was not
insalubrious, and that in our own it need not be so. Population and
cultivation are perhaps all that are needed to render tracts healthy
that now are pestilential, but which assuredly were not so in the time
of the ancient Romans, since many of them, we know, were their
favourite sites for patrician villas. Much might be done by an
intelligent and active government, and especially by a good sanitary
commission. There was one clever gentleman who wrote that Rome
was ill fitted to be the capital of Italy on account of its deficiency in
buildings suitable for government offices! Where good reasons are
not to be found silly ones may be resorted to, but they of course only
weaken the cause they are intended to prop. And if it were to be
urged that all the worst plagues flesh is heir to, combine to render
Rome for the present impossible as capital of Italy, the most we
could admit, by way of compromise, and borrowing a well-known
answer, would be, “non tutti, ma Buona parte.”
CAXTONIANA:
A SERIES OF ESSAYS ON LIFE,
LITERATURE, AND MANNERS.
By the Author of ‘The Caxton Family.’
PART XV.
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
ebookmass.com