Download the full version of the ebook at

https://ebookgate.com

Theory of Neural Information Processing

Systems A. C. C. Coolen

https://ebookgate.com/product/theory-of-neural-
information-processing-systems-a-c-c-coolen/

Explore and download more ebook at https://ebookgate.com

 Recommended digital products (PDF, EPUB, MOBI) that
you can download immediately if you are interested.

Chemical Dependency A Systems Approach C. Aaron Mcneece

https://ebookgate.com/product/chemical-dependency-a-systems-approach-
c-aaron-mcneece/

ebookgate.com

Systems Thinking Michael C. Jackson

https://ebookgate.com/product/systems-thinking-michael-c-jackson/

ebookgate.com

The Image Processing Handbook 5th Edition John C. Russ

https://ebookgate.com/product/the-image-processing-handbook-5th-
edition-john-c-russ/

ebookgate.com

The Image Processing Handbook 7th Edition John C. Russ

https://ebookgate.com/product/the-image-processing-handbook-7th-
edition-john-c-russ/

ebookgate.com
Digital Image Processing Using MATLAB R Rafael C. Gonzalez

https://ebookgate.com/product/digital-image-processing-using-matlab-r-
rafael-c-gonzalez/

ebookgate.com

Using Information Technology 7th Edition Stacey C. Sawyer

https://ebookgate.com/product/using-information-technology-7th-
edition-stacey-c-sawyer/

ebookgate.com

Principles and Practice of Brachytherapy Using

Afterloading Systems 1st Edition C. A. Joslin

https://ebookgate.com/product/principles-and-practice-of-
brachytherapy-using-afterloading-systems-1st-edition-c-a-joslin/

ebookgate.com

Neural Based Orthogonal Data Fitting The EXIN Neural

Networks Adaptive and Learning Systems for Signal
Processing Communications and Control Series 1st Edition
Giansalvo Cirrincione
https://ebookgate.com/product/neural-based-orthogonal-data-fitting-
the-exin-neural-networks-adaptive-and-learning-systems-for-signal-
processing-communications-and-control-series-1st-edition-giansalvo-
cirrincione/
ebookgate.com

Memory Evolutive Systems Hierarchy Emergence Cognition 1st

Edition A C Ehresmann

https://ebookgate.com/product/memory-evolutive-systems-hierarchy-
emergence-cognition-1st-edition-a-c-ehresmann/

ebookgate.com
Theory of Neural Information
Processing Systems
This page intentionally left blank
Theory of Neural
Information Processing
Systems

A. C. C. Coolen, R. Kühn, P. Sollich

King’s College, London

1
 3
Great Clarendon Street, Oxford OX2 6DP
Oxford University Press is a department of the University of Oxford.
It furthers the University’s objective of excellence in research, scholarship,
and education by publishing worldwide in
Oxford New York
Auckland Cape Town Dar es Salaam Hong Kong Karachi
Kuala Lumpur Madrid Melbourne Mexico City Nairobi
New Delhi Shanghai Taipei Toronto
With offices in
Argentina Austria Brazil Chile Czech Republic France Greece
Guatemala Hungary Italy Japan Poland Portugal Singapore
South Korea Switzerland Thailand Turkey Ukraine Vietnam
Oxford is a registered trade mark of Oxford University Press
in the UK and in certain other countries
Published in the United States
by Oxford University Press Inc., New York
© Oxford University Press, 2005
The moral rights of the authors have been asserted
Database right Oxford University Press (maker)
First published 2005
All rights reserved. No part of this publication may be reproduced,
stored in a retrieval system, or transmitted, in any form or by any means,
without the prior permission in writing of Oxford University Press,
or as expressly permitted by law, or under terms agreed with the appropriate
reprographics rights organization. Enquiries concerning reproduction
outside the scope of the above should be sent to the Rights Department,
Oxford University Press, at the address above
You must not circulate this book in any other binding or cover
and you must impose the same condition on any acquirer
British Library Cataloguing in Publication Data
Data available
Library of Congress Cataloging in Publication Data
Data available
Typeset by Newgen Imaging Systems (P) Ltd., Chennai, India
Printed in Great Britain
on acid-free paper by
Biddles Ltd., King’s Lynn, Norfolk
ISBN 0–19–853023–4 978–0–19–853023–7
ISBN 0–19–853024–2(Pbk.) 978–0–19–853024–4 (Pbk.)
1 3 5 7 9 10 8 6 4 2
Preface

The study of the principles behind information processing in complex

networks of simple interacting decision-making units, be these units cells
(‘neurons’) in brains or in other nervous tissue, or electronic processors
(or even software) in artificial systems inspired by biological neural net-
works, is one of the few truly interdisciplinary scientific enterprises. The
field involves biologists (and psychologists), computer scientists, engin-
eers, physicists, and mathematicians; over the years these have all moved
in and out of the centre stage in various combinations and permutations,
modulated and triggered by advances in experimental, mathematical, or
computational techniques. The reason for its unique interdisciplinary char-
acter is that this multifaceted area of research, which from now on we will
simply denote as the study of ‘neural information processing systems’, is one
of the few which meets the fundamental requirement of fruitful interdiscip-
linary science: all disciplines get something interesting and worthwhile out
of the collaboration. The biologist benefits from tapping into the mathem-
atical techniques offered by the more quantitative sciences, the computer
scientist or engineer who is interested in machine learning finds inspiration
in biology, and the theoretical physicist or applied mathematician finds new
and challenging application domains for newly developed mathematical
techniques.
We owe the knowledge that brain tissue consists of complicated net-
works of interacting brain cells mainly to the work (carried out towards
the end of the nineteenth century) of two individuals, who shared the
1906 Nobel Prize in medicine in recognition of this achievement: Camillo
Golgi, who invented a revolutionary staining method that for the first time
enabled us to actually see neurons and their connections under a micro-
scope, and Santiago Ramón y Cajal, who used this new technique to map
out systematically and draw in meticulous and artful detail the various
cell types and network structures which were now being revealed (in fact
Cajal had originally wanted to be an artist). Initially and for several dec-
ades neural networks continued to be regarded as a branch of medicine
and biology. This situation changed, however, with the birth of program-
mable computing machines around the time of the Second World War,
when the word ‘computer’ was still used to denote a person doing com-
putations. It came to be realized that programmable machines might be
made to ‘think’, and, conversely, that human thinking could perhaps be
vi Preface

understood in the language of programmable machines. This period also

saw the conception of ‘information theory’, which was largely the brain
child of Claude Shannon. Probably the first to focus systematically on
the information processing capabilities of neural networks were Warren
McCulloch and Walter Pitts, who published in 1943 a paper (‘A Logical
Calculus of the Ideas Immanent in Nervous Activity’) that can safely be
regarded as the starting point of our research field. Looking back, one can-
not help observing that McCulloch and Pitts were surprisingly typical of
the kind of scientist that henceforth would tend to be drawn into this area.
McCulloch had studied philosophy and psychology, then moved into medi-
cine, and ended up in a laboratory of electronic engineering. Pitts, who was
only 20 at the time when ‘A Logical Calculus’ was published, initially stud-
ied mathematics and also ended up in electronic engineering, but he never
received a formal academic degree. It is not unreasonable to take the view
that bringing together these disparate scientific backgrounds and interests
was crucial to the achievement of McCulloch and Pitts.
The field never lost the interdisciplinary flavour with which it was born.
Since the 1940s its popularity peaked at (roughly) 20-year intervals, with
a second wave in the 1960s (the launch of the perceptron, and the explor-
ation of learning rules for individual neurons), and a more recent wave
in the 1980s (which saw the development of learning rules for multilayer
neural networks, and the extensive application of statistical mechanics tech-
niques to recurrent ones). Extrapolation of this trend would suggest that
interesting times might soon be upon us. However, the interdisciplinary
character of neural network research was also found to have drawbacks: it
is neither a trivial matter to keep the disciplines involved connected (due to
language barriers, motivation differences, lack of appropriate journals etc.),
nor to execute effective quality control (which here requires both depth and
unusual breadth). As a result, several important discoveries had to be made
more than once, before they found themselves recognized as such (and hence
credit was not always allocated where in retrospect it should have been).
In this context one may appreciate the special role of textbooks, which allow
those interested in contributing towards this field to avoid first having to
study discipline specific research papers from fields in which they have not
been trained.
Following the most recent wave of activity in the theory of neural
information processing systems, several excellent textbooks intended
specifically for an interdisciplinary audience were published around 1990.
Since then, however, the connectivity between disciplines has again
decreased. Neural network research still continues with energy and passion,
but now mostly according to the individual scientific agendas, the style, and
the notation of the traditional stake-holding disciplines. As a consequence,
those neural network theory textbooks which deal with the progress which
has been achieved since (roughly) 1990, tend to be of a different character.
 Preface vii

They are excellent expositions, but often quite specialized, and focused
primarily on the questions and methods of a single discipline.
The present textbook aims to partly remedy this situation, by giving an
explicit, coherent, and up-to-date account of the modern theory of neural
information processing systems, aimed at students with an undergraduate
degree in any quantitative discipline (e.g. computer science, physics,
engineering, biology, or mathematics). The book tries to cover all the major
theoretical developments from the 1940s right up to the present day, as they
have been contributed over the years by the different disciplines, within a
uniform style of presentation and of mathematical notation. It starts with
simple model neurons in the spirit of McCulloch and Pitts, and includes
not only the mainstream topics of the 1960s and 1980s (perceptrons,
multilayer networks, learning rules and learning dynamics, Boltzmann
machines, statistical mechanics of recurrent networks etc.) but also the more
recent developments of, say, the last 15 years (such as the application
of Bayesian methods, Gaussian processes and support vector machines)
and an introduction to Amari’s information geometry. The text is fully
self-contained, including introductions to the various discipline-specific
mathematical tools (e.g. information theory, or statistical mechanics), and
with multiple exercises on each topic. It does not assume prior familiar-
ity with neural networks; only the basic elements of calculus and linear
algebra, and an open mind. The book is pitched at the typical postgraduate
student: it hopes to bring students with an undergraduate degree to the level
where they can actually contribute to research in an academic or industrial
environment. As such, the book could be used either in the classroom as
a textbook for postgraduate lecture courses, or for the training of indi-
vidual PhD students in the first phase of their studies, or as a reference
text for those who are already involved in neural information processing
research. The material has been developed, used, and tested by the authors
over a period of some 8 years, split into four individual one semester lecture
courses, in the context of a one-year inter-disciplinary Master’s programme
in Information Processing and Neural Networks at King’s College London.

London, January 2005 Ton Coolen, Reimer Kühn, Peter Sollich

This page intentionally left blank
Acknowledgements

It is our great pleasure to thank all those colleagues who, through dis-
cussions, joint research, joint teaching, or otherwise, have directly or
indirectly contributed towards this textbook. Especially we would like to
thank (in alphabetical order): David Barber, Michael Biehl, Siegfried Bös,
Zoubin Ghahramani, Dieter Grensing, Leo van Hemmen, Heinz Horner,
Wolfgang Kinzel, Charles Mace, Jort van Mourik, Hidetoshi Nishimori,
Manfred Opper, Hamish Rae, David Saad, David Sherrington, Nikos
Skantzos, Chris Williams, and Annette Zippelius.
This page intentionally left blank
Contents

Part I Introduction to neural networks 1

1 General introduction 3

1.1 Principles of neural information processing 3

1.2 Biological neurons and model neurons 7
1.3 Universality of McCulloch–Pitts neurons 21
1.4 Exercises 25

2 Layered networks 29

2.1 Linear separability 29

2.2 Multilayer networks 33
2.3 The perceptron 36
2.4 Learning in layered networks: error backpropagation 46
2.5 Learning dynamics in small learning rate perceptrons 52
2.6 Numerical simulations 58
2.7 Exercises 64

3 Recurrent networks with binary neurons 69

3.1 Noiseless recurrent networks 70

3.2 Synaptic symmetry and Lyapunov functions 77
3.3 Information processing in recurrent networks 81
3.4 Exercises 87

4 Notes and suggestions for further reading 89

Part II Advanced neural networks 93

5 Competitive unsupervised learning processes 95

5.1 Vector quantization 95

5.2 Soft vector quantization 98
xii Contents

5.3 Time-dependent learning rates 110

5.4 Self-organizing maps 114
5.5 Exercises 122

6 Bayesian techniques in supervised learning 127

6.1 Preliminaries and introduction 127

6.2 Bayesian learning of network weights 136
6.3 Predictions with error bars: real-valued functions 144
6.4 Predictions with error bars: binary classification 152
6.5 Bayesian model selection 156
6.6 Practicalities: measuring curvature 163
6.7 Exercises 164

7 Gaussian processes 169

7.1 The underlying idea 169

7.2 Examples of networks reducing to Gaussian processes 172
7.3 The ‘priors over functions’ point of view 176
7.4 Stationary covariance functions 177
7.5 Learning and prediction with Gaussian processes 180
7.6 Exercises 182

8 Support vector machines for binary classification 185

8.1 Optimal separating plane for linearly separable tasks 185

8.2 Representation in terms of support vectors 189
8.3 Preprocessing and SVM kernels 197
8.4 Exercises 202

9 Notes and suggestions for further reading 205

Part III Information theory and neural networks 207

10 Measuring information 209

10.1 Brute force: counting messages 209

10.2 Exploiting message likelihood differences via coding 212
10.3 Proposal for a measure of information 218
 Contents xiii

11 Identification of entropy as an information measure 223

11.1 Coding theory and the Kraft inequality 223

11.2 Entropy and optimal coding 229
11.3 Shannon’s original proof 233

12 Building blocks of Shannon’s information theory 235

12.1 Entropy 235

12.2 Joint and conditional entropy 240
12.3 Relative entropy and mutual information 245
12.4 Information measures for continuous random variables 251
12.5 Exercises 258

13 Information theory and statistical inference 261

13.1 Maximum likelihood estimation 261

13.2 The maximum entropy principle 264
13.3 Exercises 270

14 Applications to neural networks 273

14.1 Supervised learning: Boltzmann machines 273

14.2 Maximum information preservation 281
14.3 Neuronal specialization 285
14.4 Detection of coherent features 294
14.5 The effect of non-linearities 297
14.6 Introduction to Amari’s information geometry 299
14.7 Simple applications of information geometry 306
14.8 Exercises 311

15 Notes and suggestions for further reading 313

Part IV Macroscopic analysis of dynamics 315

16 Network operation: macroscopic dynamics 317

16.1 Microscopic dynamics in probabilistic form 318

16.2 Sequential dynamics 324
16.3 Parallel dynamics 338
16.4 Exercises 345
xiv Contents

17 Dynamics of online learning in binary perceptrons 349

17.1 Probabilistic definitions, performance measures 349

17.2 Explicit learning rules 353
17.3 Optimized learning rules 368
17.4 Exercises 382

18 Dynamics of online gradient descent learning 385

18.1 Online gradient descent 385

18.2 Learning from noisy examples 392
18.3 Exercises 394

19 Notes and suggestions for further reading 397

Part V Equilibrium statistical mechanics of neural networks 399

20 Basics of equilibrium statistical mechanics 401

20.1 Stationary distributions and ergodicity 401

20.2 Detailed balance and interaction symmetry 408
20.3 Equilibrium statistical mechanics: concepts, definitions 413
20.4 A simple example: storing a single pattern 419
20.5 Phase transitions and ergodicity breaking 425
20.6 Exercises 430

21 Network operation: equilibrium analysis 437

21.1 Hopfield model with finite number of patterns 438

21.2 Introduction to replica theory: the SK model 447
21.3 Hopfield model with an extensive number of patterns 460
21.4 Exercises 480

22 Gardner theory of task realizability 489

22.1 The space of interactions 489

22.2 Capacity of perceptrons—definition and toy example 494
22.3 Capacity of perceptrons—random inputs 498

23 Notes and suggestions for further reading 507

 Contents xv

Appendix A: Probability theory in a nutshell 511

A.1 Discrete event sets 511

A.2 Continuous event sets 513
A.3 Averages of specific random variables 515

Appendix B: Conditions for the central limit theorem to apply 517

B.1 Moment condition 517

B.2 Lindeberg’s theorem 518

Appendix C: Some simple summation identities 521

Appendix D: Gaussian integrals and probability distributions 523

D.1 General properties of Gaussian integrals 523

D.2 Gaussian probability distributions 527
D.3 A list of specific Gaussian integrals 531

Appendix E: Matrix identities 537

E.1 Block inverses 537

E.2 The Woodbury formula 538

Appendix F: The δ-distribution 539

F.1 Definition 539

F.2 δ(x) as solution of the Liouville equation 540
F.3 Representations, relations, and generalizations 541

Appendix G: Inequalities based on convexity 543

Appendix H: Metrics for parametrized probability distributions 549

H.1 Local distance definitions 549

H.2 The triangular inequality 550
H.3 Global distance definitions 551
xvi Contents

Appendix I: Saddle-point integration 553

References 555

Index 563
 Part Introduction to neural
I networks

This first part of the text more or less covers the main principles and
developments in neural information processing theory from its beginnings
in the 1940s up to the mid-1980s (albeit expanded in places with further
calculations, examples, and pointers to later developments, in order to link
the material to that in subsequent chapters). The particular cutoff point
chosen more or less marks the stage where the field became significantly
more mathematical in nature. As a result, this first part is mathematically
less demanding than the others.
We start with an introduction to the basic characteristics and principles
of information processing by networks of interacting nerve cells (neurons),
as observed in biology. The next stage is to use this knowledge to define
simplified versions (mathematical models) of neuron-type information
processing units and their connections, hopefully capturing the essence
of their functioning, followed by a demonstration of the universality
(in information processing terms) of one specific important neuron model:
the McCulloch–Pitts neuron.
We then turn to the so-called perceptrons (McCulloch–Pitts neurons
equipped with clever ‘learning rules’), proving convergence and also
presenting a preliminary analysis of the dynamics of these learning rules,
and to the so-called error backpropagation learning algorithm for mul-
tilayer feed-forward networks of model neurons. In addition we explore
via a number of simple examples the dynamical properties and the possible
use in information processing tasks of recurrent neural networks, including
a brief discussion of the crucial role of synaptic symmetry.
This page intentionally left blank
 General introduction
1
1.1 Principles of neural information processing
The brain is a piece of hardware that performs quite sophisticated
information processing tasks, using microscopic elements and operations
which are fundamentally different from the ones on which present-day
computers are based. The microscopic processors in the brain, the nerve
cells or neurons (see Figure 1.1), are excitable brain cells which can
be triggered to produce electrical pulses (spikes), by which they commun-
icate with neighbouring cells. These neurons are rather noisy elements,
which operate in parallel. They do not execute a fixed ‘program’ on a
given set of ‘data’, but they communicate signals through relay stations
(the synapses), located at the junctions where the output channel (axon)
of one neuron meets an input channel (dendrite) or the cell body of
another. The strengths of these relay stations, or synaptic efficacies, are
continuously being updated, albeit slowly. The neurons of each given brain
region are organized and wired in a specific network, the structure of
which can vary from very regular (especially in regions responsible for
pre-processing of sensory data) to almost amorphous (especially in the
‘higher’ regions of the brain, where cognitive functions are performed),
see Figure 1.2. The dynamic relay stations, or synapses, in combination
with some adjustable intrinsic neuron properties, represent both ‘data’ and
‘program’ of the network. Hence program and data change all the time.
We can roughly summarize the main similarities and differences between
conventional computer systems and biological neural networks in the
following table:

Computers (specifications as of 2004) Biological neural networks

Processors Neurons
operation speed ∼ 109 Hz operation speed ∼ 102 Hz
signal/noise  1 signal/noise ∼ 1
signal velocity ∼ 108 m/s signal velocity ∼ 1 m/s
connections ∼ 10 connections ∼ 104
Sequential operation Parallel operation
Program and data Connections and neuron characteristics
External programming Self-programming and adaptation
Not robust against hardware failure Robust against hardware failure
Cannot deal with unforeseen data Messy, unforeseen, and inconsistent data
Another random document with
no related content on Scribd:
 CONTINUOUS PROGRESS.
Already the new bakery was becoming too small for the work, and
in August 1908 it was agreed to complete the central portion of the
building by the addition of other two storeys at a cost of £2,263. Just
prior to this time, however, the committee felt compelled to dispense
with the services of their foreman baker, and Mr W. H. Bell, who had
been his assistant, was appointed in his place. For several years
Belfast Society used the stable at the bakery for stabling their horses,
while they were also permitted to erect a cart shed and a shoeing
forge. Later this society erected stables, etc., for themselves on
property adjoining the bakery, and the ground which had been
occupied by them was utilised by the Federation for necessary
extensions to the bakery.
By March 1909 the extended premises were ready for occupation.
It was not long ere the directors were in the mortar tub again,
however, for another extension costing almost £4,000 was entered
on in the autumn of 1910; and not long after it was completed there
came, in December 1911, an urgent request from the advisory
committee for more ovens, and another extension, at a total cost of
almost £5,000, was entered on and completed in 1912. These various
extensions practically completed the bakery as it stands at present,
except for alterations, minor in themselves, which were made from
time to time during the war period with the object of increasing the
working facilities.
In July 1909 the committee recorded their satisfaction that the
average weekly sales from the bakery amounted to £1,219, while in
March 1910 the weekly turnover had reached 500 sacks; and to meet
the increasing demand three new ovens had to be erected. The
Bakery continued to win prizes at the Agricultural Hall and other
exhibitions, thus proving that Co-operative bread baking on a large
scale was equal to producing bread of the finest quality.
Early in 1910 the directors were saddened by the news that one of
the members of the first advisory committee, Mr Crook, of Lisburn,
had passed away.
 Although most of the societies in the North were doing well, there
were one or two which were in a bad way. In 1908 Lurgan Society
had to close its doors. The Federation were creditors to the extent of
£114, and when settling day came it was found that the assets of the
dead society were only capable of returning 1/6 in the pound. In 1910
Newry Society went the same way. The Federation were creditors to
the extent of £200, and it was expected that the assets would realise
10/ per pound. In Newry the Federation made temporary
arrangements to carry on the bread trade, as had been done in
Banbridge, but after some time this course was abandoned.
Shortly after the new bakery was opened the two Dublin societies
were in consultation with the committee of the Federation about the
erection of a bakery, and the committee agreed to assist them. The
two societies were unable to agree, however, and the idea of a federal
bakery for Dublin was departed from. In 1910 Dublin Industrial
Society erected a bakery for themselves, and the opening of this
bakery was followed in a short time by the amalgamation of the two
societies. The Dublin Society, however, experienced considerable
difficulty in acquiring the knack of baking good bread—their position
in this respect recalls some of the earlier experiences of the U.C.B.S.
—and the Federation readily consented to Mr Bell or his assistant
visiting Dublin to put them on right lines. In 1913, when the distress
due to the strike was at its height in Dublin and the Trade Unions
Congress was coming to the rescue with financial support, the
Industrial Society received a contract for the supply of from 3,000 to
5,000 loaves daily. As they were unable to handle the contract in
their own bakery they secured the assistance of the U.C.B.S. bakery
in Belfast, which supplied them with the needed quantity of bread
during the period covered by the contract. Later, during the period of
the war, the Industrial Society was in considerable difficulty for a
time, and those responsible for its management were exceedingly
anxious that the U.C.B.S. should take over the bakery. This was not
done, however, and fortunately the society was able to maintain and
even to improve its position.
 GOVERNMENT CONTRACTS.
At the beginning of 1909 the Bakery secured a contract for the
supply of bread and flour to Newtownards Barracks, and since then
have supplied from time to time that Barracks, as well as those of
Holywood, Belfast, and Kilroot, while, either directly or through local
societies, they have been successful in securing contracts for various
local institutions, one society during the war securing the contract
for the Admiralty.
In 1912 the Society had loaned some motor lorries for an
excursion, and during the day a painful accident took place whereby
one child was killed and three others were severely injured. The
Society’s motorman was completely exonerated from blame for the
accident, but in token of their sympathy the Society paid the doctor’s
fees and granted £55 to the relatives. In 1913 two of the societies
were experiencing difficulty in selling the bread because of the unfair
competition to which they were being subjected in attempts to cause
them to give up the bread trade, but the Federation came to their
assistance by allowing them a little additional discount on their
purchases. In 1911 the Irish Agricultural Wholesale Society was
appointed agent in Ireland for the Federation, and some time later it
was stated that since that federation had become agent the trade had
increased. At the Dublin Congress in 1914 the Federation erected a
scone and oatcake baking plant as part of their exhibit in the
Congress exhibition, and this exhibit was an object of much interest
to the Dublin visitors to the exhibition.
 THE WAR AND ITS EFFECTS.
In Ireland, as in Scotland, the commanding position of the
U.C.B.S. had a steadying effect on the price of bread at the outbreak
of war. The Society carried good stocks of flour, and by adopting the
fixed policy of regulating the price of bread by the average price of
the flour in stock was able to maintain the price at a lower level than
the current price of flour warranted. In this way, while it was
possible to maintain full stocks of flour, the Bakery was able to keep
the price of bread at a halfpenny per 4–lb. loaf below the price at
which other bakers wished to sell it, and so saved the people of the
North many thousands of pounds.
The difficulties of transport were experienced by the branch,
however, in a much more marked degree than by the parent body. All
coal, flour, sugar, etc., used in the bakery had to be imported, and as
transport costs went up so also did prices, until bread was being sold
in Belfast at one shilling for the 4–lb. loaf. So short did the supply of
coal become that in 1915 Belfast Corporation was unable to supply
the Bakery with coke for firing purposes and a supply had to be sent
over from Glasgow. Flour and sugar also became very scarce, but
notwithstanding those facts the output of the Bakery continued to
increase rapidly until the coming of Government Regulation flour,
with its huge proportion of offal and other nastinesses, created a
distaste for bread amongst the public. At the outbreak of war the
output of the Bakery averaged 766 sacks per week, while by the end
of 1916 this had risen to 892 sacks per week, an increase of 16½ per
cent. in two and a half years. From then it gradually declined, until at
the end of the 63rd quarter of the working life of the branch it was
only 683 sacks per week. From that time and during the next six
months the trade again increased, so that by the end of the period
with which this history deals, although it had not reached the high-
water mark of 1916, all the evidence went to show that that point
would soon be left behind.
In 1918, with the object of relieving the congestion at M‘Neil
Street, it was resolved to establish an oatcake baking department at
Belfast, and for this purpose several hot-plates were transferred to
the Belfast bakery, as well as several girls from the M‘Neil Street
oatcake factory. This department has since proved a valuable adjunct
to the branch, the plant having had to be augmented and the number
of bakers employed increased.
Just at the end of the period Belfast became involved in one of the
most widespread strikes in the history of the Labour movement, and
practically all work except that of bread baking was brought to a
standstill. The electricity workers of the Corporation came out on
strike along with the other workmen, but fortunately for the branch
it had a power-generating plant of its own, and so was able to
continue at work. By this means the trade of the branch was
materially increased. The fact that the Federation has always been
willing to meet the requests of the employees in a reasonable manner
has always enabled the branch to maintain its popularity with the
Belfast workmen, and so impressed were the executives of the
Operative Bakers’ Society with the fairness and even liberality of the
committee that on one occasion an official letter was sent from the
union in which the board of management were thanked for the
assistance which their prompt action in agreeing to the request of the
union for a shorter working week had been in enabling that
concession to be imposed in the baking trade of the city.
When the branch was established in Belfast the distinct
understanding on which the U.C.B.S. took action was that as soon as
convenient it should be taken over by the local societies, but as the
years pass the likelihood of this being done seems to become more
remote. As has already been stated, the branch has been of great
assistance not only to Belfast Society but also to the majority of other
societies in the North. It has fostered the Co-operative spirit and Co-
operative idealism. From a weakling which was very much in need of
the fostering care of the movement in Scotland Co-operation in
Ireland, and especially in the North of Ireland, has grown to be a
strong and healthy organisation, with its centre in Belfast and with
branches scattered all over Ulster. It is probable that the Irish
societies are strong enough now to take over the branch if they were
so disposed, but the farseeing spirits amongst them see that there is
other work lying to their hand to which, relieved of the working of
the bakery, they can turn their undivided attention.
 On the other hand, the proposal, adopted in 1917, that the
Federation should proceed, as and when expedient, to plant
branches in other centres of Co-operation in Ireland, will in course of
time cause the U.C.B.S. in Ireland to cease to be an exclusively North
of Ireland concern. It will do much, just as the other federation in
Ireland, the I.A.W.S., is doing much, to weld together into one united
movement the whole of the Co-operative societies in the country. It
will serve for Ireland, as it and its kindred associations in Scotland
and England are serving to-day, as an illustration of the fact that Co-
operation is not merely a principle for adoption by a few farmers or a
few workmen in a given district, but is also a principle on which
these isolated groups can be united into one powerful whole, by
becoming an institution where all can meet on a common platform.
 CHAPTER XV.
A NEW PRESIDENT.

A GREAT DEMONSTRATION—A NEW TYPE OF OVEN—ST

MUNGO HALLS—TEAROOMS BURNED DOWN—MR
M‘CULLOCH RETIRES; MR GERRARD ELECTED
PRESIDENT—AN AMALGAMATION PROPOSAL—PURVEY
AND CONTRACT DEPARTMENT—THE FIRST MOTOR VAN
—A BREAD EXPERIMENT—PROPAGANDA AND PUBLICITY
—UNEMPLOYMENT—THE EFFECT ON CO-OPERATION—
CO-OPERATION THE REMEDY—SOME BUSINESS ITEMS—
THE SOCIETY’S POSITION.

In tracing the history of the two branches at Clydebank and Belfast

we have wandered far ahead of the general development of the
Society, and must now retrace our steps to the main road of our story
at the point at which we left it. In the period which began with the
beginning of the thirty-third year the directors were kept very busy
with business which was connected with the development of the two
new branches, but the parent institution was not allowed to suffer
from neglect. The great demonstration which was held in the
summer of 1902 to celebrate the attainment of an average output of
3,000 sacks per week proved a great success. The demonstration was
in two parts. In the first place, the employees and their families and
friends, to a number which filled three special trains, went picnicing
in the morning to Milngavie. In the second place, a gigantic vehicular
procession, which included about 100 vehicles and 150 horses,
paraded through the city and out the Milngavie Road as far as
Canniesburn Toll, returning to M‘Neil Street by another route. This
procession was headed by a charabanc containing the members of
the Society’s band, and the gaily decorated lorries and vans attracted
much attention as they passed through the streets. All the mottoes
displayed in the procession had some reference to Co-operation.
But, while it is good to demonstrate and advertisement has
certainly its uses, bread must also be baked, and the object of a Co-
operative society is, or ought to be, to produce the best possible
bread at a minimum cost. This was a fact of which there was not
much danger that the directors would lose sight, and even if they did
the representatives of the societies would very quickly remind them
of the lapse. They were about to build a new bakery; and, with the
view of testing the efficiency of a type of oven then practically
unknown in Scotland, they decided to erect two draw-plate ovens at
M‘Neil Street. The tests seem to have been satisfactory, for in both
Clydebank and Belfast branches this type of oven formed the
majority of those installed.
 ST MUNGO HALLS.
Meantime other matters were engaging their attention. When the
new stables had been erected on the land purchased at the south
corner of South York Street and Govan Street a considerable portion
of the ground—more than half indeed—remained unbuilt on, and
early in 1902 the educational committee of the Society came forward
with the recommendation that in any further building which might
be erected a hall which could be used for educational purposes
should be included. The committee also were desirous that the
Society should have a hall of their own, and later in the same year the
architects were instructed to prepare plans for the utilisation of this
vacant land which should include halls and accommodation for the
headquarters of the purvey department. At the quarterly meeting in
March 1903 power was given to proceed with the erection of the
buildings. These were to consist of five halls, containing
accommodation for from 200 to 1,500 people, with the necessary
siderooms, etc., and with ample accommodation for the purvey
department. Permission was also given for the erection of three
tenements of dwellinghouses, with shops on the ground floor; the
total cost to be from £14,000 to £15,000 for the halls and purvey
department buildings, and £6,400 for the tenements. Eleven months
passed, however, before the building of the halls was begun, and they
were not completed until 1906.
Toward the end of 1903 the Paisley Road tearooms were destroyed
by fire, and in restoring them the committee decided to add another
storey. The landlord agreed to bear a proportion of the cost, and on
the reconstruction being completed they were named the
“Wheatsheaf” tearooms.
 MR M‘CULLOCH RETIRES.
For several years the chairman had been desirous of retiring, but
had been induced to remain in office until the schemes on which he
had set his heart—the erection of Clydebank and Belfast branches—
were well on the way. In 1904, however, he definitely decided to end
his official connection with the Society. For fifteen years he had acted
as president. He was first appointed to the board by his society in
June 1888, and had continued to act on the board until his retiral.
Mr Daniel H. Gerrard was elected president in succession to Mr
M‘Culloch. For a short time he had represented St George Society on
the board prior to his election, while the active part which he had
taken, as a member of the Scottish Sectional Board, in combating the
boycott in the later “’nineties” and his position as chairman of St
George Society had made him well known to the majority of Scottish
Co-operators.
 AN AMALGAMATION PROPOSAL.
In the autumn of 1903 an interesting proposal was made by the
directors of Hamilton Baking Society. This was to the effect that the
two societies should become amalgamated. During the following year
several meetings took place between representatives of the two
societies, but ultimately negotiations were broken off, the members
of Hamilton Baking Society having expressed disapproval of the
proposal. If the scheme could have been carried through it would
have left the Baking Society in the position of being the only
federated society producing bread in the West, with the exception of
Chapelhall, and might have paved the way for that society to come in
also.
 PURVEY AND CONTRACT DEPARTMENT.
Meantime the work of the purvey department was growing. For the
two years 1902 and 1903 the department secured contracts from
Glasgow Corporation for the supply of bags of eatables on Children’s
Day. In the aggregate the bags supplied numbered 190,000, of a total
value of about £1,800, and requiring somewhere about twenty-five
tons of flour for their manufacture. When the “Wheatsheaf”
tearooms were reopened after the fire an “at home” was held to
which the customers were invited. The purvey department was also
an offerer on most occasions when purveys on a large scale had to be
carried through, often with success. The department was successful
in securing the purveying contract for the tearooms in the East-End
Exhibition. It was also a successful offerer in 1904 for the supply of
bread to Gailes Territorial Camp, and in the following year
Jamestown Camp was supplied from Clydebank.
The tearooms continued to give cause for a considerable amount of
anxiety on the part of the committee. Sometimes a small profit was
made on the working and sometimes a loss resulted, but there never
was that increase in trade for which the board thought they had a
right to look. With the erection of St Mungo Hall and the
transference of the catering headquarters to South York Street the
need for Main Street as a depot for the catering department
disappeared, but for some time it was carried on as a workmen’s
tearoom, always without any signs of assured success, however.
Finally, in 1904, it was given up altogether. Nor were the London
Street halls or the Union rooms much more successful. Several
experiments were made at London Street, with the object of making
the place more popular. The whole of the three upper flats were
taken by the Society, and several trade union and friendly society
branches made the rooms their headquarters, but the place was
never really popular. An attempt which was made to run the second
flat as workmen’s dining and tea rooms did not meet with much
success, and it seemed as if anything the board could do was not of
much use in popularising the place. Nor were the Union rooms much
more successful. They just managed to keep going, but they did not
become, as had been hoped, a rendezvous for the men and women of
the Co-operative movement. Nevertheless the committee did not
despair. They always kept on hoping that the tide would turn and
renewed the lease time after time, still looking for the Co-operative
patronage which never came in sufficient quantity to make the place
a success.
 THE FIRST MOTOR VAN.
For practically the whole of the nineteenth century the power-
driven road vehicle had to struggle against the bigotry of the people
and the interested opposition of the proprietors of other systems of
locomotion. Steam-driven road cars and wagons were in use long
before Stephenson had designed and completed his first railway, but
they met with strong and unreasoning opposition on the part of
many people instigated by the owners of post and passenger coaches,
who saw in the new method of locomotion a menace to their welfare,
and also by other horse owners, who found these steam-driven, noisy
vehicles intolerable nuisances which frightened into panic the high-
strung horses unfortunate enough to meet them on the roads. This
opposition translated itself into Acts of Parliament which imposed
heavy taxation on the newer mode of locomotion, and culminated in
1836 in the famous “man with the red flag” provision in the Act of
that year, which was the means of clearing practically every “faster
than walking pace” power-driven vehicle off the roads. This Act,
passed at the instance and in the interests of the railways, had a most
detrimental effect on the development of road and also of canal
traffic, and left the country completely at the mercy of the railway
companies until the “red flag” restriction was practically laughed out
of existence in the last decade of the century by the development of
petrol-driven motors, and the prohibition was removed in 1896.
After 1896 the development of power-driven vehicles for road
traffic made rapid strides, but it was not until some seven years later
that the U.C.B.S., on the instigation of the makers, put a bread motor
van on the road for a trial. After working for some months, the
committee decided to send a note of their experience of its working
to the makers, and at the same time to point out some defects which
they had discovered. Gradually, however, the new means of delivery
superseded the old, until all the Society’s long-distance work was
done by means of motor vans. For many years, however, the
construction of the motor engines placed a serious obstacle in the
way of the adoption of motor vans for short journeys or for journeys
which entailed frequent stops. The engine, from its nature, requires
that the cylinder should be charged and the charge compressed
before ignition can take place. Drivers were therefore faced with the
alternatives of putting the engine out of gear and leaving it running—
at a considerable expenditure for fuel which gave no return in work
done—or of having much laborious cranking for the purpose of
charging the cylinder and compressing and igniting the gas, the latter
plan, particularly in cold weather, often entailing considerable
expenditure of time as well as of energy. Nowadays, however, most
up-to-date motor engines have a small electric engine for doing this
work.
 A BREAD EXPERIMENT.
For a long time it had been the wish of the manager and directors
of the Baking Society to introduce a natural working day into the
bread trade, and they had done their best some years earlier to have
the matter taken up by the societies and the public, but without
success. Now, in the autumn of 1905, another attempt was made, a
squad of bakers beginning work at 9 a.m.; the bread thus baked
being delivered on the following morning. A number of the societies
in Glasgow and neighbourhood were induced to take up the sale of
this bread, with the result that the sales speedily rose; but after a trial
which lasted several months the scheme was abandoned, as the
directors found that instead of helping the general work of delivery it
was proving a hindrance to that work.
This decision of the committee met with a considerable amount of
opposition from delegates to the quarterly meeting, but the directors
were not in a position to do other than they had done. The real
obstacle was to be found in the fact that the public insisted on having
new bread, and with a number of the bakers beginning work late in
the day for the production of “natural working day” bread, as it had
been named, there was not labour enough in the early morning to
meet the demand for new bread by ten o’clock. Thus the second
attempt of the Baking Society directors to introduce a natural
working day into the baking trade came to naught, through no fault
of theirs, but because of a public who would insist on having bread
steaming from the ovens.
 PROPAGANDA AND PUBLICITY.
During these years the directors continued to adopt every
reasonable method of keeping the Federation and its productions
before the public. They took advantage of every opportunity offered
by exhibitions to show the quality of goods which were produced by
the Society; they exhibited also at shows, and were occasionally
successful in securing prizes, although the rush conditions under
which their bread had to be produced in order to cope with the ever-
increasing demands of the societies did not provide the conditions
necessary for producing bread of show quality.
At the same time the entertaining of women’s guilds, conference
associations, and other Co-operative organisations was further
developed, and many Co-operators who before had but a very hazy
idea of the size and importance of the U.C.B.S. were enlightened as a
result of these visits. The cake shows, also, held annually, were
excellent propaganda. Here, inside one hall, were to be found
specimens of all the productions of the three bakeries; and these
productions came latterly to include many novelties which were not
to be found in everyday use, as well as the more common specimens
of cakes, buns, bread, etc., and a large variety of biscuits. Thus the
cake show came to be regarded as the annual exhibition of Bakery
productions, which indeed it was, and it was one of the most eagerly
anticipated Co-operative functions of the year.
 UNEMPLOYMENT.
Unfortunately, during the years 1904–1905 there was a very severe
economic crisis which was the cause of widespread unemployment,
and Glasgow and the West suffered severely. Such economic crises
had been periodic during the past century, and differed materially
from those which occurred in earlier centuries. Until the beginning
of the nineteenth century economic crises were really due to actual
scarcity of foodstuffs, and people went hungry as much because there
was not enough food to go round as because they had no money with
which to buy their share of what there was. Thus, in Scotland, until
about the close of the Napoleonic wars, the periods of comparative
plenty or of scarcity depended largely on the character of the
seasons. While it is true that the common people were always in
want, in bad seasons they died of starvation.
With the coming of the industrial era, however, there took place a
gradual change. While prices still depended on the seasons, and were
moderately or extravagantly high as the seasons were good or bad,
there entered into the problem a new factor, and the people became
poor and were unable to purchase because they had produced too
much and there was not an effective market for the goods. As it has
been tersely put: “The shoemaker’s children went barefoot because
their father had produced too many boots; and the tailor’s, naked,
because he had made too many clothes.”
This was a new phenomenon for which history provides no
parallel, and it has persisted, ever increasing in intensity, until
towards the end of the first decade of the twentieth century as many
as 42 per cent. of the breadwinners in a respectable working-class
district in the East-End of Glasgow have been found to be
unemployed at the same time.
These periods of unemployment seem to be in the form of more or
less regularly recurring cycles. There is first a gradually increasing
inflation of the volume of trade. New works are started, old works are
enlarged, and everywhere there is a boom, until the zenith is
reached. Then comes a gradual slackening off. The supply of goods
has outstripped the limits of effective demand and sales gradually
decline as warehouses become full. The rate of slackening increases;
statisticians begin to watch the rapidly ascending unemployment
curve, which in itself does not record the full slackening, as many
workers are on short time. This increase of unemployment still
further weakens the effective demand for goods and still further
accelerates the growth of unemployment, with the result that in
about another year the unemployment crisis is reached, soup
kitchens and relief works open; thousands of hitherto steady
workmen become derelicts—and the huge commercial and industrial
concerns take advantage of the slackness to squeeze small rivals out
of business or to swallow them up, while at the same time improving
their own machinery of production.
Another feature of these industrial phenomena is the acceleration
of their periodicity. During the decades which marked the end of the
nineteenth century and the beginning of the twentieth, the number
of years which elapsed between one crisis and the next was becoming
smaller and the periods were appreciably shorter than those of the
earlier cycles. Fourteen, twelve, or ten years used to be the intervals
between crises, but in the last forty years this interval has gradually
shortened, until only three years intervened between the end of one
crisis and the beginning of the next. This is in strict accordance with
what we would expect to find. The means of supplying the effective
demand of the people for goods has increased out of all proportion to
the increase in that effective demand, with the natural result that the
market gets choked full to overflowing in an ever-shortening time.
When to this is added the continuous perfecting of processes and
equally continuous speeding up of production, we see that under the
conditions which obtain it necessarily follows that the periods which
must elapse before the markets are choked up with goods must get
progressively shorter, while the periods necessary to relieve the glut
again grow gradually longer. Thus, we had a crisis in the late
“’seventies,” another in 1888–89; another in 1893, of a milder type;
another in 1898; another, very acute, in 1904–05; and another, the
worst the industrial world in Britain has known, in 1908–09–10;
while evidences were not wanting when war broke out that we were
again on the downward sweep of the cycle.
 THE EFFECT ON CO-OPERATION.
The Co-operative movement in general is profoundly influenced by
conditions of unemployment, which is our excuse for what has been
written above. The vast majority of Co-operators depend for their
incomes on their being employed in ordinary industrial
undertakings, and when the machinery of industry breaks down their
spending power is affected. Those in fortunate positions can carry on
economically for a longer or shorter period on their investments in
their societies, but those fortunate individuals constitute but a small
proportion of the whole membership. When an unemployment crisis
comes it means distress to a large number of Co-operators, for those
who have large savings to fall back on are usually in more or less
important positions in the concerns by which they are employed and
are amongst the last to be dismissed—are, in fact, rarely out of
employment at all. Sales of Co-operative societies go down, while
working expenses continue. In this respect, however, the Baking
Society has always been the last to suffer, for people continue to buy
bread when they have almost ceased to be able to purchase anything
else. But there is always during such crises a proportion of Co-
operators, varying in different districts, whose incomes are barely
sufficient to meet physical needs while they are in constant
employment and who are thrown on their beam ends by even a
fortnight of unemployment, and the hardship of these crises is that
this is the class of people on whom the curse of unemployment falls
first and on whom it rests longest.
Fortunately, the milk of human kindness is not quite dried up in
Co-operators, and so soon as genuine cases of distress are known
steps are taken to ameliorate the condition. In this process of
amelioration the Baking Society has always played a big part. We
have already seen that during the miners’ and engineers’ strikes and
during periods of distress due to unemployment the Society
distributed thousands of pounds worth of bread, and now, during
1904–05, the same policy was pursued.