COMPUTING

Practical Neural se support

Networks (4) ur
co www. http://
w

eb
compi rg
nt/pag u.ac.uk/eng

sit
e.cfm? /
pge=6

e
732
Part 4 (final): applications and large neural nets
By Chris MacLeod and Grant Maxwell

In this final part of the series, we will consider some of the other
applications of neural nets and have a look at a few of the more difficult
questions facing neural net researchers.

During the series we’ve concentrated mostly Logic Input 1

on image recognition. This is because it is not
only one of the most important applications
of neural nets, but also the one that’s easy to
Logic Input 2
illustrate, as we can “see” the images.
Trainable truth
However, as explained in Part 1, the table output
neural net can be considered as a universal
logic system, capable (providing that the net- Logic Input 3
020324 - 4 - 11
work has three layers) of learning to produce
any truth table you like, see Figure 1.
If the output of the neurons is a sigmoid Figure 1. Universal trainable logic.
function, then it acts rather like ‘fuzzy logic’,
and is able to produce analogue outputs —
which can be useful for handling problems, in Input 1
Drive to left wheel
the real world, which are not ‘black and white’. (left bump sensor)

Applications in Robotics Input 2

(right bump sensor) Drive to right wheel
To illustrate how a trainable truth table is
020324 - 4 - 12
useful, consider a robot controller as shown
in Figure 2.
The network can learn to make the robot Figure 2. Robot controller.
behave as we wish if we train it to respond
to the sensor inputs and produce outputs to
control its motors. This is a simple example of strengthening the weights which is called Evolutionary Algorithm for
what is termed an Artificial Nervous System caused this behaviour. This can be Re-enforcement Learning (EARL).
(ANS) and an animal-like robot controlled accomplished automatically in the
through a learning control system is some- robot itself (of course we need to
times referred to as an Animat. make some judgements about Applications in Control
You could teach this robot to behave as what’s good and bad behaviour and We can extend the control aspects
you wish with the Back Propagation Algo- program these in, so that the robot mentioned above to other systems.
rithm. Alternatively, you could have it learn has a value system). For example, suppose we want a
to survive by itself, using a re-enforcement A slight variant on this is to use a system to control a DC motor, see
learning algorithm. Re-enforcement methods Genetic Algorithm to evolve network Figure 3.
work by rewarding behaviour which benefits types from a population which lead You might recognise the inputs
the robot (e.g., avoiding obstacles) and to good behaviour — this algorithm as the typical PID types used in

28 Elektor Electronics 4/2003

 COMPUTING
Input 1 train the network with. The ANN will sort
(position sensor) itself out during the training process to make
sense of the relationships and transforma-
Input 2 Output to motor tions between inputs and outputs.
(speed sensor) One future application for this type of
‘intelligent sensing’ system is a human to
machine interface. If you were to try, for
Input 3 example, and make realistic prosthetic limbs
(acceleration sensor)
020324 - 4 - 13 and interface them with part of the nervous
system, you’d have to try and make sense of
Figure 3. Motor control. the outputs of the many nerves (and perhaps
other biometric inputs and noise), all firing
with different frequencies. Some of these
Voltage inputs may be irrelevant and others related
to each other in subtle ways. Processing
information from this type of complex and
Sample 1 Sample n non-linear system is obviously something
which is right up the neural net’s ‘street’.

Recognising Waveforms
In the man-machine interface discussed
above, you’d have to process the waveforms
generated by nerves. And indeed, the Neural
Net can recognise patterns in time (like
these) as well as in space (like images), by
sampling them and feeding them to the
Network trained inputs; Figure 4 shows the idea.
to recognise
waveform.
Of course this leads to the problem,
already mentioned in part 2, of the image or,
in this case, waveform, having to be exactly
the right size and being placed in the middle
Output of the grid.
020324 - 4 - 14

Figure 4. A sampled waveform fed to a neural net. Pre-processing

Sizing and centring an image or data, so that
it’s suitable for a network, is called pre-pro-
conventional controllers. A neuro- Man Machine Interface cessing. There are many different ways of
controller like this might learn with and Intelligent Sensing achieving it. In the case of a sampled wave-
BP, from a conventional controller, form, the network is looking at a ‘window’ of
by using its outputs as targets. In the control system above, the net- time, and as the waveform passes it by, one
work responded to sensors and pro- sample at a time, it will eventually lie in the
Alternatively, and more interest- duced an appropriate output. Neural centre of the window and the network will
ingly, it could learn from experience, Nets are good at untangling complex recognise it.
using a Re-enforcement Learning interrelated inputs like this — you Exactly the same thing can be said of an
Algorithm or a Genetic Algorithm to don’t have to understand the theory image-recognising network. It to can also be
set its weights. of the system, just have examples to scanned across the image, as shown in Figure
5. Again, during this process, the pattern to be
recognised will eventually end up in the cen-
When scan reaches end of row, it
tre of the grid. Actually this is not dissimilar to
Network starts moves down one pixel and starts what our eyes do when we study a scene, we
scanning image at back
top left. Moving
(unconsciously) scan the image, looking for
one pixel at a time patterns to recognise. One important point
about training a network for this sort of task,
is that we have to train it to recognise ‘noise’
or irrelevant data as well as the wanted pat-
Scan finishes when it has
reached bottom right. tern or it will give false positives.
So much for centring the image; what
020324 - 4 - 15 about making it the correct size? Well, again
this can be done by pre-processing. If we
Figure 5. Image scanning. start with a large window and scan it across

4/2003 Elektor Electronics 29

COMPUTING
the image, we can make the window pro- Image information is reduced to
gressively smaller, so that sooner or later the correct size for network inputs by
pattern we want to recognise is correctly Network input area pixel averaging or similar method.
starts as whole
sized on the grid, see Figure 6. image size
Since the neural net inputs are of a fixed
size, fitting different sizes of image into it
When one size has
means manipulating the original image size scanned the image,
until it fits onto the grid. This is easily done network input area
Input area keeps reducing until it
by averaging adjacent pixels together until reduces in size
is smallest practical size.
the image is the correct size for the network.
020324 - 4 - 16

Modular Networks Figure 6. Image sizing.

The neural networks in the brain recognise
objects through features. Take the letter L, for
example. It has two features, a vertical line the description of the Artificial Neu- systems are easier to work with
and a horizontal line. No matter where it is ron given in part 1 doesn’t bear too when information is coded like this
placed on a grid, nor how small or larger it is, much resemblance to the Biological (or pulse-width coded).
it still has these two features. one described. In particular, the bio- Artificial models have been made
If we could have a network which recog- logical neural net frequency-coded to operate in this manner and are
nises verticals and another which recognises its activity, as shown in Figure 8. often called ‘spiky’ neurons. Figure
horizontals, then a third can integrate them Some scientists think that this is 9 shows two simple models, one of a
together to form a whole — have a look at important and that, what we term ‘spiky’ (frequency modulated) neu-
Figure 7. ‘thought’ and ‘consciousness’ are in ron and the other of a pulse-width
This is a simple example of a modular net- fact sequences of these pulses rac- modulated neuron. The weights and
work. Rather than one large and fully con- ing around the nervous system, other parameters of these are easily
nected network trying to learn everything, interacting with one another. On a set using a GA (or if you fancy a bit
we break the task down into smaller chunks, more pragmatic note, many control of mathematics, by modifying BP).

Image

Net 1: Net 2:
Vertical Horizontal
detector detector

A neuron which is only weakly

A neuron which is highly stimulated (receiving little signal
Net 3: stimulated produces many
Combines outputs through its dendrites) produces
pulses per second few pulses per second
of 1 and 2
Voltage Voltage
020324 - 4 - 17

Figure 7. Modular networks. Time Time

each one tackled by a smaller net. Research

has shown that the brain is organised in this
way — not as one huge net, but as lots of
smaller modules acting on their own tasks
and integrating their results together into a Output Output
coherent whole.
Research is ongoing at the moment to see
if it is possible to use Evolutionary Algo-
rithms to grow groups of such modules and Time Time
form an integrated system. Strongly stimulated Weakly stimulated Artificial
Artificial neurons produce Neurons produce low outputs.
high outputs. 020324 - 4 - 18

Neural Function
The sharp-eyed reader will have spotted that Figure 8. Activity in a biological and artificial neuron.

30 Elektor Electronics 4/2003

 COMPUTING
circuits are essentially laid flat — two dimen-
sional.
It may seem that both the density of neu-
rons and their wiring are insurmountable
Output Output
problems if we wish to make biologically real-
1 1
istic neural nets. But, it should be remem-
bered that signals travel through real neurons
t1 t2 t1 t2
0 0 at only a couple of hundred metres a second,
Time Time whereas in the artificial equivalent they can
t2 is inversely proportional t1 is proportional to Sum of travel at sixty percent the speed of light. So,
to Sum of inputs. inputs. t2 + t1 is a constant. what the artificial net loses in connectivity, it
Frequency modulated Pulse width modulated
gains in speed.
It might be possible to solve the connec-
020324 - 4 - 19
tivity problem, even with current technology,
if each neuron is given an address like a host
Figure 9. Spiky neurons. in a communications network and this is used
for routing signals to and from neurons rather
than having delegated wiring for each signal.
Connections On top of this, each of these neu- Making hardware neural nets is a problem
and technology rons may be connected with 1,000 as there are no integrated circuit technologies
others (some neurons have 100,000 readily available for the amateur to experi-
One final area of interest relates to connections). This means that there ment with (large discrete neural nets require
the size of the biological neural net are perhaps 100 trillion connections many components) — particularly ones with
compared with the artificial one. You in the brain. reconfigurable wiring, which is required for
may remember that the brain is built Modern technology cannot match Evolutionary ANNs. Some experimenters
from 100 billion neurons (the same either the density of neurons or their have tried Field Programmable Gate Arrays
number of stars as there are in our connections. The neuron is actually (FPGAs), but these are not ideally suited for
galaxy). This easily makes it the about the same size as a transistor, the task as each neuron takes up a great deal
most complex structure in the but is arranged in a dense three of chip ‘real estate’. This type of problem
known universe. dimensional structure. Of course, our exists because the neural net was modelled
on the brain without regard for electronics —
and so is inefficient to implement. We also
need to develop a neuron (in the sense of a
Correction general purpose processing unit, capable of
In part 2, some errors have crept into the text which explains Figure 4. Some of learning) which is electronically feasible and
the equations were spread over two lines and the resulting hyphen is easily con- efficient.
fused with the minus sign. Also, one of the equations contained a mistake. The
correct equations are given below.
Finally
1. Calculate errors of output neurons We hope you’ve enjoyed this trip around the
Errorα = outα (1 – outα) (Targetα – outα)
world of neural nets enough to try experi-
Errorβ = outβ (1 – outβ) (Targetβ – outβ)
menting with some yourself. Listed below are
2. Change output layer weights some basic references, which are a good
W+Aα = WAα + ηErrorα outA starting point for further exploration.
W+Aβ = WAβ + ηErrorβ outA (020324-4)

W+Bα = WBα + ηErrorα outB

W+Bβ = WBβ + ηErrorβ outB
W+Cα = WCα + ηErrorα outC Acknowledgements
W+Cβ = WCβ + ηErrorβ outC The authors would like to express their gratitude
to Ann B. Reddipogu, Niccolo Capanni and Sethu-
3. Calculate (back-propagate) hidden layer errors raman Muthuraman for their help putting together
ErrorA = outA (1 – outA) (ErrorαWAα + ErrorβWAβ) these articles.
ErrorB = outB (1 – outB) (ErrorαWBα + ErrorβWBβ)
ErrorC = outC (1 – outC) (ErrorαWCα + ErrorβWCβ) Further reading
Some popular books on Neural Nets for those
4. Change hidden layer weights
who’d like to delve deeper:
W+λA = WλA + ηErrorA inλ
W+ΩA = WΩA + ηErrorA inΩ
K. Gurney, An Introduction to Neural Net-
W+λB = WλB + ηErrorB inλ
W+ΩB = WΩB + ηErrorB inΩ works, UCL Press, 1997.
W+λC = WλC + ηErrorC inλ I. Pratt, Artificial Intelligence, MacMillan, 1994.
W+ΩC = WΩC + ηErrorC inΩ P. D. Wasserman, Neural Computing: theory
and practice, Van Nostrand Reinhold, 1989.

4/2003 Elektor Electronics 31