A Model for Multimodal Humanlike Perception based on

Modular Hierarchical Symbolic Information Processing,

Knowledge Integration, and Learning
Rosemarie Velik
Vienna University of Technology
Gusshausstrae 27-29/384
1040 Vienna
+43 158801-38463
[email protected]








ABSTRACT
Automatic surveillance systems as well as autonomous robots are
technical systems which would proIit Irom the ability oI
humanlike perception Ior eIIective, eIIicient, and Ilexible
operation. In this article, a model Ior humanlike perception is
introduced based on hierarchical modular Iusion oI multi-sensory
data, symbolic inIormation processing, integration oI knowledge
and memory, and learning. The model is inspired by Iindings
Irom neuroscience. InIormation Irom diverse sensors is
transIormed into symbolic representations and processed in
parallel in a modular, hierarchical Iashion. Higher-level symbolic
inIormation is gained by combination oI lower-level symbols.
Feedbacks Irom higher levels to lower levels are possible.
Relations between symbols can be learned Irom examples. Stored
knowledge inIluences the activation oI symbols. The model and
the underlying concepts are explained by means oI a concrete
example taken Irom building automation.
Keywords
Bionics, Humanlike Perception, Multisensory Integration,
Symbolic InIormation Processing, Learning, Knowledge-based
Systems, Building Automation.
1. INTRODUCTION
Over the last decades, automation technology has made serious
progress in observing and controlling processes in order to
automate them. In Iactory environments, where the number oI
possible occurring situations and states is quite limited and well
known, observation and controlling oI most industrial processes
do no longer pose an unsolvable problem. However, the situation
changes iI we go Irom the observation oI industrial processes to
the detection oI objects, events, and scenarios in a real-world
environment. Here, the number oI possible occurring objects,
events, and situations is almost inIinite. As research Irom image
processing and audio-data processing has shown, Ior a machine,
recognition oI real world situations is a task Iar Irom trivial. On
the other hand, humans, even small children, can perceive such a
'real-world environment almost eIIortlessly. The challenging
question is what gives humans the ability to perIorm these tasks
and how to design machines that perIorm in a similarly eIIicient
way.

In this article, a model Ior humanlike perception is introduced,
which is inspired by Iindings Irom neuroscience. The proposed
model is based on hierarchical modular Iusion oI multi-sensory
data, symbolic inIormation processing, integration oI knowledge
and memory, and learning. The model and the underlying
concepts are explained by means oI a concrete example taken
Irom building automation.
2. RELATED WORK
The technical model Ior humanlike perception proposed in this
article is based on modular Iusion oI multi-sensory data, symbolic
inIormation processing, integration oI knowledge and memory,
and learning. There have already been some attempts to model
human perception, Iuse multi-sensory data, process inIormation
symbolically, and to integrate diIIerent Iorms oI knowledge. The
approaches suggested in literature to solve these problems are
disparate.
In |16|, a mathematical model oI the human perception process is
presented. The proposed systems theoretical Iramework describes
the principles oI human perception as a concatenation oI
nonlinear vector mappings. |18| introduce a model Ior distributed
perception systems Ior ubiquitous computing applications using a
layered architecture. |4| suggest a neural network Ior multi-
sensory perception. This network processes auditory and visual
inIormation separately in the Iirst layers beIore combining it in
the next layers. |7| outline a strategy and a control architecture to
allow a mobile robot to navigate in an indoor environment on a
planned path. The navigation system oI the mobile robot
integrates the position estimation obtained by a vision system
with the position estimated by odometry, using a Kalman Iilter
Iramework. Obstacle detection is perIormed by means oI a set oI
ultrasonic sensors. |24| document the rationale and design oI a
multimodal interIace to a pervasive/ubiquitous computing system

Permission to make digital or hard copies oI all or part oI this work Ior
personal or classroom use is granted without Iee provided that copies are
not made or distributed Ior proIit or commercial advantage and that
copies bear this notice and the Iull citation on the Iirst page. To copy
otherwise, or republish, to post on servers or to redistribute to lists,
requires prior speciIic permission and/or a Iee.
Bionetics 07, December 10-13, 2007, Budapest, Hungary
Copyright 2007 ICST 978-963-9799-11-0.
.
that supports independent living by older people in their own
homes.
|21| introduce a multimodal system Ior recognizing oIIice activity
that relies on a cascade oI Hidden Markov Models (HMMs). The
HMMs are used to diagnose states oI user activity based on real-
time streams oI evidence Irom video, audio, and computer
(keyboard and mouse) interactions. |22| attempts to develop a
scientiIic and technological Ioundation Ior interactive
environments. ThereIore, machine perception techniques using
acoustics, speech, computer vision, and mechanical sensors are
employed. |5| points out that many human activities Iollow a
loosely deIined script in which individuals assume roles. A
layered, component-based, soItware architecture model is
proposed and illustrated with a system Ior real-time composition
oI synchronized audio-video streams Ior recording activity within
a meeting or lecture.
|9| claim that it is impossible to reconstruct the environment
'bottom-up Irom the sensory inIormation alone and that prior
knowledge is needed to interpret ambiguous sensory inIormation.
Bayesian inIerence is suggested to combine prior knowledge with
observational, sensory evidence to inIer the most probable
interpretation oI the environment. |10| mention that diIIerent
sources oI inIormation do not always keep the same relative
reliability and that a rational perceptual system should adjust the
weights that it assigns to diIIerent inIormation sources. A
Bayesian approach is suggested to understand how the reliability
oI diIIerent sources oI inIormation, including prior knowledge,
should be combined by a perceptual system.
|2| exploit location inIormation about sound signals to conclude
Irom what source a detected sound originates. For example, a
sound originating Irom the manipulation oI dishes is likely to be
detected in the kitchen near the sink. |25| describe a system Ior
the recognition oI mixtures oI noise sources in acoustic input
signals. The problem is approached by utilizing both bottom-up
signal analysis and top-down predictions oI higher-level models.
|8| presents a prediction-driven approach to interpret sound
signals. The analysis is a process oI reconciliation between the
observed acoustic Ieatures and the predictions oI an internal
model oI the sound-producing entities in the environment. |6|
propose a scheme where perception crucially involves comparison
processes between incoming stimuli and expected perceptions
built Irom previous perceptions.
Attempts to process sensor inIormation symbolically have been
made by |23| and |13| who suggest a layered architecture Ior this
purpose. |15| attempt to achieve symbol grounding by adding a
sensory concept to an abstract symbol.
3. NEUROSCIENTIFIC BACKGROUNDS
As the technical model Ior humanlike perception is inspired by
neuroscientiIic insights about the perceptual system oI the human
brain, this section summarizes backgrounds which serve as
archetype Ior the model. When elaborating such a technical
model, it has to be considered that in neuroscience up to now
there do not exist complete, uniIied concepts oI the perceptual
system oI the brain. There are still many blind spots and
controversial viewpoints, which make it diIIicult Ior an engineer
to transIorm the neuroscientiIic model into a technical one.
3.1 Bottom-up and top-down processes in
perception
Perception is the result oI top-down processing and bottom-up
processing working together. |12| Bottom-up processing, also
reIerred to as data-based processing, is based on incoming data
Irom the receptors oI the human sense organs. Incoming data are
always the starting point Ior perception. Without incoming data,
there is no perception. Top-down processing, also labeled
knowledge-based processing, is based on knowledge. This
knowledge can be Iactual knowledge about objects, pre-
experience, knowledge about the context in which an object
occurs, or expectation.
3.2 Modular hierarchical architecture for
information processing
Human perception does not rely on a single modality but involves
diIIerent perceptual systems visual perception, auditory
perception, somatosensory perception, olIactory perception, and
gustatory perception. The somatosensory system actually
comprises a whole group oI sensory systems, responsible Ior
cutaneous sensations, proprioception, and kinesthesis. Each oI
these senses is served by a speciIic type oI receptor and projects
separately to the brain. OI most interest Ior the model proposed in
section 4 are the cutaneous sensations, which are based on the
stimulation oI receptors in the skin responsible Ior tactile
sensation, vibration sense, temperature sense, and pain sense. |11|
The perceptive system oI the brain has a modular hierarchical
structure and consists oI at least three cortical zones built one
above the other. They are reIerred to as primary, secondary, and
tertiary area. |17| A scheme oI the structural organization oI the
human perceptive system is depicted in Iigure 1.

Fig. 1. Structural organization of the human perceptive
system.
Each sensory modality has its own primary and secondary area
located in a speciIic area oI the brain. The primary areas receive
impulses Irom the periphery. They consist mainly oI neurons
which have extremely high speciIicity. The neurons oI the
primary cortical visual system only respond to the narrowly
specialized properties oI visual stimuli like the character oI lines,
shades oI color, or the direction oI movement. Neurons oI the
primary auditory cortex only respond to highly diIIerentiated
properties oI acoustic stimuli. The primary areas are surrounded
by systems oI secondary cortical zones. In the secondary areas,
inIormation coming Irom the primary zones is processed. The
cells oI the secondary cortical zones have a much lower degree oI
speciIicity. Neurons oI the secondary visual cortex respond to
complex visual images like Iaces. Neurons oI the secondary
auditory context respond to complex acoustic signals like
melodies. The tertiary zones, also reIerred to as zones oI
overlapping, are responsible Ior enabling groups oI several
analyzers to work concertedly. Here, the inIormation coming
Irom the diIIerent sense organs being processed separately and in
parallel until now in the particular primary and secondary zones is
merged. The great majority oI neurons oI these zones are
multimodal in character. An example Ior a processing result oI
this area would be the association oI the visual image oI a person
to the auditory perception oI the person talking.
From the lowest to the highest layer, inIormation oI increasing
complexity is processed. The lowest level is Iully developed at an
early stage oI development oI the human brain whereas higher
levels oI inIormation processing are not Iully developed in a
child`s brain until the age oI 7 years. Connections and
associations oI lower layers have to be learned and evolved Iirst,
beIore higher layers can evolve their Iull Iunctionality.
3.3 Unimodal and multimodal binding of
information
As just described, the perceptive system oI the brain processes
inIormation in a modular hierarchical Iashion. Simple Ieatures
extracted Irom sensory inIormation are combined to more
complex unimodal representations and are then merged to a
uniIied multimodal perception. The Iundamental question, which
is Iar Irom trivial, is how these combinations are perIormed. In
neuroscience, this question is reIerred to as binding problem.
Binding occurs in many diIIerent kinds oI brain processes. To
explain how coherent representations can be Iormed oI
inIormation that is distributed throughout the brain, diIIerent
binding mechanisms have been hypothesized which are not
mutually exclusive. In literature, basically Iour diIIerent potential
solutions to the binding problem are suggested and usually
discussed in the context oI visual perception: combination coding,
population coding, temporal coding, and attention. |27|, |20| The
true solution oI how binding is solved in the brain is still not
known and might be a combination oI some or all oI these
mechanisms.
3.4 Knowledge integration
Perception is Iacilitated by knowledge: Iactual knowledge about
objects, pre-experience, knowledge about the context in which the
object occurs, and expectation. Much oI what we take Ior granted
as the way the world is as we perceive it is in Iact what we
have learned about the world as we remember it. Much oI what
we take Ior perception is in Iact memory. We Irequently see
things that are not there, simply because we expect them to be
there. Adults project their expectations onto the world all the
time. They largely construct rather than perceive the world around
them. |26| A Iundamental question is, on which level knowledge
interacts with sensory perception. The answers to this question are
controversy.
3.5 Neural versus symbolic information
processing
InIormation processing in the human brain is generally considered
as being carried out by interacting neurons as well as chemicals
like hormones and peptides. However, due to the complexity oI
mental processes, it is complicated iI not impossible to project
mental states to low-level explanations on the behavior oI
neurons, synapses, and chemicals.
To reduce complexity, inIormation processing in the human brain
can also be considered on the more abstract level oI symbols.
According to the theory oI symbolic systems, the mind is a
symbol system and cognition is symbol manipulation. |14|
Examples Ior symbols are objects, characters, Iigures, sound, or
colors used to represent abstract ideas and concepts.
4. TECHNICAL MODEL FOR HUMAN-
LIKE PERCEPTION
In this section, a technical model Ior a humanlike perceptive
system is presented. A humanlike perception system has a wide
range oI potential applications. Autonomous robots with such
capabilities could navigate selI-suIIiciently in their environment,
automatic surveillance systems could help to pare down personnel
Ior monotonous observation tasks and increase saIety oI e.g.
patients in a hospital. The application envisioned in this article is
located in the Iield oI building automation and automatic
surveillance systems. For perceiving complex events and
scenarios in buildings, diIIerent sensors are needed. The proposed
system shall be capable oI handling and processing such a Ilood
oI inIormation. However, to clariIy the underlying concepts,
simpliIied examples are used. The principles oI the model are Iirst
described generally, Iollowed by a description by means oI a
concrete example: it shall be detected iI a person is entering a
room, leaving the room, walking around in the room, standing in
the room, opening the door, or closing it.
First, the pursued strategy Ior sensor selection and inIormation
processing is outlined. Next, an architecture Ior modular,
hierarchical processing oI incoming sensor data is introduced
Iollowed by a description oI how sensor data can be transIormed
into symbolic inIormation. Fourth, a concept oI how to integrate
knowledge into symbolic inIormation processing is draIted. Last,
it is reIlected about how to bind symbolic inIormation and how
learning can be integrated into this process.
4.1 Sensor selection for multi-sensory
information processing
To perceive the environment, the human body has diverse sensory
receptors. Accordingly, a technical system has to be equipped
with diverse sensors to perceive events, situations, and scenarios
in a building. The sensors to be chosen strongly depend on the
desired situations to be recognized.
To perceive the diIIerent scenarios mentioned beIore, a room is
equipped with the Iollowing sensor types: a motion detector, two
tactile Iloor sensors, two light barriers, a door contact sensor, a
video camera, and a microphone. The sensors are mounted at the
positions depicted in Iigure 2 and have the property to have partly
overlapping sensory Iields oI perception and to provide partly
redundant inIormation. The motion detector, tactile sensors, light
barriers, and door contact sensor provide a binary sensor signal
(zero or one). The motion detector detects movement in the room.
The tactile Iloor sensors detect iI an object is present in the room
and whether it is present in the leIt or the right halI oI the room.
The two light barriers detect whether an object passed the door
and by combining the inIormation oI both in which direction
it passed. The door contact sensor determines whether the door is
open or closed. The video camera evaluates whether a person is
present in the room. The microphone detects the noise that occurs
when someone walks or when the door is opened or closed.

Fig. 2. Test bed: Room equipped with different sensors.
The inIormation gathered Irom each sensor type is intentionally
kept simple. The intended strategy is to utilize diverse sensors,
extract inIormation Irom these sensors, and gather inIormation
about complex coherencies by merging the inIormation provided
Irom diIIerent sensor types. By relying not only on one sensory
modality but on diIIerent ones, the robustness and reliability oI
perception is increased.
4.2 Modular hierarchical information
processing
Processing oI vast amounts oI sensory inIormation requires a well
thought out inIormation processing structure. The inIormation
processing structure proposed in this paper is inspired by the
modular hierarchical organization oI the human brain as described
in section 3.2. The sensory receptors have their analogy in the
diIIerent sensor types. In a Iirst stage similar to the inIormation
processing perIormed in the primary cortices oI the diIIerent
sensory modalities the sensory raw data are pre-processed to
extract Ieatures suitable Ior a Iurther processing. In a second
stage, which corresponds to the inIormation processing perIormed
in the secondary cortices, the extracted Ieatures are combined to
result in unimodal perceptions. As outlined Ior the cutaneous
senses, each unimodal perceptual system can be Iurther divided
into subsystems. The processing oI the Iirst and second stage as
well as its sub-stages is perIormed separately and in parallel Ior
each sensory modality. In a third stage analogous to the
processing in the tertiary cortex the inIormation coming Irom
the unimodal perceptive systems is combined and merged to
result in a uniIied multimodal representation oI the environment.
Figure 3 illustrates the modular hierarchical inIormation
processing structure Ior our example. Multimodal perception is
achieved by combination oI data Irom three unimodal perceptive
systems in analogy to its biological archetypes reIerred to as
visual perception, auditory perception, and cutaneous perception.
The cutaneous perceptive system integrates inIormation coming
Irom Iour cutaneous sub-systems corresponding to the Iour
diIIerent sensor types used.
4.3 Symbolic information processing
In section 3.5 it was mentioned that the human mind can be
regarded as a symbolic processing system. This strategy oI
symbolization is applied to our model. For this purpose, sensory
inIormation is transIormed into symbolic inIormation. In
similarity to inIormation processing in the primary cortex oI the
brain, relevant Ieatures have to be extracted Irom the sensory raw
data in a Iirst pre-processing step. These extracted Ieatures are
termed feature symbols. In the next stage, the Ieature symbols oI
each sensory domain are either directly merged to unimodal
symbols or are Iirst combined to sub-unimodal symbols which are
then processed to unimodal symbols. In a Iurther processing
stage, unimodal symbols are merged to multimodal symbols. All
symbols can have properties, which comprise inIormation that
speciIies the symbols in more detail. The concept oI properties
reduces the number oI necessary diIIerent symbols to detect the
deIined possible situations.
The structure oI the individual symbols used in our model has its
paragon in the organization oI biological neurons. Each symbol
can receive input inIormation Irom several other symbols which
corresponds to the Iunction oI dendrites oI neurons. The input
inIormation contains among others the activation grade oI the
symbol it origins Irom. The activation grades oI all incoming
symbols are summed up likewise in the cell body oI a nerve cell.
II this sum exceeds a certain threshold, the symbol passes a signal
to other symbols it is connected to in analogy to the axon oI a
neuron. Like in the brain where many neurons are active at the
same time, diIIerent symbols can also process inIormation in
parallel.
II a system is equipped with many diIIerent sensors and their
values have to be processed, a lot oI calculations have to be
perIormed every instant. To pare down computational power, our
system only reacts to changes oI sensor values. New Ieature
symbols are only generated and old Ieature symbols are only
expired iI sensor values are subject to changes. The same also
holds true Ior all higher-level symbol types as they result Irom a
combination oI the diverse sensor values. At initial system
startup, no Ieature symbol and no other symbol are active
unimportant oI the current values oI the sensors.
The strategy oI combining diverse lower-lever symbols to one
higher-lever symbol corresponds to a bottom-up data processing
principle. Through Ieedbacks, the generation oI symbols can be
inIluenced in a top-down manner. Additionally, symbols can be
inIluenced top-down by knowledge that is stored in the system.
The concept oI knowledge integration and memory symbols will
be described in more detail in section 4.4.
Applying the concept oI symbolic inIormation processing to our
example, we get a modular hierarchical symbol structure as
depicted in Iigure 4. As the explanation shall be kept simple, it is
assumed that there is always only one person present that can
trigger sensors. That way, only one oI the six multimodal symbols
will be generated at a certain time.
In a Iirst processing stage, Ieature symbols pictured as squares
are extracted Irom the sensory raw data. In case oI the sensors oI
the cutaneous senses, the associations between the sensor values
and the Ieature symbols are very simple due their binary sensor
output values and the Iact that the amount oI sensors per sensor
type is limited. However, iI more sensors oI each binary sensor
type or analogous sensors were used, the relations between
sensory raw data and Ieature symbols would become more
complex. Conceivable Ieature symbols in the visual inIormation
processing Ilow are edges, lines, curves, colors, moving Iorms,
etc. Feature symbols Ior auditory processing can be the Irequency
components or the loudness oI a sound signal.

Fig. 3. Architecture for modular hierarchical information
processing of multi-sensory data.
Out oI the Ieature symbols, sub-unimodal symbols in case oI the
cutaneous senses and unimodal symbols in case oI the visual and
auditory sense are Iormed. From the visual Ieature symbols it is
detected whether a person is present in the room. From the
auditory Ieature symbols it is extracted whether the characteristic
noise oI steps or the noise oI an opening or closing door is
perceived. It has to be mentioned that visual image processing and
auditory data processing are huge research Iields. There might
already exist workable solutions to recognize persons directly
Irom images or to detect the noise oI steps or opening and closing
doors directly Irom audio data. II this is the case, it is
recommendable to use these existing solutions to generate
unimodal symbols and skip the step oI explicitly generating
Ieature symbols. However, implicitly, these algorithms also
extract Ieatures out oI the raw data which correspond to our
Ieature symbols.
The sub-unimodal and unimodal symbols oI the cutaneous
perceptive system describe the states and activities oI objects.
They are not directly associated with states and activities oI a
person, because the sensors could also be triggered by something
else like an animal moving in the room or an object positioned in
the room. Some oI the sub-unimodal symbols contain properties
which speciIy the symbol in more detail. The symbol 'person
present has the property 'location l, which indicates whether an
object is present in the room and at what position (leIt or right
halI oI the room) it is located. The symbol 'object passes has the
property 'direction d which comprises the inIormation Irom
what direction the object passes the door. In other words, whether
the object entered or leIt the room. The property 'status s oI the
symbol 'door status comprises the inIormation whether the door
is opened or closed. The sub-unimodal cutaneous symbols are
combined to uniIied unimodal cutaneous symbols. What higher-
lever symbols are generated by which combination oI lower-lever
symbols can be read out Irom Iigure 4. The symbols 'object
enters and 'object leaves are both Iormed Irom the same
symbols. The decision which oI the two symbols is Iormed is only
dependent on the property 'direction d oI the symbol 'object
passes. The same holds true Ior the symbols 'door is opened
and 'door is closed which depend on the property 'status s oI
the symbol 'door status. Some oI the unimodal cutaneous
symbols also depend on the property 'location l oI the symbol
'person present.
Using this bottom-up data processing, a critical point that has to
be considered is the Iact that whenever one oI the symbols 'object
enters, 'object leaves, 'object moves, 'door is opened, or
'door is closed is generated, there is also activated the symbols
'object stands, because it results Irom a combination oI a subset
oI the same sub-unimodal cutaneous symbols. Additionally, each
activation oI the symbol 'object enters or 'object leaves also
triggers the symbol 'object moves. To overcome the undesired
activation oI more than one symbol at a certain moment,
inhibitory Ieedbacks are inducted. This structure is comparable to
the neural Ieedbacks between diIIerent layers oI neurons in the
brain. In Iigure 4, these inhibitory Ieedback connections are
depicted as dotted lines.
From the unimodal symbols oI the visual, auditory, and cutaneous
system, one oI the six multimodal symbols is generated. The
conditions under what circumstances which symbol is generated
can be read out Irom Iigure 4. The generation oI multimodal
symbols is also inIluenced top-down by knowledge. The
underlying concept thereIore is described in section 4.4.
4.4 Knowledge integration
As outlined in section 3.4, perception does not only rely on
sensory inIormation but to a great extent also on knowledge.
Integration oI knowledge and awareness oI what happened until
now into the perceptive process can greatly Iacilitate perception.
In a realistic situation, not only one out oI six diIIerent scenarios
has to be detected but one out oI thousands. In such a case it may
happen that a scenario cannot unambiguously be perceived Irom
the current sensor values, because two or more scenarios might be
triggered Irom the same or very similar sensor values. In such a
case, knowledge in diIIerent Iorms as well as awareness oI what
has happened beIore may lead to an unambiguous decision. In our
model oI symbolic inIormation processing, knowledge can
inIluence the generation or non-generation oI symbols.
One example Ior knowledge integration to our test scenarios
would be to let the system know that a person can only walk
around in the room, stand in the room, or leave the room iI he
entered beIore. This means that the symbols 'person walks,
'person stands, and 'person leaves can only be generated iI the
symbol 'person enters was generated beIore. A second example
would be to memorize iI the door was opened or closed. The
knowledge that a person can never enter or leave a room iI the
door is closed can help to lead to a resolution oI ambiguous
scenarios where a person comes close enough to the door to
trigger the light barriers but does not enter or leave the room.
Furthermore, there cannot be detected a 'person closes door or
'person opens door scenario iI the door is already closed or
open.
A utilization oI inIormation oI that kind requires a sort oI memory
to store important inIormation Irom past events. In our example,
up to the multimodal symbol layer, symbols are activated and
deactivated when sensor values change. In these layers, there
exists no memory that stores past states. Storage oI events
happened in the past is achieved in an additional layer with so
called memory symbols.
A Iundamental question is on what level knowledge should
inIluence perception. It could either already inIluence unimodal
perception or subsystems oI unimodal perception or interact not
until multimodal perception. The answers coming Irom
neuroscience are controversy. The best way to Iind out is
probably to try out the diIIerent possibilities and to evaluate what
works most eIIiciently Ior a certain application. In our model, Ior
a Iirst implementation, knowledge only inIluences multimodal
perception. The states oI memory symbols are set or reset when
certain multimodal symbols are activated. The symbol 'person
present is set aIter the symbol 'person enters was activated. It is
reset aIter the symbol 'person leaves occurred. The symbol
'door open is set by the symbol 'person opens door and reset
by the symbol 'person closes door. The activation oI certain
multimodal symbols triggered bottom up by sensor values can be
inhibited by top-down inIluence oI semantic knowledge in Iorm oI
rules in combination with stored states oI memory symbols. In
Iigure 4, inhibitory connections Irom the knowledge layer to the
multimodal layer are depicted as dotted lines.
4.5 Symbol binding through learning
For our perceptive model, symbolic inIormation processing was
suggested. A Iundamental question is how symbols can be Iormed
Irom sensor values and how symbols can be Iormed by
association oI other symbols. This problem is adequate to the
neuroscientiIic binding problem outlined in section 3.3.
For a Ilexible system it is desirable not to have to predeIine
everything but to learn coherences preIerable Irom examples.
However, similar to the human brain, not everything can be
learned during operation. Certain patterns must already be
predeIined at the system`s initial start-up. The answers Irom
neuroscience to what has to be predeIined and what can be
learned are controversy.
In our model, the symbols oI the diIIerent hierarchy levels as well
as their principal properties are predeIined. Likewise it is Iixed at
initial start-up what Ieature symbols can be extracted Irom the
sensory raw data. In contrast, there exists the possibility to learn
Irom examples what combinations oI Ieature symbols generate a
sub-undimodal or unimodal symbol, which sub-unimodal symbols
merge to unimodal symbols, and what combinations oI unimodal
symbols generate a multimodal symbol. As already mentioned,
depending on the values oI the properties oI a symbol, this
symbol can generate diIIerent higher-level symbols. It can also be
learned which value oI a property generates which higher-level
symbol. It is also learnable what multimodal symbols trigger


Fig. 4. Structure of hierarchical symbolic information processing.
which memory symbol. However, what inhibitory inIluence the
activation oI a knowledge symbol has on multimodal symbols is
deIined by explicit rules given by the system designer and stored
as semantic knowledge.
The process oI learning is divided into diIIerent phases. In section
3.2 it was outlined that higher cortical levels can only evolve iI
lower levels are already sophisticated. This strategy is applied to
our model. In a Iirst step it is learned what combinations oI
Ieature symbols Iorm which sub-unimodal or unimodal symbols.
Next, it is determined what sub-unimodal symbols Iorm which
unimodal symbols. At this level, certain unimodal symbols can
have an inhibitory Ieedback on other unimodal symbols. These
Ieedback connections are learned next. AIter having Iully
Iunctioning unimodal levels, the relation between unimodal and
multimodal symbols is learned. Finally, it is extracted out oI
examples by means oI which multimodal symbols memory
symbols are set and reset. The learning oI symbol connections can
be achieved by diIIerent methods. In a Iirst implementation, quite
simple statistical methods are used. However, also other
techniques are conceivable. To set connections between Ieature
symbols and sub-unimodal or unimodal symbols, neural networks
could be applied. For higher levels, methods like Iuzzy inductive
reasoning (FIR) might also be possible. |3|, |19|
5. IMPLEMENTATION
To test the model oI humanlike perception just introduced, it was
implemented and simulated in AnyLogic. The modeling language
oI AnyLogic has proven successIul in the modeling oI large and
complex systems. |1| The main building block oI the AnyLogic
model is the active object. Active objects can be used to model
very diverse objects oI the real world: processing stations,
resources, people, hardware, physical objects, controllers, etc.
AnyLogic supports the programming language Java. Active object
classes map to Java classes. Three Iurther elements oI AnyLogic
that were acquired to simulate our model are ports, connections,
and messages.
To simulate our model, active objects, ports, connections, and
messages are used in the Iollowing way: The diIIerent symbols
are modeled as active objects. Each active object has an input port
and an output port. Through the output port, messages containing
the actual activation grade oI the symbol and the values oI
eventual properties are passed to other symbols to which this port
is connected to. Through the input port, the symbol can receive
inIormation Irom other symbols which have a connection to it.
Each oI the incoming messages contains the activation grade oI
the symbol it was sent Irom. The activation grades are summed
up. II this sum exceeds a certain threshold, the current symbol is
activated and a message is passed to other symbols it is connected
to via the output port. In the simulation, symbols that receive
sensor values at the same time are computed in parallel.
As the system shall be able to learn connections between symbols
Irom example, additional units have to be added that are only
active during learning time and have the Iunction to memorize the
presented examples during the training phase until the correct
connections are set. AIter the learning phase, they do no longer
take inIluence on the system. However, due to shortage oI space,
the acquired concept Ior learning is not described in Iurther detail
in this article.
6. CONCLUSION AND OUTLOOK
Automatic surveillance systems as well as autonomous robots are
technical systems which would proIit Irom the ability oI
humanlike perception Ior eIIective, eIIicient, and Ilexible
operation. In this article, a model Ior humanlike perception was
introduced based on hierarchical modular Iusion oI multi-sensory
data, symbolic inIormation processing, integration oI knowledge
and memory, and learning. The model was inspired by Iindings
Irom neuroscience. InIormation Irom diverse sensors is
transIormed into symbolic representations and processed in
parallel in a modular, hierarchical Iashion. Higher-level symbolic
inIormation is gained by combination oI lower-level symbols.
Feedbacks Irom higher levels to lower levels are possible. What
symbols inIluence the activation oI other symbols can be learned
Irom examples. Stored knowledge oI diIIerent Iorms inIluences
the generation oI symbols.
The model presented here is a Iirst suggestion Ior a humanlike
perceptive system. The Iollowing Iurther investigations are
planned:
Timing behavior oI incoming inIormation has only been
considered marginally until now. The model also has to be
extended to perceive scenarios that stretch over longer time
periods and where the succession oI events is important.
Until know, knowledge has been provided as explicit predeIined
rules. It would be interesting to extract these rules out oI
examples presented to the system.
Real human perception is also strongly inIluenced by emotions
which have the Iunction to evaluate the scenarios being perceived.
Such an evaluation becomes especially important iI the system
needs to react adequately depending on the scenarios being
perceived. An implementation oI emotions into the model might
be a step towards a perceptive system that is aware oI the
situation it perceives.
Until now, the model proposed has only been tested and veriIied
on the PC with simulated parallel processing. To truly take
advantage oI the parallel distributed structure proposed, it would
be interesting to implement the model into a chip which can
perIorm real parallel processing.
7. REFERENCES
|1| AnyLogic Users Manual. Technologies Company Ltd, 2004.
|2| Bian, X., Abowd, G. D., and Rehg, J. M. Using Sound
Source Localization to Monitor and InIer Activities in the
Home. GVU Technical Reports, 2004.
|3| Cellier, F. E. and GreiIeneder, J. Continuous System
Modeling. Springer-Verlag New York, 1991.
|4| Costello, M. C. and Reichle, E. D. LSDNet: A Neural
Network Ior Multisensory Perception. In Proceedings of the
Sixth International Conference on Cognitive Modeling, 2004,
p.341.
|5| Crowley, J. Situated Observation oI Human Activity. In
Proceedings of the Computer Vision for Interactive and
Intelligent Environment, 2005, 97-108.
|6| Datteri, E., Teti, G., Laschi, C., Tamburrini, G., Dario, P.,
and Guglielmelli, E. Expected Perception: An Anticipation-
Based Perception-Action Scheme in Robots. In Proceedings
of the International Conference on Intelligent Robots and
Systems, 2003, 934-939.
|7| D`Orazio, T., Ianigro, M., Stella, E., Lovergine, F. P., and
Distante, A. Mobile Robot Navigation by Multi-Sensory
Integration. IEEE International Conference on Robotics and
Automation, 1993, Vol. 2, 373-379.
|8| Ellis, D. P. W. Prediction-driven Computational Auditory
Scene Analysis. PhD Thesis at the Massachusetts Institute oI
Technology, 1996.
|9| Ernst, M. O. and BlthoII, H. H. Merging the Senses into a
Robust Percept. TRENDS in Cognitive Sciences, 2004, Vol.
8, 162-169.
|10| Geisler, W. S. and Kersten, D. Illusions, Perception and
Bayes. Nature Neuroscience, 2002, Vol. 5, 508-510.
|11| Goldstein, E. B. Wahrnehmungspsychologie, Chapter 14.
Wadsworth Publishing, 2002.
|12| Goldstein, E. B. Sensation and Perception, Chapter 1.
Spektrum Akademischer Verlag, 2007.
|13| Gtzinger, S. O. Processing and Symbolization of Ambient
Sensor Data. Master Thesis, Vienna University oI
Technology, 2006.
|14| Hanard, S. The Symbol Grounding Problem. Physica D,
1990, Vol. 42, 335-346.
|15| Joyce, D., Richards, L., Cangelosi, A., and Coventry, K. R.
On the Foundations oI Perceptual Symbol Systems:
SpeciIying Embodied Representations via Connectionism. In
Proceedings of the Fifth International Conference on
Cognitive Modeling, 2003, 147-152.
|16| Kammermeier, P., Buss, M., and Schmidt, G. A Systems
Theoretical Model Ior Human Perception in Multimodal
Presence Systems. IEEE/ASME Transactions of
Mechatronics, 2001, Vol. 6, 234-244.
|17| Luria, A.R. The Working Brain: An Introduction to
Neuropsychology, Chapter 2. Basic Books, 1973.
|18| Michahelles, F., AntiIakos, S., Schmidt, A., Schiele, B., and
Beigl, M. Towards Distributed Awareness - An ArtiIact
based Approach. In Proceedings of the Sixth IEEE Workshop
on Mobile Computing Systems and Applications, 2004.
|19| Mirats Tur, J. M. Qualitative Modelling of Complex Systems
by Means of Fuzzy Inductive Reasoning. Variable Selection
and Search Space Reduction. Ph.D. Thesis, Universitat
Politecnica de Catalunya, Barcelona, Spain, 2001.
|20| Muller, H. Neural Binding of Space and Time: Spatial and
Temporal Mechanisms of Feature-Object Binding.
Psychology Press, 2001.
|21| Oliver, N. and Horvitz, E. S-SEER: Selective Perception in a
Multimodal OIIice Activity Recognition System. In
Proceedings of the First International Workshop of Machine
Learning for Multimodal Interaction, 2004, 122-135.
|22| Rhne-Alpes. Project-Team PRIMA Perception,
Recognition and Integration for Interactive Environments.
Activity Report, 2005.
|23| Pratl, G., Processing and Symbolization of Ambient Sensor
Data. Ph.D Thesis, Vienna University oI Technology, 2006.
|24| Perry, M., Dowdall, A., Lines, L., and Hone, K. Multimodal
and Ubiquitous Computing Systems: Supporting
Independent-Living Older Users. IEEE Transactions on
Information Technology in Biomedicine, Vol. 8, No. 3, 2004.
258-270.
|25| Sillanp, J., Klapuri, A., Seppne, J. and Virtanen, T.
Recognition oI Acoustic Noise Mixtures by Combined
Bottom-up and Top-down Processing. In Proceedings of the
European Signal Processing Conference EUSIPCO, 2000,
Vol. 1, 335-338.
|26| Solms, M. and Turnbull, O. The Brain and the Inner World:
An Introduction to the Neuroscience of Subjective
Experience, Chapter 5. Other Press, New York, 2002.
|27| Zimmer, H. D., Mecklinger, A., and Lindenberger, U.
Handbook of Binding and Memory. Perspectives from
Cognitive Neuroscience. OxIord University Press, 2006.