Towards A Human Behavior Model Based On Instincts

Victor C. Hung
Avelino J. Gonzalez
University of Central Florida
School of Electrical Engineering and Computer Science
Orlando, FL 32816
321-356-3398, 407-823-1374
[email protected], [email protected]

Keywords:
artificial intelligence, human behavior, instincts, modeling

ABSTRACT: This paper presents the feasibility of implementing an instinct-based behavioral model in an artificial
agent. This research serves as a collection of information related to devising an artificially intelligent entity whose
decisions are based on human instinct. The treatment of instincts will be defined by the bounds of psychology and
ethnology. The research focuses on two concepts: (1) the psychological and ethological implications of instinctive
behavior, and (2) the use of instinct-like functional primitives in artificial intelligence systems. The primary goal of this
paper is to survey the existing research that may reveal insight in producing a realistic behavior model based on
instincts. Such a model would serve as the basis for a human behavior model to be used in simulation and training
scenarios.

1. Introduction certain job will convey a strong sense of realism to the

simulation. This specialized nature of knowledge-based
Modeling human behavior is a time-consuming effort, HBRs, however, also has its disadvantages. For one, the
often requiring meticulous attention to small details (St. process of detailing behavioral intricacies consumes a
Amant et al, 2006). Emulating human behavior for certain significant amount of time and energy (Fu et al, 2003)
tasks and environmental interactions calls upon countless (Garcia and Griffith, 2005) Moreover, an agent that is
hours of development and testing, a process similar to narrowly centered on a certain environment will behave
producing any other types of software systems. These unpredictably if it is exposed to a situation it is not
knowledge-based human behavior representations (HBRs) programmed to handle. Hence, these knowledge-based
depend on specifically abstracted behaviors to enable an HBRs do very little in addressing the issue of placing
agent's successful interaction with its environment. It is agents in unknown situations (Soffker, 2001).
often the case where a representation may only be valid
for a particular environment. A fundamental alternative to these knowledge-based
systems would be an HBR that is sufficiently general and
The demand for high quality HBRs is very prevalent in broad to cover all situations. Such a representation would
the simulation and training community, especially those eliminate the extensive development process of injecting
groups involved in military applications (Giordano et al, specific knowledge into an agent. One approach to
2004). A simulation agent design life cycle typically creating such a generalized HBR is to produce an agent
includes a substantial amount of time examining the whose behavior is based upon some general behavior
specific behaviors pertaining to a particular skill, such as primitives that are universally relevant. It could be
urban combat (Jones et al, 2002). Software tools have proposed that instinctive reactions may serve as the basis
even been developed to facilitate this behavior authoring for behavioral primitives. Following are the issues
process (Fu et al, 2003). As a result, these knowledge- involved in building an HBR based solely on instincts, as
based HBRs end up as artificial agents that are defined by the bounds of psychology and ethology.
exceptionally well prepared to perform a narrow, but very
specialized, set of tasks. A specific application for instinct-based HBRs is the
action-response systems of artificial agents in simulations.
This practice of skill extraction is a common theme in Agents may exist as autonomous, interactive entities in
both military training simulations and commercial video training scenarios, such as those found in military
games alike (Diller et al, 2004). The advantage of exercises or in video games. In many cases, the agents are
utilizing these knowledge-based HBRs is their implemented with scripted behaviors, resulting in actions
appearance of intelligence for a particular task. An agent dictated by a simple input-output lookup table (Spronck et
that has been programmed with fine-grained detail for a al, 2004).
 natural, innate behaviors in artificial agents (Brooks,
A problem arises when an input stimulus does not exist in 1991) (Rosenblatt and Payton, 1989) have been
the response repertoire. The artificial agent may find itself suggested, but these efforts have not been widely
in a situation in which it cannot provide a proper expanded. A major cause of this reluctance to effectively
response, usually because of unknown locations or newly emulate nature is the fact that proclaiming something to
introduced stimuli. The responses to these foreign inputs be a naturally occurring response is often too subjective,
can make or break a simulation. A default response and thus, quite difficult to quantitatively defend. This
system might come into play during such incidents, which problem does not occur with the nurturing side of
may be another scripted behavior. Often, these default behavior because much of the research is deeply based in
behaviors do not depict a realistic response and taint the logically sound, mathematical methods. The bottom line
quality of a simulation exercise. The proposed behavioral remains that psychology asserts that an autonomous
architecture addresses the agent response issue by system's behavior, whether artificial or biological, is
providing an infrastructure that is based on a system of characterized by the combined efforts of both innate,
instinctive primitive interactions. When engaging a natural factors and nurtured, experiential components.
foreign environment, a series of basic behavioral The latter of these two portions is quite well-versed in the
primitives will comprise the agent's actions. Thus, as the artificial intelligence community, while the former has
entity encounters a situation of unknown circumstances, it suffered a noticeable developmental lag in comparison.
will rely on its instincts as its default response system. With this in mind, the pertinent task in this paper is to
capture the essence of the aforementioned natural side of
The following sections provide an overview of the behavior and complete the nature-nurture loop by
relevant domains researched to build this instinctive implementing it into an artificially intelligent agent.
behavior model. Specifically, they touch upon the
subjects of Human Instincts and Primitives-Based AI Instincts differ from reflexes in that the latter is simply is
Systems. an intrinsic, physiological impulse, usually triggered at a
muscular, rather than at a cognitive, level (Kantor, 1920).
2. Human Instincts It is also noted that instincts are widely regarded in
psychology as general guidelines or motivations, rather
A key component of the suggested human behavior model than specific skills or actions. Furthermore, Kantor (1920)
is the definition of human instincts. Instincts offer humans proclaims that instincts serve as the basis for more
a baseline behavior to which they can interface with their complex human behaviors, especially those involving
physical environments (Balkenius, 1994). An infant can intelligence.
only rely on its own innate instincts as its total knowledge
base. As a person matures, she or he eventually Both Angell (1906) and Mead (1909) offer their
deemphasizes this dependence on instincts, instead opting discussion of the entire spectrum of human instincts.
for its experience-based intuition and learned knowledge Angell's account gives a series of instinct groupings in the
to control behavior (Balkenius, 1994). Hence, from order in which they manifest in a person's development.
infancy, a set of instincts must exist before the eventual The first such instinct grouping is anger and fear, which
knowledge base can be established. The relationship represents the fundamental fight or flight response. Next,
between instincts and experience has evolved from the he lists shyness and sociability, instincts that dictates an
nature versus nurture debate (Galton, 1874) to a mutual individual's desire to keep to one's self, or to seek the
involvement of both naturally inherited traits and support of others. The third pair of Angell's instincts is
nurtured, learned experiences. curiosity and secretiveness. Curiosity represents a
person's instinct of arousing interest in some matter, while
The nurturing, experiential component of behavior is a secretiveness serves to consciously suppress such interest.
well-versed aspect of artificial intelligence. Observational Acquisitiveness and rivalry make up the next grouping,
learning, pattern recognition and neural network systems yielding two instincts that affect a person's comfort level
have long been implemented as a means to execute within local bounds. Acquisitiveness represents that
machine-learning methods. In fact, machine learning has person's need to take ownership of something, while
been a staple of artificial intelligence research for the last rivalry consists of the individual's yearning to be better
two decades. Hence, computational techniques needed to than a peer in some aspect of life. A fifth set of instincts
process experiences have been well studied. exists in jealousy and envy, where both dictate a person's
need to covet another person's life or possessions,
2.1 Instincts in Psychology respectively. Angell (2006) lists sexual instincts and
parental love as another instinctive pairing. The final
Artificial re-creation of the nature portion of behavior has group of instincts is three-fold: play, imitation and
remained quiet over the years. Some attempts at creating constructiveness. Play is an individual's yearning to
expend energy in a manner that is not necessarily in that brief span of time in a person's infancy when she or
productive toward achieving a certain goal. Imitation he must rely on raw instinct to survive (Balkenius, 1994).
refers to a person's instinct to subconsciously follow the With this understanding of instincts, it can be asserted that
behaviors found in his or her social environment. Finally, a default set of behaviors must be established before a
constructiveness describes one's creativity instinct, the high-level of learned behaviors can be ascertained.
need to shape the physical world.
Some of the described instincts, such as fear and anger,
Mead (1909) identifies eleven instincts, many of which can be classified as emotions. Will (Moffat and Frijda,
overlap with Angell's (1906) instincts. Flight (or fear, 1994) and EMA (Gratch and Marsella, 2004) are two
under Angell's interpretation) appears to be an undisputed artificial intelligence architectures that incorporate
instinct. Mead asserts that pugnacity, subjection, self- emotion as a system input. In fact, Marsella and Gratch
display, the parental instinct, the instinct of reproduction, (2003) assert that emotions serve as the central controller
and the gregarious instinct represent the socially- for decision-making. Their work dictates that an
dependent instincts, while repulsion, curiosity, the instinct emotionally driven agent will first appraise its situation,
of acquisition, and the instinct of construction could determining its relationship with the current environment,
feasibly be subsumed by the rest of the instincts (1909). in terms of five basic values: (1) relevance of actions to
Mead makes the argument that social instincts are better goals, (2) desirability of consequences, (3) likelihood of
representatives of instinctive behaviors because they are consequences, (4) identification of causations, and (5)
triggered and witnessed by the outside world. Essentially, coping capability. Coping refers to an agent’s ability to
Mead's work serves to limit the study of instincts to only reverse an adverse consequence or maintain a positive
those behaviors that can be evaluated and qualified in a result after it commits to an action. It is this coping step in
manner external to a person's self (1909). This idea the appraisal process that an agent must consider its
describes the basis for social psychology practices, and it emotional state. Moreover, this coping strategy exists as
is often reflected in the artificial intelligence field, where the key step in determining agent behavior, hence
external behavior is debated to be the only way to assess a emphasizing the importance of the emotions. Under the
machine's intelligence. direction of Marsella and Gratch’s work, an HBR based
on instincts would benefit from a similar control
Another discussion of instincts comes from the work of architecture centered on emotion.
Maslow and his treatment of instincts as a set of internal
drives (1954). In his famous pyramid, he created five This discussion of instincts in the psychology context
types of needs, ordered from basic (bottom) to complex relies on the idea that these behaviors remain independent
(top): physiological, safety, belonging, esteem, and self- of cultural influence. The learned behaviors resulting
actualization (1954). A human experiences the entire from different societal norms are an example nurtured
spectrum of these motivations at any given time of her or conditioning. Ekman et al (1987) assert that certain facial
his life. The emphasis behind the triangle structure is that expressions, such as smiles and frowns, are universally
a person must first achieve a comfortable stasis point identified as signs of happiness or sadness, respectively.
before attempting to satisfy the next echelon of needs. The instincts in a HBR must similarly capture this
Maslow describes the behaviors required to achieve these universality to remain a general as possible. It is the job of
stasis points as instinctoids. With Maslow's hierarchy of the nurturing mechanisms to culturally influence the
needs in place, the resulting model of human behavior agent.
involves a set of five motivation-based goals (the five
levels of the triangle), with a set of finer sub-goals 2.2 Instincts in Ethology
(instinctoids) necessary to reach these main goals.
Maslow's triangle infrastructure can be criticized for its Ethology describes the study of animal behavior from a
lack of detail in delineating the specific natural behaviors biological, as opposed to psychological, point of view.
involved in human instincts. Hence, ethology is a separate field from psychology, that
offers its own stance on instinctive behavior. In general,
Kantor suggests a strong correlation between instinctive ethologists examine instincts under an empirical,
behavior and animal intelligence (Kantor, 1920). As physiological light, while psychologists treat instincts
mentioned in the nature/nurture discussion, there remains with a more abstract, comparative approach.
a fuzzy boundary where raw, instinctive reactions meets
learned, experiential responses. Kantor (1920) asserts that As mentioned before, the development of instincts in
instincts make up the building blocks of a person's learned animals is viewed as an evolutionary process - Lorenz
behaviors, yet he concedes that a vast majority of the way established this standpoint as a staple of ethological
people behave are these derived, complex behaviors, studies (Brigandt, 2005). Over long periods of time,
rather than acting purely from instinctive drive. It is only multitudes of organisms are spawned with varying flavors
of innate tendencies. As per the mechanics of evolution, subsumption-based agents. The following section
only those organisms that are good enough to survive will describes these efforts.
be able to pass on their genetic make-up to the next
population. Within this genetic coding lies the set of Brooks' work offers a paradigm shift in behavioral
intrinsic instincts. Tinbergen asserts that these instincts architectures. Named subsumption, this infrastructure
exist as Fixed Action Patterns (FAPs) (1948). FAPs are does away with any centralized processing unit, opting for
those behaviors that are unlearned but are essential for an a set of independent simple behavioral mechanisms
animal’s survival. They are triggered by innate releasing operating simultaneously (1991). The paper begins with
mechanisms. Examples of FAPs include bees’ mating an overview of the state of the machine intelligence
dances, gulls’ egg-laying patterns, and minnows’ feeding research to date. He claims that a majority of this effort
behaviors. has consisted of simply programming human abstractions
of specific subjects into machines, a practice that
With the advent of evolutionary algorithms in computer jeopardizes the original intent of artificial intelligence. To
science, it is easily envisioned how such solutions may be discontinue this methodology, Brooks proposes the
applied to this problem of devising instinctive behaviors creation of entities that abide by the following set of
for artificial entities. Stanley, Bryant, and Miikkulainen engineering specifications: they must perform in real-
(2005) and Portegys (2006) have approached this idea of time, they must adapt to the environment in a piece-wise
performing genetic algorithms on initial, innate behavior fashion, they must have the capability to juggle
generation. simultaneous goals, and they must serve some sort of
usefulness. To achieve these design requirements, Brooks
3. Primitives-Based AI Systems maintains that these agents must be comprised of a set of
activity-based behavior mechanisms, rather a series of
Instincts may be viewed as fundamental building blocks function-based components. Moreover, he suggests the
of behavior. Hence, they may serve as functional use of a layered behavioral system, where the lower level
primitives for more complex responses. Analogous behaviors feed into the higher levels, and thus, add to the
systems based on functional primitives exist in artificial complexity of the higher-tiered activities. A major feature
intelligence; this section presents the state-of-the-art AI of the paper is the idea of a lack of internal representation
techniques that rely on primitive components. Each of throughout this layered infrastructure. Brooks asserts that
these methodologies can be associated with Reactive the simple low-level behavior primitives do not require
Agents, Deliberative Agents or Evolutionary Agents. internal memory constructs.
Reactive Agents use a set of primitive reactions to
directly control their behavioral outputs. In Deliberative The above description of an automated behavioral
Agents, a set of fundamental rules or motivations is architecture is an invention Brooks calls subsumption. He
considered in the planning of the agent's behavior. For implements this methodology in four robots, each using a
Evolutionary Agents, a series of elementary behaviors are specific type of processing hardware. Each robot uses a
evolved over time to generate an optimal behavior set. three-layer subsumption architecture, with the
The common thread amongst the agents under each of characteristic lack of a central arbiter. Brooks was able to
these different architectures is that a collection of simple instill into each entity the simple abilities of turning,
elements is working together to produce an agent capable stopping, moving forward, and detecting obstacles, as
of complex behavior. well as the more complex capabilities of wandering and
running away. Brooks' proposal of subsumption in this
3.1 Reactive Agents paper serves as a wake-up call to the artificial intelligence
community by denouncing the use of abstraction as a
Adhering to the idea that human instincts may be means to inject knowledge into machines. Nevertheless,
regarded as a series of simple, input-output responses, the behaviors he has instantiated in his robots seem a bit
instinctive behaviors can be compared to those found in crude, and thus, effectively rendering them as trivial
reactive agents. Often referred to as behavior-based, these examples. Brooks, however, emphasizes the infancy of
agents define their behavior using a set of simple reflexes his technology, and he offers reassurance that his work
that are triggered by the outside world. The subsumption will serve as a strong baseline for more complex
architecture embodies this school of thought. behaviors.
Subsumption deals with the idea that smaller, reactive
behaviors can exist with each other within an agent to Rosenblatt and Payton (1989) modify Brooks' original
produce an aggregated, complex behavior. Brooks (1986) robot architecture (1986) by augmenting the behavioral
first suggested this type of architecture, and many layering system. For traditional subsumption, the
researchers have followed his lead in developing operational emphasis is shifted to the external stimulus,
rather than an agent's internal central planning. Each low
level behavior independently interacts with the magnitudes (in both spatial and computational complexity
environment, supplying an appropriate response, given contexts). The main shortcoming of reinforcement
the state of the external world. This is the basis for a learning remains the idea that such a process was not
vertical decomposition of behavior. Moreover, higher meant for real-time, physical processing, such as those
behavior levels exist beyond the lowest behavioral level found on a robot. Instead, the originators envisioned it to
to add complexity to the agent's functionality. As be used as a pre-planning mechanism in simulation-based
mentioned above, the most pertinent portion of Brooks' entities.
subsumption lies in the idea that an internal representation
of the world is not a requirement. Rosenblatt and Payton Behavior-based decision-making addresses this issue by
point out certain deficiencies regarding Brooks' work. involving simple mechanical constructs that directly
They criticize subsumption for its inability to associate interact with its environment, thus allowing for a real-
priorities with certain low level tasks. Additionally, the time response time. Unfortunately, high-level intelligence
lack of an internal memory system severely limits its is not associated with such systems, as these impulsive
capability to efficiently provide repeatable behaviors in input-output actions ultimately serve as simple robotic
the upper behavioral layers. A final shortcoming of reflexes. Konidaris and Hayes mention the conception of
subsumption is its lack of central processing of high-level layered learning, an early technique that attempted to
functionalities, which allows for the existence of bring higher levels of complexity to robot behavior by
unwanted behavioral conflicts. using a layering infrastructure of machine learning-based
constructs. Unfortunately, layered learning lacked real-
Rosenblatt and Payton suggest a series of remedies to deal time performance. To remedy these downsides of
with these downfalls of subsumption. They introduce the machine learning, the authors define the behavior-based
idea of fine-grained behaviors, which break down the reinforcement learning architecture. With this
low-level reflexes into more simple control structures. infrastructure, an agent can make goal-oriented, as well as
Each of these units is attributed with an analog input real-time assessments of its external world through the
signal and a priority weight, and a set of these units is construction of an internal topological map. At the lowest
combined to achieve an activation level to fire an action. level of behavior, simple environment sensors reveal
Another modification to subsumption is the addition of a information about the agent's immediate position. A
centralized command arbiter. A set of fine-grained second layer compiles this data into a map representation,
behavior-enhanced actions feed into this mechanism, and consisting of a set of nodes and edges. This internal
the activation levels of these behaviors are prioritized. abstraction allows the robot to work within a confined
The third augmentation of subsumption is an expanded state space, rather than depend on the information
accessibility to the internal states of each level within overload that is delivered by its external sensors.
each state. This allows higher behavior levels access to Moreover, reinforcement learning is present at this layer
the status information of lower levels, and vice versa. To to associate rewards with certain nodes of the map, thus
demonstrate these enhancements, Rosenblatt and Payton allowing for simple path planning capability. It is noted
apply their fine-grained behavioral subsumption to a robot that each of these behavioral layers interacts with each
platform. In the lowest layer, the speed of the robot is other in an asynchronous and distributed fashion.
controlled using a series of obstacle and position sensors
interacting within a speed adjustment arbiter. In the next To test the effectiveness of their architecture, Konidaris
highest behavior layer, travel routing is conducted using a and Hayes subjected a robot to three different mazes,
fine-grained gradient field. Throughout this level, a where the objective is to retrieve a puck and return to
constant channel of communication between the lower home base. The robot tested three decision-making
behavior levels is established to allow for a real-time methods: random, layered-learning, and behavior-based
execution of this path planning. In this research, reinforcement learning. The first maze basically
Rosenblatt and Payton augment Brooks' subsumption to eliminated the random decision-making as a valid
provide a more robust and predictable control system. behavioral policy. The second and third mazes compared
the two types of learning mechanisms. When the puck
Konidaris and Hayes (2005) propose a behavior-based reward was removed, the asynchronous behavior-based
reinforcement learning methodology that combines the reinforcement proved to be more flexible in path planning
efforts of reinforcement learning with behavior-based than the synchronous version. Moreover, the former
decision-making. Their efforts resemble much of the work methodology tended to converge to a more efficient path
done by other subsumption researchers. The authors note plan than the layered-learning algorithm. The robotic
that reinforcement techniques provide a natural learning architecture that is presented by Konidaris and Hayes
feedback by using an internal drive value system. A offers an innovative and advanced internal representation
problem with reinforcement learning is the fact that the system, similar to that of the human mind. Furthermore,
agent's learning scope tends to be inflated to unwieldy the experimental results appear to reflect its effectiveness.
The authors could improve their findings by not limiting adapted for such endeavors as emotion-based control and
their cognitive infrastructure to a physical topological group-level behavior modeling (Giordano et al, 2004).
map, generalizing internal representation for any type of Nevertheless, Giordano et al (2004) point out certain
data. applications that may never be achieved within the realms
of Soar, ACT-R, and COGNET. Specifically, they regard
Mataric (1995) adapted subsumption to a social completely integrating emotion, cognition and behavior as
environment. Her work saw the formation of group an infeasible task. Their main argument behind this
behaviors, derived from a basic behavior set. These assertion is the fact that human behavior is too complex to
behaviors are characterized as either spatial awareness plausibly capture in a single representation. This
responses (such as avoiding objects or following) or complexity is heavily based upon co-existence of
group dynamics movements (such as congregating or rationality and irrationality of human nature. Giordano et
dispersing). The resulting population-level behaviors al’s statement, however, speaks upon the near
allowed for cooperation amongst the individuals and an impossibility of creating a complete representation. It is
enhanced information flow within the group. Henceforth, not the immediate focus of the instinct-based HBR to
Mataric’s work exhibited the social applications of reliably predict all human behavior. With that being said,
subsumption. it is definitely a possibility that a production system, such
as Soar, could be utilized to produce an HBR based on
In each of the discussed subsumption systems, the instincts.
researchers implemented a set of independent primitive
behaviors whose conglomeration leads to the agent Samsonovich and De Jong (2004) propose a three-part
behavior as a whole. This mindset can be paralleled to the model of the human mind's architecture. The first part
idea that the co-existence of individual and simple human consists of the schema processor. The authors utilize
instincts within a person results in her or his total Kantian definition of schema to describe a basic unit of
behavior. This subsection's common theme of semantic knowledge. A real-world instantiation of a
subsumption, however, is not a main theme of this schema is regarded as a state. It is, however, not
dissertation. The subsumption architecture is necessary that the state occur in a tangible, physical form;
characteristically a reactive method, while the aim of this it may exist as an abstract mental construct. The second
research is to provide an agent whose behavior is closer to component of the model is the multichart infrastructure.
a hybrid of the reactive and deliberative representations. A group of states that occur simultaneously is referred to
In fact, the mention of subsumption in this dissertation as a chart. Internally, a series of charts are maintained,
serves to emphasize the need for a hybrid agent. It is seen where each chart is interconnected with other charts.
that the weakness of the purest subsumption agent is its Cognition occurs during the journey between charts,
inability to provide adequate internal planning. Because hence the name, multichart. The final element to the
of the internalized nature of instincts, it is determined that architecture is the self concept. The self is implemented as
they cannot simply be treated as immediate reactive a series of overall ruling axioms. These rules maintain a
attributes. Nevertheless, Brooks' basic idea with his global policy on the construction of schemas and charts.
subsumption architecture was to define behavior as a set While Samsonovich and De Jong present a very novel
of primitive behaviors, which is a concept that is innovation in the self concept, their paper fails to produce
analogous to attributing instincts as primitive units of a robust set of experiments and results to demonstrate
behavior at a cognitive level. their new cognition architecture. Even the simplest of
examples would have been sufficient as an initial
3.2 Deliberative Agents presentation of proof of concept. Regardless,
Samsonovich and De Jong present a system in which a set
Instincts may be regarded as an interaction of basic of pre-defined operating rules motivates the decisions of
motivations. The following section describes different the agent, very much like the pre-programmed nature of
behavior representation models whose decision-making is human instincts.
dictated through a set of fundamental rules or
motivations. These deliberative agents utilize these As an extension to their research, Samsonovich and De
primitives when planning their behavioral responses. Jong (2005) present an autonomous learning system,
based on biologically-inspired mechanisms, called a self-
Standard industry HBR architectures, such as Soar (Laird aware neuromorphic hybrid. This approach produces an
et al, 1987), ACT-R (Anderson, 1993) and COGNET agent that evolves from an initial embryonic state. The
(Zachary et al, 1991) operate under an organization of authors describe the five key features for their system: (1)
basic rules. These production systems break down agent a capability for autonomous discovery, (2) a fundamental
behavior to a series of primitive motivations. There memory infrastructure, (3) a sense of social
already exist instances in which these systems have been existentialism, (4) an interface protocol with the external
world, and (5) an ability to process emotion. Under the same phenomenon is analogous in nature, where animals
direction of these five capabilities, the neuromorphic use the same set of muscles for all behaviors, regardless
hybrid is implemented as a system of memory modules of the behavior's cognition complexity.
and a reward-punishment processor that relies on the
interaction between the aforementioned self rule-base and Chrysanthakopoulos and Marks (1998) implement the
schema mechanisms. To demonstrate the capabilities of Org model, which endows its agents with a set of five a
this paradigm, Samsonovich and De Jong employ the priori rules. These rules, known as instincts, directly
hybrid agent as an intrusion detector. In these experiment, affect the agent's behavior as follows: (1) seek evasive
the neuromorphic agent must defend a computer objects, (2) flee from aggressive objects, (3) slowly step
network's 24 nodes by preventing a hacker agent from away from non-moving objects, (4) move toward
visiting any of those nodes. The neuromorphic hybrid is unknown objects, and (5) move in a random fashion if no
able to autonomously derive different courses of action object is present. The Orgs are equipped with sensors for
for this particular environment for various scenarios. The distance, speed, color and size. Furthermore, the Orgs
authors assert that the hybrid's internal self rules allows may augment their instinctual rule-set with a behavior
for such autonomy, which produced the same strategies pattern-matching learning method. To examine their
that game theory, evolutionary computing, and orthodox instinctive organisms, Chrysanthakopoulos and Marks
machine learning would yield, without the aid of human placed a pair of Orgs in the same environment. Because
intervention. of the simplistic nature of this experiment, the authors
were not able to capture conclusive results about their
In each paper, Samsonovich and De Jong (2004) (2005) Orgs. They did, however, observe that their agents did not
present a system in which a set of pre-defined operating behave in a predictable, scripted manner. This work can
rules, the self component, motivates the decisions of the be further improved by expanding its complexity,
agent, a manner very much like the internally pre-defined especially in its instinct definitions, and by providing
nature of human instincts. The authors' dependence on the more thorough experiments.
self concept, however, does present a weakness, as their
explanation of rules within this mechanism are not well- Andrianmasinoro (2004) introduces the idea of
defined. It may be the case that their self is an application- incorporating a set of instinct-like motivations into a
specific definition, hence lacking the desired hybrid agent. He notes that many reactive agents have
characteristic of universality found success in having pre-configured instincts to drive
their behaviors. Implementing such drives into a
Burt laid out the ideal characteristics of a hybrid agent, cognitively-active agent, such as a hybrid one, is
stating that its reactive capabilities must not overpower its fundamentally different since the instincts do not directly
cognitive abilities (Burt, 1998). He asserts that both dictate behavioral reaction, but rather influence the
reaction and cognition must be well balanced as to behavioral planning process. The paper presents
prevent trivializing either aspect of the agent. Kadlecek MASLOW, a hybrid agent whose internal drive system
and Nahodil (2001) adhere to this mentality with their relies on its namesake's legacy, the Maslow hierarchy of
hybrid agent design, maintaining that intelligent life is needs (Maslow, 1954). Alongside this motivation
both a behavior-based and a knowledge-based existence. network, MASLOW contains a network of actions and a
The hybrid agent that is built contains both reactive and Need Importance Manager (NIM), which serves as the
cognitive mechanisms, all of which are controlled by arbiter between the system's actions and needs. Each level
three major, independent components: the vegetative of the need hierarchy is split into High-Needs (HNs) and
system block, the action selection block and the Low-Needs (LNs). The needs of a particular level of the
conception block. The vegetative block is the portion of pyramid are considered satisfied when both the LNs and
the agent that interprets the low-level perceptions and HNs are fulfilled. The action network is comprised of two
categorizes them into drives. This block tends to the categories of responses: primitives and composed actions.
agent's need to maintain homeostasis. The action selection Primitives are the most basic of actions that MASLOW
block takes an accumulation of the agent's tendencies at a can execute, while composed actions combine a set of
particular time and computes the optimal action to primitives and/or other composed actions. The NIM acts
execute in the current environment. This component is resolve which needs should be satisfied first and which
reflects the agent's instinctive and reflexive actions. The actions are to be used to provide this satisfaction.
conception block remains the cognitive force of the agent, Andrianmasinoro tests MASLOW using the ADK
where learning and planning occur. This final block acts environment, a system originally meant for reactive
as more intelligent action selector, allowing for the agents. The experiments are a search-and-rescue-type
presence of complex behaviors. Kadlecek and Nahodil format designed to compare the behaviors of a pre-
proclaim that the glue that binds these three blocks is they programmed agent, a MASLOW-based reactive agent,
all share the same primitive actions. They assert that this and a MASLOW cognitive agent. The reactive agent was
able to emulate the performance of the pre-programmed to keep task scopes to limit computational complexity.
agent, yet without the extra layer of human intervention Portegys insists that these tasks are a result of an
for behavioral preparation. The author concedes that the internalized motivation rather than an ultimate end state.
cognitive agent exhibited different results from its Within each motivation, Mona breaks down its behavior
reactive counterpart, but he insists that the motivational into smaller sub-goals, each of which relies on a defined
component of both robots, which is the centerpiece of his input-output negotiation. Furthermore, it implements
research, are identical. The major contribution from these major motivation control structures as a set of
Andrianmasinoro's work is his use of Maslow's hierarchy neural networks. These networks are composed of
as the mechanism to prioritize needs, which, in turn, receptor, mediator, and motor neurons, each representing
dictate behavior. Henceforth, this work demonstrates that an input, intermediate, and output node, respectively.
not only can motivations be treated in a piecemeal Receptors serve as stimulus sensors, while motors serve
fashion, but they can also be organized in a manner as response actuators. Mediators act as bridges that
similar to satisfying instinctual drives. It is very likely that designate the cause and effect relationships between
Andrianmasinoro's work may be extended to increase its neurons. A sub-goal is completed once its motor is fired
versatility as a generalize motivation organization as a result of satisfying the preceding receptor and
architecture for a variety of agent applications. mediator network structure. Portegys asserts that the true
strength of Mona is its ability of need-reduction. The idea
Konidaris and Barto (2006) address motivation-based behind this concept is that an immediate need serves as
behavior by tackling reinforcement learning when a the basis to trigger the mediator nodes within Mona's
multitude of reward priorities are present. Their paper neural nets. This need is continuously applied to the
presents a robotic platform in which many immediate system until a satisfying response is achieved, thus,
goals exist at one time. The authors propose a framework quashing the need-based motivation. Portegys uses a
that incorporates a set of drives, where each drive has its simple ant-feeding behavior to demonstrate Mona's
own satiation level and priority level. As time progresses, operation. The resulting response trace shows that the ants
the drive's reward value equals the product of the change would be able to fetch their food source using the Mona
in satiation over time with the drive priority level. The motivation model.
goal of the system remains to optimize the summation of
all the drive rewards. To test this motivation-based In a second paper, Portegys (2006) presents his
architecture, a simulated circular robot was placed into a experimental results in evolving an instinctive behavioral
rectangular pit. Two types of resource pellets exist in the set for the Monkey and Bananas problem. The work
pit, and the robot must efficiently move about the arena to begins with a general discussion of the realm of
satiate the drives caused by both resource types. Three instinctive behavior, making the claim that instinct is
types of priority schemes were implemented: simple drive often used in nature as a more efficient method of
reduction, equal prioritization, and dynamic priority deploying learned behaviors, rather than acquiring the
scheduling. The simple, non-priority-based robot knowledge through experiential means. Portegys proposes
consumed whatever resource was readily available. The that a genetic algorithm be implemented to create instincts
equalized priority scheme resulted in an improved that simulate those of simple animals. He explains that the
distribution of resource acquisition, although one drive Monkey and Bananas problem consists of a monkey that
did appear to be favored over the other. The final, full- must stack a series of boxes to reach a bunch of bananas.
blown motivational robot with dynamic priorities was the This particular problem serves as his test bed for his
most capable of achieving a more balanced distribution of evolutionary algorithm for instinctive behavior
drive reduction. Konidaris and Barto’s research aligns to generation, with his Mona architecture as the
the theme of using prioritized need satisfaction as a experimental platform. The evolutionary algorithm
behavioral motivation. Their general infrastructure can be consisted of a population of 20 monkeys for each
expanded to involve a larger array of resources to satisfy, generation, with a fitness function that measured how
such as those of instinctive drives. quickly each monkey could reach the bananas. The base
case of no initial instincts yielded no monkey that could
Portegys (1999) describes his Mona architecture, which complete the task. The first substantial result compared
models human cognition with special emphasis on the instinct-only performance versus instinct plus experiential
concept of behavioral motivation, where motives, rather learning performance. This experiment showed that
than goals, drive action. Motives are the general tendency instincts played a major role in the completion of the task,
to satisfy immediate homeostatic needs (those bodily where about 90% of the successful behavior was
functions required to maintain physiological balance). attributed to solely innate behaviors. A subsequent
Mona incorporates a standard internal-external interface experiment concerned the application of a monkey's
through a series of sensory data and response exchanges. behavior set on new environments. Portegys concluded
A short-term memory is internally maintained as a means that a monkey should acquire a fine-tuned behavior set on
a known environment before exposing the organism to a time-critical version of the NeuroEvolution of
foreign stage. A final experiment tested the effect of Augmenting Topologies (NEAT) architecture. NEAT is a
evolution-driven behavior sets versus directly specialized genetic algorithm that utilizes historical
programmed custom behaviors. The paper claims that the markings to improve the evolutionary process. These
behavior set based on evolutionary computation yielded historical markings encode an evolutionary time stamp
better fitness scores over the handcrafted behavior sets into an individual's genotype. Individuals that share
when each type of monkey was exposed to a randomly similar historical markings can be considered to be of the
generated environment. Portegys' work presented an same species. This speciation allows individuals to be
interesting aspect of cognition modeling, namely, the compared amongst similarly structured candidates, rather
notion of instinctive behaviors. The paper, however, than the global population, thus allowing for certain
would been more valuable if multiple scenarios had been species to fully realize their strengths instead of dying out
used, rather than just the Monkey and Bananas problem. too early. To implement NEAT in real-time, an abridged
The notion of implementing instincts in a universal sense version of population replacement is instantiated, thus
was not captured when dealing with only this particular adjusting to the time constraints involved in video game
problem domain. processing. The platform in which rtNEAT is
implemented is the NeuroEvolving Robotic Operatives
In each of Portegys' papers (1999) (2006), Mona is (NERO) game. In NERO, the user serves as a trainer,
instantiated as a motivation-driven response generator. monitoring the skills that the newly spawned on-screen
This use of motivations to produce behavior is very agents exhibit. At the start of the game, the robots lack
similar to how humans rely on instincts to satisfy their any knowledge, only bearing the ability to learn and a set
internal needs, which, in turn, dictates their environmental of sensors (enemy detectors, target locators, and
responses. rangefinders). During a series of training exercises, the
agents are exposed to both battlefield obstacles and a
3.3 Evolutionary Agents human "God" that may manually eliminate weaker
candidates. These environmental factors determine
The term Evolutionary Agents refers to artificial agent whether or not certain individuals have adopted incorrect
systems that dynamically adapt to their environments with skills. Internally, the topology of each agent's neural
little or no use of scripted behaviors (Bossomaier et al, network is evaluated, where the inputs are sensor data and
2006). To capture this dynamic nature, an agent's the output is an action (move left, right, forward or back,
response system is broken down to a fine-grained or fire weapon). As the training progresses, newly
definition of inputs and outputs. A formulation of proper spawned populations take on the surviving individuals'
outputs with respect to certain inputs is then evolved over neural network topologies and perform genetic mutations
time. This model of behavior simulates nature's tendency to achieve more fit organisms. Stanley et al.'s work
to converge toward certain instinctive behaviors concludes with an overview of a series of actual NERO
(Brigandt, 2005). Additionally, these evolving agents treat training runs, documenting the successful training of
high-level behaviors as an aggregation of smaller input- different robot teams and behaviors. This paper provides
output reactions, which adheres to the instincts-as- an innovative mix of genetic programming with training
functional-primitives ideology. Past efforts in the and simulation techniques. The idea of evolving behaviors
evolutionary agent realm have even included generating hints at the application of rtNEAT in the evolution of
physical robot behaviors (Floreano and Mondada, 1994). instinctive behaviors.
The following section focuses on two software-baed
evolving behavioral agent implementations. Spronck, Sprinkhuizen-Kuyper and Postma propose a
methodology to improve the artificial intelligence of
Stanley, Bryant and Miikkulainen present the idea of video game agents (2004). The authors specify that the
evolving agents to improve their survival fitness in a major complaint of game players is that the computer AI
video game environment (2005). The paper asserts that can be beaten once its pre-scripted behaviors are
the video game realm serves as an ideal platform for identified. They argue that these scripts can adapt sets of
testing artificial intelligence techniques. The authors new, effective responses via post-release learning, This
recognize five concerns involved in such a virtual conception, called dynamic scripting, describes the
environment: an expansive state/action space, multiple process where a computer agent adjusts its behavior
behavior permutations, an expected behavior consistency, according to its interactions with human players. Spronck
short learning times, and a recollection of historical et al. assert that dynamic scripting must exhibit speed,
actions. Stanley et al. propose the use of their effectiveness, robustness, and efficiency to achieve
reinforcement-learning infrastructure, Real-time successful results. They choose an agent learning
NeuroEvolution of Augmenting Topologies (rtNEAT), to methodology that begins with a base set of behavioral
address the issues posed by these challenges. rtNEAT is a rules. Each of these rules is selected using a certain
probability-based weight. As the agent gains experience response. The two agents' performances will be analyzed,
through its interactions with the environment, especially with particular emphasis on the capabilities of the
with its human opponents, it updates these weights in instinct-only agent. Specifically, metrics pertaining to an
relation to the reward-based reinforcement it associates agent's survivability will be examined; these measures
with each rule. Because of the slow performance of quantify an agent's ability to evade obstacles or threats
traditional reinforcement learning, a simplified weight- and the agent's ability to prolong its life span. It is
updating technique is proposed. To test this dynamical expected that the instinct-only agent will not be as
scripting, the paper cites seven archetypes of human effective as the knowledge-based entity. This research
tactics that the agents must deal with. The amount of time will, however, provide insights into the effectiveness of
required by the AI opponent to defeat the human player an instinct-only model. Specifically, insights into what
was recorded. It is discovered that the computer player tasks or missions that can be effectively managed by an
was indeed able to achieve victory over each of the instinct-only agent can be useful. Ultimately, an adequate
tactical representatives, with the longest amount of time default behavior and when it is called for can be
for defeat belonging to the strategy that most closely determined from the proposed research.
resembled that of an actual human. While Spronck et al.
have described a promising technique in improving video 5. Summary
game AI, the idea of a base set of rules conveys the idea
that a pre-scripted routine is still in effect. The paper itself The preceding literature review surveys different research
describes these rule sets as domain-specific information, efforts that give insight into building an HBR based on
conveying the idea that they have basically constructed a instincts. The impetus of establishing instinctive
case-based reasoning system or perhaps a dynamic expert primitives in an artificial agent is to complete the
system. An improvement to their work would be to nature/nurture relationship, as featured in psychology's
incorporate a more generalized set of rule primitives, depiction of behavior. The AI research has provided many
instead of pre-determined behaviors. Also, the results did solutions for the nurturing side, where learning techniques
not convince the reader that their dynamically scripted have been fiercely explored for years. The nature side of
computer opponent would defeat a typical human player, behavior has yet to be explored with such thoroughness.
as the human player tactics did not completely capture the The point of this paper is to lay down the groundwork for
essence of normal playing strategies. A useful experiment a new paradigm that enhances artificial agent behavior,
to add would pit actual human players against the AI. one in which simulations are enhanced by the existence of
instinctive constructs. These constructs will reflect a
The preceding discussion covers two methods in which generalized nature, as to remain generic enough to apply
agent behavior sets are optimized via evolutionary to any type of environmental input data, whether foreign
processes. Agents resulting from these architectures are or recognizable.
equipped with a priori behaviors that have withstood the
test of generational survival. This procedure of evolving The fundamental problem in building an HBR based on
behaviors resembles that of the evolutionary development instincts remains in defining the computational
of instincts in animals. It is quite reasonable that a representation of instincts and engineering that
universal behavior representation could incorporate this representation in an artificial intelligence architecture.
evolutionary spirit to maintain a sense of generality. This paper has presented a general treatment of instincts,
under the both psychological and ethological standpoints,
4. Future Work as well as a series of AI methods that share ideology of
using basic building blocks for behavior. The next step in
This paper serves as a review of the psychological, this research endeavor is to build the proposed instinct-
ethological and artificial intelligence components related based HBR under the guidelines of these findings.
to producing an HBR based on instincts. The works cited
will contribute to the actual model in a simulation and 6. Acknowledgments
training environment.
This paper was made possible by the AT&T Corporation
The instinctive HBR will be applied in two types of through the Institute for Simulation and Training at the
agents: (1) an instinct-only agent, and (2) a knowledge- University of Central Florida.
based behavior agent with instincts as a default behavior.
The instinct-only agent will behave purely on its
instinctive primitives, while the knowledge-based agent
7. References
will have a priori knowledge on how to behave for certain
pre-defined stimuli. When the knowledge-based entity Anderson, J. R. (1993). Rules of the Mind. Hillsdale, NJ:
encounters a new stimulus, it defaults to its instincts for a Erlbaum.
Andriamasinoro, F. (2004). Modeling natural motivations Galton, F. (1874, 1895). English men of science: Their
into hybrid artificial agents. Fourth International nature and nurture. New York: D. Appleton and
ICSC Symposium on Engineering of Intelligent Company.
Systems Garcia, C. J. and Griffith, T. W. (2005). A Composable
Angell, J. R. (1906). The Important Human Instincts. Behavior Modeling System for Rapidly
Chapter 16 in Psychology: An Introductory Constructing Human Behaviors. Proceedings of
Study of the Structure and Function of Human the Industry/Interservice, Training, Simulation
Conscious, Third edition, revised. New York: and Education Conference.
Henry Holt and Company, 294-309. Giordano, J. C., Reynolds, P. F. and Brogan, D. C.
Balkenius, C. (1994). Biological Learning and Artificial (2004). Exploring The Constraints of Human
Intelligence. Lund University Cognitive Studies, Behavior Representation. Proceedings of the
30. 2004 Winter Simulation Conference, 912–921.
Bossomaier, T., Knittel, A., Harre, M. and Snyder, A. Gratch, J. and Marsella, S. (2004). A domain independent
(2006). An Evolutionary Agent Approach to framework for modeling emotion. Journal of
Dots-and-Boxes. Software Engineering Cognitive Systems Research, 5 (4), 269-306.
Applications, 514. Jones, R., Laird, J., Nuxoll, A., and Wray, R. (2002).
Brigandt, I. (2005). The instinct concept of the early Intelligent opponents for virtual reality trainers.
Konrad Lorenz. Journal of the History of Proceedings of the Interservice/Industry
Biology. 38, 571–608 Training, Simulation and Education Conference.
Brooks, R. (1986). A robust layered control system for a Kadlecek, D. and Nahodil, P. (2001). New Hybrid
mobile robot. Robitcs and Automation, 2(1), 14- Architecture in Artificial Life Simulation. ECAL,
23. 143–146.
Brooks, R. (1991). Intelligence Without Representation. Kantor, J. R. (1920). A Functional Interpretation of
Artificial Intelligence, 47, 139–159. Human Instincts. Psychological Review (27), 50-
Burt, A. (1998). Modelling Motivational Behaviour in 72.
Intelligent Agents in Virtual Worlds. Proc. of the Konidaris, G. D. and Barto, A. (2006). An Adaptive
1998 Conference on Virtual Worlds and Robot Motivational System. Animals to Animats
Simulation. 9: Proceedings of the 9th International
Chrysanthakopoulos, G. and Marks, R. J. (1998). Conference on Simulation of Adaptive Behavior.
Simulated autonomous agents utilizing instincts Konidaris, G. D., & Hayes, G. M. (2005). An
and behavior learning: Orgs in Orgland. IEEE Architecture for Behavior-Based Reinforcement
International Conference on Evolutionary Learning. Adaptive Behavior - Animals, Animats,
Computation, 727–734. Software Agents, Robots, Adaptive Systems,
Diller, D. E., Ferguson, W., Leung, A. M., Benyo, B. and 13(1), 5-32.
Foley, D. (2004). Behavior Modeling in Laird, J. E., Newell, A. and Rosenbloom, P. S. (1987).
Commercial Games. Behavior Representation in Soar: An Architecture for General Intelligence.
Modeling and Simulation. Artificial Intelligence, 33, 1-64.
Ekman, P., Friesen, W. V., O'Sullivan, M., Chan, A., Marsella, S. and Gratch, J. (2003). Modeling coping
Diacoyanni-Tarlatzis, I., Heider, K., Krause, R., Behavior in Virtual Huamns: Don’t Worry, Be
LeCompte, W. A., Pitcairn, T., Ricci-Bitti, P. E., Happy. Procedings of Second International Joint
Scherer, K., Tomita, M., & Tzavaras, A. (1987). Conference on Autonomous Agents and
Universals and cultural differences in the Multiagent Systems.
judgments of facial expressions of emotion. Maslow, A. (1954). Toward a Psychology of Being, 3rd
Journal of Personality and Social Psychology, Edition. Edited by Lowry Richard, J. Wiley &
53 (4), 712-717. Sons, Inc, November 1998.
Floreano, D. and Mondada, F. (1994). Automatic creation Mataric, M. J. (1995). Designing and Understanding
of an autonomous agent: Genetic evolution of a Adaptive Group Behavior. Adaptive Behavior
neural-network driven robot. Proc. of the 3rd Int. 4(1), 51-80
Conf. on Simulation of Adaptive Behavior, 421– Mead, G. H. (1909). Social Psychology as Counterpart to
430. Physiological Psychology. Psychological
Fu, D., Houlette, R., Jensen, R. and Bascara. O. (2003). A Bulletin (6), 401- 408.
Visual, Ob ject-Oriented Approach to Simulation Moffat, D. and Frijda, N. (1994). Where There's a Will
Behavior Authoring. Proceedings of the There's an Agent. ECAI Workshop on Agent
Industry/Interservice, Training, Simulation and Theories, Architectures, and Languages, 245-
Education Conference. 260.
Portegys, T. E. (1999). A connectionist model of Author Biographies
motivation. IJCNN’99 Proceeding, 80– 85.
Portegys, T. E. (2006). Instinct Evolution in a Goal- VICTOR HUNG is a Ph. D. student at the University of
Seeking Neural Network. IASTED International Central Florida in Orlando, Florida. He is pursuing
Conference on Computational Intelligence. research in developing a human behavior model based on
Rosenblatt, K., & Payton, D. (1989). A Fine-Grained instincts.
Alternative to the Subsumption Architecture for
Mobile Robot Control. Proceedings of the IEEE AVELINO GONZALEZ is a full-time professor of
International Conference on Neural Networks. Computer Engineering at the University of Central
St. Amant, R., McBride, S. P. and Ritter, F. E. (2006). AI Florida. He is currently the interim head chair of the Civil
support for building cognitive models. and Environmental Engineering department.
Proceedings of the Twenty-First National
Conference on Artificial Intelligence (AAAI,
Nectar track), 1663–1666.
Samsonovich, A. V., and De Jong, K. A. (2004). A
general-purpose computational model of the
conscious mind. Proceedings of the Sixth
International Conference on Cognitive Modeling
ICCM-2004, 382-383.
Samsonovich, A. V. and De Jong, K. A. (2005).
Designing a self-aware neuromorphic hybrid.
AAAI-05 Workshop on Modular Construction of
Human-Like Intelligence, 71–78.
Stanley, K. O., Bryant, B. D., & Miikkulainen, R. (2005).
Evolving Neural Network Agents in the NERO
Video Game. Proceedings of the IEEE 2005
Symposium on Computational Intelligence and
Games (CIG'05).
Soffker, D. (2001). From Human Machine-Interaction
Modeling to New Concepts Constructing
Autonomous Systems: A Phenomenological
Engineering-Oriented Approach. Journal of Intel
ligent and Robotic Systems, 32(2), 191–205.
Spronck, P., Sprinkhuizen-Kuyper, I., & Postma, E.
(2004). Online Adaptation of Game Opponent
AI with Dynamic Scripting. International
Journal of Intelligent Games and Simulation,
3(1), 45–53.
Tinbergen, N. (1948). Social releasers and the
experimental method required for their study.
Wilson Bulletin, 60, 6-52.
Zachary, W., Ross, L. and Weiland, M. Z. (1991).
COGNET and BATON: An Integrated Approach
to Embedding User Models in Complex
Systems. Proc. of the IEEE International
Conference on System, Man and Cybernetics,
689-694.