Blue Eyes Technology Dept of Computer.

Science

INTRODUCTION

Imagine yourself in a world where humans interact with computers. You are
sitting in front of your personal computer that can listen, talk, or even scream
aloud. It has the ability to gather information about you and interact with you
through special techniques like facial recognition, speech recognition, etc. It
can even understand your emotions at the touch of the mouse. It verifies
your identity, feels your presents, and starts interacting with you .You ask
the computer to dial to your friend at his office. It realizes the urgency of the
situation through the mouse, dials your friend at his office, and establishes a
connection.

Human cognition depends primarily on the ability to perceive, interpret, and

integrate audio-visuals and sensoring information. Adding extraordinary
perceptual abilities to computers would enable computers to work together
with human beings as intimate partners. Researchers are attempting to add
more capabilities to computers that will allow them to interact like humans,
recognize human presents, talk, listen, or even guess their feelings.

The BLUE EYES technology aims at creating computational machines that

have perceptual and sensory ability like those of human beings. It uses some
sensing method, employing most modern video cameras and microphones to
identify the users actions through the use of imparted sensory abilities. The
machine can understand what a user wants, where he is looking at, and even
realize his physical or emotional states.

The Oxford College Of Engineering 1

Blue Eyes Technology Dept of Computer. Science

1. KEY FEATURES OF THE BLUE EYES TECHNOLOGY

 Visual attention monitoring (eye motility analysis)

 Physiological condition monitoring (pulse rate, blood

oxygenation)

 Operator’s position detection (standing, lying)

 Wireless data acquisition using Bluetooth technology

 Real-time user-defined alarm triggering

 Physiological data, operator's voice and overall view of the

control room recording

 Recorded data playback

The Oxford College Of Engineering 2

Blue Eyes Technology Dept of Computer. Science

2. System overview

BlueEyes system provides technical means for monitoring and recording the
operator’s basic physiological parameters. The most important parameter is
saccadic activity1, which enables the system to monitor the status of the
operator’s visual attention along with head acceleration, which accompanies
large displacement of the visual axis (saccades larger than 15 degrees).
Complex industrial environment can create a danger of exposing the
operator to toxic substances, which can affect his cardiac, circulatory and
pulmonary systems. Thus, signals taken from the forehead skin surface, the
system computes heart beat rate and blood oxygenation.
The BlueEyes system checks above parameters against abnormal (e.g. a low
level of blood oxygenation or a high pulse rate) or undesirable (e.g. a longer
period of lowered visual attention) values and triggers user-defined alarms
when necessary.
Quite often in an emergency situation operators speak to themselves
expressing their surprise or stating verbally the problem. Therefore, the
operator’s voice, physiological parameters and an overall view of the
operating room are recorded. This helps to reconstruct the course of
operators’ work and provides data for long-term analysis.

1
Saccades are rapid eye jumps to new locations within a visual environment assigned predominantly by the

conscious attention process.

The Oxford College Of Engineering 3

Blue Eyes Technology Dept of Computer. Science

This system consists of a mobile measuring device and a central analytical

system. The mobile device is integrated with Bluetooth module providing

D a t a A c q u i s it i o n U n it C e n t r a l S y s t e m U n it

8 0 5 1 f a m il y B lu e t o o t h B lu e t o o t h C o n n e c tio n M a n a g e r D a ta L o g g e r
m i c r o c o n t r o lle r d e v ic e d e v ic e M o d u le M o d u le

D a ta b a s e
V o ic e
P h y s io lo g i c a l in t e r fa c e D a ta A n a ly s is V is u a lis a tio n
p a ra m e te rs M o d u le M o d u le
sensor

Figure 1. Overall system diagram

wireless interface between sensors worn by the operator and the central unit.
ID cards assigned to each of the operators and adequate user profiles on the
central unit side provide necessary data personalization so different people
can use a single mobile device (called hereafter DAU – Data Acquisition
Unit). The overall system diagram is shown in Figure 1.
The tasks of the mobile Data Acquisition Unit are to maintain Bluetooth
connections, to get information from the sensor and sending it over the
wireless connection, to deliver the alarm messages sent from the Central
System Unit to the operator and handle personalized ID cards. Central
System Unit maintains the other side of the Bluetooth connection, buffers
incoming sensor data, performs on-line data analysis, records the
conclusions for further exploration and provides visualization interface.

The Oxford College Of Engineering 4

Blue Eyes Technology Dept of Computer. Science

3. Data Acquisition Unit (DAU)

In this section we describe the hardware part of the BlueEyes system with
regard to the physiological data sensor, the DAU hardware components and
microcontroller software.
Atmel 8952 micro controller is the core of the Data Acquisition Unit since it
is a well-established industrial standard and provides necessary functionality
(i.e. high speed serial port) at a low price. The figure below shows the other
DAU components.

m ic ro p h o n e M C 145483
P C M
P C M codec
e a rp h o n e

J A Z Z M u l ti s e n s o r
B lu e to o th
m o d u le

Beeper
UART
A tm e l 8 9 5 2
m ic r o c o n tr o lle r
S m a ll a lp h a n u m e ric M A X232
L C D d is p la y L e v e l s h i ft e r

L E D in d ic a to rs

B lu e E ye s
ID c a rd m e c h a n ic S im p le D a t a A c q u is it io n U n it
in te rfa c e k e y b o a rd

ID c a r d

Figure 6. DAU hardware diagram

Since the Bluetooth module uses synchronous voice data transmission (SCO
link) therefore it used hardware PCM codec to transmit operator’s voice and
central system sound feedback.

The Oxford College Of Engineering 5

Blue Eyes Technology Dept of Computer. Science

The codec that has been employed reduces the microcontroller’s tasks and
lessens the amount of data being sent over the UART. Additionally, the
Bluetooth module performs voice data compression, which results in smaller
bandwidth utilization and better sound quality. We have chosen Motorola
MC145483 linear 13-bit 3.3V codec to be used in the prototype, since it is
voltage-level compatible with the Bluetooth module.
Communication between the Bluetooth module and the microcontroller is
carried on using standard UART interface. MAX232 Level Shifter does the
RS232 ↔ TTL voltage level conversion. The speed of the UART is set to
115200 bps in order to assure that the entire sensor data is delivered in time
to the central system (sensor data along with Bluetooth-related control data
take up approx. 70% of the bandwidth).

3.1 The alphanumeric LCD display -(two 8-character lines) gives more
information of incoming events and helps the operator enter PIN code. In the
prototype we used inexpensive Hitachi command-set LCD display.

3.2 The LED indicators -show the results of built-in self-test, power
level and the state of wireless connection.

3.3 The simple keyboard -is used to react to incoming events (e.g. to
silence the alarm sound) and to enter PIN code while performing
authorization procedure.

3.4 ID card interface- helps connect the operator’s personal identification

card to the DAU. After inserting the card authorization procedure starts.

The Oxford College Of Engineering 6

Blue Eyes Technology Dept of Computer. Science

In the prototype as the ID card an EEPROM (24C16) was used since it

provides easy programming interface and is capable of holding the entire
necessary operator’s data. In the commercial release a cryptographic
processor should be used instead. Each ID card is programmed to contain:
operator’s unique identifier, device access PIN code the operator enters on
inserting his ID card and system access PIN code that is used on connection
authentication. The operator’s unique identifier enables the supervising
system to distinguish different operators.

3.5 Jazz multi sensor- To provide the Data Acquisition Unit with
necessary physiological data we decided to purchase an off-shelf eye
movement sensor -Jazz Multisensor. It supplies raw digital data regarding
eye position, the level of blood oxygenation, acceleration along horizontal
and vertical axes and ambient light intensity.

The Oxford College Of Engineering 7

Blue Eyes Technology Dept of Computer. Science

4. Central System Unit (CSU)

In this section we describe the four main CSU modules (see Fig. 1):
Connection Manager, Data Analysis, Data Logger and Visualization. The
modules exchange data using specially designed single-producer-multi-
consumer buffered thread-safe queues. Any number of consumer modules
can register to receive the data supplied by a producer. Every single
consumer can register at any number of producers, receiving therefore
different types of data.

The Oxford College Of Engineering 8

Blue Eyes Technology Dept of Computer. Science

4.1 Connection Manager

Connection Manager’s main task is to perform low-level Blue

tooth communication using Host Controller Interface commands.

It is designed to cooperate with all available Bluetooth devices in

order to support roaming.

4.1.1 Transport Layer Manager

Hides the details regarding actual Bluetooth physical transport interface.

4.1.2 Bluetooth Connection Manager

Is responsible for establishing and maintaining connections using all
available Bluetooth devices. It periodically inquires new devices in an
operating range and checks whether they are registered in the system
O p e ra to r
D a ta D a ta D a ta p ro d u c e r
D a ta b a s e d e m u l t i p le x e r queues
M anager

T ra n sp o rt D a ta s tre a m
B lu e t o o t h O p e ra to r
Layer
m o d u le M anager r a w d a ta b u ffe r Logger
M anager
A le rt b u ffe r A le rts

... ... ...

B l u e t o o th
... ... C o n n e c tio n ...
M anager
... ... ...
T ra n sp o rt D a ta s tre a m
B lu e t o o t h O p e ra to r
Layer
m o d u le M anager r a w d a ta b u ffe r Logger
M anager
A le rt b u ffe r A le rts

Figure 11. Connection Manager Components

database. Only with those devices the Connection Manager will

The Oxford College Of Engineering 9

Blue Eyes Technology Dept of Computer. Science

communicate. After establishing a connection an authentication procedure

occurs. The authentication process is performed using system PIN code
fetched from the database. Once the connection has been authenticated the
mobile unit sends a data frame containing the operator’s identifier.

4.1.3 Operator Manager.

At the startup it communicates with the Operator Data Manager in order to
get more detailed personal data. The most important Operator Manager’s
task is to buffer the incoming raw data and to split it into separate data
streams related to each of the measured parameters. The raw data is sent to a
Logger Module. Furthermore, the Operator Manager provides an interface
for sending alert messages to the related operator.

4.1.4 Operator Data Manager

provides an interface to the operator database enabling the other modules to
read or write personal data and system access information.

4.2 Data Analysis Module

The module performs the analysis of the raw sensor data in order to obtain
information about the operator’s physiological condition. The separately
running Data Analysis Module supervises each of the working operators.
The module consists of a number of smaller analyzers extracting different
types of information. An analyzer can be either a simple signal filter (e.g.
Finite Input Response (FIR) filter) or a generic data extractor (e.g. signal
variance, saccade detector) or a custom detector module. In tree induction

The Oxford College Of Engineering 10

Blue Eyes Technology Dept of Computer. Science

algorithm, The computed features can be e.g. the operator’s position

(standing, walking and lying) or whether his eyes are closed or opened. As
built-in analyzer modules we implemented a saccade detector, visual
attention level, blood oxygenation and pulse rate analyzers.

4.2.1 The saccade detector-

Since saccades are the fastest eye movements the algorithm calculates eye
movement velocity and checks physiological constraints.

The algorithm we used performs two main steps:

 User adjustment step

 On-line analyzer flow

 User adjustment step. -The phase takes up 5 s. After buffering

approx. 5 s of the signal differentiate it using “Three point central
difference algorithm”, which will give eye velocity time series. Sort
the velocities by absolute value and calculate upper 15% of the border
velocity along both X – v0x and Y – v0y axes. As a result v0x and v0y are
cut-off velocities.

 On-line analyzer flow. Continuously calculate eye movement

velocity using three-point central difference algorithm. If the velocity

The Oxford College Of Engineering 11

Blue Eyes Technology Dept of Computer. Science

excess pre calculated v0 (both axes are considered separately) there is

a possibility of saccade occurrence. Check the following conditions (if
any of them is satisfied do not detect a saccade)

• The last saccade detection was less than 130 ms ago (physiological
constraint – the saccades will not occur more frequently)
• The movement is nonlinear (physiological constraint)
• Compare the signal with accelerometer (rapid head movement may force
eye activity of comparable speed)
• If the accelerometer signal is enormously uneven consider ignoring the
signal due to possible sensor device movements.

If none of the above conditions is satisfied – signal the saccade occurrence.

4.2.2 visual attention level analyzer- uses as input the results produced
by the saccade detector.

Low saccadic activity (large delays between subsequent saccades) suggests

lowered visual attention level (e.g. caused by thoughtfulness). Thus, we
propose a simple algorithm that calculates the visual attention level (Lva): Lva
= 100/ts10, where ts10 denotes the time (in seconds) occupied by the last ten
saccades. Scientific research has proven [1] that during normal visual
information intake the time between consecutive saccades should vary from
180 up to 350 ms. This gives Lva at 28 up to 58 units. The values of Lva lower
than 25 for a longer period of time should cause a warning condition. The

The Oxford College Of Engineering 12

Blue Eyes Technology Dept of Computer. Science

following figure shows the situation where the visual attention lowers for a
few seconds.
Pulse rate analyzer

Registers for the oxy-hemoglobin and de-oxy-hemoglobin level data

Eye movement velocity
250 saccade
200
150
100
Velocity

50
0
-50
-100
-150
-200
-250
0 8000 16000 24000 32000
Time [ms]

V
Attention level (Lva)

60
50
Figure 12. Saccade occurrences and visual attention level

40
4.3 Data Logger Module
30
The module registers as a consumer of the data to be stored in the database.
20 data is recorded by a separate instance of the Data
Each working operator’s
Logger. Apart from the raw or processed physiological data, alerts and
10
operator’s voice are stored. The raw data is supplied by the related Operator
0 the Data Analysis module delivers the processed
Manager module , whereas
data. A Voice Data Acquisition
0 module delivers the voice data. The module
8000
registers as an operator’s voice data consumer and optionally processes the
sound to be stored (i.e. reduces noise or removes the fragments when the

The Oxford College Of Engineering 13

Blue Eyes Technology Dept of Computer. Science

operator does not speak). The Logger’s task is to add appropriate time
stamps to enable the system to reconstruct the voice.

4.4 Visualization Module

The module provides user interface for the supervisors. It enables them to
watch each of the working operator’s physiological condition along with a
preview of selected video source and his related sound stream. All the
incoming alarm messages are instantly signaled to the supervisor. Moreover,
the visualization module can be set in the off-line mode, where all the data is
fetched from the database. Watching all the recorded physiological
parameters, alarms, video and audio data the supervisor is able to reconstruct
the course of the selected operator’s duty.

5. ADVANTAGES

 Preventions from dangerous incidents

 Minimization of

 Ecological consequences

 Financial loss

 Threat to human life

The Oxford College Of Engineering 14

Blue Eyes Technology Dept of Computer. Science

 The reconstruction of the course operators

6. DISADVANTAGES

 Threshold

 If one does not look at a target continuously for a set threshold

(e.g., 200 ms), the target will not be successfully selected. On
the other hand, if one stares at an object for more than the set
threshold, the object will be selected, regardless of the user’s
intention.

 Exactness

 Using eye gaze for exact pin pointing does not appear to be
very promising.

Summary

We have decided to develop the BlueEyes system because of the need for a
real-time monitoring system for a human operator. The approach is
innovative since it helps supervise the operator not the process, as it is in
presently available solutions. We hope the system in its commercial release
will help avoid potential threats resulting from human errors, such as
weariness, oversight, tiredness or temporal indisposition. However, the
prototype we have developed is a good estimation of the possibilities of the
final product.

The Oxford College Of Engineering 15

Blue Eyes Technology Dept of Computer. Science

We find it possible still to improve our project. The use of a miniature

CMOS camera integrated into the eye movement sensor will enable the
system to calculate the point of gaze and observe what the operator is
actually looking at. Introducing voice recognition algorithm will facilitate
the communication between the operator and the central system and simplify
authorization process.
Despite considering in the report only the operators working in control
rooms, our solution may well be applied to everyday life situations.
Assuming the operator is a driver and the supervised process is car driving it
is possible to build a simpler embedded on-line system, which will only
monitor conscious brain involvement and warn when necessary. As in this
case the logging module is redundant, and the Bluetooth technology is
becoming more and more popular, the commercial implementation of such a
system would be relatively inexpensive. The final thing is to explain the
name of our system. BlueEyes emphasizes the foundations of the project –
Bluetooth technology and the movements of the eyes. Bluetooth provides
reliable wireless communication whereas the eye movements enable us to
obtain a lot of interesting and important information.

CONCLUSION

The Blue eyes technology is convenient way of simplifying the life by more
delicate and user-friendly facilities in computing device. The day is not far
when this technology will push its way into your house and making human
work easy.

The Oxford College Of Engineering 16

Blue Eyes Technology Dept of Computer. Science

References

1. Carpenter R. H. S., Movements of the eyes, 2nd edition, Pion Limited,

1988, London.
2. Quinlan J. R., C4.5 programs for machine learning, Morgan Kaufmann
Publishers, 1993, San Mateo, California.
3. Horowitz P., Hill W., The art of Electronics, Cambridge University Press,
1989.
4. Bluetooth specification, version 1.0B, Bluetooth SIG, 1999.

The Oxford College Of Engineering 17

Blue Eyes Technology Dept of Computer. Science

5. ROK 101 007 Bluetooth Module, Ericsson Microelectronics, 2000.

6. ST14C16 memory card IC Datasheet, ST Microelectronics, 1999.
7. Leinecker R. C., Archer T., Visual C++ Bible, IDG Books, 1999.
8. MSDN Library, Microsoft (Visual Studio 6.0).
9. Intel Signal Processing Library – Reference Manual.
10. http://www.cs.put.poznan.pl/csidc/2001/en_download.html

11. http://www.almaden.ibm.com/cs/BlueEyes/index.html

The Oxford College Of Engineering 18