BLUE EYES TECHNOLOGY

INDEX
1. INTRODUCTION...........................................................................................................2 2. OVERALL SYSTEM OVERVIEW................................................................................4 3. PARTS OF BLUE EYES TECHNOLOGY....................................................................5 3.1 DATA AQUISITION UNIT 3.2 CENTRAL SYSTEM UNIT 4. TOOLS USED...............................................................................................................16 5. ADVANTAGES AND DISADVANTAGES................................................................18 6. APPLICATIONS...........................................................................................................20 7. FUTURE TRENDS.......................................................................................................21 8. CONCLUSION.24 9. REFERENCE.25

DEPT.OF ECE, GSSSIETW, MYSORE

Page 1

BLUE EYES TECHNOLOGY

INTRODUCTION
BlueEyes system provides technical means for monitoring and recording the operators basic physiological parameters. The most important parameter is saccadic activity, which enables the system to monitor the status of the operators visual attention along with head acceleration, which accompanies large displacement of the visual axis. Complex industrial environment can create a danger of exposing the operator to toxic substances, which can affect his cardiac, circulatory and pulmonary systems. Thus, on the grounds of plethysmographic signal taken from the forehead skin surface, the system computes heart beat rate and blood oxygenation. The BlueEyes system checks above parameters against abnormal (e.g. a low level of blood oxygenation or a high pulse rate) or undesirable (e.g. a longer period of lowered visual attention) values and triggers user-defined alarms when necessary. This paper is about the hardware, software, benefits and interconnection of various parts involved in the blue eye technology. Toward the end, the Blue Eyes aims at creating computational devices with the sort of perceptual abilities that people take for granted. Blue eyes is being developed by the team of Poznan University of Technology & Microsoft. It makes use of the blue tooth technology developed by Ericsson.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 2

BLUE EYES TECHNOLOGY

What Is Blue Eyes?

BLUE EYES is a technology, which aims at creating computational machines to have perceptual and sensory abilities like those of human beings. The basic idea behind this technology is to give computer human power. For example, we can understand humans emotional state by his facial expressions. If we add these perceptual abilities to computers, we would enable them to work together with human beings as intimate partners. It provides technical means for monitoring and recording human-operators physiological condition.

Key features of the system

Visual attention monitoring (eye motility analysis). Physiological condition monitoring (pulse rate, blood oxygenation). Operators position detection (standing, lying). Wireless data acquisition using Bluetooth technology. Real-time user-defined alarm triggering. Physiological data, operator's voice and overall view of the control room recording. Recorded data playback.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 3

BLUE EYES TECHNOLOGY

Why its named Blue Eyes?

BlueEyes emphasizes Bluetooth technology and the movements of the eyes. Bluetooth provides reliable wireless communication whereas the eye movements enable us to obtain a lot of interesting and important information.

WORKING

Figure1.block diagram

The major parts in the Blue eye system are Data Acquisition Unit and Central System Unit. The tasks of the mobile Data Acquisition Unit are to maintain Bluetooth connections, to get information from the sensor and sending it over the wireless connection, to deliver the alarm messages sent from the Central System Unit to the operator and handle personalized ID cards. DEPT.OF ECE, GSSSIETW, MYSORE Page 4

BLUE EYES TECHNOLOGY Central System Unit maintains the other side of the Bluetooth connection, buffers incoming sensor data, performs on-line data analysis, records the conclusions for further exploration and provides visualization interface.

PARTS OF BLUE EYE TECHNOLOGY

The main parts in the Blue eye system are 1. Data Acquisition Unit 2. Central System Unit

Data Acquisition Unit (DAU)

Data Acquisition Unit is a mobile part of the Blue eyes system. Its main task is to fetch the physiological data from the sensor and to send it to the central system to be processed. To accomplish the task the device must manage wireless Bluetooth connections (connection establishment, authentication and termination). Personal ID cards and PIN codes provide operator's authorization. Communication with the operator is carried on using a simple 5-key keyboard, a small LCD display and a beeper. When an exceptional situation is detected the device uses them to notify the operator. Voice data is transferred using a small headset, interfaced to the DAU with standard mini-jack plugs.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 5

BLUE EYES TECHNOLOGY

1. Physiological data sensor

Figure 2. Jazz Multisensor

DEPT.OF ECE, GSSSIETW, MYSORE

Page 6

BLUE EYES TECHNOLOGY

To provide the Data Acquisition Unit with necessary physiological data we decided to purchase an off-shelf eye movement sensor Jazz Multisensor. It supplies raw digital data regarding eye position, the level of blood oxygenation, acceleration along horizontal and vertical axes and ambient light intensity. Eye movement is measured using direct infrared oculographic transducers.The eye movement is sampled at 1kHz, the other parameters at 250 Hz. The sensor sends approximately 5.2kB of data per second.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 7

BLUE EYES TECHNOLOGY

2. Hardware specification
We have chosen Atmel 8952 microcontroller to be the core of the Data Acquisition Unit since it is a well-established industrial standard and provides necessary functionality (i.e. high speed serial port) at a low price.The figure below shows the other DAU components.

Figure 3. DAU hardware diagram Since the Bluetooth module supports synchronous voice data transmission (SCO link). we decided to use hardware PCM codec to transmit operators voice and central system sound feedback. The codec that we have employed reduces the microcontrollers tasks and reduces the amount of data being sent over the UART. Additionally, the Bluetooth module performs voice data compression, which results in smaller bandwidth utilization and better sound quality. Communication between the Bluetooth module and the microcontroller is carried on using standard UART interface.. DEPT.OF ECE, GSSSIETW, MYSORE Page 8

BLUE EYES TECHNOLOGY

The alphanumeric LCD display (two 8-character lines) gives more information of incoming events and helps the operator enter PIN code . The LED indicators show the results of built-in self-test, power level and the state of wireless connection. The simple keyboard is used to react to incoming events (e.g. to silence the alarm sound) and to enter PIN code while performing authorization procedure. ID card interface helps to connect the operators personal identification card to the DAU. After inserting the card authorization procedure starts. In the commercial release a cryptographic processor should be used instead. Each ID card is programmed to contain: operators unique identifier, device access PIN code, the operator enters on inserting his ID card and system access PIN code that is used on connection authentication. The operators unique identifier enables the supervising system to distinguish different operators.

3. Microcontroller software specification

All the DAU software is written in 8051 assembler code, which assures the highest program efficiency and the lowest resource. At a low-level design stage we modeled the software using a state diagram. Such a diagram facilitates implementation, debugging and testing phases.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 9

BLUE EYES TECHNOLOGY

Central System Unit (CSU)

Figure 4.CSU Central System Unit hardware is the second peer of the wireless connection. The box contains a Bluetooth module and a PCM codec for voice data transmission. The module is interfaced to a PC using a parallel, serial and USB cable. The audio data is accessible through standard mini jack sockets.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 10

BLUE EYES TECHNOLOGY

To program operator's personal ID cards we developed a simple programming device. The programmer is interfaced to a PC using serial and PS/2(power source) ports. Inside, there is Atmel 89C2051 microcontroller, which handles UART transmission and I2C EEPROM (ID card) programming. In this section we describe the four main CSU modules Connection Manager, Data Analysis, Data Logger and Visualization.

1. Connection Manager
The Connection Manager handles: communication with the CSU hardware. searching for new devices. It is responsible for managing the wireless communication between the mobile Data Acquisition Units and the central system in the covered range . establishing Bluetooth connections . connection authentication . incoming data buffering . sending alerts .

2. Data Analysis Module

The module performs the analysis of the raw sensor data in order to obtain information about the operators physiological condition. The separately running Data Analysis Module supervises each of the working operators.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 11

BLUE EYES TECHNOLOGY

The module consists of a number of smaller analyzers extracting different types of information. Each of the analyzers registers at the appropriate Operator Manager or another analyzer as a data consumer and, acting as a producer, provides the results of the analysis. An analyzer can be either a simple signal filter (e.g. Finite Input Response (FIR) filter) or a generic data extractor (e.g. signal variance, saccade detector) or a custom detector module. As we are not able to predict all the supervisors needs, the custom modules are created by applying a supervised machine learning algorithm to a set of earlier recorded examples containing the characteristic features to be recognized. In the prototype we used an improved decision tree induction algorithm. The computed features can be e.g. the operators position (standing, walking and lying) or whether his eyes are closed or opened. As built-in analyzer modules we implemented a saccade detector, visual attention level, blood oxygenation and pulse rate analyzers.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 12

BLUE EYES TECHNOLOGY

Figure 5. Saccade occurrences and visual attention level The saccade detector registers as an eye movement and accelerometer signal variance data consumer and uses the data to signal saccade occurrence. Since saccades are the fastest eye movements the algorithm calculates eye movement velocity and checks physiological constraints.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 13

BLUE EYES TECHNOLOGY The visual attention level analyzer uses as input the results produced by the saccade detector. Low saccadic activity (large delays between subsequent saccades) suggests lowered visual attention level (e.g. caused by thoughtfulness). Thus, we propose a simple algorithm that calculates the visual attention level (Lva): Lva = 100/ts10, where ts10 denotes the time (in seconds) occupied by the last ten saccades. Scientific research has proven that during normal visual information intake the time between consecutive saccades should vary from 180 up to 350 s. this gives Lva at 28 up to 58 units. The values of Lva lower than 25 for a longer period of time should cause a warning condition. The following figure shows the situation where the visual attention lowers for a few seconds. The Pulse rate analyzer registers for the oxyhemoglobin and deoxyhemoglobin level data streams. Since both signals contain a strong sinusoidal component related to heartbeat, the pulse rate can be calculated measuring the time delay between subsequent extremes of one of the signals

3. Data Logger Module

The module provides support for storing the monitored data in order to enable the supervisor to reconstruct and analyze the course of the operators duty. The module registers as a consumer of the data to be stored in the database. Each working operators data is recorded by a separate instance of the Data Logger.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 14

BLUE EYES TECHNOLOGY

3. Visualization Module

Figure 6.visualization module The module provides user interface for the supervisors. It enables them to watch each of the working operators physiological condition along with a preview of selected video source and his related sound stream. All the incoming alarm messages are instantly signaled to the supervisor. Moreover, the visualization module can be set in the off-line mode, where all the data is fetched from the database.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 15

BLUE EYES TECHNOLOGY

The physiological data is presented using a set of custom-built GUI controls: a pie-chart used to present a percentage of time the operator was actively acquiring the visual information A VU-meter showing the present value of a parameter time series displaying a history of selected parameters' value

TYPES OF USERS
Users belong to three categories: Operators Supervisors System administrators

Operator:
Operator is a person whose physiological parameters are supervised. The operator wears the DAU. The only functions offers to the operator are Authorization to the system and receiving alarm alerts. Authorization: Operator has to enter his personal PIN into DAU, if PIN is accepted, authorization is said to be complete.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 16

BLUE EYES TECHNOLOGY Receiving Alerts: This function supplies the operator with the most important alerts about his and his co-workers condition and mobile device state.

Supervisor:
He is the person responsible for analyzing operators condition and performance. The supervisor receives tools for inspecting present values of the parameters (online browsing) as well as browsing the results of the long-term analysis (off-line browsing).

System Administrator:
He is the user that maintains the system. The administrator is delivered tools for adding new operators to the database. Defining alarm conditions. Configuring logging tools. Creating new analyzer modules.

TOOLS USED
During the implementation of the DAU there was a need of a piece of software to establish and test Bluetooth connections. Therefore a tool had been created called BlueDentist (Fig. 7).This tool provides support for controlling the currently connected Bluetooth device. Its functions are: local device management and connection management. To test the possibilities and performance of the remaining parts of the Project Kit (computer, camera and database software) BlueCapture had been created (Fig. 8).This DEPT.OF ECE, GSSSIETW, MYSORE Page 17

BLUE EYES TECHNOLOGY tool supports capturing video data from various sources (USB web-cam, industrial camera) and storing the data in the MS SQL Server database. Additionally, the application performs sound recording. After filtering and removing insignificant fragments (i.e. silence) the audio data is stored in the database. Finally, the program plays the recorded audiovisual stream. The software was used to measure database system performance and to optimize some of the SQL queries. Since all the components of the application have been tested thoroughly they were reused in the final software, which additionally reduced testing time. A simple tool was created for recording Jazz Multisensor measurements. The program reads the data using a parallel port and writes it to a file.

Figure 7. BlueDentist

DEPT.OF ECE, GSSSIETW, MYSORE

Page 18

BLUE EYES TECHNOLOGY

Figure 8. BlueCapture

To program the operators personal ID card a standard parallel port is used, as the EEPROMs and the port are both TTL-compliant. A simple dialog-based application helps to accomplish the task.

ADVANTAGES
Prevention from dangerous incidents Minimization of ecological consequences financial loss a threat to a human life

BlueEyes system provides technical means for monitoring and recording Human-operators physiological condition.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 19

BLUE EYES TECHNOLOGY

The key features of the system are:

visual attention monitoring (eye motility analysis) physiological condition monitoring (pulse rate, blood oxygenation) operator's position detection (standing, lying) wireless data acquisition using Bluetooth technology real-time user-defined alarm triggering physiological data, operator's voice and overall view of the control room recording recorded data playback

BlueEyes system can be applied in every working environment requiring permanent operator's attention: at power plant control rooms at captain bridges at flight control centers

Data security - This system implies data security which is require in the modern network system. Only registered mobile devices can connect to the system Bluetooth connection authentication & encryption Access rights restrictions Personal and physiological data encryption

DEPT.OF ECE, GSSSIETW, MYSORE

Page 20

BLUE EYES TECHNOLOGY

DISADVANTAGES
Doesnt predict nor interfere with operators thoughts. Cannot force directly the operator to work.

APPLICATIONS
1. It can be used in the field of security & controlling, where the contribution of human operator required in whole time.

2. Engineers at IBM's office:smart tags Research Center in San Jose, CA, report that a number of large retailers have implemented surveillance systems that record and interpret customer movements, using software from Almaden's BlueEyes research project. BlueEyes is developing ways for computers to anticipate users' wants by gathering video data on eye movement and facial expression. Your gaze might rest on a Web site heading, for example, and that would prompt your computer to find similar links and to call them up in a new window. But the first practical use for the research turns out to be snooping on shoppers.

3. Another application would be in the automobile industry. By simply touching a computer input device such as a mouse, the computer system is designed to be able to determine a person's emotional state. DEPT.OF ECE, GSSSIETW, MYSORE Page 21

BLUE EYES TECHNOLOGY 4. Current interfaces between computers and humans can present information vividly, but have no sense of whether that information is ever viewed or understood. In contrast, new real-time computer vision techniques for perceiving people allows us to create "Faceresponsive Displays" and "Perceptive Environments", which can sense and respond to users that are viewing them. Using stereo-vision techniques, we are able to detect, track, and identify users robustly and in real time. This information can make spoken language interface more robust, by selecting the acoustic information from a visually-localized source. Environments can become aware of how many people are present, what activity is occuring, and therefore what display or messaging modalities are most appropriate to use in the current situation. The results of our research will allow the interface between computers and human users to become more natural and intuitive.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 22

BLUE EYES TECHNOLOGY

FUTURE TREND

What can we do with blue eyes technology?

It has the ability to gather information about you and interact with you through special techniques like facial recognition, speech recognition, etc. It can even understand your emotions at the touch of the mouse. It can verify your identity, feel your presence, and start interacting with you. The machine can understand what a user wants, where he is looking at, and even realize his physical or emotional states. It realizes the urgency of the situation through the mouse. For instance if you ask the computer to dial to your friend at his office, it understands the situation and establishes a connection. It can reconstruct the course of operators work.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 23

BLUE EYES TECHNOLOGY

DEPT.OF ECE, GSSSIETW, MYSORE

Page 24

BLUE EYES TECHNOLOGY

DEPT.OF ECE, GSSSIETW, MYSORE

Page 25

BLUE EYES TECHNOLOGY

CONCLUSION
BlueEyes need for a real-time monitoring system for a human operator. The approach is innovative since it helps supervise the operator not the process, as it is in presently available solutions. This system in its commercial release will help avoid potential threats resulting from human errors, such as weariness, oversight, tiredness or temporal indisposition. In future it is possible to create a computer which can interact with us as we interact each other with the use of blue eye technology. It seems to be a fiction, but it will be the life lead by BLUE EYES in the very near future. Ordinary household devices -- such as televisions, refrigerators, and ovens -- may be able to do their jobs when we look at them and speak to them.

DEPT.OF ECE, GSSSIETW, MYSORE

Page 26

BLUE EYES TECHNOLOGY

REFERENCE
1. www.cs.put.poznan.pl/achieve/load/CSIDC01Poznan.ppt 2. ndt-equipment-supply.com/.../blue-eyes-monitoring-human-operator-system 3. www.cs.put.poznan.pl/csidc/2001/images/html/en_jazz.html 4. www.slideshare.net/jainshef/blue-eye-technology 5. www.seminar4u.com 6. www.howstuffswork.com 7. www.cs.put.poznan.com 8. www.whitepapers.com 9. www.ibmresearchcenter.com 10. www.almaden.ibm.com

DEPT.OF ECE, GSSSIETW, MYSORE

Page 27