c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678

journal homepage: www.intl.elsevierhealth.com/journals/cmpb

A neural-based remote eye gaze tracker under natural

head motion

Diego Torricelli , Silvia Conforto, Maurizio Schmid, Tommaso DAlessio

Department of Applied Electronics, University Roma TRE, Italy

a r t i c l e i n f o a b s t r a c t

Article history: A novel approach to view-based eye gaze tracking for human computer interface (HCI) is
Received 16 September 2007 presented. The proposed method combines different techniques to address the problems of
Received in revised form 6 June 2008 head motion, illumination and usability in the framework of low cost applications. Feature
Accepted 8 June 2008 detection and tracking algorithms have been designed to obtain an automatic setup and
strengthen the robustness to light conditions. An extensive analysis of neural solutions
Keywords: has been performed to deal with the non-linearity associated with gaze mapping under
Eye gaze tracking free-head conditions. No specic hardware, such as infrared illumination or high-resolution
Human machine interaction cameras, is needed, rather a simple commercial webcam working in visible light spectrum
Neural networks sufces. The system is able to classify the gaze direction of the user over a 15-zone graphical
Eye detection interface, with a success rate of 95% and a global accuracy of around 2 , comparable with
the vast majority of existing remote gaze trackers.
2008 Elsevier Ireland Ltd. All rights reserved.

1. Introduction The usability requirements that an ideal EGT system

should satisfy to be applied as an assistive technology device
Eye gaze tracking (EGT) estimates the direction of gaze of a per- in the interaction with general computer interfaces [11] are
son [1]. In biomedicine, estimating the direction of gaze is of listed below and will focus our attention during the discussion
utmost importance not only in diagnostic contexts, but also in of this paper:
the framework of assistive technology to facilitate interaction
with interfaces for people with disabilities [2]. To accomplish tracking accuracy and reliability;
this task, different approaches have been proposed over the robustness to light conditions and head movements;
last three decades [3]. Most of them have been proven as accu- non-intrusiveness (i.e. cause no harm or discomfort);
rate, and some of the techniques and systems gave rise to real-time implementation;
commercially available products [47]. minimal burden time for calibration;
To enhance the accuracy of these techniques, the most cost-effectiveness.
common solution is based on the presence of intrusive
devices, such as chin rests to restrict head motion, and some of Among all the EGT systems, remote eye gaze trackers
them need the user to wear equipment, such as ad hoc contact (REGT) represent an effective non-intrusive solution and
lenses [8], electrodes [9] or head-mounted cameras [10]. are thus becoming prevalent in the context of assistive

Disclosure: All the subjects who performed the trials gave their informed consent to the experiments, according to the Helsinki
declaration, and the local committee approved the experimental procedure.

Corresponding author at: Department of Applied Electronics, University Roma TRE, Via della Vasca Navale 84, I-00146 Roma, Italy.
Tel.: +39 06 5733 7298; fax: +39 06 5733 7026.
E-mail address: [email protected] (D. Torricelli).
0169-2607/$ see front matter 2008 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.cmpb.2008.06.008
 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678 67

technology. With these systems, no devices are in physical dened as

contact with the user. It is possible to estimate the gaze 
direction by tracking some typical visual features of the sx = a0 + a1 x + a2 y + a3 xy + a4 x2 + a5 y2
(1)
eye. sy = b0 + b1 x + b2 y + b3 xy + b4 x2 + b5 y2
Starting from an image sequence (acquired either by a sin-
gle camera or by a set of multiple cameras), a preliminary where (sx , sy ) are the screen coordinates, and (x, y) are the
task has to be accomplished, that is the detection of face and pupil-glint vector components. A calibration procedure is thus
eyes within the image frames. Some local features of the eyes needed to estimate the 12 unknown variables {a0 , . . ., a5 , b0 , . . .,
are then needed to estimate the position of the pupil with b5 }. To perform the calibration, in Ref. [13] the user is requested
respect to the eye socket. Based on this information, a map- to look at nine points on the screen. Since each calibration
ping between the eye conguration and the direction of gaze point denes two equations, the system is over constrained
can be computed. with 12 unknowns and 18 equations, and can be solved, e.g.
Following the aim of providing an effective solution to through least squares analysis.
this issue, the proposed work presents a novel approach The accuracy of the aforementioned IR-based methods
to view-based REGT systems, which is based on neural corresponds to a standard deviation of about 11.5 , which rep-
computing and commercially available devices. The system, resents a good gure for the requirements of the vast majority
developed to target people with disabilities, was tested in of the interactive applications. Moreover, the calibration is rel-
terms of robustness to head motion and changes in light atively fast and easy, and the required computational time
conditions. allows real-time interaction. Thus, IR-based REGT systems are
The following section describes previous REGT research. prevalent in both research and commercial environment. Nev-
In Section 3 the proposed method will be introduced and ertheless, besides the high cost, several important issues have
described in detail. In Section 4 the experimental setup and yet to be solved, and at present hinder IR-based REGT systems
testing will be presented. Finally, the experimental results will usability:
be discussed in Sections 5 and 6.

Changes in light conditions, especially in presence of

2. Background sun light, which interferes with the infrared illumina-
tion, generating other kinds of reective phenomena
Two classes of approaches exist among the REGT systems: [1,3]. Some solutions have been proposed for this issue
one widespread and effective solution is based on active [19].
infrared (IR) illumination [1215]; the second relies on the anal- Bright eye effect variability in different subjects. This effect
ysis of videos capturing eye movements under natural light has been veried as preventing IR-based techniques from
conditions, and is commonly referred to as view-based or providing consistent and reliable results in gaze direction
appearance-based [1618]. Both approaches are described in estimation [4].
the following.
Moreover, there is agreement that EGT systems may nd
2.1. Infrared-based REGT use as future computer input devices only if they become con-
venient and inexpensive [22]. This is one of the reasons why a
The IR-based techniques utilize active illumination from different class of REGT systems has become more frequent in
infrared light sources to enhance the contrast between the research literature to meet these requirements, i.e. view-based
pupil and the iris. The light beam produces two effects on REGT. These will be described in the following.
the eye. The rst one, called bright-eye effect, is similar
to the red-eye effect in photography: the camera records a 2.2. View-based REGT
bright circle, which is the light passing through the pupil
and reected by the retina. The second effect is the reec- In the classical view-based approaches, intensity images of the
tion of the light on the corneal surface (the external surface eyes are grabbed by traditional image acquisition devices and
of the eye), seen by the camera as a small bright point then processed to extract information on the eye congura-
called glint or corneal reection. Assuming that the eye tion. No ad hoc hardware is usually needed. On the other hand,
is a sphere that rotates around its centre, the glint does more challenging efforts are required in terms of image pro-
not move with the eye rotation, whereas the pupil does. cessing, as compared to the IR-based REGT systems, in order
For this reason the glint can be considered as a reference to detect face and eyes.
point. In the work by Baluja and Pomerleau [16] no explicit
After grabbing the eye image, pupil and glint can features are required. Each pixel of the image is consid-
be detected by appropriate image processing algorithms ered as an input parameter to the mapping function. A
[4,19]. Gaze detection is then computed by applying a 15 40 pixels image of the eye is processed, corresponding to
mapping function that, starting from the 2D pupil-glint vec- a 600-dimension input space vector. An articial neural net-
tor, calculates the point observed on the screen. To this work (ANN), trained to model the relationship between the
latter aim, different mapping functions have been pro- pixel values and the 2D coordinates of the observed point on
posed [1,20,21]. Among them, the most commonly used the screen, is used as the mapping function. In the calibration
[13] is based on a second order polynomial function procedure, the user is requested to look at a cursor moving
68 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678

on the screen along a known path made of 2000 positions. An the head. Hence, they proposed a different set of eye features:
accuracy of 1.5 is reported. beside the pupil-glint vector, other four factors are chosen for
Similar to the previous work, Xu et al. [18] present a neu- the gaze calibration to get the mapping function. A generalized
ral technique as well. The image of the eye is segmented to regression neural network (GRNN) is then used to accomplish
precisely locate the eye and then processed in terms of his- the task, by using a normalized input vector comprising the
togram normalization to enhance the contrast between the six factors. The GRNN can classify one of eight screen regions
eye features. As in Baluja and Pomerleaus method, the ANN with a success of 95%.
receives 600-image pixel values and returns a vector of pos- Solving the issue related to head motion is thus common to
sible (x, y) coordinates of the estimated gaze point with an both IR-based and view-based REGT systems: if it is likely that
accuracy of around 1.5 . Three thousand examples are used this problem is close to the solution for the rst ones, work
for the training procedure. still needs to be done in view-based systems. Even if Baluja
Tan et al. [17] proposed a method that considers the and Pomerleaus technique allows for small head movements,
image as a point in a multi-dimensional space, by using an the only approach specically addressed to solve this issue
appearance-manifold technique: an image of 20 20 pixels has been recently investigated by the authors of the present
can be considered as a 400-component vector in a 400- work [28]. In the cited work, a multilayer perceptron has been
dimensional space. In this work, 252 images from three trained to account for little head movements that occur when
different subjects were taken, using each one as the test and the user is asked to maintain the head still and look to a cur-
the others as the manifold. The calibration is done by look- sor on the screen. The encouraging results persuaded us to
ing at markers on a screen, while taking the picture of the eye deepen the eld of the neural approach for robust gaze track-
for each position. The reported accuracy, as calculated by the ing within less constrained conditions.
leave-one-out approach, is very good (0.38 ).
Zhu and Yang [21] propose a method for gaze estimation 2.4. Design remarks/considerations
based on feature extraction from an intensity image. Both
the irises and the inner corners of the eye are extracted and The presented eye-gaze tracker has been designed to meet
tracked with sub-pixel accuracy. Then, through a linear map- most of the requirements mentioned as of utmost importance
ping function, they calculate the gaze angle, using only two in Section 1. In particular, the design has been focused on the
points for the calibration. The reported accuracy is 1.4 . robustness to light conditions and head pose. A procedure for
an automatic initialization is also proposed. In the following
2.3. Head motion section the method will be presented and described in detail.

Compensating for head motion still represents the greatest

limitation for most of the remote eye gaze trackers [23]. In 3. System description
principle, as the head moves from its original position, a new
calibration is needed. Thus, a strong requirement for the sys- As displayed in Fig. 1, the procedure for the estimation of
tem to work properly is to maintain the head perfectly still. the gaze is composed of two blocks: the rst one makes use
At the same time, this requirement is by far very restrictive of image processing algorithms to extract and track the fea-
for most of the applications. Several researchers over the last tures of the eyes; the second block accomplishes the task
years have dedicated their efforts to the solution of this issue, of mapping the geometric visual information with the gaze
with the goal of allowing the user to move the head freely, yet direction, by using articial neural networks. Merging these
maintaining the accuracy in an acceptable range. two techniques has a double purpose: on the one hand to
To compensate for head movements, most of the related both guarantee robustness to light changes and minimize
methods are based on a 3D model of the eye [12,15,2426]. In user intervention in the init process. On the other hand the
such methods, two or more cameras are used to estimate the neural approach has the goal of account for the high non-
3D position of the eye. Through the use of mathematical trans- linearity of the mapping function. To the authors knowledge
formations, the line of gaze is then computed. In some of these
systems [13,24], pan and tilt cameras with zoom lenses are
required to follow the eye during head movements. An original
approach proposed by Yoo and Chung [15] uses a multiple-light
source that gives rise to a calibration-free method. Accurate
results are also obtained by Park [27], which uses a three-
camera system to compensate for head motion.
Even if these approaches seem to solve the problem of head
motion, the complexity of the proposed systems, driven by
the need of additional hardware, prevents them from being
routinely used in a home environment. To accommodate this Fig. 1 General scheme of a procedure for a view-based
need, Ji and Zhu proposed a fairly simple method to com- REGT system. The dashed arrows indicate the classical
pensate for head motion, based on articial neural networks view-based techniques. The solid arrow shows the
(ANN), and a single IR-based camera [20]. They criticize the proposed approach, where feature detection is combined
linear/quadratic mapping functions because they cannot take with neural mapping to make the system robust to light
into account the perspective projection and the orientation of changes and head movements.
 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678 69

Fig. 2 The initialization procedure. Three consecutive steps have been adopted and integrated in the initialization
procedure: blink detection, iris detection and corner detection. This procedure allows the system to extract the feature
templates to be used by the subsequent tracking procedure.

this approach has never been presented for view-based REGT into account, to allow for redundancy in the mapping function,
systems before. thus providing a set of six eye features.
In the next subsection, the rst portion of the proposed Starting from an image sequence, three steps are per-
REGT procedure will be described and denoted as feature formed, as displayed in Fig. 2. With a preliminary blink
detection and tracking. Then we will focus on the mapping detection procedure, eye locations are estimated through
section, which will be denoted as neural mapping. image differencing. Irises are then detected with the use of
a modied Hough transform algorithm [29]. At last, a corner
3.1. Feature detection and tracking detection block extracts the corner positions by analysing and
segmenting the eye-blobs resulting from the blink detection
To gather information on the conguration of the eyes, two module. This procedure permits to automatically initiate the
kinds of features are needed, i.e. the pupil and one or more tracking process, based on template matching.
reference points on the face. Since no infrared light sources
have been used, such reference points could not be repre- 3.1.1. Blink detection
sented by corneal reections. As it will be claried later, two This block is mainly based on the analysis of eyelid move-
reference points per-eye have been considered, i.e. the inter- ments, in order to identify the shape of the eye as the area of
nal and external corners of the eye socket. Both eyes are taken the image where high changes of gray level intensity occur.

Fig. 3 Blink detection. The image difference between two consecutive frames is followed by a gray-to-black and white
conversion to detect the presence of a blink.
70 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678

The user is just requested to blink in front of the camera. The edges is detected within the eye image, making the discrim-
algorithm then performs both a frame differencing between ination of the correct iris edge very difcult. With the Sobel
couples of subsequent frames, and a gray-to-binary conver- operator a lower number of edges is detected. Nevertheless,
sion. The resulting binary images (see Fig. 3) show the zones thank to its high contrast, the iris edge always belongs to
with high changes in pixel value (blobs in the following). the detected borders, making it possible to automatically
A ve-step sequence is then carried out to verify that each choose the correct iris border with Hough transform tech-
blob actually represents the eyes: nique, as explained in the following.
Modied Hough transform. The Hough transform is a fea-
(i) Erode and dilate to account for fragmentation of the eye ture extraction technique aimed at nding a certain class
blobs. of shapes, such as lines, circles and ellipses, within an
(ii) Compute the number of clusters: only the images with image [29]. In the case of circumferences, for a given value
two clusters can be considered as eye-candidates. of radius, the algorithm will nd the best circumference
(iii) Calculate the ratio between the size of the two clus- candidates. A voting number is assigned to each candi-
ters and the distance between them to account for the date, representing the number of pixels of the image that
anatomy of the face, and to make the technique indepen- belongs to that circumference. Since the exact value of
dent from the camera-user distance. the radius is unknown, the algorithm has been applied
(iv) Calculate the inclination of the line passing through the iteratively for different radius values around a rst guess
centroids of the clusters. A 30 range is allowed to value obtained automatically by dividing the inter-eyes dis-
account for a natural head pose. tance by a constant factor (coming from morphological
(v) Delete false positives. It is a post-processing step that observations). To estimate the correct pair of circumfer-
aims at removing such images that contain estimated ences from the set of candidates, a modication of the
eyes with a size too different from the size of the eye algorithm has been applied. Each group of similar circum-
candidates present in the other images. ferences, in terms of radius and centre position, is merged
into a new circumference and a new vote is assigned to
This sequence of steps will lead to the denition of a set of it, coming from a weighted sum of the single ones. The
binary images containing the shapes of the eyes. This set of most voted circumferences of each eye are then com-
images will represent the basis of the next two blocks, i.e. iris pared with the candidates of the other eye to choose the
and corner detection. pair of circumferences with the biggest total vote (Fig. 4).
This process has shown a good behaviour over different
3.1.2. Iris detection light conditions even with asymmetric illumination of the
Once each eye has been detected from the previous step, the face.
gray intensity images of the eyes are extracted from the last
frame of the initialization video and then processed by a two 3.1.3. Corner detection
step algorithm, constituted of: To determine the eye corners, a further processing of the
results of the procedure of blink detection is needed. As men-
Edge detection: Based on the observation that the iris and the tioned, the frame-to-frame difference images highlight the
sclera present high contrast in brightness, the edge detec- zones in which a movement occurs, based on the fact that dur-
tion can easily serve as a detector of the iris. Different kinds ing a movement the corresponding pixel of the image changes
of edge detectors have been evaluated. Among different its value. We are interested in getting the shape of the eye
image frame operators (see e.g. [30,31]) the Sobel operator as accurately as possible by separating the zones with large
accomplished the task with a higher specicity in terms of inter-frame motion (eyelids) from those with small one. To
border detection: even if Canny operator results to be in gen- choose the appropriate threshold value in an automatic way,
eral more robust to light changes, a very high number of the difference images, each one with its absolute values, are

Fig. 4 Iris detection. The Sobel operator is applied to detect the edges in the image (a) and a modied Hough transform
algorithm is then used to choose the two most voted circumferences among the candidates (b).
 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678 71

Fig. 5 Inner corner detection. From the knowledge of the position of the irises, a search window is created. Then the blobs
coming from the blink detection block are ltered to detect the inner extreme of the eyelid movements, i.e. the inner corner
of the eye.

summed. The resulting matrix is then ltered to determine tracking procedure, which is based on a hierarchical procedure
the eye corner, as explained in the following. combining template matching [32] and sub-pixel approxima-
For the inner corner, a search window is generated, not tion [33].
including the iris. Within the window, the mean value and
the standard deviation of the values of the image are used to 3.2. Neural mapping
dene the threshold for the image binary process. As shown in
Fig. 5 the most lateral pixel of the binary image is considered The second part of the REGT procedure is based on the use
as the estimated inner corner. of articial neural networks designed to map the relation
For the external corner a search window is created over the between the eye features and the gaze direction. The neu-
external area of the eye, starting from the estimated iris posi- ral approach has been chosen for the ability of the nets to
tion. Ten-level quantization is then applied to the intensity learn by examples, to smoothly adapt to changing contexts
image. By eliminating the brighter levels, the line of the upper and to generalize for unseen congurations. At the same
eyelid can be easily identied. The external extremity of this time, due to the underlying complexity, the neural architec-
shape will be considered as the external corner (see Fig. 6). In ture requires an accurate design that regards the choice of
the experimental testing section, the results of this technique, the typology of the input, and the selection of the internal
over different light conditions and distance of the user from structure of the net, i.e. the number of hidden layers and
the camera, will be shown. neurons. The proposed approach arises from the belief that,
once provided with a proper set of input space vectors and
3.1.4. Feature tracking training examples, the net will be able to generalize for differ-
The initialization process above discussed has the aim of auto- ent head positions. In the next paragraphs the chosen input
matically detecting the features of the eyes, thus avoiding set, the different nets and the training procedures will be
the presence of manual selection that could compromise the detailed.
accuracy, repeatability, speed and convenience of the setup.
The detected features represent the starting point for the 3.2.1. Input space selection
The choice of the input set has the role of giving the net the
appropriate information about the relative position of the eye
within the eye-socket like on the pose of the head with respect
to the screen. This information is combined by the net to build
a non-linear regression function that takes into account those
head movements that can occur during the visual task. Basi-
cally two features are needed: one refers to the pupil, and the
other one, called reference point, to appropriately describe the
movement of the head. The latter has been identied as the
corner of the eye.
As opposed to what happens with infrared-based tech-
niques, the image of the eye in visible light spectrum does
not permit to identify a reference point on the surface of the
eye. Whereas the glint of IR-based systems is pretty insen-
sitive to the rotation of the eyes around its optic axis, the
vector connecting the eye corner with the pupil turns out to
be extremely sensitive to small head rotations. For this reason,
and for the unavoidable uncertainty of feature tracking with
Fig. 6 External corner detection. The gray level image of low-resolution images, a redundancy has been introduced by
the eyes is enhanced using histogram equalization. A considering, for each eye, two vectors connecting, respectively,
binary image (containing the white areas) is created by the pupil with the inner and the external corner. The resulting
eliminating the pixels with higher luminance. The external vector of inputs thus consists of 12 parameters: eight of them
extremes are then taken as the external corners of the eyes. come from the magnitudes and angles of the distance vec-
72 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678

Fig. 7 Geometric features of the eyes (numbered from 1 to

6). Magnitudes and angles of the distance vectors represent
8 of the 12 inputs. The remaining 4 inputs are the x and y
coordinates of the external corners of the eyelids. For the
sake of clarity, only one of the angle features is represented.

tors previously mentioned; the remaining four consist of the x

and y coordinates of the two external corners. All of them are
depicted in Fig. 7.

3.2.2. The neural structure

Two different architectures of neural networks have been used
to explore and approximate the mapping properties of the
gaze function. The rst net, a multilayer perceptron (MLP)
[34], is considered to have a strong ability to provide compact
and robust representation of mapping in real-world problems. Fig. 9 The training procedure. The upper panel represents
The other net is represented by a general regression neu- the screen with the 15 positions that the user is asked to
ral network (GRNN) [35], which is recognized to have better look at during one trial. For each position, 5 video frames
performance in function approximation tasks with respect to are processed and analyzed to build up a 75 columns (5x15)
traditional neural networks. A GRNN has also been used in one matrix, each column representing a 12 parameters vector.
previous work [14], based on infrared-based techniques. Both The whole training matrix is represented in the lower
networks have a 12-neuron input layer receiving the vector of panel.
the eye parameters. Each architecture consists of two distinct
nets that separately calculate the xp and yp coordinates of the
screen point.
The MLP design aims at nding the optimum conguration eter called spread that determines the width of an area in
with the following variables: number of hidden layers, number the input space to which each neuron responds: the higher
of neurons, number and typology of training trials and number the spread, the smoother the function. The choice of the
of training epochs. To evaluate the best conguration, accu- best conguration lies basically in the optimal estimation of
racy is taken as the performance index. The structures of the the spread, which constitutes the only user-dened internal
chosen MLP are displayed in Fig. 8. parameter.
The GRNN, based on the radial basis architecture, con-
sists of an input layer, a hidden layer, a summation layer and 3.2.3. The training procedure
an output layer (see Fig. 8). The hidden layer has as many The input set is composed of examples from different tri-
neurons as there are input/target vectors, while the num- als with changes in head position. For each trial the subject
ber of nodes of the summation layer equals the number of is requested to look at a white cursor moving on the screen
output neurons plus one. The hidden layer contains a param- over 15 different positions. During the task, the webcam grabs
seven frames for each observed position. Each frame will be
then processed to extract the eye features. During the task
the subject is requested to maintain the head in a com-
fortable position, so that natural movements are allowed.
Among the seven frames grabbed during each position, ve
are taken to build up the training set, while the remaining
two are used to validate the training procedure. As shown
in Fig. 9, the resulting training set is then constituted of 75
(15 positions, 5 frames per position) vectors for each video
trial.
The number of positions covered by the cursor has been set
Fig. 8 The multilayer perceptron (MLP, left) and the general at 15 for the following reasons:
regression network (GRNN, right), both with a 12 input layer
and a single output neuron for the calculation of either the (1) HCI interfaces used for people with disability usually
x or the y coordinate of the observed point on the screen. contain a reduced number of icons of relatively large
 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678 73

4. Experimental testing

The experimental tests have been carried out to evaluate the

accuracy and robustness of the method under conditions that
mimic a realistic context of use.
This section has been organized as follows:

Testing of the algorithms of feature detection and tracking

over different subjects, light conditions, and distance to the
camera.
Analysis of different neural congurations: The MLP and the
GRNN have been tested by varying the number of neurons
and layers in order to set the congurations with the best
performance.
Fig. 10 Head movements. Dashed line: range of head Analysis of the accuracy and robustness of the global sys-
movement in the training phase. Dotted line: head motion tem.
during the test trials. Performance comparison with other methods applicable to
the view-based context. In particular a polynomial map-
ping function [13] has been implemented and tested and
the results compared to the neural solution.
dimensions, which can roughly be approximated by divid-
ing the screen in 815 areas. The overall hardware composing the system includes
(2) A high number of positions leads to time consuming train- a commercial webcam (QuickCam Communicate STXTM ,
ing procedures; increasing the number of areas would Logitech , 30 fps, 640 480 pixels), a personal computer with
unnecessarily increase the burden time for calibration. a Pentium-4 3.0 GHz processor, and a 17 in. monitor. The web-
cam has been located over the central zone on the screen, to
To give the ANN the possibility to generalize, a wide range minimize the errors coming from perspective distortions.
of head displacements is needed. Five head positions have
been chosen for this purpose: one in central position, and four 4.1. Feature detection and tracking
in upper-down and left-right positions, as shown in Fig. 10.
The input set has been provided together with the desired A set of separate experiments has been dedicated to validate
outputs, represented by the x and y coordinates of the centre the initialization technique. Five subjects with age ranging
of the cursor. A resilient back propagation training algorithm from 22 to 57 years have been analyzed under different con-
(RBP) has been chosen. At each epoch of training two kinds ditions of illumination and distance to the camera. For each
of errors have been evaluated and compared each other. The subject ve videos have been analyzed (see Fig. 11): one very
rst one is internally calculated by the RBP algorithm at each close to the camera (25 cm), one pretty far (80 cm) and three
iteration, and the second one, the validation error, is obtained videos at a normal distance (40 cm) with different conditions
by using the validation set (two frames per position) as input. of illumination, high (900 lx), low (250 lx) and asymmetrical,
The end of the training is established by comparing these thus resulting in a total number of 25 trials. During the trial
two values with predened thresholds, ranging between 104 the subject is requested to blink three times while looking at
and 103 . the camera.

Fig. 11 Illumination and distance modications. Five different combinations of illumination and distance from the camera
have been tested for the algorithm. Below each picture, the histogram representation of the luminance distribution is
depicted.
74 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678

Table 1 Feature detection: assigned performance

values corresponding to the distance between the
automatically and manually detected feature (for a
80 40 pixels eye image)
Distance between Performance Correctness
manual and automatic value value (%)
detection

01 (pixels) 4 100
23 3 90
46 2 50
79 1 20
>9 0 0

The accuracy has been evaluated in terms of percentage of

correct estimation: for each video, a value ranging from 0 to
4 was assigned to each estimated position, based on the dis-
tance between the estimated position and the one manually
determined by one independent researcher. As an example,
considering a 80 40 eye image, the relation between perfor-
mance values and distance ranges is depicted in Table 1. The
percentage of correct recognition is reported in Table 2. As
illustrated, the method shows a good performance in non-
extreme conditions, even for high and low illumination. A
strong lateral illumination makes the technique fail in the
corner identication.

4.2. Neural congurations

In order to detect the most appropriate structure, multiple

congurations of MLP and GRNN have been tested. In this part
of the study the performance of each net has been evaluated
in terms of the ability to classify the gaze direction over the
Fig. 12 MLP classication performance as a function of the
15 zones on the screen, expressed by the percentage of cor-
number of neurons of the net across the different number
rectly estimated zones. More than 50 congurations of MLP
of hidden layers.
have been tested, with one, two and three hidden layers and
a total number of neurons ranging from 5 to 420.
The experimental protocol has been designed as follows:
for each trial session the user is asked (1) to seat in front of trials with the head approximately centred with respect to the
the screen, (2) to execute a repetition of three blinks to permit trained volume, and slight movements allowed (Fig. 10). In par-
the system to automatically detect the eye features, and then ticular, the head translation lies within an area of 3 3 cm in
(3) to perform the visual task looking at the cursor moving on x and y directions, and the variation along z direction is of
the 15-position path. At the end of the trial the subject is asked 1 cm.
to stand up and move away from the seat, and then go back As shown in Fig. 12, the mean performance does not vary
again in front of the screen to start a new trial. During the ses- signicantly with the number of neurons. A slight improve-
sion the subject is free to move the head in a natural way, still ment (23%) occurs in the case of two and three hidden layers.
avoiding sudden and/or large movements, at a distance from Among the overall set, some nets reach higher recognition per-
the screen of approximately 46 cm (corresponding to a dis- centages. In particular, for the calculation of the x coordinates,
tance from the camera of 43 cm). The test set includes eight some congurations achieve a performance of 9698%, while

Table 2 Performance of the feature detection algorithm

Percentage of successful detections

Blink (%) Iris (%) Inner corner (%) External corner (%)

Close (25 cm) 100 100 90 100

Far (80 cm) 100 80 100 80
Medium distance (40 cm) (high lighting) 100 90 100 80
Medium distance (40 cm) (low lighting) 100 90 100 100
Medium distance (cm) (asymmetric lighting) 100 90 30 60
 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678 75

According to the obtained results we can state that the MLP

structure gives more accurate results for a given distance to
the camera, while the GRNN is more robust than MLP even for
changes in the z direction, still maintaining the percentage of
correct recognition at a high level. For this reason we consider
the GRNN structure more suitable for free head conditions.
In the following, an analysis of the global accuracy and
robustness of the GRNN will be presented. The accuracy has
been calculated in terms of mean and standard deviation of
the gaze error, i.e. the error between the real observed position
and the estimated values, expressed in terms of pixel (epxl ) and
angular degrees (edegree ), according to the following equation:
Fig. 13 GRNN classication performance as a function of
the spread parameter.
epxl
edegree = arctan
dpxl

Table 3 Performance of the neural networks for with dpxl representing the distance between the subject and
different distances from the camera the screen plane expressed in pixels. As in the previous para-
Percentage of zone recognition zones graph, the percentage of zone recognition will be used as a
measure of the accuracy of the system.
X (%)a Y (%)a X (%)b Y (%)b Two kinds of results are reported, rst considering each trial
MLP 45 40 39 55 separately to determine the robustness to head movements
GRNN 100 100 80 60 (Table 4), then by reporting the accuracy on each zone over the
different trials (Table 5 and Fig. 14) to highlight the distribution
a
Far (53 cm). of the accuracy over the screen.
b
Near (41 cm).
The estimated mean accuracy is approximately 1.6 on
x direction and 2.6 on y direction with a standard devia-
for the y coordinate the best score (96.7%) is reached with a tion, respectively, 1.4 and 1.9 , leading to a rate of successful
three-layer conguration. The GRNN has been tested for dif- recognition of 94.7%. Exceptions have occurred in three zones,
ferent values of the spread. As Fig. 13 shows, the optimum where the performance decreases to values under 90%.
value of the spread has been set at 0.3 for the x and 0.33 for the
y, leading to a performance of, respectively, 96.7% and 93.3% 4.3. Performance comparison
of zone recognition performance.
The nets above selected have been tested again, loosen- In this section we have compared the performance of the
ing some of the constraints to the head motion, in particular neural approach with a polynomial mapping function that
concerning the distance to the camera. The test set is in this has been often used in the context of gaze tracking based on
second situation composed of two head positions, one far, at geometric feature relations. In particular, Eq. (1) used in the
53 cm from the screen and the other closer, at 41 cm from infrared-based systems [13] has been implemented.
the camera. The different MLP congurations have shown a As mentioned in Section 2.1, the pupil-glint vector used in
low capability of calculating the gaze correctly for these two the infrared-based techniques is not reproducible in the view-
distances, while the GRNN has shown a very good perfor- based approaches, and thus a new vector has been considered
mance for the higher distance. In the case of close distance, as a proxy of the pupil-glint: it is obtained by connecting the
some errors occur especially for the y coordinate estimation pupil with the midpoint of the segment connecting the two
(Table 3). corners.

Table 4 Accuracy of the GRNN for nine different test trials

Trial Distance (cm) Mean value standard deviation ( , pixels) Percentage of zone recognition

X Y X Y

1 48 1.6 1.2, 36 26 3.3 2.1, 78 46 100 80

2 47 1.4 0.9, 32 21 1.2 0.8, 31 20 100 100
3 46 2.2 1.3, 47 27 4.1 3.0, 91 63 93.3 66.7
4 46,5 1.7 1.2, 38 27 1.3 1.1, 32 26 100 100
5 45 1.3 1.1, 29 24 1.5 0.8, 37 19 100 100
6 45 2.0 1.8, 45 40 3.5 2.1, 80 45 93.3 93.3
7 45 1.5 1.0, 34 25 1.8 1.8, 43 42 100 100
8 45 1.9 1.3, 61 28 2.4 1.5, 59 33 86.6 100
9 53 1.2 1.3, 29 30 2.8 1.2, 66 26 100 100
Mean 1.7 1.2 2.4 1.6 97 93.3
76 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678

Table 5 Accuracy of the GRNN on the 15 zones

Zone Mean value standard Zone
deviation ( ) recognition (%)

X Y

1 1.8 1.8 2.1 1.3 93.3

2 0.7 3.1 3.4 2.2 93.3
3 1.0 1.5 2.5 1.0 100
4 1.4 1.7 1.8 1.3 100
5 1.7 1.0 1.2 1.6 100
6 2.8 0.7 3.5 3.1 93.3
7 0.2 2.0 2.6 3.1 93.3
8 0.4 1.1 2.8 2.4 93.3
9 0.3 2.0 2.5 2.7 93.3
10 4.2 1.5 2.3 2.8 80
11 3.9 1.4 3.5 1.8 100
12 0.3 2.5 2.7 1.8 100
13 0.3 1.9 2.8 2.3 100
14 0.1 1.9 4.0 2.5 93.3
15 2.7 1.6 5.9 3.1 86.7

Mean 1.4 1.7 2.9 2.2 94.7

The quadratic function takes as inputs the two compo-

nents of the vector returning the coordinates of the screen Fig. 15 Comparison to the quadratic mapping.: mean
point. A 15-point calibration and a least square solution have values (histograms) and standard deviations of the error in
been used to solve the over-constrained problem, as pro- degrees for the different trials.
posed by Morimoto et al. [13]. The results depicted in Fig. 15
demonstrate the better performance of the neural approach as
compared to the polynomial interpolation method in terms of 5. Discussion
mean values and standard deviations, showing an accuracy of
1.7 1.2 for the x direction and 2.4 1.6 for the y direction, The proposed REGT system shows reliable and accurate global
while the accuracy of the quadratic function, respectively, of results. The uncertainty of gaze estimation has been proven
2.9 1.9 and 3.6 1.9 for x and y directions. to come from two main factors. The rst one refers to the eye

Fig. 14 Accuracy evaluation. White squares represent the observed positions. The crosses correspond to the estimated
values (1 for each test trial). The small circles represent mean values among the trials, while the ellipses stand for the
standard deviations in x and y direction.
 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678 77

features tracking: in some extreme cases, if the gaze is directed The core of the proposed approach overcomes most of the
towards the very lowest part of the screen and off the centre, issues induced by head movement, that is one of the most seri-
the iris tracking algorithm does not achieve a high accuracy, ous problems in REGT systems. Two neural network structures
due to occlusions from the eyelids and signicant changes in have been designed to learn the gaze direction from the user,
the iris shape, so that the template matching is not as accurate under natural head motion: head shift was performed during
as for the intermediate positions. This is what happens for the the training sessions and a test session with head motion in
zones 14 and 15 (low right in the screen, see Fig. 14) where the 3D has been performed. The aim of the testing was to test the
estimation accuracy increases to higher, yet acceptable, values system under natural head movements during a visual task,
of error. maintaining a comfortable posture.
The second source of inaccuracy is due to the nature of The obtained results (global average accuracy of around 2
the input set, which might describe a non-predictable dis- and classication rate of 95% over 15 zones) conrm the relia-
tribution over the 12-dimensional space in presence of head bility and robustness of the proposed system, in which the
movements. The neural mapping seems to represent a proper neural mapping has been proven to outperform traditional
solution for this problem, overcoming the high non-linearity of quadratic approaches.
headeye movements. The GRNN structure has been proven to The proposed system makes use of simple, low cost and
be more effective than MLP in compensating for head motion commercially available hardware, since no further device than
along the z direction, while the MLP network seems to achieve a webcam is needed.
higher values of accuracy for distances very similar to the Future efforts will be devoted to develop new strategies to
ones present in the training set. Since the system is aimed address the issue of larger head movements. In this context
at compensating for movement in a 3D space, the GRNN has a convincing solution can be represented by a combination
been considered the most appropriate solution. The neural of a decision tree approach with a neural network design, to
mapping outperforms the quadratic mapping function, as one explore and exploit the classication and approximation abil-
would expect by considering the possibility of having the user ities of the neural systems in the direction of making remote
look at the same point on the screen by having different eye eye gaze trackers effective solutions both in the assistive tech-
feature congurations with different head positions. nology framework and in rehabilitation contexts.
The system has also been proven to be robust to light
changes. By using an initialization procedure based on the
Conicts of interest
movements of the eyelids during blinking, the feature extrac-
tion and tracking is almost independent from the presence of
The authors allege that no nancial or personal relationships
light changes, both in intensity and direction. Moreover, this
with any other people and organisation have inappropriately
preliminary procedure attains to automatically initialize the
inuenced the work here submitted.
procedure, avoiding an external user to select the features to
track.
The estimated accuracy of the system is pretty good, yield- Acknowledgements
ing values comparable with most of view-based REGT ones,
except for the work of Xu et al. [18], where the accuracy is, how- This research activity has been partially funded by the Italian
ever, calculated through the leave-one-out procedure, whose Ministry of University and Research (MiUR). The help of the
use is controversial, since it usually underestimates the true two anonymous reviewers is greatly acknowledged. A special
prediction error [36]. thanks to Roberto Moschini and Saba Gervasio (and family) for
As for the classication performance, results are in favour their technical help and suggestions.
of a very high classication rate under natural head motion
with a 15-zone user interface environment, which represents references
a valuable trade-off between usability and timing cost.
The ve-trial calibration procedure, necessary to let the
system learn and correctly estimate the gaze direction, is user [1] C.H. Morimoto, M.R.M. Mimica, Eye gaze tracking techniques
specic and requires about 8 min (5 min of visual task plus for interactive applications, Computer Vision and Image
1 min of rest between two consecutive trials): the burden time Understanding 98 (2005) 424.
is acceptable for user comfort. [2] A.T. Duchowski, A breadth-rst survey of eye tracking
applications, Behavior Research Methods, Instruments, and
Computers 34 (2002) 455470.
[3] D.W. Hansen, A.E.C. Pece, Eye tracking in the wild, Computer
Vision and Image Understanding 98 (2005) 155181.
6. Conclusions and future goals [4] T. Hutchinson, K.J. White, K. Reichert, L. Frey,
Humancomputer interaction using eye-gaze input, IEEE
The proposed remote eye gaze tracker is based on a novel Transactions on Systems, Man, and Cybernetics 19 (1989)
approach in which eye feature extraction and tracking is com- 15271533.
[5] Tobii Technology AB, Stockholm, Sweden,
bined with neural mapping, with the purpose of improving
http://www.tobii.se.
robustness, accuracy and usability under natural conditions.
[6] SMI, SensoMotoric Instruments GmbH, Teltow, Germany,
By working on visible light spectrum and by extracting fea- http://www.smi.de.
tures based on blink detection, the method has shown not to [7] Eye Response Technology, Charlottesville, USA,
be affected by light changes. http://www.eyeresponse.com.
78 c o m p u t e r m e t h o d s a n d p r o g r a m s i n b i o m e d i c i n e 9 2 ( 2 0 0 8 ) 6678

[8] D.A. Robinson, A method of measuring eye movement using promises, in: R. Radach, J. Hyona, H. Deubel (Eds.), The
a scleral search coil in a magnetic eld, IEEE Transactions on Minds Eye: Cognitive and Applied Aspects of Eye Movement
Biomedical Engineering 10 (1963) 137145. Research, North-Holland/Elsevier, Boston, MA, 2003, pp.
[9] M.J. Coughlin, T.R. Cutmore, T.J. Hine, Automated eye 573605.
tracking system calibration using articial neural networks, [23] Z. Zhu, Q. Ji, Novel eye gaze tracking techniques under
Computer Methods and Programs in Biomedicine 76 (2004) natural head movement, IEEE Transactions on Biomedical
207220. Engineering 99 (2007) 111.
[10] L.E. Hong, M.T. Avila, I. Wonodi, R.P. McMahon, G.K. Thaker, [24] R. Newman, Y. Matsumoto, S. Rougeaux, A. Zelinsky, Real
Reliability of a portable head-mounted eye tracking time stereo tracking for head pose and gaze estimation, in:
instrument for schizophrenia research, Behavioural Proceedings of the fourth IEEE international conference on
Research Methods 37 (2005) 133138. automatic face and gesture recognition, 2000, pp. 122128.
[11] M. Adjouadi, A. Sesin, M. Ayala, M. Cabrerizo, in: K. [25] D. Beymer, M. Flickner, Eye gaze tracking using an active
Miesenberger, J. Klaus, W. Zagler, D. Burger (Eds.), Computers stereo head, in: Proceedings of the IEEE Conference on
Helping People with Special Needs, Springer, Computer Vision and Pattern Recognition, vol. II, Madison,
Berlin/Heidelberg, Germany, 2004, pp. 761769. WI, 2003, pp. 451458.
[12] S. Shih, J. Liu, A novel approach to 3d gaze tracking using [26] T. Ohno, N. Mukawa, A Free-head, Simple Calibration, Gaze
stereo cameras, IEEE Transactions on Systems, Man, and Tracking System That Enables Gaze-Based Interaction, ETRA
Cybernetics 3 (2003) 112. 2004: Eye Tracking Research and Applications Symposium,
[13] C. Morimoto, D. Koons, A. Amir, M. Flickner, Pupil detection 2004, pp. 115122.
and tracking using multiple light sources, Image and Vision [27] K.R. Park, A real-time gaze position estimation method
Computing 18 (2000) 331336. based on a 3D eye model, IEEE Transactions on Systems,
[14] Q. Ji, Z. Zhu, Non-intrusive eye and gaze tracking for natural Man and Cybernetics 37 (2007) 199212.
human computer interaction, MMI-Interaktiv 6 (2003), ISSN [28] D. Torricelli, M. Goffredo, S. Conforto, M. Schmid, T.
14397854. DAlessio, A novel neural eye gaze tracker, in: Proceedings of
[15] D.H. Yoo, M.J. Chung, A novel non-intrusive eye gaze the 2nd International Workshop on Biosignal Processing and
estimation using cross-ratio under large head motion, ClassicationBiosignals and Sensing for Human Computer
Computer Vision and Image Understanding 98 (2005) 2551. Interface (BPC 2006), 2006, pp. 8695.
[16] S. Baluja, D. Pomerleau, Non-intrusive gaze tracking using [29] T. Kawaguchi, D. Hidaka, M. Rizon, Detection of eyes from
articial neural networks, in: J.D. Cowan, G. Tesauro, J. human faces by Hough transform and separability lter, in:
Alspector (Eds.), Advances in Neural Information Processing International Conference on Image Processing 2000
Systems (NIPS), vol. 6, Morgan Kaufmann Publishers, San Proceedings, 2000, pp. 4952.
Francisco, CA, 1994, pp. 753760. [30] Parker, R. James, Algorithms for Image Processing and
[17] K.H. Tan, D.J.J. Kriegman, N. Ahuja, Appearance-based eye Computer Vision, John Wiley & Sons, Inc., New York, 1997,
gaze estimation, in: Proceedings of the IEEE Workshop on pp. 2329.
Applications of Computer Vision-WACV02, 2002, pp. [31] J. Canny, A computational approach to edge detection, IEEE
191195. Transactions on Pattern Analysis and Machine Intelligence
[18] L.Q. Xu, D. Machin, P. Sheppard, A novel approach to PAMI-8 (6) (1986) 679698.
real-time non-intrusive gaze nding, in: Proceedings of the [32] W.K. Pratt, Digital Image Processing, Wiley, New York,
9th British Computer Vision Conference-BMVC, 1998, pp. 1978.
428437. [33] M. Goffredo, M. Schmid, S. Conforto, T. DAlessio, A
[19] Y. Ebisawa, Improved video-based eye-gaze detection markerless sub-pixel motion estimation technique to
method, IEEE Transactions On Instrumentation and reconstruct kinematics and estimate the centre of mass in
Measurement 47 (1998) 948955. posturography, Medical Engineering & Physics 28 (2006)
[20] Z. Zhu, Q. Ji, Eye gaze tracking under natural head 719726.
movements, in: Proceedings of the 2005 IEEE Computer [34] S. Haykin, Neural Networks: A Comprehensive Foundation,
Society Conference on Computer Vision and Pattern Prentice Hall, 1998.
Recognition (CVPR05), 2005, pp. 918923. [35] D.F. Specht, A general regression neural network, IEEE
[21] J. Zhu, J. Yang, Subpixel eye gaze tracking, in: Proceedings of Transactions on Neural Networks 2 (1991) 568
the 5th IEEE International Conference on Automatic Face 576.
and Gesture Recognition, 2002, pp. 124129. [36] H.A. Martens, P. Dardenne, Validation and verication of
[22] R.J.K. Jacob, K.S. Karn, Eye tracking in human-computer regression in small data sets, Chemometrics and Intelligent
interaction and usability research: ready to deliver the Laboratory Systems 44 (1998) 99121.