Remote Gaze Tracking System on a Large Display

1. Introduction
Recently, intelligent televisions such as the Internet protocol television
(IPTV) and the smart TV have been widely popularized. The intelligent TV provides not only
conventional broadcasting but also many services such as video on demand (VOD), web surfing,
shopping, teleconferencing, and social network services (SNS) . This development has caused the
conventional remote control to become exceedingly complex because many additional buttons have
been included to perform various functions on the remote control . In addition, people must learn
what functions are performed according to the type of buttons on the remote control. This issue
adversely affects user convenience. In order to solve these problems, we propose a novel gaze
tracking system as an intelligent TV interface for a large display. Gaze tracking is a technology used
to detect where a user is looking, and gaze tracking systems have generally been divided into two
categories in previous studies. The first category is a wearable gaze tracking system, and the second
is the remote gaze tracking system. In the case of a wearable gaze tracking system, a user must wear
a device that includes an eye-tracking camera. However, by requiring a total of four cameras, their
system has the disadvantage of high cost and large size. In addition, their system requires the
additional procedure of camera calibration. However, they positioned the gaze tracking camera close
to users face (60 cm in front of the display), and it can be inconvenient to the user by hindering the
users natural line of sight. In general, gaze tracking for an intelligent TV with a large screen is
different from that for a desktop computer, and the following factors should be considered:

2. Proposed Gaze Tracking System
2.1. Overview of the Proposed Gaze Tracking System

When the proposed system starts, the users image is captured by a wide view camera
(WVC). Then, the face and the eye are detected in the captured image, and the X and Y positions of
the users eye can be obtained in the WVC image. In addition, on the basis of the detected faces
width, the Z distance between the proposed gaze tracking device and the user is estimated in the
captured image by the WVC. Then, the NVC of the proposed gaze tracking device captures the
users eye by panning, tilting, and focusing using the information of the X, Y, and Z positions of the
users eye . From this information, an enlarged and focused eye image can be obtained by the NVC,
and the pupil and four specular reflections (SRs) are detected in the image captured by the NVC. For
accurate gaze tracking in this study, user-dependent calibration is performed, which includes two
sub-steps of focus-calibration and gaze-calibration. In the first step, user-dependent information
regarding the focus is obtained. In the second step, angle kappa is calculated, which is the difference
between the pupillary and visual axes. If the user-dependent calibration stage is executed, the gaze
position of the user is calculated by using the detected pupil center and the four specular reflections
on the eye.




2.2. Proposed Gaze Tracking Device
The proposed gaze tracking device is located in front of and below the IPTV. In order
to obtain the NVC eye image, which is illuminated only by the near infrared (NIR) illuminators, a
visible light passing filter of the NVC is removed and an NIR passing filter is incorporated in the
NVC .This filter was incorporated because the pupil is not easily discriminated from the iris by
visible light in Asian people, who have dark-colored eyes. In addition, a great deal of noise from
external visible lights can be excluded in the NVC camera by the NIR passing filter and the NIR
light. Four near infrared (NIR) illuminators are attached at the four corners on the TV. The WVC
can capture the user in a wide field of view (FOV) and has an image resolution of 1,280 1,024
pixels, as shown in Figure 4a. The ranges of the viewing angles are approximately 33 on the
horizontal axis and 25 on the vertical axis. The NVC can capture an enlarged eye image with
an image resolution of 1,600 1,200 pixels by using a lens with high magnification.




Proposed gaze tracking device. (a) Panning and tilting structures. (b) Parallel structure of the NVC and
WVC.

(a) (b)
3. Proposed Gaze Tracking Algorithms
In the first step, the face and the eye are detected in the image captured by the WVC.
To detect the face and the eye positions of the user, an adaptive boosting (Ada Boost) algorithm is used
The input image of 1,280 1,024 pixels is reduced in size to 640 480 pixels in order to enhance the
face detection processing speed. With the detected face region, eye detection is performed by combining
the Ada Boost algorithm and the adaptive template matching (ATM) method. Because information of
precise eye position is required for accurate panning and tilting of the NVC, eye detection is performed in
the original input image of 1,280 1,024 pixels within the eye searching area, which is defined on the
basis of the detected face region. Similar to the procedure for face detection, if the Ada Boost method
fails to detect the eye region, ATM-based eye detection is performed. In order to reduce the change in
illumination in the NVC image, illumination compensation is performed by changing the pixel value on
the basis of the average mean value of the image.


3.2. Estimating Z Distance
After face and eye detection, Z distance estimation is performed. For higher accuracy of gaze
detection, the accurate detections of the pupil center and centers of corneal SR are required, and a focused
eye image provided by the NVC is inevitably necessary. In general, when the camera captures a
magnified image, the depth of field (DOF) of the camera lens becomes smaller. The DOF represents the Z
distance range where a focused image can be obtained. Because the NVC of our system acquires a
magnified eye image, as shown in Figure 4b, the DOF is small; consequently, the Z distance between the
camera and the user is required for accurate auto-focusing of the NVC.


3.3. Detection of Pupil and SR in NVC Images
Firstly, the image with a resolution of 1,600 1,200 pixels that is captured by the NVC is
sub-sampled down to an image with a resolution of 400 300 pixels to reduce the processing time. Then,
the rapid eye detection method is carried out. On the basis of the pupil area detected by the RED, the
pupil center and four SR centers are detected. An image that was captured by the NVC. To calculate the
pupil center, the corneal SRs around the pupil are removed. Then, we performed histogram stretching to
enhance the distinct difference in brightness between the pupil and the iris. Next, a gray morphological
operation using a 7 7 pattern is performed to smooth the removed region of the SRs. The pattern has a
circular shape similar to the pupil, and the removal of the SR, which has a bright gray color, can be
performed by iterating the procedure of erosion and dilation. Subsequently, the CED is performed, and its
center is regarded as the pupil center. The red circle is a result of the CED, and red point inside the red
circle is the pupil center. On the basis of the pupil center, the search region for detecting the SR is
defined. Within the search region, the four SRs are detected by binarization, component labeling, and
calculating the geometric center,


(a) (b) (c)


(d) (e) (f)

Steps for obtaining pupil center and four SR centers. (a) Original image. (b) Results of
removing SRs. (c) Results of histogram stretching. (d) Results of the morphological operation. (e)
Detection results of the pupil using CED. (f) Detection results of four SRs.

Conclusions

We have proposed a novel gaze tracking system for the interface of an intelligent
TV. A gaze tracking device with a WVC and an NVC was designed and operated on the basis of panning
and tilting mechanisms. The proposed system could stably track the gaze of the user at a distance from 1.8
to 2.2 m. In addition, it can be used for monitoring the users interest among the various contents
displayed or the audience rating of TV watchers. And it can be used for the intelligent interface for the
handicapped or patient, etc. Since these functionalities cannot be executed just once but are continuously
performed, especially when the user watches the TV, eye-gaze should be tracked in the successive
frames. If the gaze detection error is large, it is difficult to select from a small menu. In future work, we
would like to research a method using an additional device such as an NIR-based distance measuring
device or a high-cost laser scanner in order to measure the Z distance more accurately.