EEG SIGNAL PROCESSING FOR EYE TRACKING

Mohammad Reza Haji Samadi, Neil Cooke

Interactive Systems Engineering Research Group, University of Birmingham, U.K.

ABSTRACT 2. BACKGROUND

Head-mounted Video-Oculography (VOG) eye tracking is vi- 2.1. Eye Movement and Tracking
sually intrusive due to a camera in the peripheral view. Elec-
trooculography (EOG) eye tracking is socially intrusive be- The eye tracking problem can be defined as estimating a per-
cause of face-mounted electrodes. In this work we explore son’s focus of foveal attention [3, 4, 5] and eye movement
Electroencephalography (EEG) eye tracking from less intru- types such as saccades (large and rapid movements), smooth
sive wireless cap scalp-based electrodes. Classification algo- pursuits (slow movements), fixation (little movement) and
rithms to detect eye movement and the focus of foveal at- blinking.
tention are proposed and evaluated on data from a matched Eye tracking techniques can be classed as either Video-
dataset of VOG and 16-channel EEG. The algorithms utilise oculography (VOG) or non-optical. Electrooculography
EOG artefacts and the brain’s steady state visually evoked (EOG) [6] is a non-optical method for eye movement sensing
potential (SSVEP) response while viewing flickering stim- based on changes measured in the steady corneoretinal po-
ulus. We demonstrate improved performance by extracting tential difference. The potential changes are measured with a
features from source signals estimated by Independent Com- set of electrodes mounted around the eyes. If the eye moves
ponent Analysis (ICA) rather than the traditional band-pass from its central position, the positive pole of the eye (cornea)
preprocessed EEG channels. The work envisages eye track- moves to one of the electrodes while the negative pole (retina)
ing technologies that utilise non-facially intrusive EEG brain approaches the opposite electrode. The dipole orientation
sensing via wireless dry contact scalp based electrodes. change effects the EOG signal amplitude. Although EOG
can estimate 2◦ visual angle [5], it can be cumbersome and
Index Terms— ICA, SSVEP, VOG, eye tracking, visual uncomfortable, normally limited to laboratory experiments
attention and they be employed in daily life and mobile circumstances.
VOG records an eye image sequence with high resolu-
tion cameras. They rely on the software processing system
1. INTRODUCTION for obtaining the eye position [5, 7]. Cameras are either head
mounted or remote. VOG is more comfortable and less intru-
The prevalent method for eye tracking is a camera pointed at sive than EOG. However, remote eye trackers limit the visual
the eye - Video-Oculuography (VOG). Eye image capture is field and mobility. Head-mounted trackers require users to
inherently noisy due to the sensor technology and the unstable wear devices which potentially limit physical activity.
physical space between the camera and eye from occlusion,
movement and varying light conditions; state-of-the-art head
mounted cameras have 0.5◦ visual error and remote systems 2.2. Electroencephalography
2◦ visual error [1]. The variation of the surface potential distribution over the
Brain signal processing via Electroencephalography scalp reflects the brain’s functional activities which are
(EEG) provides two additional eye movement information recorded by placing a set of electrodes (sensors) on the scalp
sources- Oculomuscular movements via Electrooculography and measuring the voltage differences between electrode
(EOG), and the brain’s response to flickering stimuli - the pairs. The resulting data is called the Electroencephalogra-
Steady-State Visual Evoked Potential (SSVEP) [2]. In this phy (EEG) [8, 9].
work, these two information sources are combined to develop EEG signals are contaminated by artefacts rendering them
an EEG-based eye tracking solution. less usable. There are two categories of artefact- environ-
Section 2 reviews eye tracking methods and EEG. Sec- mental and biological [10]. Environmental artefacts originate
tion 3 describes the proposed EEG signal processing method. from outside of the body such as 50Hz or 60Hz power line
Section 4 reports the evaluation including details of the noise. Biological artefacts are electrical activities originate
EEG/VOG dataset collected specifically for this study. Sec- from non-cerebral origins such as muscle artefacts (facial and
tion 5 discusses future work. corporeal) and EOG.

Authorized licensed use limited to: Banasthali Vidyapith. Downloaded2030

on March 10,2021 at 07:34:46 UTC from IEEE Xplore. Restrictions apply.
 In Neuroscience and Brain-Computer Interface (BCI) and apriori knowledge of the positions of stimuli in the visual
studies, the EOG signal is an unwanted source which is field. The following signal processing stages are proposed:
normally removed to reveal ‘clean’ EEG containing neural Pre-processing: EEG channels are band-pass filtered (1
activity only. Averaging, filtering and linear regression meth- –40Hz) to remove the slow drifts and high-frequency noise
ods are commonly used to reject EOG but they typically (e.g. mains hum)
either under compensate leaving residual artefacts, or over Feature Extraction: Features are extracted from the pre-
compensate removing neural information from the signal. processed EEG channels. For eye movement type classifica-
Improvement in EEG artefact rejection is demonstrated tion (EOG-based) each channel is epoched into 400ms tem-
with the Blind Source Separation (BSS) method Indepen- poral windows with 50% overlap and two features extracted
dent Component Analysis (ICA) [11] which assumes that the per channel - the minimum and maximum amplitude. For
observed EEG signals from electrode channels are a linear SSVEP response estimation, each channel is epoched into
mixture of independent source signals (components). The 2000ms windows with 60% overlap and 2 features per stimu-
EEG signal channels are decomposed into multiple compo- lus are extracted per channel - the spectral power at the stimuli
nents with a time-invariant invertible mixing matrix which frequency (typically either 7Hz, 10Hz or 12Hz) and its sec-
is estimated by criterion to identify different sources from ond harmonic. The Power Spectral Density is estimated with
signal characteristics such as entropy or mutual information. Welch’s method [15]. The extracted features from all chan-
nels are concatenated at the end.
Independent Component Analysis: Source component
2.3. Steady-State Visual Evoked Potential signals are extracted from channels with the Blind Source
The Steady-State Visual Evoked Potential (SSVEP) is an Separation (BSS) algorithm Extended-Infomax [16] Indepen-
electrical response in the brain when the retina is stimu- dent Component Analysis (ICA). The same features are ex-
lated by a flickering visual stimulus at a frequency higher tracted from the component signals as from the pre-processed
than 6Hz [12]. SSVEP responses are an oscillation in the EEG channels.
EEG signals at the same frequency as the visual stimula- Classification: Classifiers (reported in this paper k-
tion. SSVEP can be detected reliably with minimal between- Nearest Neighbour (kNN) with Euclidean distance metric)
person variation by analysing the EEG signal’s spectral con- are trained for eye movement type (4 classes - blink, fixa-
tent. SSVEP has become a popular technique for BCI [13, 14] tion, saccade and pursuit) and to detect SSVEP (4 classes -
and enables a person’s focus of attention (FoA) to be es- 0Hz, 7Hz, 10Hz and 12Hz). The value k is set to 1 (i.e.
timated, providing the focus/stimulus is flickering. Thus, 1-N N ). Separate classifiers for features extracted from EEG
augmenting the presentation of a focus with a flickering lumi- channels and ICA components are trained and compared in
nosity enables detection of the focus of foveal visual attention the evaluation.
without the use of VOG.
4. EVALUATION
2.4. Novelty To evaluate the EEG-based Eye Tracker, 5 users’ EEG and
Most studies in EEG are motivated by neuroscience and BCI VOG is recorded under controlled cued tasks tracking stimuli
rather than eye tracking; eye movement is seen as an un- which requires fixation, saccadic and pursuit eye movements.
wanted EOG artefact. They normally use laboratory-grade Participants: The 5 healthy users have normal or corrected-
EEG devices with a high number of channels (e.g. 64), each to-normal vision and no neurological disorders. They sit in a
channel sensed by a wet-contact electrode. Scalp-based elec- comfortable chair 1m from a 1900 computer monitor. They are
trodes are normally complemented by others on the face and instructed to relax and retain as placid as possible, to avoid
chest to assist with artefact rejection. Consequently, such sys- the EEG channel contamination with electrical interference
tems are costly and not suited for real-world wearable appli- from muscles.
cations. This study captures data specifically for eye tracking Apparatus: Both EEG and VOG apparatus are non-
with a wearable consumer device with a lower number (16) intrusive and designed for mobile (wireless) situations outside
of wireless dry contact scalp-only electrodes. of laboratory settings. EEG signals are recorded from the 16
electrode wireless Emotiv EPOC EEG headset which has a
128Hz sample rate. This device has credible albeit lower per-
3. EEG EYE TRACKING SIGNAL PROCESSING formance than the higher-channel laboratory-based devices
[17] although performance degrades over time [18]. VOG is
There are two types of eye tracking information that can be captured from Tobii Eye Glasses, a head-mounted monocular
extracted from EEG channels - eye movement type from EOG eye tracker which samples at 30Hz and records visual angles
(e.g. saccades, fixations and pursuit) and eye position in re- up to 56◦ horizontally and 40◦ vertically. EEG and VOG data
lation to the visual field from the SSVEP response detection are synchronised by inserting timestamps in the VOG data.

Authorized licensed use limited to: Banasthali Vidyapith. Downloaded2031

on March 10,2021 at 07:34:46 UTC from IEEE Xplore. Restrictions apply.
 ent frequencies suitable for SSVEP detection (7Hz, 10Hz or
12Hz [19]) for another 14s. To elicit fixations the checker-
board remains still. To elicit pursuit it moves in one of 8 direc-
tions towards the screen edge. Each user performs 8 fixations
and 8 pursuit movements for each flickering frequency (total
48 movements over approximately 11 minutes.

4.2. Classifiers
Eight kNN Classifier pairs (labelled C1 − 8 in the result
Table 1) are trained on the EEG session data following the
process outline in section 3. Each pair consists one classifier
trained on EEG pre-processed channels and one classifier
trained on source components estimated by ICA. C1 and C2
classify long and short saccade directions respectively from
Fig. 1. Display to user over time (left to right). Row (a) Pos-
EOG features (8 class problem). C3 and C4 classify FoA
sible stimuli locations; (b) Elicit short saccade; (c) Elicit fix-
fixations and pursuit movement respectively from SSVEP
ation in the display centre; (d) Elicit a long saccade; (e) Elicit
features (2 class problem). C5 and C6 repeat this but for
fixation at the display edge; (f) Possible stimuli locations (g)
SSVEP detection for a specific flickering frequency (4 class
Static SSVEP stimulation for fixations where stimulus is fixed
problem assuming 3 potential FoA flickering at different fre-
at the screen centre; (h) SSVEP stimulation for pursuit where
quencies). C7 and C8 classifiers classify pursuit direction
stimulus moves from the screen centre to the sides.
with and without SSVEP present (2 class problem).
The saccade and pursuit direction detection performance
The 30Hz VOG data is up-sampled to match the 128Hz EEG is assessed by the Root Mean Square Error (RMSE) of the
data by linear interpolation; this approximates the saccades’ difference between the detected angle and the real angle asso-
trajectory in VOG. ciated with the stimuli’s position:
v
u
u1 X N
4.1. Sessions recorded
RM SE = t (θt − θ̂t )2 (1)
N t=1
Fixation, Saccade and Pursuit direction detection from EOG:
Fixations and horizontal, vertical and diagonal saccades are
elicited by visual cue. The EEG collected is used to detect and ,where N is the time length, and θt , θ̂t are the real and esti-
classify pursuit and saccade direction from the EOG features mated angular stimuli position at time t, respectively.
as outlined in section 3. Refer to Fig. 1, users are presented
with a black square display and there are 9 possible stimulus 4.3. Results
locations around the screen edge. Sessions start with a white
fixation cross appearing. To elicit short saccades, the cross Table 1 summarises the classification performance results ob-
appears in the centre. To elicit long saccades, the cross ap- tained through different classifiers (C1-C2). The classifiers’
pears at the edge. After a random time between 500ms and accuracy and RMSE before and after applying ICA are re-
900ms, the stimulus is presented at one of the nine edge lo- ported. As its evident from the results, the all classifiers’
cations. The cross disappears at a random interval 500ms to accuracy is improved and the between-person variation de-
900ms after stimulus presentation which provides the cue for creased significantly if features are extracted from ICA com-
users to make a saccade to the stimulus location. Stimuli are ponents (Fig. 2).
presented for a further 1s after the cue. Each user performs For C1, short saccade direction classification accuracy is
112 cued fixations and 112 cued saccades which lasts approx- almost 74% with a 14% improvement if features are extracted
imately 11 minutes. from ICA components. C2 benefits about 2% more from ap-
Focus of foveal Attention (FoA) during fixations and pur- plying ICA than C1, result in 89% accuracy for long saccade
suit movements: Fixations and smooth pursuits are elicited by direction classification. Considering RMSE, similar trend is
flickering stimuli. The EEG collected is used to detect and obtained when comparing C1 and C2 (Fig. 2).
classify FoA fixations and pursuits direction from features For C3 and C4, FoA estimation (i.e. SSVEP detection
outlined in section 3. Similar to the previous session, refer during fixations and pursuits respectively) accuracy again is
to Fig. 1, the screen is a black square. Again, sessions start shown to be improved (e.g. C3 by 6% to 97.90%) with ICA
with a white fixation cross appearing. After 5s the stimulus - a component features. Similarity between results suggests that
10x10 checkerboard - appears flickering at one of three differ- the SSVEP response during pursuit movements is no harder

Authorized licensed use limited to: Banasthali Vidyapith. Downloaded2032

on March 10,2021 at 07:34:46 UTC from IEEE Xplore. Restrictions apply.
 Table 1. Eye Movement Classification Results.
RMSE (◦ ) Accuracy (%)
Classifer Description
Before ICA After ICA Before ICA After ICA
C1 Short saccade in one of 8 directions 19.02 9.02 73.84 87.58
C2 Long saccade in one of 8 directions 19.30 7.81 73.45 89.25
C3 Fixation from SSVEP detection (all frequencies) (2 class) N/A N/A 91.38 97.90
C4 Pursuit from SSVEP detection (all frequencies)(2 class) N/A N/A 92.27 97.79
C5 Fixation-FoA from SSVEP detection (7Hz, 10Hz ,12Hz)(4 class) N/A N/A 83.87 96.38
C6 Pursuit-FoA from SSVEP detection (7Hz, 10Hz ,12Hz) (4 class) N/A N/A 86.59 96.05
C7 Pursuit direction (no SSVEP) (8 class) 11.51 1.79 81.64 97.34
C8 Pursuit direction (SSVEP present) (8 class) 12.44 1.63 82.88 97.57

Fig. 2. , (a) RMSE for saccadic eye movement direction detection; C1: Short saccade; C2: Long saccades, grey bars represent
RMSE when features are extracted from original EEG channels, black bars represents RMSE when features are extracted from
ICA estimated sources; (b) SSVEP frequency classification accuracy;(c) Pursuit direction detection; C7: without SSVEP, C8:
with SSVEP. The error bars represent standard errors.

to detect than fixations despite the potential for more EEG and evaluated. Feature extraction from EEG source compo-
signal contamination due to EOG. nents estimated by ICA results in better performance com-
For C5 and C6, FoA estimation for SSVEP given 3 po- pared to feature extraction from pre-processed EEG channels;
tential stimuli flickering has 96% accuracy for pursuit move- notably between-person variation reduces. This work is the
ments and fixations with features extracted from ICA compo- first we are aware that uses source components for SSVEP
nents. classification rather than EEG channels. Future work will
Finally, results from C7 and C8 demonstrate that the pres- refine the eye movement type classification in the presence
ence of SSVEP in the visual field does not hinder the classi- of other EEG artefacts such as body movement. The VOG
fication of pursuit direction with both classifiers show similar data captured in this study affords the opportunity for hybrid
results. VOG/EEG multimodal eye tracking. The modulation of real-
Overall, eye movement direction classification accuracy scenes with flickering stimuli via augmented reality technolo-
is best in C8 (i.e. 97.57%) with the least RMSE (i.e. 1.63◦ ) gies to estimate focus of foveal visual attention will poten-
where there is pursuit eye movement rather than saccadic eye tially lead to VOG-free eye tracking, without the social and
movement. physical intrusion of face mounted electrodes.

5. DISCUSSION 6. REFERENCES

A signal processing scheme for extracting eye tracking data [1] Dan Witzner Hansen and Qiang Ji, “In the eye of the be-
from wireless portable scalp-based EEG has been proposed holder: A survey of models for eyes and gaze,” Pattern

Authorized licensed use limited to: Banasthali Vidyapith. Downloaded2033

on March 10,2021 at 07:34:46 UTC from IEEE Xplore. Restrictions apply.
 Analysis and Machine Intelligence, IEEE Transactions [13] Leonard J Trejo, Roman Rosipal, and Bryan Matthews,
on, vol. 32, no. 3, pp. 478–500, 2010. “Brain-computer interfaces for 1-d and 2-d cursor con-
trol: designs using volitional control of the eeg spec-
[2] Matthew Middendorf, Grant McMillan, Gloria Calhoun, trum or steady-state visual evoked potentials,” Neural
and Keith S Jones, “Brain-computer interfaces based on Systems and Rehabilitation Engineering, IEEE Trans-
the steady-state visual-evoked response,” Rehabilitation actions on, vol. 14, no. 2, pp. 225–229, 2006.
Engineering, IEEE Transactions on, vol. 8, no. 2, pp.
211–214, 2000. [14] Po-Lei Lee, Jyun-Jie Sie, Yu-Ju Liu, Chi-Hsun Wu,
Ming-Huan Lee, Chih-Hung Shu, Po-Hung Li, Chia-
[3] Kyung-Nam Kim and RS Ramakrishna, “Vision-based Wei Sun, and Kuo-Kai Shyu, “An ssvep-actuated brain
eye-gaze tracking for human computer interface,” in computer interface using phase-tagged flickering se-
Systems, Man, and Cybernetics, 1999. IEEE SMC’99 quences: a cursor system,” Annals of Biomedical En-
Conference Proceedings. 1999 IEEE International Con- gineering, vol. 38, no. 7, pp. 2383–2397, 2010.
ference on. IEEE, 1999, vol. 2, pp. 324–329.
[15] Peter Welch, “The use of fast fourier transform for the
[4] Kevin Smith, Sileye O Ba, Daniel Gatica-Perez, and estimation of power spectra: a method based on time av-
Jean-Marc Odobez, “Tracking the multi person wan- eraging over short, modified periodograms,” Audio and
dering visual focus of attention,” in Proceedings of the Electroacoustics, IEEE Transactions on, vol. 15, no. 2,
8th international conference on Multimodal interfaces. pp. 70–73, 1967.
ACM, 2006, pp. 265–272.
[16] Anthony J Bell and Terrence J Sejnowski, “An
information-maximization approach to blind separation
[5] Carlos H. Morimoto and Marcio R. M. Mimica, “Eye
and blind deconvolution,” Neural computation, vol. 7,
gaze tracking techniques for interactive applications,”
no. 6, pp. 1129–1159, 1995.
Comput. Vis. Image Underst., vol. 98, no. 1, pp. 4–24,
Apr. 2005. [17] Matthieu Duvinage, Thierry Castermans, Mathieu
Petieau, Thomas Hoellinger, Guy Cheron, and Thierry
[6] Ali Bülent Usakli, Serkan Gurkan, Fabio Aloise, Gio- Dutoit, “Performance of the emotiv epoc headset for
vanni Vecchiato, and Fabio Babiloni, “On the use of p300-based applications,” Biomedical engineering on-
electrooculogram for efficient human computer inter- line, vol. 12, no. 1, pp. 56, 2013.
faces,” Computational Intelligence and Neuroscience,
vol. 2010, pp. 1, 2010. [18] L Mayaud, M Congedo, A Van Laghenhove, M Figère,
E Azabou, and F Cheliout-Heraut, “A comparison of
[7] Arantxa Villanueva and Rafael Cabeza, “Models for recording modalities of p300 event related potentials
gaze tracking systems,” Journal on Image and Video (erp) for brain-computer interface (bci) paradigm,” Neu-
Processing, vol. 2007, no. 3, pp. 4, 2007. rophysiologie Clinique/Clinical Neurophysiology, 2013.

[8] Hans Berger, “Über das elektrenkephalogramm des [19] Danhua Zhu, Jordi Bieger, Gary Garcia Molina, and
menschen,” European Archives of Psychiatry and Clin- Ronald M Aarts, “A survey of stimulation methods used
ical Neuroscience, vol. 87, no. 1, pp. 527–570, 1929. in ssvep-based bcis,” Computational intelligence and
neuroscience, vol. 2010, pp. 1, 2010.
[9] Saeid Sanei and Jonathon A Chambers, EEG signal pro-
cessing, Wiley. com, 2008.

[10] John N Demos, Getting started with neurofeedback,

WW Norton & Company, 2005.

[11] Ricardo Nuno Vigário, “Extraction of ocular artefacts

from eeg using independent component analysis,” Elec-
troencephalography and clinical neurophysiology, vol.
103, no. 3, pp. 395–404, 1997.

[12] Zhonglin Lin, Changshui Zhang, Wei Wu, and Xiaorong

Gao, “Frequency recognition based on canonical corre-
lation analysis for ssvep-based bcis,” Biomedical En-
gineering, IEEE Transactions on, vol. 53, no. 12, pp.
2610–2614, 2006.

Authorized licensed use limited to: Banasthali Vidyapith. Downloaded2034

on March 10,2021 at 07:34:46 UTC from IEEE Xplore. Restrictions apply.