Proceedings of the 3rd International FrD3.

10
IEEE EMBS Conference on Neural Engineering
Kohala Coast, Hawaii, USA, May 2-5, 2007

Spelling with Steady-State Visual Evoked Potentials

Abstract— In this work, the real-time performance of a novel relative merits of different technologies their respective state-
method for detecting Steady-State Visual Evoked Potentials of-the-art performances must be compared, and this work can
(SSVEP) is evaluated in a brain-computer interface (BCI) be seen as a step towards such a goal. Moreover, it may be
spelling task. At the core of this method is a spatial filtering
algorithm for extracting SSVEP responses, which in previous the case that hybrid methods will turn out to work well, e.g.,
off-line studies has shown significantly improved classification a combination of SSVEP and eye-tracking.
performance. The on-line performance is investigated by letting There are two main goals in this work. First, in [7] a
a group of 11 healthy subjects spell the word ’BRAINCOM- novel spatial filtering algorithm was presented, with which
PUTERINTERFACE’. An average information transfer rate of it is possible to detect SSVEP responses more accurately
27 bits/minute was obtained in this task and the probability
of correctly classifying the user’s intention was estimated to compared to previous methods. One goal is to investigate
97.5%. In addition, two different letter layouts and selection the information transfer rate that can be obtained with this
schemes tailored for SSVEP BCI’s are compared. improved detection algorithm in a real BCI application. The
second goal is to investigate spelling applications tailored for
I. I NTRODUCTION SSVEP-based BCI’s. Two different spelling layouts and letter
selection methods are evaluated. The outline of the following
A main goal in brain-computer interfacing is to pro-
sections are as follows: First, the implementation of the
vide means of communication for severely disabled peo-
above-mentioned SSVEP detection algorithm is described.
ple. The currently most successful BCI applications are
Next, the two letter selection methods are presented. Finally,
spelling devices which allow users to write text messages
the information transfer rates and the letter selection methods
by voluntarily controlling their brain signal patterns. The
are investigated in a group of 11 healthy subjects.
first BCI spelling application was based on the P300 signal
pattern [1]. A number of other brain signals have since II. R EAL - TIME SSVEP DETECTION
been successfully utilized for spelling, e.g., Slow Cortical
The SSVEP response to visual stimulation with a light
Potentials [2] and ERD/ERS [3]. The BCI presented in this
source flickering with a frequency f0 is well modeled as a
work is based on Steady-State Visual Evoked Potentials
superposition of a number of sinusoids with frequencies f0 ,
(SSVEP), which is a signal evoked mainly in the visual and
2f0 , 3f0 ,. . . , i.e., the fundamental stimulation frequency and
parietal cortexes as a response to flickering visual stimuli
its harmonics. In [7], a novel spatial filter which extracts
[4]. The SSVEP phenomenon has gained interest in the BCI
the SSVEP response from multiple electrodes placed over
community because it provides advantages in terms of speed
the visual and parietal cortexes is presented. This proce-
and robustness. These advantages can be attributed to the
dure is briefly explained below. First, let Y be a matrix
fact that SSVEP responses are strong and robust signals
where each column contains the signal from one electrode
which can be well modeled and detected. An SSVEP-based
location. Let Xf = [sin(2πf t) cos(2πf t)] be a model
BCI relies on external visual input in form of an array of
matrix containing a sine and a cosine of frequency f and
light sources, where each light source is flickering with
let X = [Xf0 X2f0 X3f0 ] be a matrix where the columns
a distinct frequency [5], [6]. By analyzing the frequency
span the signal subspace in which the SSVEP response
content in recorded electroencephalography (EEG) signals,
is assumed to reside. The goal is to find a spatial filter
the light source on which the user is focussing the visual
vector w that cancels as much of the interference and
attention can be inferred. Herein lies the main limitation
nuisance signals in Y as possible. To
¢−1this Tend, a modified
of an SSVEP-based BCI; the subject controlling the BCI
signal matrix Ỹ = (I − X XT X
¡
X )Y, where the
must be able to move the eyes in order to focus on one
energy in the SSVEP frequencies has been removed, is first
of several light sources. This restriction excludes users who
calculated. The filter vector w is subsequently found as
do not have this ability and who would benefit the most
the eigenvector belonging to the smallest eigenvalue of the
from the BCI technology. Clearly, competing technologies
matrix ỸT Ỹ. A spatially filtered signal is finally calculated
can be used when the subject is able to move the eyes, e.g.,
as Yw, where it is important to emphasize that the filter is
a communication device based on eye-tracking. To assess the
applied to the original signals in Y. The SSVEP response
This research project has been supported by a Marie Curie Transfer strength is quantified by calculating a SSVEP-response-to-
of Knowledge Fellowship of the European Communitys Sixth Framework noise ratio (SNR) for the filtered signal, i.e., the SSVEP
Programme under contract number MTKD-CT-2004-014211. response strength is contrasted with an estimate of the noise
The authors are with the Institute of Automation, University of Bremen,
Otto-Hahn Allee 1, 28359 Bremen, Germany. Corresponding author is O. power in the same frequencies. The noise power is estimated
Friman [email protected] by means of an autoregressive AR(5) model fitted to the

1-4244-0792-3/07/$20.00©2007 IEEE. 354

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO ESPIRITO SANTO. Downloaded on December 9, 2009 at 06:36 from IEEE Xplore. Restrictions apply.
 selecting the row and then the column of the desired letter.
1 2 3 4 5 L That is, the selection of a letter requires two decisions. In
1 A B C D E B R J the alternative layout in Fig. 1b, which is referred to as the
Rhombus layout, letters are instead selected by navigating
2 F G H I J V M I F K the cursor right, left, up and down until the desired letter is
3 K L M N O U N A E O S C reached. This letter is then selected using the fifth SSVEP
frequency. Once a letter has been selected, the cursor starts
4 P R S T U P W T G Y over at the center position. That is, except for the letter E,
5 V W X Y Z X H Z two or more commands must be executed to select a letter
and this layout is therefore expected to be slower than the
D Row-Column layout. On the other hand, the Rhombus layout
a) Row-Column layout b) Rhombus layout is more forgiving against error-classifications, for example,
an accidental ”up”-command can easily be corrected by
Fig. 1. a) The Row-Column layout, where letters are chosen by first the user. The Rhombus layout was constructed with the
selecting the row containing the desired letter and then the column. b) The
Rhombus layout, in which a cursor is navigated right, left, up and down to assumption that it is easier to keep a constant visual attention
reach the desired letter. Once the letter has been chosen, the cursor jumps on one flickering light source for an extended time to produce
back to the center position. repeated selections of the same command than to switch
attention between the light sources. To ensure that common
letters are reached with a minimum of effort, the letters
spatially filtered signal. When applied off-line to recorded were ordered with respect to both their occurrences in the
SSVEP data, this algorithm gave the best performance in English language as well as to the number of attention-
a comparison between six different detection methods [7]. switches between light sources required. For example, in the
Note that the algorithm does not rely on any calibration Rhombus layout in Fig. 1b, two commands are required to
or training data. For the current work, to obtain real-time reach the letters ’M’ (left-up) and ’N’ (left-left). Since the
performance, a BCI system was implemented in C++ where letter ’N’ is about twice as common in the English language
the detection algorithm described above is included as a as the letter ’M’ (relative frequencies about 6% and 3%
class. This code is available upon request. respectively), the letter ’N’ is in the layout reached with
the left-left command sequence which is assumed faster to
III. S PELLING LAYOUTS
execute than the left-up sequence. The letters in Fig. 1b were
Depending on the number of distinct brain patterns that ordered manually with the above considerations in mind and
can be distinguished, different strategies for selecting letters it should therefore be seen as a good ordering rather than
in a spelling application are required. When only two dif- an optimal ordering. More importantly, these considerations
ferent patterns can be discriminated, a tree layout may be lead to the rhombus shape instead of a square shape of the
suitable [8], e.g., a letter is selected by recursively partition- layout, because more letters can be reached without any
ing the alphabet until only the desired letter remains. Another attention switches at all in the rhombus shape than in a square
layout suitable for binary commands is the recently proposed layout. The spelling layouts were also implemented in C++
Hex-o-Spell [9], which also employs a recursive partitioning and they receive commands from the analysis software via
scheme. In spelling applications designed for P300 BCI’s, a TCP/IP connection.
the alphabet is typically arranged in a 2-dimensional grid.
The characters in the grid are randomly flashed and a P300 IV. S UBJECTS AND DATA ACQUISITION
response is evoked every time the target letter is highlighted 11 healthy subjects (1 female and 10 male) in the ages
[1]. With a BCI based on the SSVEP response, a relatively 21-58 years (mean age 30.1 years) were recruited for the
large number of commands can be distinguished. In the current work. Only 1 of the subjects (Subject #1) had prior
current work, 5 different commands are used, requiring five experience of the BCI and the spelling task. 6 gold EEG
different light sources that flicker with different frequencies. electrodes were placed over the visual and parietal cortex (2
In Fig. 1, two different spelling layouts designed to be electrodes at positions OZ , PZ according to the international
controlled with these 5 commands are shown. The layouts 10-20 system and 4 electrodes 2.5 cm above and under the
contain 25 of the 26 letters in the English alphabet (the letter positions O1 and O2 respectively). These electrodes were
Q is excluded). It should be stressed that the two presented referenced to a ground electrode placed at position FZ . The
layouts are used for evaluation purposes only and that more measured signals were amplified using a g.BSamp amplifier
symbols are required in a final layout. This can be achieved from g.tec and filtered through analog high-pass and low-
by extending the layouts and/or introducing more SSVEP pass filters with cut-off frequencies of 0.5 Hz and 32 Hz
frequencies that encode more commands. For example, a respectively. Electrode impedances were at all times kept
delete function could be implemented using a sixth light below 5 kΩ. The amplified signals were digitized using
source. The layout in Fig. 1a is here referred to as the Row- a National Instruments acquisition card DAQ6024E with a
Column layout, and it is similar to the design commonly used sampling rate of 128 Hz and further processed on a regular
in P300 spelling programs [1]. Letters are chosen by first laptop computer with a 1.73 GHz Pentium M processor. Both

355

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO ESPIRITO SANTO. Downloaded on December 9, 2009 at 06:36 from IEEE Xplore. Restrictions apply.
 Row-Column layout Rhombus layout
the SSVEP processing application described in Section II Minimum 44 commands Minimum 58 commands
and the spelling programs described in Section III ran on Time Errors Bits/min Time Errors Bits/min
the same computer. For visual stimulation a custom made Subject #1 2:40 0 38 3:01 0 43
Subject #2 3:02 2 31 3:32 0 38
array with 5 red light emitting diodes (LED’s) was used. Subject #3 2:27 1 40 5:11 0 26
These LED’s were flickering with the five frequencies 13, Subject #4 3:20 0 31 4:25 0 30
14, 15, 16 and 17 Hz respectively. These frequencies encode Subject #5 3:56 1 25 4:29 1 29
Subject #6 3:52 0 26 4:40 0 29
the five commands needed to operate the spelling layouts, Subject #7 3:53 1 25 4:40 2 29
and they were chosen because previous studies have shown Subject #8 4:34 1 21 5:14 2 24
that the strongest SSVEP responses are evoked in this range Subject #9 5:16 1 19 4:34 0 29
Average 3:40 28.4 4:25 30.6
[10]. Harmonics of the stimulation frequencies larger than Subject #10 - - - 9:06 6 13
32 Hz (cut-off of the low-pass filter) were not used for the Subject #11 - - - 11:13 8 11
SSVEP detection. Each LED covered an area of about 2×4 Average - - 5:28 27.2
cm. The subjects were placed in a comfortable chair with TABLE I
about 0.5 m viewing distance to the LED’s and the screen E XPERIMENTAL RESULTS
of the laptop. During the acquisition the room was dimmed.

V. M ETHOD
was instructed to rewrite the desired letter so that all letters
The 11 subjects were given the task to write the word in the task word were written.
’BRAINCOMPUTERINTERFACE’ (22 letters) with the two
spelling layouts presented in Section III. In the Row-Column VI. R ESULTS
layout, 2 commands are required to select a letter, and The resulting times, errors and information transfer rates (
therefore a total of 44 commands are required to write this [11]) for the 11 subjects are reported in Table I. The subjects
word. In the Rhombus layout it takes a minimum of 58 are sorted according to their performance, i.e., Subject #1
commands to write the task word. The SSVEP detection had the highest average bit-rate and Subject #11 the lowest.
algorithm was set to take 2 seconds of EEG data into account Moreover, Subject #10 and Subject #11 were only able to
for detecting SSVEP responses in the recorded signals, as complete the task within reasonable time (< 15 minutes)
this length has been observed to give a good speed vs. using the Rhombus layout and not with the Row-Column
detection accuracy trade-off. Likewise, an idling period of layout. The reason for this is that these two subjects had weak
2 seconds was used after a classification to acquire fresh SSVEP responses and that they consequently had problems
data for the next classification. An SNR threshold was used producing SNR’s exceeding the threshold. As briefly touched
for detecting significant SSVEP responses. This threshold upon previously, false-positive errors cause less damage in
was initially set to a default value of 4, but for some the Rhombus layout since in most cases an error only means
subjects who had very strong SSVEP responses the threshold that the cursor moves one step in the wrong direction and
was raised to 6 to further reduce the probability of false only rarely is the select command triggered by chance.
positive classifications. Via Monte Carlo simulations it can Hence, even if it took some time, these two subjects were
be shown that the probabilities that the SNR of any of the able to write the entire task word using the Rhombus layout
five stimulation frequencies exceeds 4 and 6 by chance are with very few actual spelling errors. In the Row-Column
0.05 and 0.002 respectively. layout all commands are implicitly select-commands, and
Since the users were new to the spelling task and the Subject #10 and Subject #11 therefore experienced problems
spelling layouts, they were asked to play around with the with this layout.
system and each spelling layout for a few minutes, e.g., by Studying the 9 remaining subjects who wrote the task
writing their names, before the actual spelling task. Based word with both spelling layouts, the average times reveal that
on the strength of SSVEP responses (which were plotted the Row-Column layout is faster than the Rhombus layout
on-line) generated in this initial testing, the classification (paired t-test p = 0.036). The main reason is that fewer
threshold was adjusted for strong performing subjects as commands must be executed to complete the spelling task
mentioned above. All subjects felt ready for the actual task with the Row-Column layout. This layout was also preferred
within 5 minutes. The subjects were then asked to first write by most of the subjects. Note, however, that the bit-rate
the task word using the Rhombus layout and then with the is higher for the Rhombus layout and the average time to
Row-Column layout. To evaluate the performance, the time generate a correct command using the Row-Column layout
required to complete the spelling task was measured and the is 5 seconds and 4.6 seconds for the Rhombus layout (paired
number of errors made was recorded. An error was counted t-test p = 0.29). From the raw numbers it is not possible to
when the BCI made a classification that did not agree with tell if this is due to a fatigue effect (the Rhombus layout
the user’s intention, e.g., a cursor-movement to the left when was always used before the Row-Column layout), due to the
the intention was an up-movement. The subjects were asked design of the Rhombus layout which was made so that the
to report such errors to the person monitoring the experiment. sequence of commands is easy and fast to execute, or due
If an error resulted in a false letter being written the subject to chance. However, since the experiments were quite short

356

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO ESPIRITO SANTO. Downloaded on December 9, 2009 at 06:36 from IEEE Xplore. Restrictions apply.
 and included pauses, one may guess that the difference is is in general weaker and more difficult to detect than the
not due to fatigue. SSVEP response. Finally, to maximize writing speed, a
final spelling application should of course be supported by
VII. D ISCUSSION stronger language models that can make informed guesses
There are two main aims in this work. The first is to of the next letter as well as the entire next word.
evaluate a novel SSVEP detection algorithm in a real-time
VIII. C ONCLUSIONS
BCI application. The second aim is to present spelling appli-
cations tailored for SSVEP-based BCI’s and to evaluate their A real-time implementation of a novel SSVEP detection
merits. The speed obtained using the novel detection algo- algorithm has been evaluated in a brain-computer interface
rithm ranges between 11 and 43 bits/minute, with an average spelling application. Information transfer rates of up to 43
of about 27-30 bits/minute. It is difficult to do meaningful bits/minute using 5 different commands were obtained. Two
comparisons with previously reported speeds (due to, for different spelling layouts designed for SSVEP-based BCI’s
example, different number of LED’s and different tasks), but were also evaluated, and it was found that the Row-Column
the speed must be considered as high. The high classification layout was faster than the Rhombus layout, especially for
accuracy of about 97.5 % should also be emphasized, as well strong performing subjects. For lower-performing subjects,
as that very little subject-specific adaptation of the BCI is the Rhombus layout may be more suitable since it is more
required (only the detection threshold was adjusted in this forgiving against false-positive error classifications.
work). Thus, the BCI can be said to follow the plug-and- R EFERENCES
play principle. This does, however, not mean that training
[1] L. Farwell and E. Donchin, “Talking off the top of your head:
and tuning would not improve the performance further. As toward a mental prosthesis utilizing event-related brain potentials,”
an example, it was observed that several of the subjects Electroencephalogr Clin Neurophysiol, vol. 70, no. 6, pp. 510–523,
had problems and spent lots of time trying to produce an 1988.
[2] A. Kübler, B. Kotchoubey, T. Hinterberger, N. Ghanayim, J. Perel-
SSVEP signal with the lowest 13 Hz stimulation frequency, mouter, M. Schauer, C. Fritsch, E. Taub, and N. Birbaumer, “The
whereas the higher frequencies did not pose any problems thought translation device: a neurophysiological approach to commu-
at all. These subjects would have completed the spelling nication in total motor paralysis,” Experimental Brain Research, vol.
124, no. 2, pp. 223–232, 1999.
task considerable faster had the stimulation frequencies been [3] B. Obermaier, G. Mller, and G. Pfurtscheller, “”Virtual keyboard”
shifted to the interval 14 - 18 Hz instead. Moreover, a controlled by spontaneous EEG activity,” IEEE Transactions on Neural
majority of the subjects spontaneously commented that they Systems and Rehabilitation Engineering, vol. 11, no. 4, pp. 422–426,
2003.
would be able to complete the spelling task much faster with [4] D. Regan, Human Brain Electrophysiology: Evoked Potentials and
some practise because they would then know the locations Evoked Magnetic Fields in Science and Medicine. Elsevier, New
of the letters in the layouts, and there would be no need to York, 1989.
[5] M. Middendorf, G. McMillan, G. Calhoun, and K. Jones, “Brain-
switch the attention between the LED’s and the computer computer interfaces based on the steady-state visual-evoked response,”
screen. In the literature it is commonplace to report the IEEE Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp.
number of characters that can be written per minute with the 211–214, 2000.
[6] M. Cheng, X. Gao, S. Gao, and D. Xu, “Design and implementation of
BCI system. This is clearly a very relevant quantity for the a brain-computer interface with high transfer rates,” IEEE Transactions
end-user. In this work, Subject #3 was able to write 9 letters on Biomedical Engineering, vol. 49, no. 10, pp. 1181–1185, 2002.
per minute with the Row-Column layout. As a comparison, [7] O. Friman, I. Volosyak, and A. Gräser, “Multiple channel detection
of steady-state visual evoked potentials for brain-computer interfaces,”
in [12] a speed of 7.8 characters per minute and a 80% IEEE Transaction on Biomedical Engineering, vol. 54, no. 4, pp. 742–
classification accuracy are reported with a P300-based BCI 750, 2007.
and a similar row-column layout. It should be mentioned that [8] J. Perelmouter, B. Kotchoubey, A. Kübler, and N. Birbaumer, “Lan-
guage support program for thought-translation devices,” Automedica,
this latter layout consisted of 6 rows and columns, whereas vol. 18, pp. 67–84, 1999.
the Row-Column layout in this work consists of only 5 rows [9] B. Blankertz, G. Dornhege, M. Krauledat, M. Schrder, J. Williamson,
and columns. However, extending the current layout with R. Murray-Smith, and K.-R. Mller, “The Berlin brain-computer inter-
face presents the novel mental typewriter Hex-o-Spell,” in Proceedings
one extra row and column, and an extra LED, would not of the 3rd International Brain-Computer Interface Workshop and
pose any problems for the high-performance subjects in this Training Course. Verlag der Technischen Universität Graz, 2006.
study. As was discussed in the Introduction, SSVEP-based [10] C. Herrmann, “Human EEG responses to 1-100 Hz flicker: resonance
phenomena in visual cortex and their potential correlation to cognitive
BCI’s can be expected to be faster than, for example, P300- phenomena,” Experimental Brain Research, vol. 137, no. 3-4, pp. 346–
based BCI’s. On the other hand, as also already discussed, 353, 2001.
P300-based BCI’s are potentially useful for a larger group of [11] J. Wolpaw, N. Birbaumer, W. Heetderks, D. McFarland, P. Peckham,
G. Schalk, E. Donchin, L. Quatrano, C. Robinson, and T. Vaughan,
people than SSVEP-based BCI’s. In the comparison between “Brain-computer interface technology: a review of the first interna-
the Row-Column layout and the Rhombus layout, the Row- tional meeting,” IEEE Transactions on Rehabilitation Engineering,
Column layout was found to be significantly faster, the vol. 8, no. 2, pp. 164–173, 2000.
[12] E. Donchin, K. Spencer, and R. Wijesinghe, “The mental prosthesis:
main reason being that fewer commands must be executed assessing the speed of a P300-based brain-computer interface,” IEEE
to select a letter in the Row-Column layout. However, the Transactions on Rehabilitation Engineering, vol. 8, no. 2, pp. 174–
Rhombus layout was found better for subjects with weaker 179, 2000.
SSVEP signals. Thus, the Rhombus layout may be useful in
combination with a P300 BCI, because the P300 response

357

Authorized licensed use limited to: UNIVERSIDADE FEDERAL DO ESPIRITO SANTO. Downloaded on December 9, 2009 at 06:36 from IEEE Xplore. Restrictions apply.