Neural Networks 22 (2009) 1329–1333

Contents lists available at ScienceDirect

Neural Networks
journal homepage: www.elsevier.com/locate/neunet

2009 Special Issue

The MindGame: A P300-based brain–computer interface game

Andrea Finke a,b,∗ , Alexander Lenhardt b , Helge Ritter a,b
a
Research Institute for Cognition and Robotics - CoR-Lab, Bielefeld University, Germany
b
Neuroinformatics Group, Faculty of Technology, Bielefeld University, Germany

article info abstract

Article history: We present a Brain–Computer Interface (BCI) game, the MindGame, based on the P300 event-related
Received 1 October 2008 potential. In the MindGame interface P300 events are translated into movements of a character on a three-
Received in revised form 30 June 2009 dimensional game board. A linear feature selection and classification scheme is applied to identify P300
Accepted 8 July 2009
events and calculate gradual feedback features from a scalp electrode array. The classification during the
online run of the game is computed on a single-trial basis without averaging over subtrials. We achieve
Keywords: classification rates of 0.65 on single-trials during the online operation of the system while providing
Brain–computer interface gradual feedback to the player.
BCI
© 2009 Elsevier Ltd. All rights reserved.
EEG
P300
Virtual game
Neurofeedback

1. Introduction studied extensively. Polich (2003, 2007) provides detailed reviews

on the P300 potential.
Various types of brain–computer interfaces (BCI) have been in- The first P300-based BCI was introduced by Farwell and
troduced and evaluated in great detail in the past two decades. The Donchin (1988). Using data from only one electrode (Pz) and
vast majority of these approaches relies on electroencephalogra- simple, model-based classifiers, they already succeeded in de-
phy (EEG) to acquire signals from the brain – for many good rea- veloping a slow but usable communication device for severely
sons: EEG technology is non-invasive and highly portable while paralyzed patients. Their original paradigm, the P300 Speller
acquisition and maintenance are inexpensive. Several components Paradigm, has since then become the standard benchmark case
can be extracted from the human EEG that are suited as input for P300-based BCIs. It has been successfully implemented by
to a brain interface, among them the P300 event-related poten- several researchers (e.g. Lenhardt, Kaper, & Ritter, 2008; Serby,
tial (ERP). For a general overview on current BCI research refer to Yom-Tov, & Inbar, 2005) in offline and online mode. These ap-
the book of Dornhege, del R Millán, Hinterberger, McFarland, and proaches usually target at the improvement of information trans-
Müller (2007). Wolpaw et al. (2002) gives an introduction on the fer rates. Others, like Bell, Shenoy, Chalodhorn, and Rao (2008),
ideas guiding BCI research. try to control robots with the P300. In the past BCI competitions
The P300 is time-locked to a stimulus and characterized by a I to III data from the P300 Speller Paradigm was provided for anal-
positive polarity. However, it is usually embedded in EEG noise and ysis (see http://ida.first.fraunhofer.de/projects/bci/competition_iii
superposed by other EEG events. It is triggered by unpredictable for an overview on the last competition with P300 data).
thus surprising events in a series of background stimuli (Duncan- A very important and so far often ignored advantage of the P300
Johnson & Donchin, 1977). Decades of psychological research have over other EEG components is that its application in a BCI offers a
shown that the P300 can be elicited reliably in every neurologically discrete selection rather than a continuous control mechanism as
healthy human. The experimental parameters influencing the most of the other components. In a recent report Wolpaw (2007)
properties of the P300, which are amplitude and latency, have been called this distinction goal-selection versus process control. He
argued that the goal-selection approach resembles the natural
functioning of the brain much closer since it does not require the
brain to learn something completely new. The P300 is produced
∗ Corresponding author at: Research Institute for Cognition and Robotics - CoR-
unconsciously without large additional cognitive load. On a
Lab, Bielefeld University, Germany. Tel.: +49 521 10612116.
cognitive level the P300 can be seen as a measure for alertness and
E-mail addresses: [email protected] (A. Finke),
[email protected] (A. Lenhardt), [email protected]
attention and thus reflecting a subject’s general arousal level (Datta
(H. Ritter). et al., 2007).

0893-6080/$ – see front matter © 2009 Elsevier Ltd. All rights reserved.
doi:10.1016/j.neunet.2009.07.003
1330 A. Finke et al. / Neural Networks 22 (2009) 1329–1333

in the symbol matrix of the original P300 Speller Paradigm. One

difference, however, is that there are no longer group-wise stimuli
highlightings in the form of rows or columns. Instead, one turn of
the game consists of the consecutive highlighting of all 12 target
fields. Target and background stimuli are coded by different colors,
namely red for the target and yellow for non-targets. The player
focuses attention on the target such that a red flash will elicit
a P300 while a yellow flash will not. Fig. 1 shows the graphical
setup of the MindGame. Immediately following each turn1 the
classifier will try to identify the correct target. In the case of a
successful recognition the output of the classifier will be translated
into a movement of the character, which is modulated gradually.
Based on the quality of the detected P300 the character is moved
a certain number of steps. The number of steps is not fixed in
advance, but is decided according to the ‘‘confidence’’, which can
be interpreted as a measure of the quality of the P300. Listing 1
shows the pseudocode for the basic MindGame loop.

Algorithm 1 MindGame: Online classification and game control.

1: Initialize k ← 0, set ISI ← 120 {k is the subtrial index}
2: Load pre-trained classifier Φ and PCA matrix P
3: while k < 12 do
4: Flash all 12 stimuli in random order
5: Create subtrial matrix Xs
6: Apply PCA matrix: X̂s = Xs ∗ P
7: Apply classifier to feature vectors: y = Φ (x̂si )
8: if y > 0 then
9: Calculate feedback λs
10: Move character λs steps
11: if Target reached then
12: k ← k + 1 Transform tree
13: end if
14: end if
15: end while

Fig. 1. Screenshots from the MindGame. The subject controls the character by
Relevant parameters of the MindGame system are mainly
selecting the target of the next move in a P300-based paradigm. the amplitude of the P300, the ISI and the target probability.
These parameters have a direct impact on the difficulty of the
2. The MindGame approach discrimination task. Longer ISIs are known to allow for larger
amplitudes. However, as a tradeoff between large amplitudes and
Our point of departure was the original P300 Speller Paradigm, the subtrial duration we chose an ISI of 120 ms. Target probability
where a 6 × 6 symbol matrix containing letters and digits is was set to p = 0.08, experimental studies revealed a maximal
presented to the subject. The subject has to attend to the symbol amplitude around a value p = 0.1 (Polich, 1987).
while the rows and the columns are highlighted in random order.
Since the highlighting events are unpredictable for the subject, 3. Methods
they elicit P300 events whose detection allows us to identify
the row and column that intersect at the target symbol. The The classification procedure used in the MindGame system
idea that guided the presented study was to extend the most relies on supervised learning. We acquired labeled training data
common P300-based approach in BCI research to a game paradigm using a simple oddball task designed as a two-dimensional image
to provide richer interaction possibilities, in particular to study matrix containing 3 × 4 images of comic characters. The basic
the role and adaptivity of neurofeedback while playing a ‘‘thought task for the classifier is the discrimination between positive epochs
controlled‘‘game. We employed an adaptive classifier for the belonging to a target stimulus, i.e. the P + class and negative epochs
identification of P300 events from a scalp electrode array, together belonging to a background stimulus, i.e. the P − class. A training
with a training scheme, for optimizing the classifier to each player. session consisted of 500 subtrials, leading to a dataset containing
Additionally, we incorporated gradual feedback into the game to 500 P + epochs and 5500 P − epochs. A balanced dataset was used
explore the benefits of cognitive control. for training, such that 500 epochs were randomly selected from the
Paradigm. The MindGame is played on a checkerboard-styled whole set of P − epochs, resulting in a total of 1000 feature vectors.
game board with 28 × 18 fields and 12 randomly positioned trees. These vectors are comprised of 10 EEG channels with an epoch
A ‘‘comic character’’ that is able to move from field to field is placed
on this board. Fields with trees constitute potential target stimuli in
terms of the oddball paradigm. Starting from the front of the board,
1 The following BCI-specific terminology is common: The time window following
the player’s task is to move the character from tree to tree along
each stimulus flash – the base unit for the translation algorithm - is called an epoch.
the z-axis, until each tree has been visited. The brain interface for The inter-stimulus interval (ISI) is the time span between two successive stimulus
the movement of the character is realized by exploiting the trees onsets. The sequence of highlighting all 12 stimuli is a subtrial, in the MindGame
as potential target stimuli, in an analogous fashion as the symbols equal to one turn.
 A. Finke et al. / Neural Networks 22 (2009) 1329–1333 1331

length of 800 ms. Given a sample rate of 256 Hz, this constitutes Table 1
vectors of the dimension d800 · 256/1000e· 10 = 2050. Each vector Mean results from the linear and the quadratic classifier. Results are averages over
11 subjects and 5 foldings from cross-validation.
belongs to one stimulus highlighting.
The classification should be efficient and cause low computa- FLDA + PCA QDA + PCA

tional cost for testing and, with regard to a future extension to an Accuracy 0.849 ± 0.030 0.735 ± 0.027
online-learning capable system, training. In the presented study Sensitivity 0.855 ± 0.029 0.649 ± 0.045
we applied Fisher’s Linear Discriminant Analysis (FLDA) (Fisher,
1936) for classification. The FLDA requires a matrix inversion of the mean, µ+ , of all positive epochs, which is calculated on the whole
pooled covariance matrix and thus suffers from nearly singular ma- trainset. With these references, a measure for the quality of the
trices in very high-dimensional spaces. Consequently, we added an P300 is incorporated into the game, determined by an average
extra dimension reduction step to our method. quality represented by µ+ and a maximal quality represented by
Principle component analysis. Calculating PCA on the data yref .
acquired during the training session revealed that only 137 to In the online MindGame session the data matrix is created after
146 principal components depending on the subject accounted for every subtrial. The classification function returns a value for the
99.9% variance. This constitutes a dimension reduction from 2050 projection of each feature vector onto the weight vector together
to 142 on average. Fig. 2 shows a tempo-spatial plot of the 16 with a class label. The value of the target projection is mapped
largest principal components (PC) with respect to the eigenvalues by the translation algorithm onto a stepwidth that will determine
of the data covariance matrix as obtained from the training data set the next move of the character in the game. The distance between
of one subject. The first PCs represent specific frequency bands for the class border and the maximal projection yref is divided into
all electrode locations with higher frequencies found in the later bins, each bin representing a different stepwidth of the character.
PCs. In later PCs different activation for the electrode sites occurs. The basic idea of the control algorithm is to reward projection
This basic pattern is stable across sessions and subjects. Hence, the values larger than the projection of the class mean obtained from
directions of the basis vectors of the subspace are approximately the training. Hence, the distance between the class border and
the same for all data sets recorded under the same experimental the projection of the mean is divided only into two equally sized
conditions. This finding encourages the usage of a pre-computed bins, the first bin containing all projections in the range 0 < y <
PCA matrix for online classification. Thus, we calculated the PCA 0.5 · µ+ and the second bin covers the range 0.5 · µ+ ≤ y < µ+ ,
matrix needed for online classification on the training data matrix given that all projections have been moved such that the class
and stored it for later usage in the game. border is zero. The distance between µ+ and yref is divided into
Classification. For classifier training, balanced sets were applied, smaller, also equally sized bins to allow for finely grained steps.
containing 500 epochs of the P + and 500 epochs of the P − class. The granularity, i.e. the number of bins, is an adjustable parameter
in the MindGame.
Data was bandpass filtered 0.5 to 10 Hz and scaled to an interval
of [−1, 1]. A five-fold cross-validation was applied to assess the Experimental setup. The MindGame implementation was evalu-
classification accuracies. FLDA has only one parameter, the bias, ated in an experimental series with 11 subjects, 9 male, 2 female,
which was assessed by scanning a total of 400 values; the final aged 24–39 years. 10 electrodes were applied to the scalp accord-
bias used later for classification was calculated as the mean of ing to the international 10–20 system at the locations Fz, Cz, Pz,
the 5 values obtained from the foldings. In the case of identical Oz, C3, C4, P3, P4, PO7 and PO8 and referenced to both ear lobes.
covariance matrices the bias can be calculated directly as the The data was sampled at 256 Hz using the EEG amplifier Mindset24
mean of the projections of the two class means onto the weight by Nolan Computer Systems LLC. The subjects completed the train-
vector. However, the assumption of identical covariance matrices ing session and the MindGame session consecutively in one exper-
for the two classes in the MindGame system cannot be made. The imental block. The MindGame session consisted of 4 rounds of the
relation between the duration of one epoch, 800 ms, and the ISI, game, each round having a variable length, i.e. a variable number
120 ms, leads to highly overlapping epochs. All positive epochs of subtrials, depending on the subject’s performance.
start at the onset of the target stimulus, they all contain the P300
waveform in an almost identical temporal course, leading to a 4. Results
small variance of the P300 class, caused mainly by noise in the
data and a minor variability in the P300 between subtrials. The Offline classifier evaluation. The performance of the classifier was
negative epochs, in contrast, are subject to large variance, because estimated from the cross-validation of all 11 training data sets.
up to six negative epochs in one subtrial following the target Data sets were preprocessed with PCA and classified with FLDA as
stimulus contain a different part of the P300, while the preceding described in the previous section. Table 1 lists the overall means
ones do not contain any P300 at all. These considerations imply for all subjects. For comparison, a Quadratic Discriminant Analysis
that distinct covariance matrices must be assumed and a cross- (QDA) which can by itself cope with unequal covariance matrices
validation strategy is needed. was also tested. Data preprocessing was computed in the same
Game control. The control mechanism in the game relies on fashion as in the previous case. Obviously, the accuracies obtained
reference values obtained from the training data. These values with a quadratic discriminant stay clearly below those achieved
serve as the basis for the evaluation of the P300. Two reference with linear separation mean accuracy 0.735 ± 0.027 in case of QDA
values are needed for the P + class to adjust the stepwidth, the with PCA versus 0.849 ± 0.030 for FLDA with PCA). The pitfall with
largest projection and the projection of the class mean. They QDA seems to be an overfitting effect on the training data leading to
are obtained from the training data during the cross-validation worse generalization capabilities on the test data. Apparently, the
procedure. Within each folding, the projections of the testset data general properties of the class distributions are best represented
onto the weight vector obtained from the trainset are calculated. with a linear class border.
= maxi wT xi , i =


From each fold k, the maximal projection ymaxk Online gaming. A mean single-trial classification rate of 0.659 was
1, . . . , n is obtained, where n is the number of rows in the testset achieved, considering all four rounds of the MindGame from all
matrix. The reference value is then calculated as the mean of these subjects. Compared with the prior probability of p = 0.08, this is a
maximal projections with the class border moved to zero: yref = value clearly beyond chance. A mean classification rate of this scale
1
P5 max
5 k=1 yk + b. The second reference is the projection of the is actually very high for an online operated BCI. Fig. 3a shows the
1332 A. Finke et al. / Neural Networks 22 (2009) 1329–1333

PC 1 PC 2 PC 3 PC 4
2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
200 400 600 200 400 600 200 400 600 200 400 600
PC 5 PC 6 PC 7 PC 8
2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
200 400 600 200 400 600 200 400 600 200 400 600
PC 9 PC 10 PC 11 PC 12
2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
200 400 600 200 400 600 200 400 600 200 400 600
PC 13 PC 14 PC 15 PC 16
2 2 2 2
4 4 4 4
6 6 6 6
8 8 8 8
10 10 10 10
200 400 600 200 400 600 200 400 600 200 400 600

Fig. 2. The first 16 principle components (PC) representing the 16 largest eigenvalues of the covariance matrix calculated from data of subject 6. Apparently, the PCs reflect
mainly activity in certain frequency bands with increasing frequency towards higher PCs. The labels on the y-axes refer to different electrode locations in the order: Fz, Cz,
Pz, C3, C4, P3, P4, PO7, PO8, Oz. The x-axes show the time of one epoch in ms.

1
0.9
3
0.8
0.7
0.6
feedback
accuracy

0.5 2
0.4
0.3
0.2
1
0.1
0
1 2 3 4 5 6 7 8 9 10 11 0.4 0.5 0.6 0.7 0.8 0.9 1
subject accuracy

Fig. 3. (a) Comparison of the mean classification results obtained in offline classifier training (blue line) and the mean classification results obtained in online classification
during the game session. (b) Relation between classification accuracy and feedback. Large steps resp. projections correlate with high classification rates.

single-trial accuracies as obtained during the online operation of obtained from the training data. This boundary corresponds to a
the MindGame system for all subjects. For comparison, the offline feedback stepwidth of 3. Hence, it was desired to achieve a mean
accuracies obtained from the training sets are also plotted. performance above this boundary. An overall mean of 3.23 steps
Usually, P300-based BCIs rely on averaging across several was achieved, the means over all subjects for the particular rounds
subtrials, using either a static or a dynamic number of subtrials. also exceeded the boundary of 3 steps, though in the second round
The classification of one target is then calculated as the average only marginal. Thus, with the interactive MindGame paradigm
of the subtrials. In the work of Lenhardt et al. (2008), where classification results that are equal or even superior to a passive
a dynamic subtrial limitation algorithm was used, an accuracy standard oddball paradigm can be achieved.
of 0.6 could not be achieved before the third subtrial. In the
MindGame approach all classifications are computed purely on 5. Conclusion
single-trial level. However, given a standard deviation of 0.146,
the results expose a very high variance among subjects which is Brain–Computer Interfaces (BCIs) were originally developed as
typical for online operated BCIs. A combination of the stepwidth communication devices for severely paralyzed patients. Within the
and the single-trial classification rate determined the duration scope of the presented study we developed a BCI based on the
of each round of the game. Classification rate is plotted against P300 event-related potential as a device for game control, where
the stepwidth in Fig. 3b. The number of subtrials the subjects signals from the brain replace the common control devices like
accomplished before reaching the goal of the game varied between mice or joysticks. Thus, the game is controlled without any motor
a minimum of 47 subtrials for subject 5 in the 4th round up to actions. The current approaches in BCI research targeting the field
149 subtrial for subject 10 in round 2. We were interested if the of multimedia control utilize several different EEG components,
subjects’ performance would improve in the game where active but none of them the P300. While game and multimedia control
feedback was constantly given, compared to the performance in with a BCI is inherently a challenging field, it potentially also
the training session without feedback. The critical boundary or opens the door for interesting applications in a clinical context.
reference value for defining improvement was the projection of The proposed system implements a gradual control mechanism
the mean vector of the positive epochs onto the weight vector as for a game where the gradual decisions are derived from the
 A. Finke et al. / Neural Networks 22 (2009) 1329–1333 1333

output of the FLDA classifier. Following the hypothesis that the Datta, A., Cusack, R., Hawkins, K., Heutink, J., Rorden, C., Robertson, I. H., & Manly,
P300 potential is a marker of attention, the MindGame could in T. (2007). The P300 as a marker of waning attention and error propensity.
Computational Intelligence and Neuroscience, 2007.
the future serve as a neurofeedback system allowing for training Dornhege, G., del R Millán, J., Hinterberger, T., McFarland, D., & Müller, K.-R. (2007).
attention. Showing true neurofeedback effects requires long-term Toward brain–computer interfacing. MIT Press.
Duncan-Johnson, C., & Donchin, E. (1977). On quantifying surprise: The variation
studies with multiple subjects and multiple sessions and is beyond of event-related potentials with subjective probability. Psychophysiology, 14,
the scope of this work. However, we believe that the insight gained 456–467.
can guide further development of similar or more challenging Farwell, L., & Donchin, E. (1988). Talking off the top of your head: Towards a mental
prosthesis utilizing event-related brain potentials. Electroencephalography and
games. An appealing setting for future research is the development clinical Neurophysiology, 70, 510–523.
of games which can be played competitively by two or more Fisher, R. (1936). The use of multiple measurements in taxonomic problems. Annals
players and thus allowing for a competition in terms of cognitive of Eugenics, 7, 179–188.
Lenhardt, A., Kaper, M., & Ritter, H. (2008). An adaptive P300-based online
performance. brain–computer interface. IEEE Transactions on Neural Systems and Rehabilita-
tion Engineering, 16, 121–130.
Polich, J. (1987). Task difficulty, probability, and inter-stimulus interval as
Acknowledgement determinants of P300 from auditory stimuli. Electroencephalography and Clinical
Neurophysiology, 68, 311–320.
Polich, J. (2003). Detection of change: Event-related potential and fMRI findings.
Andrea Finke gratefully acknowledges the financial support Boston: Kluwer Academic Press, (Chapter 5).
from Honda Research Institute Europe for the project ‘‘A Brain- Polich, J. (2007). Updating P300: An integrative theory of P3a and P3b. Clinical
Neurophysiology, 118, 2128–2148.
Robot Interface for Controlling ASIMO’’. Serby, H., Yom-Tov, E., & Inbar, G. F. (2005). An improved P300-based Brain-
Computer Interface. IEEE Transactions on Neural Systems and Rehabilitation
Engineering, 13, 89–98.
References Wolpaw, J. R. (2007). Brain–computer interfaces as new brain output pathways.
Journal of Physiology, 579, 613–619.
Bell, C. J., Shenoy, P., Chalodhorn, R., & Rao, R. P. N. (2008). Control of a humanoid Wolpaw, J. R., Birbaumer, N., McFarland,, Dennis, J., Pfurtscheller, G., & Vaughan,
robot by a noninvasive brain–computer interface in humans. Journal of Neural T. (2002). Brain–computer interfaces for communication and control. Clinical
Engineering, 5, 214–220. Neurophysiology, 113, 767–791.