PeyeDF: an Eye-Tracking Application for Reading and

Self-Indexing Research

MARCO FILETTI, University of Helsinki, Finland

arXiv:1904.12152v1 [cs.HC] 27 Apr 2019

HAMED R. TAVAKOLI, Aalto University, Finland

NIKLAS RAVAJA, Aalto University, Finland and University of Helsinki, Finland
GIULIO JACUCCI, University of Helsinki, Finland
PeyeDF is a Portable Document Format (PDF) reader with eye tracking support, available as free and open
source software. It is especially useful to researchers investigating reading and learning phenomena, as it
integrates PDF reading-related behavioural data with gaze-related data. It is suitable for short and long-
term research and supports multiple eye tracking systems. We utilised it to conduct an experiment which
demonstrated that features obtained from both gaze and reading data collected in the past can predict reading
comprehension which takes place in the future. PeyeDF also provides an integrated means for data collection
and indexing using the DiMe personal data storage system. It is designed to collect data in the background
without interfering with the reading experience, behaving like a modern lightweight PDF reader. Moreover, it
supports annotations, tagging and collaborative work. A modular design allows the application to be easily
modified in order to support additional eye tracking protocols and run controlled experiments. We discuss the
implementation of the software and report on the results of the experiment which we conducted with it.
CCS Concepts: • Information systems → Document collection models; Open source software; Desktop
search; Data stream mining; Ontologies; • Hardware → Emerging tools and methodologies;
Additional Key Words and Phrases: eye tracking, reading, personal data storage, PDS, learning, support vector
machines, SVM
ACM Reference Format:
Marco Filetti, Hamed R. Tavakoli, Niklas Ravaja, and Giulio Jacucci. 2019. PeyeDF: an Eye-Tracking Application
for Reading and Self-Indexing Research. 1, 1 (May 2019), 26 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1 INTRODUCTION
Eye tracking is a valuable, non-invasive means of investigation. It has seen increased adoption
rates in the past decade, thanks to gradual but steady technological and cost-related improvements
[35]. Many fields of research have been taking advantage of its capabilities, ranging from design
[19], collaboration [22], learning [29], cognition [11, 40, 68] and usability [35].
To date, the experiment design, experiment procedure and utilization of the eye tracking devices
have been a laborious task for users and researchers, in particular for a long run experiment. In many
circumstances, most of the users fail having a tangible connection with the software user interface
and the researchers have to spend huge amount of time on implementing the experiments using eye
tracking software. In the context of providing an appealing user experience and easier experiment
design procedure, in particular for long run experiments, we present PeyeDF, an open-source
eye-tracking enabled reading tool. PeyeDF is suitable for the conduction of eye-tracking-based
Authors’ addresses: Marco Filetti, University of Helsinki, Helsinki Institute for Information Technology HIIT, Gustaf
Hällströmin katu 2b, Helsinki, Uusimaa, 00014, Finland, [email protected]; Hamed R. Tavakoli, Aalto University,
Department of Computer Science, Espoo, Finland, [email protected]; Niklas Ravaja, Aalto University, Department
of Information and Service Economy, Helsinki, Finland, University of Helsinki, Department of Social Research, Helsinki,
Finland, [email protected]; Giulio Jacucci, University of Helsinki, Helsinki Institute for Information Technology HIIT,
Helsinki, Finland, [email protected].

2019. XXXX-XXXX/2019/5-ART $15.00

https://doi.org/10.1145/nnnnnnn.nnnnnnn

, Vol. 1, No. 1, Article . Publication date: May 2019.

2 Filetti et al.

reading and learning experiments conducted over long periods of time. For users, PeyeDF is easy
to use, lightweight, stable software that provides support for storing easily-performed manual
document annotations. For experimenters, it provides out-of-the box support for multiple eye
tracking platforms.

1.1 Related work

Deciphering the state of one’s mind by performing inferences on eye movements is a research
topic that has been undertaken for various purposes. The span of this research area is wide-range
and includes observer task inference [12], tagging image pleasantness [47], inferring traits [34],
predicting one’s age [70], information retrieval [48], etc. Reviewing all the related work is beyond
the intent of this article, for brevity, we refer the readers to [22]. A brief account of the most relevant
work is, however, presented.
Investigating the relationship between eye movements and learning can provide great insight
into cognitive functions. To investigate one’s cognitive performance during learning, reading has
been a central focus in conjunction with the eye tracking. For example, it has been demonstrated
that the duration and number of fixations increase along with the difficulty of the text being read
[49]. Such fixation-level measurements is utilised to assess how sentence construction affects
comprehension difficulty [64] or to estimate the reader’s language expertise [39]. More examples
are given in a review by [51].
Within the concept of learning via “Lifelogging”, storing our data in personal databases [28]
and retrieving previously manually annotated information is demonstrated to reinforce memory
[44]. Annotations might also be performed automatically with attentive documents [15]. Both
are valid strategies for the location of important passages that exploit manually or implicitly
(automatically) annotated information. One drawback of automated annotation approaches is that
the reinforcement of learning induced by the act of manually annotating passages is lost. In this
context, the possibility of manually annotating passages is a feature that must be provided by all
learning platforms [2]. The PeyeDF tries to alleviate this shortcoming by allowing both manual
and implicit annotation schemes.
Electronic devices have introduced new research questions for learning, such as the investigation
of screen-based reading behaviour. There are opposing views in favour [17, 42] and against such
technologies [1, 43] in learning and education. Many factors such as meta-cognitive abilities, culture,
familiarity, habit, improved display technologies and age are argued to influence performance of
users in digital reading [4, 25, 63]. While electronic reading is gradually replacing paper-based
reading [2, 42], the reading experience is key to success in reading. In other words, being comfortable
with digital tools can improve learning [31]. We, thus, need to have well-designed software, akin to
PeyeDF, for learning studies.
Designing software for the collection of user data during reading and learning can be cumbersome;
a number of software systems have been developed to facilitate such tasks. These are summarised
in Table 1.
Adele is a e-learning suite that allows synchronisation of eye tracking data with course material
[29]. Text 2.0 [9] and Eyebook [8] are two frameworks for web browsers designed to provide
interactive feedback to gaze data. These two systems, along with that presented by [16], can distin-
guish between skimmed versus read text. Eye an Pen [3] enables synchronisation of handwriting
with gaze data. Ogama is a system for slideshow study designs [67]. iTrace is an environment
that enables eye tracking research on software systems, rather than static text [56]. Eye tracking
software prototypes have also been developed for tablets and smartphones, although these are
limited in accuracy and speed due to current hardware [38, 69].

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 3

Task Tracer is a Windows application for tracking user activity across various tasks [21]. It
provides access to an open database that stores computer usage data. “Stuff I’ve Seen” is a system
for personal data storage and retrieval [23]. Presto is a document storage system that organises
material based on its contents, in addition to the traditional folder-based approach [20]. Xlibris is a
tablet-based proprietary system for document storage and annotation [54]. Note that some these
applications do not support eye tracking and are not open-source. They are, thus, not among the
best choices for researchers.
Among the open-source applications, Varifocal Reader is an application that allows navigation
of large text documents at various level of detail (e.g. headers, sub-headers) [37]. It is aimed at
exploring individual large documents (books), while PeyeDF is aimed at storing information about
collections of documents of any size. GazeParser is an open-source library for low-cost eye tracking
and data analysis; it consists of a video-based eyetracker and libraries [59]. It is not aimed at reading
nor does it support long-term data storage and retrieval. Eyerec [53] and openEyes [41] are two
open-source hardware specification for eye tracking.
To summarize, the current eye tracking-based systems, in one hand, do not support personal
data storage (PDS) and can not support lifelogging. On the other hand, the PDS based systems do
not support any eye tracking and can not be used in eye tracking research. The PeyeDF software
addresses these shortcommings by providing support for eye tracking and PDS. Moreover, it is an
open source, lightweight PDF reader that can be customized for numerous research purposes.

Table 1. Comparison of software related to PeyeDF

Name Reference Purpose Eye Tracking PDS* Open-source

Adele [29] Learning ✓
Text 2.0 [9] Reading ✓
Eyebook [8] Reading ✓
Eye and Pen [3] Learning ✓
Ogama [67] Presentations ✓
iTrace [56] Programming ✓
Eyetab [69] Tablets ✓
My Reading Life [38] Phones ✓
Varifocal Reader [37] Large documents ✓ ✓
Task Tracer [21] Office work ✓
Stuff I’ve Seen [23] PDS ✓
Xlibris [54] Tablets ✓
GazeParser [59] Software library ✓ ✓
Eyerec [53] Hardware ✓ ✓
Openeyes [41] Hardware ✓ ✓
PeyeDF Reading ✓ ✓ ✓
*Supports or includes a Personal Data Storage system.

1.2 Contribution
We introduce PeyeDF. It is an open-source eye-tracking enabled reading tool. The PeyeDF is
designed to support personal data storage systems (PDS) and eye tracking support in combination
with a lightweight document viewer that enhances user experience during learning experiments.
Some of the unique properties of the PeyeDF includes,
• tracking viewport position,

, Vol. 1, No. 1, Article . Publication date: May 2019.

4 Filetti et al.

• tracking zoom factors,

• support for multiple eye tracking devices,
• implicit data annotation
• extendable modules,
• supporting DiMe, a personal data storage system,
• lightweight and user friendly interface
The PeyeDF eases the life of both experimenters and users in long run reading and learning
experiments. It has been tested over a long (3+ years) at Aalto University, Finland and to run the
experiment described in section 5. It can be also used as a stand alone self-indexing system when
an eye tracker is not available.

2 IMPLEMENTATION
PeyeDF is available only for macOS and it has been highly optimised for the platform. All computa-
tion related to eye tracking data and behavioural data collection takes place in the background, so
that the main thread (which updates the User Interface) is left unburdened, resulting in a smooth
user experience; this is enabled by the implementation described in section 2. It also supports
spotlight (described in subsection 3.3) and URL Types (explained in subsection 4.3). It implements
the Swift API guidelines1 and the Apple Human Interface guidelines2 .

2.1 Installation
PeyeDF is a standard macOS application; as such, it can be installed by downloading a disk image
from https://github.com/HIIT/PeyeDF/releases and dragging PeyeDF to the Applications folder.
However, in order to function it requires the installation of DiMe and, optionally, a connection to a
supported eye tracker. These steps are described below. An overview of the full system is given in
Figure 1.

2.2 Prerequisites
PeyeDF requires DiMe for data storage and retrieval. DiMe is an open-source Personal Data Storage
(PDS) system that supports multiple applications and data types [58]. DiMe itself requires the
following software to be installed in order to be compiled.
• Xcode3 or Command Line Tools 4
• Java SE5 JDK version 8 or above
• Node.js, downloadable from the web6 or via Homebrew7 , if installed

2.3 Setup
Once the required software is installed, DiMe should be compiled and run. PeyeDF must then be
configured to utilise DiMe using a predefined user and password. The procedure to do so is detailed
below and the related code can be pasted into a Terminal (note that individual commands are
separated by ‘;’ and otherwise each statement should be written in a single line).
(1) Clone the open-source DiMe repository:
git clone --recursive https://github.com/HIIT/dime-server
1 https://swift.org/documentation/api-design-guidelines/
2 https://developer.apple.com/design/human-interface-guidelines/macos/overview/themes/
3 https://itunes.apple.com/app/xcode/id497799835
4 xcode-select --install
5 http://www.oracle.com/technetwork/java/javase/downloads
6 https://nodejs.org
7 brew install node

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 5

Fig. 1. System overview. PeyeDF requires two way communication with DiMe. It supports the listed protocols
for eye tracking communication. As discussed in subsection 4.7, the EyeDataProvider protocol generalises
eye tracking support. It is also possible to export / import eye tracking and reading data via JSON files.

(2) Run DiMe:

cd dime-server; make run
(3) DiMe is ready Once ‘fi.hiit.dime.Application: Started’ appears; first-time compilation may
take a few minutes.
(4) Navigate to http://localhost:8080 and create a user. For a test run, use Test1 as user and 123456
for password as these are the PeyeDF defaults.
(5) PeyeDF can now be started and should be fully functional. If a different user rather than the
suggested Test1 was created during the previous step, navigate to PeyeDF Preferences, select
the ‘DiMe’ tab and enter the chosen username and password.
Dime creates a .dime directory under the user’s home. This directory contains the full database;
deleting this directory will reset DiMe. If the directory is copied or moved to another machine an
installation of DiMe on that machine will be able to read the database (assuming that username
and password match). DiMe can also be installed on a (local) network server so that it can be used
by multiple users simultaneously.

3 USAGE
PeyeDF works and behaves like a lightweight PDF reader – similarly to the stock ‘Preview’ applica-
tion on macOS. PDF files can be opened with PeyeDF by default, if desired. Eye tracking support
is not required for the application to work, although it is the main motivation for using PeyeDF,

, Vol. 1, No. 1, Article . Publication date: May 2019.

6 Filetti et al.

Fig. 2. Screenshot and paragraph detection. This figure shows a screenshot of PeyeDF (an instance of
DocumentWindowController) with overlays representing fixations and their related paragraphs. A fixation is
assigned to a paragraph when its height can be contained within 3° of visual angle from the fixation. PeyeDF
stores all fixation and paragraph information in sets of ReadingEvents, as described in subsection 4.1.

along with its data storage and indexing capabilities. A screenshot of the applications is provided
in Figure 2, which also shows how PeyeDF captures the currently read paragraph while using an
eye tracker.
PeyeDF supports two eye tracking hardware systems: SMI8 and Pupil Labs9 . Hardware support
is enabled via data input from the LSL10 , zeroMQ11 and MIDAS12 protocols. If desired, PeyeDF can
be extended to support additional protocols (and hence hardware) as discussed in subsection 4.7.
To activate eye tracking while using PeyeDF, it is sufficient to select the desired protocol in the
Preferences > Experiment dialog. An error message will be displayed if a valid connection was
not be established. All gaze data is fetched in real time; at this stage, PeyeDF does not support
off-line gaze recorders such as GazeParser [60]. If eye tracking is currently being used, a tick for
‘Eye Tracking’ appears in the ‘Connections’ menu.
In the ‘Connections’ menu, a tick also indicates whether PeyeDF is successfully connected to
DiMe. In case a connectivity problem occurs during usage, one of the images displayed in Figure 3
is displayed over the reading area to attract the user’s attention.
PeyeDF collects eye tracking and reading data in the background, without noticeably affecting
the reading experience. This is enabled by asynchronous processing of behavioural and eye tracking
data in background queues (as described in subsection 4.6).

8 https://www.smivision.com
9 https://www.pupil-labs.com
10 https://github.com/sccn/labstreaminglayer
11 http://zeromq.org
12 https://github.com/bwrc/midas

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 7

Fig. 3. Error indicators. These images are shown over the reading area when a connection error is detected,
in order to capture the user’s attention. The ‘Not receiving eye tracking data’ image is also shown when the
user’s eyes have not been detected for an extended period of time (e.g. in case the user left the workstation).

PeyeDF provides support for rapid annotations. Annotation support is critical for reading appli-
cations as actively highlighting passages can increase their retention [24]. It has been previously
suggested that annotations in the digital world may be less effective than paper-based annotation
when they are too complex to use [6]. Providing a means for quickly annotating sections of relevant
text is then beneficial. To enable quick annotations, the ‘quick mark’ button (visible on the toolbar)
should be activated. Annotations are then performed with a double-click on a paragraph to mark
an “important” paragraph in yellow. A triple click would highlight the area as “critical”, using
a stronger colour (red). It is also possible to annotate text by selecting a range of text and right
clicking.
Tagging is also supported: users can assign tags (corresponding to short strings) to any range of
text. Tags are stored in DiMe, and can subsequently searched and retrieved by PeyeDF or other
supporting applications.

3.1 Refinder
It has been suggested that lack of visual cues in digital applications might impede reading [1, 43].
PeyeDF implements a Refinder functionality, which utilises colours, tags and overviews in order
to provide spatiotemporal markers to previously read and annotated text. The Refinder has been
included in PeyeDF for two reasons. Firstly, it improves the reading experience by providing a
simple means to revisit previously highlighted text; this may motivate experiment participants in
utilising PeyeDF rather than a default reader. Secondly, the Refinder has been designed in order to
be extensible: it can be integrated into third party applications (as described in subsection 4.3) or it

, Vol. 1, No. 1, Article . Publication date: May 2019.

8 Filetti et al.

Fig. 4. Refinder screenshot. A screenshot of an instance of RefinderWindowController is shown in this

figure. There can only be one such window in an instance of PeyeDF. Colours represent the type of annotations
created by users (yellow for “important” and red for “critical”). In this example, blue represents unannotated
paragraphs which contained at least three fixations. The colour bars represent the overall proportion of the
given document which is marked in the given colour. Note that in the overview, blocks are overlaid over text,
while in the main window annotations are displayed as a bar on the left side of the related paragraph.

can be modified to investigate novel refinding approaches (all Refinder functionality is collected in
the ‘Refinder’ folder in our repository13 ).

3.2 Collaboration
PeyeDF also supports collaborative reading and tagging. To enable it, two (or more) collaborators
must be connected to the same local network. Detected collaborators can be displayed by activating
the Connections > Show network readers menu. One can then initiate collaboration by pressing
the ‘invite’ button. Collaboration also requires DiMe. This allows users to “track” one another, if
desired, showing exactly which paragraph is being read by the collaborator using eye tracking.
Real time sharing of gaze information has been shown to increase collaboration quality during
problem solving [55, 61], collaborative visual search [13] and a planning task [18]. It has also been
shown to be useful, for example, in remote non-collaborative scenarios, such as virtual lectures
[57] or virtual conferencing [66]. Support for collaborative work in reading software is a desirable
feature according to [2].
Application elements unique to the collaboration feature are shown in Figure 5. This feature is
provided to keep the application on par with modern office packages and potentially allows for
research in collaborative environments, with and without eye tracking.

3.3 Spotlight integration

A minor but useful feature is spotlight integration. When searching for text on a macOS system,
opening a PDF with PeyeDF will open it while searching for the previously searched text. Searched-
for text is stored in SummaryReadingEvents. Events are explained in the next section.
13 http://github.com/HIIT/PeyeDF/tree/master/PeyeDF/Refinder

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 9

Fig. 5. Collaboration features. On the left, the list of active connections is shown; in this case, there is only
one active connection. This displays the filename and title of the document currently being read by the
collaborator(s). This item will also display a ‘Track’ checkbox upon the start of a collaborative reading session;
this allows one collaborator to follow the gaze and / or reading viewport of the other. On the right, the
overview of the collaborative session. Colours identify sections of the document which have been read by
the local user, the collaborator, or both. This overview is a small window displayed along with the main
DocumentWindowController

4 DATA STRUCTURES AND APIS

This section describes the format of PeyeDF data structures, APIs, events and their usage. Figure 6
summarises the data types supported by DiMe. This information is also documented in the dime-
server repository14 and in the corresponding wiki pages15 .
For a generic user, accessing the stored data of an experiment is likely the most critical function-
ality of the software. Data can be retrieved in two ways: (1) by right clicking an entry (session) in
the Refinder and choosing ‘Extract JSON’, which allows to save all ReadingEvents related to the
selected session in a JSON file, and (2) by utilising the DiMe API, to be described in subsection 4.2.
While the first approach is relatively easy, the second requires better understandings of the internals
of the software, and it also enables one to further extend the software.

4.1 Data Types: Events and Information Elements

There exists two main data types, which should be distinguished from each other, (1) Events and (2)
InformationElements. Events correspond to snapshots of user activity, while InformationElements
14 https://github.com/HIIT/dime-server/tree/master/src/main/java/fi/hiit/dime/data
15 https://github.com/HIIT/PeyeDF/wiki/Data-Format

, Vol. 1, No. 1, Article . Publication date: May 2019.

10 Filetti et al.

Event DiMeData

timeCreated: Date Tag

start: Date InformationElement
timeModified: Date
end: Date text: String
user: User
duration: Double uri: String time: Date
tags: Tag [0..*]
targettedResource: InformationElement plainTextContent: String auto: Boolean
… … weight: Double
title: String
isStoredAs: String …
appId: String
ReadingEvent Rect contentHash: String
Size

sessionId : String width: Double ReadingTag

origin: Point
previousSessionId: String height: Double rects: Rect [1..*]
size: Size
dpi: Integer
pageIndex: int
pageNumbers: Integer [1..*]
readingClass: int
pageLabels: String [1..*] ScientificDocument
classSource: int
pageRects: Rect [1..*]
plainTextContent: String
pageEyeData: PageEyeData [0..*] authors: Person [0..*]
floating: Boolean
plainTextContent: String booktitle: String
scaleFactor: Double
screenDistance: Double Point keywords: String [0..*]
unixt: Long [1..*] firstPage: int
x: Double lastPage: int
SummaryReadingEvent y: Double year: int
address: String
proportionSeen: Double publisher: String
PageEyeData
proportionRead: Double volume: int
proportionInteresting: Double doi: String
Xs: Double [1..*]
proportionCritical: Double Person
Ys: Double [1..*]
readingTime: Double
Ps: Double [1..*]
foundStrings: String [0..*] firstName
startTimes: Long [1..*]
lastName
endTimes: Long [1..*] Legend
middleNames
durations: Double [1..*]
…
pageIndex: int Inheritance

scaleFactor: Double Relationship
unixt: Long

Fig. 6. Data class diagram. Shows the relationships and format of data classes that PeyeDF exports. DiMeData
is top-level class from which all other inherit; this is used in PeyeDF to identify classes which can be stored in
DiMe. Field names were designed to be self-explanatory. Event is the most relevant class for retrieving eye
data (along with its subclasses); this is explained in subsection 4.1. If needed, more details are provided, in
the dime-server repository data folder14 or the PeyeDF repository wiki15 .

represent documents or other information retrievable with a URI. In PeyeDF, there is only one value
for InformationElement, which is ScientificDocument, a link to the PDF files to be retrieved.
An InformationElement contains also a ContentHash and an appId. The first, a SHA hash of the
textual content of a PDF, is used to identify textual similarity between PDFs. appId is simply a
ContentHash with ‘PeyeDF_’ prefixed to distinguish between PeyeDF and other applications.
The data from experiments will be mostly stored in the Event and its variants. We, thus, dedicate
most of this section to the Events data type. The software consists of several event data structures,
namely Event, ReadingEvent, and SummaryReadingEvent. The first and most important is Event
that contains Event.targettedResource field which contains the associated element of a given
event, i.e., a pointer to InformationElement data type corresponding to the current event.
During the software operation, when a user wants to access a document for reading, the reading
event is triggered to handle the retrieval of the relevant document and starts recording the action of
reading. ReadingEvent contains data collected from the moment the user started the act of reading
until the reading stopped. A ReadingEvent may last a few seconds or minutes. We assume reading
action is stopped as soon as the window is scrolled or moved. A more detailed explanation of how
we estimate whether the user is reading is provided in subsection 4.6.

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 11

Fig. 7. Viewports. This figure displays the position of the viewport under different page arrangements. Note
that origin is always at bottom left. When the viewport covers more than one page, the pageLabels and
pageNumbers lists of a ReadingEvent will contain more than one value. The crop box represents the visible
section of a PDF page (if a page has not been previously cropped, the crop box will correspond to the full
page size). Viewports are specified using Rects within each ReadingEvent. Rects representing viewports are
identified using a readingClass of 10 and a classSource of 1.

Each ReadingEvent has a sessionId in terms of a string UUID in PeyeDF. During a reading
session, the same sessionId is allocated to all ReadingEvents of that specific reading session.
A session starts when the user opens a PDF document and terminates when the PDF is closed.
Normally, one session comprises many events.
A ReadingEvent has the following fields: pageNumbers, pageLabels, pageRects and plainTextContent
to convey the visible information within a session. pageNumbers and pageLabels contains the
same number of elements and represent which page(s) that were visible to user. Page number is
intended to be a page index starting from 0, while page label is the page number written on the page
extracted from the PDF. If page label can not be extracted from PDF source the same indexing as
page index will be used. plainTextContent contains all text that was visible on in the viewport(s).
pageRects is an array of Rects, which represent positions of paragraph of various nature. These
specify the viewport position when the event was generated and other paragraph-level information.
Viewports are illustrated in Figure 7.
To distinguish Rects of different nature, the two fields of readingClass and classSource are
introduced. The classSource defines what is the source of a Rect, e.g., eye tracker, a click, a
viewport, etc. readingClass contains the information about the importance of the event, e.g., an
important pargraph vs. a regular one.
During eye tracking, a ReadingEvent can include one or several PageEyeData fields to contain
information for fixation information. The fixations’ data is specified within multiple in-order arrays
(e.g. horizontal and vertical positions, pupil size, duration). Their coordinates are specified in page
space, with pageIndex identifying the page on which these took place. There is a pageEyeData
entry for each page for which gaze data were detected. When using eye tracking, PeyeDF detects the
paragraph of interest for the given fixation, using the implementation described in subsection 4.5.

, Vol. 1, No. 1, Article . Publication date: May 2019.

12 Filetti et al.

A SummaryReadingEvent is triggered once a document is closed, i.e., a reading session is termi-

nated. In consequence, the PeyeDF computes a summary for the reading session and adds it to the
database. This information will be further used for future retrievals. The summary information
consists of all the searched keywords and search hits in a document, along with their statistical
information that may be useful for further relevance information retrieval within documents.

4.2 DiMe API

DiMe is the central backbone of PeyeDF data storage. PeyeDF populates DiMe using HTTP Rest
API calls (POST or DELETE) in JSON; these data can be subsequently retrieved by consumers using
the GET calls described in this subsection. PeyeDF itself uses GET calls for displaying data within
the Refiner. The full DiMe API and configuration instructions are described by [58].
Using the DiMe API, one can query the data structures discussed above. The queries useful in the
PeyeDF context are: data/event, data/events, data/informationelement, data/informationelements,
eventsearch and search. The singular endpoints (data/event and data/informationelement)
are used to retrieve a single event or element as JSON. These terminate with the unique integer id
that one wants to retrieve (e.g. data/informationelement/1 will fetch the first element stored in
DiMe). The plural data endpoints retrieve 0 or more JSON objects and can be filtered using GET
parameters. These are discussed more in detail in the next two subsections.

4.2.1 data/events. This endpoint is used to fetch ReadingEvents. This is used when fetching
reading data created by PeyeDF in the format described in section 4. Parameters can be used to
filter by type, actor, sessionId, elemId or contentHash. Query parameters can be chained.
Possible queries:
• type allows to filter by type. PeyeDF uses two types: #ReadingEvent and #SummaryReadingEvent.
Note that the type has to be prefixed by http://www.hiit.fi/ontologies/dime/ (this is
to distinguish between types created by different organisations).
• actor filters by the “creator” of the event. In the case of PeyeDF, there is only one actor:
PeyeDF.
• sessionId as previously indicated, each reading session, per document (from window open
to window close) is assigned one sessionId. This query parameter can be used to fetch all
events generated during a single session. This is an alphanumeric string.
• elemId filters only events related to a given information element Id (an integer). This is used
to obtain all events related to an individual PDF file.
• contentHash retrieves all events which were related to a specific file. Using hashes to identify
files allows them to be found after renaming or moving.

Example. A full query could then be http://localhost:8080/api/data/events?type=http:

//www.hiit.fi/ontologies/dime/#ReadingEvent&actor=PeyeDF
This query will fetch all events created by PeyeDF. The targettedResource field within each
returned event is used to identify the document related to a given event. This allows one to parse
the list while searching for specific metrics and relate the given event(s) to a document.

4.2.2 data/informationelements. This endpoint fetches Scientific Documents (which are the only
type of information element used by PeyeDF). This endpoint is used to fetch document(s) with a
specific contentHash, of a specific type or with a given tag.
• contentHash is an alphanumeric string uniquely identifying the document or file.
• type; PeyeDF uses only type: #ScientificDocument. As with event queries, PeyeDF prefixes
type with http://www.hiit.fi/ontologies/dime/.

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 13

• tag allows to retrieve only documents for which a user assigned the given tag. The tag can
appear in any part of the document.
Example. http://localhost:8080/api/data/informationelements?type=http://www.hiit.
fi/Fontologies/dime/#ScientificDocument&tag=done fetches all ScientificDocuments (PDF
files) that the user tagged with ‘done’.
4.2.3 eventsearch. To perform textual searches across all events, use the eventsearch endpoint.
The query requires a parameter for the given query; optionally, any of the event filter parameters
described above can be used.
Example. http://localhost:8080/api/eventsearch?query=pipe&type=http://www.hiit.
fi/ontologies/dime/#ReadingEvent looks for ReadingEvents for which the word “pipe” ap-
peared in the viewport. This can be used to retrieves parts of a document which have been read (or
likely to have been read) by the user. The found results can be manually filtered to retrieve only
events which contained fixations, for example.
4.2.4 search. To find documents that contain given the text use the search endpoint.
Example. http://localhost:8080/api/eventsearch?query=pipe&type=http://www.hiit.
fi/ontologies/dime/#ScientificDocument finds all PDFs that contain the word “pipe” in any
part of their body and have been previously opened by the user.

4.3 URL types

PeyeDF also supports the native macOS ‘URL Types’ corresponding to interprocess communication
protocol. That is, third party applications can ask PeyeDF to open documents and optionally focus
on given areas. Is possible to send messages to PeyeDF using URLs starting with the peyedf://
protocol. A custom URL allows for relatively simple interaction between applications.
There are two modes that can be used:
• reader opens a file for a new reading session. It can be used by a search engine to direct to a
specific search result.
• refinder opens a previously read file. Useful for personal indexing services which access
own reading data.
These appear where the host would normally appear in a URL (e.g. peyedf://refinder/. . . )
4.3.1 Reader. The format of the request is:
peyedf://reader/[path_on_disk|contentHash|appId|sessionId]?search=search_query[&page=
n][&rect=(x,y,w,h)|&point=(x,y)]
After reader/ one can refer to a file using a full path. If the file is already known by DiMe, it is
also possible to use a contentHash, a given sessionId or appId.
The search query is optional. If enclosed in double quotes (" or a %22 escape sequence) it is
treated as an exact phrase search (as opposed to an “all words” search).
Optionally, one can focus to a specific block of text using the page and rect parameters. Other-
wise, point and page can be used to focus on a broad area of text (to focus on a section heading,
for example). The page parameter must be present if either point or rect is used. It is also possible
to use page alone, to focus on the beginning of a given page.
Examples. peyedf://reader/Users/marco/Downloads/Wyble\%20et\%20al\%202009.pdf?search=
\%22attentional\%20blink\%22
This opens the ‘Wyble et al 2009.pdf’ file and searches for the “attentional blink” phrase.

, Vol. 1, No. 1, Article . Publication date: May 2019.

14 Filetti et al.

peyedf://reader/Users/marco/Downloads/Yan\%20et\%20al\%202007.pdf?rect=(200,400,
200,100)\&page=3
This opens the ‘Yan et al 2007.pdf’ file and focuses on the given area with origin 200 (x) , 400 (y)
and size 200 (width), 100 (height).
peyedf://reader/Users/marco/Downloads/Yan\%20et\%20al\%202007.pdf?rect=(200,400,
200,100)&page=3
This opens the same file as before, but instead navigates to a point approximately in the middle
of page 2 (or 1, if we start counting from 0).

4.3.2 Refinder. The format of the request is:

peyedf://reader/Users/marco/Downloads/Yan\%20et\%20al\%202007.pdf?rect=(200,400,
200,100)&page=3
sessionId is the unique id assigned to each reading session, as described in subsection 4.1.
Optionally, a specific block of text can be focused upon using the page and rect parameters.
Otherwise, point and page can be used to focus on a broad area of text (to focus on a section
heading, for example). The page parameter must be present if either point or rect is used. It is
also possible to use page alone, to focus on the beginning of a given page.

Examples. peyedf://refinder/4c12e273e7be240f9eca1f?point=(200,200)&page=1
Opens the same reading session and in addition focuses to that point on page 2 (if starting from
1).
peyedf://refinder/c7b4d50487d47ce88?rect=(100,400,200,100)&page=3
Opens another reading session and focuses on the given block of text on page 4 (if starting from
1). The given block of text could be something previously annotated, for example.

4.4 Data collection

We, here, explain the mechanisms that are used for collecting data. We first explain how and when
a ReadingEvent is generated and which parts of the software are involved. We then discuss the
support for eye tracking devices, followed by explaining how an experiment is built using the
software.

4.4.1 ReadingEvent generation. The central classes involved in data collection are HistoryManager,
DocumentWindowController, PDFReader, to be described below. Their interaction is visually de-
picted in Figure 8.
The flow of events begins with Notification events, which are generated by user activity.
The notifications are collected by an active DocumentWindowController, which triggers the
HistoryManager by identifying itself as the centre of the user’s attention. When the software
detects a user has finished reading (e.g. switched to another window), the active controller notifies
the HistoryManager again.
Upon first call of the HistoryManager, it sets up a short timer (entryTimer 2 seconds, defined
in PeyeConstants.minReadTime, that is keep timing meanwhile the user is reading, preventing
noisy data from being recorded. The timer asks the DocumentWindowController to generate a
ReadingEvent, containing the viewport information and eye tracking data when available. This
event will be sent to DiMe later. The timer stops upon exitTimer which is normally triggered only
when eye tracking is off and the user leaves the workstation.

4.4.2 HistoryManager. The HistoryManager class is a singleton class. That is, only one instance
of it is created and is shared across the application. This class acts as a central buffer for be-
havioural and eye tracking data, which are frequently sent to DiMe. These data are stored in the

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 15

Fig. 8. Generation of ReadingEvents. This diagram depicts the flow involved in the generation of
ReadingEvents. As detailed in subsection 4.6, this allows PeyeDF to run smoothly even when large amounts of
eye tracking data are being collected and re-referenced to text and paragraphs. Events are initially dispatched
via application-wide notifications (top right). These notifications are generated by events representing user or
eye tracking activity. This is described in more detail in subsubsection 4.4.1.

currentReadingEvent field; this field is cleared and its previous contents are sent to DiMe every
time exit() is called.
4.4.3 DocumentWindowController. Every document window displayed by PeyeDF is an instance of
DocumentWindowController. It calls the entry() and exit() methods in HistoryManager to record
all the information. Upon appearing, the window sets up a number of Notifications, which
monitor user and eye tracking activity. Every time the user “starts reading”, the DocumentWin-
dowController calls entry() in HistoryManager passing itself as a parameter. This identifies the

, Vol. 1, No. 1, Article . Publication date: May 2019.

16 Filetti et al.

current window as the window the user is currently paying attention to. When the user “stops
reading” exit() is called.
The “start reading” and “stops reading” actions are defined by the type of Notification that are
being triggered. For example, if the window starts moving, loses focus or it is otherwise not visible
it is assumed that the user stopped reading. Similarly, exit() is called when the user’s gaze was lost
for a relatively long amount of time (defined in PeyeConstants.maxReadTime). Otherwise, when
the window stops moving, or a scroll event is terminated, it is assumed that the user started reading,
and entry() is called. This sets up a short timer, (entryTimer, 2 seconds) which upon fires calls the
HistoryManager.entryTimerFire() method, which signals that the user is actively reading. This
generates a ReadingEvent in the background containing PDF information (e.g. viewports), ready
to be sent to DiMe once the user stops reading.
4.4.4 PDFReader. Every DocumentWindowController contains a reference to PDFReader. This is
a subclass of PDFBase, which represents any instance of a PDF file displayed in PeyeDF. PDFReader
is specialised in reading detection and interaction with the user. PDFReader contains the getView-
PortStatus() method, called by HistoryManager.entryTimerFire(). PDFReader also contains conve-
nience methods for creating annotations and tagging while reading. The PDFBase superclass con-
tains anything relatable to a PDF, such as annotations, tags, searching support. Each DocumentWindowControlle
controls one instance of PDFReader.

4.5 Paragraph detection

PeyeDF uses the distance of the user from the screen to determine which paragraph is cur-
rently being read (or annotated) by the user. Distance is obtained from the eye tracker, if avail-
able – otherwise it defaults to 60 cm. When a fixation is detected and it falls within text, the
PDFBase.pointToParagraphRect() is called by DocumentWindowController to determine the height
covered by 3° of visual angle, which should cover abundantly the area covered by the fovea [45].
This only covers the area corresponding to the current textual paragraph; in other words, dou-
ble newlines are skipped at the edges, if present (detected using textual from within the PDF
information).
This information is stored in the EyeRectangle class, which is converted into Rects and sent
to DiMe within ReadingEvents. When this happens, the splitAndCrop function is called to split
EyeRectangles which are too big (as defined by maxHeight). To sum up, this procedure unites
whole paragraphs unless they reach maxHeight.
Algorithms and data structures regarding paragraph operations are defined in Paragraphs.swift16 .
This includes uniting colliding rectangles, and converting Rects stored in DiMe back to annotated
blocks of text.

4.6 Performance
PeyeDF was designed to maximise both usability of the software and reliability of eye tracking data.
High usability is achieved by behaving similarly to a lightweight PDF reader and providing tagging
and Refinder functionality. Eye tracking and behavioural data are collected using background Grand
Central Dispatch17 (GCD) queues. The main queue for eye data processing is called eyeQueue in
HistoryManager. This has a default priority, which in GCD is third, below userInteractive and
userInitiated. Each eye tracker implementation also has its own default priority queue (e.g.
LSLManager has a queue field to which all data callbacks are dispatched).

16 https://github.com/HIIT/PeyeDF/blob/master/PeyeDF/Model/Paragraphs.swift
17 https://developer.apple.com/guides/threading.html

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 17

To prevent consuming too much CPU power and reduce unnecessary data consumption, PeyeDF
is conservative regarding the creation of events. It is assumed that the user is not reading unless
no interaction with the software happened for a predefined amount of time (2 seconds, defined in
PeyeConstants.minReadTime) while the window is still currently being focused (i.e. in front).

4.7 Eye tracker support

PeyeDF currently supports three types of protocols to enable eye tracking. These are MIDAS, LSL
(for SMI eye trackers) and zeroMQ (for Pupil Labs glasses).
The Extras folder in our repository18 , provides supporting files for SMI trackers and Pupil Lab
glasses. For SMI, we provide a script that converts SMI network packets into LSL streams. For Pupil
Labs, we provide a plugin that allows surfaces (2-D frames) to send fixations over zeroMQ. This
is because PeyeDF requires fixation-level information. In case these are not available, a possible
alternative is to use the MIDAS framework to calculate them in real time, which we support. This
framework is described by [32].
Assuming fixation-level data can be obtained, implementing support for additional eye trackers
requires two steps: 1) creating a class that adopts the EyeDataProvider protocol and 2) adding the
given class to the EyeDataProviderType enum.
4.7.1 Adopting the EyeDataProvider protocol. The protocol generalises eye tracking behaviour by
providing a number of implemented methods (protocol extensions) that should simply be called
by the adopter at the correct times; these are sendLastRaw(), sendFixations(), eyeStateChange()
and eyeConnectionChange(). Normally, these are be called in response to receipt of gaze data (i.e.
within data callbacks, depending on the specific implementation). The protocol adopter should
also provide start() and stop() methods, which control activation and deactivation of the data
stream. In addition, it should hold references to the fixationDelegate, available, eyesLost and
fixationDelegate fields. The provided adopters (e.g. LSLManager19 can be used as a reference for
new implementations.
4.7.2 The EyeDataProviderType enum. The enum is used a reference that informs PeyeDF on the
current types of eye trackers supported. Hence, in addition to protocol implementation, a minor
sequence of two-line edits to the EyeDataProviderType enum are required to fully integrate the
eye tracker into PeyeDF. These are a case statement in the EyeDataProviderType, a new instance
of the class under associatedTracker, a string for description and an increased count variable.
For example, a tracker called NewTracker would be associated to a .newtracker enum which
would call NewTracker(), a description of "New Tracker" while count should be increased from 4
to 5. The current version of EyeDataProviderType can be used as a reference. Completing this step
will add the eye tracker to the Preferences > Experiment > Eye Tracker menu and will immediately
provide eye tracking data upon selection.

4.8 Experiment creation

PeyeDF can be utilised to conduct user experiments. The approach we recommend is to create a git
branch (in github, or a private local branch). Subsequently it is suggested to create an additional
target that will be used to run the experiment. It is possible to duplicate the original ‘PeyeDF’ target
into another (e.g. ‘PeyeDF Experiment 1’).
Under the Xcode target’s build settings, additional flags can be added to identify the target in
the application’s code. That is, under ‘Other Swift Flags’ create a flag using the -Dkeyword syntax.
18 https://github.com/HIIT/PeyeDF/tree/master/Extras
19 https://github.com/HIIT/PeyeDF/tree/master/PeyeDF/Eye%20Tracking/LSLManager.swift

, Vol. 1, No. 1, Article . Publication date: May 2019.

18 Filetti et al.

For example, a flag called ExperimentalTarget requires a -DExperimentalTarget flag. One can
then add target-specific code using:

# i f ExperimentalTarget
/ / anything experiment − s p e c i f i c
# endif

This way, patches can be “pushed upstream” or “pulled downstream” independently of experiment-
specific implementations. We provide the ‘PeyeDF Questions’ target as an example of how to define
experimental targets. This target starts the Refinder and will run through a predefined list of papers,
asking a given set of multiple-choice questions. It was utilised to run the second session of the
experiment described in the following section.

5 EXPERIMENT
The experiment tested the possibility of utilising fixation data collected in the past to predict
reading comprehension which takes place in the future. That is, fixation data collected during a
first reading of a given passage was utilised to predict performance (answer correctness) that took
place during a subsequent reading of the same passage. The experiment was ran utilising PeyeDF,
while data analysis was performed in Matlab 2018a.
The study was divided into two sessions. During the first session, participants were instructed
to read a sequence of papers and find specific information within each. The second session took
place exactly one week after the first; participants revisited the papers, answering multiple-choice
questions that referred to the information they had to locate during the first session. We demonstrate
that fixation data collected during the first session predicts performance in the second.

5.1 Question creation

We selected four papers, and in relation to each we created a number of questions that referred to a
specific passage. We selected papers with the following requirements: they had to be relatively easy
to read, while covering topics of general interest (e.g. health). They also had to contain enough
text and information to allow the creation of a total of 24 questions. The questions also had to be
grouped so that 4 consecutive questions were related to the same topic (and were therefore located
within the same section / paragraph). We gave the name “target topic” to the topic covered by a
sequence of four related questions. This way, we obtained 6 “target topics” per paper. We then split
the 6 target topics into two groups of three. The resulting two “target topic groups” were called A
and B. The split was done randomly but did not change across participants (e.g. group A in paper 1
always referred to the same set of questions).
To summarise, each target topic group (A or B) contained 3 target topics, each containing 4
questions. One participant would then be assigned only one target topic group within a paper.
Splitting into target topic groups allowed us to assign a different set of questions to every participant,
even though they all read the same papers.

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 19

The four papers we utilised were:

(1) The Impact of Asthma and Allergic Diseases on Schoolchildren: Are They at Increased Risk
of Absenteeism and Poor School Performance? [7]
(2) Acceptability of psychotherapy, pharmacotherapy, and self-directed therapies in Australians
living with chronic hepatitis C [62]
(3) The effectiveness of a motivational interviewing primary-care based intervention on physical
activity and predictors of change in a disadvantaged community [30]
(4) Choice and placebo expectation effects in the context of pain analgesia [52]
An additional paper (titled Waterpipe smoking among US university students, [46]) was used
for practice. This contained only 2 topics with 2 questions each in total, all belonging to the same
group (group A).
The resulting set of questions were saved in JSON files and are available in our GitHub reposi-
tory20 .
We used PeyeDF’s tagging functionality to identify the location in the text corresponding to the
answer to each question. This information was stored in JSON files, available in our repository21 .
Each JSON file contains the answer location for a given paper (1 to 4), group (A or B) and target
topic (1 to 3). The answer location corresponds to a page and a rectangle (text box) which encloses
the text containing the answer (along with the actual text of the answer).
5.1.1 First session. Each participant was assigned the set of four papers, randomly shuffled and
with randomly assigned target topic groups. An assignment could then be represented as 1A, 2B,
4B, 3B, indicating that the participant had to start from paper 1 and would only be shown questions
from group A, then continue to paper 2, group B, etc.
Participants were given 15 minutes to freely familiarise themselves with each paper before being
asked to perform the information finding task. After this familiarisation phase, participants were
asked to find specific information by showing to them instructions on a computer screen. The
following example depicts instructions for paper 1, group A, target topics 1 and 2:
• In Bener2011_asthma.pdf
– An overall (worldwide) view of asthma (target topic 1)
∗ Overall asthma prevalence
∗ Factors correlated to asthma
∗ Lifestyle disease modifier factors
∗ Number of people affected by pollutants
– Findings about asthma and allergic diseases (target topic 2)
∗ Most accepted method to measure asthma prevalence
∗ Correlation between age and pulmonary infection
∗ Relationship to sex
∗ Relationship between rhinitis prevalence and sex
The text shown for each item (identified by asterisk in the example) is called “summary” and is
saved in the JSON files available in our repository20 .
We utilised the practice paper to guide participants in this task, ensuring they understood the
procedure correctly.
Participants were asked to simply look up the information, without putting any additional effort
in memorising it. Participants were instructed to proceed to the next paper after finding all the
necessary information. They were also asked to advise the experimenter immediately in case
20 https://github.com/HIIT/PeyeDF/tree/master/Extras/Experiment/Questions
21 https://github.com/HIIT/PeyeDF/tree/master/Extras/Experiment/Answer_Location_Tags

, Vol. 1, No. 1, Article . Publication date: May 2019.

20 Filetti et al.

they found it difficult to locate the information they were asked to find (pilot testing was used to
verify that all information could be found autonomously by participants). Once finished, they were
instructed to advise the experimenter.
5.1.2 Second session. The second session was run using PeyeDF’s Refinder functionality (previously
shown in Figure 4). This feature automatically displayed the papers the given participant read during
the first session, in order. The Refinder funcionality was modified using the ‘PeyeDF Questions’
target22 . This modification displays a multiple choice question below the Refinder. Participants had
to select one out of three possible answers and confirm their answer before proceeding to the next
question. They performed this task in the same order as session one.
The following example refers to paper 1, group A, target topic 1, question 1:
• In western countries, how high can the prevalence of asthma and allergies be (among chil-
dren)?
– 1/3
– 1/4
– 1/5
The first item of the Question JSON files20 always corresponds to the correct answer. Answers
were displayed to participants in random order. Participants were trained for this task using the
practice paper, along with the experimenter.
Note that questions do not exactly match the ‘summary’ participants were shown during the
first session. Our aim was to induce participants into re-reading the text, rather than simply rely
on their memory. This is because we were interested in assessing whether fixation data collected
in the past predicts reading comprehension in the future (correlations between fixation duration
and memory have already been demonstrated in [33], for example).
Response correctness and time passed in relation to each question were recorded and exported
to JSON for analysis.

5.2 Participants
In total, we recruited 12 participants. We rejected participants that spent less than 30 minutes or
more than 90 minutes to complete the first session. We also rejected participants when eye tracking
calibration error was above 1° of visual angle. This resulted in 7 valid participants. We also set a
threshold for rejection at 80 % correct answers given in the second session; however, none of the
remaining participants were below this threshold.
Each participant was paid with two movie tickets.

5.3 Data analysis

As mentioned in subsection 5.1, we used Tags to store the location within the text (page number
and text box) corresponding to each answer. We utilised these to identify all fixations collected
in the first session that fell within 3° of visual angle from the centre of a given answer location.
This way, we obtained a set of fixation positions and durations in relation to each answer, for each
participant. No fixation data from the second session was utilised.
We populated a Matlab table in which each row corresponded to an answer. Each column
represented information related to the answer (correctness, time spent to answer) and the question
(target topic number, group, paper number). Each column also contained features calculated from
eye tracking data such as average fixation duration and travel. We created three Support Vector

22 https://github.com/HIIT/PeyeDF/tree/master/PeyeDF/Questions

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 21

Machine classifiers that predicted response correctness using a given set of columns; the classifiers
and related features (table columns) were:
• Eye: classifier for gaze data
– Average fixation duration
– Median fixation duration
– Sum of fixation durations
– Distance of eyes from screen
– Total travel (sum of saccades)
– Forward travel (sum of saccades moving left or towards the bottom of the screen)
– Backward travel (sum of saccades moving upwards or towards the top of the screen)
• Topic: classifier for question data
– Paper number (1 to 4)
– Group identity (A or B)
– Target topic number (1 to 3)
• All: classifier that considered all table columns.
All classifiers were trained using leave-one-out cross validation. We utilised the Area Under
the ROC Curve (AUC) as the performance measure for each classifier, as this has shown to be
appropriate for unbalanced data data [36]. Our data was unbalanced since the predicted variable
(response correctness) was skewed, since 87% of responses were correct.
We utilised a permutation test [26] to compute a p-value for each classifier. That is, every classifier
was run an additional 1000 times with the predicted variable (response correctness) randomly
permuted across table rows. A p-value is obtained by dividing the number of times that a randomised
AUC was greater than the true-observed AUC by the number of permutations.

5.4 Results and discussion

Of the tree classifiers, only All was significant (AUC: .62, p = 0.041). Topic (AUC: .37, p > 0.5) and
Eye (AUC: .50, p > 0.5) failed to classify response correctness above chance levels. This indicates that
gaze data significantly contributed to predictions only when coupled to question-related information.
However, question related information alone (the Topic classifier) was not sufficient to predict
response correctness. Similarly, gaze data alone is not sufficient. This indicates collecting gaze data
for future use, which the main feature of PeyeDF, is valuable for reading-based experiments, as
long as some text-related information is also collected. We believe this is because not all target
topics were of identical difficulty. Combining topic data with gaze-related data provides enough
information to discriminate between topics which were been previously read carefully and were
relatively easy to understand, agains topics which were not carefully read and / or were harder to
understand.
Note that topic data was linked to the design of the experiment in this example. In more realistic
scenarios, topic features could be replaced with natural language metrics (such as number nouns
and verbs) [10] or approximate location (page number, section) of the text currently read by the
user.
During pilot testing, SMI eye tracker reported a pupil size of 0, indicating that pupil size could
not be estimated reliably (this could be due to lighting in our lab or the fact that we used a highly
interactive application). Gaze-based classification could be further improved by including pupil size
measurements, as these correlate to workload [5]. Despite this limitation, we demonstrated that
fixation data provides valuable information and can significantly help in discriminating between
correctly and incorrectly comprehended text.

, Vol. 1, No. 1, Article . Publication date: May 2019.

22 Filetti et al.

New frameworks recently released for the macOS platform such as the Basic neural network
subroutines (BNSS - part of the Accelerate framework), the Convolutional Neural Networks (CNN -
part of the MetalPerformanceShaders framework) or the upcoming Create ML framework could be
integrated in the future, making deployment of machine learning applications and experiments
integrated within PeyeDF, improving user-friendliness.
Collaborative functions are included in PeyeDF. The experimental design we utilised in the
previous section could be adapted for collaborative reading research. In particular, we identified
two independent “target topic groups” within a number of papers. Each paper could then be read
by two participants, one focusing on group A and the other on group B. Gaze and behavioural data
could then be used to investigate wether their features can distinguish between the two participants,
for example.
PeyeDF is a desktop application; however, many of its functions are based on PDFKit, which
is a framework recently made available for iOS. This means that PeyeDF could be extended to
run on tablets or smartphones, given compatible eye-tracking hardware. It could also run as an
eye-tracking data viewer only.
PeyeDF enables longer term research than traditional experimental software. That is, machine
shutdowns and startups do not interfere with the data, which is stored in DiMe. Hence, it can
be used to track how user habits change over time using its integration of behavioural and eye
tracking data stored within ReadingEvents. For example, it has been suggested that expertise
affects gaze over time, on the same participants [65]. PeyeDF would be suitable for experiments
in this directions, as experiments can be run over long periods of time. In educational settings,
PeyeDF could be used to assess retention of read material depending on eye tracking data [50].
The Refinder functionality included in PeyeDF is relatively basic and could be extended to
investigate how different implementation affect information retrieval and retention. For example, it
would be possible to obtain read paragraph information from ReadingEvents and use it to compute
a cloud of potentially related documents as suggested by [14, 15].
MIDAS [32], which is supported by PeyeDF, allows real-time processing of sensor data. It could
be used to gather data collected from additional sensors, enabling research on the relationship
between physiology and emotions experienced during reading, as suggested by [27].
Native annotation support in PeyeDF is useful for annotation-directed research. That is, different
types of annotation can be proposed to users; subsequent DiMe queries can be used to determine
which were the most used annotations and experiments can be designed to assess information
retention depending on annotation type.
PeyeDF can also be used for more controlled, short-term studies. PDF pages contained the
required stimuli can be created and read by PDF. Full screen support allows to display the required
stimuli covering the whole screen, if required. Page movements could then be triggered by code,
rather than user activity.

CONFLICT OF INTEREST STATEMENT

The authors declare that the research was conducted in the absence of any commercial or financial
relationships that could be construed as a potential conflict of interest.

AUTHOR CONTRIBUTIONS
MF wrote the paper, designed the software and the related experiment. HT contributed to the
writing of the article and to the analysis of eye tracking data. NR and GJ supervised the software
and experimental design.

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 23

FUNDING
The present work has been supported by the Finnish Funding Agency for Innovation (TEKES) under
the project Revolution of Knowledge Work (funding decisions 558/31/2013 and 5159/31/2014).

REFERENCES
[1] R Ackerman and M Goldsmith. 2011. Metacognitive regulation of text learning: on screen versus on paper. J Exp
Psychol Appl 17, 1 (2011), 18–32.
[2] Annette Adler, Anuj Gujar, Beverly L Harrison, Kenton O’Hara, and Abigail Sellen. 1998. A diary study of work-related
reading: design implications for digital reading devices Proceedings of the SIGCHI conference on Human factors in
computing systems. ACM Press/Addison-Wesley Publishing Co., 241–248.
[3] D Alamargot, D Chesnet, C Dansac, and C Ros. 2006. Eye and pen: A new device for studying reading during writing.
Behavior Research Methods 38, 2 (2006), 287–299.
[4] Robert Ball and Juan Pablo Hourcade. 2011. Rethinking Reading for Age From Paper and Computers. International
Journal of Human-Computer Interaction 27, 11 (2011), 1066–1082.
[5] Jackson Beatty, Brennis Lucero-Wagoner, et al. 2000. The pupillary system. Handbook of psychophysiology 2, 142-162
(2000).
[6] Gal Ben-Yehudah and Yoram Eshet-Alkalai. 2014. The Influence of Text Annotation Tools on Print and Digital Reading
Comprehension. In Proceedings of the 9th Chais Conference for the Study of Innovation and Learning Technologies:
Learning in the Technological Era, Y. Eshet-Alkalai, A. Caspi, N. Geri, Y. Kalman, V. Silber-Varod, and Y. Yair (Eds.). The
Open University of Israel, 28–35.
[7] Abdulbari Bener. 2011. The Impact of Asthma and Allergic Diseases on Schoolchildren: Are They at Increased Risk of
Absenteeism and Poor School Performance? In Advanced Topics in Environmental Health and Air Pollution Case Studies.
InTech.
[8] Ralf Biedert, Georg Buscher, and Andreas Dengel. 2010. The eyeBook – Using Eye Tracking to Enhance the Reading
Experience. Informatik Spektrum 33, 3 (2010), 272–281.
[9] Ralf Biedert, Georg Buscher, Sven Schwarz, Manuel Möller, Andreas Dengel, and Thomas Lottermann. 2010. The text
2.0 framework. In Workshop on Eye Gaze in Intelligent Human Machine Interaction. Citeseer, 114–117.
[10] Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the
natural language toolkit. O’Reilly Media.
[11] MR Blair, MR Watson, RC Walshe, and F Maj. 2009. Extremely selective attention: eye-tracking studies of the dynamic
allocation of attention to stimulus features in categorization. Journal of experimental psychology. Learning, memory,
and cognition 35, 5 (2009), 1196–1206.
[12] Ali Borji and Laurent Itti. 2014. Defending Yarbus: Eye movements reveal observers’ task. Journal of Vision 14, 3 (2014),
29. https://doi.org/10.1167/14.3.29 arXiv:/data/journals/jov/932817/i1534-7362-14-3-29.pdf
[13] SE Brennan, X Chen, CA Dickinson, MB Neider, and GJ Zelinsky. 2008. Coordinating cognition: the costs and benefits
of shared gaze during collaborative search. Cognition 106, 3 (2008), 1465–1477.
[14] Georg Buscher and Andreas Dengel. 2009. Gaze-based filtering of relevant document segments. In International World
Wide Web Conference (WWW). 20–24.
[15] Georg Buscher, Andreas Dengel, Ralf Biedert, and Ludger V. Elst. 2012. Attentive documents: Eye tracking as implicit
feedback for information retrieval and beyond. ACM Trans. Interact. Intell. Syst. 1, 2 (2012), 1–30.
[16] Christopher S Campbell and Paul P Maglio. 2001. A robust algorithm for reading detection. In Proceedings of the 2001
workshop on Perceptive user interfaces. ACM, 1–7.
[17] Guang Chen, Wei Cheng, Ting-Wen Chang, Xiaoxia Zheng, and Ronghuai Huang. 2014. A comparison of reading
comprehension across paper, computer screens, and tablets: Does tablet familiarity matter. J. Comput. Educ. 1, 2-3
(2014), 213–225.
[18] Mauro Cherubini, Marc-Antoine Nüssli, and Pierre Dillenbourg. 2010. This is it!: Indicating and looking in collaborative
work at distance. Journal of Eye Movement Research 3 (2010), 1–20.
[19] Soussan Djamasbi, Marisa Siegel, and Tom Tullis. 2010. Generation Y, web design, and eye tracking. International
Journal of Human-Computer Studies 68, 5 (2010), 307–323.
[20] Paul Dourish, W Keith Edwards, Anthony LaMarca, and Michael Salisbury. 1999. Presto: an experimental architecture
for fluid interactive document spaces. ACM Transactions on Computer-Human Interaction (TOCHI) 6, 2 (1999), 133–161.
[21] Anton N Dragunov, Thomas G Dietterich, Kevin Johnsrude, Matthew McLaughlin, Lida Li, and Jonathan L Herlocker.
2005. TaskTracer: a desktop environment to support multi-tasking knowledge workers Proceedings of the 10th
international conference on Intelligent user interfaces. ACM, 75–82.
[22] AT Duchowski. 2002. A breadth-first survey of eye-tracking applications. Behav Res Methods Instrum Comput 34, 4
(2002), 455–470.

, Vol. 1, No. 1, Article . Publication date: May 2019.

24 Filetti et al.

[23] Susan Dumais, Edward Cutrell, Jonathan J Cadiz, Gavin Jancke, Raman Sarin, and Daniel C Robbins. 2016. Stuff I’ve
seen: a system for personal information retrieval and re-use ACM SIGIR Forum, Vol. 49(2). ACM, 28–35.
[24] Robert L Fowler and Anne S Barker. 1974. Effectiveness of highlighting for retention of text material. Journal of
Applied Psychology 59, 3 (1974), 358.
[25] Juliane Franze, Kim Marriott, and Michael Wybrow. 2014. What academics want when reading digitally, Steven Simske
and Sebastian Rönnau (Eds.), Vol. the 2014 ACM symposium. ACM Press, New York, New York, USA, 199–202.
[26] Phillip Good. 2013. Permutation tests: a practical guide to resampling methods for testing hypotheses. Springer Science &
Business Media.
[27] Arthur C. Graesser and Sidney D’Mello. 2012. Moment-To-Moment Emotions During Reading. Read Teach 66, 3 (2012),
238–242.
[28] Cathal Gurrin, Alan F Smeaton, and Aiden R Doherty. 2014. Lifelogging: Personal big data. Foundations and Trends(R)
in Information Retrieval 8, 1 (2014), 1–125.
[29] Christian Gütl, Maja Pivec, Christian Trummer, Victor Manuel Garcia-Barrios, Felix Mödritscher, Juergen Pripfl, and
Martin Umgeher. 2005. Adele (adaptive e-learning with eye-tracking): Theoretical background, system architecture
and application scenarios. European Journal of Open, Distance and E-Learning 8, 2 (2005).
[30] Sarah Hardcastle, Nicola Blake, and Martin S Hagger. 2012. The effectiveness of a motivational interviewing primary-
care based intervention on physical activity and predictors of change in a disadvantaged community. Journal of
behavioral medicine 35, 3 (2012), 318–333.
[31] B.L. Harrison. 2000. E-books and the future of reading. IEEE Comput. Grap. Appl. 20, 3 (2000), 32–39.
[32] Andreas Henelius and Jari Torniainen. 2018. MIDAS: Open-source framework for distributed online analysis of data
streams. SoftwareX 7 (2018), 156–161.
[33] Andrew Hollingworth, Carrick C Williams, and John M Henderson. 2001. To see and remember: Visually specific
information is retained in memory from previously attended objects in natural scenes. Psychonomic Bulletin & Review
8, 4 (2001), 761–768.
[34] Sabrina Hoppe, Tobias Loetscher, Stephanie A. Morey, and Andreas Bulling. 2018. Eye Movements During Everyday
Behavior Predict Personality Traits. Frontiers in Human Neuroscience 12 (2018), 105. https://doi.org/10.3389/fnhum.
2018.00105
[35] Robert JK Jacob and Keith S Karn. 2003. Eye tracking in human-computer interaction and usability research: Ready to
deliver the promises. In The mind’s eye. Elsevier, 573–605.
[36] Laszlo A. Jeni, Jeffrey F. Cohn, and Fernando De La Torre. 2013. Facing Imbalanced Data Recommendations for the Use of
Performance Metrics. In 2013 HUMAINE ASSOCIATION CONFERENCE ON AFFECTIVE COMPUTING AND INTELLIGENT
INTERACTION (ACII) (International Conference on Affective Computing and Intelligent Interaction). Humaine Assoc;
IEEE Comp Soc; Comp Vis; Multimedia Lab; Univ Geneva, Swiss Ctr Affect Sci; GFK Verein; Technicolor; Telono; Brain
Prod; Inst Telecom; Telecom ParisTech; Swiss Natl Sci Fdn; Soc Academique Geneve; Amer Assoc Artificial Itelligence,
245–251. https://doi.org/{10.1109/ACII.2013.47} 5th Biannual Conference of the Humaine-Association on Affective
Computing and Intelligent Interaction (ACII), Geneva, SWITZERLAND, SEP 02-05, 2013.
[37] Steffen Koch, Markus John, Michael Worner, Andreas Muller, and Thomas Ertl. 2014. VarifocalReader — In-Depth
Visual Analysis of Large Text Documents. IEEE Trans. Visual. Comput. Graphics 20, 12 (2014), 1723–1732.
[38] Kai Kunze, Shoya Ishimaru, Yuzuko Utsumi, and Koichi Kise. 2013. My reading life. In Proceedings of the 2013 ACM
conference on Pervasive and ubiquitous computing adjunct publication - UbiComp ‘13 Adjunct. ACM Press, New York,
New York, USA.
[39] Kai Kunze, Hitoshi Kawaichi, Kazuyo Yoshimura, and Koichi Kise. 2013. Towards inferring language expertise using
eye tracking. In CHI ‘13 Extended Abstracts on Human Factors in Computing Systems on - CHI EA ‘13. ACM Press, New
York, New York, USA.
[40] Sébastien Lallé, Cristina Conati, and Giuseppe Carenini. 2016. Predicting Confusion in Information Visualization from
Eye Tracking and Interaction Data.. In International Joint Conference on Artificial Intelligence. 2529–2535.
[41] Dongheng Li, Jason Babcock, and Derrick J Parkhurst. 2006. openEyes: a low-cost head-mounted eye-tracking solution.
In Proceedings of the 2006 symposium on Eye tracking research & applications. ACM, 95–100.
[42] Ziming Liu. 2005. Reading behavior in the digital environment: Changes in reading behavior over the past ten years.
Journal of documentation 61, 6 (2005), 700–712.
[43] Anne Mangen, Bente R. Walgermo, and Kolbjørn Brønnick. 2013. Reading linear texts on paper versus computer
screen: Effects on reading comprehension. International Journal of Educational Research 58 (2013), 61–68.
[44] Kenton O’Hara, Fiona Smith, William Newman, and Abigail Sellen. 1998. Student readers’ use of library documents:
implications for library technologies Proceedings of the SIGCHI conference on Human factors in computing systems.
ACM Press/Addison-Wesley Publishing Co., 233–240.
[45] Helena Ojanpää, Risto Näsänen, and Ilpo Kojo. 2002. Eye movements in the visual search of word lists. Vision Research
42, 12 (2002), 1499–1512.

, Vol. 1, No. 1, Article . Publication date: May 2019.

PeyeDF: an Eye-Tracking Application 25

[46] Brian A Primack, Ariel Shensa, Kevin H Kim, Mary V Carroll, Mary T Hoban, E Victor Leino, Thomas Eissenberg,
Kathleen H Dachille, and Michael J Fine. 2012. Waterpipe smoking among US university students. Nicotine & Tobacco
Research 15, 1 (2012), 29–35.
[47] Hamed R.-Tavakoli, Adham Atyabi, Antti Rantanen, Seppo J. Laukka, Samia Nefti-Meziani, and Janne Heikkilä.
2015. Predicting the Valence of a Scene from Observers’ Eye Movements. PLOS ONE 10, 9 (09 2015), 1–19. https:
//doi.org/10.1371/journal.pone.0138198
[48] Hamed R.-Tavakoli, Hanieh Poostchi, Jaakko Peltonen, Jorma Laaksonen, and Samuel Kaski. 2016. Preliminary Studies
on Personalized Preference Prediction from Gaze in Comparing Visualizations. In Advances in Visual Computing,
George Bebis, Richard Boyle, Bahram Parvin, Darko Koracin, Fatih Porikli, Sandra Skaff, Alireza Entezari, Jianyuan
Min, Daisuke Iwai, Amela Sadagic, Carlos Scheidegger, and Tobias Isenberg (Eds.). Springer International Publishing,
Cham, 576–585.
[49] Keith Rayner, Kathryn H Chace, Timothy J Slattery, and Jane Ashby. 2006. Eye Movements as Reflections of Compre-
hension Processes in Reading. Scientific Studies of Reading 10, 3 (2006), 241–255.
[50] Keith Rayner, Kathryn H Chace, Timothy J Slattery, and Jane Ashby. 2006. Eye movements as reflections of compre-
hension processes in reading. Scientific Studies of Reading 10, 3 (2006), 241–255.
[51] Daniel C. Richardson, Rick Dale, and Michael J. Spivey. 2007. Eye Movements in Language and Cognition. John
Benjamins Publishing Company.
[52] Jason P Rose, Andrew L Geers, Heather M Rasinski, and Stephanie L Fowler. 2012. Choice and placebo expectation
effects in the context of pain analgesia. Journal of behavioral medicine 35, 4 (2012), 462–470.
[53] Thiago Santini, Wolfgang Fuhl, Thomas Kübler, and Enkelejda Kasneci. 2016. Eyerec: An open-source data acquisition
software for head-mounted eye-tracking. In International Conference on Vision Theory and Applications (VISAPP).
[54] Bill N Schilit, Gene Golovchinsky, and Morgan N Price. 1998. Beyond paper: supporting active reading with free
form digital ink annotations. In Proceedings of the SIGCHI conference on Human factors in computing systems. ACM
Press/Addison-Wesley Publishing Co., 249–256.
[55] Bertrand Schneider and Roy Pea. 2013. Real-time mutual gaze perception enhances collaborative learning and
collaboration quality. Intern. J. Comput.-Support. Collab. Learn. 8, 4 (2013), 375–397.
[56] Timothy R. Shaffer, Jenna L. Wise, Braden M. Walters, Sebastian C. Müller, Michael Falcone, and Bonita Sharif. 2015.
iTrace: enabling eye tracking on software artifacts within the IDE to support software engineering tasks. In Proceedings
of the 2015 10th Joint Meeting on Foundations of Software Engineering - ESEC/FSE 2015. ACM Press, New York, New
York, USA.
[57] Kshitij Sharma, Patrick Jermann, and Pierre Dillenbourg. 2015. Displaying Teacher’s Gaze in a MOOC: Effects on
Students’ Video Navigation Patterns. In Design for Teaching and Learning in a Networked World. Springer, 325–338.
[58] Mats Sjöberg, Hung-Han Chen, Patrik Floréen, Markus Koskela, Kai Kuikkaniemi, Tuukka Lehtiniemi, and Jaakko
Peltonen. 2016. Digital Me: Controlling and Making Sense of My Digital Footprint. In Proceedings of the 5th International
Workshop on Symbiotic Interaction.
[59] H Sogo. 2013. GazeParser: an open-source and multiplatform library for low-cost eye tracking and analysis. Behavior
Research Methods 45, 3 (2013), 684–695.
[60] H Sogo. 2013. GazeParser: an open-source and multiplatform library for low-cost eye tracking and analysis. Behav Res
Methods 45, 3 (2013), 684–695.
[61] Randy Stein and Susan E Brennan. 2004. Another person’s eye gaze as a cue in solving programming problems. In
Proceedings of the 6th international conference on Multimodal interfaces. ACM, 9–15.
[62] Benjamin JR Stewart, Deborah Turnbull, Antonina A Mikocka-Walus, Hugh AJ Harley, and Jane M Andrews. 2013.
Acceptability of psychotherapy, pharmacotherapy, and self-directed therapies in Australians living with chronic
hepatitis C. Journal of clinical psychology in medical settings 20, 4 (2013), 427–439.
[63] Sakari Taipale. 2014. The affordances of reading/writing on paper and digitally in Finland. Telematics and Informatics
31, 4 (2014), 532–542.
[64] Matthew J Traxler, Robin K Morris, and Rachel E Seely. 2002. Processing Subject and Object Relative Clauses: Evidence
from Eye Movements. Journal of Memory and Language 47, 1 (2002), 69–90.
[65] Tamara van Gog and Katharina Scheiter. 2010. Eye tracking as a tool to study and enhance multimedia learning.
Learning and Instruction 20, 2 (2010), 95–99.
[66] Roel Vertegaal. 1999. The GAZE groupware system: mediating joint attention in multiparty communication and
collaboration. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. ACM, 294–301.
[67] A Vosskühler, V Nordmeier, L Kuchinke, and AM Jacobs. 2008. OGAMA (Open Gaze and Mouse Analyzer): open-source
software designed to analyze eye and mouse movements in slideshow study designs. Behav Res Methods 40, 4 (2008),
1150–1162.
[68] Glenn F Wilson and Christopher A Russell. 2003. Real-Time Assessment of Mental Workload Using Psychophysiological
Measures and Artificial Neural Networks. Human Factors 45, 4 (2003), 635–643.

, Vol. 1, No. 1, Article . Publication date: May 2019.

26 Filetti et al.

[69] Erroll Wood and Andreas Bulling. 2014. Eyetab: Model-based gaze estimation on unmodified tablet computers. In
Proceedings of the Symposium on Eye Tracking Research and Applications. ACM, 207–210.
[70] A. T. Zhang and B. O. Le Meur. 2018. How Old Do You Look? Inferring Your Age from Your Gaze. In 2018 25th IEEE
International Conference on Image Processing (ICIP). 2660–2664. https://doi.org/10.1109/ICIP.2018.8451219

Received April 2019; revised -; accepted -

, Vol. 1, No. 1, Article . Publication date: May 2019.