UPTEC E 18 007

Examensarbete 30 hp
Mars 2018

Distributed machine learning

for embedded devices

Niclas Gyllsdorff
 Abstract
Distributed machine learning for embedded devices

Niclas Gyllsdorff

Teknisk- naturvetenskaplig fakultet

UTH-enheten The goal of the master thesis is to investigate the feasibility of
having distributed machine learning on embedded devices and to
Besöksadress: analyse how the architecture of such a system can look like. A system
Ångströmlaboratoriet
Lägerhyddsvägen 1 is proposed which enables machine learning running on multiple
Hus 4, Plan 0 embedded devices to communicate with an end application. The
application communicates with the distributed machine learning via a
Postadress: gateway, which decouples the application. The proposed system is
Box 536
751 21 Uppsala implemented as a proof of concept system, which utilizes distributed
machine learning to achieve gesture recognition. The Intel Curie
Telefon: module was selected as the embedded device, as it provides a hardware
018 – 471 30 03 accelerated implementation of the machine learning algorithms
Telefax: K-Nearest Neighbour and Radial Basis Function. This module also
018 – 471 30 00 provides accelerometer/gyroscope sensors for collecting gesture data
as well as Bluetooth Low Energy, which enables wireless communication
Hemsida: with other devices. The implemented system shows that it is feasible
http://www.teknat.uu.se/student
to implement distributed machine learning on embedded devices if the
machine learning is hardware accelerated. If a hardware accelerator
was not used the computational load on the embedded device will
increase, which will increase the overall power consumption. For a
low powered and constrained devices, hardware accelerated machine
learning is likely the best approach to implement edge intelligence.

Handledare: Aitor Hernandez Herranz, Soma Tayamon

Ämnesgranskare: Thiemo Voigt
Examinator: Tomas Nyberg
ISSN: 1654-7616, UPTEC E 18 007
Populärvetenskaplig sammanfattning
Detta examensarbete har utförts på Ericsson AB på Research and Development, Device
Technologies. Ett utav Ericssons långsiktiga mål är att arbetar mot ett Networked Society, en
värld där allting är anslutet i realtid. I takt med att antalet uppkopplade enheter ökar, så ökar
även också belastningen på närverken. I strävan mot ett Networked Society är en av
utmaningarna att kunna skifta en del av beräkningsbelastningen bort från molnet
(centraliserade datacentrar) genom att använda Edge computing. Edge computing är när
beräkningsbelastningen ligger vid kanten av nätverket istället för centralt på servrar. Genom att
utföra en viss databehandling vid kanten av nätverket, nära källan till data, reduceras den totala
belastningen på nätverken.

Målet med examensarbetet är att undersöka möjligheten att ha distribuerad maskininlärning på

inbyggda system och att utvärdera hur arkitekturen i ett sådant system kan se ut. En arkitektur
beskrivs och implementeras för att bevisa att det är möjligt att ha distribuerad maskininlärning.
Det implementerade systemet utnyttjar distribuerad maskininlärning för att uppnå
gestigenkänning. Intel Curie-modulen valdes att användas som det inbyggda systemet i
arkitekturen, eftersom den har en hårdvaruaccelererad implementering av
maskininlärningsalgoritmerna K-Nearest Neighbors och Radial Basis Function. Dessutom så
tillhandahåller Curie-modulen accelerometer/gyroskop sensorer för att samla rörelse data samt
Bluetooth Low Energy, vilket möjliggör trådlös kommunikation med andra enheter.

Den implementerade arkitekturen visar att det är möjligt att genomföra distribuerad
maskininlärning på inbyggda system om maskininlärningen är hårdvaruaccelererad. Om en
hårdvaruaccelerator inte används ökar beräkningsbelastningen på det inbyggda systemet, vilket
ökar den totala strömförbrukningen. För enheter med begränsad processorkraft som behöver
vara energisnåla så är hårdvaruaccelererad maskininlärning troligen det mest lämpliga sättet att
implementera maskininlärning.
Abbreviations
Table 1. The abbreviations used in the report.

Abbreviation Description

Curie Intel Curie module

Quark Intel Quark SE microcontroller C1000
RPi Raspberry Pi 3 Model B
DM Device manager
IP Internet Protocol
WPAN Wireless Personal Area Network
one-time programmable non-volatile
OTP NVM
memory
IMU Inertial Measurement Unit
PME Pattern Matching Engine
RBF Radial Basis Function
KNN K-Nearest Neighbour
BLE Bluetooth low energy
MQTT Message Queue Telemetry Transport
RTT Round Trip Time
PoC Proof of Concept
NKB NeuroMem Knowledge Builder
Table of Contents
1. Introduction .......................................................................................................... 1
Research questions............................................................................................. 2
Feasibility of distributed machine learning on embedded devices .................... 2
Architecture of the system .............................................................................. 2
Outline ............................................................................................................... 2
2. System overview.................................................................................................... 3
Embedded device ............................................................................................... 3
Fusion ................................................................................................................ 4
Device manager ................................................................................................. 4
Gateway ............................................................................................................ 4
Application......................................................................................................... 4
3. Background ........................................................................................................... 5
Intel Curie .......................................................................................................... 5
Boards using the Curie module ........................................................................ 6
Intel Quark C1000 ........................................................................................... 6
Inertial Measurement Unit .............................................................................. 8
Pattern Matching Engine ................................................................................ 9
Bluetooth Low Energy ................................................................................... 22
Battery charger............................................................................................. 26
Raspberry Pi 3 model B ..................................................................................... 26
MQTT ............................................................................................................... 27
Publish/Subscribe ......................................................................................... 27
4. Implementation ................................................................................................... 28
Proof of Concept system ................................................................................... 28
Hardware ..................................................................................................... 28
Programming languages used ....................................................................... 29
Implementation steps ................................................................................... 30
Embedded device ............................................................................................. 31
One-handed gesture vocabulary ................................................................... 31
 Gathering of sensor data .............................................................................. 32
Preprocessing ............................................................................................... 33
Detect motion ............................................................................................... 36
Base model for PME ...................................................................................... 37
Result and Discussion .................................................................................... 39
Device manager ............................................................................................... 50
Gesture service ............................................................................................. 51
Data service .................................................................................................. 55
Result and Discussion .................................................................................... 58
Fusion .............................................................................................................. 64
Two-handed gesture vocabulary ................................................................... 65
Gateway .......................................................................................................... 65
Application....................................................................................................... 66
Result ........................................................................................................... 67
5. Conclusion ........................................................................................................... 68
Future work ..................................................................................................... 69
6. References ........................................................................................................... 71
7. Appendix ............................................................................................................. 76
Embedded device ............................................................................................. 76
Accuracy of the PME ..................................................................................... 76
1. Introduction
This thesis project has been conducted at Ericsson AB on Research and Development,
Device Technologies. Ericsson is a world-leading provider of communications technology
and services. Their offerings include services, consulting, software and infrastructure
within information and communications technology. Ericsson is working towards a
Networked Society: a world where everything is connected in real time [1].

To enable a Networked society one of the challenges is to be able to shift some of the
computation load away from the cloud (centralized datacentres) by utilizing edge
computing. Edge computing is when the computational load lies at the edge of the
network, at the end nodes. By performing some of the data processing at the edge of the
network, near the source of the data, the overall communication bandwidth needed is
reduced. It can be an AI recognition running on a video feed from a camera. If the AI is
located in the cloud there will always be latency, but if the camera itself was performing
the AI the system would not have the transmission latency. This way the data volume that
must be moved is decreased, thereby the transmission costs, latency and the overall quality
of service all improve [2].

The goal of the master thesis is to investigate the feasibility of having distributed machine
learning on embedded devices and to analyse how the architecture of such a system can
look like. Machine learning is the field of computer science that studies algorithms that can
learn from and make predictions from data. The data is divided into training sets and test
sets. The training sets are used to train the algorithms to make more accurate predictions of
the test sets. A machine learning algorithm contains a model, which define the knowledge
that is used to make the predictions. A model becomes more accurate the more the
algorithm is trained, and a more accurate model makes more accurate predictions.

There are many different categories of machine learning, but this thesis will focus on
classification using supervised learning. Classification means that the algorithm's
prediction will be divided into discrete categories, e.g., a spam filter will have the
categories spam and not spam. Supervised learning means that the algorithms will be
trained with training sets and their desired predictions.

A centralized approach to machine learning could be to have external nodes sense and
detect certain aspects of the world surrounding the node. The nodes continually send the
collected data to a central device that will make a decision based on the data. The gain of
this approach is that each external node does not need to do any demanding processing, it
is all done at the central device. The drawback is that more bandwidth needs to be used in
order to send all the data to a central device. A distributed approach to machine learning
would instead be to have the external nodes perform edge computing by both collect and
do the analysis of the data. This reduces the amount of data that the nodes need to send as
well as reduce the workload of the central device.

1
 Research questions

Feasibility of distributed machine learning on embedded devices

The feasibility is evaluated by applying machine learning on embedded devices. An

embedded device is a constrained device with limited storage and processing capabilities.
These limitations set the conditions for which machine learning algorithm that is plausible
to implement on the device. A comparison between a software and a hardware-
implemented machine learning on embedded devices is conducted.

Architecture of the system

The architecture of a system which applies distributed machine learning is evaluated by

implementing the aforementioned system. The necessary components and their relationship
to each other are identified. When the system is implemented, the limitations of each
hardware platform and the software is reviewed.

Outline
2. System overview

This chapter gives a general overview of a system which executes distributed machine
learning. The tasks of the components, which make up the system, are explained and how
they interact with each other.

3. Background

The purpose of this chapter is to describe possible hardware and software which can be
used to implement the system from section 2. There are multiple ways the system can be
implemented, but this chapter only contains the hardware which is used in section 4 to
implement the proposed system.

4. Implementation

Describes how the system from section 2 can be implemented and which
possibilities/limitations such an implementation entails.

5. Conclusion

Includes a discussion of what conclusions can be drawn from the thesis and what work lays
in the future for distributed machine learning on embedded devices.

2
2. System overview
The following chapter describes a general architecture of a system executing distributed
machine learning. Five components were defined; application, gateway, fusion, device
manager (DM) and embedded device. These five components are depicted in Figure 1.

Figure 1: Overview of the system.

The relationship between the different components are seen in Figure 1, arrows between
components symbolise message busses. The purpose of the system is to run distributed
machine learning, by running machine learning on each embedded device and have it
communicate with an end application, via a Gateway.

Embedded device
The task of the embedded devices is that they shall execute edge computing by running
machine learning algorithms. Figure 1 shows that the system consists of multiple
embedded devices, all of which executes their own machine learning, making the system
executing distributed machine learning.

3
 Fusion
The Fusion component in Figure 1 has the task to form a single output from the distributed
machine learning network. It works as a many-to-one component by gathering the output
of the embedded devices’ machine learning algorithms and giving a single output. The
functionality of the fusion component is implemented in software.

Device manager
The DM’s task in Figure 1 is to control, configure, upload/download models, and
synchronize the embedded devices. The models that are uploaded/downloaded from the
embedded devices are the models used in the machine learning. In the same manner as the
fusion component, the functionality of the DM component is implemented in software.

Gateway
The fusion and DM component make up the gateway, this component acts as an
intermediate between the distributed machine learning of the embedded devices and the
end application. The task of the gateway is to simplify the communication between the
application to the embedded devices by hiding the complexity of the distributed machine
learning, thus decouple the application from the distributed machine learning network.

The gateway has two wireless interfaces, one to communicate with the embedded devices
and one to communicate with the end application. The interface between the gateway and
embedded devices is a wireless personal area network (WPAN) protocol and the interface
between the gateway and the application is an internet protocol (IP). WPANs are used to
enable wireless communication between personal devices which are located in each other's
vicinity. IP, on the other hand, enables internet communication between devices which
necessarily need not be in each other's vicinity.

Application
The application could, for instance, be a PC, TV, smartphone or a cloud server that can be
controlled by or want to interact with the distributed machine learning network.

4
3. Background
The following chapter describes the hardware and software used in the implementation of
the system in Figure 1.

Intel Curie
The Intel Curie module (Curie) [3] is a small (11x8.05x1.95mm) and low powered module,
which contains Inertia Measurement Unit (IMU), Bluetooth Low Energy (BLE), Pattern
Matching Engine (PME) and battery charger. At the Curies core is an Intel Quark SE
C1000 (Quark) [4], which is a power efficient microcontroller that is ideal for “always-on”
applications [5]. Figure 2 depicts the functional blocks and discrete devices within the
Curie. For this thesis, it is the Quark, Nordic Bluetooth NRF51822, Bosch BMI160 6-axis
sensor and battery charger BQ25101H that is most interesting blocks within the Curie.

 Quark is described in section 3.1.2.

 IMU which Bosch BMI160 6-axis sensor [6] supplies is described in section 3.1.3.
 BLE which Nordic Bluetooth NRF51822 [7] supplies is described in section 3.1.5.
 The Battery charger which BQ25101H [8] supplies is described in 3.1.6.

Figure 2: Block diagram of the Curie. The figure is taken from [3].

5
 Boards using the Curie module

The Curie module used by the printed circuit boards Arduino 101 [9] and TinyTILE [10].
Arduino 101 is primarily marketed as a development board while the TinyTILE is
marketed as a board suitable for prototyping. A size comparison between the two boards is
depicted in Figure 3.

Figure 3: Scale comparison of the Arduino 101 (left) and the TinyTILE (right).

Intel Quark C1000

Figure 4 depicts the functional blocks and discrete devices within the Quark. For this
thesis, it is the Quark™ processor core, Memory Subsystem, Sensor Subsystem and the
PME that is most interesting blocks within the Quark.

 Quark™ processor core characteristics are presented in this section.

 Memory Subsystem characteristics are presented in this section.
 Sensor Subsystem is presented in this section.
 PME is a short presentation in this section and is covered in greater detail in section
3.1.4.

6
 Figure 4: Block diagram of the Quark. The figure is taken from [4].

Processor

 Intel Pentium x86 ISA-compatible without x87 floating point unit

 32-bit processor, 32-bit data bus and 32-bit address bus
 32 MHz clock frequency
 8 kB L1 instruction cache
 Low-latency data tightly-coupled memory (TCM) interface to on-die SRAM
 Integrated local APIC and I/O APIC

Memory

 384 kB of on-die flash + 8 kB on-die OTP NVM

 80 kB of on-die SRAM

Pattern matching engine

The PME allows the device to learn to recognize different patterns in data, therefore
making the Quark able to make an intelligent decision based on input patterns. The PME is
covered in detail in section 3.1.4.

7
Sensor subsystem

The sensor subsystem functions as a sensor hub, which can manage multiple sensors. By
having external sensors controlled by a sensor hub, the main CPU (the Pentium processor)
can sleep until the sensor hub wakes it up based on programmable cues. This result in low
energy consumption. The sensor hub is based on an ARC EM4 DSP processor [11].

Technical specification of the ARC EM4 DSP core-based sensor subsystem

 32-bit processor, 32-bit data bus and 32-bit address bus
 32 MHz clock frequency

Inertial Measurement Unit

The Curie includes a Bosch BMI160 [6] 16-bit six-axis sensing device which provides
accelerometer and gyroscope in x-, y- and z-axis. The direction of the axis on the
TinyTILE is shown in Figure 5. The IMU is controlled by the sensor subsystem, which is
covered in section 3.1.2.

Measurements returned from the BMI160 is 16 bit signed integers (range

between [−32768, 32767]), but the Curie module stores this values as 32 bit signed
integers. This means that the memory needed to store a measurement from the BMI160 is
twice as big as it needs to be, measurements can, therefore, be stored as 16 bits signed
integers without losing any precision.

The accelerometer can be set to the range ±2ɡ, ±4ɡ, ±6ɡ, ±8ɡ or ±16ɡ were ɡ denotes g-
force. For instance, if the range is set to ±2ɡ then the minimum value -32768 will denote
-2ɡ and the maximum value 32767 will denote +2ɡ [12] [13].

The gyroscope can be set to the range ±125°/s, ±250°/s, ±500°/s, ±1000°/s or ±2000°/s
were °/s denotes the angular velocity. For instance, if the range is set to ±250°/s then the
minimum value -32768 will denote -250°/s and the maximum value 32767 will denote
+250°/s [12] [13].

Figure 5: Visualization of the direction of the 6 sensor axis. The TinyTILE is drawn with
its USB port and Intel Curie.

8
 Pattern Matching Engine

The PME inside the Quark has the ability to learn and recognize patterns in arbitrary sets
of data. This hardware is based on a technology called Neuromorphic Memories
(NeuronMem) [14] which is developed by General Vision Inc [15]. The recognition is
done by first training the PME with example patterns were each pattern has an associated
numeric category. The PME can then recognize this pattern in new datasets which it has
never seen before. When it recognizes a pattern it will return the category which is
associated with the pattern. It achieves this by using 128 parallel neurons, shown in Figure
6, each neuron contains the fields; context, category, influence, minimum influence, and
pattern. The neuron fields are described in Table 1.

Figure 6: Visualization of the parallel neurons and the fields inside each neuron.

Table 1: Brief description of the usage of each field inside a neuron, more detailed
explanation of each field is covered further down in the report.

Context Only neurons with the same context as the global context are used in the
recognition/learning. Neurons with different context turn idle.

Category A numeric value that the pattern stored in the neuron is associated with.
If an input pattern is classified to be recognized as the same pattern as
the pattern stored in a neuron, then the input pattern is set to have the
same numeric category as that neuron.

Influence Also called active influence. When using the RBF classifier, if the
distance between an input pattern and a pattern stored in a neuron is
smaller than the active influence of the neuron, the input pattern is
recognized as the neuron’s pattern.
9
 Minimum The minimum influence which the active influence can shrink to. When
influence minimum influence is equal to active influence the neuron is tagged as
degenerated.

Pattern Pattern point which a neuron stores. Is a 128-byte array.

The PME has two different modes, learn/classify and save/restore, shown in Figure 7.

 Learn/classify: In this mode, the hardware can learn to associate different patterns
as numeric categories. It can then classify patterns which it has never seen before as
the categories that it has been trained to recognize.
 Save/restore: In this mode, the neuron data of the 128 neurons used for
classification can be retrieved from the hardware and saved to an external location,
e.g., SIM-card or sent through the BLE. Neuron data stored at an external location
can also be restored to the hardware so the PME can recognize the patterns learned
at an earlier time.

Figure 7: The PME can be in train/classification mode or Save/restore mode.

In Learn/classify mode, the input pattern to the PME is a 𝑁 element byte array, were 𝑁 can
be 1 to 128. This 𝑁 element byte input pattern will be used as a training pattern or the
hardware will try to classify it, depending on if it is inputted to the training or the
classification function. To be able to classify different patterns as different categories, the
PME will first need to construct a decision space during a training phase. During the
10
training phase, the PME will store training patterns which it cannot classify. The training
patterns are stored in the neurons, one new neuron is committed for each new training
pattern. The decision space is defined by the patterns stored in these neurons, these patterns
are represented as points in a 128 axis coordinate system. When the PME tries to classify
an input pattern it is also represented by a point in this 128 axis coordinate system and its
category will depend on how close it is to the training points stored in the neurons.
Category input during training must be an integer in the interval [0, 32766]. Category equal
to zero (Null category) has special attributes. Category output during classification can be
an integer in the interval [1, 32767]. Category equal to 32767 (0x7FFF) indicates that no-
match were found during classification.

A classification from the PME can be certain, uncertain or unknown:

 Certain: when only neurons of the same category recognized a pattern.

 Uncertain: when neurons of different categories recognized a pattern.
 Unknown: when no neurons recognized a pattern.

The PME supports two different classifiers, K-Nearest Neighbours (KNN) or Radial Basis
Function (RBF) when it evaluates which category an input pattern belongs to. It also
supports two different kinds of distance norms, Manhattan and Chebyshev distance, which
are two methods to measure the difference between an input pattern and the patterns stored
inside the neurons.

The term used to indicate that a neuron recognizes an input pattern is that the neuron fires.
Which neurons that will fire given an input pattern is determined by which classifier and
which distance norm is used.

Decision space

The decision space is defined as the space that spans up by 128 orthogonal vectors,
equation (1) describes the unit vectors of the decision space and equation (2) describes an
arbitrary point in this space.

𝑒̅0 = (1, 0, … , 0, 0)
𝑒̅1 = (0, 1, … , 0, 0)
..
. (1)
𝑒̅126 = (0, 0, … , 1, 0)
𝑒̅127 = (0, 0, … , 0, 1)

𝑉 = (𝛼0 , 𝛼1 , … , 𝛼126 , 𝛼127 ) 𝑤𝑒𝑟𝑒 { 𝛼𝑖 𝜖𝒁: 0 ≤ 𝛼𝑖 ≤ 255} (2)

Were 𝑒̅𝑖 is the unit vector along axis 𝑖 and 𝑉 is an arbitrary point inside the decision space.
Thus the decision space is a 128 axis coordinate system with its axis limited to integers
ranging from 0 to 255. The range limit on the axis come from the fact each neuron stores
its pattern as a 128 byte array.
11
The PME can have a maximum of 128 committed neurons that are used to recognize
different patterns. Before any learning have been done it starts with 0 committed neurons
and as more examples are trained it will increase the number of committed neurons. A
training pattern that is trained will be represented as a point in the decision space and is
stored inside one neuron. The decision space can contain 128 points which can be used for
classification because the PME has 128 neurons available for use. If the user has defined
their pattern as, e.g., a 2-dimensional value (x, y), then it will be represented as a point in
the 2D plane with the coordinates (x, y, 0, …, 0) (126 zeros after the value x and y). In
order to create graphical illustrations of how the PME functions, the decision spaces
visualized in this report is in 2D and were created using the RBF Space mapping tool from
General Vision [16].

Distance norm

The distance norm determines how the distance between an input pattern 𝑉 and the training
patterns 𝑃 is calculated. The training patterns are the patterns that are stored inside the
neurons.

Manhattan Distance 𝐷𝐿1

127

𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 = ∑ |𝑉𝑖 − 𝑃𝑖 | (3)

𝑖=0

In the two dimensional space, the points located at a distance 𝐷𝐿1 from 𝑃 such that
𝐷𝐿1 < 𝐶 form a diamond centred round 𝑃 and with side equal to √2𝐶, shown in Figure 8
[17].

Figure 8: The Manhattan distance is the greyed area shaped as a diamond that fulfils the
condition DL1<C [17].

Chebyshev Distance 𝐷𝐋𝐒𝐮𝐩

12
 𝐷𝑖𝑠𝑡𝑎𝑛𝑐𝑒 = 𝑚𝑎𝑥 |𝑉𝑖 − 𝑃𝑖 | (4)

In the two dimensional space, the points located at a distance 𝐷𝐿𝑆𝑢𝑝 from P such that
𝐷𝐿𝑆𝑢𝑝 < 𝐶 form a square centered round 𝑃 and with side equal to 2𝐶, shown in Figure 9
[17].

Figure 9: The Chebyshev distance is the greyed area shaped as a square that fulfils the
condition DLSup<C [17].

K-Nearest Neighbours

KNN is the simplest of the two classification functions that the PME can use. KNN works
by using the K nearest neurons to the input pattern point to decide its category. For
instance, if K=1, then only the closest neuron decides the category. If K=2, only closest
and the second closest neuron decide the category, and so on. Classification with K=2 and
K=5 is visualized in Figure 10 and the decision space, with K=1, is visualized in Figure 11.
In the report the syntax “KNN K=x” will be used, where x is a positive integer value which
denotes how many of the nearest neurons that will be used by the KNN classifier.

In contrast to the RBF classifier, the KNN classifier disregards the neurons’ influence
fields and instead say that every neuron recognizes the input pattern. The distance from
every neuron to the input pattern is calculated and sorted so that only the k-nearest can be
used. This means the entire decision space is mapped to different categories. Uncertainty
areas do not exist and unknown areas only exist when no neurons have been committed.
KNN mode should not be active during training of the decision space, instead, RBF mode
should be used. If KNN were be used during training the result would be that only one
neuron per category would be committed. The reason for this is that as soon as one neuron
with category A is trained, it will fire along with all the other committed neurons and
prevent the learning of new examples of category A [17].

13
 Figure 10: The circle is the input pattern which should be classified as a triangle or a
square. If K=3 (solid circle) it is assigned the triangle category (2 triangles vs. 1 square).
If K=5 (dashed circle) then it is assigned the square category (3 squares vs. 2 triangles).

Figure 11: 2D visualization how the decision space for the KNN (K=1) classification
function changes as more examples are trained. The first square has no training example
stored, thus classification will result in a no match. For the other squares, each training
example is shown in the decision space with its own colour representing its category. The
distance between neurons is in this example calculated with the Manhattan distance norm.

Radial Basis function

RBF is the second classifier that the PME can use and it uses the influence fields of each
neuron to determine the input vectors category. A neuron’s influence field determines the
space in which an input pattern is recognized as the same category as the neuron. This is
done by having the distance norm calculate the distance between the input pattern and the
pattern stored by the neuron. If the distance is less than the influence field of the neuron,
then the input pattern is recognized by the neuron.

If only neurons of, e.g., category A recognizes a pattern, then the pattern is said to have
category A. If neurons of different categories recognized the pattern then the category of
the pattern is uncertain. If no neuron recognizes the pattern then its category is said to be
unknown. Which area that is classified as certain, uncertain and unknown changes as more
patterns are trained, as shown in Figure 12.

14
Figure 12: 2D visualization how the decision space for the RBF classification changes as
more examples is trained. Each point is a pattern stored inside a neuron and the category
of the neuron is shown by the colour of its influence field. Grey areas are classified as no
match and areas were colours overlap are classified as uncertain. The distance between
neurons is in this example calculated with the Manhattan distance norm.

When a new training pattern V with its category A is sent to the PME, one of four things
will happen:

1) The training pattern is recognized as belonging to the correct category A. Then no

action is taken.
2) No neuron recognizes the training pattern V. Then a new neuron is committed to
storing pattern point V with category A. Its influence field will be set so it does not
overlap with the closest neuron.

𝑖𝑛𝑓𝑙𝑢𝑒𝑛𝑐𝑒 = 𝑑𝑖𝑠𝑡𝑎𝑛𝑠_𝑡𝑜_𝑐𝑙𝑜𝑠𝑒𝑠𝑡_𝑛𝑒𝑢𝑟𝑜𝑛 – 1 (5)

3) The training pattern V is recognized by neurons with the incorrect category. A new
neuron with pattern point V is committed and the influence field of both the new
neuron and the neurons that recognized it will be set to:

𝑖𝑛𝑓𝑙𝑢𝑒𝑛𝑐𝑒 = 𝑑𝑖𝑠𝑡𝑎𝑛𝑠_𝑡𝑜_𝑐𝑙𝑜𝑠𝑒𝑠𝑡_𝑛𝑒𝑢𝑟𝑜𝑛 – 1 (5)

4) The training pattern V is recognized by several categories, both correct and

incorrect. The PME will then shrink the influence fields of the neurons with the
incorrect category to exclude the training example, but no new neuron will be
committed because it’s already recognized as the correct category A.

𝑖𝑛𝑓𝑙𝑢𝑒𝑛𝑐𝑒 = 𝑑𝑖𝑠𝑡𝑎𝑛𝑠_𝑡𝑜_𝑐𝑙𝑜𝑠𝑒𝑠𝑡_𝑛𝑒𝑢𝑟𝑜𝑛 – 1 (5)

Iterative learning

The PME will reshape and adjust the decision space as new training patterns are trained,
either by committing new neurons or shrinking the influence fields of the already
committed neurons. When training additional training patterns the decision space will
continue to change and it is possible that a previous correctly categorized pattern will no
longer fall inside the influence field of the correct category. Or it might even fall inside the
influence field of a newly committed neuron with the incorrect category. Therefore it may

15
no longer be recognized as the correct category. An example of this scenario is visualized
in Figure 13. The decision space is thus dependent on the order in which examples are
trained and it is recommended to train patterns repeatedly, in a different order, until the
decision space stabilizes. It is stabilized when no new neurons are committed and no
changes in influence fields occur between iterations of the training patterns [17].

Figure 13: 2D visualization of how a previously correctly classified pattern might get
misclassified after new patterns are trained, this is the reason for why iterative training
might be important in order to create an accurate decision space.

Null category

Training using category 0 (also known as the Null category) has special functionality in
that it only reduces existing neurons’ influence fields and no new neurons will be
committed. Training with the Null category is equivalent to showing a counter-example.
The Null category is used to keep the neurons from overgeneralizing the decision space
and stop them from recognizing patterns that should not be recognized [17].

Minimum influence fields and degenerated neurons

Each neuron has a minimum influence field which the influence field cannot shrink below.
This minimum value is set when the neuron is committed and cannot be changed
afterwards. If the influence field reaches the minimum influence value during training, then
the neuron will be tagged as degenerated. A degenerated neuron sill behaves like any other
committed neuron. This tag indicates that a neurons pattern point lay close to the boundary
of another neuron with a different category. This can be an indication that its response
should be weighted differently than the response of other non-degenerated neurons [18].

16
Figure 14: bit order of the category register. Bit number 15 determines if the neuron is set
as degenerated or not.

After a learning phase is completed, degenerated neurons might indicate that the learned
patterns from different categories might be too similar and the PME can’t discriminate
between the categories. This might also pinpoint errors were patterns were assigned the
wrong category or that a different context should be used for the category [18].

Contexts

Context enables the PME to have dedicated neurons for different sets of data. For instance,
some of the neurons can be trained to recognize voice commands and some to recognize
gestures. Two contexts will be used to toggle between the two sub-networks, one for voice
recognition and one for gesture recognition. The maximum allowed number of contexts is
127.

Each neuron has its own Neuron Context Register (NCR) which determines which context
the neuron belongs to. The current context used is selected by writing a context value to
the Global Context Register (GCR). Committed neurons with its own NCR different from
the GCR turns idle and will not participate in any learning or recognition operation. The
context of value 0, is an exception, which enables all the neurons without regards to their
context [18].

Libraries

Currently, 3 implemented libraries exist which enable usage of the PME of the Intel Curie:

 CurieNeurons free library: developed by General Vision and includes RBF

classifier, single context and no access to the neurons registers. Has two
classification functions: one which returns the category, distance and identifier of
nearest firing neuron which recognizes the input, it also returns the classification
status (unknown, identified or uncertain). The second classification function returns
lists of the categories, distances and identifiers of the K nearest firing neurons
which recognizes the input [19] [20].
 CurieNeurons PRO library: developed by General Vision and includes RBF and
KNN classifier, multiple contexts and full access to the neurons registers (both read
and write to registers). Has three classification functions, the two functions
included in the free library and a function which only returns the classification
status (unknown, identified or uncertain). [19] [20].
 CuriePME: the library that is used in the report. It is a free open source library
which includes the RBF and KNN classifiers, multiple contexts and limited access
17
 to the neurons registers (only read registers). Has two classification functions,
classify and writeVector [21].
The classify function returns the category of the nearest firing neuron and if no
neuron fires then the returned category is no-match. For the KNN classifier, this
means that K is always set to 1. The RBF classifier uses its influence fields to
determine which neurons recognize a pattern, it then does a KNN classification
with K=1 over these neurons. This means that the closest neuron that recognizes the
input pattern with its influence field determines the category, making the RBF
classifier act as an RBF+KNN (with K=1) classifier.

Figure 15: comparison of the RBF classifier, which has the uncertain areas, with the RBF
+ KNN (K=1) classifier which is used in CuriePME. The RBF + KNN has one figure with
the bounds set by both classifiers drawn and one only showing the decision space.

But if the classification status is desired to be known while using the classify
function, then the Neuron Status Register (NSR) needs to be read with the getNSR
function after the classify function. NSR=0 indicates unknown-status and no-
match were returned from the classify function. NSR=4 indicates uncertain-status
and occurs when more than one category is fired in the classification, the
classify function returned the closest category. NSR=8 indicates identified-
status and occurs when only one category fires, this category is returned by the
classify function.
The degenerated bit is masked away from the category when the classify function is
used, so if this bit is of interest then the writeVector function must be used
[21]. The writeVector function only calculates the distances between the firing
neurons and the input pattern. The classification then needs to be done by calls to
the PME’s different registers. This makes the writeVector function more
versatile than the classify function, but the user needs to have an understanding
of how the registers of the PME works [21].

18
Usage of CuriePME writeVector function

The use of this function is not well documented and it requires some knowledge of the
PME’s registers in order to use it successfully. A call to writeVector will calculate the
distance between each firing neuron and the input pattern. In KNN mode all neurons will
fire so the distance between all neurons will be calculated. In RBF mode only the neurons
with a larger influence field than the distance will fire [18] [21].

A call to the function getIDX_DIST after writeVector will sort the internal list of
the firing neurons by distance and return the distance to the nearest neuron. Next call to
getIDX_DIST will return the second closest distance, and so on. When getIDX_DIST
returns 0xFFFF it means that there are no more firing neurons [18] [21].

The function getCat returns the category of the top firing neuron, the second call returns
the second firing neuron and so on. When getCat returns 0xFFFF it means that there are
no more firing neurons. Observe that getIDX_DIST needs to be called before getCat
in order to sort the internal neuron list by distance. If getIDX_DIST is not called first,
then getCat will return the smallest category of the firing neurons, without regard to
distance. The category returned from getCat includes the degenerated bit, thus enabling
the user to weight these neurons differently than non-degenerated neurons response [18]
[21].

CuriePME.writeVector(pattern_vector, vector_length);

for(int i=0; i<K; i++)

{
distance[i] = CuriePME.getIDX_DIST();
category[i] = CuriePME.getCAT(); // & 0x7FFF;

if(category[i]==0xFFFF)
break;
}
// distance and category of the top K firing neurons.
// if less than K neurons fired then the value 0xFFFF will signal
// that there are no more category/distance in the list

Code 1: example code for getting the category and distance of the K nearest firing
neurons. 0x7FFF can be used to mask away the degenerated bit of the category. The
algorithm visualized in Figure 10 can implemented using this method [13] [15].

19
 CuriePME.writeVector(pattern_vector, vector_length);

// sort and return the distance to the nearest firing neuron

uint16_t distance = CuriePME.getIDX_DIST();
// return the nearest category
// Note: getIDX_DIST() needs to be called before getCAT()
uint16_t category = CuriePME.getCAT(); // & 0x7FFF;

switch(CuriePME.getNSR())
{
case 0:
// handle unknown case
break;
case 4:
// handle uncertain case
break;
case 8:
// handle identified case
break;
}

Code 2: example code how the Neuron Status Register can be read in order to determined
unknown, uncertain and identified status while using the RBF classifier. 0x7FFF can be
used to mask away the degenerated bit of the category [13] [15].

Limitations

Each neuron can store 128 bytes, this means that the patterns the hardware can learn to
recognize can only be 128 values long and every value must be in the range 0 to 255 (the
range of one byte). The maximum number of neurons that can be committed is 128, which
limit how complex the decision space can be. These 128 neurons are divide up into the
categories of each context. Complex patterns require more neurons than simpler patterns to
describe its decision space and if the neurons need to be divided between too many
categories and contexts then the PME will not be able to accurately describe the decision
space. Fewer contexts and categories enable more neurons to define each category.

The PME which is integrated into the Quark uses the NeuronMem technology from General Vision
to perform its pattern recognition. General Vision provides various other boards with NeuroMem
if it were such that Curie's limitations would make it inappropriate for a certain pattern
recognition problem.

20
Table 2 shows a comparison of the different chips from General Vision. The NM500 [22]
and the CM1K [23] chips can be cascaded by connecting multiple chips together to form
large networks of neurons, with a capacity of thousands, tens of thousands and even
millions of neurons. These chips can thus be used for more complex data, text, image
analytics and sensor fusion than the Quark can.

21
 Table 2: comparison between NeuronMem in the Quark, NM500 and CM1K chips.

Neuron attributes Quark NM500 CM1K

Neuron capacity 128 576 1,024
Memory capacity per neuron 128 bytes 256 bytes 256 bytes
Categories 15 bits 15 bits 15 bits
Distances 16 bits 16 bits 16 bits
Contexts 7 bits 7 bits 7 bits

Bluetooth Low Energy

The Curie includes the hardware Nordic NRF51822 [7] which provides BLE 4.1 to the
device. This section describes how BLE works in general and how it works specifically for
the NRF51822.

Internally the NRF51822 is connected to the Quark. BLE is a WPAN protocol which is
optimized for low power use at low data rates. The functionality of the NRF51822 is
determined by which SoftDevice stack is used. SoftDevice is Nordic Semiconductors name
for the software protocol stack that the BLE uses. The NRF51822 supports three different
SoftDevices [7]:

 S110: a peripheral/broadcaster protocol stack.

 S120: a central/observer protocol stack solution supporting up to eight

simultaneous central role connections.

 S130: a concurrent multi-link protocol stack solution supporting simultaneous

Central/Peripheral/Broadcaster/Observer role connections. There are different
versions of the S130, which are described in section 0. It is this SoftDevice that is
included within the Curie, but it is never stated which version [3].

The library that the Curie provides for interfacing with the NRF51822 and its SoftDevice is
called CurieBLE [24].

Each BLE device acts as either a bulletin board (peripheral or broadcast role) or as a reader
(central or observer role), shown in Figure 16. The peripheral/broadcaster device post data
on the bulletin board for all central/observer devices to read. Data on the bulletin board is
called characteristics and can be at maximum 20 bytes long. The characteristics are
grouped into services. In the analogy, the services are notices posted on the bulletin board
and characteristic are the individual paragraphs on those notices.

22
 Figure 16: Visualization of the bulletin board (peripheral/broadcasting device), services,
characteristics and readers (central/observer devices) [24].

Broadcaster/observer role

A device that is in broadcast mode is constantly advertising its data and does not connect
with other devices, therefore it cannot receive data. An observer is the opposite of a
broadcaster, it passively listens to other BLE devices in the area to collect their advertised
data without connecting to the devices. A device in observer mode cannot send data. In a
broadcaster and observer network, the communication is one way [25].

Central/peripheral role

A peripheral device has similar functionalities as a broadcaster, with the exception that
devices need to connect to it before communication can be made. In the same way, a
central device has similar functionalities as an observer, but with the exception that a
connection needs to be made. This connection enables two-way communication between
the peripheral and the central device. The peripheral device sets up and advertises the
services and characteristics it provides. It is then up to a central device to connect to the
peripheral device and begin the communication.

Properties of a characteristic

The peripheral/broadcaster defines the characteristics and properties it has. These

properties define how a central device or an observer device can interact with a
characteristic:

23
  Broadcast: is used to define that the device shall act as a broadcaster for the
characteristic.
 Read: Central asks the peripheral device to send the current value of the
characteristic. Often suitable for characteristics that do not change very often.
 Write: Central is able to change the value of the characteristic. Suitable for
commands were the central device controls the peripheral device.
 Notify: When a sender (can be either peripheral or central device) writes to a
characteristic the new value is then automatically sent to the receiver, without the
receiver needing to execute a read command. Often used for streaming data, e.g.,
sensor data.
 Indicate: Works similarly as notify but the reader sends an acknowledgement of
the pushed data [24].

Versions of SoftDevice S130

Currently, there are two different versions of S130, version 1 and version 2. They have
some differences but for the scope of this thesis the most interesting ones are:

S130 Version 1
Supports up to three connections as a central, one connection as a peripheral, an observer,
and a broadcaster all running concurrently [26].

S130 Version 2
The SoftDevice can be configured with one connection as a peripheral and up to eight
connections as a central. The SoftDevice supports running one peripheral or broadcaster
and one central or observer concurrently with the BLE connections [27].

Star topology

Regardless which version of S130 that is used, a peripheral device can only connect to a
single central device. This means that the architecture of the BLE system will be star
topology, there will be a central device in the middle with multiple peripherals devices
connected to it, as depicted in Figure 17.

Figure 17: star configuration with Curies. Central device in the middle, with the connected
peripheral devices surrounding it.
24
Multirole

Multirole is when a device acts as multiple roles of peripheral, central, broadcaster and
observer. This enables the Curies to form a network built up by multiple smaller star
networks. Communication with the network can then be done with an unconnected
peripheral within the network. The Curie that peripheral belongs to then act as a gateway
for the whole network, as shown in Figure 18.

Figure 18: with the use of multirole the Curies can form large networks. Without multirole,
the network size would be limited by maximum connections to a central.

Limitations

Slow
BLE is designed with the primary goal of consuming low power. This means that the
technology was not optimized to be able to send data at high data rates [24].

20 bytes per characteristic

A BLE characteristic can at maximum only contain 20 bytes of data. Meaning that if more
than 20 bytes need to be sent, then the data needs to be divided into multiple packages.

Max 6 characteristics
The Intel Curies BLE device can at maximum have 6 characteristics in total, regardless if
they are placed in different services or not. If more than 6 characteristics are used, then the
device will freeze when the function BLE.advertise() is called. The reason for this is
that the system has only 8 KB of heap memory and too many characteristics will result in
the heap filling up. The amount of heap memory that is allocated is defined by the Curie’s
firmware. The issue is under investigation and might be solved in the future. For now, the
Curie can at maximum have 6 characteristics [28].

25
Device hangs when the BLE and the IMU are used
Currently there a bug in the Intel Curie source code that makes the device crash if the IMU
and BLE are used at the same time [29], which is what we want to do. The solution for the
implemented system, described in section 4, was to disable all interrupts for the Curie
when the IMU is being used and reactivate them afterwards.

Battery charger

The Curie includes the hardware BQ25101H [8] which makes the Curie able to charge
connected batteries, but unfortunately, the pins of the charger are not exported to any pin
on the Arduino 101 or the TinyTILE. This means that the battery charging capabilities of
the Curie cannot be used on any of the two commercially available boards and battery
charging is only available if an external battery charger is used [30] [31]. If a custom board
using the Curie module is designed and the internal battery charger is desired to be used,
then refer to Intel® Curie™ Module Design Guide [32] for guidance on how to design a
board which enables the use of the charger.

Raspberry Pi 3 model B
Raspberry Pi 3 Model B (RPi) [33] is a small single-board computer with its own operating
system. It has many of the same functionalities that a normal PC has: create documents,
browse the internet, programming different programming languages, play games and so on.
The RPi has a 1.2GHz 64-bit Quad-Core ARM Cortex-A53 processor with 1GB RAM. It
is also equipped with 10/100 Mbit/s Ethernet, 802.11n wireless, Bluetooth 4.1. This
enables the RPi to communicate both via the WPAN protocol BLE and via IP protocols,
which are the wireless interfaces needed by the gateway component [34]. Figure 19 depicts
the layout of the RPi and some of its features.

Figure 19: Layout of the RPi. The figure is taken from [35].
26
 MQTT
Message Queue Telemetry Transport (MQTT) is a lightweight machine to machine
publish/subscribe protocol which is used on top of the TCP/IP protocol. The protocol is
suitable for the Internet of Things and connections with remote locations because of its
small code footprint and low power usage. It is designed for constrained devices and low-
bandwidth, high-latency or unreliable networks [36].

This thesis demands a protocol that enables communication between the gateway and the
application, but it does not specify which protocol this must be. MQTT is selected to be
used in this communication. But the system could as well be implemented using another
protocol that enables communication over an internet connection, such as CoAP [37].
Therefore only a brief description of MQTT is included to give the reader an understanding
of the most important aspects of the protocol.

Publish/Subscribe

The MQTT is a publish/subscribe protocol, which means that devices can publish
messages to topics and other devices can subscribe to these topics. In MQTT topics are the
addresses to the data that is sent amongst various clients. Both publisher and subscribers
are called clients in this system and a client can be both a publisher and subscriber.
Multiple clients connect to a broker and it is this broker that keep tracks of the topics that
the clients publish and subscribe to. The broker acts as an intermediate so the clients do not
need to know anything about each other. They only need to know which topics they want
to subscribe to. An example of multiple clients and topics are shown in Figure 20 [38].

Figure 20: sensor 1 publish a message to topic acceleration and sensor 2 publish a
message to topic temperature. The broker forwards this information to all clients that have
subscribed to each topic. In this case, the application has subscribed to the acceleration
topic.

27
4. Implementation
The following chapter describes the how the system in Figure 1 was implemented on the
hardware.

Proof of Concept system

The implemented system is a Proof of Concept (PoC) system, which shows that a system
with an architecture like the one shown in Figure 1 is feasible. The PoC utilizes distributed
machine learning to achieve gesture recognition. A user attaches one embedded device to
each wrist. These devices collect sensor data of each hand while the user performs
gestures. The machine learning algorithms of each embedded devices are trained to
classify which gesture a user performs from the collected sensor data. Thus the embedded
device attached to the right wrist is able to detect and determine which gesture the right-
hand preform and the left embedded device is able to do the same for the left hand. The
determined gestures are then sent to the Fusion component, which combines the gesture
from each hand to form more complex two-handed gestures. These gestures are then
forwarded to an end application, which is controlled by the gestures.

Hardware

Figure 21 illustrates on which hardware each component in the system is implemented on.
Communication between the embedded devices and the gateway is done via BLE, and
communication between the gateway and the application is done via IP using MQTT as
messenger bus. Figure 22 shows the PoC system running on the hardware.

Figure 21: Hardware used in the PoC system.

28
  Curie as embedded device: The systems requirements specification for the
embedded device is that it executes a machine learning task. There are three main
reasons that make the Curie model suitable as the device component in the PoC
system:
o IMU provides accelerometer and gyroscope data for detecting and
collecting data that describes gestures.
o PME provides pattern matching capabilities accelerated by hardware that
makes it able to solve machine learning problems.
o BLE provides a wireless interface for communicating with other
components in the system.
From now on in the thesis report, embedded device will be referred to as Curie.
 RPi as DM and Fusion: To simplify the implementation, the two components
were integrated to be run a single program run on the RPi. The reason for having
the components as the same program is that the Curies can only have one BLE
connection at a time. For instance, if the DM and the Fusion component were
implemented as different programs, then the Curies would need to switch the
connection back and forth between the two components, depending on which
component needs to communicate with the Curies.
 The application runs on a computer: A Snake game [39] was developed to show
that the PoC is feasible, by having the game being controlled by gestures.

Figure 22: The hardware running the PoC system. To the left is two Curies, in the middle
is an RPi acting as a Gateway and to the right is the end application which runs a snake
game.

Programming languages used

C/C++ [40]: Software for the Curie is programmed in C/C++ and developed in the
Arduino IDE [41]. A program created in the Arduino IDE which runs on an Arduino
compatible board is called a sketch.

29
Python 3.6 [42]: The DM and Fusion components were all implemented into a single
Python program, which runs on the RPi. The programming language selected for
programming these components were Python because it provides libraries for both BLE
and MQTT communication. BLE communication was implemented using the library
Bluepy [43] and the MQTT client was implemented using the library Paho [44].

The Application was also written in Python, it runs on a PC. For MQTT client the library
Paho was used.

Implementation steps

Implementation was done in 5 separate steps, each separate step is described below.

Step 1: Embedded device

The first step was to get machine learning working on the embedded devices, by
implementing gesture recognition on a single Curie. The recognition is done by collecting
accelerometer and gyroscope data from a user’s gestures, the data are sent to the PME
which is trained to distinguish different gestures from the sensor data. One Curie can then
detect and classify gestures done by the hand it is attached to. By attaching one Curie to
each wrist of a user, the Curies can detect and classify gestures of each hand.

Step 2: Device manager

The second step was to implement a DM in order to control the two Curies via a BLE
interface. Uploading/downloading of models, synchronization and control of the Curies via
BLE were implemented.

Step 3: Fusion

Prior to this step, each of the recognized gestures of the Curies was handled separately,
making the system unable to detect gestures performed with two hands. The gesture
vocabulary was extended to include two-handed gestures and the fusion component was
implemented, thus making the system able to determine if two gestures were, in fact, a
two-handed gesture.

Step 4: Gateway

The fourth step was to implement communication via MQTT, which is the interface for
which the end applications can communicate with the distributed machine learning.

Step 5: Application

The last step was to implement a test application which can make use of the gesture
recognition. A snake game was implemented which is controlled by a user’s gestures.

30
 Embedded device
The task of the Curies in the PoC is to solve the machine learning problem of recognizing
gestures done by one hand. A gesture recognition sketch was developed for the Curie
module, which is based on the open source code Drawing in the air by Intel [45]. That
code demonstrates how the PME can use accelerometer data to recognize letters that are
drawn in the air. The sketch developed for the PoC follows the same idea but uses both the
IMU’s accelerometer and gyroscope sensors for gathering gesture data. Recognized
gestures are sent via BLE to the other components in the PoC system. The workflow of the
gesture recognition sketch is shown in Figure 23.

Figure 23: General idea of how gesture recognition can be done on a Curie.

A shown in Figure 23, gesture recognition on a Curie can be done in three steps:

1) When a user performs a gesture the IMU will collect sensor data describing the
gesture, both accelerometer and gyroscope are used for this. Configuration for the
IMU is described in section 4.2.2. The sensor data is compressed to fit into 128
bytes via the preprocessing step, which is described in section 4.2.3
2) These 128 bytes are sent to the PME for classification of the gesture. The PME
returns the category of the gesture as an integer value. The gestures vocabulary that
the PME can recognize is described in section 4.2.1, and the generation of the
model that the PME uses in the recognition is described in section 4.2.5.
3) Lastly, BLE is used to communicate the category from the classification to the
gateway component. The implementation of the BLE is discussed in section 4.3.

One-handed gesture vocabulary

For the gesture recognition to work, then the PME must first be trained with the gestures it
should recognize. Therefore a gesture vocabulary was defined, containing 9 distinctive
gestures that can be performed with one hand. These gestures are shown in Figure 24. Each
gesture was given a number which acts as its category. It is this category number that is
returned from the PME after a successful classification.

31
 Figure 24: Gesture vocabulary for gesture recognition with one hand. The vocabulary is
the same for both right and left hand.

Two more categories also exist, which does not correspond to real gestures, these
categories are the no-match and the uncertain. The no-match category is returned when the
sensor data doesn’t match any of the other 9 gestures and it has the category 32767
(0x7FFF). The uncertain category is returned when the sensor data matches more than one
gesture and thus the PME cannot decide which category it belongs to. In this case, the
PME will return category 32768 (0x8000) which is the uncertain category. The uncertain
category only exists if the RBF classifier is used, because there are no uncertainty areas
when the KNN classifier is used. As explained in section 0, the current official release of
the CuriePME library does not have uncertainty areas in the RBF classifier. The library
was modified to be able to return an uncertainty category.

Gathering of sensor data

One Curie is strapped on each wrist of a user in order to gather the sensor data describing
the user’s gestures. As the user performs a gesture, samples from the IMU’s accelerometer
and gyroscope are gathered. One sample consists of 6 values: x, y, z for the accelerometer
and x, y, z for the gyroscope.

32
 𝑠𝑖 = (𝑎𝑥 , 𝑎𝑦 , 𝑎𝑧 , 𝜔𝑥 , 𝜔𝑦 , 𝜔𝑧 ) (6)

𝑠𝑖 denotes sample number 𝑖

𝑎𝑥 , 𝑎𝑦 , 𝑎𝑧 denotes accelerometer value in x, y, z axis

𝜔𝑥 , 𝜔𝑦 , 𝜔𝑧 denotes gyroscope data in x, y, z-axis

Accelerometer configuration
Sample rate = 200 Hz
Range = ±2ɡ

Gyroscope configuration
Sample rate = 200 Hz
Range = ±250°/s

Preprocessing

The main idea behind the gesture recognition is to gather accelerometer and the gyroscope
data, in x-, y- and z-axis, as a user performs a gesture. The PME then uses the gathered
data determine which gesture the user performed.

As explained in section 0, the PME have some hardware limitations that must be taken into
account. Mainly that the input to the PME is limited to maximum 128 bytes. For instance,
with a sampling rate of 200 Hz were each sample consists of 6 values (3 accelerometers
128 𝑏𝑦𝑡𝑒
and 3 gyroscopes) would fill up the 128-byte input in 6 𝑏𝑦𝑡𝑒 ∗ 200 𝐻𝑧 ≈ 0.11𝑠, which is too
short time for a user to make a gesture. Therefore preprocessing were implemented to
counteract the limitations and make the senor data fit as input to the PME. The
preprocessing is divided into the steps depicted in Figure 25: dynamic range,
undersampling and averaging filter.

Figure 25: The steps that constitute the preprocessing.

Dynamic range

Measurements from the IMU can be stored as 16-bit integers (2 bytes) with full precision
maintained, as explained in section 3.1.3, but the input to the PME is a byte array.
This means that the 2-byte measurements need to be stored as 1 byte. The range of the
sensor data is therefore changed from [−32768, 32767] (2 bytes) to [0, 255] (1 byte). This
entails a loss of precision of the sensor data, with 2 bytes the data can assume 65536
different values but with 1 byte it can only assume 256 different values. By using 1 byte to

33
 256
store the values it will result in accuracy of 65536 ∗ 100% = 0.39% comparing to if we
used 2 bytes.

The 1-byte sensor values are then stored in a sample queue which fit 400 samples. With a
sample rate of 200 Hz this queue can fit 2 seconds worth of sensor data at maximum, thus
setting the maximum length of a gesture to 2 seconds. A queue is used so that when over
400 samples are collected it will start to overwrite the earliest collected samples, thus
ensuring that only the latest 2 seconds worth of data is stored.

Undersampling

The queue that is used to store the collected gesture data can at maximum store 400
samples, where one sample consists of 6 values (3 accelerometers and 3 gyroscopes).
Maximum array size that can be inputted into the PME is 128, therefore the 400 samples
128
need to be squeezed down to ≈ 21 samples without losing too much information in the
6
data. This is done by first undersampling the gesture samples. For instance, if 400 samples
400
are gathered, then only every = 19𝑡ℎ sample will be stored.
21

𝑛
𝑠𝑡𝑒𝑝 𝑠𝑖𝑧𝑒 = 𝑟𝑜𝑢𝑛𝑑 𝑑𝑜𝑤𝑛 ( ) (7)
𝑠
Equation (7) describes how the step size, which is used in undersampling, is calculated.
Here 𝑛 denotes the total number of collected samples and 𝑠 denotes the number of samples
per vector, which is 21 if all 6 sensor axis are used. The step size is rounded down to the
closest integer.

The sample window is centred by setting the start sample as given by equation (8). Half of
a step (rounded down) and half of 𝑛 𝑚𝑜𝑑𝑢𝑙𝑜 21 (rounded down) will be put on the left
side of the sample window and the rest on the right side. A visualization of how
undersampling works is shown in Figure 26.

𝑠𝑡𝑒𝑝 𝑠𝑖𝑧𝑒 𝑟𝑒𝑚𝑖𝑛𝑑𝑒𝑟

𝑠𝑡𝑎𝑟𝑡 𝑠𝑎𝑚𝑝𝑙𝑒 = 𝑟𝑜𝑢𝑛𝑑 𝑑𝑜𝑤𝑛 ( ) + 𝑟𝑜𝑢𝑛𝑑 𝑑𝑜𝑤𝑛 ( ) (8)
2 2
𝑟𝑒𝑚𝑖𝑛𝑑𝑒𝑟 = 𝑛 𝑚𝑜𝑑 21

34
 Figure 26: Sensor data collected for the clockwise gesture. The sensor data consists of 6
sensor axis (x, y and z for accelerometer and gyroscope). Undersampling is done for all
axis, but it is only shown for the z-axis too not overcrowd the plot. Note that the sample
window is centred in the sampled data as equation (8) describes.

Averaging filter

If the undersampling was the last stage of the preprocess, then most of the gathered
samples would be discarded and never used in the gesture recognition. For instance, in
Figure 26 the application collects 272 samples which are undersampled to 21 samples. In
this case, the undersampling stage wants to discard 221 samples. Instead of just throwing
away these samples that are in-between the undersampling steps, the averaging filter uses
them by taking the average of them.

∑𝑠𝑡𝑒𝑝 𝑠𝑖𝑧𝑒
𝑖=−𝑠𝑡𝑒𝑝 𝑠𝑖𝑧𝑒 𝑠𝑗−𝑖
𝑎𝑣𝑒𝑟𝑎𝑔𝑒𝑗 = (9)
𝑠𝑡𝑒𝑝 𝑠𝑖𝑧𝑒 ∗ 2 + 1

Equation (9) shows how the average value of sample number 𝑗 is calculated, were 𝑠𝑗−𝑖 is
the sample number 𝑗 − 𝑖 and 𝑗 = [0,20]. As a remainder, a sample consists of 6 values (1
35
for every sensor axis) and the average from equation (9) is calculated for every one of
these values. This averaging filter will ensure that the information in the in-between
samples are not lost and will smooth out eventual noise from the sensors. In simple terms,
equation (9) calculates the value of sample 𝑗 as the average value of all samples between
sample 𝑗 − 1 and sample 𝑗 + 1. An example of how undersampling works is shown in
Figure 27.

Figure 27: Shows the raw, undersampled, undersampled + averaged values of the z-axis
for the gyroscope data from a clockwise gesture. Note that the x-axis of the plot refers to
the samples of the raw gesture data. The undersampled and undersampled + average data
has only 21 samples and thus is stretched in the plot.

Detect motion

Sensor samples are continuously gathered and stored in the queue, these values are then
used to evaluate if the Curie has started moving. This is done by looking through the last
15 sensor values for each of the 6 sensor axes and see if any of the axis has had a change
over a certain threshold (the threshold used in the PoC is 3). If it had then the Curie has
started moving, which means that gesture has started. The value 15 and 3 was chosen by
simply testing out the motion detection function with different values, it was found that
these values gave the function adequate sensitivity in its detection.

36
Detect motion 𝑇ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 = 3 < |𝑉𝑎𝑥𝑖𝑠 𝑘, 𝑣𝑎𝑙𝑢𝑒 𝑗 − 𝑉𝑎𝑥𝑖𝑠 𝑘, 𝑣𝑎𝑙𝑢𝑒 𝑗−𝑖 | (10)

Detect zero motion 𝑇ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑 = 3 ≥ |𝑉𝑎𝑥𝑖𝑠 𝑘, 𝑣𝑎𝑙𝑢𝑒 𝑗 − 𝑉𝑎𝑥𝑖𝑠 𝑘, 𝑣𝑎𝑙𝑢𝑒 𝑗−𝑖 | (11)

𝑖 = 1,2, . . ,14
𝑘 = 1,2, , . . ,6
𝑗 = 𝑖𝑛𝑑𝑒𝑥 𝑜𝑓 𝑡ℎ𝑒 𝑙𝑎𝑠𝑡 𝑐𝑜𝑙𝑙𝑒𝑐𝑡𝑒𝑑 𝑣𝑎𝑙𝑢𝑒

Equation (10) describes the criterion which the application uses to determine if the Curie is
in motion, if it’s true for any combination of 𝑖 and 𝑘 then the Curie is assumed to be in
motion. 𝑉𝑎𝑥𝑖𝑠 𝑘, 𝑣𝑎𝑙𝑢𝑒 𝑗 denotes the newest sampled value of axis k, and
𝑉𝑎𝑥𝑖𝑠 𝑘, 𝑣𝑎𝑙𝑢𝑒 𝑗−𝑖 denotes the last 14 values of axis k.

After a motion has been detected by using equation (10), the Curie will now use equation
(11) for detecting when the motion stops. Equation (10) and equation (11) are almost
identical, except that inequality-operator has been changed. The samples that were
gathered between the start and stop of the motion will be the gesture.

Base model for PME

The model defines which gestures the PME can and cannot recognize. The models contain
the neuron data of the PME: context, influence, minimum influence, category and the
pattern point of each of the committed neurons.

A system was developed in order to train the PME hardware with gesture data (training
sets) stored on a computer. Figure 28 shows a general visualization of the system and
Figure 29 shows an implementation of the system. The application running on the
computer was written in Python. The training is done in an iterative manner, meaning that
each training set is used multiple times in the training, but the order of the sets are different
in each iteration. See Figure 13 for a visualization of why iterative training is important.
When the training is done, the model is sent back to the computer for storage.
Communication between the Curie and the computer are done via serial communication
over USB.

Figure 28: Visualization of a system able to train the PME with training sets and retrieve
the model afterwards.
37
 Figure 29: Real setup of the system in Figure 28. T/C stands for Training/classification
and S/L stand for sending model from Curie/loading model to Curie.

The training sets were collected with two different approaches; many gestures from one
user and few gestures from many users.

Many gestures from one user

In order to investigate the maximum accuracy of the PME, 90 set of each gesture was
collected. All gestures of this approach were collected from a single user. To stop the PME
from over generalising the decision space and giving false positives, 90 sets for the Null
category was also collected. This was done for both hands of the user, 900 sets of the left
hand and 900 sets for the right hand were collected. Table 3 summarizes the gathered
gestures for this approach.

90 training sets per gesture were selected because the PME was deemed accurate enough to
successfully classify most gestures without making the iterative training process take too
long. From Table 8 it can be seen that the training process for one hand with 90 sets per
gesture takes roughly 70 minutes to complete.

Table 3: Collected training sets for the “many gestures from one user”. Gesture category
0 denotes the number of gestures collected of the Null category.

Gesture category 0 1 2 3 4 5 6 7 8 9
User 1 right hand 90 90 90 90 90 90 90 90 90 90
User 1 left hand 90 90 90 90 90 90 90 90 90 90

38
Few gestures from many users

10 training sets of each gesture for 7 different users were collected in order to evaluate the
accuracy of the PME when it is trying to recognize gestures from new users. This was only
done with the right hand of the users. The dataset included a total of 630 gestures. Table 4
summarizes the gathered gestures for this approach.

10 training sets per gesture were selected because the user would need to perform 90
gestures in total, in general, the users began to become sloppy in their execution if more
gestures were required of them. No gestures of the Null category was collected from the
users due to the fact it is more tedious to collect them;

1. Category 1 to 9 needs to be collected.

2. The PME trained with the collected gestures.
3. Gestures which should be classified as no-match but the PME misclassifies as
gestures are collected as category zero.

Table 4: Collected training sets for the “few gestures from many users”. No gestures of the
Null category were collected.

Gesture category 1 2 3 4 5 6 7 8 9
User 1 10 10 10 10 10 10 10 10 10
User 2 10 10 10 10 10 10 10 10 10
User 3 10 10 10 10 10 10 10 10 10
User 4 10 10 10 10 10 10 10 10 10
User 5 10 10 10 10 10 10 10 10 10
User 6 10 10 10 10 10 10 10 10 10
User 7 10 10 10 10 10 10 10 10 10

Result and Discussion

KNN is unsuitable as a classifier for gesture recognition

When classifying a negative sample, the KNN will incorrectly classify the sample as a
category and thus give a false positive. A negative sample, in the context of gesture
recognition, is a sample which should not correspond to a valid gesture, i.e., it should be
classified as a no-match. Because the no-match category does not exist when the KNN is
used, the classifier will give false positives whenever negative samples are presented. For
the PoC this means that the PME will try to match every little movement a user does as a
gesture, even though most of a user’s movements will likely not correspond to a gesture in
the gesture vocabulary. This makes the KNN unsuitable as a classifier for gesture
recognition application.
39
Accuracy of the PME

Max Accuracy

In order to evaluate the maximum accuracy the PME can achieve, the data set many
gestures from one user (described in section 4.2.5) was used. The dataset contains, for both
the right and left hand, 90 sets of each of the 9 gestures and 90 sets with the Null category.
The sets of each hand were divided into two sets; half as a training set and half as a test set.
Thus one training set and one test set for the left hand and one training and one test set for
the right hand. Each of the two types sets contained 45 sets of gesture 1 to 9 and 45 sets of
the Null category. The training sets were used to train the PME which patterns to
recognize. Afterwards, the PME was used to try to correctly classify the gestures of the
verifications sets, the result can be seen in Figure 30. The presented result is the average
from the left and the right test sets. The average was used because the result from the left
and right test sets had no significant difference between them. The raw data can be found
in the appendix, under 7.1.1 in Table 12 to Table 15. Classification was done 4 times for
each hand, with different classification algorithms;

 RBF which cannot be uncertain (RBF certain). Originally included in CuriePME.

 RBF which can be uncertain (RBF uncertain). Added in a modified CuriePME.
 KNN with K=1. Originally included in CuriePME.
 KNN with K=3. Added in a modified CuriePME.

Note that the only difference between the RBF certain and RBF uncertain is that the
neuron status register is read after classification for RBF uncertain, as shown in Code 2.
This means that RBF uncertain will try to classify the wrong categories of the certain
version as uncertain categories. But it will also mistakenly classify some of the correctly
classified categories as uncertain. RBF uncertain might give better performance if RBF
certain has a high percentage of wrong classifications.

Observe that when the PME tries to classify a gesture of the Null category, the correct
classification will be the no-match category. This is because the Null category is used to
tell the PME which input patterns should not be classified as gestures. When the KNN K=1
or the KNN K=3 classifier are being used, the no-match category does not exist, which
means that gestures of the Null category cannot be classified as the no-match category and
are instead classified as gestures. Therefore the test sets used to calculate the results in
Table 5 does not include the Null category when KNN is being used, but it is included
when the RBF is used.

40
 Figure 30: Average correct classification by different classification algorithms when the
PME tries to classify the gestures from a user it has been trained to classify. The KNN
cannot correctly classify the Null category as a no-match because this category does not
exist when that classifier algorithm is being used.

Table 5: Average percentage of no-match, uncertain, wrong category and accuracy of

each classifier when the PME tries to classify the gestures from a user it has been trained
to classify.

RBF certain RBF Uncertain KNN K=1 KNN K=3

No-match 1% 1% 0% 0%
Uncertain 0% 0.67% 0% 0%
Wrong category 1.89% 1.89% 2.35% 11.23%
Accuracy 97.11% 96.44% 97.65% 88.77%

In Table 5 it is shown that the KNN K=1 is the most accurate classifier when presented
with test sets that contain positive samples. But this percentage will decrease if the
classifier is presented with negative samples (which should be classified as no-match) as it
cannot correctly classify those gestures. The second most accurate classifier is the RBF
certain, which is the classifier that will most likely perform best in the PoC system. RBF
uncertain might be a better alternative if the RBF certain has a high percentage of wrong
classifications, but as this is not the case here, thus the RBF certain has better performance.
The classifier with the worst performance is the KNN K=3.

41
New user

In order to evaluate how accurate the PME is to classify gestures from users which it has
not been trained to recognize, the data set few gestures from many users (described in
section 4.2.5) was used. The dataset contains 10 sets from each of the 9 gestures, from 7
different users. The PME was trained with the gestures from 6 of these users. After the
training was done, the PME was tasked with trying to classify 7 of the gestures from the
seventh user, the user which was excluded from the training set. This was done 7 times,
with each user excluded from the training one time and used as test set. A clarification of
how the training/verification was done can be seen in Figure 31.

Figure 31: Training and test sets for the seven training iterations when evaluating the
accuracy of the PME when encountering a new user. Each table in the figure corresponds
to one training iteration.

The classification was done 4 times for each excluded user, with different classification
algorithms; RBF certain, RBF uncertain, KNN with K=1 and KNN with K=3. The average
correct classification for the 7 training iteration with each classification algorithm can be
seen in Figure 32. The raw data from which the average was calculated from can be found
in the appendix, under 7.1.1 in Table 16.

42
 Figure 32: Average correct classification by different classification algorithms when
encountering a new user.

Table 6: Average percentage of no-match, uncertain, wrong category and accuracy of

each classifier when encountering a new user.

RBF certain RBF Uncertain KNN K=1 KNN K=3

No-match 13.15% 13.15% 0% 0%
Uncertain 0% 8.84% 0% 0%
Wrong category 3.86% 2.27% 7.94% 14.06%
Accuracy 82.99% 75.74% 92.06% 85.94%

From Table 6 it can be noted that KNN K=1 is the most accurate classifier when
encountering a new user, but a KNN classifier is not suitable for gesture classification as
mentioned on page 39. The main problem for RBF certain is that it has a high percentage
of no-match classifications. If RBF certain had a high percentage of wrong category
classifications, then RBF uncertain can be used instead, as it can classify some of the
wrong categories as uncertain categories. But in this case, RBF certain has a low
percentage of wrong categories, which means that RBF uncertain will perform poorly,
which it does according to Table 6. As suspected the accuracy of the PME is worse if it
tries to classify gestures from a new user, comparing to if it tries to classify gestures from a
user which it been trained to classify. One important aspect of the results from when the
PME tries to recognize a new user’s gestures is that the PME has not been trained with the

43
Null category. This means that the model of the PME might be overgeneralized. But the
high percentage of no-match for the RBF classifiers indicates that this might not be the
case.

New user with online training

The accuracy of the PME from Table 6 can be increased by letting the PME be trained
with some gestures from the new user before classification. This resembles a real scenario
because a gesture recognition application will most likely come with a base model, which
has been designed by the developers. A user will then fine-tune the model to better
recognize the user’s gestures by training the machine learning algorithm with only a few
gestures. A clarification of how the training/verification was done can be seen in Figure 33.

Figure 33: Training and test sets for the seven training iterations when evaluating the
accuracy of the PME when encountering a new user. The PME is trained with 3 sets of
each gesture from the new user. Each table in the figure corresponds to one training
iteration.

44
The average correct classification for the 7 training iteration with each classification
algorithm can be seen in Figure 34. The raw data from which the average was calculated
from can be found in the appendix, under 7.1.1 in Table 17.

Figure 34: Average correct classification by different classification algorithms when

encountering a new user. The PME is trained with 3 sets of each gesture from the new
user.

Table 7: Average percentage of no-match, uncertain, wrong category and accuracy of

each classifier when encountering a new user. The PME is trained with 3 sets of each
gesture from the new user.

RBF certain RBF Uncertain KNN K=1 KNN K=3

No-match 6.58% 6.58% 0% 0%
Uncertain 0% 8.62% 0% 0%
Wrong category 1.81% 0.67% 5.67% 11.34%
Accuracy 91.61% 84.13% 94.33% 88.66%

From Table 7 it can be seen that the accuracy is increased when the PME is trained with 3
sets of each gesture from the new user. By comparing Table 6 and Table 7, it can be seen
that the accuracy increased by ~10% for the RBF classifiers and ~2.5% for the KNN
classifiers. If more sets per gesture from the new user was used in the training, then the
accuracy will converge closer to the maximum accuracy from Table 5.

45
The results of this section prove that it is feasible to develop a gesture recognition
application using the PME that will have acceptable accuracy with the current gesture
vocabulary. It is important to remember that the current gesture vocabulary has quite large
gestures with strict start/end point. For instance, a clockwise circle must start at the top and
end at the top, see Figure 24. If the gestures were more fine-grained and/or had varied
starting/end points, then the accuracy of the PME is suspected to decrease noticeably. How
noticeable this decrease would be is out of the scope of this thesis.

Model used in the PoC

As described on page 16, the decision space of the PME depends on the order of which
training sets are used, it is therefore recommended to train with a random order and
multiple iterations until the decision space stabilizes. If the decision space does not
stabilize, then it means that the PME cannot accurately describe the supplied patterns and
their categories.

In order to generate the most accurate model, as many gestures as possible was used in an
offline training process. The setup that was used can be seen in Figure 28. For this reason,
all gathered data in many gestures from one user (described in section 4.2.5) was used as
training sets when models for the PoC were created. This means that 900 training sets of
the left hand and 900 training sets for the right hand were used in the offline training
process. For each iteration in Figure 35, one training set was picked at random and sent to
the PME to be used in the training. If a new neuron is committed then the sent set are
removed from the training sets, but if no new neuron is committed then the sent set is
reused again later in the training. The training was considered done once no new neurons
are committed for “number of training sets used”-iterations (900 iterations in this case).

Figure 35: Learning curve for when the model for the right and left hand were created by
offline training the PME. Both learning curve converges to a stable value, indicating that
the PME is able to distinguish between the different gestures in the gesture vocabulary.

46
Table 8: Total number of committed neurons that the offline training process converged to,
the time the training process took and the number of iterations, for both the left and right
hand.

Committed Neurons Time elapsed Iterations

Left hand 52 70.71 minutes 4189
Right hand 50 67.07 minutes 3980

From Table 8 it can be seen that less than half of the neurons are used. This means that the
model for the left and right hand could be implemented as different contexts, the PME has
enough neurons to store both models. This approach comes with the benefit that no new
model needs to be uploaded when the Curie switches between recognizing the left or right
hand. Having few committed neurons also means that the gesture vocabulary could be
extended with more gestures, as long the PME can distinguish between all gestures.

From Table 8 it can also be seen that the offline training process is slow, taking roughly
~70 minutes, for both the left and right hand. Training was slow because it was done for
many iterations, and for every iteration, a training set had to be sent via the serial USB
port, which is a slow means of communication. The training process would be faster if the
training sets did not have to be sent to the hardware for each iteration, this could be
achieved by using a software version of the hardware algorithms. General Vision provides
a software called NeuroMem Knowledge Builder [46] which provides this functionality,
this software is discussed in greater detail in section 5.1.

CuriePMe vs SoftPME

CuriePME is a library which provides the interface with which the programmer can
communicate and control the hardware PME. The neurons which the PME utilizes are all
connected in parallel, which means that the PME have the same execution time whether it
has 1 committed neuron or 128 committed neurons. Thus CuriePME has 𝑂(1) complexity.

To analyse the performance of the CuriePME, a software version of it were implemented

called SoftPME. It uses the same machine learning algorithms as CuriePME (KNN or
RBF), the only differences are that CuriePME uses hardware for its calculation and
SoftPME uses a software implementation. Three cases were studied: CuriePME run on a
Curie (it can only be run on a Curie), SoftPME run on a PC and SoftPME run on a Curie.

Size of the data stored in a neuron is 136 bytes, 128 bytes for the pattern point and 2 bytes
each for context, influence, minimum influence, and category. In the SoftPME library,
these 136 bytes per neuron needs to be stored as static data in the Curies SRAM. If no
other static variables are used in the sketch, the SoftPME can at maximum support 70

47
neurons due to the memory limitations of the Curie. But if the program uses other global
variables this number will decrease. For example, if SoftPME is used in the gesture
recognition sketch that is executed on the Curies, the maximum number of neurons is only
42. This is due to the fact that that sketch uses other variables which also takes up memory,
e.g., variables to keep track of the BLE characteristics.

To evaluate the performance differences between the hardware CuriePME and the software
SoftPME, the average time for 1 classification was calculated by doing 1000 consecutive
classifications for three cases;

1. CuriePME running on a Curie using the RBF classifier, which can at maximum
have 128 neurons.

2. SoftPME running on a Curie using the RBF classifier, which can at maximum have
70 neurons. The processor characteristic of the Curie is described in section 3.1.2.

3. SoftPME running on a PC using the RBF classifier, the processor of the PC was an
Intel Core i5-2380P CPU 3.10GHz. The maximum number of neurons was not
investigated for this device but is much higher than for case 1 or 2 because the
device does not have any memory limitations (in comparison to an embedded
device).

Figure 36: Comparison between the execution time for 1 classification using CuriePME,
SoftPME on a PC and SoftPME on a Curie. The topmost graph contains all three
execution times, the bottom graph contains execution time for CuriePME and SoftPME on
a PC.

48
Table 9: Speed boost by using CuriePME, SoftPME on a PC and SoftPME on a Curie. The
dotted line shows when CuriePME gets faster than SoftPME on a PC.

𝑺𝒐𝒇𝒕𝑷𝑴𝑬𝑪𝒖𝒓𝒊𝒆 𝑺𝒐𝒇𝒕𝑷𝑴𝑬𝑪𝒖𝒓𝒊𝒆 𝑪𝒖𝒓𝒊𝒆𝑷𝑴𝑬

Neurons
𝑪𝒖𝒓𝒊𝒆𝑷𝑴𝑬 𝑺𝒐𝒇𝒕𝑷𝑴𝑬𝑃𝐶 𝑺𝒐𝒇𝒕𝑷𝑴𝑬𝑃𝐶
10 28.4 258.0411 9.0860

20 56.7 269.8458 4.7592

30 85 274.0168 3.2237
40 113.32 263.3634 2.3241
50 141.62 267.0262 1.8855
60 169.92 269.5346 1.5862
70 198.22 267.6912 1.3505
80 1.1487

100 0.9518
120 0.7931
128 0.7403

From Figure 36 is seen that CuriePME have a constant execution time regardless of the
number of neurons committed, this is due to the fact that hardware neurons are all
connected in parallel. SoftPMEPC is fastest of the three cases when there are fewer than
~100 neurons committed after that CuriePME is the fastest. SoftPMECurie is the slowest of
the three cases. The execution time for SoftPMEPC and SoftPMECurie are both linear, if the
number of the committed neurons are doubled then the execution time is also doubled. The
complexity of SoftPME is 𝑂(𝑛). Table 9 shows that the SoftPME is ~270 times faster
when it is run on a PC

Even though SoftPMECurie has the slowest execution time, it still fast enough to be used in
the PoC system to classify gestures without noticeable slowing the system down. But if
machine learning was to be used in a system where classification was executed at a faster
rate, then the SoftPME would most likely not be suitable. The drawback of using the
SoftPME in the PoC system is that it can only have 42 committed neurons. Making it
unable to fit the 52 neurons for the left-hand model or the 50 neurons for the right-hand
model. If the SoftPME were to be used in PoC, then its models would need to be reduced
to be using a maximum of 42 neurons. This would decrease the accuracy of the PME.
Another aspect of using a software implementation instead of hardware is that the
computational load on the device increases, which also increases the overall power
consumption. No measurements of the power consumption were made for the thesis, but it
is likely that a software version of the PME will consume more power than the low power
optimized hardware version.

49
A software implemented machine learning algorithm is mainly limited by the memory
constraints of the device it is implemented on. But a software implementation is also more
versatile in how the available memory is used; it can be implemented to have few neurons
with large feature vectors of each neuron or have many neurons with shorter feature
vectors. A hardware implemented algorithm are more rigid in that is comes with a fixed
number of neurons and length of the feature vectors. Thus, a software implemented
algorithm can more easily be shaped to fit the problem at hand. The one main strength of
the hardware accelerated machine learning is that is not affected by the same memory
limitations and is able to provide many neurons with long feature vectors. Other benefits of
the hardware implementation are that it is faster and has an optimized power consumption.

Device manager
The duty of the DM is to control, configure, upload/download models, and synchronize the
Curies. Communication between the Curies and the DM are done with BLE. The Curies
need to be able to send the category data and upload their PMEs neuron data to the DM.
DM, on the other hand, needs to be able to control the Curies and be able to send new
neuron data to them. Therefore the system needs to be able to have two-way
communication between the Curies and the DM, which means peripheral/central
relationships is needed. The Curies are run in peripheral mode and the DM runs in central
mode. As seen in Figure 37, the BLE interface that was implemented is divided into 2
distinct services: a gesture service and a data service.

Figure 37: visualization of the BLE communication of the system.

50
 Gesture service

The gesture service has three characteristics: category characteristic, command

characteristic and synchronization characteristic.

Category characteristic

The category characteristic provides information about the last classified gesture
(category, start and end time) and the properties of the PME. The DM can read, write and
be notified when this characteristic changes values. It uses 14 of its 20 bytes to send
information.

Figure 38: Distribution of how the 20 bytes of the category characteristic is used and
distribution of the bits of the settings byte.

 Category byte[0:1]: The category of the gesture, is updated whenever the Curie
classifies a new gesture. The DM also uses this field to set which category the
Curie will train when it enters a training mode. The byte string is read from right to
left, with the most significant byte at index 0.
 Start time byte[2:5]: The start time of the gesture, is updated whenever the Curie
classifies a new gesture. Byte[2] is the most significant byte.
 End time byte[2:5]: The end time of the gesture, is updated whenever the Curie
classifies a new gesture. Byte[6] is the most significant byte.
 Settings byte[10]: Contain information about the current settings of the PME.
o Right arm bit[0]: Indicates whether or not the Curie is worn on the right
arm. Bit[0]=1 Curie worn on the right arm and bit[0]=0 Curie worn on the
left arm.
o Classifier bit[1]: Indicates which classifier function the Curie uses.
Bit[1]=1 Curie uses KNN and bit[1]=0 Curie uses RBF.
o Distance norm bit[2]: Indicates which distance norm the Curie uses.
Bit[2]=1 Curie uses Chebyshev distance and bit[2]=0 Curie uses Manhattan
distance.
o K bits[3:7]: The K-value used when the KNN classifier is being used. The
DM uses this field to set the value of K. Approved values for K is [1,21].

51
Command characteristic

The command characteristic provides an interface to control which state the Curie is in.
This characteristic is an unsigned char which enables the DM to write a number to the
Curie, this number then decides what the Curie should do.

 Classify, command = 1: puts the Curie in classification state. The Curie will
continuously collect sensor data and sense for starts and stops of gestures. The data
that is collected between the starts and stops will be sent to the PME for
classification. The category, start time and end time bytes in the category
characteristic will then be updated accordingly. When the category characteristic
has been updated a notification to the connected central device will be sent.
 Train, command = 2: puts the Curie in training state. The Curie will start to sense
after a gesture, this gesture data will then be used to train the PME. The category
that the data is trained with will be the category stored in the category bytes of the
category characteristic. Thus the category bytes needs to be set to the desired value
before the command characteristic is set to start the training.
 Initialize neuron data, command = 3: puts the Curie into initiation state, which
means that the base model is loaded into the PME. There are two different versions
of this model, one if the Curie is wielded on the left wrist and one for the right
wrist. Which model that is loaded is decided by which direction the gravitational
pull is affecting the accelerometers y-axis. If ay is negative the Curie is tilting to the
right and is assumed to be worn on the right wrist. If ay is positive then the Curie is
tilting to the left and is assumed to be worn on Curie left wrist. The category
characteristic is also updated to indicate which arm the Curie is worn on.
 Upload neuron data via BLE, command = 4: tells the Curie to start uploading its
neuron data to the DM. This is done with the out characteristic of the data service.
 Upload neuron data via serial, command = 5: tells the Curie to start uploading its
neuron data to the DM. This is done through the serial port of the Curie.
 Download neuron data via BLE, command = 6: tells the Curie to download
neuron data that is sent from the DM. This is done with the in characteristic of the
data service.
 Download neuron data via serial, command = 7: tells the device to download
neuron data that is sent from the DM. This is done through the serial port of the
Curie.
 Forget neuron, command = 8: tells the Curie to forget all its neuron data. After
this command, the Curie cannot recognize any of the gestures.
 Change classifier, Command = 9: change the classification function that the Curie
uses, between KNN and RBF. The category characteristic is also updated to
indicate which classification function the Curie uses.
 Set K value, Command = 10: Set the K value which is used in the classification
with the KNN function. The K value that is set is the value stored in the K byte in
the category characteristic. Thus the DM needs to set this byte to the desired value
before the command characteristic is set to 10.
52
Synchronization characteristic

The PoC system consists of 4 separate hardware devices: two Curies, one gateway (which
the DM is part of) and one application. Each hardware device has its own internal clock
with different start points. Synchronization of these clocks is necessary to determine if
gestures are concurrent or not and to determine the latency of the system.

Figure 39: Visualization of the variables used for calculating a mutual synchronization
point in time for all devices.

Synchronization is done by the DM by calculating a mutual synchronization point in time

0 0 0 0
for all devices, called 𝑡𝐶1 , 𝑡𝐶2 , 𝑡𝐷𝑀 and 𝑡𝐴𝑝𝑝 . Because all devices start their internal clock at
different points in time, these synchronization points will have different timestamps. The
DM sends out a synchronization massage to each device and measures the Round Trip
Time (RTT), which is the time it takes for a message to be returned from the device. The
message returned to the DM contains a timestamp of the moment that the other device
1 2 2 1 2 2
received the message, called 𝑡𝐶1 , 𝑡𝐶2 and 𝑡𝐴𝑝𝑝 .The DM thus has access to: 𝑡𝐶1 , 𝑡𝐶2 , 𝑡𝐴𝑝𝑝 ,
0 1 2 3 4 5
𝑡𝐷𝑀 , 𝑡𝐷𝑀 , 𝑡𝐷𝑀 , 𝑡𝐷𝑀 , 𝑡𝐷𝑀 and 𝑡𝐷𝑀 . The remaining variables in Figure 39 can be calculated
with the following equations:

53
Equations Curie 1
1 0
𝑅𝑇𝑇𝐶1 = 𝑡𝐷𝑀 − 𝑡𝐷𝑀 (12)

0 1
𝑅𝑇𝑇𝐶1
𝑡𝐶1 = 𝑡𝐶1 − (13)
2
Equations Curie 2
3 2
𝑅𝑇𝑇𝐶2 = 𝑡𝐷𝑀 − 𝑡𝐷𝑀 (14)

1 2
𝑅𝑇𝑇𝐶2
𝑡𝐶2 = 𝑡𝐶2 − (15)
2
0 1 0 2
𝑡𝐶2 = 𝑡𝐶2 + 𝑡𝐷𝑀 − 𝑡𝐷𝑀 (16)

Equations Application
5 4
𝑅𝑇𝑇𝐴𝑝𝑝 = 𝑡𝐷𝑀 − 𝑡𝐷𝑀 (17)

1 2
𝑅𝑇𝑇𝐴𝑝𝑝
𝑡𝐴𝑝𝑝 = 𝑡𝐴𝑝𝑝 − (18)
2
0 1 0 4
𝑡𝐴𝑝𝑝 = 𝑡𝐴𝑝𝑝 + 𝑡𝐷𝑀 − 𝑡𝐷𝑀 (19)

Equation (13), (15) and (18) work under the assumption that the transmission time to and
from a device takes the same amount of time. For instance, for the case of Curie 1, this
𝑅𝑇𝑇𝐶1
means that one transmission takes .
2

0 0 0 0
The DM calculates the synchronization points, 𝑡𝐶1 , 𝑡𝐶2 , 𝑡𝐷𝑀 and 𝑡𝐴𝑝𝑝 , for all devices and is
there for aware of their internal clocks. The other devices on the other hand is not aware of
this information. The DM uses this information to convert the timestamps of the gestures
from their respective Curies’ internal clock into the internal clock of the application, as
seen in Figure 40. There are two reasons for this:

1) If the gestures from the two Curies have the same internal clock, then the fusion
component can determine if the gestures are concurrent. More on this in section
4.4.
2) If the timestamps of the Curies gestures are synchronized after the applications
internal clock, then the application is aware at what point in time a gesture
classification is performed. For some systems, this knowledge might be vital.

54
 ′ ′ ′ ′
Figure 40: 𝑡𝐶1 , 𝑡𝐶2 , 𝑡𝐷𝑀 𝑎𝑛𝑑 𝑡𝐴𝑝𝑝 is the synchronized time between all devices.

Data service

The data service has two characteristics: an in characteristic and an out characteristic. The
naming is done from the devices point of view, the in characteristic is for data that is sent
to the device and the out characteristic is for data that is sent out. The data that is sent
through these characteristics is the neuron data for the PME.

Size of the data stored in a neuron is 136 bytes, 128 bytes for the pattern point and 2 bytes
each for context, influence, minimum influence, and category. A characteristic can send at
maximum 20 bytes at a time, this means that the neuron data needs to be divided into
different packages. These packages have the same structure for both the in- and the out
characteristics. Context, influence, minimum influence and category (2 bytes each) have
the byte order big-endian, meaning that the most significant byte is the first byte in the byte
string.

55
 Figure 41: Neuron data is sent as packages with the in and out characteristics. All
packages contain package number, neuron number, the total number of neurons and a part
of the neuron’s pattern point. The exception is the first package which also contains the
neuron’s context, influence, minimum influence and category.

As seen in Figure 41 it takes 8 packages to send the neuron data of one neuron to/from the
Curie. This needs to be done for every committed neuron. For instance, 30 committed
neurons mean that 30*8=240 packages need to be sent. The neuron number indicates how
many neurons we have sent and the neuron total indicates how many neurons the DM or
Curie is going to send. When the eight package with neuron number equal to neuron total
has been received, then the receiver knows that all data has been sent.

While uploading neuron data, the Curie is the sender and the DM is the receiver, and while
downloading the roles are reversed. Upload and download can be implemented in two
different methods; with acknowledgement and without acknowledgement.

Figure 42 visualize how transmissions are done without acknowledgement, the sender will
send each package as fast as possible without regard for the receiving device. In the worst
case scenario, the receiving device will not have enough time to handle all the data in time.
With this method, there is no guarantee that the receiving device will be done handling a
package before the next package is sent.

56
 Figure 42: uploading and downloading neuron data without acknowledgements. For
uploading: ΔtCurie is the measured time taken for the Curie to send the neuron data, ΔtDM is
the measured time taken for the DM to receive and handle the neuron data. For
downloading: ΔtC is the measured time taken for the Curie to receive and handle the
neuron data, ΔtDM is the measured time taken for the DM to send the neuron data.

Figure 43 visualize how transmissions are done with acknowledgement, the sender will
wait for the receiver to acknowledge that a package has been read before it will send the
next package. There will, therefore, be twice as many messages needed because for each
package sent an acknowledgement also needs to be sent. This case will be slower than
without acknowledgement, but it is also more robust. If a transmission suffers from
package loss then the receiver can ask the sender to resend a package with the
acknowledge message. This is not implemented, instead, the acknowledge message is only
used to tell the sender when it can send the next package. The receiver is assured that it
only needs to handle one package at a time, but the system is still vulnerable to package
loss.

57
 Figure 43: uploading and downloading neuron data with acknowledgements. For
uploading: ΔtCurie is the measured time taken for the Curie to upload its neuron data, ΔtDM
is the measured time taken for the DM to receive and handle the neuron data. For
downloading: ΔtC is the measured time taken for the Curie to receive and handle the
neuron data, ΔtDM is the measured time taken for the DM to send the neuron data.

Result and Discussion

Synchronization of the system

To evaluate the synchronization characteristics of the system, a synchronization command

is carried out every minute for six hours. For every synchronization, the current time of
each internal clock is calculated. A clock drift for each internal clock is then calculated
with equation (20). The clock drift compares the elapsed time between the device (Curie 1,
Curie 2 and DM) and the application. Calculated clock drift is displayed in Figure 44.

𝑐𝑙𝑜𝑐𝑘 𝑑𝑟𝑖𝑓𝑡𝑑𝑒𝑣𝑖𝑐𝑒 = 𝑡𝑖𝑚𝑒 𝑒𝑙𝑎𝑝𝑠𝑒𝑑𝑑𝑒𝑣𝑖𝑐𝑒 − 𝑡𝑖𝑚𝑒 𝑒𝑙𝑎𝑝𝑠𝑒𝑑𝑎𝑝𝑝 =

𝑖 0 𝑖 0
= (𝑡𝑑𝑒𝑣𝑖𝑐𝑒 − 𝑡𝑑𝑒𝑣𝑖𝑐𝑒 ) − (𝑡𝑎𝑝𝑝 − 𝑡𝑎𝑝𝑝 ) (20)
𝑖 = 𝑖𝑡𝑒𝑟𝑎𝑡𝑖𝑜𝑛, 𝑑𝑒𝑣𝑖𝑐𝑒 = [𝐶𝑢𝑟𝑖𝑒 1, 𝐶𝑢𝑟𝑖𝑒 2, 𝐷𝑀]

58
 Figure 44: clock drifts for Curie 1, Curie 2 and DM. The drift is defined as the time
difference between a device and the application.

From Figure 44, after six hours the time difference between the devices (Curie 1, Curie 2
and DM) and the application is:

 Both Curies’ clocks run roughly 1.1 seconds faster than the applications clock.
 DM’s clock is roughly 0.38 seconds faster than the applications clock.

How accurate the synchronization must be in a system, in order for the system to function
properly, depends on the task of the system. In the PoC system, we set that the internal
clocks cannot differ more than 0.1 seconds from each other. A limit of 0.1 seconds will
give the system acceptable synchronization accuracy. Equation (21) and (22), calculates
how often the system needs to synchronize the Curies and DM to keep them within the 0.1
second limit of the application clock. The Curies need to be synchronized every 32.72
minutes and the DM need to be synchronized every 94.74 minutes.

𝑙𝑖𝑚𝑖𝑡𝐶𝑢𝑟𝑖𝑒 0.1𝑠 1
= ∗ = 32.72 𝑚𝑖𝑛𝑢𝑡𝑒𝑠 (21)
𝑑𝑟𝑖𝑓𝑡 𝑝𝑒𝑟 𝑠𝑒𝑐𝑜𝑛𝑑 1.1𝑠⁄ 60
(6 ∗ 60 ∗ 60)𝑠

𝑙𝑖𝑚𝑖𝑡𝐷𝑀 0.1𝑠 1
= ∗ = 94.74 𝑚𝑖𝑛𝑢𝑡𝑒𝑠 (22)
𝑑𝑟𝑖𝑓𝑡 𝑝𝑒𝑟 𝑠𝑒𝑐𝑜𝑛𝑑 0.38𝑠⁄(6 ∗ 60 ∗ 60)𝑠 60

The calculations used in the synchronization work under the assumption that the RTT is
symmetrical, meaning that the synchronization message takes as much time from the DM
59
to the device as it takes from the device to the DM. The fluctuating RTT values shown in
Figure 45 might indicate that this assumption might not be correct. This fluctuation can be
seen in Figure 44 as the noise in the clock drift, but this noise has a smaller magnitude than
the drift because the test was run for a long period of time.

Average RTT for Curie 1 and Curie 2 are 108.48ms and 85.27ms, respectively. RTT of the
Curies is dependent on outside factors, such as location/orientation of devices, and
background noise from other devices. Both Curies were lying still during the test, with the
same distance to the DM, thus the test was expected show no significant difference
between the two Curies. From Figure 45 it can be noted that the shape of the graphs of
Curie 1 and Curie 2 differ. Curie 2 have RTT ~70ms in most cases, with a few cases were
the RTT double to around ~140ms. The doubled RTT might be caused by lost packages in
the transmission, which demands that the RTT message is resent. But this explanation was
never confirmed and should thus be considered a speculation. Curie 1, on the other hand,
start at ~70ms and steadily increases its RTT until it reaches ~140ms, at this point, it starts
over at ~70ms again. A feasible explanation for why the two Curies behave differently was
never found and is therefore left for future work. One reasonable

Figure 45: RTT for the devices.

Figure 46 shows the time difference between the two Curies and a linear equation fitted to
the data called 𝑑𝑟𝑖𝑓𝑡𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒 . The fluctuating values in Figure 46 are a result from the fact
that the fluctuating RTTs values (depicted in Figure 45) are used in the calculations of the
60
Curie’s current time. The large start value of ~4060 ms means that Curie 1 started ~4
seconds before Curie 2 started. 𝑑𝑟𝑖𝑓𝑡𝑒𝑠𝑡𝑖𝑚𝑎𝑡𝑒 gives an approximation of the clock drift
between the Curies; the clock of Curie 2 runs 0.1003ms faster than the clock of Curie 1
every minute. This means that the two Curies needs to be synchronized every 16.62 hours
if their clocks should be within 0.1 seconds of each other, according to equation (23).

𝑙𝑖𝑚𝑖𝑡𝐸𝐷 0.1𝑠 1
= ∗ = 16.62 ℎ𝑜𝑢𝑟𝑠 (23)
𝑑𝑟𝑖𝑓𝑡 𝑝𝑒𝑟 𝑠𝑒𝑐𝑜𝑛𝑑 0.1003⁄ 𝑠 60
1000

Figure 46: calculated time difference between Curie 1 and Curie 2.

Upload and download via BLE

Upload and download speed of the PoC were analysed by transmitting the data of 50
neurons (neurons needed for the right-hand model) 10 times, this result is shown in Figure
47 and Figure 48. The average speed for transmitting the neuron data of one neuron was
then calculated, which is shown in Table 10. The transmission time, for a Curie (ΔtC) and
the DM (ΔtDM), was measured as shown in Figure 42 and Figure 43. All measurements
were done without moving the devices. Should the measurements be done again the result
would most likely be slightly different. This is because the BLE wireless transmission is
affected by localization of the devices and if there is background noise from other devices
61
present. Results of measured time, therefore, serve as an indication of the order of
magnitude for the time needed for uploading and downloading neuron data.

Figure 47: Measured time for uploading the neuron data from 1 neuron via BLE.

Figure 48: Measured time for downloading the neuron data from 1 neuron via BLE.

62
Table 10: The average time for uploading and downloading neuron data from 1 neuron via
BLE. These values are represented as dotted lines in Figure 47 and Figure 48.

𝐀𝐯𝐞𝐫𝐚𝐠𝐞 𝚫𝐭 𝐂𝐮𝐫𝐢𝐞 [ms] 𝐀𝐯𝐞𝐫𝐚𝐠𝐞 𝚫𝐭 𝐃𝐌 [ms]

Uploading with acknowledgement 875 877
Uploading without acknowledgement 402 403
Downloading with acknowledgement 929 928
Downloading without acknowledgement 168 3

As expected, Table 10 shows that transmitting without acknowledgement is faster than

transmitting with acknowledgement. This is because transmission with acknowledgement
requires that an extra acknowledgement message is transmitted for every package.

It is reasonable to expect that transmission time for uploading and downloading will be of
the same magnitude because it is the same amount of data that is being transmitted. The
only difference is the direction. But this is not the case when the transmission is done
without acknowledgement. This is because the Curie was implemented to have a delay of
50ms between each sent package while uploading. Without this delay, the connection
between the Curie and DM will be lost once the uploading process starts. A short delay
would increase the risk of losing the connection to the system. Thus a delay of 50ms was
chosen as a trade-off between the transmission time and risk of connection being lost. This
limitation is probably due to the library [43] used to implement the BLE communication
for the DM. This conclusion is based on the fact that tests were performed that showed that
a Curie is able to send its neuron data to another Curie without the transmitting device
having the 50ms delay, which means that the limitation probably is not at the Curie.

In the current implementation, a lost connection means that the DM needs to be restarted in
order to re-establish the connection. In a real system, this is not optimal and it is left for
future work to implement a system which can re-establish lost connections automatically.

Upload and download via Serial

Uploading and downloading of neuron data via serial are done in the same way, the
difference is the direction that the data is transmitted. The transmission is done by first
sending the total number of neurons that is going to be transmitted. The receiving device
will den be able to tell when all neurons have been transmitted. Neuron data of each
neuron is then sent sequentially via the serial port in the order: context, influence,
minimum influence and pattern point.

63
Figure 49: Measured time for uploading and downloading the neuron data from 1 neuron
via serial

Table 11: The average time for uploading and downloading neuron data from 1 neuron via
serial. These values are represented as dotted lines in Figure 49.

𝐀𝐯𝐞𝐫𝐚𝐠𝐞 𝚫𝐭 𝐂𝐮𝐫𝐢𝐞 [ms] 𝐀𝐯𝐞𝐫𝐚𝐠𝐞 𝚫𝐭 𝐃𝐌 [ms]

Uploading 35 35
Downloading 19 10

From Table 11 it can be seen that uploading takes the same amount of time for both
devices, while the DM takes roughly half the time as the Curie takes while downloading.
Uploading and downloading via serial is significantly faster than doing it via BLE.

Fusion
The task of the fusion component is to take the one-handed gesture output from the Curies
and determined if they match any of the gestures of the two-handed gesture vocabulary.
For the gestures to be classified as two-handed gestures, then they need to be performed
simultaneously. Two gestures are said to be simultaneous if the condition of equation (24)
is met. If gestures were not simultaneous, then they are treated as separate gestures.

|𝑠𝑡𝑎𝑟𝑡 𝑡𝑖𝑚𝑒𝑟𝑖𝑔ℎ𝑡 − 𝑠𝑡𝑎𝑟𝑡 𝑡𝑖𝑚𝑒𝑙𝑒𝑓𝑡 | < 250 𝑚𝑠 (24)

|𝑒𝑛𝑑 𝑡𝑖𝑚𝑒𝑟𝑖𝑔ℎ𝑡 − 𝑒𝑛𝑑 𝑡𝑖𝑚𝑒𝑙𝑒𝑓𝑡 | < 250 𝑚𝑠
64
 Two-handed gesture vocabulary

With two Curies, one for each arm, the system can detect gestures done with two hands.
The gesture vocabulary was extended with 4 additional gestures which required two hands
to perform, seen in Figure 50. These additional gestures were defined as a combination of
the original 9 gestures, e.g., a zoom in is a right swipe with the right hand and a left swipe
with the left hand. Thus the PME was not needed to be trained with new gestures in order
to make the system able to detect the two-handed gestures.

Figure 50: Extended gesture vocabulary for gesture recognition with two hands.

Gateway
The gateway is made up of the DM and fusion component. The task of the gateway is to
simplify the communication between the application to the Curies by acting as an
intermediary. By doing this it hides the complexity of the distributed machine learning and
thus decouples the application from the distributed machine learning network.

The gateway has two wireless interfaces, one BLE interface to communicate with Curies
and one MQTT interface communicate with the end application. The BLE interface is
covered in section 4.3 and the MQTT interface is covered in this section. The MQTT
interface only uses one topic, called gateway/gesture. Data sent via the gesture topic are

65
formatted as JavaScript Object Notation (JSON). JSON format enables data to be
transmitted as attribute-value pairs. The attributes of the gesture topics are; category,
startTime, endTime, and hand.

 category: The possible values for the category attribute is 1 to 13 (corresponds to

the category of the gesture seen in Figure 24 and Figure 50), 32767 (0x7FFF) for
unknown gestures and 32768 (0x8000) for when the Curies are uncertain of which
gesture is performed.
 startTime: Start time of the gesture.
 endTime: End time of the gesture.
 hand: Which hand that performed the gesture. Possible values are: left, right and
both (when a two-handed gesture is performed).

Application
The application is a modified version of the Wormy game by Al Sweigart [47]. Wormy is
basically a Snake-clone in which a snake must collect apples in order to grow longer. The
modifications make the game controllable with gestures.

 Clockwise, category 2: Pause game

 Counterclockwise, category 3: Unpause game
 Cross, category 4: Exit game
 Swipe down, category 6: Turn the snake downwards
 Swipe left, category 7: Turn the snake left
 Swipe right, category 8: Turn the snake right
 Swipe up, category 9: Turn the snake upwards
 Zoom in, category 10: Zoom in towards the snakes head by increasing the size of
each cell in the map, making the window fit fewer cells. A visualization of zoom
in/out is shown in the top pictures in Figure 51.
 Zoom out, category 11: Zoom out from the snakes head by decreasing the size of
each cell in the map, making the window fit more cells. A visualization of zoom
in/out is shown in the top pictures in Figure 51.
 Rotate left, category 12: Rotate the map 90° to the left, see bottom pictures in
Figure 51. The snake rotates around its own head, while the apple rotates around
the centre of the map.
 Rotate right, category 13: Rotate the map 90° to the right, see bottom pictures
Figure 51. The snake rotates around its own head, while the apple rotates around
the centre of the map.

66
 Figure 51: Example of how zoom in/out and rotate left/right works.

Result

Thanks to the PoC system, a user is able to control the Snake game via gestures. But if the
game is run at the same speed as it does regularly, the user will not have enough time to
make the control gestures. This is because of how the PoC collects its sensor data, the
gathering of data follows the pattern:

1) Curie stands still  no recording

2) Curie starts to move  record gesture
3) Curie stands still  stops recording and send the gesture to get classified

This means that a gesture needs to start with standing still and end with standing still,
which makes the gesture recognition slow. Since Snake is a game where it is beneficial to
have quick reactions and responsive controls, the slow gesture recognition makes it hard to
play. In order to counteract this limitation, the game is run at a slower speed when gestures
are used as controls.

As stated in section 4.2.4, the equation to detect zero motion requires that the Curie must
be standing still for 15 samples before zero motion is detected. Section 4.2.2 defines that
the sampling frequency of the sensors is 200 Hz. This means that a Curie must be standing
still for 0.075 seconds (Eq. (25)) before the gesture recording stops and the classification
can begin.

1
∗ 15 𝑠𝑎𝑚𝑝𝑙𝑒𝑠 = 0.075 𝑠 (25)
200 𝐻𝑧

67
5. Conclusion
As of July 18, 2017, Intel announced that the Curie module and the development board
Arduino 101 will be discontinued [48] [49]. The TinyTILE, which have the Curie as its
core, will also be discontinued [50]. As of November 25, 2017, no news about if the Quark
will be discontinued or not has been released. If the Quark were to be discontinued in the
future, then its product specification page [51] will likely be updated with this information.
The reader is advised to investigate for himself/herself if the Quark is still commercially
available at the time of reading this thesis report. An alternative to the Quark is the other
chips from General Vision, NM500 and CM1K, which supplies more neurons with lager
feature vectors in each neuron than the Quark does. More complex machine learning
problems can be solved at the edge nodes if one of these chips from General Vision was
utilized.

The Curie module can be replaced with the Quark, motion sensor BMI160 [6] and BLE
NRF51822 [7] since these three devices provides the same functionality needed for the
PoC. This means that the PoC still can be implemented even though the Curie has been
discontinued. The PoC and the Curie show that hardware accelerated machine learning for
wearable embedded devices is feasible to achieve.

NeuroMem is not the only hardware technology to bring low power and high-performance
machine learning to edge nodes. For instance, the Movidius™ Neural Compute Stick
(NCS) [52] which provides hardware accelerated Convolutional Neural Network
capabilities. NCS is designed to look like a USB storage device and can be plugged into
the USB port of x86_64 computer running Ubuntu 16.04 or an RPi. In the future, it is
reasonable to believe that there will be even more devices with hardware accelerated
machine learning.

As shown in section 4.2.6, it is possible to implement machine learning on an embedded

device without it being hardware accelerated, but this might not be the best approach for
most situations. Machine learning implemented as software is limited by the memory
constraints of an embedded device, there might not be enough memory to solve the desired
machine learning problem. This approach also increases the computational load on the
device, which will increase the overall power consumption. For a low powered and
constrained devices, hardware accelerated machine learning is likely the best approach to
implement edge intelligence.

SoftPME have its limitations if it is being deployed on an embedded device, but a similar
system to the PoC could be implemented, if a more powerful smartwatch were to be used
instead of the Curie, for instance, the following setup is plausible:

 Smartwatch instead of the Curie: running SoftPME in an app, this app could, for
instance, be used to perform gesture recognition.
 Smartphone instead of Gateway: Communicates with the smartwatch and
forwards the result of the SoftPME to an end application.
68
  TV/console or some other device as an end application: For instance, if the
smartwatch is performing gesture recognition, then the TV could be controlled by
the gestures (volume up/down, change channel etc.).

Future work
There are two points of improvements to be done in order to get the synchronization of the
PoC to work properly;

1) The system needs to be able to synchronize itself when the internal clocks of the
devices drift too far from each other. In its current state, a user must command the
DM to synchronize the system but it is desirable to have a self-regulating system.
2) An explanation for the strange fluctuating RTT values shown in Figure 45 needs to
be found. If the RTT cannot be measured with acceptable accuracy the
synchronization of the system will perform poorly.

The implementation of the training function for SoftPME has never been tested if it
functions properly, i.e., that it creates an identical model that a Curie would if it were
trained with the same training sets. It might be a good idea to determine this functionality
so that the SoftPME can be used when the base models are created. In this thesis, the
models were created by sending training data over the serial port to a Curie, but as shown
in Table 8, this is a time-consuming process. But if the SoftPME were to function properly,
then the whole training process could be done on a computer, thus shortening the training
process from a ~hour to a couple of minutes. An alternative for shortening the training
process is to use the software called NeuroMem Knowledge Builder [46] (NKB) from
General Vision. The NKB allows offline learning, recognizing, clustering patterns and
generating diagnostics reports about the performance of the NeuroMem neural network. It
achieves this by simulating the architecture of the NeuroMem with 4096 neurons with 256
bytes feature vectors. This software was not used for this thesis, but a reader which is
interested in using the NeuroMem is advised to look into this application.

Gestures defined in the gesture vocabulary for the PoC are obtrusive, in that they require
relatively large hand movements to be perceived. This worked fine for some purposes, e.g.,
if they are used to play a game, but for everyday use, it might be a bit cumbersome to make
these large gestures. For everyday use, it could be better if the system could recognize
small and fine-grained gestures, e.g., pinching, tapping or rubbing one's fingers. Whether
or not the PoC would be able to perceive gestures of this kind was never evaluated and is
there for left for future work.

In the current implementation, the gesture recognition uses an activation to start recording
gestures (Curie starts moving) and a deactivation to stop recording and send the data to
classification (Curie stops moving), this makes the gesture recognition slower than it has to
be. An improvement would be that if gesture recognition were done without needing some

69
kind of activation/deactivation, by recognising gestures from a continuous flow of samples
from the sensors.

Figure 45 shows that the RTT measurements have some strange behaviour and the
accuracy of these measurements are vital to how well the synchronization preforms. An
analysis of the accuracy of the synchronization was never performed, but it might be
valuable to perform one in order to get a sense of how accurate the synchronization method
actually is. It is also left for future work to investigate how the synchronization is affected
by interference, localization and the number of devices that needs to be synchronized. The
PoC would benefit if the system itself could tell when the internal clocks of the system
have drifted too far from each other and then perform an automatic synchronization.

70
6. References

[1] Ericsson, “Ericsson - Networked society,” 13 October 2017. [Online]. Available:

https://www.ericsson.com/en/networked-society.

[2] M. P. Kumar, “Find out how AI Everywhere is key to the infinite edge,” 7 September 2017.
[Online]. Available: https://www.ericsson.com/thinkingahead/the-networked-society-
blog/2017/09/07/ai-everywhere-edge-compute-startups/.

[3] Intel, “Intel Curie module datasheet,” [Online]. Available:

http://www.intel.com/content/dam/support/us/en/documents/boardsandkits/curie/intel-
curie-module-datasheet.pdf. [Accessed 10 May 2017].

[4] Intel, “Intel Quark SE Microcontroller C1000 datasheet,” February 2017. [Online]. Available:
https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/quark-
c1000-datasheet.pdf. [Accessed 5 August 2017].

[5] Intel, “Intel® Curie™ Module - Intel news fact sheet,” 18 August 2015. [Online]. Available:
http://download.intel.com/newsroom/kits/ces/2015/pdfs/Intel_CURIE_Module_Factsheet.p
df.

[6] Bosch, “BMI160 datasheet,” [Online]. Available: http://www.mouser.com/ds/2/783/BST-

BMI160-DS000-07-786474.pdf. [Accessed 27 July 2017].

[7] Nordic Semiconductor, “nRF51822 - Overview,” [Online]. Available:

https://www.nordicsemi.com/eng/Products/Bluetooth-low-energy/nRF51822. [Accessed 28
May 2017].

[8] “BQ25101H datasheet,” 25 November 2017. [Online]. Available:

http://www.ti.com/product/BQ25101H.

[9] Arduino, “Arduino 101,” [Online]. Available: https://store.arduino.cc/genuino-101. [Accessed

5 August 2017].

[10] element14, “tinyTILE,” [Online]. Available:

https://www.element14.com/community/docs/DOC-82913. [Accessed 27 July 2017].

[11] Intel, “Intel Quark SE microcontroller - Product brief,” 25 November 2017. [Online].
Available: https://www.intel.com/content/dam/www/public/us/en/documents/product-
briefs/quark-se-soc-product-brief.pdf.

[12] Intel, “CurieIMU,” Intel, [Online]. Available:

https://www.arduino.cc/en/Reference/CurieIMU. [Accessed 18 May 2017].

71
[13] Intel, “Github - CurieIMU,” Intel, [Online]. Available: https://github.com/01org/corelibs-
arduino101/tree/master/libraries/CurieIMU. [Accessed 18 May 2017].

[14] General Vision, “NeuroMem - General Vision Inc,” [Online]. Available: http://www.general-
vision.com/datasheet/DS_NeuroMem_Techno.pdf. [Accessed 11 May 2017].

[15] P. Clarke, “Intel discusses neural network inside Quark MCU,” 7 July 2016. [Online].
Available: http://www.eenewsanalog.com/news/intel-discusses-neural-network-inside-
quark-mcu. [Accessed 15 July 2017].

[16] General Vision, “RBF Tutorial,” 16 dec 2017. [Online]. Available: http://www.general-
vision.com/download/rbf-tutorial/.

[17] General Vision, “NeuroMem - RBF Decision Space Mapping,” [Online]. Available:
http://www.general-
vision.com/documentation/TM_NeuroMem_Decision_Space_Mapping.pdf. [Accessed 10
May 2017].

[18] General Vision, “NeuroMem Technology Reference Guide,” [Online]. Available:

http://www.general-
vision.com/documentation/TM_NeuroMem_Technology_Reference_Guide.pdf. [Accessed
10 May 2017].

[19] General Vision, “CurieNeurons Library,” General Vision, [Online]. Available:

http://www.general-vision.com/software/curieneurons/. [Accessed 19 May 2017].

[20] General Vision, “CurieNeurons presentation,” [Online]. Available: http://www.general-

vision.com/publications/PR_CurieNeuronsPresentation.pdf. [Accessed 17 May 2017].

[21] Intel, “Github - CuriePME,” Intel, [Online]. Available: https://github.com/01org/Intel-Pattern-

Matching-Technology#curiepmereadneuron. [Accessed 18 May 2017].

[22] General Vision, “NM500 - NeuroMem,” 25 November 2017. [Online]. Available:

http://www.general-vision.com/hardware/nm500/.

[23] General Vision, “CM1K - NeuroMem,” 25 November 2017. [Online]. Available:

http://www.general-vision.com/hardware/cm1k/.

[24] Intel, “CurieBLE library,” Intel, [Online]. Available:

https://www.arduino.cc/en/Reference/CurieBLE. [Accessed 11 May 2017].

[25] G. B. Alexandru, “A view on Bluetooth Low Energy stack roles,” 9 October 2014. [Online].
Available: https://community.nxp.com/thread/332319. [Accessed 28 May 2017].

[26] Nordic Semiconductor, “S130 SoftDevice v1.0.0 Specification,” 12 June 2015. [Online].
Available: http://infocenter.nordicsemi.com/pdf/S130_SDS_v1.0.pdf. [Accessed 8 June

72
 2017].

[27] Nordic Semiconductor, “S130 SoftDevice v2.0 Specification,” 8 April 2016. [Online].
Available: http://infocenter.nordicsemi.com/pdf/S130_SDS_v2.0.pdf. [Accessed 7 June
2018].

[28] “Github forum - CurieBLE: Adding more than 6 characteristics to a service causes device to
crash when attempting to advertise,” 20 February 2017. [Online]. Available:
https://github.com/01org/corelibs-arduino101/issues/446. [Accessed 26 May 2017].

[29] “Git - Device locks up using CurieIMU,” 23 March 2017. [Online]. Available:
https://github.com/01org/corelibs-arduino101/issues/495. [Accessed 6 June 2017].

[30] f. -. F. Moderator, “Arduino 101 Battery charger,” 4 January 2016. [Online]. Available:
https://forum.arduino.cc/index.php?topic=368275.0.

[31] shabaz, “TinyTILE Battery charge,” 23 March 2017. [Online]. Available:

https://www.element14.com/community/docs/DOC-84366/l/tinytile-differencespdf.

[32] Intel, “Intel® Curie™ Module Design Guide,” 25 November 2017. [Online]. Available:
https://www.intel.com/content/dam/support/us/en/documents/boardsandkits/curie/intel-
curie-module-design-guide.pdf.

[33] Raspberry Pi Foundation, “Raspberry Pi 3 Model B datasheet,” [Online]. Available:

http://docs-europe.electrocomponents.com/webdocs/14ba/0900766b814ba5fd.pdf.
[Accessed 5 August 2017].

[34] Raspberry Pi, “RASPBERRY PI 3 MODEL B,” Raspberry Pi Foundation, [Online]. Available:
https://www.raspberrypi.org/products/raspberry-pi-3-model-b/. [Accessed 7 June 2017].

[35] RS Components, “Raspberry Pi 3 Model B SBC,” 25 November 2017. [Online]. Available:

https://uk.rs-online.com/web/p/processor-microcontroller-development-kits/8968660/.

[36] “MQTT,” [Online]. Available: http://mqtt.org/. [Accessed 7 June 2017].

[37] X. C. (. p. w. u. t. g. P. R. Jain), “Constrained Application Protocol for Internet of,” 5 May 2014.
[Online]. Available: https://www.cse.wustl.edu/~jain/cse574-14/ftp/coap.pdf. [Accessed 16
July 2017].

[38] R. Light, “MQTT — MQ Telemetry Transport,” Mosquitto, [Online]. Available:

http://mosquitto.org/man/mqtt-7.html. [Accessed 7 June 2017].

[39] Wikipedia, “Snake (video game),” 22 July 2017. [Online]. Available:

https://en.wikipedia.org/wiki/Snake_(video_game). [Accessed 6 August 2017].

[40] Arduino, “Arduino programming Language,” [Online]. Available:

73
 https://www.arduino.cc/en/Reference/HomePage. [Accessed 6 Augusti 2017].

[41] Arduino, “Arduino IDE,” [Online]. Available: https://www.arduino.cc/en/Main/Software.

[Accessed 6 Augusti 2017].

[42] Python Software Foundation, “Python 3.6,” [Online]. Available: https://docs.python.org/3.6/.

[Accessed 6 Augusti 2017].

[43] I. Harvey, “Bluepy - Python interface to Bluetooth LE on Linux,” [Online]. Available:

https://github.com/IanHarvey/bluepy. [Accessed 16 May 2017].

[44] Eclipse, “Paho™ MQTT Python Client,” [Online]. Available:

https://github.com/eclipse/paho.mqtt.python. [Accessed 20 May 2017].

[45] Intel, “Drawing in the air,” 13 March 2017. [Online]. Available:

https://github.com/01org/Intel-Pattern-Matching-
Technology/blob/master/examples/DrawingInTheAir/DrawingInTheAir.ino. [Accessed 23
May 2017].

[46] General Vision, “NeuroMem Knowledge Builder,” 18 October 2017. [Online]. Available:
http://www.general-vision.com/documentation/TM_NeuroMem_KB.pdf.

[47] A. Sweigart, “Wormy game,” [Online]. Available:

https://inventwithpython.com/pygame/chapter6.html. [Accessed 6 Augusti 2017].

[48] Intel, “Intel Curie discontinued,” 17 July 2017. [Online]. Available:

https://qdms.intel.com/dm/i.aspx/43A2E4E9-B349-4FF6-AB6B-237BE2F07D2C/FPCN115578-
00.pdf.

[49] Intel, “Arduino 101 discontinued,” 17 July 2017. [Online]. Available:

http://qdms.intel.com/dm/d.aspx/c84df553-5bec-4813-b415-ff18ff6017e0/PCN115582-
00.pdf.

[50] Mouser Electronics, “TinyTILE discontinued,” 24 November 2017. [Online]. Available:

https://eu.mouser.com/ProductDetail/tinyTILE/6010662/?qs=u4fy%2fsgLU9OLFHZm0LFWW
A%3d%3d.

[51] Intel, “Intel® Quark™ SE C1000 Microcontroller - Product Specifications,” 25 November 2017.
[Online]. Available: https://ark.intel.com/products/91949/Intel-Quark-SE-C1000-
Microcontroller.

[52] Movidius , “Movidius Neural Compute Stick,” 25 November 2017. [Online]. Available:
https://developer.movidius.com/.

[53] A. Munoz, “Machine Learning and Optimization,” Courant Institute of Mathematical

Sciences, New York, 2014.

74
[54] Nordic Semiconductor, “SoftDevice S130 for nRF51822,” [Online]. Available:
https://www.nordicsemi.com/eng/Products/S130-SoftDevice. [Accessed 7 June 2017].

[55] Arduino, “Arduino sketch,” [Online]. Available: https://www.arduino.cc/en/tutorial/sketch.

[Accessed 6 August 2017].

[56] K. Yeung, “Arduino 101 vs TinyTile,” 25 November 2017. [Online]. Available:

https://create.arduino.cc/projecthub/kitty-yeung/arduino101-tinytile-ble-match-making-
sunglasses-65f65e.

75
7. Appendix
Embedded device

Accuracy of the PME

Max Accuracy

Table 12: Result from the right test set, the set contained 45 gestures of each category. The
numbers of the topmost row are the gesture category.

RIGHT 0 1 2 3 4 5 6 7 8 9
RBF certain 37 45 44 44 45 44 42 45 45 45
RBF uncertain 37 45 41 44 45 44 42 45 45 45
KNN K=1 0 45 45 45 45 42 45 45 45 45
KNN K=3 0 45 45 42 45 0 39 45 45 45

Table 13: Result from the right test set, the set contained 45 gestures of each category. The
Null category was excluded when the KNN classifiers were being used.
RIGHT No-match Uncertain Wrong category Accuracy
RBF certain 5 0 9 96.89%
RBF uncertain 5 3 9 96.22%
KNN K=1 0 0 4 99.01%
KNN K=3 0 0 54 86.67%

Table 14: Result from the left test set, the set contained 45 gestures of each category. The
numbers of the topmost row are the gesture category.

LEFT 0 1 2 3 4 5 6 7 8 9
RBF certain 37 45 45 45 44 45 43 44 45 45
RBF uncertain 37 42 45 45 44 45 43 44 45 45
KNN K=1 0 45 45 45 32 45 45 45 45 43
KNN K=3 0 45 45 44 17 39 45 45 45 43

Table 15: Result from the left test set, the set contained 45 gestures of each category. The
Null category was excluded when the KNN classifiers were being used.

LEFT No-match Uncertain Wrong category Accuracy

RBF certain 4 0 8 97.33%
RBF uncertain 4 3 8 96.67%
KNN K=1 0 0 15 96.30%
KNN K=3 0 0 37 90.86%
New user

76
 Table 16: Result from when each user was used as test set. 7 sets of each gesture from the
test user was used as test sets. See Figure 31 for clarification of how the training and
testing were done.

User 1 as test set No- Wrong

1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 1 0 2 4 6 3 6 6 4 24 0 7 50.79%
RBF uncertain 1 0 2 4 6 3 0 0 4 24 16 3 31.75%
KNN K=1 5 6 6 5 7 6 6 7 4 0 0 11 82.54%
KNN K=3 5 7 4 5 7 2 1 7 4 0 0 21 66.67%
User 2 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 5 6 6 6 6 7 7 6 7 4 0 3 88.89%
RBF uncertain 4 5 0 6 6 7 7 6 7 4 10 1 76.19%
KNN K=1 4 7 7 6 6 7 7 7 7 0 0 5 92.06%
KNN K=3 5 7 7 5 6 7 7 7 7 0 0 5 92.06%
User 3 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 7 7 7 7 7 7 6 7 7 1 0 0 98.41%
RBF uncertain 4 7 6 7 7 6 6 7 7 1 5 0 90.48%
KNN K=1 7 7 7 7 7 7 7 7 7 0 0 0 100%
KNN K=3 7 7 7 7 5 7 5 7 7 0 0 4 93.65%
User 4 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 5 7 7 7 7 7 7 7 7 2 0 0 96.83%
RBF uncertain 5 7 7 7 7 7 6 7 7 2 1 0 95.24%
KNN K=1 7 7 7 7 7 7 7 7 7 0 0 0 100%
KNN K=3 7 7 7 7 7 7 6 7 7 0 0 1 100%
User 5 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 7 6 7 7 7 6 7 7 7 1 0 1 96.83%
RBF uncertain 7 6 7 7 7 6 7 6 7 1 1 1 95.24%
KNN K=1 7 6 7 5 3 7 7 7 7 0 0 7 88.89%
KNN K=3 7 7 7 4 3 7 7 7 6 0 0 8 87.30%
User 6 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 6 6 6 7 6 7 7 7 3 7 0 1 87.30%
RBF uncertain 4 4 5 7 6 7 7 7 3 7 6 0 79.37%
KNN K=1 7 7 6 7 6 7 7 7 6 0 0 3 95.24%
KNN K=3 7 7 6 7 4 7 7 7 6 0 0 5 92.06%
User 7 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 3 4 4 5 6 7 7 2 1 19 0 5 61.90%
RBF uncertain 3 4 4 5 6 7 7 2 1 19 0 5 61.90%
KNN K=1 7 7 7 5 7 7 6 6 2 0 0 9 85.71%
KNN K=3 7 7 2 5 7 7 0 7 3 0 0 18 71.43%

77
 New user with online training

Table 17: Result from when each user was used as test set. 7 sets of each gesture from the
test user was used as test sets and 3 sets of each gesture from the test user was used as
training sets. See Figure 33 for clarification of how the training and testing were done.

No- Wrong
User 1 as test set 1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 3 6 6 4 6 7 6 7 4 8 0 6 77.78%
RBF uncertain 3 6 2 4 6 7 3 4 4 8 14 2 61.90%
KNN K=1 5 7 7 5 7 6 6 7 4 0 0 9 85.71%
KNN K=3 3 7 7 5 7 6 0 7 2 0 0 19 69.84%
User 2 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 7 7 7 5 6 7 7 6 7 4 0 0 93.65%
RBF uncertain 4 7 4 5 6 7 7 6 7 4 6 0 84.13%
KNN K=1 7 7 7 6 6 7 7 7 7 0 0 2 96.83%
KNN K=3 7 7 7 5 6 7 7 7 7 0 0 3 95.24%
User 3 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 7 7 7 7 7 7 6 7 7 1 0 0 98.41%
RBF uncertain 5 7 6 7 7 6 6 7 7 1 4 0 92.06%
KNN K=1 7 7 7 7 7 7 7 7 7 0 0 0 100%
KNN K=3 7 7 7 7 5 7 5 7 7 0 0 4 93.65%
User 4 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 7 7 7 7 7 7 7 7 7 0 0 0 100%
RBF uncertain 7 7 7 7 7 7 6 7 7 0 1 0 98.41%
KNN K=1 7 7 7 7 7 7 7 7 7 0 0 0 100%
KNN K=3 7 7 7 7 7 7 6 7 7 0 0 1 98.41%
No- Wrong
User 5 as test set 1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 7 6 7 7 7 6 7 7 7 2 0 0 96.83%
RBF uncertain 7 6 7 7 7 6 7 6 7 2 1 0 95.24%
KNN K=1 7 6 7 5 3 7 7 7 7 0 0 7 88.89%
KNN K=3 7 7 7 3 2 7 7 7 7 0 0 9 85.71%
User 6 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 6 7 6 7 6 7 7 7 4 5 0 1 90.48%
RBF uncertain 4 5 5 7 6 7 7 7 4 5 6 0 82.54%
KNN K=1 7 7 6 7 6 7 7 7 6 0 0 3 95.24%
KNN K=3 7 7 6 7 6 7 7 7 6 0 0 3 95.24%
User 7 as test set No- Wrong
1 2 3 4 5 6 7 8 9 match Uncertain category Accuracy
RBF certain 3 6 6 5 6 7 7 6 7 9 0 1 84.13%
RBF uncertain 3 4 6 5 6 7 7 6 3 9 6 1 74.60%
KNN K=1 7 7 7 5 7 7 6 6 7 0 0 4 93.65%
KNN K=3 7 7 7 5 7 7 0 7 5 0 0 11 82.54%

78