Input Devices

Rynson W.H. Lau

 Table of Content
1. Basic of 3D Tracking
2. Types of 3D Trackers
 Mechanical Trackers
 Magnetic Trackers
 Ultrasonic Trackers
 Optical Trackers
 Inertial Trackers
 3D Navigation/Manipulation Devices

3. Gesture Devices
 1. Basic of 3D Tracking

 Tracker: a device to measure (or to track) the

position and/or orientation of an object in real-time.
 Typical targets for tracking in VR include the head,
hands, and limbs of the user.
 The tracked position and orientation are typically
specified relative to a reference (or world)
coordinate system.
 Degrees of Freedom (DOFs)

The degrees of freedom indicate the number of

independent motions (or measurements) to be tracked.

Examples devices of different DOFs:

 2 DOFs (e.g., 2D mouse)
 3 DOFs (e.g., accelerometers for tracking position or
gyroscopes for tracking orientation)
 6 DOFs (e.g., most mobile phones use both
accelerometers and gyroscopes)
 Coordinate Matrix
Given a 3D point, we can determine its location in
the image space as follows:

v2D = Mproj * Mview * Mworld * v3D

where v3D is the input 3D point,

Mworld is the model matrix of the 3D point,
Mview is the view matrix, and
Mproj is the projection matrix.
Mworld is the model matrix that transforms a
3D point in the model’s local coordinate
system to the world coordinate system
(or reference coordinate system).

Mview is the view matrix that transforms a

3D point in the world coordinate system to
the view coordinate system (i.e., the user’s
view).

Mproj is the projection matrix that projects a

3D point in the view coordinate system to a
point on the 2D image.
 Tracking Location and Orientation

Assuming that we want to track the user’s view

point, we may attach a tracker to the user’s head.

We can determine the view matrix (i.e., location

and orientation of the tracker relative to the
world/reference coordinate system) as follows:

Mview = R(-roll, -pitch, -yaw) * T(-eye)

To determine the location of the tracker, we use a
translation matrix to indicate the distance of the sensor
from the reference coordinate system.

This translation can be specified as T(-eye),

where eye refers to the 3D location of the eye point.
To specify the orientation of the tracker, we use a
rotation matrix to indicate it, relative to the reference
coordinate system.

This rotation matrix is:

R(-roll, -pitch, -yaw)
= Rroll(-roll) * Rpitch(-pitch) * Ryaw(-yaw).
 Tracking Accuracy

Tracker performance is typically measured using

the following parameters:
 Accuracy
 Jitter
 Drift
 Latency
 Tracker
Accuracy

1. Accuracy
 Accuracy refers to the
similarity of the object’s actual
position and the measurement
by the tracker.
 Accuracy is typically not a constant, and it degrades
as the distance from the origin of the reference
coordinate system increases.
 The distance at which the accuracy is acceptable
defines the tracker’s operating range.
 Tracker
Jitter

2. Jitter

 Jitter refers to the changes in

tracker output, even when the
tracked object is stationary.

 These changes are typically random and fluctuate

around the true value.
 The amount of jitter may be different over the tracker’s
workspace, and is influenced by environmental
conditions.
 Tracker
Drift

3. Drift

 Drift refers to the inaccuracy

that grows as time passes,
due to factors such as error
accumulation.

 It may require periodically resetting the error by a

secondary tracker.
 4. Latency

 Latency refers to the time delay

between the moment of action
and that of receiving the signal.
 In the case of a 3D tracker, latency refers to the time
between the change in object position or orientation
and the time the tracker outputs it.
 A large delay between the motion and the visual
feedback causes simulation sickness, such as
headaches, nausea, and dizziness.
Possible methods to reduce latency:
 Using faster communication lines.
 Reducing the processing delay of each step.
 Synchronizing the tracker measurement,
communication, rendering, and display loops.
– This helps reduce the update rate of the whole system.
 Tracker Update Rate
 This is the number of measurements that the
tracker reports every second, typically over 30.
 The larger this update rate is, the better the
dynamic response of the simulation.
 If a single tracking system needs to support more
trackers (i.e., sensors) to track multiple objects or
body parts, the sampling rate may need to be
proportionally reduced.
2. Types of 3D Trackers
 Mechanical Trackers
 Magnetic Trackers
 Ultrasonic Trackers
 Optical Trackers
 Inertial Trackers
 3D Navigation/Manipulation Devices
 2.1. Mechanical Trackers
 A serial or parallel kinematic structure composed
of links, interconnected using sensorized joints.
 The system knows the structure of the kinematic
chain in advance.
 The position and orientation of one part of the
mechanical tracker relative to the other are
obtained by the tracker’s joint sensors.
 By attaching one end of the tracker to the desk or
floor, the computer can derive the object’s position
at the other end.
 Gypsy Motion Capture

 It consists of a sensorized exoskeleton worn on a suit.

 User’s 15 joint positions are measured by 42 single-
turn conductive plastic precision potentiometers with an
accuracy of 0.08o.
 Wires from each sensor are routed through hollow
aluminium housings at the shoulders and the hips.
 The workspace of tracking is unlimited.
 The mechanical tracker is constructed from:
 aluminium plates
 swivel clips
 a rubber suit

 It weighs 5kg to 7kg.

 To capture motion such as walking, running, and any
other body position change relative to an external
system, it uses footstep micro-switches.
 The system can detect which foot is landing on the
ground.

 It calculates relative position of one foot to the other.

 The Gypsy 2.5 tracking suit includes a gyroscope to
track the rotation of the hip.
Advantages of Mechanical Trackers

General advantages of mechanical trackers:

 Accurate
 Small latency

Mechanical Mocaps:
 No line-of-sight restriction
 Long recording can be done – small drift
 Workspace is unlimited (however, captured motion
only relative to the base of the structure)
 Drawbacks of Mechanical Trackers
 It has limited range of motion (motion relative to the
base of the structure) due to the mechanical structure.
 As the mass of the links increase, unwanted
mechanical oscillation increases, due to inertia.
 The heavy weight of the system can lead to fatigue as
well as a diminishing sense of immersion.
 The tracker itself may interfere with the user’s motion
and reduce the user’s freedom of motion.
 Mechanical trackers are intrusive to the motion,
while non-contact motion trackers (such as magnetic
trackers, ultrasonic trackers, optical trackers, and
accelerometer/gyroscopes) do not have this problem.
 2.2. Magnetic Trackers
 They use magnetic field produced by a stationary
transmitter to determine the real-time position of a
moving receiver (the tracker).
 The transmitter consists of three antennas formed by
three mutually orthogonal coils wounded on a
ferromagnetic cube.
 Video: Electromagnetic Induction

transmitter structure receiver structure

 These coils are excited sequentially to produce three
orthogonal magnetic fields, which are either alternating
fields of 7-14kHz (for AC trackers) or pulsed fields (for
DC trackers).
 The fields reaching the receiver produce a signal that
consists of nine voltages.
 The receiver consists of three small orthogonal coils
when AC magnetic fields are used. It consists of
magnetometers when DC magnetic fields are used.
 The receiver voltages are sampled by the tracking
system, which determines the position (x, y, z) and
  
orientation (x , y, z ) of the receiver in relation to the
transmitter.
 Time multiplexing is used so that more than one
sensor can be connected to the tracking system for
motion tracking.
AC Magnetic Trackers

 Polhemus Isotrak AC magnetic tracker was the

first one (1991).
 Latency was large (~ 30ms).
 Large jitter noise (almost 1 degree rotation).
 This caused the shaking of the virtual hand and
blurred graphics in some earlier systems.
 Polhemus Fastrak used DSP architecture.
 It can measure up to 120 measurements / second
when using only one receiver.
 The sampling rate drops by half when two receivers
are used.
 The operating range is 75cm between the
transmitter and the receiver. A long ranger option
could triple the range.
Block diagram of the Fastrak system.
 Problems with AC Magnetic Trackers
 When the strength of the magnetic field changes,
eddy currents are induced in surrounding metal.
 The induced voltage, Vr , can be computed as:

where is the derivative of

the amplitude of the magnetic field.
 Eddy currents are produced when a conductor is exposed
to a changing magnetic field.
 These currents are induced in the conductor to oppose
the change in magnetic flux that generated them.
 These currents generate new magnetic fields that oppose
the change of the original magnetic field.
 The tracker receiver gets a distorted magnetic field, and
accuracy is degraded.

• B: magnetic field
• I: electric current
• d: distance from the center
• o: the magnetic constant
 DC Magnetic Trackers

 To alleviate the eddy current problem, the DC

magnetic field was proposed to replace the AC one.
 Short time delay between the excitation of the
transmitter allows the eddy currents to disappear.
 An embedded processor controls the amplitude of
the transmitter’s DC magnetic pulses.
 The multiplexer allows the processor to cycle the
current sources; three orthogonal magnetic
sources are produced one at a time.
 There are four time periods to cycle, which are
defined by T0-T1, T1-T2, T2-T3, and T3-T4.
 T0 – T1: all source currents are off.
 T1 – T2: the X field is on.
 T2 – T3: the Y field is on.
 T3 – T4: the Z field is on.
 The transmitter fields induce voltages in the three
orthogonal antennas in the receiver.
 During T0-T1, the voltage due to the earth’s DC
magnetic field is obtained.
 The voltage due to the earth’s magnetic field is
subtracted from the measurements collected from
other periods.
 The processor then determines the position /
orientation of the sensor, using the calibration
algorithm.
 Ascension Flock of Birds was first available in
1998.
 It uses distributed computing architecture to reduce
computation time and maintain high update rates.
 Each receiver is controlled by one processor. As
receivers’ data are processed in parallel, update
rates are not degraded with added receivers.
 It maintains an update rate of 144 measurements
per second, even when 30 receivers are used.
 Ascension MotionStar is a wireless system and
was first available in 2001.
 An extended range controller drives two extended
range transmitters (ERT1 and ERT2) in a time-
sequential way.
 First, ERT1 sends a sequence of DC magnetic
pulses, then ERT2, and then ERT1, and so on.
 The two transmitters can be either facing each
other or placed side by side.
 Total of 10 to 20 DC magnetic receivers are put on
the user’s body.
 All the sensors are wired to a backpack unit, which
weighs 1.7kg.
 The backpack contains the battery, processing
electronics, and a wireless modem for data
communication.
 The base station computes the position/orientation
of the receivers using the magnetic data.
 The base station knows which readings are from
the ERT1 field and which are from ERT2 field.
 More weighting is put on the data by the closer
transmitter to improve accuracy.
 Several backpack-base station pairs can be used
to track up to five users (each 100 data sets/sec).
Problems with DC Magnetic Trackers
Ambient Error:
 The intensity of the magnetic field falls with the
cube of the distance from the transmitter.
 The position measurement error due to ambient
noise is given by:
Error = K * d4
where K is a constant, and d is the distance
between the transmitter and the receiver.
Metal Interference:
 Eddy currents can be produced by a time-variant
magnetic field.
 DC trackers are unaffected by non-ferromagnetic
metals such as brass, aluminium, and stainless steel.
 AC and DC trackers are both affected by the
ferromagnetic metals such as mild steel and ferrite,
which will become magnets and in turn affect the
magnetic field.
 The high permeability of ferromagnetic metals
allows the induced magnetic fields to exist for
AC/DC trackers.
 Both are affected by copper; its high conductivity
allows eddy currents to exist longer than the time
delay used in DC currents.
Advantages/Limitations of Magnetic Trackers

+ Accurate, but not as accurate as mechanical trackers

+ The sensor is light weight
+ No line-of-sight restriction

- Affected by metallic objects/surfaces

 2.3. Ultrasonic Trackers
 It is a non-contact position/orientation tracking
device that uses an ultrasonic signal produced by
a stationary transmitter to determine the real-time
position of a moving receiver (or tracker).
 The transmitter is a set of three ultrasonic speakers
mounted about 30cm apart from each other on a rigid
fixed triangle frame.
 The receiver is a set of three ultrasonic microphones
mounted on a smaller rigid triangle frame.
 Due to its simplicity, ultrasound trackers represent a
cheaper alternative to magnetic ones.
 For a given temperature, the speed of sound is
known and can be used to measure the distance
based on the time of flight.
 Each speaker is activated in cycle and the three
distances from it to the three microphones in the
receiver are calculated.
 A total of nine distances is measured in order to
determine the position and orientation of the plane
that contains the three microphones.
 The update rate of ultrasonic trackers is about 50
data sets/sec (less than half of magnetic trackers).
 The reason for the low update rates is the need to
wait 5-100ms to allow echoes from previous
measurement to die out before a new start.
 The operating range of ultrasonic trackers depends
on the ultrasound attenuation due to air absorption –
typically 1.52m from the transmitter.
 Direct line of sight is required between the
transmitter and the receiver.
 Background noise must be eliminated.
 The motion volume can be increased by spatially
multiplexing several transmitters with a single
receiver.
 The tracking volumes must overlap for seamless
tracking.
 To avoid mutual interferences,
only one transmitter can be
turned on at a time.
Advantages/Limitations of Ultrasonic Trackers
+ No interference with metal
+ Small and light weight
+ Relatively inexpensive

- Line-of-sight restriction
- Accuracy depends on sound velocity (which in turn
depends on temperature, pressure, humidity, etc.)
- Susceptible to noise, e.g., wall properties.
 2.4. Optical Trackers
 An optical tracker:
• is a non-contact position measurement device
• is based on optical sensing
• calculates real-time position/orientation of an object

 Requires direct line of sight

 High update rates
 Small latency
 1. Outside Looking In

 In this technique, we fix the

sensor, e.g., camera, at a
fixed location. We put some
markers or light beacons on
the user whose motion is to
be tracked.
An Example – Optical Motion Capture Systems
 A popular motion capture system is MotionAnalysis
(www.motionanalysis.com/).

 Infrared lights are placed around the work space.

 Markers (infrared reflectors) are attached to
various parts of an object or a user to be tracked.
 A number of infrared cameras (say 8) are used to
capture the reflection from the markers.
 This will produce 8 images (from the 8 infrared
cameras.)
 Recovering the 3D position of a 2D marker
• Given the 8 captured images, we first identify the same
marker across these images.
• Based on the marker’s 2D locations in different images,
the 3D location of the marker can be computed.
• Note that it is often not possible to find the same marker
in all 8 images, due to occlusion/ambiguity. Luckily, we
only need to find the marker in 3 or 4 images.
• A demo of matching image points:
https://www.youtube.com/watch?v=VDTJmKE0EyA
 Recovering 3D body posture:
• Given a marker, if we know its 2D position in each of
the captured images, we can calculate its 3D position.

• Using three markers on each body segment, it is

possible to compute the segment’s position and
orientation.

• By doing this for all the body segments, it is possible to

compute the posture of the whole body.
 With the captured 3D motion data, a virtual object can
be programmed to follow the movement of a real object.
 For example, to model a virtual human playing baseball:
• Markers are attached to different body parts of a real baseball
player to capture his movement. <video>
• The captured motion information <video> can then be used to
guide the movement of the virtual baseball player. <video>

 By exaggerating the captured information, it is even

possible to produce a more interesting effects. <video>
Some more motion capture animations: <video>
 Tracking sensitivity is degraded as:
• distances among the markers decrease – reduced
computation accuracy.
• distance between the user and the camera
increases – increased computation error.
• distance between adjacent cameras increases –
increased discrepancy among camera views due to
occlusion. (Reducing the distance between adjacent
cameras, however, will require more cameras and
higher processing costs.)
 Optical Motion Capture System Pipeline

There are five steps in the motion capture process:

1. Calibration – We need to know where each camera is
located in the reference coordinate system, by
capturing the locations of some special markers with
known positions. This can be done using a checkboard
or a calibration frame.
2. Motion Capture – This is the actual motion capturing
step, where the actor(s) performs the action to be
captured.
3. 3D Position Reconstruction – Occlusion can occur during
the motion capture. To reconstruct the 3D position of each
marker in the reference coordinate system, it must be seen
by at least two cameras.
4. Fitting the Skeleton – This is to associate the markers to a
predefined skeleton, if available. This can help improve the
accuracy.
5. Post Processing – After the motion has been captured, we
may need to correct / refine / improve the motion, e.g.,
removing noise, changing the speed of a certain part of the
motion, and motion editing.
 Advantages/limitations of Optical Trackers

+ High precision: as this is the standard method for

motion capturing, many techniques have been
proposed to improve the precision:
• Camera calibration, tracking algorithms, ….

- Occlusion problem: body parts may obscure each

other. Obscured body parts may not be recovered.
• If the 3D model of the object being tracked is available,
it can significantly reduce the error.
 2. Inside Looking Out
 In this technique, we attach the
sensor, e.g., the camera, to the
object or user being tracked.
We put markers on the walls.
 Its sensitivity is maximized for
changes in orientation.
 Theoretically, the work space
can be scaled to infinity.
 An Example – HiBall Tracker from 3rd Tech
 It consists of two HiBall units, LED beacon arrays,
ceiling-HiBall interface, and a host computer.
 The HiBall sensing unit incorporates IR-filtering lenses,
lateral-effect photodiodes (LEPDs), and miniaturized
electronic circuits.
 There are six narrow-view (6 degrees) lenses arranged
in the six sectors of a hemisphere.
 The LEPD determines the x-y coordinates of a light spot
that is the image of a pulsed LED on the beacon array.
HiBall 3000 wide area tracker

HiBall Optical Sensor HiBall Optical Sensor interior

6 photodiodes

6 optical lenses

signal conditioning
electronics

(courtesy of 3rdTech Inc.)

HiBall 3000 tracker
on an HMD

Lateral effect
photo diodes
Beacon array modules
(6 strips with 8 LED/strip)
 The beacon arrays consist of 48 LEDs arranged
in six parallel strips over an 8-ft2 surface.
 A number of beacon arrays can be added to
increase the tracking envelope of the HiBall from
8x8 ft to 40x40 ft.
 The update rate is 2000 data sets / sec.
 Advantages: high precision and large tracking
space.
 2.5. Inertial Sensors

There are three types of inertial sensors:

 Gyroscope measures angular velocity in degrees
per second.
 Accelerometer measures linear acceleration in
m/s2.
 Magnetometer measures magnetic field strength in
uT (micro Tesla).
 Gyroscopes
 The rate of change in the sensor
orientation (or angular velocity) can
be measured by the Coriolis-typed
gyroscopes.
 Example: Fujitsu Gyro Sensor
 The sensor is shaped like a tuning fork, and vibrates
continuously.
 As the sensor rotates and the Coriolis force also vibrates
the two elements in the direction perpendicular to the
continuous vibration.
 The vibration due to the Coriolis force can be converted
into a voltage proportional to the angular velocity of
rotation for measurement.
 Three sets of such gyroscope are combined and
mounted on mutually orthogonal axes.
 The 3D orientation of the sensor can then be
calculated by integration of the angular velocities
obtained from the three gyroscopes.
 The measurements from the gyroscopes are affected
by temperature, and drift over time.
 They are accurate in the short term, but suffer from
drift in the longer term.
 Accelerometers
 Accelerometers measure the translational velocity or
acceleration.
 Three accelerometers are mounted coaxially with the
three gyroscopes.
 The 3D position of the sensor can be calculated by
double integration of the accelerations, after
subtracting the effects by gravity and rotation.
 Accelerometers are accurate in the long term. They
do not suffer from drift.
 However, they are unreliable in the short term due to
noise.
 As a result, their measurements can be
complementary to the measurements from the
gyroscopes.
 Magnetometers

 It measures earth’s magnetic field in uT.

 Again, there are three orthogonal axes.

 The actual direction that the magnetometers point to

may change depending on its latitude and longitude.

 Its accuracy is affected by metallic objects nearby.

 They are less general, and mainly used in special

trackers.
 Inertial Measurement Unit (IMU)

 InvenSense MPU-9250 is a 9 DoF IMU.

 It has 3-axis gyroscopes, 3-axis accelerometer, and

3-axis magnetometer all in one chip.

 It includes 9 16-bit ADCs for digitizing the 9DoF

data.
The coordinate systems used in MPU-9250.
 Advantages of inertial trackers:
+ Unlimited range of tracking
+ No line-of-sight constraints
+ Low jitter
 Drawbacks of inertial trackers:
– Rapidly accumulating errors or drift.
– Gyroscope bias leads to an orientation error that increases
proportionally with time due to integration.
– Accelerometer bias induces an error that increases with
the square of time.
– Commercial devices : 40mm error in 2s.
– Expensive ones: 40mm error in 200s.
– Solution: periodically reset the error using another type of
tracker.
 A good summary of the IMU:
https://www.youtube.com/watch?v=eqZgxR6eRjo
2.6. 3D Navigation/Manipulation Devices
 Navigation: interactive change of view in the
virtual environment for exploration.
 Manipulation: selecting an object for some form
of modification.
 Logitech Magellan
 Sensorized cylinder that measures three forces and
three torques applied by the user’s hand on the cylinder.
 Six LEDs are placed at the center of the inside of the
cylinder.
 Six photo sensors are placed on the shell of the inside of
the cylinder.
 When the user applies forces or torques on the moving
shell, the shape of the cylinder is distorted, which would
affect the output of the photo sensors.
 Hence, the photo sensor output is used to measure the
forces and torques applied by the user.
 Disadvantages:
– Trackballs suffer from sensor coupling: although the
user may wish to translate the object, it often rotates
at the same time
– Not so user-friendly
 MicroScribe 3D: a 3D probe
 Consists of 6 rotary joints; thus
the probe has 6 DOFs, allowing
simultaneous positioning and
orienting at the tip.
 A counterbalance is placed
close to the base to minimize
the user's fatigue.
 Can be used to digitize objects.
 The tip position is calculated using direct kinematics,
based on sensor values and the lengths of the links.
 A binary switch on the foot pedal is used to select or
deselect virtual objects, navigate, or mark a point on the
real object surface.
 Possible to have over 1,000 samples/sec.
 The kinematic arm accumulates errors from the base
toward the tip.
 Calibration is needed to minimize errors.
 There are similar devices available.
 3. Gesture Devices
 They measure the real-time positions of the user's
fingers.
 They allow gesture-based interactions with the
virtual environment.
 Pinch Gloves
 Incorporate electrodes in the form of conductive fibre
patches:
 at the fingertips
 at the back of fingers
 in the palm
 Gestures are detected as establishing and breaking of
electrical contacts between the fingers and palms.
 Only detect whether contacts are made or not.
 The 5DT Data Gloves
 Able to measure finger
joint angles.
 One flex sensor per finger.
 A tilt sensor to measure
wrist orientation.
 Each finger has a fibre-optic sensor loop routed
through attachments.
 An LED is placed at one end of the fibre, while a
photo-transistor is place at the other end to capture the
light emitted from the LED.
 The fibre walls are treated to change their refractive
index, such that light will escape upon finger flexion.
 Finger bending can be measured indirectly based on
the intensity of the returned light.
 The glove uses an 8-bit A/D converter.
 It has a resolution of 256 intermediate positions between
a flat and a fisted hand.
 The sensor data needs to be normalized first in order to
account for the effect of varying hand sizes.
 The normalization is carried out as follows:

rawval  rawmin
out   256
rawmax  rawmin
where rawmax – rawmin is the dynamic range for a given
flex sensor and a given user.
 Calibration is done by having the user to flex its hand
a few times, while the system updates the value of the
glove’s dynamic range.
 It generates a look-up table that maps the sensor
outputs to postures.
 Thus, real-time detection of the finger joint angles can
be achieved.
 Didjigloves
 Ten capacitive bend sensors are used to measure
the positions of the user’s fingers.
 Each capacitive sensor consists of two layers of
conductive polymer separated by a dielectric.
 Each conductive layer is arranged in a comb-like
fashion such that the overlapping surface of the
electrode is directly proportional to the amount of
sensor bending.
 Since capacitance is directly proportional to the
overlapping surface of the electrode, the bending
angle can be measured electrically.
 CyberGloves
 Incorporate thin electrical strain gauges placed on an
elastic nylon glove.
 The glove sensors are either rectangular (for the flexion
angles) or U-shaped (for adduction-abduction angles).
 There are either 18 or 22 sensors in a glove. They are
used to measure finger flexion, abduction, and wrist
yaw and pitch.
 Joint angles are measured indirectly by a change of
resistance in a pair of strain gauges.
 During finger motion one of the strain gauges is
under compression while the other is under tension.
 Their change in resistance produces a change in
voltage between them.
 These differential voltages are then amplified, and
digitized by an A/D converter.
 The CyberGlove was very popular for high-
performance hand measurements.

 This is due to its large number of sensors, its good

programming support, and the available
extensions into more complex haptic gloves.

 <video>