VR Input Devices

SWE405 Virtual Reality Technology and Application

Prepared by Yap Shook Chin 1
VR Interface
◦ A virtual world is mapped into a representation
that is then rendered and fed to the user via
display devices.
◦ The rendering process takes into account the
movements of the user to create the immersive
point of view.
◦ The user can affect the virtual world through
inputs programmed to interface with certain
aspects of the world.
◦ For augmented reality systems, rendering of the
virtual world is mixed with a view of the real
world.

Prepared by Yap Shook Chin 2

VR Interface
Rendering
◦ A virtual world is mapped into a representation
that is then rendered and fed to the user via
display devices.
◦ The rendering process takes into account the
movements of the user to create the immersive Display
point of view. Devices
◦ The user can affect the virtual world through
inputs programmed to interface with certain
aspects of the world.
◦ For augmented reality systems, rendering of the
virtual world is mixed with a view of the real
world.

Prepared by Yap Shook Chin 3

Interface to the Virtual World - Input
◦ The VR system needs input to
◦ monitor the user in order to provide the user information necessary to make a
display immersive.
◦ enable user interact with the virtual world.
◦ to gather nonparticipant information about the world (real or virtual), which the
user might want to investigate or interact with
◦ Input devices are just the physical tools (Hardware interface) that are used to
implement various interaction techniques.

Prepared by Yap Shook Chin 4

Types of Input
◦ User monitoring
◦ the real-time monitoring of the participant's actions in a VR experience.
◦ World monitoring
◦ Supplies real-world input and information about change in the virtual world,
brought about through time or user manipulation.

Prepared by Yap Shook Chin 5

User Monitoring
◦ For the user to tell the system what they want and to track at least some part of their
body.
◦ Includes the continuous tracking of both user movements and user-initiated actions,
such as pressing a button or issuing a voice command to the system
◦ Active ways: the user inputs information into the system
◦ Include the use of spoken commands, physical controls like wands, joysticks, steering
wheels, dashboards, and keyboards, props, and platforms.
◦ Passive ways: information is supplied to the computer to tell the computer how and
where the participant is moving and where they are looking.
◦ Include tracking of the body (hands, eyes, and feet) and position tracking that tells the
computer the participant’s location and orientation.

Prepared by Yap Shook Chin 6

World Monitoring
◦ Information can be gathered and brought into the experience from sources not directly related to
the participant.
◦ The world being monitored might be the real world or a persistent virtual world (a virtual world
that evolves over time, with or without participants).
◦ Sometimes monitoring is necessary because a server and multiple user interfaces are being used,
and information needs to go between them.
◦ A virtual world server may contain the position of other active participants and/or the current
state of the virtual world.
◦ Transducers are used to retrieve real-world data, e.g. microphones, weather stations, LIDAR
(light detection and ranging), video cameras, and electromagnetic position sensors.
◦ In augmented reality systems: a video feed of the physical world

Prepared by Yap Shook Chin 7

Three Degrees of Freedom (DoF)
◦ This is the simplest form of user tracking
◦ Allow a device to measure how it is moving in three types of directional rotation
◦ Relies entirely on the inbuilt sensors (accelerometers, gyroscopes and magnetometers) that
phones use to measure movement.

Prepared by Yap Shook Chin 8

Six Degrees of Freedom (DoF)
◦ Incorporates the three rotational measurements (rolling, pitching and yawing) and adds three
further directional movements that allow a person to physically move around in x, y, z axes
◦ Crucial for whole-room VR experiences and gives the user a lot more freedom to explore
locations, inspect details and to dodge virtual bullets.
◦ Both the headset and the controllers worn by the user are tracked.

Prepared by Yap Shook Chin 9

User Monitoring

10
Position Tracking
◦ A position sensor is a device that reports its location and/or orientation to the
computer.
◦ Often position sensors are used to track the participant's head and one of the
participant's hands.
◦ Most important tracking device of any VR system.
◦ Limitations arise from the technology used to determine the relationship from
some fixed origin and the sensor.

Prepared by Yap Shook Chin 11

Position Tracking
◦ Factors to consider
◦ accuracy/precision and speed of the reported sensor position
◦ interfering media (e.g., metals, opaque objects)
◦ encumbrance (wires, mechanical linkages)
◦ Noise and low accuracy in the position sensor caused time lag, decrease the
realism or immersiveness of the experience and can lead to nausea in some
participants.

Prepared by Yap Shook Chin 12

Types of Tracking
◦ Electromagnetic
◦ Mechanical
◦ Optical
◦ Videometric
◦ Ultrasonic
◦ Inertial
◦ Neural

Prepared by Yap Shook Chin 13

Electromagnetic Tracking
◦ Transmitter – generate magnetic field, then
generate current in the coils of receiver
◦ Fixed at known position
◦ Receiver – worn by user
◦ signal in each coil in the receiver is measured
to determine its position relative to the
transmitter
◦ The signal weakens as the receiver move away
from transmitter
◦ The strength of the signal changes based on
the relative orientation between transmitter
and receiver coils.

Prepared by Yap Shook Chin 14

Electromagnetic Tracking
◦ Electromagnetic tracking (EMT) is used in several medical fields to determine
the position and orientation of dedicated sensors, e.g., attached to surgical
tools.

Prepared by Yap Shook Chin 15

Electromagnetic Tracking
◦ Limitations:
◦ metal in the environment can cause magnetic interference.
◦ Short range of the generated magnetic field. The receivers will operate with reasonable
accuracy only within 3-8 feet of the transmitter, depending on the specific model.
◦ Advantage:
◦ electromagnetic systems have no line of sight restriction, allows the users to move about in
a space that might have multiple visual or sonic obstacles
◦ wireless systems have become available, reducing encumbrances on the participant.

Prepared by Yap Shook Chin 16

Mechanical Tracking
◦ Accomplished through mechanical means
◦ In addition to being a tracking device, this type of system is generally used for
the head as a visual display or for the hands as a haptic I/0 device
◦ Motors can be connected to create a force display (a sense of touch or haptic
display that provides pressure and resistance effects)

Prepared by Yap Shook Chin 17

Mechanical Tracking
◦ For example, BOOM by Fakespace Systems,
Inc
◦ An articulated armlike boom is used to
measure the head position.
◦ The boom follows their movements within a
limited range, each elbow joint and
connecting link of the boom is measured to
help calculate the user's position.
◦ The joints in the boom arm are flexible and
allow user movement in the area within reach
of the boom arm.
Prepared by Yap Shook Chin 18
Mechanical Tracking
◦ Advantage:
◦ The rotational and linear measurements of the mechanical linkages can be made quickly,
accurately, and precisely.
◦ Disadvantage
◦ the physical linkages restrict the user to a fixed location in the world.
◦ There are some residual effects, because the inertia of a heavy display can take some effort
to move fluidly--especially when a large mass (like a pair of CRT displays) is attached to the
linkages.
◦ can't be used for both head and hands; thus, a second tracking system is required if both
head- and hand-tracking are important.

Prepared by Yap Shook Chin 19

Optical Tracking
◦ Make use of visual information to
track the user.
◦ Make use of a video camera that acts
as an electronic eye to "watch" the
tracked object or person.
◦ The video camera is in a fixed location.
Computer vision techniques are then
used to determine the object's
position based on what the camera
"sees."

Prepared by Yap Shook Chin 20

Optical Tracking
◦ When using a single sensing device, the position can be reported in only two
dimensions, no depth information (2D optical tracking)
◦ Using multiple sensors allow the system to provide three-dimensional position
information, enables full 6-DoF.
◦ Light-sensing devices other than video cameras can be used, e.g. laser and
LED

Prepared by Yap Shook Chin 21

Optical Tracking
◦ Advantages:
◦ best for accuracy right now
◦ Latency is better, making it less likely you'll feel nauseous.
◦ Limitation:
◦ the line of sight between the tracked person or object and the camera must always
be clear.
◦ keeping the user within the sight of the camera limits the participant's range of
movement, user is only tracked in its field of view.

Prepared by Yap Shook Chin 22

Videometric (Optical) Tracking
◦ The camera is attached to the object being tracked (HMD) and watches the
surroundings
◦ The VR system analyzes the incoming images of the surrounding space to
locate landmarks and derive the camera's relative position to them.
◦ The location of the landmarks in the space must be known
◦ The computer vision algorithm track multiple points and distinguish them
from surrounding objects, these fiducial landmarks act as known reference
points in the world.

Prepared by Yap Shook Chin 23

Videometric (Optical) Tracking
◦ Infrared light sources are placed
throughout the ceiling to be sensed by
multiple infrared-sensitive cameras
mounted on the tracked object (HMD)
◦ An alternate approach for creating
landmarks is placing easily distinguishable
patterns in places that are likely to fall
within the camera's view.

Prepared by Yap Shook Chin 24

Ultrasonic Tracking
◦ Uses high-pitch sounds emitted at timed intervals to determine the distance
between the transmitter (a speaker) and the receiver (a microphone).
◦ Three transmitters combined with three receivers provide enough data for the
system to triangulate the full 6-DOF position of an object.
◦ Example: Logitech ultrasonic tracker

Prepared by Yap Shook Chin 25

Ultrasonic Tracking
◦ Advantages
◦ relies on common technology such as speakers, microphones, and a small computer, it is
inexpensive

◦ Disadvantages
◦ Tracking performance can be degraded when operated in a noisy environment.
◦ The sounds must have an unobstructed line between the speakers and the microphones to
accurately determine the time (and therefore distance) that sound travels between the
two.
◦ These transmitters and receivers must be separated by a certain minimum distance.

Prepared by Yap Shook Chin 26

Inertial Tracking
◦ Uses electromechanical instruments to detect the relative motion of sensors
by measuring change in gyroscopic forces, acceleration, and inclination.
◦ Includes accelerometer, inclinometer, gyroscopes
◦ The inertial tracking device is a small sensor that is attached to the object
being tracked.
◦ Inertial tracking systems can be combined with other tracking systems to
provide the best of both methods.

Prepared by Yap Shook Chin 27

Inertial Tracking
◦ Advantages
◦ They are self-contained units that require no complementary components fixed to a known
location, so there is no range limitation.
◦ They work relatively quickly compared with many of the other tracking methods and,
therefore, introduce little lag into the system.
◦ Inexpensive and have been incorporated directly into some inexpensive head-based
displays
◦ Limitation:
◦ In practical applications of virtual reality, these tracking systems are typically limited to
orientation-only measurement.

Prepared by Yap Shook Chin 28

Neural (Muscular) Tracking
◦ A method of sensing individual body-part movement, relative to some other part of
the body.
◦ It is not appropriate for tracking the location of the user in the venue, but it can be
used to track movement of fingers or other extremities.
◦ This form of VR system tracking has
not been deeply explored.

Prepared by Yap Shook Chin 29

Body Tracking
◦ To sense position and actions of the participants.
◦ Tracking the head
◦ Tracking the hand and fingers
◦ Tracking the eyes
◦ Tracking the torso
◦ Tracking the feet
◦ Tracking other body parts
◦ Indirect tracking

Prepared by Yap Shook Chin 30

Tracking the Head
◦ The VR headset has a motion sensor (IMU) inside that detects spatial acceleration and rotation
rates on all three axes, providing what's called the six degrees of freedom (6DOF).
◦ This is the same technology that is commonly found in mobile phones and some console game
controllers, but perhaps with higher sensitivity and accuracy.
◦ With the IMU integrated into your headset, when you move your head, the current viewpoint
is calculated and used when the next frame's image is drawn. This is referred to as motion
detection.

Prepared by Yap Shook Chin 31

Mobile Phone Motion Detection
◦ The previous generation of mobile motion sensors was good enough for us to play mobile
games on a phone, but for VR, it's not accurate enough.
◦ These inaccuracies (rounding errors) accumulate over time, as the sensor is sampled thousands
of times per second, and you may eventually lose track of where they were in the real world.
◦ This drift was a major shortfall of the older, phone-based Google Cardboard VR.
◦ It could sense your head’s motion, but it lost track of your head's orientation.
◦ The current generation of phones, such as Google Pixel and Samsung Galaxy, which conform to
the Daydream specifications, have upgraded sensors.

Prepared by Yap Shook Chin 32

High-end HMD Motion Detection
◦ High-end HMDs account for drift with a separate positional tracking
mechanism:
◦ Inside out
◦ Outside in
◦ Spatial mapping

Prepared by Yap Shook Chin 33

Inside-Out
◦ The original Oculus Rift CV1
◦ The original Oculus Rift CV1 did this with inside-out positional tracking, where an
array of (invisible) infrared LEDs on the HMD were read by an external optical sensor
(infrared camera) to determine your position. You need to remain within the view of
the camera for the head tracking to work.

Prepared by Yap Shook Chin 34

Outside-In
◦ The Steam VR VIVE Lighthouse technology uses outside-in positional tracking, where two or
more dumb laser emitters are placed in the room (much like the lasers in a barcode reader at
the grocery checkout), and an optical sensor on the headset reads the rays to determine your
position.

Prepared by Yap Shook Chin 35

Spatial Mapping
◦ The Oculus Rift-S, Vive Cosmos, and Windows
MR headsets use no external sensors or
projectors.
◦ Rather, the headset itself contains all the
integrated cameras and sensors needed to
perform spatial mapping of the local
environment around you in order to locate
and track your position in real-world 3D space.

Prepared by Yap Shook Chin 36

Tracking the Hand
◦ To give the user a method of interacting with the world
◦ In multiparticipant spaces, hand gestures can provide communication between
participants.
◦ A hand can be tracked by attaching a tracker
unit near the wrist or through the use of a
tracked handheld device.
◦ A glove input device is used to track the
positions of the user's fingers and other
flexions of the hand.

Prepared by Yap Shook Chin 37

Tracking the Hand
◦ Tracked hand-held controllers are typically 6 DoF devices (known as “wands”
in the VR research community
◦ The Sixense STEM (left) and Oculus Touch (right) tracked hand-held controllers

Prepared by Yap Shook Chin 38

Tracking the Eyes
◦ To monitor the direction of the gaze to allocate computer resources.
◦ Eye tracking could be used as part of the interface with the world itself, objects might be
selected or moved based on the movement of the eyes.
◦ Left: Eyegaze system, a non-intrusive approach which uses an infra-red source that reflects off
of the pupil, developed by LC Technologies.
◦ Right: iView, a head-mounted eye tracking device developed by SensoMotoric Instruments.

Prepared by Yap Shook Chin 39

Full Body Tracking
◦ Typically done with a motion capture suit, similar to what is used in the film industry.
◦ Different suits utilize different technologies such as electromagnetic sensors, retro-
reflective markers, and inertial sensors.
◦ Depth cameras such as Microsoft Kinect can theoretically track the entire body, but it
can be difficult to capture the entire body unless multiple cameras are used

Prepared by Yap Shook Chin 40

Other Body Parts
◦ Other body parts can be tracked and used to control
various aspects of a virtual world.
◦ These items include body functions, such as
temperature, perspiration, heart rate, respiration
rate, emotional state, and brain waves.
◦ The user wears a body suit, which provides a means
for the computer system to monitor the wearer’s
physiological attributes

Prepared by Yap Shook Chin 41

Physical Input Devices
◦ Keyboard, joystick, mouse, tablets
◦ Wand: hand-held controller
◦ The Interaction Slippers are a custom made device which allows users to perform toe and
heel tapping for invoking commands
◦ The Pinch Glove system developed by Fakespace (Left picture)
◦ A puppeteer uses a waldo hand-input device to control the computer graphics representation
of Gonzo (right picture)

Prepared by Yap Shook Chin 42

Platform
◦ Platform: the part of the VR system where the participant is situated
◦ For Locomotion tracking
◦ Can provide sense-of-balance (vestibular) feedback by mounting them on a motion
base. A motion base is an active floor or cockpit that is moved by hydraulics.
◦ E.g. Slidemills

Virtuix Omni
Prepared by Yap Shook Chin 43
Platform

Driving simulator: Caterpillar Virtual

VR Kiosk
Prototyping System in a CAVE VR display

Prepared by Yap Shook Chin 44

Platform
◦ Treadmills

Aperium Pod: single-direction treadmill Infinadeck: omnidirectional treadmill

Prepared by Yap Shook Chin 45

Platform
◦ Omnideck: gold standard of VR
treadmills
◦ Over 3 meters in diameter and
almost 1000kg in weight. This is the
currently the only device that can
truly simulate walking
◦ The Omnideck is also the most
mature product on this list, having
been on the market since 2011.

Prepared by Yap Shook Chin 46

Other Inputs
◦ Painting table
◦ Used in CavePainting application
◦ Users can dip the paint brush prop into the colored
cups to change brush strokes
◦ Speech
◦ Provides complement to other devices
◦ A natural way to combine different modes of input

Prepared by Yap Shook Chin 47