MagicMouse: an Inexpensive 6-Degree-of-Freedom Mouse

Eric Woods Paul Mason Mark Billinghurst

HIT Lab NZ Lincoln University, New Zealand HIT Lab NZ
[email protected] [email protected] [email protected]

Abstract mechanical and radio frequency devices can be attached to a hand

to track it, but most only determine position, require the user to be
An inexpensive computer input device was developed that allows tethered to the machine and are expensive.
the user to operate within both 2D and 3D environments by This paper offers a cheap, intuitive, 6 DOF alternative using a
simply moving and rotating their fist. Position and rotation black cardboard square, a standard USB video camera and some
around the X, Y and Z-axes are supported, allowing full six computer vision software (ARTOOLKIT 2002).
degree of freedom input. This is achieved by having the user
wear a glove, to which is attached a square marker. Translation 2 The MagicMouse
and rotation of the hand is tracked by a camera attached to the
computer, using the ARToolKit software library. Extraction, The MagicMouse has two key software components: ARToolKit
calibration, normalisation and mapping of the data converts hand which generates a transformation matrix representing the hand;
motion into meaningful operations within 2D and 3D and functions that extract Xpos, Ypos, Zpos, Xrot, Yrot and Zrot
environments. Four input scenarios are described, showing that from the matrix, normalise them to range from –1 to 1 (using
the mapping of the position and rotation data to 2D or 3D calibration data) and map them to a variety of operations.
operations depends heavily on the desired task. ARToolKit is a free, open-source C software library that uses
computer vision techniques to calculate camera pose (position and
Keywords: Input Device, six degrees of freedom, camera, 2D, 3D orientation) relative to a black square marker. It is fast enough to
perform this calculation at 30 frames per second on a normal
1 Introduction desktop PC (800Mhz Pentium III). ARToolKit is typically used
for applications that augment video of the real world with
With the constant increase in availability and performance of 3D computer- generated objects (Figure 1a).
graphics cards, there comes a parallel need to be able to
intuitively perform operations in 3D environments such as 2.1 Extraction of Data from ARToolKit
navigation, and control. These operations benefit from the ability
to translate along and rotate about the X, Y and Z axes, defined The user wears a glove with a marker attached (Figure 1b). The
here as Xpos, Ypos and Zpos, and Xrot, Yrot and Zrot ARToolKit library analyses each video input frame and creates an
respectively. Previous research has shown that manipulation OpenGL transformation matrix that describes the position and
operations are most effective with the ability to operate in six rotation of the marker relative to the camera. Code was written
degrees of freedom (DOF) simultaneously (WARE 1990). that converts the transformation matrix into Xpos, Ypos, Zpos,
Three-dimensional input devices are available in many forms, Xrot, Yrot and Zrot. Once these 6 values are normalised they are
but generally they are either limited or very expensive, or both. very easily mapped to operations in 2D or 3D environments.
The keyboard can be used to navigate in three dimensions, but it
is not designed for this, so it lacks both intuitive and analogue 2.2 Calibration and Normalisation
control. Joysticks are the cheapest dedicated tools available, and The system requires calibration so that incoming data can be
can be analogue; however, they only offer more than 3 DOF by normalised to a value ranging from –1 to 1 for each DOF.
adding "hat" buttons (mini joysticks) and sliders, which are not The position values are determined by calculating the position
always very intuitive. A variety of ultrasound, magnetic, of the marker within the viewing volume (Figure 2). Objects
must be within this volume to be visible, so a calibration
procedure was created that determined the maximum and
minimum X and Y extents at which the marker could be seen.

Figure 1a: An ARToolKit “marker” and a 3D object placed on it.

Figure 1b: The MagicMouse glove, marker and webcam.

Figure 2: The viewpoint of the camera (the grey box), and the
viewing volume, normalised using calibration data.
Users can set the front and back Z-depth of the volume. To
ensure the extents of –1 and 1 could be reached under all
conditions, the measured maxima and minima were reduced by 3D Absolute-Mode Mouse
10%. The resultant values were used to assemble linear equations Xpos, Ypos, Zpos, Xrot, Yrot and Zrot are mapped directly to the
for the boundary planes as a function of the raw z-value. For a respective properties of a 3D cursor in a fixed volume. This
given z-value the maximum and minimum x- and y-values, can be scenario is best suited to fixed-volume operations, e.g. viewing a
calculated. Knowing these values correspond to 1.0 and –1.0, we 3D object from any position or direction. It is not well suited to
can then normalise the raw x- and y-values obtained from the navigation within an “infinite” volume like a flight simulator. It
transformation matrix. The raw z-values are normalised using the also has restricted rotation if the Xrot, Yrot and Zrot values of –1
maximum and minimum values collected from the front and back. and 1 are mapped directly to -70° and 70°. This is easily
The rotation values range from –1.0 to 1.0 about each axis overcome if values of –1 and 1 are mapped to -180° and 180°.
relative to the minimum and maximum defined extents. Hand 3D Relative-Mode Mouse (like a 3D Joystick)
rotation is limited by the flexibility of the wrist, so the rotation Similar to the 3D Absolute-Mode Mouse, except movements and
extents ranged from –70° to 70°, with 0 being the resting pose. rotations “push” the 3D cursor in that direction or rotation. This
scenario is closest to conventional joysticks (which work in
2.3 Mapping of Data to Operations Relative-Mode) except it has an amazing 6 DOF. This makes it
The normalised values of X, Y and Z position and rotation could well suited to navigation within an “infinite” volume like a flight
be mapped to operations in many different applications like robot simulator or a “run and shoot” game, where the 3D cursor
control and 2D and 3D navigation. The way the data is mapped to controls the position, rotation and subsequent point of view of the
operations is critical in ensuring an intuitive interface. aeroplane or person etc. It is completely intuitive and surprisingly
We tested simple 2D and 3D operations using both absolute accurate as the incremental nature of the scenario smooths out
and relative input. Absolute navigation moves the “cursor” in natural hand jitters. Complex navigation such as rotating to the
direct response to the position of the controller (in this case, the left while moving to the right is much easier than comparative
marker). Relative navigation constantly moves the “cursor” in a operations using mouse and keyboard combinations.
direction equivalent to the position of the marker relative to the
centre of the control space, and in a magnitude that is equivalent 3 Conclusions and Future Work
to the distance from the marker to the centre of the control space. This paper has demonstrated that for the cost of a USB camera it
A standard mouse is absolute, while a standard joystick is relative. is possible to create an intuitive, 6-degrees-of-freedom mouse.
2D navigation was implemented as mouse motion and button Considerable effort was invested in calibrating and normalising
clicks, which were synthesised using the Windows API function the values for Xpos, Ypos, Zpos, Xrot, Yrot and Zrot so that they
mouse_event. This provides control of any standard windows could be easily mapped to various operations. Use of four
application that responds to standard mouse events. mapping scenarios showed that the mapping of the position and
3D navigation was implemented as keyboard presses, which rotation data to 2D or 3D operations depends heavily on the task.
were synthesised using the Windows API function This initial version of the MagicMouse did not synthesize
keybd_event. While using keyboard events limits the Windows joystick input. Creating an implementation of this
response and real world integration of this device, it provides device that acts as a true joystick driver would maximise the
sufficient functionality to trial the device in an application. potential of this device, making it both analogue and far easier to
Where relative-mode movement was used, a threshold was integrate into third party applications.
implemented around the resting pose so that minor movement Relative-mode thresholds were implemented as a square area
within the threshold would not make the cursor move. (+-20 units in the X and Y axis, making a 40x40 square). Circular
There was also experimentation with having positive Zrot areas should be investigated in case they are more intuitive.
mapped to a Left Mouse Click and negative Zrot mapped to a As the resolution of the camera is lower than that of the
Right Mouse Click. Zrot was given a rotational threshold: beyond computer display, undersampling can affect the accuracy of the
which it synthesised a “button down” event, and below which it absolute-mode scenarios. Combined with natural hand instability,
synthesised a “button up” event. This was not very practical, as jitter of a few pixels was commonplace. While this is of little
the X and Y positions often changed as the marker was rotated, consequence for some applications, noise reduction algorithms
causing confusion (accidental drags etc). An alternative would be could be applied to this data to reduce jitter. As higher resolution
to do such actions with the spare hand or a small wireless device. cameras become commonplace, higher frequency sampling should
also be possible, making the system more responsive.
2.4 Resultant Operations and their Usefulness The ARToolKit has the ability to associate different markers
Four different scenarios were tested as described below: with different actions. With a variety of scenarios possible, it
would be attractive if the user could simply switch between them
2D Absolute-Mode Mouse by simply switching markers, e.g. the user could pick up a marker
This uses Xpos and Ypos to synthesise standard mouse with the pattern of a musical note to play music.
movements and Xrot to synthesise the scroll button. It is A variety of other limitations have been recognised with the
particularly good for painting programs where normal mouse current implementation. The necessity to keep a fist raised for
operation can be clumsy. If the camera resolution is low, the extended periods of time may cause fatigue; however, the current
cursor may have an error of up to 4 pixels. The ability to use Xrot dangers of repetitive micro movements may be combated by the
for scrolling in text documents was found to be very intuitive. ability to transform the computer interface experience into a more
2D Relative-Mode Mouse (like a 2D Joystick) physical one. This may require alternatives to the desk mounted
Either Xpos and Ypos or Xrot and Yrot are used to “push” the screen and chair be investigated. The ability to click buttons with
cursor across the screen, and Xrot is used to synthesise the scroll the hand that is wearing the marker would be quite attractive.
button. This is very similar to the 2D Absolute-Mode Mouse;
however, the Relative-Mode offers finer control over the cursor. References
Due to the incremental nature of this scenario, natural hand jitter
is minimised, producing surprisingly steady control. Use of the WARE, C. 1990. Using Hand Position for Virtual Object
mouse_event function makes analogue control possible. For Placement. Visual Computer 6 (5): 245-253.
example, a large Xpos makes the cursor move rapidly right. ARTOOLKIT 2002 HTTP://WWW.WASHINGTON.EDU/ARTOOLKIT/