Hacking The Nintendo Wii Remote
Hacking The Nintendo Wii Remote
Hacking the
Nintendo Wii Remote
The global hacking community has collectively reverse-engineered a
significant portion of one of the world’s most sophisticated and common
input devices. And they’re putting it to uses its designers never intended.
I
n November 2006, Nintendo released its torials to demonstrate interaction techniques
fifth home videogame console, the Nin- supported or enabled by the Wii remote. In just
tendo Wii. The company’s previous game a few weeks, these tutorials have received over
console, the Gamecube, hadn’t fared six million unique views and generated over
well in terms of market share against 700,000 software downloads. In this article,
the much higher-powered alternatives released I will talk about the Wii remote’s technology,
by its competitors, Microsoft and Sony. At first cover what’s involved in developing custom ap-
the Wii also seemed significantly underpowered plications, describe intended and unintended
relative to its competitors. However, one year interaction techniques, and outline additional
later it became the market leader of its console uses of the device.
generation, selling over 20 million units world-
wide.1 This success is largely attributable to the Inside the Wii remote
innovative interactive technology and game-play Although the Wii remote’s official specifications
capabilities introduced by the console’s game are unpublished, the global hacking community
controller, the Wii remote, shown in Figure 1. has collectively reverse-engineered a significant
The Nintendo Wii remote, or Wiimote, is a portion of the technical information regarding
handheld device resembling a television remote, its internal workings. Much of this work has
but in addition to buttons, it been collected in online wikis at http://wiili.org
Johnny Chung Lee contains a 3-axis accelerom- and http://wiibrew.org. The body of knowledge
Carnegie Mellon University eter, a high-resolution high- at these sites represents contributions from nu-
speed IR camera, a speaker, a merous individuals and constitutes the source
vibration motor, and wireless for most of the information presented in this
Bluetooth connectivity. This section.
technology makes the Wii remote one of the Because many low-level details are available
most sophisticated PC-compatible input de- online and, furthermore, are likely to be refined
vices available today; together with the game and updated as more information is uncovered,
console’s market success, it’s also one of most the following descriptions of each major Wii re-
common. At a suggested retail price of US$40, mote component represent only higher-level de-
the Wii remote is an impressively cost-effective tails relevant to building custom applications.
and capable platform for exploring interaction
research. Software applications developed for Infrared camera tracker
it have the additional advantage of being read- In the tip of each Wii remote is an IR camera sen-
ily usable by millions of individuals around the sor manufactured by PixArt Imaging, shown in
world who already own the hardware. Figure 2. The camera chip features an integrated
I’ve recently begun using Internet video tu- multiobject tracking (MOT) engine, which
40 Authorized
P ER E computing
VA SI Vlicensed
use limited www.computer.org/pervasive
to: Politecnico di Milano. Downloaded on September 23,2023 at 18:39:40 UTC from IEEE Xplore. Restrictions apply.
developing custom applications. The bar” is a misnomer because the device text of bowling, boxing, or playing ten-
remote’s compatibility with the Blue- doesn’t contain sensors; rather, it con- nis, baseball, or golf. The game appears
tooth HID specification manifests on tains two groups of infrared LEDs. The to register subtle variations in swing dy-
the host computer as a joystick. Wii remote’s IR camera sees the two namics and thus affect the simulation.
Software libraries for connecting to groups and provides a method of laser- Though the motion recognition might
a Wii remote, parsing the input report pointer-style input. The software can not necessarily be accurate, the expe-
data, and configuring the controller are transform the x, y coordinate dot pairs rience is quite compelling. However,
available for nearly every major devel- to provide an averaged x, y coordinate the majority of existing games simply
opment platform on Windows, Mac pair, a rotation, and a distance. The x, employ shake recognition to trigger an
OS, and Linux. The open development y, and rotation values correspond to the event similar to a button press.
community has created these libraries, controller’s yaw, pitch, and roll, respec- As in other game consoles, the but-
and you can download them for free. tively, and the distance is estimated us- tons of the Wii remote are heavily
Because these software APIs are in ac- ing the known physical separation of employed for triggering input events.
tive development and might change rap- the two IR groups and the camera’s Frequently, games use the Nunchuk at-
idly, I won’t discuss them in detail. Visit fixed field of view. tachment, which is designed to be held
http://wiili.org and http://wiibrew.org Common Wii game interactions us- in the nondominant hand and adds
for more information. ing the controller as a pointer include more buttons, an analog joystick, and
Accessing the data is usually as sim- selection, navigation, aiming a weapon another accelerometer for independent
ple as reading values from an array or or tool, drawing, rotating objects, and motion sensing in each hand. In total,
an appropriately named member vari- push-pull interactions. Although the the Wii remote with Nunchuck attach-
able of a Wii remote class object, such remote is frequently used as a pointer, ment provides 13 digital inputs, 12 ana-
as accelX = remote.accelerometer.x. The com- no game currently makes a significant log controls, and auditory, visual, and
puter receives input reports 100 times attempt to ensure that the cursor’s on- tactile feedback.
per second, providing low-latency data. screen position accurately matches the
As the software libraries evolve, they screen plane’s intersection with the ray Remote interaction
might support event queues, derivative defined by the axis of the Wii remote. techniques without
values, and utilities that compute useful Assumptions are made regarding the the Wii console
transformation matrices or recognize screen’s visual angle and the scale of The Wii remote’s rich level of input
gestures, thereby simplifying applica- movement. However, this doesn’t ap- and output combined with the ease of
tion development. pear to have a significant impact on us- PC connectivity have made it a popu-
In many cases, the most difficult part ers’ pointing ability in most contexts. lar platform for exploring alternative
of this process is getting the Bluetooth This might be because the game pro- control schemes for existing applica-
pairing to occur successfully. Because vides constant visual feedback of the tions. Many initial projects in the de-
the Wii remote isn’t 100 percent HID cursor position, which lets users rely veloper community involved using the
compliant, it might work only with cer- on relative movements rather than ab- motion- and tilt-sensing capabilities for
tain Bluetooth chipsets and driver soft- solute aiming. robotic control and synthesized musical
ware. However, once a pairing is suc- Use of the accelerometer data within performance. For example, see the Wii
cessful, the configuration is typically
quite reliable. After you’ve connected
the Wii remote and installed the soft- The open development community has created
ware library, developing custom appli-
cations is straightforward. software libraries for connecting to a Wii remote
The projects I describe in this article
are C# Windows software applications for nearly every major development platform.
using Brian Peek’s Managed Library
for the Wiimote. 2 Wii games varies from basic shake trig- remote’s use in making the iSobot per-
gering, to tilt-and-balance control, to form combat motions (www.robodance.
Wii console simple gesture recognition. WiiSports, com/Nintendo-wii-i-sobot.php) and in
interaction techniques the mini-game that comes with the Wii composing music in the Kyma X devel-
Wii users hold the remote controller console, might currently involve the opment environment (www.youtube.
in one hand and point it at a television most intricate use of the accelerometer com/watch?v=ESDzYYl0__s).
that has a Wii sensor bar either above data. WiiSports encourages players to Software libraries to replicate the
or below the screen. The term “sensor swing the remote in the imaginary con- remote’s cursor-pointing capabilities
July–September 2008 use limited to: Politecnico di Milano. Downloaded on September 23,2023 at 18:39:40 UTC from IEEE P
Authorized licensed Xplore.
ER VA SI V E computing
Restrictions apply. 41
The Hacking Tradition
42 Authorized
P ER E computing
VA SI Vlicensed
use limited www.computer.org/pervasive
to: Politecnico di Milano. Downloaded on September 23,2023 at 18:39:40 UTC from IEEE Xplore. Restrictions apply.
Figure 4. Interactive display. (a) Infrared
LED pens used as a stylus for (b) an
interactive whiteboard.
July–September 2008 use limited to: Politecnico di Milano. Downloaded on September 23,2023 at 18:39:40 UTC from IEEE P
Authorized licensed Xplore.
ER VA SI V E computing
Restrictions apply. 43
The Hacking Tradition
nearby surfaces. This field of research the camera pose would enable registra- IR glyphs
is called spatial augmented reality. tion onto surfaces in 3D space. Because the Wii remote can track up
For fixed objects, you can manu- to four points simultaneously, we could
ally align projected imagery onto a Other projects use spatial and temporal multiplexing
surface’s physical features. 5 However, I hope these few projects I’ve described of IR emitters to create unique identi-
projecting imagery onto moving ob- so far have demonstrated the immense fiers. This would let the remote discover
jects requires very-low-latency, high- utility the Wii remote provides. The fol- the identity of an object it’s pointing at,
resolution tracking to ensure sufficient lowing are additional project concepts which means it could control arbitrary
registration quality to make the illusion that haven’t yet been implemented, but instrumented objects in the environ-
compelling. The Wii remote’s low-cost, could further increase the remote’s pos- ment simply by pointing. The remote
high-performing camera provides an sible applications. could manipulate lights, electronic
attractive option for this application. doors, vehicles, appliances, or other
Unfortunately, the remote can track 3D motion tracking objects in the environment.
only up to four points simultaneously, By using two Wii remotes, we can ap- If the identifiers are associated with
which limits the number of objects you ply stereo-vision techniques to acquire computer displays, individuals could
can track and the geometric complex- 3D tracking data from individual IR use their personal Wii remotes to inter-
ity of surfaces before you must make emitters. Multiple remotes could cover a act with any participating display in an
significant assumptions. larger tracking volume and a wider range intuitive and immediate manner. Using
For example, four points are sufficient of occlusion conditions. IR illumination each controller’s unique identity, seam-
to track a quadrilateral surface’s general and reflective tags could support the as- less file manipulation and management
orientation. However, if the surface is sembly of a low-end motion-capture sys- across displays and computers would
known to be square or is constrained tem for a couple hundred dollars. be possible through a centralized in-
to a table surface, you can use fewer formation server.
points to track the surface orientation Tracking objects with ID
and use the remaining points for input. One limitation of camera-based track- Laser tag
Figure 6 shows examples of a display ing is the inability to easily detect emit- If IR emitters are attached to each re-
projected to a foldable newspaper and ter identity. Researchers have explored mote, each remote can see the others’
a folding fan. A video demonstration of temporal variations in emitter behavior locations. This would support a laser-
this work is available at www.cs.cmu. to communicate identity. However, the tag-style interaction in which individu-
edu/~johnny/academic. data rate of ID transmission is directly als hold their own remotes. Your in-
You can use a four-point homogra- related to a camera’s frame rate, which tended target could be discovered by
phy calibration, similar to the interac- has typically been in the range of 30 blinking each player’s IR emitter in
tive whiteboard application, to regis- Hz. The Wii remote’s 100 Hz refresh some identification or hit-validation
ter surfaces that are constrained to a and several bits of IR dot size or inten- pattern.
plane. However, if you know the pro- sity provide an opportunity for higher
jector parameters, you could leverage data rates resulting in faster recognition Gesture recognition
the epipolar geometry of the projector- of a larger set of trackable and identifi- The gesture recognition in Nintendo Wii
camera pair and compute the funda- able objects. games, using either the accelerometer
mental matrix.6 Given four points of Alternatively, we could couple the data or camera-tracking data, has been
known geometric relationship, com- Wii remote with high-speed IR receiv- rather limited relative to what’s possible
bining the matrix with a solution for ers, similar to those used for remote in contemporary research systems. We
44 Authorized
P ER E computing
VA SI Vlicensed
use limited www.computer.org/pervasive
to: Politecnico di Milano. Downloaded on September 23,2023 at 18:39:40 UTC from IEEE Xplore. Restrictions apply.
might explore how to adapt gesture- the AUTHORS
recognition algorithms to the particular Johnny Chung Lee is a researcher in the Applied Sciences group at microsoft-
characteristics of the accelerometer and Hardware, although the work reported in this article was done while he was a
orientation data that the Wii remote’s phD student at Carnegie mellon university. His research focuses on exploring
novel techniques that enhance the practicality and accessibility of interactive
camera provides. The data from these technology. lee received his phD in human-computer interaction from Carn-
inputs presents unique challenges for egie mellon university. Contact him at [email protected].
recognition systems to properly param-
eterize variations in speed, size, and ori-
entation for a given gesture. A number
of developers are currently exploring
this issue, but it still remains an open
research problem. However, a robust community’s energy and imagination www.cs.unc.edu/~tracker/media/pdf/
method for performing accelerometer- will lead to countless more uses than I SIGGRAPH2001_CoursePack_11.pdf.
based gesture recognition would be a could possibly list here. 4. E.W. Weisstein, “Homography,” Math-
significant contribution to a wide vari- World—A Wolfram Web Resource, http://
ety motion-sensing applications in both mathworld.wolfram.com/Homography.
html.
the industrial and research domains.
REFERENCES
5. R. Raskar, G. Welch, and K.-L. Low,
T
1. Nintendo, Consolidated Financial High- “Shader Lamps: Animating Real Objects
lights, 24 Jan. 2008, www.nintendo. with Image-Based Illumination,” Proc.
he Wii remote’s rich I/O ca- Eurographics Workshop on Rendering,
co.jp/ir/pdf/2008/080124e.pdf.
pabilities clearly support a Springer, 2001, pp. 89–102.
wide range of potential ap- 2. Managed Library for Nintendo’s Wiimote,
blog, http://blogs.msdn.com/coding4fun/ 6. R. Hartley and A. Zisserman, Multiple
plications beyond its origi- View Geometry in Computer Vision,
archive/2007/03/14/1879033.aspx.
nal intended use. Its low cost and easy Cambridge Univ. Press, 2003.
Bluetooth connectivity have made it an 3. B.D. Allen, G. Bishop, and G. Welch,
ideal platform for the developer com- “Tracking: Beyond 15 Minutes of
Thought,” Proc. 28th Ann. Conf. Com- For more information on this or any other com-
munity to create custom control and puter Graphics and Interactive Tech- puting topic, please visit our Digital library at
tracking applications. I’m certain the niques (Siggraph 01), ACM Press, 2001, www.computer.org/csdl.
The #1 AI Magazine
www.computer.org/intelligent
IEEE
July–SEptEmbEr 2008 use limited to: Politecnico di Milano. Downloaded on September 23,2023 at 18:39:40 UTC from IEEE P
Authorized licensed Xplore.
ER VA SI V E computing
Restrictions apply. 45