Kinect Open Source Programming Secrets: Hacking the Kinect with OpenNI, NITE, and Java

Introduction

It was at my university’s 2011 Open Day that I realized the importance of the Kinect sensor. Returning from lunch, I saw a very long line of people waiting to get into one of our computer labs. A lab of PCs is nothing special to today’s computer-savvy kids, so I walked over to see what the fuss was about. People wanted to dance in a virtual disco, boogying on down in front of a Kinect.

To my mind, the Kinect has achieved two remarkable things. One is that it introduced the wonders of computer vision to a public waiting for the next Big Thing. Also, the Kinect is the first consumer-level (affordable) device that makes it comparatively easy to program with motion sensing, skeletal tracking, and natural user interfaces using gestures. That’s why the Kinect has caught the imaginations of the hacker/developer community, won a slew of awards for best gadget and innovative engineering, and been awarded a Guinness World Record for the fastest selling consumer electronics device.

But notice that I said comparatively easy to program. It’s fair to say that there has been a rather dismal lack of documentation on the Kinect’s APIs. This book has grown out of my online tutorials (at http://fivedots.coe.psu.ac.th/∼ad/kinect), which aim to remedy that problem.

Why Buy This Book?

I can suggest four reasons for buying this book:

 It offers a unique choice of Kinect programming tools.

 It explains the official Java wrappers for those tools.

 It covers topics not found elsewhere.

 It provides depth, but with brevity.

Unique Programming Tools

This is the only book on programming the Kinect using the OpenNI library, NITE, and Java (as of April 2012, when this book went to press). Of course, this claim won’t mean much if you don’t know OpenNI or NITE, so let me briefly describe them.

OpenNI (or more correctly, the OpenNI framework) is open source software for accessing the Kinect’s sensors. OpenNI has been ported to multiple platforms, such as Windows, Mac OS X, and Linux. It was developed by PrimeSense, the company that worked with Microsoft on building the Kinect. NITE adds higher-level features to OpenNI, including user tracking and hand gestures. It was also developed by PrimeSense, and although it isn’t open source, it is available for free. OpenNI and NITE have been around since the early days of the Kinect, and a vast number of fun projects use their APIs. The OpenNI and NITE developer forums are very active, providing a lot of help.

Official Java Wrappers

This is the only book that explains the official Java wrappers for OpenNI and NITE (again, as of April 2012). By official, I mean that these bindings were developed by PrimeSense. Obvious advantages of Java include object-orientation, cross-platform support, availability for free, and many people (including you, probably) knowing how to program with it. Most important, programming in Java gives you access to a massive number of libraries—for graphics, networking, and beyond—that can be linked to the Kinect without much effort. For example, I’ll demonstrate how to use the Java 3D graphics library and the Java binding for the OpenCV computer vision package.

Some Kinect books employ the Processing language, a simplified version of Java aimed at new programmers interested in generating graphics. Processing has many interesting libraries, but nothing like Java’s vast collections.

There are other Java bindings for OpenNI, created by third-party developers. These APIs tend to have limited functionality and are not consistently maintained.

The main drawback of using the PrimeSense Java wrappers is their lack of documentation. As I explain in Chapter 1, I had to decompile the libraries’ JAR files, and work out the correspondences between the Java source and the somewhat better documented C++ OpenNI/NITE APIs. (This is why including Secrets in the book’s title isn’t too excessive.)

A Wide Range of Topics

This book covers programming topics not found elsewhere. I start off with the basics, of course, with chapters on depth, infrared, and RGB imaging, point clouds, skeletal user tracking, hand tracking, and gesture support. Moving beyond that, I cover several novel and unusual features, including the following:

 Kinect gaming based around a version of the classic Breakout video game.

 Controls for the Kinect motor, LED, and accelerometer, which are not part of the standard OpenNI API. In fact, their absence is often held up as a serious drawback of the API. It’s actually quite easy to add these capabilities using a custom-built USB driver.

 3D graphics programming in the point cloud and skeletal tracking examples, using Java 3D.

 A computer vision example that demonstrates how to link the Kinect to the popular (and powerful) OpenCV library.

 The creation of new body gestures (inspired by the FAAST system), which are not part of the limited NITE repertoire.

 A new type of GUI component controlled by hand gesturing, illustrated with three examples: a button, dial, and slider. These components are controlled without the help of mouse or keyboard.

Depth with Brevity

This book describes a lot of complicated code but, unlike some rather hefty programming tomes, you won’t find all the code tediously printed on these pages. Instead, you can download it from the book’s website at http://fivedots.coe.psu.ac.th/∼ad/kinect/. In addition, I’ve been adding supplementary chapters to the website, including ones discussing speech recognition and the Kinect microphone array.

I’ve also cut down on the page count by assuming that you have some Java programming experience (perhaps you’ve taken a one-semester course on Java at college). That means that I don’t explain how to create a JFrame or search for substrings. I don’t describe every line of every program, but concentrate on the important Kinect-specific material.

What This Book Is Not About

The fact that I’m powering ahead without wasting time explaining about classes, objects, and inheritance could be viewed as a disadvantage. This is not a book for first-time Java programmers.

Also, I don’t have the space to seriously explain the topics of 3D graphics or computer vision. I introduce them (in the form of Java 3D and JavaCV) in enough detail so that you can understand my examples, but there is a lot of material that I don’t mention. When I get to these topics, I’ll suggest resources for finding more information.

Chapter 1

Getting Started

This chapter introduces the Kinect sensor, explains how to install the developer software on Windows (which can be a bit tricky), looks at some examples that come with the download, and provides an overview of the Java API for programming the Kinect. It even includes a very simple Java program for testing the API.

Later chapters will start programming in earnest, looking at imaging (depth detection, camera, and infrared), point clouds, skeletal user tracking in 2D and 3D, tilt motor control, hand tracking, Kinect games, gesture support, the Kinect and OpenCV, and gesture-controlled graphical user interface (GUI) components.

I’ll be programming the Kinect with OpenNI and NITE, not OpenKinect, CL NUI, or Microsoft’s Kinect for Windows SDK. The 3D features (in the point cloud and skeletal tracking examples) are implemented using Java 3D. My OpenCV coding will be with its JavaCV binding. I’ll be using the Java API released with OpenNI and NITE. I won’t be using Processing libraries such as Simple-OpenNI.

You can find detailed code examples at my website: http://fivedots.coe.psu.ac.th/∼ad/kinect/.

The Kinect Sensor Hardware

The Kinect sensor was originally intended to be a motion-sensing input device for the Xbox 360, allowing the user to control games via gestures and spoken commands. Key hardware components are an RGB camera, a depth sensor, multiarray microphones, a tilt motor, and a three-axis accelerometer. Figure 1-1 illustrates the components of the Kinect sensor.

FIGURE 1-1 Components of the Kinect sensor

Soon after the Kinect’s launch in November 2010, third-party developers started releasing software to allow it to be used on platforms other than just the Xbox (such as Windows, Linux, and the Mac). Microsoft eventually came out with a Windows 7-based SDK in June 2011. In February 2012, Microsoft released a new version of the Kinect specifically aimed at desktop PCs running Windows 7 or 8, aptly called Kinect for Windows, featuring a new Near Mode that lets gesture control be used as close as 40 cm. However, all the examples in this book use the older (and cheaper) original Kinect.

The Kinect’s depth sensor consists of an infrared (IR) light source and a laser that projects a pattern of dots, which are read back by a monochrome CMOS IR sensor. The sensor detects reflected segments of the dot pattern and converts their intensities into distances. The sensor’s depth range is from about 50 cm to 3 m, with a resolution of about 1 cm along the z axis, while spatial resolution (along the x and y axes) is in millimeters. The range data isn’t linear, as explained in an excellent blog post by Nathan Crock at http://mathnathan.com/2011/02/03/depthvsdistance/.

Each frame generated by the depth sensor is at VGA resolution (640 × 480 pixels), containing 11-bit depth values, which provides 2048 levels of sensitivity. The output stream runs at a frame rate of 30 Hz. The RGB video stream also uses VGA resolution and a 30 Hz frame rate.

There are also high-resolution versions for IR and video (1280 × 1024 pixels), which run at about a 10 Hz frame rate. The high-resolution camera image can be useful when more complex computer vision algorithms are being used with the Kinect.

The audio array consists of four microphones, with each channel processing 16-bit audio at a sampling rate of 16 KHz. The hardware includes ambient noise suppression.

The Kinect draws 12 watts, which is quite a bit more than the 2.5-watt power provided by a standard USB port. This means that it’s necessary to plug the device into both the PC’s USB port and a separate power supply for the Kinect to function properly.

Microsoft suggests that you allow about 6 feet (1.8 m) of empty space between you and the sensor. Standing too close or being in a cluttered environment confuses the depth sensor.

The iFixit website, at http://www.ifixit.com/Teardown/Microsoft-Kinect-Teardown/4066/1, has a detailed teardown of the Kinect’s hardware, with plenty of interesting pictures of its internals.

Kinect Development Software

Four main Kinect development libraries are available:

 OpenKinect’s libfreenect (http://openkinect.org/wiki/Main_Page)

 CL NUI (http://codelaboratories.com/nui/)

 OpenNI (http://www.openni.org/)

 Microsoft’s Kinect for Windows SDK (http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/default.aspx)

OpenKinect’s libfreenect is derived from a reverse-engineered/hacked Kinect driver, and supports Windows, Linux, and MacOS X. It was the work of Hector Martin, who delivered it just six days after the Kinect’s release, thereby winning the Adafruit Hack the Kinect contest.

CL NUI is aimed at Windows only, but allows multiple Kinect sensors to work together. It’s really not fair to compare libfreenect or CL NUI with OpenNI, which is intended to act as an interface between a variety of sensors (including a Kinect) and middleware for manipulating the sensor data in a higher-level manner (for example, for recognizing user gestures as commands).

Figure 1-2 shows OpenNI’s three-layer format (based on an image from the OpenNI user guide at http://www.openni.org/documentation).

FIGURE 1-2 The layers in the OpenNI framework

I don’t need OpenNI’s multidevice, multimiddleware capabilities, as I’m using only the Kinect and NITE middleware. OpenNI’s Kinect driver, SensorKinect, is derived from open sourced (LGPL) code from PrimeSense, the company behind Kinect’s depth-sensor technology. The SensorKinect driver is also known as the Avin2 driver, after its principal developer, Avin.

One advantage of libfreenect is API support for controlling the Kinect’s tilt motor, status light, and accelerometer, which is missing from the SensorKinect driver. However, it’s easy to add the necessary support using USB programming techniques. I’ll describe the details in Chapter 6.

An important reason for using OpenNI is its support for middleware. The NITE library understands different hand movements as gesture types based on how hand points change over time. NITE gestures include pushing, swiping, holding steady (no hand movement), waving, and hand circling. Chapters 7 through 10 explain how to program with gestures.

OpenKinect and OpenNI users have created numerous interesting, fun examples. Dedicated sites for Kinect hacking projects include http://www.kinecthacks.com/, http://kinect.dashhacks.com/, http://www.kinecthacks.nl/, and http://kinect-hacks.net/. More general maker sites with Kinect sections include http://hackaday.com/category/kinect-hacks/ and http://www.engadget.com/tag/kinect,hack. YouTube is a good source for ideas and inspiration; search for kinect and hacks.

The 800-Pound Gorilla Enters

Eight months after the Kinect’s release, Microsoft brought out a Kinect SDK (http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/download.aspx), which covers much of the same ground as OpenNI and NITE. The low-level API gives access to the depth sensor, image sensor, and microphone array, while higher-level features include skeletal tracking, audio processing, and integration with the Windows speech recognition API. A new version of the SDK was released in February 2012, but it mostly consists of minor enhancements and bug fixes. Two interesting improvements are per-user skeletal tracking and a Near Mode for the depth camera.

The main area where the SDK wins over OpenNI is audio. Although OpenNI contains audio-related APIs, as of this writing, none of them are currently implemented. The situation is somewhat better for OpenKinect, which has proof-of-concept audio code at https://github.com/OpenKinect/libfreenect/tree/master/examples (micview.c and wavrecord.c).

A nice aspect of Kinect audio support in Microsoft’s SDK is that it treats the microphones array as a USB audio device. This makes it possible for the Java sound API to connect to it and capture audio input. If you’re interested in audio and OpenNI, I’ve written two chapters looking at speech recognition, microphone capture, recording, and beamforming. I’m afraid they were finished too late for this book, but can be found online at the book’s website, http://fivedots.coe.psu.ac.th/∼ad/kinect/.

Other pluses for Microsoft’s SDK are its extensive documentation and examples, and its ease of installation on Windows 7. As you’ll see, both the documentation and installation steps for OpenNI could be improved.

The main drawback of the SDK is that it works only on Windows 7, not XP or any non-Windows platform. Also, applications must be written in C++, C#, or Visual Basic; naturally, I want to use Java. These are the main reasons I did not use the SDK for the examples in this book.

Programming the Kinect in Java

Both libfreenect and OpenNI have Java and Processing bindings. Processing (http://processing.org/) is a sketching language that’s translated into Java at compile time.

libfreenect lists two Java wrappers at http://openkinect.org/wiki/Main_Page; the JNA version comes with more information. A Processing binding called, somewhat confusingly, OpenKinect is available from http://www.shiffman.net/p5/kinect/.

OpenNI has an excellent Processing binding called Simple-OpenNI (http://code.google.com/p/simple-openni/), which comes with good examples, clear installation instructions, and an active developer forum.

If you want to employ only OpenNI’s Kinect driver, SensorKinect, then JKinect is a suitable Java wrapper for the driver. In a similar vein, there’s a Java binding for the CL NUI Kinect driver, called CLNUI 4 Java (http://www.pixelnerve.com/v/2010/11/19/clnui-4-java-kinect/).

I’m not using any of these libraries because, at the start of July 2011, the OpenNI and NITE releases finally came with their own Java bindings. This means that you can program the Kinect without adding extra third-party libraries.

Kinect Software Installation

Installing OpenNI and NITE currently involves three separate packages, which must be installed in the correct order onto a system that doesn’t contain older OpenNI versions or other Kinect drivers (such as libfreenect or CL NUI). In addition, the installed libraries require some manual configuration before they will work correctly.

This may seem like a pain, but the situation in the early days of Kinect development was much trickier, and the installation steps are