A Depth Sensor to Control Pick-and-Place Robots for

Fruit Packaging
Abdul Md Mazid
Pavel Dzitac
School of Engineering and Built Environment
School of Engineering and Built Environment Central Queensland University
Central Queensland University Rockhampton, Australia
Rockhampton, Australia E-mail: [email protected]
E-mail: [email protected]

Abstract— this paper presents a powerful and inexpensive

automated object detection and localization method for pick-and- II. ARCHITECTURAL THEORY AND DEVELOPMENT
place robots using a low cost, commercial 3D depth sensor. As mentioned earlier the experimental system, for detecting
and locating individual object from the possible upper layer,
The robot control application developed for this project extracts
relevant information such as the XYZ co-ordinates of an object’s is consisted of the following:
location from a 3D depth map, generated by the depth sensor
interface software. This information is then used by the robot • A depth sensor,
control application to control a pick-and-place robot for handling • Interface software, and
object/fruit. • An image processing library
Experiments have proven that object’s projected area and
Assembly and interfacing of these equipments were performed
orientation can be extracted from the depth map provided by the
3D depth sensor. for experimental purposes that are described below.
A. Robot-sensor interface design
This type of depth sensor can be used in robotics research
projects for short range environment mapping and navigation The depth map point cloud data can be processed using 3D
and for object detection and locating. point cloud processing libraries to extract 3D information of
interest to the application being developed.
Keywords— depth sensor, depth map, pick-and-place robot,
fruit packaging For the application presented in this paper the robot-sensor
I. INTRODUCTION interface was designed such that the 3D depth map generated
by the depth sensor can be captured and processed to extract
This paper presents a prototype robotic pick-and-palce the XYZ location co-ordinates for each object of interest in the
application development that uses the Asus Xtion Pro Live 3D scene and provide this information to the controller of the
depth sensor, OpenNI interface software and the AForge pick-and-place robot. The depth map point cloud data can be
framework’s image processing library to locate fruits (melon, used to extract any relevant information about the object for
oranges, tomatoes, or pineapples) in container and direct the the purpose of robot control decision making.
pick-and-place robot with its gripper to the XYZ location of
each of the fruits on the top layer of the container. The The block diagram in Figure 1 shows the configuration of
experiments were performed successfully for detecting and the robot-sensor interface at the hardware level. The interface
locating oranges on a plain surface. between the sensor and the computer is provided via USB 2.0
communication. The interface between the computer and the
This technology can be advantageously used in packaging robot controller is provided via RS232 communication.
industry particularly for packaging fruits and vegetables such Obviously the interface between the computer and the robot
as oranges, pineapples, mangoes, tomatoes, etc. and possibly controller can be provided using any other standard
for sorting them according to sizes. Sorting according to sizes, communication hardware that is available on both the
mass, quality is an essential task particularly in fruit and computer and the robot controller. In this case the RS232
vegetable industry. Envisaged, that depth sensor method can communication is used because it is readily available on both
be applied in packaging industry for sorting and packaging and is adequate in terms of the required data throughput.
purposes efficiently.
 The robot control application presented in this paper
extracts relevant information from the depth map data supplied
by the depth sensor via the OpenNI interface software, and
uses it as an input to the control logic that determines the
actions of the robot.
B. The Asus Xtion Pro Live depth sensor
The Xtion sensor consists of a depth sensor (infrared
projector and receiver), an RGB camera and two microphones
as shown in Figure 4. The depth sensor itself consists of a
structured infrared light projector that projects a 2D array of
Figure 1. Layout of the robot-sensor hardware interface infrared dots and a receiver that decodes the dot pattern and
generates a raw 3D depth point cloud.
The second block diagram in Figure 2 displays the
configuration of the robot-sensor interface at the software
level. The interface between the sensor firmware and the user
software application running on the computer is provided by
the OpenNI interface software [2], which uses the sensor
driver to capture the raw depth point cloud. Standard
Microsoft Visual C# library functions are used to create the
interface between the user application and the robot controller. Figure 4. The Xtion Pro Live sensor

The raw depth point cloud stream is processed by the

OpenNI interface software which generates a depth map
(distance map) in millimeters after applying correction factors
to the raw depth point cloud. The specified useable range of
the depth sensor is between 800mm to 3500mm. In other
words the object has to be at least 800mm away from the
sensor and not further than 3500mm to be reliably detectable.

The Xtion sensor transmits video, depth and audio streams.

The video and depth streams are transmitted at frame rates of
60fps (320x240 resolution) and 30fps (640x480 resolution).
We ran our depth stream tests successfully at a frame rate of
Figure 2. Layout of the robot-sensor software interface
30fps using a resolution of 640x480.
Figure 3, extracted from the OpenNI User Guide [2], shows
the relationship between the hardware such as the Xtion Pro The Xtion Pro Live sensor is commercially available from
Live depth sensor, the OpenNI interface software, the Asus with a bundled SDK and is similar to Microsoft’s Kinect
middleware (OpenNI plug-in) and the user application sensor used on Xbox 360. Both sensors are based on
software. technology developed by PrimeSense [6].

C. Installation and setup

For Windows environments the software installation
package is provided for both the 32bit and 64bit machines.
The latest version of the driver & SDK package was installed
available from the Asus website. The latest version of OpenNI
software is also available from the OpenNI website.

The Xtion sensor is installed by following the installation

procedure provided by Asus.

III. ROBOT CONTROL APPLICATION DEVELOPMENT

Figure 3. The layer in the OpenNI framework The prototype robot control application was developed
using Microsoft Visual C# 2010. The easiest way to start is to
modify a sample applications provided with the SDK [3]. image, and locates islands of pixels that are separated by a
black background.
The application presented in this paper was created by
modifying the SimpleViewer sample application provided The Blob is an AForge structure that holds the various bits
with the SDK. Our application uses the OpenNI interface of information about the detected blob, such as its XY position
software to capture the depth map. The depth map is then and its area.
processed by the robot control application using the AForge
image processing library to find the objects that are nearest to A size filter can be applied to the BlobCounter to force it
the depth sensor and determine their position. to consider blobs of a certain size range and ignore the rest.
This functionality is useful and can simplify the object
identification task in the user application.
The following C# code shows the required references to
OpenNI and AForge.
The following C# code applies a filter to the instantiated
blobCounter and forces it to ignore objects that are not
using OpenNI;
within the filter constraints.
Using AForge;
using AForge.Imaging;
blobCounter.FilterBlobs = true;
using AForge.Math.Geometry; blobCounter.MinHeight = 20;
blobCounter.MinWidth = 20;
The following C# code shows the required declarations for blobCounter.MaxHeight = 60;
OpenNI. blobCounter.MaxWidth = 60;

private Context context; For the purpose of this project the depth map provided by
private ScriptNode scriptNode; the depth sensor is converted to a 2D image after the vertical
private DepthGenerator depth; Z-distance to the object is obtained. The Z-distance of interest
in this project is the Z-distance from the sensor to the top of
A Context is defined as a workspace where the application
the object. This distance is found by applying a distance
builds its OpenNI production graph and holds the information
regarding the state of the application. The production graph is threshold to the depth map data and then using the AForge
a map of all the nodes used in the application. image processing library functions to find the distance at
which the objects are detected. The distance threshold is
To use OpenNI we constructed and initialized a Context. sequentially increased from the minimum distance of 800mm
The ScriptNode allows OpenNI to create nodes and manages to a desired maximum distance, which is sufficient to detect
all nodes that it has created by using scripts. The all objects of interest.
DepthGenerator is a node that has generated the depth map
from the raw depth point cloud. When the first object is detected its Z-distance is recorded.
From this first Z-distance found the threshold is increased for
The following C# code initializes the Context workspace a further distance say 30mm. The Z-distance of each of the
using an XML file and the ScriptNode. It also initializes the objects found in the 30mm scanned region is recorded into an
DepthGenerator node. array. The AForge library is used at the same time to find the
XY location of each of the detected objects and to provide
context=Context.CreateFromXmlFile(SAMPLE_XML_FIL other information such as object’s projected area and
E, out scriptNode); orientation. Once this information is available the robot
depth = context.FindExistingNode(NodeType.Depth) control application computes the control logic and determines
as DepthGenerator; where to direct the end-effectors of the pick-and-place robot.

The following C# code shows the required declarations for The below written C# code shows the depth data reading
AForge. sequence. The first statement creates an instance of a
private BlobCounter blobCounter = new DepthMetaData. The depth metadata contains the actual
BlobCounter(); distance data (depth map) after correction factors were applied
private Blob[] blobs; to the raw depth data. Next OpenNI is instructed to wait for
update from the depth node. When the update is received
The BlobCounter is an AForge function that processes a OpenNI is instructed to get the metadata and store it in the
2D image, typically formatted as a 24bpp RGB image, finds depthMD variable of type DepthMetaData.
“blobs” in the image and computes their location, area and
other parameters. A blob is an island of pixels in the image. //Create a DepthMetaData instance
AForge thresholds the image to generate a black and white DepthMetaData depthMD = new DepthMetaData();
 //Read next available raw depth data }
context.WaitOneUpdateAll(depth); else if(blobCnt > 0)
}
//Get depth metadata from raw depth data //when blob found record its Z distance
depthMD = depth.GetMetaData(); blobZ[0] = z_distance;
z_distance = 0;
Once the metadata (depth map) is available it can be extracted }
from the depthMD structure and processed as desired. We use else
}
the AForge library to find the objects (blobs) and the Z-
//no blob found if search range exceeded
distance to each object. no_blobs = true;
z_distance = 0;
The first step is to convert the depth information to a bitmap }
image so it can be processed by AForge. This is done by
applying a threshold to the distance data and converting it to a The above code can be modified to find all objects that are
binary image. The converted bitmap image is then passed to in the selected search range. This can be done by searching for
AForge for processing and blob finding as shown in the C# objects until the specified maximum search distance is
code below. reached, and not limiting the search by the number of objects
found. When the objects are found, their XYZ location
//process the 2D image information is sent to the robot controller. The robot is then
blobCounter.ProcessImage(AForge.Imaging.Image.Cl commanded to direct the gripper to the given location, pick up
one(image,System.Drawing.Imaging.PixelFormat.For
the detected object and put it in a new desired location, such as
mat24bppRgb));
in a packaging box or a quality inspection machine.
The blob information is then retrieved from AForge as an
array of blobs, as shown in the C# code below. In this case the IV. EXPERIMENT FOR PERFORMANCE TESTING
maximum number of detected blobs is determined by the robot
control application, which can be designed to find objects
within a specified depth search range (but within the sensor The blob detection application was tested using a single
working range). layer of several oranges as objects on a horizontal plane, as
shown in Figure 5.
//Get info for each blob found
blobs = blobCounter.GetObjectsInformation(); The Xtion sensor was located about 860mm above the
horizontal plane. The aim of the test was to determine whether
Each element of the retrieved array of blobs holds the the search algorithm would detect the objects and their XYZ
information about each blob in a structure and can be queried
location reliably.
to retrieve the required information as shown in the code
below.

//Record the number of blobs found

blobCnt = blobs.Length;

//Extract the area and location info for each

blob found
for (int i = 0; i < blobs.Length; i++)
{
blobArea[i] = blobs[i].Area;
blobX[i] = blobs[i].CenterOfGravity.X;
blobY[i] = blobs[i].CenterOfGravity.Y;
}

The following C# code searches for the first blob (top-most Figure 5. Experimental setup for object detection
object) by incrementing the blob search distance (z_distance)
until the blob is found or the maximum search distance is The robot control application detected the objects as shown
reached. in Figure 6 (the front row of oranges in Figure 5 is the top row
of blobs in Figure 6). Out of fifteen oranges thirteen were
//Search for blob if(blobCnt < 1 && z_distance < detected as blobs, one was ignored as too small by the filter
830) setting in the application and one was not detected at all
{ because it was too low (outside the preset detection range).
//increment z distance until next blob found
The results on the right are for the first blob in the array of
z_distance += 2;
detected blobs The Centre of Gravity is the XY position of the measure depth is given by Nate Lowry [9].
detected object. The Z position (not shown in the picture) was
calculated separately during the vertical search for each blob. V. CONCLUSION
The two small size blobs on the two edges of the image are the A flexible and inexpensive object detection and localization
two legs of the sensor support. When detecting objects in a method for pick-and-place robots that can be developed with
container the edges of the depth map can be cropped to ignore little effort has been presented.
the sides of the container during object detection. Filters
would also reject most of the artifacts in the image. Although the Xtion and Kinect depth sensors were intended
for gaming applications, they can be used for indoor robotics
to provide useful depth information in many applications.
Depth sensors provide the robots with flexible and powerful
means of locating objects, such as boxes, without the need to
hardcode the exact co-ordinates of the box in the robot
program. This technology can be used for sorting and
packaging of various fruit and vegetables such as oranges,
apples, pineapples and grapefruit.

Further improvement can be done in this application to

synchronize Xtion’s RGB camera with the depth sensor to
Figure 6. The oranges test results using the OpenNI SimpleViewer provide color information, and to improve the XY resolution
and accuracy. This would be useful in general as many
Although this is a simple test it shows the usefulness and applications, such as fruit sorting, require or could benefit
flexibility of this sensor even when used in conjunction with from object’s color information.
the free open source point cloud and image processing
libraries. Most of such libraries are available for C++. A sophisticated detection and localization system for
randomly orientated objects in containers can be developed
Once the detected object’s data is available it can be used relatively easy using these depth sensors and the open source
by the control application to send the pick-and-place robot to libraries available for 2D image and 3D point cloud
the object’s location. The application can update the object processing.
coordinates in real time, and therefore the robot would not
have to wait at any point for the robot control application to ACKNOWLEDGMENT
update the location of the objects. The preferred spelling of the word “acknowledgment” in
America is without an “e” after the “g”. Avoid the stilted
Many of the important specifications such as the depth map expression, “One of us (R. B. G.) thanks . . .” Instead, try “R.
horizontal and vertical resolution are not stated in the basic B. G. thanks”. Put sponsor acknowledgments in the unnum-
bered footnote on the first page.
sensor documentation, so some research was required.
According to our testing, at 2m from sensor, the depth map REFERENCES
has a resolution of about 3mm along the X and Y axes, and a
depth (Z) resolution of about 10mm. This agrees with the [1] Pavel Dzitac, Abdul Md Mazid, “Principles of sensor technologies for
sensor resolution information that we found. object recognition and grasping”, Proceedings, 15th International
Conference on Mechatronics Technology, Melbourne, 2011.
[2] OpenNI User Guide, Available:
The resolution of this sensor increases as the distance from http://openni.org/Documentation/ProgrammerGuide.html.
the sensor decreases. At 1m from sensor the XY resolution is [3] Asus Xtion Pro Live driver & SDK package, Available:
about 1.5mm and the Z resolution is about 5mm. http://www.asus.com/Multimedia/Motion_Sensor/Xtion_PRO_LIVE/#d
ownload.
[4] Microsoft MSDN library – C# Programmer’s Reference. Available:
The object detection precision at 1m from sensor is very http://msdn.microsoft.com/en-us/library/618ayhy6(v=VS.71).aspx.
good for most packaging and material handling applications. [5] AForge.Net Framework. Available: http://www.aforgenet.com/
[6] PrimeSense, Available: http://www.primesense.com/#4.
[7] Pavel Dzitac, Abdul Md Mazid, “An advanced control technology for
Although these low cost sensors do not have high robotic palletising in materials handling for higher productivity”,
resolution, they are still extremely useful for locating objects Proceedings, 15th International Conference on Mechatronics
Technology, Melbourne, 2011.
in many applications. Additionally the RGB camera can be
[8] P. Dzitac, A. M. Mazid, “An Efficient Control Configuration
used in conjunction with the depth sensor to improve the XY Development for a High-speed Robotic Palletising System”, Proceedings
resolution and accuracy if necessary. of 2008 IEEE International Conference on Robotics, Automation and
Mechatronics (RAM 2008). Chengdu, China. 2008. pp140-145.
[9] Giovanna Sansoni, Marco Trebeschi, and Franco Docchio, “State-of-
A good explanation on how Kinect (and Xtion) sensors The-Art and Applications of 3D Imaging Sensors in Industry, Cultural
 Heritage, Medicine, and Criminal Investigation”, Sensor 2009, Vol. 9.
Pp. 568-601.
[10] The Kinect SensesDepth available via link:
http://nongenre.blogspot.com.au/2010/12/how-kinect-
senses-
depth.html?z#!http://nongenre.blogspot.com/2010/12/how
-kinect-senses-depth.html