Introduction

Feature Estimation

Keypoint Detection

PCL :: Features
Michael Dixon and Radu B. Rusu
July 1, 2011

Keypoints + Features

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Outline

1. Introduction
2. Feature Estimation
3. Keypoint Detection
4. Keypoints + Features

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Outline

1. Introduction
2. Feature Estimation
3. Keypoint Detection
4. Keypoints + Features

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Local Features
What is a feature?

What is a feature?
In vision/perception, the word feature can mean many
different things. In PCL, feature estimation means:
I

computing a feature vector based on each points local

neighborhood

or sometimes, computing a single feature vector for the

whole cloud

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Local Features
What is a feature?

Feature vectors can be anything from simple surface normals to

the complex feature descriptors need for registration or object
detection.
Today, well look at a couple of examples:
I

Surface normal estimation (NormalEstimation)

Point Feature Histogram estimation (PFHEstimation)

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Local Features (1-2/5)

Surface Normal Estimation. Theoretical Aspects: Basic Ingredients

Given a point cloud with x,y,z 3D point coordinates

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Local Features (1-2/5)

Surface Normal Estimation. Theoretical Aspects: Basic Ingredients

Given a point cloud with x,y,z 3D point coordinates

Select each points k -nearest neighbors, fit a local plane,

and compute the plane normal

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Local Features (3/5)

Surface Normal and Curvature Estimation

Cj =

i =

Pk

i=1 i

di2
2

(pi pj )T (pi pj ), p =

, pi outlier

1
k

p =

Pk

i=1 pi

0
0 +1 +2

1, pi inlier

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

NormalEstimation example
Estimating surface normals

void compute_surface_normals
(pcl::PointCloud<pcl::PointXYZRGB>::Ptr &points, float normal_radius
pcl::PointCloud<pcl::Normal>::Ptr &normals_out)
{
pcl::NormalEstimation<pcl::PointXYZRGB, pcl::Normal> norm_est;
// Use a FLANN-based KdTree to perform neighborhood searches
norm_est.setSearchMethod
(pcl::KdTreeFLANN<pcl::PointXYZRGB>::Ptr
(new pcl::KdTreeFLANN<pcl::PointXYZRGB>));
// Specify the size of the local neighborhood to use when
// computing the surface normals
norm_est.setRadiusSearch (normal_radius);
// Set the input points
norm_est.setInputCloud (points);
// Set the search surface (i.e., the points that will be used
// when search for the input points neighbors)
norm_est.setSearchSurface (points);

// Estimate the surface normals and store the result in "normals_out

norm_est.compute (*normals_out);
}

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

NormalEstimation results
$ ./features_demo ../data/robot1.pcd normals

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Now lets look at a more complex feature, like...

I

Point Feature Histograms (PFH)

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Point Feature Histograms

A 3D feature descriptor

For every point pair

h(ps , ns ); (pt , nt )i, let
u = ns , v = (pt ps )u, w = uv
f0 = hv , nj i

j
k
f1 = hu, pj pi i/||pj pi ||
P
fx d
x
ihist = x3
x=0 fxmax fxmin d

f2 = ||pj pi ||

f3 = atan(hw, nj i, hu, nj i)

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Point Feature Histograms

A 3D feature descriptor

f0 = hv , nj i

j
k
f1 = hu, pj pi i/||pj pi ||
P
fx d
x
ihist = x3
x=0 fxmax fxmin d

f2 = ||pj pi ||

f3 = atan(hw, nj i, hu, nj i)

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Point Feature Histograms

A 3D feature descriptor

For every point pair

h(ps , ns ); (pt , nt )i, let
u = ns , v = (pt ps )u, w = uv

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Point Feature Histograms

Points lying on different geometric primitives

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

PFHEstimation example
PFH is a more complex feature descriptor, but the code is very similar
using namespace pcl; // Lets save ourselves some typing

void compute_PFH_features
(PointCloud<PointXYZRGB>::Ptr &points, PointCloud<Normal>::Ptr &norm
float feature_radius, PointCloud<PFHSignature125>::Ptr &descriptors
{
// Create a PFHEstimation object
PFHEstimation<PointXYZRGB, Normal, PFHSignature125> pfh_est;

// Set it to use a FLANN-based KdTree to perform its neighborhood se

pfh_est.setSearchMethod (KdTreeFLANN<PointXYZRGB>::Ptr
(new KdTreeFLANN<PointXYZRGB>));
// Specify the radius of the PFH feature
pfh_est.setRadiusSearch (feature_radius);
// Set the input points and surface normals
pfh_est.setInputCloud (points);
pfh_est.setInputNormals (normals);
// Compute the features
pfh_est.compute (*descriptors_out);
}
Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

PHFEstimation example

...actually, well get back to this one later. For now, lets move
on to keypoints.

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Outline

1. Introduction
2. Feature Estimation
3. Keypoint Detection
4. Keypoints + Features

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Keypoints
What is a keypoint?

What is a keypoint?
I

A keypoint (also known as an interest point) is simply a

point that has been identified as a relevant in some way.

Whether any given point is considered a keypoint or not

depends on the *keypoint detector* in question.

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Keypoints
What is a keypoint?

Theres no strict definition for what constitutes a keypoint

detector, but a good keypoint detector will find points which
have the following properties:
I

Sparseness: Typically, only a small subset of the points in

the scene are keypoints

Repeatiblity: If a point was determined to be a keypoint in

one point cloud, a keypoint should also be found at the
corresponding location in a similar point cloud. (Such
points are often called "stable".)

Distinctiveness: The area surrounding each keypoint

should have a unique shape or appearance that can be
captured by some feature descriptor
Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Keypoints
Why find keypoints?

What are the benefits of keypoints?

I

Some features are expensive to compute, and it would be

prohibitive to compute them at every point. Keypoints
identify a small number of locations where computing
feature descriptors is likely to be most effective.

When searching for corresponding points, features

computed at non-descriptive points will lead to ambiguous
feature corespondences. By ignoring non-keypoints, one
can reduce error when matching points.

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Keypoints
Keypoint detection in PCL

There are a couple of keypoint detectors in PCL so far.

I NARFKeypoint
I
I

Requires range images

Bastian will talk more about this later

SIFTKeypoint
I

A 3D adaptation of David Lowes SIFT keypoint detector

Now lets look at an example of how to compute 3D SIFT

keypoints

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

SIFTKeypoints example
Computing keypoints on a cloud of PointXYZRGB points
using namespace pcl;
void detect_keypoints
(PointCloud<PointXYZRGB>::Ptr &points, float min_scale,
int nr_octaves, int nr_scales_per_octave, float min_contrast,
PointCloud<PointWithScale>::Ptr &keypoints_out)
{
SIFTKeypoint<PointXYZRGB, PointWithScale> sift_detect;
// Use a FLANN-based KdTree to perform neighborhood searches
sift_detect.setSearchMethod
(KdTreeFLANN<PointXYZRGB>::Ptr
(new KdTreeFLANN<PointXYZRGB>));

// Set the detection parameters

sift_detect.setScales (min_scale, nr_octaves, nr_scales_per_octave);
sift_detect.setMinimumContrast (min_contrast);
// Set the input
sift_detect.setInputCloud (points);
// Detect the keypoints and store them in "keypoints_out"
sift_detect.compute (*keypoints_out);
}
Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

SIFTKeypoint results
$ ./features_demo ../data/robot1.pcd keypoints

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Outline

1. Introduction
2. Feature Estimation
3. Keypoint Detection
4. Keypoints + Features

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Keypoints + features

Keypoints and feature descriptors go hand-in-hand.

Lets look at how to compute features only at a given set of
keypoints.

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Keypoints + features (1)

Computing feature descriptors for a set of keypoints
using namespace pcl;
void
compute_PFH_features_at_keypoints
(PointCloud<PointXYZRGB>::Ptr &points,
PointCloud<Normal>::Ptr &normals,
PointCloud<PointWithScale>::Ptr &keypoints, float feature_radius,
PointCloud<PFHSignature125>::Ptr &descriptors_out)
{
// Create a PFHEstimation object
PFHEstimation<PointXYZRGB, Normal, PFHSignature125> pfh_est;
// Set it to use a FLANN-based KdTree to perform its
// neighborhood searches
pfh_est.setSearchMethod
(KdTreeFLANN<PointXYZRGB>::Ptr (new KdTreeFLANN<PointXYZRGB>));
// Specify the radius of the PFH feature
pfh_est.setRadiusSearch (feature_radius);
// continued on the next slide ...
Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Keypoints + features (2)

// ... continued from the previous slide

// Convert the keypoints cloud from PointWithScale to PointXYZRGB
// so that it will be compatible with our original point cloud
PointCloud<PointXYZRGB>::Ptr keypoints_xyzrgb
(new PointCloud<PointXYZRGB>);
pcl::copyPointCloud (*keypoints, *keypoints_xyzrgb);

// Use all of the points for analyzing the local structure of the cl
pfh_est.setSearchSurface (points);
pfh_est.setInputNormals (normals);
// But only compute features at the keypoints
pfh_est.setInputCloud (keypoints_xyzrgb);
// Compute the features
pfh_est.compute (*descriptors_out);
}

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

What can you use this for?

Heres an example of how you could use features and keypoints

to find corresponding points between two point clouds.
int correspondences_demo (const char * filename_base)
{
// Okay, this is a bit longer than our other examples,
// so lets read the code in another window.
// (See "features_demo.cpp", lines 308-368 )
}

Point Cloud Library (PCL)

Introduction

Feature Estimation

Keypoint Detection

Keypoints + Features

Correspondence results

$ ./features_demo ../data/robot
correspondences

Point Cloud Library (PCL)