Virtual Humans – Winter 23/24

Lecture 4_1 – ICP: Iterative Closest Points

Prof. Dr.-Ing. Gerard Pons-Moll

University of Tübingen / MPI-Informatics
Non-rigid Articulated Registration

2
What is missing?
Given correspondences, we can find the optimal rigid alignment with
Procrustes.

PROBLEMS:
• How do we find the correspondences between shapes ?
• How do we align shapes non-rigidly ?

3
ICP and alignment based on optimisation
• Optimising alignment and correspondences using Iterative Closest
Point (ICP).

• Alignment through continuous optimisation.

4
How do we find correspondences?

5
How do we find correspondences?

6
How do we find correspondences?

7
How do we find correspondences?
?

compact notation: f contains translation, rotation and isotropic scale

Closest point to target shape point

The optimisation is over:

• the transform
• the correspondences

8
How do we find correspondences?
The idea was to minimise the sum of distances between the one set of
points and the other set, transformed
?

compact notation: f contains translation, rotation and isotropic scale

9
Ideas
The idea was to minimise the sum of distances between the one set of
points and the other set, transformed

compact notation: f contains translation, rotation and isotropic scale

What if we estimate the correspondences?

10
 Solution: Iteratively find correspondences
The idea was to minimise the sum of distances between the one set of
points and the other set, transformed

compact notation: f contains translation, rotation and isotropic scale

What if we estimate the correspondences?
iteration
Given current best transformation,
which are the closest correspondences?
original unsorted points
Given current best correspondences,
which is the best transformation?

11
 Alternate between finding correspondences
and finding the optimal transformation
The idea was to minimise the sum of distances between the one set of
points and the other set, transformed

compact notation: f contains translation, rotation and isotropic scale

What if we estimate the correspondences?
iteration
Given current best transformation,
which are the closest correspondences?
original unsorted points
Given current best correspondences,
which is the best transformation?

12
Make up reasonable correspondences

13
Make up reasonable correspondences

Neutral initialization.
Initializing t to align centroids
should work better!
14
Make up reasonable correspondences

15
Solve for the best transformation

solve with procrustes

16
Apply it …

17
and iterate!

18
and iterate!

19
and iterate!

20
and iterate!

21
and iterate!

22
Iterative Closest Point (ICP)
1. Initialize

23
Iterative Closest Point (ICP)
1. Initialize

2. Compute correspondences according to current best transform

24
Iterative Closest Point (ICP)
1. Initialize

2. Compute correspondences according to current best transform

3. Compute optimal transformation ( )with Procrustes

25
Iterative Closest Point (ICP)
1. Initialize

2. Compute correspondences according to current best transform

3. Compute optimal transformation ( )with Procrustes

4. Terminate if converged (error below a threshold), otherwise iterate

26
Iterative Closest Point (ICP)
1. Initialize

2. Compute correspondences according to current best transform

3. Compute optimal transformation ( )with Procrustes

4. Terminate if converged (error below a threshold), otherwise iterate

5. Converges to local minima
27
Is ICP the best we can do?
Iteration j:
• compute closest points

• compute optimal transformation with Procrustes

• apply transformation

• terminate if converged, otherwise iterate

28
Closest points
• Brute force is O(n2)
• For every source point find a neighbor point on the source shape

29
Closest points
• Tree based methods (e.g. kdtree) have avg. complexity log(n)

• Random point sampling also reduces the running time 30

ICP: Tips to avoid local minima
• Always find correspondences from target to source!
Proper data term
• Outliers —> Robust cost functions
• Use additional information (e.g. normals)
• Compute transformation based on greedy subsets of points: RANSAC

31
A much better objective: Point-to-surface
distance

y2 2 Y
<latexit sha1_base64="tZWfSSSaqlE76QU68gPmAB+fPSA=">AAACBHicbVDLSsNAFJ34rPUVddnNYBFclaT46LLgxmUF+5AmhMl00g6dTMLMRAghCzf+ihsXirj1I9z5N07aCNp64MLhnHu59x4/ZlQqy/oyVlbX1jc2K1vV7Z3dvX3z4LAno0Rg0sURi8TAR5IwyklXUcXIIBYEhT4jfX96Vfj9eyIkjfitSmPihmjMaUAxUlryzJoTIjXxgyzNvSZ0KIc/wl3umXWrYc0Al4ldkjoo0fHMT2cU4SQkXGGGpBzaVqzcDAlFMSN51UkkiRGeojEZaspRSKSbzZ7I4YlWRjCIhC6u4Ez9PZGhUMo09HVncaFc9ArxP2+YqKDlZpTHiSIczxcFCYMqgkUicEQFwYqlmiAsqL4V4gkSCCudW1WHYC++vEx6zYZ90Ti/Oau3W2UcFVADx+AU2OAStME16IAuwOABPIEX8Go8Gs/Gm/E+b10xypkj8AfGxzevzZgb</latexit>

y1 2 Y
<latexit sha1_base64="9ia0P+7dh12ye8rI1ceEXCaJDDU=">AAACBHicbVDLSsNAFJ3UV62vqMtuBovgqiTio8uCG5cV7EOaECbTSTt0MgkzEyGELNz4K25cKOLWj3Dn3zhpI2jrgQuHc+7l3nv8mFGpLOvLqKysrq1vVDdrW9s7u3vm/kFPRonApIsjFomBjyRhlJOuooqRQSwICn1G+v70qvD790RIGvFblcbEDdGY04BipLTkmXUnRGriB1maezZ0KIc/wl3umQ2rac0Al4ldkgYo0fHMT2cU4SQkXGGGpBzaVqzcDAlFMSN5zUkkiRGeojEZaspRSKSbzZ7I4bFWRjCIhC6u4Ez9PZGhUMo09HVncaFc9ArxP2+YqKDlZpTHiSIczxcFCYMqgkUicEQFwYqlmiAsqL4V4gkSCCudW02HYC++vEx6p037onl+c9Zot8o4qqAOjsAJsMElaINr0AFdgMEDeAIv4NV4NJ6NN+N93loxyplD8AfGxzeuOpga</latexit>

y0 2 Y
<latexit sha1_base64="mRy+IT2Eizo45Cpp6cu/op1+a3Y=">AAACBHicbVDLSsNAFJ3UV62vqMtuBovgqiTio8uCG5cV7EOaECbTSTt0MgkzEyGELNz4K25cKOLWj3Dn3zhpI2jrgQuHc+7l3nv8mFGpLOvLqKysrq1vVDdrW9s7u3vm/kFPRonApIsjFomBjyRhlJOuooqRQSwICn1G+v70qvD790RIGvFblcbEDdGY04BipLTkmXUnRGriB1maexZ0KIc/wl3umQ2rac0Al4ldkgYo0fHMT2cU4SQkXGGGpBzaVqzcDAlFMSN5zUkkiRGeojEZaspRSKSbzZ7I4bFWRjCIhC6u4Ez9PZGhUMo09HVncaFc9ArxP2+YqKDlZpTHiSIczxcFCYMqgkUicEQFwYqlmiAsqL4V4gkSCCudW02HYC++vEx6p037onl+c9Zot8o4qqAOjsAJsMElaINr0AFdgMEDeAIv4NV4NJ6NN+N93loxyplD8AfGxzesp5gZ</latexit>

32
Closest points: avoid local minima

Point-to-point distance

33
Closest points: avoid local minima

Point-to-surface distance

34
Is ICP the best we can do?
Iteration j:
• compute closest points

• compute optimal transformation with Procrustes

• apply transformation

• terminate if converged, otherwise iterate

35
Best transformation?
• Procrustes gives us the optimal rigid transformation and scale given
correspondences

• What if the deformation model is not rigid ?

• Can we generalise ICP to non-rigid deformation ?

36
Iterative Closest Point (ICP)
Iteration j:
• compute closest points è Which direction to move?

• compute optimal transformation with Procrustes

• apply transformation

• terminate if converged, otherwise iterate

37
Iterative Closest Point (ICP)
Iteration j:
• compute closest points è Which direction to move?

• compute optimal transformation with Procrustes è

Compute a transform that reduces the error
• apply transformation

• terminate if converged, otherwise iterate

38
Gradient-based ICP
Iteration j:
• compute closest points è Which direction to move?

• compute optimal transformation with Procrustes è

Compute descent step by linearising the energy
Jacobian of distance-based energy
• apply transformation

• terminate if converged, otherwise iterate

Gradient-based ICP

• If f is a rigid transformation we can solve this minimisation using

Procrustes

• If f is a general non-linear function ?

• Gradient descent:

• For least squares, is there a better optimisation method ?

yes: Gauss-Newton based methods.
40
Gradient-based ICP
1. Energy:

2. Consider the correspondences fixed in each iteration j+1

3. Compute gradient of the energy around current estimation

4. Apply step (gradient descent, dogleg, LM, BFGS…)

(for example )
5. terminate if converged, otherwise iterate (go to step 2)
41
Gradient-based ICP

42
Why is convergence on the left less smooth?

Point to point objective Point to surface objective

43
Gradient-based ICP
• Energy:

• Consider the correspondences fixed in each iteration j+1

• Compute gradient of the energy around current estimation

• Apply step (gradient descent, dogleg, LM, BFGS…)

• terminate if converged, otherwise iterate

44
Gradient-based ICP
• Gradient: derivative of the sum of squared
distances with respect to transformation f
parameters

45
Gradient-based ICP
• Gradient: derivative of the sum of squared
distances with respect to transformation f
parameters

• Each derivative is easy

• Who wants to writes it down?

46
Gradient-based ICP
• Gradient: derivative of the sum of squared
distances with respect to transformation f
parameters

• Each derivative is easy

• Who wants to writes it down?

• Chain rule and automatic differentiation!

47
Automatic differentiation

write as if it was numpy code

results in expression tree

with jacobians available at
each step

48
Gradient-based ICP
• Energy:

• Consider the correspondences fixed in each iteration j+1

• Compute gradient of the energy around current estimation

• Apply step (gradient descent, dogleg, LM, BFGS…)

• terminate if converged, otherwise iterate

49
Why Gradient-based ICP?
• Formulation is much more generic: the energy can incorporate other
terms, more parameters, etc
• A lot of available software for solving this least squares problem (cvx,
ceres, …)
• However, the resulting energy is non-convex for general deformation
models. Optimisation can get trapped in local minima.

50
Take-home message
• Procrustes is optimal for rigid alignment problems with known
correspondences. For other problems:

• We can compute correspondences and solve for the best

transformation iteratively with Iterative Closest Point (ICP)

51
Slide credits
• Javier Romero