Dictionary Learning in Games - GDC 2014

Dictionary Learning for Games
Manny Ko
Principal Engineer, Activision R&D
Graphics Research and Development

Outline
● K-SVD and dictionary learning
● Linear Blend Skinning
● Brief survey on automatic skinning and compression
● Dictionary learning for LBS
● Two-layer sparse compression of Le & Deng.
● This talk is about compressing skinned animations.

Frames, Sparsity and Global Illumination:
New Math for Games
GDC 2012
Robin Green – Microsoft Corp
Manny Ko – PDI/Dreamworks

Orthogonal Matching Pursuit
and K-SVD for Sparse Encoding
Manny Ko
Senior Software Engineer, Imaginations Technologies
Robin Green
SSDE, Microsoft Xbox ATG

Representing Signals
● We represent signals as linear combinations of things we already know –
the ‘basis’
× 𝛼1 +
× 𝛼2 +
× 𝛼3 + ⋯
=
× 𝛼0 +

Orthonormal Bases (ONBs)
● The simplest way to represent signals is using a set of orthonormal bases
𝑏𝑖 𝑡 𝑏𝑗(𝑡)
+∞
−∞
𝑑𝑡 =
0 𝑖 ≠ 𝑗
1 𝑖 = 𝑗

Example ONBs
● Fourier Basis
𝑏 𝑘 𝑡 = 𝑒 𝑖2𝑝𝑘𝑡
● Wavelets
𝑏 𝑚,𝑛 𝑡 = 𝑎−𝑚 2 𝑥 𝑎−𝑚 𝑡 − 𝑏𝑚
● Gabor Functions
𝑏 𝑘,𝑛 𝑡 = 𝜔 𝑡 − 𝑏𝑛 𝑒 𝑖2𝑝𝑘𝑡
● Contourlet
𝑏𝑗,𝑘,𝐧 𝑡 = λ𝑗,𝑘 𝑡 − 2 𝑗−1 𝐒 𝑘n

Benefits of ONB
● Analytic formulations
● Well understood mathematical properties
● Fast and simple algorithms for projection

Problems with ONB
● One-size-fits all – not data adaptive
● Global support cannot adapt to data locally
● Fourier support is infinite, SH support spans the sphere
● Try using Fourier to represent a step-function
● Not sparse – very few zero coefficients
● Not additive - relies on destructive cancellation.

Gibb’s Ringing – Fourier and SH

What is Overcomplete Dictionary?
● Overcomplete means the dictionary has more atoms
(columns) than the minimum required for the
dimension of the signal
● In 3D, an ONB only needs 3 basis
● A 3D dictionary can have dozens or hundreds

The Sparse Signal Model
𝐃
A fixed dictionary
𝛼
=
𝑥
𝑁 𝑁
𝐾
resulting
signal
Sparse
vector of
coefficients

Why so many atoms?
● More atoms give our algorithm a better chance to
find a small subset that matches a given signal
● Let’s look at some patches from Barbara

Dictionary Learning in Games - GDC 2014

Domain Specific Compression
● Just 550 bytes per image
1. Original
2. JPEG
3. JPEG2000
4. PCA
5. KSVD per block

Project onto Dictionaries
● Overcomplete and non-orthogonal
● interactions among atoms cannot be ignored
● How do we project?
● Sparse Coding problem

Matching Pursuit
1. Set the residual 𝑟 = 𝑥
2. Find an unselected atom
that best matches the
residual 𝐃𝛼 − 𝑟
3. Re-calculate the residual
from matched atoms
𝑟 = 𝑥 − 𝐃𝛼
4. Repeat until 𝑟 ≤ 𝜖
Greedy Methods
𝐃
𝛼
=
𝑥

Orthogonal Matching Pursuit (OMP)
● Add an Orthogonal Projection to the residual calculation
1. set 𝐼 ∶= ∅ , 𝑟 ≔ 𝑥, 𝛾 ≔ 0
2. while (𝑠𝑡𝑜𝑝𝑝𝑖𝑛𝑔 𝑡𝑒𝑠𝑡 𝑓𝑎𝑙𝑠𝑒) do
3. 𝑘 ≔ argmax
𝑘
𝑑 𝑘
𝑇
𝑟
4. 𝐼 ≔ 𝐼, 𝑘
5. 𝛾𝐼 ≔ 𝐃𝐼
+ 𝑥
6. 𝑟 ≔ 𝑥 − 𝐃𝐼 𝛾𝐼
7. end while

What is Dictionary Learning?
● select a few atoms for each signal – e.g. OMP
● Adjust the atoms to better fit those signals
● Repeat

K-SVD
● Is one of the well known dictionary learning
methods
● Check out our GDC2013 talk
● our GDC13 slides "OMP and K-SVD for Sparse Coding“
● See Jim’s talk just before this session
● Miral’s Online Learning is the other.

Overcomplete Dictionary Recap
● Importance of overcomplete dictionaries
● OMP for efficient projection onto dictionaries
● K-SVD for learning a better dictionary using samples
from the real data

Linear Blend Skinning
● 𝑣𝑖 = 𝑤𝑖𝑗(𝑅𝑗
|𝐵|
𝑗=1 𝑝𝑗 + 𝑇𝑗)
● 𝑝𝑖 is the position for the 𝑖th vertex of the rest pose
● 𝑤𝑖𝑗 ≥ 0 𝑎𝑛𝑑 𝑠𝑢𝑚𝑠 𝑡𝑜 𝑜𝑛𝑒(affinity). The non-negative
constraint makes the blend additive. The affinity
constraint prevents over-fitting and artifacts.
● 𝑅𝑗 usually is orthogonal to avoid shearing or scaling
● |𝐵| is the number of weights (usually <= 6)

Blend Skinning on GPU
GPU
cores

LBS on GPUs
● 𝑤𝑖𝑗 typically very sparse – 4-6 weights or less per-
vertex
● Ideally a group of vertices all have the same weights
to avoid thread divergence or splitting drawcalls
● These are fairly serious constraints
a) Some vertices might need more weights – e.g. very
smooth meshes or complex topology (hand)

WeightsReduction
Poisson-based Weight Reduction of Animated Meshes [Landreneau and Schaefer 2010]
 Discrete optimization:
– Impossible to find optimum solution
– Very high cost for non-optimum solution
• Fracture
• Significant increase of computing cost: nK non-zero  n(K+1) non-zero

Magic 4
● why 4 weights is too few to generate smooth
weights
● 4 vertices specifies an affine transform exactly.
● simplices in 3D contains 4 vertices for barycentric
coordinates.

Two-LayerSparseCompressionof
Dense-WeightBlendSkinning
BinhLe and ZhigangDeng
SIGGRAPH2013

Two-Layer Sparse Compression, Le & Deng 2013
● Use dictionary learning to compute a two-level
compression using bones
● Work with the weights of the bind-pose directly

Why Dictionary for LBS?
● Why dictionary learning?
● limitations of Orthonormal-basis e.g. eigen/PCA
● Not adaptive
● Not purely additive – i.e. negative weights (relies on cancellation)
● No intuitive meaning – bones extracted cannot be used to tweak the
model

Dense-WeightCompression
Input: Dense matrix
Bone Transformations
Blending
Vertices
Vertices
Bones

SparseMatrixFactorization– dictionarylearning

SparseMatrixFactorization
c=max{card(wi)}+1

SparseMatrixFactorization
c=max{card(wi)}+1
n is very large
card(A)=2n→min

Algorithm– Blockcoordinatedescent
Alternative update D and A
(Block coordinate descent)
Update D Update A

UpdateCoefficientsA
Linear least square with 2 unknowns
Use mesh smoothness
assumption to quickly
find the non-zero
candidates (virtual bones)
αi

Analysis of Two-Layer Scheme
● Use 100’s of virtual bones means we are not limited to a
sparse approximation to the original animation.
● virtual bones act as a ‘common subexpression’
● e.g. think compute shader that writes to LDS.
● Still enforce sparsity on VBs to control runtime cost and
LDS usage – but k can be 100’s.
● Per-vertex weights are
● very sparse (2 per vertex) and the same for all vertices
● good for GPU.

Learning Virtual Bones
● Virtual bones are learned from the dense vertex weights
by block-coordinate-descent (BCD):
Sparse coding: search for a few good atoms among the
input columns. Use that to project all the rest of the inputs.
● Atom update: given the sparse weights from above we
seek to adjust the atoms to make them fit the inputs that
needs them better – a series of small LS problems.
● Similar to EM/Lloyd-Max

Sparse Coding
Sparse coding:
● insert the vertex with the largest L2 norm
● add a few more vertex which has the smallest dot-
product with the 1st atom
● solve the basis-pursuit with OMP (see K-SVD) or LARS.
● solve 2x2 least-square prob. for 𝑤𝑖𝑗 to blend masters
bones

Weight Map – matrix A
● Weights and indices for each vertex to blend virtual
bones
● solving a small 2x2 linear system to minimize MSE:
● arg 𝑚𝑖𝑛 𝑥 𝐷𝑥 − 𝑤𝑖 ^2
● runtime per-vertex cost is just 2 dotp
● no bone hierarchy to worry about
● no warp divergence even for high valence vertices

Atom Updates
Atom update:
foreach vertex
● update each atom to minimize error for the set of vertices that
reference it (this is like K-SVD)
● Miral’s Online Dictionary Learning [Miral09]

Atom Updates
● Precompute A and B
● 𝐴 = 𝛼𝑖
𝑡
𝑖=1 𝛼 𝑇
● B = 𝑥𝑖 𝛼 𝑇𝑡
𝑖=1
● For all atoms
● 𝑢𝑗
1
𝐴 𝑗,𝑗
𝑏𝑗 − 𝐷𝑎𝑗 + 𝑑𝑗 − eq(5)
● 𝑑𝑗
1
max 𝑢 𝑗 2,1
𝑢𝑗. − eq 6
● 𝑢𝑗is thresholded to make sure # of non −
zero is below the # of master bones

CompressionwithExamplePoses
Without using example pose
– Minimize weights difference
With using example poses
– Minimize reconstruction error

Recap
● The two-level scheme can work with dense (hand painted)
weights or example poses (blend shape?)
● Only the vertex positions are needed
● a fixed memory footprint and uniform per-vertex cost - GPU
friendly
● Combines the quality of dense skinning and the efficiencies of
sparse-LBS. Animators can use blend-shapes or FFD more.

Recap 2
● Besides it uses dictionary learning and modern
sparsity methods – how cool is that? 
● Last year we show how good dictionary learning is
for compressing 2d images and 3d volumes
● Now we see what it can do for animation.
● Thank you!

Recap 3
● Non-negative LS and Active-set Method (ASM)
● Block-coordinate descent
● Sparsity constraints
● L1 relaxation and L0-norm constraints
● Direct solving
● These are all very useful tools.

Acknowledgements
● Binh Huy Le & Zhigang Deng kindly provided the demo and their Siggraph
materials.
● Robin Green for being my collaborator for many years.
● Igor Carron inspired me to learn sparsity methods and matrix factorization
and for his spirit of broad exploration and sharing.
● Julien Mairal for the online learning math
● Peter-Pike who inspired me to apply modern math to graphics and games.
● Carlos Gonzalez Ochoa for sharing his insight in animation.

Activision R&D is Hiring
● Our group is hiring 

References
● Alexa 2000. “As-rigid-as-possible shape interpolation”, SIGGRAPH 2000.
● Halser 2010. “Learning skeletons for shape and pose”, I3D 2010.
● Kavan, Sloan and O'Sullivan 2010. “Fast and Efficient Skinning of Animated Meshes” Comput. Graph.
Forum.
● Ko, and Green 2013 “Orthogonal Matching Pursuit and K-SVD for Sparse Encoding” GDC, Math for
Games 2013 gdc2013-ompandksvd
● Landreneau & Schaefer “Poisson-Based Weight Reduction of Animated Meshes”, CGF 28(2), 2012.
● Le & Deng 2012. “Smooth skinning decomposition with rigid bones”, ACM TOG, Vol. 31, No. 6.
● Le & Deng 2013. “Two-Layer Sparse Compression of Dense-Weight Blend Skinning”, Siggraph 2013
Paper page
● Mairal 2009. “Online dictionary learning for sparse coding” Int. Conf. on Machine Learning.

Appendix
● Kabsch/Procrutes method – use SVD to compute the
MSE minimum rotation of one point-set to another.
● Kabsch_algorithm

Dictionary Learning in Games - GDC 2014

More Related Content

What's hot (15)

Viewers also liked (19)

Similar to Dictionary Learning in Games - GDC 2014 (20)

Recently uploaded (20)

Dictionary Learning in Games - GDC 2014