Sigal Encyclopedia CVdraft
Sigal Encyclopedia CVdraft
Synonyms
Related Concepts
Definition
Human pose estimation is the process of estimating the configuration of the
body (pose) from a single, typically monocular, image.
Background
Human pose estimation is one of the key problems in computer vision that
has been studied for well over 15 years. The reason for its importance is the
abundance of applications that can benefit from such a technology. For example,
human pose estimation allows for higher level reasoning in the context of human-
computer interaction and activity recognition; it is also one of the basic building
blocks for marker-less motion capture (MoCap) technology. MoCap technology
is useful for applications ranging from character animation to clinical analysis
of gait pathologies.
Despite many years of research, however, pose estimation remains a very
difficult and still largely unsolved problem. Among the most significant chal-
lenges are: (1) variability of human visual appearance in images, (2) variability
in lighting conditions, (3) variability in human physique, (4) partial occlusions
due to self articulation and layering of objects in the scene, (5) complexity of
human skeletal structure, (6) high dimensionality of the pose, and (7) the loss
of 3d information that results from observing the pose from 2d planar image
projections. To date, there is no approach that can produce satisfactory results
in general, unconstrained settings while dealing with all of the aforementioned
challenges.
Theory and Application
Human pose estimation is typically formulated probabilistically to account
for ambiguities that may exist in the inference (though there are notable ex-
ceptions, e.g. [11]). In such cases, one is interested in estimating the posterior
Fig. 1. Skeleton Representation: Illustration of the 3d and 2d kinematic tree
skeleton representation on the left and right, respectively.
distribution, p(x|z), where x is the pose of the body and and z is a feature set
derived from the image. The key modeling choices that affect the inference are:
Next, the primary lines of research in pose estimation with respect to these
modeling choices are reviewed. It is worth noting that these three modeling
choices are not always independent. For example, some inference frameworks
are specifically designed to utilize a given representation of the pose.
The regression coefficients, A, can be learned easily from paired training samples,
D = {(xi , zi )}N
i=1 , using the least squares formulation (see [1] for details).
Parametric vs. Non-parametric: Parametric discriminative methods [1,6,12] are
appealing because the model representation is fixed with respect to the size
of the training dataset D. However, simple parametric models, such as Linear
Regression [1] or Relevance Vector Machine [1], are unable to deal with com-
plex non-linear relationships between image features and poses. Non-parametric
methods, such as Nearest Neighbor Regression [18] or Kernel Regression [18],
are able to model arbitrary complex relationships between input features and
output poses. The disadvantage of these non-parametric methods is that the
model and inference complexity are both functions of the training set size. For
example, in Kernel Regression,
N
X Kz (z, zi )
p(x|z) = Kx (x, xi ) PN , (2)
i=1 k=1 Kz (z, zk )
where Kx (·, ·) and Kz (·, ·) are kernel functions measuring the similarity of the
arguments (e.g., Gaussian kernels), the inference complexity is O(N ) (where N
is the size of the training dataset). More sophisticated non-parametric methods,
such as Gaussian Process Latent Variable Models (GPLVMs), can have even
higher complexity; GPLVMs have O(N 3 ) learning and O(N 2 ) inference com-
plexity. In practice, non-parametric methods tend to perform better but are
slower.
Dealing with ambiguities: If one assumes that p(x|z) is uni-modal [1], conditional
expectation can be used to characterize the plausible configuration of the person
in an image given the learned model. For example, for linear regression in Eq. (1),
E[x|z] = A[z − µz ] + µx ; for kernel regression in Eq. (2),
N
X Kz (z, zi )
E[x|z] = xi PN . (3)
i=1 k=1 Kz (z, zk )
In practice, however, most features under standard imaging conditions are am-
biguous, resulting in multi-modal distributions. Ambiguities naturally arise in
image projections, where multiple poses can result in similar, if not identical, im-
age features (e.g., front- and back-facing poses yield nearly identical silhouette
features). To account for these ambiguities, parametric mixture models were in-
troduced in the form of Mixture of Experts [6,12]. Non-parametric alternatives,
such as Local Gaussian Process Latent Variable Models (LGPLVM) [25], clus-
ter the data into convex local sets and make uni-modal predictions within each
cluster, or search for prominent modes in p(x|z) [15,22].
Learning: Obtaining the large datasets that are required for learning discrim-
inative models that can generalize across motions and imaging conditions is
challenging. Synthetic datasets often do not exhibit the imaging characteristics
present in real images, and real fully-labeled datasets are scarce. Furthermore,
even if large datasets could be obtained, learning from vast amounts of data is
not a trivial task [6]. To address this issue, two solutions were introduced: (1)
learning from small datasets by discovering an intermediate low dimensional la-
tent space for regularization [15,22] and (2) learning in semi-supervised settings,
where a relatively small dataset of paired samples is accompanied by a large
amount of unlabeled data [12,15,22].
Limitations: Despite popularity and lots of successes, discriminative methods
do have limitations. First, they are only capable of recovering a relative 3d
configuration of the body and not its position in 3d space. The reason for this
is practical, as reasoning about position in 3d space would require prohibitively
large training datasets that span the entire 3d volume of the space visible from
the camera. Second, their performance tends to degrade as the distributions of
test and training data start to diverge; in other words, generalization remains
one of the key issues. Lastly, learning discriminative models efficiently from large
datasets that cover wide range of realistic activities and postures remains a
challenging task.
Inference (generative): Alternatively, one can take a generative approach and
express the desired posterior, p(x|z), as a product of a likelihood and a prior:
Among the earliest successes along this line of research is the work of Lee and
Cohen [13]. Their approach focused on obtaining proposal maps for the locations
of individual joints within an image. These proposal maps were obtained based
on a number of features that were computed densely over the image. For example,
face detection was used to obtain hypotheses for the location of the head; head-
shoulder contour matching, obtained using a deformable contour model and
gradient descent, was used as evidence for shoulder joint locations; elliptical
skin regions, obtained using skin-color segmentation, were used to determine
the locations of the lower arms and lower legs. In addition, second-derivative
(ridge) observations were used as evidence for other limbs of the body. Given
proposals for the different joints, weighted by the confidence of corresponding
detectors, a data-driven Markov Chain Monte Carlo (MCMC) approach was
used to recover 3d configurations of the skeleton. This inference relied on direct
inverse kinematics (IK) obtained from 2d proposal maps. To further improve the
results, a kinematic jump proposal process was also introduced. The kinematic
jump proposal process involves flipping a body part or a set of parts (i.e., the
head, a hand, or an entire arm) in the depth direction around its pivotal joint.
Other part-based approaches try to assemble regions of an image into body
parts and successively construct those parts into a body. Prime examples of such
methods are introduced by Mori et al . [14] and Ren et al . [17]. In [14] super-pixels
were first assembled into body parts based on the evaluation of low-level image
cues, including contour, shape, shading, and focus. The part-proposals were then
pruned and assembled together using length, body part adjacency, and clothing
symmetry. A similar approach was taken in [17], but line segments were used
instead of assembling super-pixels. Parallel lines were assembled into candidate
parts using a set of predefined rules, and the candidate parts were in turn as-
sembled into the body with a set of joint, scale, appearance, and orientation
consistency constraints. Unlike [14], the search for the most probable body con-
figurations was formulated as a solution to an Integer Quadratic Programming
(IQP) problem.
The most traditional and successful approach, however, is to represent the
body using a Markov Random Field (MRF) with body parts corresponding to
the nodes and constraints between parts encoded by potential functions that
account for physical and statistical dependencies (see Figure 3). Formally, the
posterior, p(x|z), can be expressed as:
p(x|z) ∝ p(z|x)p(x)
= p(z|{x1 , x2 , ..., xM })p({x1 , x2 , ..., xM })
M
Y Y
≈ p(z|xi ) p(x1 ) p(xi , xj ) . (6)
i=1 (i,j)∈E
| {z }| {z }
likelihood prior
In this case, pose estimation takes the form of inference in a general MRF net-
work. The inference can be solved efficiently using message-passing algorithms,
such as Belief Propagation (BP). BP consists of two distinct phases: (1) a set of
message-passing iterations are executed to propagate consistent part estimates
within a graph, and (2) marginal posterior distributions are estimated for every
body part [2,8,16]. A typical formulation looks at the configuration of the body
in the 2d image plane and assumes discretization of the pose for each individual
part, e.g., xi = {τi , θi , si } where τi ∈ R2 is the location and θi ∈ R1 and si ∈ R1
are orientation and scale of the part i (represented as a rectangular patch) in the
image plane. As a result, the inference is over a set of discrete part configurations
li ∈ Z (for part i), where Z is the enumeration of poses for a part in an image (li
is a discrete version of xi ). With an additional assumption of pair-wise potentials
that account for kinematic constraints, the model forms a tree-structured graph
known as the Tree-structured Pictorial Structures (PS) model. An approximate
inference with continuous variables is also possible [20,21].
Inference in the tree-structured PS model first proceeds by sending recur-
sively defined messages of the form:
X Y
mi→j (lj ) = p(li , lj )p(z|li ) mk→i (li ), (7)
li k∈A(i)\j
where mi→j is the message from part i to part j, with p(li , lj ) measuring the
compatibility of poses for the two parts and p(z|li ) the likelihood, and A(i) \ j is
the set of parts in the graph adjacent to i except for j. Compatibility, p(li , lj ), is
often measured by the physical consistency of two parts at the joint, or by their
statistical (e.g., angular) co-occurrence with respect to one another. In a tree-
structured PS graph, these messages are sent from the outermost extremities
inward and then back outward.
Once all of the message updates are complete, the marginal posteriors for all
of the parts can be estimated as:
Y
p(li |z) ∝ p(z|li ) mj→i (li ). (8)
j∈A(i)
Similarly, the most likely configuration can be obtained as a MAP estimate:
One of the key benefits of the Pictorial Structures (PS) paradigm is its
simplicity and efficiency. In PS exact inference is possible in the time linear
to the number of discrete configurations a given part can assume. Because of
this property, recent implementations [2] can handle the pixel-dense configura-
tions of parts that result in millions of potential discrete states for each body
part. The linear complexity comes from the observation that a generally com-
plex non-Gaussian prior over neighboring parts, p(xi , xj ), can be expressed as
a Gaussian prior over the transformed locations corresponding to joints, mainly
p(xi , xj ) = N (Tij (xi ); Tji (xj ), Σij ). This is done by defining a transformation,
Tij (xi ), that maps a common joint between parts i and j, defined in the part i’s
coordinate frame, to its location in the image space. Similarly, Tji (xj ) defines the
transformation from the same common joint defined in the j’s coordinate frame
to the location in the image plane. This transformation allows the inference to
use an efficient solution that involves convolution (see [8] for more details).
Performance: Recently, it has been shown that the effectiveness of a PS model
is closely tied to the quality of the part likelihoods [2]. Discriminatively trained
models [16] and more complex appearance models [2] tend to outperform models
defined by hand [8]. Methods that learn likelihood cascades, corresponding to
better and better features tuned to a particular image, have also been explored
for both superior speed and performance [16]. Most recent discriminative formu-
lation of PS model allows joint learning of part appearances and model structure
[26] using structural Support Vector Machine (SVM).
Speed: Cascades of part detectors serve not only to improve performance, but
also to speed up the inference (e.g., [23]). Fast likelihoods can be used to prune
away large parts of the search space before applying more complex and computa-
tionally expensive likelihood models. Other approaches to speed up performance
include data-driven methods (e.g., data-driven Belief Propagation). These meth-
ods look for the parts in an image first and then assemble a small set of the part
candidates into the body (akin to the methods of [14,17]). The problem with
such approaches is that any occluded parts are missed altogether because they
cannot be detected by the initial part detectors. Inference can also be sped up
by using progressive search refinement methods [9]. For example, some methods
use upper body detectors to restrict the search to promising parts of the image
instead of searching the whole image.
Non-tree structured extensions: Although tree-structured PS models are compu-
tationally efficient and exact, they generally are not sufficient to model all the
necessary constraints imposed by the body. More complex relationships among
the parts that fall outside of the realm of these models include non-penetration
constraints and occlusion constraints [20]. Incorporating such relationships into
the model adds loops corresponding to long-range dependencies between body
parts. These loops complicate inference because: (1) no optimal solutions can
be found efficiently (message passing algorithms, like BP, are not guaranteed to
converge in loopy graphs) and (2) even approximate inference is typically com-
putationally expensive. Despite these challenges, it has been argued that adding
such constraints is necessary to improve performance [4]. To alleviate some of the
inference complexities with these non-tree-structured models, a number of com-
peting methods have been introduced. Early attempts used sampling techniques
from the tree-structured posterior as proposals for evaluation of a more complex
non-tree-structured model [7,8]. To obtain optimality guarantees, branch-and-
bound search was recently proposed by Tian et al . [24], with the tree-structured
solutions as a lower bound on the more complex loopy model energy.
Open problems
Despite much progress in the field, pose estimation remains a challenging
and still largely unsolved task. Progress has been made in estimating the con-
figurations of mostly unoccluded and isolated subjects. Open problems include
dealing with multiple, potentially interacting people (e.g., [7]), and tolerance to
unexpected occlusions. Future research is also likely to expand on the types of
postures and imaging conditions that the current algorithms can handle.
To date, most successful pose estimation approaches have been bottom-up.
This observation applies to both discriminative approaches and part-based ap-
proaches. However, it seems short-sighted to assume that the general pose esti-
mation problem can be solved purely in a bottom-up fashion. Top-down infor-
mation may be useful for enforcing global pose consistency, and a combination of
top-down and bottom-up inference is likely to lead to success faster. The recent
success of combining bottom-up part-based models with 3d top-down priors [3]
is encouraging and should be built upon to produce models that can deal with
more complex postures and motions. Earlier attempts at building hierarchical
models [27] may also be worth revisiting with the newfound insights.
Finally, there is significant evidence suggesting that successfully estimating
pose independently at every frame is a very ill-posed problem. Spatio-temporal
models that aggregate information over time [3] are emerging as a way to reg-
ularize performance obtained in individual frames and smooth out the noise in
the estimates. Leveraging all sources of generic prior knowledge, such as spatial
layout of the body and temporal consistency of poses, and rich image observation
models is critical in advancing the state-of-the-art.
Recommended Readings