Object Co-skeletonization with Co-segmentation

Koteswar Rao Jerripothula1,2 , Jianfei Cai2 , Jiangbo Lu3,4 and Junsong Yuan2
1 2
Graphic Era University, India Nanyang Technological University, Singapore
3 4
Advanced Digital Sciences Center, Singapore Shenzhen Cloudream Technology, China
[email protected], {asjfcai,jsyuan}@ntu.edu.sg, [email protected]

Abstract

Recent advances in the joint processing of images have Source images

certainly shown its advantages over the individual process-
ing. Different from the existing works geared towards co-
segmentation or co-localization, in this paper, we explore Co-skeletonization Co-segmentation

a new joint processing topic: co-skeletonization, which is

defined as joint skeleton extraction of common objects in
a set of semantically similar images. Object skeletoniza-
tion in real world images is a challenging problem, because
there is no prior knowledge of the object’s shape if we con-
sider only a single image. This motivates us to resort to the
Joint Optimization
idea of object co-skeletonization hoping that the common-
ness prior existing across the similar images may help, just Skeletons Segmentations
as it does for other joint processing problems such as co-
segmentation. Noting that skeleton can provide good scrib- Figure 1. Object co-skeletonization with co-segmentation. Skele-
bles for segmentation, and skeletonization, in turn, needs tons are in yellow.
good segmentation, we propose a coupled framework for
co-skeletonization and co-segmentation tasks so that they
learn [25, 20] to perform skeletonization on test images.
are well informed by each other, and benefit each other
The recent deep learning based method [22] requires not
synergistically. Since it is a new problem, we also con-
only the skeleton location information but also the skeleton
struct a benchmark dataset for the co-skeletonization task.
scale information that accounts for shape information. The
Extensive experiments demonstrate that proposed method
skeleton scale is basically the distance between a skeleton
achieves very competitive results.
point and the nearest boundary point of the object.
In contrast, in this paper we consider the skeletoniza-
tion problem with weak supervision, i.e. co-skeletonization,
1. Introduction
which does not need pre-segmentation or groundtruth
Our main objective in this paper is to exploit joint pro- skeletons of training images. Particularly, we leverage
cessing [30, 13, 6] to extract objects’ skeletons in images the existing idea of object co-segmentation to help co-
of the same category. We call it object co-skeletonization. skeletonization. It turns out that co-skeletonization can also
By objects, we mean something which interests the im- help co-segmentation in return by providing good scribbles.
age viewer more compared to the stuff like sky, roads, In this way, both co-skeletonization and co-segmentation
mountains, sea, etc, in its presence. Automatic skele- benefit each other synergistically. We couple these two
tonization of such objects has many applications such as tasks to achieve what we call “Object Co-skeletonization
image search, image synthesis, generating training data with Co-segmentation” as shown in Fig. 1.
for object detectors, etc. However, it is difficult to solve There are several challenges involved in performing
this problem as a standalone task, because it requires ob- co-skeletonization and the coupling with co-segmentation.
jects shape information as well. Existing methods either First, existing skeletonization algorithms [21, 17, 3, 19]
need pre-segmentation [3, 21] of the object in the im- can yield a good skeleton if a good and smooth shape is
age or groundtruth skeletons for the training images to provided, but they are quite sensitive to the given shape,

6205
 (a) (b) (c)

(d) (e)
Figure 3. Inherent interdependencies of co-skeletonization and co-
Figure 2. Example challenges of co-skeletonization. The quality segmentation can be exploited to achieve better results through a
of segmentation affects the quality of skeletonization. (b) The re- coupled iterative optimization process.
sult of [21] for (a). (c) Our result. Skeletons lie on homogeneous
regions, such as in (d) and (e), which are difficult to be detected
achieves state-of-the-art co-skeletonization performance in
and described.
the weakly supervised setting.

as shown for the image in Fig. 2(a) which has unsmooth 2. Related Work
segmentation. The skeleton produced by [21] in Fig. 2(a) Skeletonization: The research on skeletonization can
has too many unnecessary branches, while a more desirable be divided into three categories. First, there are some al-
skeleton to represent the cheetah would be the one obtained gorithms [17, 3, 19] which can perform skeletonization if
by our method in Fig. 2(c). Thus, the quality of the provided the segmentation of an object is given. Generally, these al-
shape becomes crucial, which is challenging for the con- gorithms are quite sensitive to the distortions of the given
ventional co-segmentation methods because their complex shape. However, this problem can be tackled through re-
way of co-labeling many images may not provide good and cent methods such as [21]. Second, there are also some tra-
smooth shapes. Second, joint processing of skeletons across ditional image processing methods [28, 29, 11] which can
multiple images is quite tricky. Because most of the skele- generate skeletons by exploiting gradient intensity maps.
ton points generally lie on homogeneous regions as shown They generate skeletons even for stuffs like sky, sea, etc,
in Fig. 2(d) and (e), it is not easy to detect and describe which usually need some object prior to be suppressed.
them for the purpose of matching. Third, how to couple the Third, there are also some supervised learning based meth-
two tasks so that they can synergistically assist each other ods which require groundtruth skeletons of training images
is another challenge. for learning. This class of methods includes both tradi-
Our key observation is that we can exploit the inher- tional machine learning based methods [25, 20] and the
ent interdependencies of two tasks to achieve better results recent deep learning based methods [27, 22]. The perfor-
jointly. For example, in Fig. 3, although the initial co- mance of the traditional machine learning based methods is
segmentation produces a poor result, most of the skeleton not satisfactory due to the limited feature learning capabil-
pixels still remain on the horse, which gradually improve ity in homogeneous regions. On the other hand, the recent
the segmentation by providing good seeds for segmentation deep learning based methods have made great progress in
in the subsequent iterations of joint processing. In turn, co- the skeletonization process as reported in [22] at the cost of
skeletonization also becomes better as the co-segmentation requiring complex training process on a substantial amount
improves. Our another observation is that we can exploit of annotated data. In contrast, our method is a weakly su-
the structure-preserving quality of dense correspondence to pervised one, although it can utilize the annotated data as
overcome the skeleton matching problem. well, if available.
To the best of our knowledge, there is only one dataset Segmentation: Image segmentation is a classical prob-
where co-skeletonization could be performed in a weakly lem, and there are many types of approaches like interac-
supervised manner, i.e. WH-SYMMAX dataset [20], and tive segmentation [15, 24], image co-segmentation [4, 7, 5],
it only contains horse images. To extensively evaluate semantic segmentation [18], etc. While interactive segmen-
co-skeletonization, we construct a new benchmark dataset tation needs human efforts, image co-segmentation exploits
called CO-SKEL dataset, which consists of images rang- weak supervision in the form of requiring the association of
ing from animals, birds, flowers to humans with total 26 same category images and uses an inter-image prior to help
categories. Extensive experiments show that our approach segment each individual image. Semantic image segmenta-

6206
tion not only segments objects but also provides a label for skeleton pruning problem and can be solved using the ap-
each pixel. In the past few years, deep learning based meth- proach similar to [21], where branches in the skeleton are
ods such as fully convolution networks (FCN) have greatly iteratively removed as long as it reduces the energy. Sim-
o
advanced the performance of semantic image segmentation. ilarly, if we combine both the inter-image prior ψpr and
o
Recently, [10] proposed a joint framework to combine in- the skeleton prior ψin as the data term, (3) become a stan-
teractive segmentation with FCN based semantic segmen- dard MRF-based segmentation formulation, which can be
tation [18] so as to help each other. In a similar spirit, in solved using GrabCut [15]. Thus, compared with the ex-
this work, we propose coupling of co-skeletonization and isting works, the key differences of our formulation lie in
co-segmentation to assist each other. the designed inter-image prior terms as well as the interde-
pendence terms, which link the co-skeletonization and co-
3. Proposed Method segmentation together.
Iteratively solving (2) and (3) requires a good initial-
In this section, we discuss our joint framework of co-
ization. We propose to initialize O by Otsu thresholded
skeletonization and co-segmentation in detail.
saliency maps and K by the medial axis mask [3]. Alg. 1
3.1. Overview of Our Approach summarizes our approach, where (ψpr + ψin + ψsm )(t) de-
notes the objective function value of (1) at the tth iteration
Given a set of m similar images belonging to the same k o k o k o
and ψpr = ψpr +ψpr , ψin = ψin +ψin , ψsm = ψsm +ψsm .
category, denoted by I = {I1 , I2 , · · · , Im }, we aim to
provide two output sets: K = {K1 , K2 , · · · , Km } and
O = {O1 , O2 , · · · , Om }, comprising skeleton masks and Algorithm 1: Our approach for solving (1)
segmentation masks, respectively, where Ki (p), Oi (p) ∈ Data: An image set I containg images of the same
{0, 1} indicating whether a pixel p is a skeleton pixel category
(Ki (p) = 1) and whether it is a foreground pixel (Oi (p) = Result: Sets O and K containing segmentations and
1). skeletons of images in I
Our overall objective function for an image Ii is defined (0)
Initialization: ∀Ii ∈ I, Oi = Otsu thresholded
as (0) (0)
saliency map and Ki = ma(Oi );
min λψpr (Ki , Oi |Ni ) + ψin (Ki , Oi |Ii ) + ψsm (Ki , Oi |Ii ) Process: ∀Ii ∈ I,
Ki ,Oi
do
s.t. Ki ⊆ ma(Oi ) 1) Obtain Oi
(t+1)
by solving (3) using [15] with
(1) (t) (t)
where the first term ψpr accounts for the priors from the set O and Ki .
(t+1)
of neighbor images denoted as Ni , the second term ψin is 2) Obtain Ki by solving (2) using [21] with
(t) (t+1) (t+1) (t+1)
to enforce the interdependence between the skeleton Ki and K and Oi , s.t. Ki ∈ ma(Oi ).
the shape / segmentation Oi in image Ii , the third term ψsm while
is the smoothness term to enforce smoothness, and λ is a (λψpr + ψin + ψsm )(t+1) ≤ (λψpr + ψin + ψsm )(t) ;
parameter to control the influence of the inter-image prior O ← O(t) and K ← K(t)
term. The constraint in (1) means the skeleton must be a
subset of medial axis (ma) [3] of the shape.
We resort to the typical alternative optimization strategy 3.2. Object Co-skeletonization
to solve (1), i.e., dividing (1) into two sub-problems and
solve them iteratively. In particular, one sub-problem is as As shown in Alg. 1, the step of object co-skeletonization
follows. Given the shape Oi , we solve co-skeletonization is to obtain K (t+1) by minimizing (2), given the shape
by O(t+1) and the previous skeleton set Kt . Considering the
(t+1) (t+1)
constraint of Ki ∈ ma(Oi ), we only need to
k k k
min λψpr (Ki |Ni ) + ψin (Ki |Oi ) + ψsm (Ki ) search skeleton pixels from the medial axis pixels. We build
Ki
(2) up our solution based on [21], but with our carefully de-
s.t. Ki ⊆ ma(Oi ). signed individual terms for (2) as explained below.
k
The other sub-problem is that given the skeleton Ki , we Prior Term (ψpr ): In the object co-skeletonization, a
solve co-segmentation by good skeleton pixel will be the one which is repetitive
o o o
across images. To account for this repetitiveness, we need
min λψpr (Oi |Ni ) + ψin (Oi |Ki , Ii ) + ψsm (Oi |Ii ). (3) to find corresponding skeleton pixels in other images. How-
Oi
ever, skeleton pixels usually lie on homogeneous regions
k
If we treat both the inter-image prior term ψpr and the shape (see Fig. 2(d)&(e)) and are thus difficult to match. Thus,
k
prior term ψin as a combined prior, (2) turns out to be a instead of trying to match sparse skeleton pixels, we make

6207
 Hump?

Missing parts

Source Shape & Reconstructed

image medial axis shape & skeleton Leg?

Figure 5. Shape reconstruction from skeleton. Compared to the

reconstructed shape from the medial axis (2nd column), the recon-
structed shape (3rd column) from our simplified skeleton is sim-
pler and smoother while still preserving the main structure. Nev-
ertheless, we do not want an over-simplified skeleton, which will
result in missing important parts in the corresponding shape recon-
struction (4th column).

each pixel p to account for the rough skeleton alignment

across the images.
k
Interdependence Term (ψin ): Our interdependence
(a) (b) (c)
term is similar to the traditional data term in skeleton prun-
Figure 4. Dense correspondences preserve the skeleton and seg- ing, i.e., it enforces that the skeleton should provide a good
mentation structures roughly. Here (a) is warped to generate (b) to reconstruction of the given shape, which medial axis al-
be used as a prior for (c). ready does well. However, a medial axis often contains
spurious branches, while the noisy shapes obtained from
imperfect co-segmentation only make this worse. To avoid
use of dense correspondences using SIFT Flow [12], which spurious branches, we prefer a simplified skeleton, whose
preserve the skeleton and segmentation structures well, as reconstructed shape is expected to be smooth while still pre-
shown in Fig. 4. serving the main structure of the given shape (see Fig. 5
Once correspondence is established, we utilize the for example). On the other hand, we do not want an over-
warped skeleton pixels from neighboring images to develop simplified skeleton, whose reconstructed shape is likely to
the prior term. Particularly, we align all the neighboring im- miss some important parts (see the 4th column of Fig. 5).
ages’ tth iteration’s skeleton maps to the concerned image Therefore, we expect the reconstructed shape from the
Ii , and generate a co-skeleton prior at the (t + 1)th iteration skeleton to match the given shape, but not necessary to be
as exactly the same as the given shape. In this spirit, we define
(t) P (t) k
Ki + Wji (Kj ) our interdependence term ψin as
Ij ∈Ni
e (t+1) =
K (4)
i
|Ni | + 1 k |R(Ki , Oi ) ∩ Oi |
ψin (Ki |Oi ) = −α log (6)
|R(Ki , Oi ) ∪ Oi |
where we align other skeleton maps using a warping func-
tion Wji [12] and then average them with Ii ’s own skele- where we use IoU to measure the closeness between the re-
ton map. Note that the neighborhood Ni is developed sim- constructed shape R(Ki , Oi ) and the given shape Oi , and
ply based on the GIST distance [14]. For simplicity, we α is the normalization factor as defined in [21]. The recon-
drop the superscriptions such as (t + 1) in all the following structed shape R(Ki , Oi ) is basically the union of maximal
derivations. disks at skeleton pixels [21], i.e.,
Considering that the corresponding skeleton pixels from [
other images may not exactly align with the skeleton pixels R(Ki , Oi ) = d(p, Oi ) (7)
of the considered image, we define our inter-image prior p∈ma(Oi )
term as
X  X  where d(p, Oi ) denotes the maximal disk at skeleton pixel
k
ψpr (Ki |Ni ) = −Ki (p) log 1 + Ke i (q) . p for the given Oi , and the maximal disk is the disk that
p∈ma(Oi ) q∈N(p) exactly fits within Oi with skeleton pixel p as the center.
k
(5) Smoothness Term (ψsm ): To ensure a smoother and
(5) essentially measures the consistency among image Ii ’s simpler skeleton, we aim for a skeleton whose: (i) branches
own skeleton mask and the recommended skeleton mask are less in number and (ii) branches are long. Our criteria
from its neighbor images. Note that we accumulate the discourage skeletons with spurious branches while at the
co-skeleton prior scores in a certain neighborhood N(p) for same time encouraging skeletons with structure-defining

6208
branches. This is different from the criteria in [21] which Denoting θ(Ki , Ii ) as the developed appearance models,
o
only aims for less number of skeleton pixels. Specifically, we define the interdependence term ψin as
k
we define the smoothness term ψsm as   
X
|b(Ki )|
o
ψin (Oi |Ki , Ii ) = − log P Oi (p) | θ(Ki , Ii ), Ii (p)
X 1
k   p∈Di
ψsm (Ki ) = |b(Ki )| × (8)
u=1 length bu (Ki )   (11)
where P Oi (p) | θ(Ki , Ii ), Ii (p) denotes how likely a
where b(Ki ) = {b1 (Ki ), · · · , b|b(Ki )| (Ki )} denotes the pixel of color I(p) will take the label Oi (p) given θ(Ki , Ii ).
set of branches of the skeleton Ki . In this way, we pun- o
ψin is similar to the data term in the interactive segmenta-
ish skeletons with either large number of branches or short- tion method [15].
length branches. Smoothness Term (ψsm o
): For ensuring smooth fore-
3.3. Object Co-segmentation ground and background segments, we simply adopt the
smoothness term of GrabCut [15], i.e.,
The object co-segmentation problem here is as follows. X 2
Given the skeleton Ki , find the optimal Oi that minimizes o
ψsm (Oi |Ii ) = γ [Oi (p) 6= Oi (q)]e(−β||Ii (p)−Ii (q)|| )
the objective function defined in (3). The individual terms (p,q)∈Ei
in (3) are defined in the following manner. (12)
o
Prior Term (ψpr ): We generate an inter-image co- where Ei denotes the set of neighboring pixel pairs in the
segment prior, similar to that for co-skeletonization. In par- image Ii , and γ and β are segmentation smoothness related
ticular, we align segmentation masks of neighboring images parameters as discussed in [15].
and fuse them with that of the concerned image, i.e.,
P 3.4. Implementation Details
Oi + Wji (Oj )
Ij ∈Ni We use the saliency extraction method [2] for initializa-
Oei = (9)
|Ni | + 1 tion of our framework in our experiments. We use the same
default setting as that in [15] for the segmentation parame-
where Wji is the same warping function from image j to ters γ and β in (12) throughout our experiments. For the pa-
ei , we define our inter-
image i. Then, with the help of O rameters of SIFT flow [12], we follow the setting in [16] in
image prior term as order to handle the possible matching of different semantic
  objects. The parameter λ in both (2) and (3), which controls
X 1 X
o
ψpr (Oi |Ni ) = − Oi (p) log ei (q)
O the influence of joint processing, is set to 0.1.
|N(p)|
p∈Di q∈N(p)
   
! 4. Experimental Results
1 X
+ 1 − Oi (p) log 1 − ei (q)
O 4.1. Datasets and Evaluation Metrics
|N(p)|
q∈N(p)
(10) Datasets: There is only one publicly available dataset,
which encourages the shape to be consistent with O ei . Here i.e. WH-SYMMAX dataset [20], on which weakly super-
again we account for pixel correspondence errors by neigh- vised co-skeletonization can be performed, but it contains
borhood N(p) (in the pixel domain Di ) averaging. only the horse category of images. In order to evaluate
Interdependence Term (ψin o
): For the co-segmentation the co-skeletonization task extensively, we develop a new
process to benefit from co-skeletonization, our basic idea is benchmark dataset called the CO-SKEL dataset. It con-
to build up foreground and background appearance models sists of 26 categories with total 353 images of animals,
based on the given skeleton Ki . Particularly, we use GMM birds, flowers and humans. These images are collected
for appearance models. The foreground GMM model is from the MSRC dataset, CosegRep, Weizmann Horses and
learned using Ki (i.e., treating skeleton pixels as foreground iCoseg datasets along with their groundtruth segmentation
seeds), whereas the background GMM is learned using the masks. Then, we apply [21] (with our improved terms) on
background part of Ki ’s reconstructed shape R(Ki , Oi ). In these groundtruth masks, in the same manner as the WH-
this manner, the appearance model is developed entirely us- SYMMAX dataset has been generated from the Weizmann
ing the skeleton. Note that at the beginning it is not robust Horses dataset [1]. Fig. 6 shows some example images, and
to build up the GMM appearance models in this manner their skeletons using [21] and our improvement of [21]1 . It
since the initial skeleton extracted based on saliency is not can be seen that our skeletons are much smoother and better
reliable at all. Thus, at initialization, we develop the fore- in representing the shapes.
ground and background appearance models based on the 1 We will make our dataset with groundtruths and code publicly avail-

inter-image priors K e i and O

ei , respectively. able.

6209
 Image [21] Ours Image [21] Ours Image [21] Ours

Figure 6. Given the shape, we improve skeletonization method [21] using our improved terms in their objective function. It can be seen
that our skeletons are much smoother and better in representing the shape. We use these improved results as groundtruths in our CO-SKEL
dataset.

Since our method searches for k-nearest neighbors first Method F0 F1 F2 F3 J

and then performs joint processing, our method can also Ours(0) 0.095 0.229 0.282 0.319 0.412
work in an unsupervised way as long as there are a suffi- Ours (w/o ψin ) 0.168 0.337 0.391 0.434 0.649
cient number of images of same category objects or visu- Ours 0.189 0.405 0.464 0.506 0.721
ally similar objects. Thus, our method can also be applied
to datasets like the SK506 dataset [22], which consists of Table 1. Comparisons of the co-skeletonization and co-
many uncategorized images. segmentation results of our method and its two baselines on the
Metrics: For evaluation of skeletonization and segmen- WH-SYMMAX dataset. Ours(0) : our initialization baseline us-
tation, we calculate F-measure (including precision and re- ing Otsu thresholded saliency maps [2] for segmentation and [21]
for skeleton. Ours (w/o ψin ): our method without the interde-
call) and Jaccard Similarity, respectively. Considering it
pendence terms, i.e. running co-segmentation followed by skele-
is very difficult to get a resultant skeleton mask exactly tonization.
aligned with the groundtruth, if a resultant skeleton pixel is
nearby a groundtruth skeleton pixel, it should be considered F0 F1 F2 F3 J
as a hit. Therefore, we consider a resultant skeleton pixel as (0)
correct if it is at a distance of d ∈ {0, 1, 2, 3} pixels from a Ours 0.129 0.306 0.371 0.416 0.600
groundtruth skeleton pixel, for which we denote F d as the Ours (w/o ψin ) 0.236 0.426 0.484 0.522 0.725
corresponding F-measure. Jaccard Similarity (denoted as Ours 0.237 0.435 0.495 0.535 0.741
J) is basically the IoU of groundtruth and our segmentation Table 2. Comparisons of the co-skeletonization and co-
result. segmentation results of our method and its two baselines on our
4.2. Weakly Supervised Results CO-SKEL dataset.

We report our overall co-skeletonization and co-

segmentation results on WH-SYMMAX and our CO-SKEL It can be seen that our method outperforms this base-
datasets in Tables 1 and 2, respectively. Note that since we line on both the datasets. Marginal improvement on the
do not perform any kind of training, we combine both train- CO-SKEL dataset may be due to already good initializa-
ing and test images of the WH-SYMMAX dataset, and then tion. Specifically, it can be seen that J for initialization is
obtain the results. It can be seen that our method greatly already 0.600 in the CO-SKEL dataset compared to 0.412
improves over our initialization baseline. To demonstrate in the WH-SYMMAX dataset, suggesting that there is rela-
the importance of considering the interdependence between tively less room for improvement.
co-segmentation and co-skeletonization, we also compare We also evaluate how our method performs at differ-
the proposed method with another baseline, Ours (w/o ψin ), ent iterations in Fig. 8 on the WH-SYMMAX dataset. It
where we remove the interdependence, i.e., running co- can be seen that our method first improves the performance
segmentation first and then performing skeletonization from swiftly and then it becomes somewhat steady. This sug-
the resultant foreground segments. gests that 2-3 iterations are good enough for our method.

6210
Figure 7. Some examples of steadily improving skeletonization and segmentation after each iteration. The top-right example shows that
our model continues to reproduce similar results once the optimal shape and skeleton are obtained.

m F0 F1 F2 F3 J 0.8
bear 4 0.075 0.1714 0.213 0.246 0.846 0.7
iris 10 0.363 0.600 0.658 0.698 0.837 0.6
camel 10 0.224 0.353 0.395 0.432 0.674 0.5
F0
cat 8 0.118 0.360 0.469 0.523 0.733 F1
0.4
cheetah 10 0.078 0.221 0.287 0.335 0.735 F2
0.3
cormorant 8 0.351 0.545 0.606 0.642 0.768 F3
J
cow 28 0.142 0.437 0.580 0.669 0.789 0.2

cranesbill 7 0.315 0.619 0.670 0.696 0.935 0.1

deer 6 0.214 0.366 0.407 0.449 0.644 0

0 1 2 3 4 5
desertrose 15 0.360 0.662 0.721 0.759 0.934
dog 11 0.122 0.356 0.457 0.522 0.746
egret 14 0.470 0.642 0.669 0.693 0.760 Figure 8. Performance v/s Iteration plot. It can be seen that the
firepink 6 0.416 0.685 0.756 0.805 0.918 performance improves swiftly at first and then becomes steady.
frog 7 0.163 0.358 0.418 0.471 0.734
geranium 17 0.299 0.633 0.716 0.764 0.940 Please refer to Fig. 7 for examples where the results im-
horse 31 0.217 0.435 0.490 0.529 0.726 prove steadily with each iteration. Fig. 9 shows some sam-
man 20 0.144 0.246 0.274 0.295 0.385 ple results of our method along with groundtruths from the
ostrich 11 0.298 0.530 0.592 0.634 0.752 WH-SYMMMAX and CO-SKEL datasets.
panda 15 0.037 0.102 0.140 0.174 0.696 We also show our results on individual categories and the
pigeon 16 0.181 0.326 0.361 0.382 0.590 variance in performance across the categories of our CO-
seagull 13 0.257 0.461 0.520 0.562 0.662 SKEL dataset in Table 3. Low variances for both F d and J
seastar 9 0.440 0.649 0.681 0.702 0.750 metrics suggest that our method is quite reliable.
sheep 10 0.078 0.249 0.342 0.401 0.769
snowowl 10 0.089 0.222 0.268 0.306 0.543 4.3. Supervised Results
statue 29 0.306 0.506 0.542 0.564 0.681 In the literature, since only the fully supervised skele-
woman 23 0.305 0.463 0.503 0.533 0.674 tonization methods are available, for fair comparison, we
variance 0.015 0.028 0.029 0.030 0.016 follow the original process but with a change in the initial-
ization. We replace the saliency initialization with ground
Table 3. Categorywise number of images and our weakly super-
vised results on the CO-SKEL dataset. truth initialization for training images. This will help de-
velop better joint processing priors for remaining images
which are the test images. We do the comparisons on test
images of the WH-SYMMAX and SK506 datasets in Ta-

6211
 Image Groundtruth Ours Image Groundtruth Ours Image Groundtruth Ours Image Groundtruth Ours

Figure 9. Sample co-skeletonization results along with our final shape masks. It can be seen that both are quite close to the groundtruths.

Methods WH-SYMMAX SK506 4.4. Limitations

[9] 0.174 0.218
Our method has some limitations. First, for initializa-
[8] 0.223 0.252
tion, our method requires common object parts to be salient
[26] 0.334 0.226
in general across the neighboring images if not in all. There-
[23] 0.103 -
fore, it depends on the quality of the neighboring images.
[25] 0.365 0.392
The second limitation lies in the difficulty during warping
[29] 0.402 -
process. For example, when the neighboring images con-
Ours(0) 0.322 0.261
tain objects at different sizes or at different viewpoints, the
Ours 0.530 0.483
warping processing will have difficulty in aligning the im-
Ours (S) 0.594 0.523
ages. Such a situation will not be crucial when there is a
large number of images to select from. Another issue is that
Table 4. Comparisons of the results of F d of our methods with
smoothing the skeleton may cause missing out some impor-
supervised methods. Ours(0) : our initialization baseline. Ours
tant short branches.
(S): our method with groundtruth
p initialization on training images.
Note that here d = 0.0075× width2 + height2 following [22].
5. Conclusion
The major contributions of this paper lie in the newly
defined co-skeletonization problem and the proposed joint
ble 4. Note that to make the distinction between our su- co-skeletonization and co-segmentation framework, which
pervised method (groundtruth initialization) and our weakly effectively exploits inherent interdependencies between the
supervised method (with saliency initialization), we denote two to assist each other synerergistically. Extensive experi-
the results of our supervised approach as “Ours (S)”. It can ments demonstrate that the proposed method achieves very
be seen that not only our supervised method comfortably competitive results on a few benchmark datasets.
outperforms all the traditional supervised methods, but also Acknowledgements This research is supported by the
our weakly supervised (unsupervised for SK506) approach National Research Foundation, Prime Ministers Office, Sin-
is able to do so. Note that the other performance values re- gapore, under its IDM Futures Funding Initiative. It is also
ported here are directly taken from [22]. We would like to supported by the HCCS research grant at the ADSC2 from
point out that the recently developed deep learning based Singapore’s A*STAR.
supervised method [22] reports much better performance.
We did not compare with it since our method essentially is 2 This work was partly done when Koteswar and Jiangbo were interning

a weakly supervised approach. and working in ADSC

6212
References In Computer Vision and Pattern Recognition (CVPR), pages
1939–1946. IEEE, 2013.
[1] E. Borenstein and S. Ullman. Class-specific, top-down seg-
[17] P. K. Saha, G. Borgefors, and G. S. di Baja. A survey on
mentation. In European Conference on Computer Vision
skeletonization algorithms and their applications. Pattern
(ECCV), pages 109–122. Springer Berlin Heidelberg, 2002.
Recognition Letters, 76:3 – 12, 2016. Special Issue on Skele-
[2] M.-M. Cheng, G.-X. Zhang, N. J. Mitra, X. Huang, and tonization and its Application.
S.-M. Hu. Global contrast based salient region detection. [18] E. Shelhamer, J. Long, and T. Darrell. Fully convolu-
In Computer Vision and Pattern Recognition (CVPR), pages tional networks for semantic segmentation. IEEE Trans-
409–416. IEEE, 2011. actions on Pattern Analysis and Machine Intelligence (T-
[3] W.-P. Choi, K.-M. Lam, and W.-C. Siu. Extraction of the PAMI), 39(4):640–651, 2017.
euclidean skeleton based on a connectivity criterion. Pattern [19] W. Shen, X. Bai, R. Hu, H. Wang, and L. J. Latecki. Skeleton
Recognition, 36(3):721 – 729, 2003. growing and pruning with bending potential ratio. Pattern
[4] J. Dai, Y. N. Wu, J. Zhou, and S.-C. Zhu. Cosegmenta- Recognition, 44(2):196 – 209, 2011.
tion and cosketch by unsupervised learning. In International [20] W. Shen, X. Bai, Z. Hu, and Z. Zhang. Multiple instance
Conference on Computer Vision (ICCV). IEEE, 2013. subspace learning via partial random projection tree for local
[5] K. R. Jerripothula, J. Cai, and J. Yuan. Group saliency prop- reflection symmetry in natural images. Pattern Recognition,
agation for large scale and quick image co-segmentation. In 52:306 – 316, 2016.
International Conference on Image Processing (ICIP), pages [21] W. Shen, X. Bai, X. Yang, and L. J. Latecki. Skeleton prun-
4639–4643. IEEE, 2015. ing as trade-off between skeleton simplicity and reconstruc-
[6] K. R. Jerripothula, J. Cai, and J. Yuan. Cats: Co-saliency tion error. Science China Information Sciences, 56(4):1–14,
activated tracklet selection for video co-localization. In Eu- 2013.
ropean Conference on Computer vision (ECCV), pages 187– [22] W. Shen, K. Zhao, Y. Jiang, Y. Wang, Z. Zhang, and X. Bai.
202. Springer, 2016. Object skeleton extraction in natural images by fusing scale-
[7] K. R. Jerripothula, J. Cai, and J. Yuan. Image co- associated deep side outputs. In Computer Vision and Pattern
segmentation via saliency co-fusion. IEEE Transactions on Recognition (CVPR), pages 222–230. IEEE, 2016.
Multimedia (T-MM), 18(9):1896–1909, Sept 2016. [23] A. Sironi, V. Lepetit, and P. Fua. Multiscale centerline detec-
[8] T. S. H. Lee, S. Fidler, and S. Dickinson. Detecting curved tion by learning a scale-space distance transform. In Com-
symmetric parts using a deformable disc model. In Interna- puter Vision and Pattern Recognition (CVPR), pages 2697–
tional Conference on Computer Vision (ICCV), pages 1753– 2704. IEEE, 2014.
1760. IEEE, 2013. [24] M. Tang, L. Gorelick, O. Veksler, and Y. Boykov. Grabcut
[9] A. Levinshtein, S. Dickinson, and C. Sminchisescu. Mul- in one cut. In International Conference on Computer Vision
tiscale symmetric part detection and grouping. In Interna- (ICCV), pages 1769–1776, 2013.
tional Conference on Computer Vision (ICCV), pages 2162– [25] S. Tsogkas and I. Kokkinos. Learning-based symmetry de-
2169. IEEE, 2009. tection in natural images. In European Conference on Com-
[10] D. Lin, J. Dai, J. Jia, K. He, and J. Sun. Scribble- puter Vision (ECCV), pages 41–54. Springer Berlin Heidel-
sup: Scribble-supervised convolutional networks for seman- berg, 2012.
tic segmentation. In The IEEE Conference on Computer Vi- [26] N. Widynski, A. Moevus, and M. Mignotte. Local sym-
sion and Pattern Recognition (CVPR). IEEE, 2016. metry detection in natural images using a particle filtering
[11] T. Lindeberg. Edge detection and ridge detection with au- approach. IEEE Transactions on Image Processing (T-IP),
tomatic scale selection. International Journal of Computer 23(12):5309–5322, 2014.
Vision, 30(2):117–156, 1998. [27] S. Xie and Z. Tu. Holistically-nested edge detection. In
[12] C. Liu, J. Yuen, and A. Torralba. Sift flow: Dense corre- International Conference on Computer Vision (ICCV), pages
spondence across scenes and its applications. IEEE Trans- 1395–1403. IEEE, 2015.
actions on Pattern Analysis and Machine Intelligence (T- [28] Z. Yu and C. Bajaj. A segmentation-free approach for skele-
PAMI), 33(5):978–994, 2011. tonization of gray-scale images via anisotropic vector diffu-
[13] F. Meng, J. Cai, and H. Li. Cosegmentation of multiple sion. In Computer Vision and Pattern Recognition (CVPR),
image groups. Computer Vision and Image Understanding pages 415 – 420. IEEE, 2004.
(CVIU), 146:67 – 76, 2016. [29] Q. Zhang and I. Couloigner. Accurate centerline detec-
[14] A. Oliva and A. Torralba. Modeling the shape of the scene: A tion and line width estimation of thick lines using the radon
holistic representation of the spatial envelope. International transform. IEEE Transactions on Image Processing (T-IP),
journal of computer vision (IJCV), 42(3):145–175, 2001. 16(2):310–316, 2007.
[15] C. Rother, V. Kolmogorov, and A. Blake. Grabcut: Inter- [30] H. Zhu, F. Meng, J. Cai, and S. Lu. Beyond pixels: A com-
active foreground extraction using iterated graph cuts. In prehensive survey from bottom-up to semantic image seg-
Transactions on Graphics (TOG), volume 23, pages 309– mentation and cosegmentation. Journal of Visual Communi-
314. ACM, 2004. cation and Image Representation (JVCIR), 34:12 – 27, 2016.
[16] M. Rubinstein, A. Joulin, J. Kopf, and C. Liu. Unsupervised
joint object discovery and segmentation in internet images.

6213