SORTING LOCAL DESCRIPTORS FOR LOW BIT RATE MOBILE VISUAL SEARCH

Jie Chen Ling-Yu Duan Rongrong Ji† Hongxun Yao† Wen Gao†

Institute of Digital Media, Peking University, Beijing 100871, China
†
Visual Intelligence Laboratory, Harbin Institute of Technology, Heilongjiang, 150001, China

ABSTRACT largely degenerates the user experiences in most existing mo-

bile visual search applications. Coming with the ever grow-
State-of-the-art mobile visual search systems put emphasis on
ing computational power of the mobile devices, recent works
developing compact visual descriptors [4][6], which enables
have directly extracted visual descriptors on the mobile de-
low bit rate wireless transmission instead of delivering an en-
vices for a low cost transmission. In such case, the compres-
tire query image. In this paper, we address the orderless na-
sion rates of the traditional compressive local descriptors (e.g.
ture of the transmission set of query descriptors . We propose
SURF [11] and PCA-SIFT [12]) are still not sufficient for the
to adapt the orders of local descriptors in transmission, which
low bit rate wireless transmission.
subsequently yields more consistent statistic distributions in
each feature dimension towards more efficient residual coding Recent works have focused on extreme low bit rate vi-
based compression. Our scheme further enables lossy sorting sual descriptors such as [2][3][4][6], which are specially de-
by an adaptive quantization strategy within each feature di- signed for the mobile visual search scenarios. For instance,
mension, which largely improves the compression rates of the Chandrasekhar et al. proposed a Compressed Histogram of
residual coding in each dimension. We show that the perfor- Gradient (CHoG) [4] for compressive local feature descrip-
mance degeneration of such lossy sorting is acceptable in our tion, which adopts both Huffman Tree and Gagie Tree to com-
mobile landmark search applications. Our approach’s effec- press each local feature into approximate 50 bits. The work
tiveness and efficiency is demonstrated via extensive experi- in [6] compressed the SIFT descriptor with Karhunen-Loeve
mental comparisons to state-of-the art works in both mobile Transform, resulted into approximate 2 bits per SIFT dimen-
visual descriptors [2][4] and compact image signatures [5]. sion. Tsai et al. [10] proposed to transmit the spatial layouts
of interest points to improve the discriminability of CHoG.
Index Terms— mobile visual search, compressive visual
Two recent works also transmit the bag-of-features histogram
descriptors, sorting, lossy compression
[2][3] instead of the local feature descriptors: Chen et al. pro-
posed to compress the sparse bag-of-features [2] by encoding
1. INTRODUCTION position difference of non-zero bins. It produced an approx-
imate 2KB code per image for a vocabulary with 1M words.
With the popularization of camera embedded mobile de- The work in [3] further compressed the inverted indices of
vices and wireless Internet services, nowadays there is an vocabulary tree [1] with arithmetic coding, which further re-
emerging potential in mobile visual search and related ap- duces the maintaining memory cost in the mobile device.
plications. Generally speaking, most state-of-the-art mobile
visual search systems follow a client-server architecture. The In this paper, we focus on extracting and transmitting the
remote server maintains a large-scale photo database, where local descriptor set through the wireless network. Especially,
photos are bound with related information, e.g. locations, while such local descriptors are independent with each other,
touristic and commercial information. To enable the scalabil- we propose to sort their transmitting orders to facilitate even
ity of the ranking system, inverted indexing is usually adopt, lower bit rate wireless transmission with residual coding com-
typically built upon scalable bag-of-words models, such as paring with the state-of-the-art works [4][6]. To the best of
works in [1][8][9]. In online search, the mobile user takes a our knowledge, this serves as the first work to consider the
photo as a query, which is transmitted to the remote server to descriptor orders in the context of mobile visual search.
identify its matching photos. Consequently, the best matched The rest of this paper is organized as follows: Section 2
photos, together with their bounded information are returned introduces our proposed local descriptor sorting scheme, with
to the mobile user as the search results. a subsequent residual coding for low bit rate wireless query
In a typical scenario, the query photo transmission from transmission. Section 3 presents our mobile landmark search
the mobile devices to the remote server is often over a rela- system, with quantitative comparisons to state-of-the-arts
tively slow, bandwidth-constrained wireless network. In such [2][4][5]. We conclude this paper in Section 3 and discuss
case, sending the entire image is often time consuming, which our future research directions.

978-1-4577-0539-7/11/$26.00 ©2011 IEEE 1029 ICASSP 2011

Authorized licensed use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:10 UTC from IEEE Xplore. Restrictions apply.
 Fig. 1. The proposed low bit rate mobile visual search framework via sorting local feature descriptors [4][7].

2. SORTING LOCAL DESCRIPTORS WITH

RESIDUAL CODING

Figure 1 shows the systematic framework of the proposed

low bit rate mobile visual search system. Similar to state-
of-the-art works in generic and compact visual descriptors
[2][3][4][6], our framework emphasizes on directly extract-
ing compact local feature descriptors on a mobile device, then
transmitting more reduced-sized query data over the wire-
less network. Comparing with existing works [2][3][4][6],
our contribution lies in sorting local descriptors followed Fig. 2. The visualized example of orderless SIFT sets for
by residual coding to further reduce the transmission rate. three photo. Different query produces different amount of
Since the local descriptors are delivered in orderless manner SIFT, hence the horizontal length is not identical. Each row’s
through wireless network , their sorting can be performed in height is identical, denoting 128 SIFT dimensions. Brighter
an arbitrary order. We argue that sorting these descriptors can points denote highes values.
bring about potential benefits for subsequent coding methods
such as residual coding or wavelet transformation to further
encode the entire data amount to transmit. k ∈ [1, K] denotes the dimension of a given local descriptor,
The Proposed Mobile Visual Search Pipeline: Given a say 128 for SIFT. We select each new S Ri that minimizes the
query photo Iq , a mobile device extracts a set of local descrip- distance to its former descriptor S Ri−1 . Fixing the cost of stor-
tors as Sq = [S 1q , S 2q , ..., S qJ ]. Once this feature set arrives at ing the first local descriptor S R1 , the overall cost with current
the remote server, Scalable Vocabulary Tree (SVT) [1] is em- ordering is given by:
ployed to find the near-duplicated photos (such as landmarks 
J
or objects) in an efficient manner. SVT is well exploited in LostS ort = Residual(S Ri ) (3)
state-of-the-art works [2] [3][8][9], which uses hierarchical k- j=1
means to partition local descriptors into quantized codewords.
SVT quantizes Sq by traversing in the vocabulary hierarchy to Greedy and Lossy Learning: Learning the sorting or-
find the nearest codeword, which converts Sq into a Bag-of- ders by Equation 3 is NP hard. We resort to its greedy
Words (BoW) signature Vq = [V1q , V2q , ..., V M q
]. In each code- version: At the selection of the ith local descriptor, we at-
word’s inverted index, all co-indexed photos are included into tempt to find out the descriptor that has the minimal cost of
the similarity ranking to search near-duplicated photos. Residual(S Ri ) using Equation 2. In such case, learning is
Sorting Local Descriptors From the above pipeline, efficient, with the complexity of only O(J 2 ) to learn the best
search is independent of the sending order of these local order. Let’s step forward by look forward the N local descrip-
descriptors. Therefore, we propose to learn an ordering func- tors [S 1 , S 2 , ..., S N ], which means at each time we select a
tion [R1 , R2 , ...R J ] for Sq to sort them. Subsequently, we only batch of local descriptors as:
transmit Sq via the following residual coding: 
N
SPatch = arg minS N (Residual(S 1 ) + Residual(S n )) (4)
S R1 , Residual(S R2 ), ..., Residual(S RJ ) (1) n=2

where the Residual(S Ri ) denotes the residual coding from S Ri Such selection leads to in total C JN magnification of computa-
to its previous descriptor S Ri−1 , which is calculated as: tional cost to accomplish its greedy selection.
We further consider the case of lossy learning, which
Residual(S Ri ) = [Dis(s1i , s1i−1 ), ..., Dis(siK , si−1
K
)] (2) means that the coding residual for a given dimension is fur-

1030
Authorized licensed use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:10 UTC from IEEE Xplore. Restrictions apply.
ther quantized into M fractions. As a result, we have the
following simplification:

Residual(S Ri ) = [Qk (Dis(ski , ski−1 ))]k=1

K
(5)

3. MOBILE SEARCH SYSTEM IMPLEMENTATIONS

AND QUANTITATIVE RESULTS

Data Collection: We collected over 10 million geo-tagged

photos from photo sharing websites of Flickr and Panoramio.
Our dataset covers typical areas including Beijing, New York
City, Lhasa, Singapore, and Florence.
Ground Truth Labeling: First, within each city, we se- Fig. 3. Compression rate and ranking distortion comparing
lected the top 30 most densely geographical regions, as well with [2][4][5] using our ground truth query set. The six
as 30 random geographical regions. Second, as manually points (left to right) in Lossy Sorting correspond to M =
identifying all related photos from an identical landmark is in- 2, 4, 8, 16, 32, 64.
tensive, for each of these 60 regions, we ask our volunteers to
manually identify one or more dominant landmark views. All
their near-duplicated landmark photos are labeled in its cur- about theoretical improvements in our subsequent compres-
rent region and all the geographical nearby regions. Finally, sion rate, it can well reduce the storage/memory cost in a
we randomly sample 5 images from each region to form the mobile device. Also, incorporating CHoG instead of SIFT
ground truth. Each query and their returning results are used is shown to yield better ranking MAP. Hence, we choose
to test the retrieval performance of different methodologies. CHoG as our final local feature descriptor in our system
So that we generate 300 user query logs and their ground truth implementation.
for each city (in total 300 × 5 queries) Rate Distortion Analysis: We give the rate distortion
Parameters Tuning and Evaluation: For the landmark analysis of our approach in comparisons to alternative meth-
photo collection within each city, we extract both SIFT [7] ods and state-of-the-art works in [2][4][5]. It is obvious that
and CHoG [4] features from each photo. We build a Scal- our schemes have achieved the best performances, as shown
able Vocabulary Tree model [1] to produce the initial Vocab- in Figure 3. It is obvious that our scheme has not serious lost
ulary V, which generates a bag-of-words signature Vi for each of MAP before quantization of M = 8, which means the SIFT
database photo Ii . We denote the hierarchical level as H and descriptor used here is indeed redundant and compressible.
the branching factor as B, which gives at most M = BH words Lossy Sorting Performance Degeneration: We further
at the finest level. We stop the quantization division once investigate the performance degeneration of using our lossy
there are less than 1,000 SIFT features in a word (node). Typ- sorting strategy. In Figure 4, we investigates the MAP degen-
ically, we have H = 6 and B = 10, producing approximate eration using different quantization scheme, where Qi = M
100,000 codewords. denotes we subdivide each local descriptor dimension into M
We use Mean Average Precision at N (MAP@N) against fractions, and the setting of M = [2, 4, 8, 16, 32, 64] denotes
the compression rate to evaluate our system performance. the different division scheme in lossy quantization.
MAP@N represents the mean precision of a batch of queries, Generally speaking, the curves (M = 16, 32, and 64)
each of which reveals its position-sensitive ranking precision in Figure 4 is smooth with less MAP degeneration (verti-
within the top N ranking positions. cal viewpoint), until M = 8. The figure shows the com-
Baselines: (1) Transmitting the original query photo: As pression rates with different quantization schemes for M =
the most common strategy in most existing mobile visual [2, 4, 8, 16, 32, 64] to compare with the lossless compression
search scenario, we resort to JPEG compression to encode scheme. Also, since before M = 8 the distortion is not obvi-
an entire query photo. It serves as the first baseline with ous, we have selected M = 16 in our subsequent case study.
the highest coding length in our comparisons. (2) Transmit- Case Study: Figure 5 shows several groups of exem-
ting SIFT Descriptor Sets: Transmitting the original local plar transmission rates, in which we show the given landmark
descriptor set serves as a better solution, comparing with query as well as its corresponding transmission rate. While
transmitting the entire query photo. Since SIFT [7] feature in general the coding length is variable (corresponding to dif-
is widely adopted, we use the SIFT descriptor set as our an- ferent landmark queries), it is obvious that our approach can
other baseline. (3) Transmitting CHoG Descriptor Sets: As give the most compact visual description length comparing
an alternative approach, we replace the SIFT descriptor with with directly sending the unordered descriptor sets. Finally,
CHoG [4], which produces a more compact local descriptor the sorted local descriptor set corresponding to Figure 2 is
with approximate 50 dimensions. Although it does not bring given in Figure 6

1031
Authorized licensed use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:10 UTC from IEEE Xplore. Restrictions apply.
 Fig. 6. The 3 corresponding local descriptor sets of Figure 2
after the lossless sorting (lookforward step N = 1).

Fig. 4. The retrieval MAP lost of different lossy sorting 5. ACKNOWLEDGEMENTS

schemes by quantization scales of M = [2, 4, 8, 16, 32, 64].
This work was supported in part by grants from the Chi-
nese National Natural Science Foundation under contract No.
60902057, in part by the National Basic Research Program of
China under contract No. 2009CB320902, and in part by the
CADAL Project Program.

6. REFERENCES
[1] Nister D. and Stewenius H. Scalable recognition with a vocab-
ulary tree. CVPR. 2006. 1, 2, 3
[2] Chen D., Tsai S., and Chandrasekhar V. Tree histogram coding
for mobile image matching. DCC. 2009. 1, 2, 3
[3] Chen D., Tsai S., Chandrasekhar V., Takacs G., Vedantham R.,
Grzeszczuk R., and Girod B. Inverted index compression for
Fig. 5. Case study of transmission rates (Bytes Per Local De-
scalable image matching. DCC. 2010. 1, 2
scriptor) for representative landmark queries in Beijing. Each
[4] Chandrasekhar V., Takacs G., Chen D., Tsai S., Grzeszczuk R.,
photo on the left is the query, its right histogram is the trans-
and Girod B. CHoG: Compressed histogram of gradients a low
mission rate of transmitting the unordered local descriptors bit-rate feature descriptor. CVPR. 2009. 1, 2, 3, 4
(Green), lossless sorting (Blue), and lossy sorting M = 16
[5] Jegou H., Douze M., Schmid C., Perez P. Aggregating local
(Red), respectively. descriptors into a compact image representation. CVPR. 2010. 1,
3
[6] Chandrasekhar V., Takacs G., Chen D., Tsai S., Singh J., and
4. CONCLUSIONS AND FUTURE WORKS Girod B. Transform coding of image feature descriptors. VCIP.
2009. 1, 2
We have studied the feasibility of sorting the local descriptors [7] Lowe D. G. Distinctive image features from scale-invariant key-
at a mobile device, which enables low bit rate wireless trans- points. IJCV. 2004. 2, 3
mission in the context of mobile visual search. Such sorting [8] Irschara A., Zach C., Frahm J., Bischof H. From structure-from-
is able to facilitate the subsequent residual coding for a com- motion point clouds to fast location recognition. CVPR. 2009. 1,
pressive transmission of the original orderless local descrip- 2
tor set. We further enable lossy sorting by adaptively quantiz- [9] Schindler G. and Brown M. City-scale location recognition.
ing the residual values into different fractions, which achieves CVPR. 2007. 1, 2
even higher compression rate without serious lost in retrieval [10] Tsai S., Chen D., Takacs G., Chandrasekhar V. Location cod-
performance. We reported the superior performance over the ing for mobile image retrieval. MobileMedia. 2010. 1
state-of-the-art work in compact descriptors for visual search. [11] Bay H., Tuytelaars T., and Van Gool L. SURF: Speeded up
[4]. Currently, the residual coding only serves as an exem- robust features. ECCV. 2006. 1
plar and suggestive solution to compress our “smooth” (after [12] Ke Y. and Sukthankar R. PCA-SIFT: A more distinctive repre-
sorting) local descriptor sets. Our future work includes more sentation for local image descriptors. CVPR. 2004. 1
investigation on different coding schemes, such as wavelet or
DCT transforms, to further compress the proposed local de-
scriptor sorting scheme.

1032
Authorized licensed use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:10 UTC from IEEE Xplore. Restrictions apply.