This article has been accepted for publication in a future issue of this journal, but has not been

fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 1

BOEW: A Content-based Image Retrieval

Scheme using Bag-of-Encrypted-Words in
Cloud Computing
Zhihua Xia, Member, IEEE, Leqi Jiang, Dandan Liu, Lihua Lu, and Byeungwoo Jeon, Senior Mem-
ber, IEEE,

Abstract—Content-based Image Retrieval (CBIR) techniques have been extensively studied with the rapid growth of digital images.
Generally, CBIR service is quite expensive in computational and storage resources. Thus, it is a good choice to outsource CBIR
service to the cloud server that is equipped with enormous resources. However, the privacy protection becomes a big problem, as the
cloud server cannot be fully trusted. In this paper, we propose an outsourced CBIR scheme based on a novel bag-of-encrypted-words
(BOEW) model. The image is encrypted by color value substitution, block permutation, and intra-block pixel permutation. Then, the
local histograms are calculated from the encrypted image blocks by the cloud server. All the local histograms are clustered together,
and the cluster centers are used as the encrypted visual words. In this way, the bag-of-encrypted-words (BOEW) model is built to
represent each image by a feature vector, i.e., a normalized histogram of the encrypted visual words. The similarity between images
can be directly measured by the Manhattan distance between feature vectors on the cloud server side. Experimental results and
security analysis on the proposed scheme demonstrate its search accuracy and security.

Index Terms—Searchable encryption; Computation outsourcing; Content-based image retrieval; Bag-of-Words.

F
1 I NTRODUCTION

T HE world has witnessed a rapid development of imag-

ing devices, such as digital cameras, medical imaging
equipments, smart phones, and so on. Accordingly, the
The main contributions are summarized as follows:
1) A BOEW model is proposed for CBIR outsourcing. We
propose to encrypt images by blocks and make sure that
number of digital images increases dramatically. In order the secure and useful local features can be directly ex-
to retrieve similar images quickly from large amount of im- tracted from the encrypted blocks. k -means clustering
ages, many practical Content-based Image Retrieval (CBIR) algorithm is deployed to generate the encrypted visual
techniques have been developed. However, typical image words. The final feature vectors, also the encrypted
database is simply too large, including millions of images ones, are then constructed with the visual words. The
and each may be larger than 40 megabytes [1]. Thus, CBIR similarity between the feature vectors can be directly
service generally requires heavy storage and computation. measured by Euclidean or Manhattan distance. The
Such demands make it attractive to outsource CBIR services proposed BOEW could be a valuable model in encrypt-
to the cloud server. In this way, the image owner needs ed image processing.
not to store the image database locally, and can efficiently 2) As a case study, we propose to encrypt image by color
retrieve the desired images from the cloud server [2]. value substitution, block permutation, and intra-block
Apart from the enormous benefits of CBIR outsourcing, pixel permutation. With the specially-designed encryp-
the privacy of images becomes the biggest concern to the tion method, secure local histograms can be directly
image owner. Both the image database and the query image extracted from the encrypted images on cloud server
should be protected properly. side. The index construction can also be finished by
Contribution. In this paper, we propose an outsourced cloud server. Compared with the scheme using secure
CBIR scheme where the image content is properly protected. global histogram [3], [4], our method achieves a much
better retrieval accuracy.
• Zhihua Xia, Leqi Jiang, Dandan Liu, and Lihua Liu are with Jiangsu The rest of this paper is organized as follows. Section
Engineering Center of Network Monitoring, Jiangsu Collaborative Inno- 2 introduces the related works. Section 3 presents the tech-
vation Center on Atmospheric Environment and Equipment Technology,
and School of Computer and Software, Nanjing University of Information nical overview. Section 4 addresses the proposed scheme
Science & Technology, Nanjing, China. Zhihua Xia is also with the design. Section 5 gives the security analysis. Experiments
College of Information & Communication Engineering, Sungkyunkwan and results are presented in Section 6. Finally, conclusion is
University, Korea.
E-mails: xia [email protected], jdz [email protected], [email protected],
made in Section 7.
millie [email protected]
• Byeungwoo Jeon is with the College of Information & Communication
Engineering, Sungkyunkwan University, Korea. 2 R ELATED WORKS
E-mail: [email protected] CBIR techniques have been studied for more than twenty
Manuscript received November 9, 20**; revised August 26, **. years and shown its maturity in many real-world appli-

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 2

cations [5], [6], [7], [8]. However, it cannot be outsourced

directly to the cloud due to the privacy concern. It is
noteworthy that, in addition to the original image data, the
image features would also leak the information about image
contents if they are not well protected.
Searchable encryption (SE) enables the clients to store
the encrypted data at the cloud, meanwhile supports data
search over cipher-text domain [9]. However, many of the Fig. 1: System model. In the initialization phase, the image
existing SE schemes are designed for text documents [10], owner encrypts and uploads the images to the cloud server.
[11], [12]. Lu et al. [13] proposed the first privacy-preserving Then, the cloud server extracts features from the encrypted
CBIR scheme over the encrypted image database. The images to generate the index. In the query phase, the image
scheme utilized the set of visual words to represent images. owner uploads an encrypted query image to the cloud serv-
The similarity between images was measured by Jaccard er. Next, the cloud server extracts features from the query
distance between the sets of visual words. The min-hash image and search the index. Finally, the images similar to
algorithm and order-preserving encryption were employed the query image are returned to the image owner.
to protect the visual words. In another work [14], Lu et
al. investigated three image feature protection techniques
including bitplane randomization, random projection, and
randomized unary encoding. The bitplane randomization
and random unary encoding support the calculation of The works mentioned above are good solutions to the
Hamming distance in the encryption domain. The random CBIR service outsourcing. Nevertheless, there exists a com-
projection supports the approximate calculation of L1 dis- mon disadvantage. As the big volume of storage and large
tance in the encryption domain. In [15], Lu et al. compared computation complexity, both the image feature extraction
the three mentioned methods with the homomorphic en- and index construction are resource-consuming operations.
cryption and indicated that the homomorphic encryption In the previous outsourced CBIR scheme, it is data own-
consumed much more computation and communication er to undertake the tasks of feature extraction and index
resources. Yuan et al. [16] protected the image features using generation [13], [14], [15], [16], [17], [18], [19], [21], [22].
local sensitive hashing and Cuckoo Hashing to support Bellafqira et al. [24], [25] proposed two secure CBIR schemes
secure similarity search. This method was used to discovery in homomorphic encryption domain. The SIFT [24] and
the social connections between image owners. Xia et al. discrete wavelet transform features [25] can be extracted
[17] proposed a privacy-preserving CBIR scheme based on directly from the encrypted images. However, these features
Scale-Invariant Feature Transform (SIFT) features and Earth can only support the exact search because the tiny differ-
Mover’s Distance (EMD). The calculation of the EMD is in ences between the plaintext images can make the encrypted
fact a linear program problem. The linear transformation images completely different. In [26], Xu et al. proposed a
was utilized to protect the privacy information during the privacy-preserving CBIR scheme based on partial encryp-
solution process of EMD problem. Yuan et al. [18] designed tion. The images were orthogonally decomposed and then
an encrypted image search scheme based on the secure k NN divided into two different parts. One part was encrypted to
(k -nearest neighbors) algorithm and constructed a tree index protect the image content and the other was used to extract
to improve the search efficiency. In [19], Chen et al. proposed features for image retrieval. However, the unencrypted part
a Markov process-based retrieval scheme over encrypted can lead serious information leakage. Ferreira et al. [3], [4]
images. The image content was protected by encrypting proposed an Image Encryption Scheme which is tailor-made
the Huffman table in JPEG files. The Markov features were for constructing a CBIR scheme (IES-CBIR). In this scheme,
directly extracted from the DCT coefficients which were the color values are protected by random permutation and
decoded with the encrypted Huffman table. In [20], [21], the pixels are shuffled by rows and columns. After the
Weng et al. proposed a framework for privacy-preserving encrypted image are uploaded to the cloud server, the
multimedia retrieval. The media features were protected by HSV (Hue-Saturation-Value) color histograms are extracted
the robust hashing and partial encryption by image owner. from the encrypted images at the cloud server side. The
Then encrypted part of hash introduced search ambiguity similarities between images can be directly measured by
to server. The similar images were retrieved using the Hamming distances between the corresponding histograms.
unencrypted part of hash on the server side and refined In this way, the image owner undertakes the image en-
with the whole plaintext hash on the query user side. In cryption only. Other tasks, such as feature extraction, index
[22], Xia et al. proposed a privacy-preserving CBIR scheme. generation, and search operation, are outsourced to the
Four MPEG descriptors were used to represent the images. cloud server. However, the global histogram is too rigid for
Secure k NN algorithm was employed to protect the image image retrieval problems. Liu et al. [27] tried to improve
features. Locality-sensitive hash was used to increase the the retrieval accuracy with difference histogram. But the
search efficiency. In addition, the authors incorporated an improvement is not significant. In this paper, we propose
encryption-domain watermarking method to the scheme so an outsourced CBIR scheme based on secure local features.
as to deter the image users’ illegal distribution. In [23], The proposed scheme also outsources the tasks of feature
Zhang et al. proposed a secure outsourced CBIR scheme extraction, index construction, and search operation to the
with fine-grained access control. A key-agent was intro- cloud server, meanwhile achieving much higher retrieval
duced to identify which images can be accessed by a user. precision by using the proposed BOEW model.

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 3

3 S YSTEM OVERVIEW AND PRELIMINARIES of the visual words. Specifically, the BOW model has three
3.1 System model steps:
(i) Local feature extraction. The first step is to extract
Similar to [28], the proposed scheme involves two types of
local features from the images in database. Some typical
entities, i.e., the image owner and the cloud server, as shown
local features include SIFT [36] and SURF [37], which are
in Fig. 1.
invariant to illumination, deformation, and rotation. As the
Image owner has a large-scale image database, I =
development of cloud computing, some researchers have
{Ii }ni=1 with the corresponding identity set ID = {IDi }ni=1 ,
designed several secure SIFT feature extraction methods
to be outsourced for cost saving and convenient utilization.
which can calculate the SIFT features from the particularly
To preserve privacy, the image database needs to be encrypt-
encrypted images. However, the extracted features are also
ed before being uploaded, generating an encrypted image
the encrypted ones and cannot be directly employed for
set C = {Ci }n i=1 . Besides the image encryption, the image the image retrieval [38], [39]. For efficiency and security, we
owner would like to outsource the computation and storage
calculate the color histograms from the image blocks as local
tasks to the cloud server as much as possible.
features.
After storing the images on the cloud server, the image
(ii) Vocabulary construction. The second step is to con-
owner may want to retrieve the desired images which are
struct the vocabulary. Typically, k -means method can be
similar to a query image. In the query phase, the image
employed to cluster the local features into k classes. The
owner in our scheme needs only to encrypt the query
cluster centers are defined as visual words. The full set of
image and then uploads it. The feature extraction and search
the visual words constitute the vocabulary.
operation are finished by the cloud server. (iii) Histogram calculation. The last step is to calculate
Cloud server stores the encrypted images for the image the histogram of visual words. Please note that, all the
owner and provides CBIR service for the image owner. In local features are represented by their nearest cluster centers
addition to the search operation, the cloud server in our (visual words). Finally, each image is represented by a k -bins
scheme also takes charge of the index generation. histogram of visual words.

3.2 Security model 3.3.2 Color space

Similar to the previous SE schemes [12], [16], [28], [29], [30], Color images can be displayed in many color spaces. In
[31], an honest-but-curious cloud server is considered in this paper, we describe our system with HSV color model
our scheme. That is to say, the cloud server follows the as [3], [4]. HSV is a cylindrical-coordinate representation
designated protocol specification correctly, but may pre- of the points in RGB color model. HSV stands for hue,
serve and analyze the history of communication to acquire saturation, and value. In matlab, the components of HSV
sensitive information. In this paper, we do not consider the are measured by the decimals in [0, 1]. Here we quantize
information leakage due to the access pattern. In addition, them into integers in [0, 100] as [3], [4] for the convenience
if images Ii and Ij are clustered together or return as the of local histogram calculation. In our experiments, we also
search results to the same query, it is not difficult to deduce test our scheme in RGB and YUV color spaces.
that images Ii and Ij are similar to each other. This kind of
information leakage is also not considered in this paper as 4 T HE P ROPOSED SCHEME
many previous works [3], [4], [13], [14], [15], [17], [18], [19], 4.1 Overview of the proposed scheme
[21], [22], [24], [25].
The proposed scheme includes six algorithms which are
respectively executed by two entities, i.e., KeyGen, ImgEnc,
3.3 Preliminaries TrapGen, and ImgDec executed by the image owner, and
3.3.1 Bag-of-Word model IndexGen and Search executed by the cloud server.
CBIR methods extract visual features to represent the im- First of all, given an image database I = {Ii }n i=1 , the
ages. Then, the similarity between images will be measured image owner runs KeyGen to generate a set of secret keys K.
by the distance between feature vectors. The features can Then, the owner runs ImgEnc to encrypt the image database,
be extracted either globally from the entire image or locally generating an encrypted image database C = {Ci }n i=1 .
from the small regions. Generally, the local features are more The encrypted image database C is uploaded to the cloud
robust and likely to retrieve more accurate results [32]. One server. Upon the receipt of encrypted image database C , the
popular CBIR approach using local features is the bag-of- cloud server runs IndexGen to build a secure index Idx. To
words (BOW) model [33]. retrieve similar images, the image owner runs TrapGen to
BOW model firstly appeared in the field of natural generate a trapdoor and then submits it to the cloud server.
language processing and information retrieval. The model Once receiving the query request from the image owner, the
regards the document as a collection of words, ignoring the cloud server runs Search to find θ similar images which
word order and grammar of the documents. Afterward the are returned to the image owner as the search results. After
BOW model is widely applied in image retrieval [34], [35]. receiving the search results, the image owner runs ImgDec
In this model, local features are extracted from images in the to decrypt the retrieved images. A summarization of the
database, and then are jointly clustered. The cluster centers algorithms is presented in Fig. 2.
are defined as “visual words” to form the vocabulary. Then,
all the local features are represented by its nearest visual 4.2 Privacy-preserving CBIR protocol
word and an image can be finally denoted by a histogram Detailed algorithm processes are as following.

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 4

Image owner side:

• K ← KeyGen(1κ ) is the key secret generation algo- pmtb ← RandPerm(keyb , [1...blknum], ID). (2)
rithm which takes as input the security parameter κ,
and returns the secret key K. We apply the same block permutation secret key to the
• C ← ImgEnc(K, I, ID, blksize) is the image encryp- three components in an image, but different keys to different
tion algorithm which takes as inputs the secret key
images for better security without any influence on retrieval
K, the image database I , identity set of images ID,
and the predefined block size blksize, and returns accuracy. In the process of calculation, the ID and the secret
the encrypted image database C . key keyb are merged to be the seed of the pseudo-random
• Iq ← ImgDec(K, Rq , blksize) is the decryption al- permutation generator.
gorithm which takes as inputs the secret key K, the The secret keys {keyp ∗ }∗∈{H,S,V } are used to generate
set of result images Rq and block size blksize, and random permutations to shuffle pixels in blocks. The ran-
returns the decrypted image set Iq . dom permutations of three components are generated as
• TD ← TrapGen(K, Iq , blksize) is the trapdoor gener- follows:
ation algorithm which takes as inputs the secret key
K, the query image Iq and the block size blksize, and
returns the trapdoor TD. pmtp∗j ← RandPerm(keyp ∗ , [1...blksize], ID, j), (3)
Cloud server side:
where ∗ ∈ {H, S, V }. Here we generate different keys for
• Idx ← IndexGen(C, blksize) is the index generation different image blocks. This increases the security but has
algorithm which takes as inputs the image database
no influence on the local histogram calculation, and thus
C and the block size blksize, and returns the index
Idx. will not influence the image retrieval.
• Rq ← Search(C, Idx, TD, blksize) is the search algo- Among the secret keys above, the RandPerm, blksize
rithm which takes as inputs the encrypted database and keyv ∗ are needed to request an valid image query.
C , the index Idx, the trapdoor TD and the block size After receiving the encrypted images, all the secret keys are
blksize, and returns the encrypted similar images needed for image decryption.
Rq .
4.2.2 Image encryption
Fig. 2: Overview of the algorithms in the proposed scheme C ← ImgEnc(I, ID, K, blksize). As presented above, there
are three steps in the image encryption including color
value substitution, block permutation, and intra-block pixel
4.2.1 Key Generation
permutation. For each step, we present a sub-algorithm to
K ← KeyGen(1κ ). Images are made up of two kinds specify its process (see Algorithm 1, 2, and 3).
of information, i.e., color and texture information, both As shown in Algorithm 1, we utilize a polyalphabetic
of which should be protected properly [3], [4]. In our cipher to encrypt the color values. Polyalphabetic cipher
scheme, color information is protected by substituting the encrypts the values by substituting values with multiple
color values, while the texture information is protected by tables which in this paper are the permutations of the
shuffling the pixel positions. This scheme processes the elements in the set {0,..., 255}. In this way, the same pixel
images in HSV color space. To protect the image content, value at different positions can be substituted with different
we firstly substitute the color values in three components values, which helps to resist the statistic attack [40], [41]. As
with three different secret keys. Next, the image is divid- presented by Algorithm 2, and 3, for block permutation and
ed into non-overlapping blocks which are then shuffled. intra-block permutation, we generate random permutations
Finally, the pixels in each block are shuffled to further to shuffle the block position and intra-block pixel position. It
protect the texture information. Accordingly, we generate is quite straightforward. Finally, with these sub-algorithms,
a pseudo-random permutation generator and several secret the whole process of the image encryption is defined as in
keys for the whole encryption process, i.e., K = {RandPerm, Algorithm 4.
{keyv ∗ }∗∈{H,S,V } , keyb , {keyp ∗ }∗∈{H,S,V } }.
Here, the secret keys {keyv ∗ }∗∈{H,S,V } are utilized to 4.2.3 Index generation
generate random permutations to substitute color values
in H, S, V components, respectively. The random permu- Idx ← IndexGen(C, blksize). In our scheme, the construc-
tations are generated from the range [0...100] as follows: tion of index is also outsourced to cloud server so as to
further reduce the burden of the image owner. Inspired by
the famous BOW model, we propose a novel BOEW model
{pmtv ∗,# } ← RandPerm(keyv ∗ , [0...100]), (1) to support the image retrieval while preserving the image
privacy. Similar to the BOW model, our method consists of
where ∗ ∈ {H, S, V }, # ∈ {1, ..., Npmt }, Npmt is amount three steps.
of the random permutations for each color component. (i) Local histogram extraction. Firstly, the encrypted images
The secret key keyb is used to generate pseudo-random are divided into non-overlapping blocks as that in the image
permutations from the range [1...blknum], where blknum encryption process. Next, we calculate a local histogram
is the total number of non-overlapping blocks in an image. from each block, and denote the local histogram of j -th
Here, the random permutation generated by keyb is used to block in i-th image as hij = (bijτ )303 τ =1 , where i ∈ {1, ..., n},
shuffle image blocks and is generated as follow: j ∈ {1, ..., blknumi }. It is worth noting that the local

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 5

Algorithm 1 ValueSubstitution Algorithm 3 IntrablockPermut

Input: Image I and secret keys {keyv ∗ }∗∈{H,S,V } Input: I ′′ , ID, {keyp ∗ }∗∈{H,S,V } , blksize
Output: I ′ Output: C
1: Generate the secret permutations pmtv H,# , pmtv S,# , 1: Denote the final encrypted image as C whose compo-
and pmtv V,# , where # ∈ {1, ..., Npmt } ; nents are denoted as CH , CS , and CV . Divide the com-
2: Generate three random sequences sqntH , sqntV , sqntS . ponents CH , CS , and CV into non-overlapping blocks
c
The length of sequences are equal to the pixel amount denoted by blkH , blkSc , and blkVc ;
of the image I and the elements of these sequences are 2: Divide the components of image I ′′ into non-
′′
within the set [1, ..., Npmt ]; overlapping blocks denoted by blkH , blkS′′ , and blkV′′ ;
3: Denote pi as ith pixel in image I , and pi H , pi S , pi V as 3: for ∀∗ ∈ H, S, V do
the three components of the pixel; 4: for ∀ blk ′′ ∗j ∈ blk∗′′ do
4: Denote p′i as the corresponding pixel in the encrypted 5: Generate the secret permutation for j -th block
image I ′ , and p′iH , p′iS , p′iV as the three components of blk ′′ ∗j as pmtp ∗j ;
′′
the pixel; 6: for ∀ blk∗j [i] ∈ blk ′′ ∗j do
5: for each p′i ∈ I ′ do 7: blk ∗j [i] ← blk ′′ ∗j [pmtp∗j [i]];
c
6: p′iH ← pmtv H,sqntH [i] [pi H ], p′iS ← 8: end for
′
pmtv S,sqntS [i] [pi S ], piV ← pmtv V,sqntV [i] [pi V ] 9: end for
7: end for 10: end for

Algorithm 2 BlockPermut Algorithm 4 ImgEnc

′
Input: I , ID, keyb , blksize Input: I, ID, K, and blksize,
Output: I ′′ 1: for ∀ Ii ∈ I do
1: /* blksize denotes the size of image blocks, which is an open 2: Ii′ = ValuePermut(Ii , {keyv ∗ }∗∈{H,S,V } );
parameter in our scheme. */ 3: Ii′′ = BlockPermut(Ii′ , IDi , keyb , blksize);
2: Calculate the total number of non-overlapping blocks in 4: Ci = IntrablockPermut(Ii′′ , IDi , {keyp ∗ }∗∈{H,S,V } ,
image I ′ as blknum = imgsize/blksize; blksize);
3: Generate the secret permutation pmtb; 5: end for
4: Divide the image I ′ into non-overlapping blocks denot-
ed as blk ′ ;
5: Divide the resulting image I ′′ into the blocks denoted as the feature vector make up a linear index as shown in Table
blk ′′ ; 1.
6: for ∀blk ′′ [i] ∈ I ′′ do
7: blk ′′ [i] ← blk ′ [pmtb[i]]; 4.2.4 Trapdoor generation
8: end for
TD ← TrapGen(K, Iq , IDIq , blksize). In our scheme, the
trapdoor is generated by encrypting the query image with
the color value substitution, block permutation, and intra-
histograms in our scheme are the encrypted ones since the block pixel permutation. The encrypted query image is then
color values have been encrypted by the substitution. uploaded to the cloud server as the trapdoor.
Compared to the classic local features, such as SIFT
which is robust to scaling, rotation, affine distortion, and 4.2.5 Search operation
illumination changes, the local histogram has its limitation. Rq ← Search(C, Idx, TD). Upon receiving the trapdoor,
Some researchers proposed to outsource the calculation of i.e., the encrypted query image, the cloud server divides
SIFT features in a privacy-preserving manner [38], [39], [42]. it into blocks and calculates the set of local histograms
blknum
However, these methods need several rounds of commu- {hqj }j=1 q , where blknumq is the total number of blocks
nications between non-collusive cloud servers. In addition, in the query image Iq . Next, a feature vector fq = (fqj )kj=1 is
the applications of the outsourced SIFT in secure CBIR need calculated using the vocabulary and {hqj }. The Manhattan
the additional communication between the server and query distance between fq and the feature vector fi in the index
users, which is a undesirable burden to the users.
(ii) Vocabulary generation. Cluster all the local histograms
{hij } into k classes with the k -means clustering algorithm TABLE 1: The linear index
[43]. The k cluster centers are defined to be the encrypted
Image identity Feature vector
visual words which make up the vocabulary.
(iii) Global feature extraction. After generating the vocabu- ID(C1 ) f1 = {f11 , f12 , ...f1j , ..., f1k }
lary, all the local histograms in an image are represented by ... ...
their nearest visual words. Then, the occurrence histogram ID(Ci ) fi = {fi1 , fi2 , ...fij , ..., fik }
of the visual words are calculated and normalized to repre-
... ...
sent the image. As a result, each image is represented by a
k ID(Cn ) fn = {fn1 , fn2 , ...fnj , ..., fnk }
feature vector f = (fi )i=1 . Finally, the image identities and

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 6

can be directly calculated. Finally, the θ most similar images of the color information and texture information,
are returned to the image owner as the search results. both of which are contained in all H, S, V compo-
nents. In our scheme, this information is protect-
4.2.6 Image decryption ed respectively by the substitutions and random
Iq ← ImgDec(Rq , K, blksize). The decryption process is an permutations. Specifically, the color information in
exact reverse to the encryption process. Firstly, the intra- H, S, V components are protected by color value
block pixel permutation is reversed. Secondly, the block per- substitution with different secret keys. The color
mutation is reversed. Finally, the color value permutation is information between the real images and simulated
recovered. Here, we do not describe the decryption process ones are indistinguishable according to the property
in detail because it is quite straightforward. of random permutation. For a random permutation
with length of 101, the computational complexity of
4.2.7 Image update a distinguisher D, executed by S , in distinguishing
Idx ← ImgUpdate(Idx, ID, TD, updatetype). When the the color values is 101! because D needs to figure
owner needs to add an image, he encrypts the image and out the correct one from 101! permutations, which
submits the encrypted image and the ID to the cloud server. means a log2 (101!) ≈ 531 bits security strength. The
The server generates the corresponding feature vector as the texture information is protected by block permuta-
trapdoor generation process and adds the pair of image ID tion and intra-block pixel permutations. The secu-
and feature vector to the index. When an image needs to be rity strengths of block permutation and intra-block
deleted, the image owner just notices the server to delete the pixel permutations are equal to log2 (blknum!) and
encrypted image and the corresponding item in the index. log2 (blksize!) bits, respectively. The image content
is made up of all of these information and hence
the security strength of image encryption (SecImg )
5 S ECURITY ANALYSIS in our scheme can be calculated as:
As stated in subsection 3.2, we consider an honest-but-
curious cloud server which follows the protocol correctly, SecImg =3 × Npmt × log2 (101!)
but may analyze the owner’s data to acquire privacy in-
+log2 (blknum!) (4)
formation such as the content of encrypted images and the
image features. The security of the scheme is analyzed under +3 × blknum × log2 (blksize!)(bits)
the ciphertext-only attack (COA), known-background attack • Security of features. In our scheme, image features are
(KBA), and chosen-plaintext attack (CPA) models. calculated from the local histograms of encrypted
color values. In addition, the adversary can obtain
5.1 Security under COA model the encrypted global histogram of an image if he
In the COA model, the adversary is only able to get the wants. With a simulated image I S , S can calculate
ciphertext. For formal statement, the functionality F and the the local and global histograms of the simulated im-
corresponding information leakages of our scheme under age. The computational complexity of a distinguisher
the COA model are summarized in Fig. 3. Our security D in distinguishing the histogram is 3 × Npmt × 101!
proofs follow the paradigm in secure multi-party com- which means a 3 × Npmt × 531 bits security strength.
putations [44]. The execution of our scheme involves the • Security of query image. In our scheme, query image
interaction between cloud server and image owner, which is is encrypted in the same way as images on cloud
defined as the real experiment. In the proposed scheme, the server. Thus the security strength equals to SecImg .
honest-but-curious cloud server is the potential adversary In this scheme, the images are encrypted by the combi-
A. In an ideal experiment, a simulator S is defined as the nation of value substitution, block permutation, and intra-
one that can simulate the view of adversary A by using the block pixel permutation. The image can be well protected,
functionality F only, constructing the ideal experiment. The although the security is related to the image size. Despite
proposed scheme is proved secure once the two experiments the proposed image encryption method cannot be the best
are indistinguishable. one in terms of security, we would like to point that our en-
cryption method is designed to support the direct extraction
Theorem 1. Our scheme is secure against an honest-but-
of features from encrypted images for image retrieval.
curious probabilistic polynomial time (PPT) adversary under
the COA model. The security strength depends on the image
size, block size, and the number of permutations in color value 5.2 Security under KBA model
encryption. In the KBA model, besides the information leakage de-
fined in Fig. 3, the adversary also knows certain statistical
Proof.
properties of natural images, which degrades the security
• Security of image content. The simulator S simulates strength of the proposed scheme. For example, as illustrated
a set of images I S and the corresponding identity in the first subfigure of Fig. 4, color values do not occur
set IDS according to the storage leakage as shown uniformly, and the adjacent color values have the similar
in Fig. 3. The simulator S knows the total number occurrence probabilities. After the encryption with a single
of images and the size of each image. However, S permutation, the histogram bins have been shuffled, which
can only fill the images with the randomly generated protects the image statistical features to some extent. How-
pixels. As stated in subsection 4.2, images consist ever, the distribution statistics are still reserved as shown in

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 7

The ideal functionality F of our scheme as well as information leakages.

(i) F .StoreImage(I, ID, K, blksize):
• Functionality. Image owner encrypts all the images in I , and generates a set of encrypted images C . Next,
Image owner uploads C, ID, blksize to the cloud server.
• Storage leakage. The information leaked here includes C, ID, blksize, the size of each images, and the total
number of images.
(ii) F .IndexGen(C, blksize):
• Functionality. Cloud server extracts local histograms from images blocks, and constructs the vocabulary by
cluster algorithm, and calculates a feature vector for each image in C .
• Feature leakage. The information leaked here includes the encrypted local histograms, the similarities between
the local histograms, and the distributions of local histograms. What’s more, the cloud server can calculate the
encrypted global histogram of image if he wants.
(iii) F .Query(Iq ):
• Functionality. Image owner encrypts the query image, and submits the encrypted image to cloud server as
query trapdoor. After searching on the index, the cloud server returns the most similar images to the image
owner.
• Query leakage. The information leaked here includes the encrypted query image and the similarity between
the images in the database.

Fig. 3: The functionality F and information leakage in our framework

the second subfigure of Fig. 4, which decreases the compu- 5.4 Access pattern
tational complexity to figure out the secret permutation. In Access pattern is a high-lever privacy compared with the
our scheme, the polyalphabetic cipher is utilized to encrypt data confidentiality. The consequences caused by access pat-
the pixel values, which will flatten the color value histogram tern leakage depend on the application scenario. In an image
of the encrypted image and thus offering stronger security retrieval scheme, the cloud server knows which images
[40], [41]. In this paper, we calculate the histogram entropies are retrieved by the image owner by analyzing the access
of an original image and its encrypted versions generated pattern. Generally, the researchers do not consider the access
using different number of permutations Npmt as shown in pattern leakage in the design of an SSE scheme [3], [4], [9],
Fig. 5. The higher entropy is received with larger Npmt , [12], [13], [14], [15], [16], [17], [18], [19], [21], [22], [24], [25],
which indicates higher uniformity, i.e., better security [40], [29], [30], [31].
[41]. However, there is a trade-off between the security and Oblivious RAM techniques have been researched for
retrieval accuracy, which will be shown in next section. many years to deal with access pattern leakage, but usually
suffers to huge computation and communication burdens
5.3 Security under CPA model [47], [48]. It is possible to incorporate Oblivious RAM tech-
In the CPA model, besides the information leakage defined niques into the SE scheme to mitigate the access pattern
in Fig. 3, the adversary can obtain the ciphertexts of ar- leakage. But the computation and communication burdens
bitrary plaintexts. In fact, all the permutation-only image are big problems for the image retrieval.
ciphers can be completely broken by the chosen-plaintext
attack if images are encrypted by the same secret key 6 E XPERIMENTAL RESULTS
[45]. But in our scheme, the images are encrypted by the
This section evaluates the performance of the proposed
combination of position permutation and pixel substitution.
scheme in terms of encryption effectiveness, retrieval ac-
The secret keys of position permutation are distinct for each
curacy, and efficiency. We implement the proposed scheme
image. In addition, the pixel position is not shuffled by
with MatLab 2017a on a Win-10 operation system with Intel
rows and columns which significantly degrades the security
Core (TM) i7-6900K CPU 3.20 GHz and 64 GB memory.
[45]. Thus, the images encrypted in our scheme cannot be
Inria Holidays database [49] is used as the experiment
recovered even under the CPA model. However, in order to
database which contains 1491 color images (2.65 GB) in
support the image retrieval, the secret key of the color value
500 categories. Inria Holidays database provides a Python
substitution is the same for all images, and this secret key
evaluation package to calculate mAP and was used in
is unsafe under the CPA model [46]. This will cause infor-
privacy-preserving image retrieval schemes [4], [26], which
mation leakage about image content. With the secret key of
facilitates performance comparison. For clarity, we symbol-
color value substitution, the cloud server can generate the
ize several important parameters as shown in Table 2.
valid trapdoor for the query image chosen by itself. Then,
the cloud server can search the encrypted image database
with the trapdoor, and all the retrieved images may contain 6.1 Effectiveness of image encryption
the similar content with the chosen query image. In this way, In our scheme, the images are encrypted by color value sub-
the cloud server can figure out the image content to some stitution, block permutation, and intra-block pixel permuta-
extent. tion. Fig. 6 illustrates the separate and joint effect of the three

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 8

8.2

Original image 8
0.1
Ratio

7.8
0.05

Entropy
0 7.6
0 50 100 150 200 250 300
7.4
Npmt=1
0.1 7.2
Ratio

0.05 7
0
0 50 100 150 200 250 300 6.8
1 2 3 4 5 10 20 50 100 200 500
Npmt=5 N
pmt
0.1
Ratio

0.05
0 Fig. 5: Entropies of global histograms of encrypted images
0 50 100 150 200 250 300 using different Npmt .
Npmt=10
0.1 TABLE 2: Symbol of parameters
Ratio

0.05
0 Parameters Symbol
0 50 100 150 200 250 300
Size of block blksize
Npmt=20
0.1 Number of cluster centers k
Ratio

0.05 Total number of local histograms blknumT

0
0 50 100 150 200 250 300 Number of used local histograms in clustering blknumU
Npmt=50 Number of permutations in color value substitution Npmt
0.1
Ratio

0.05
0
0 50 100 150 200 250 300 6.2 Retrieval accuracy
Npmt=100 In our experiment, mean average precision (mAP) is used to
0.1 measure the retrieval accuracy. The Python evaluation pack-
Ratio

0.05 age of Inria Holidays Dataset is directly used to calculate the

0
0 50 100 150 200 250 300 mAP of the proposed method for fair comparison. Firstly,
Npmt=500 we use only one permutation in color value encryption,
0.1 i.e., Npmt = 1, to determine the proper parameters such as
Ratio

0.05 blksize and k . Next, we test our scheme with different Npmt
0 as the larger Npmt can protect the image content better but
0 50 100 150 200 250 300
may decrease the retrieval accuracy.
Values in HSV components

6.2.1 Accuracy with one permutation in color value encryp-

tion
The retrieval accuracy of the proposed BOEW-CBIR scheme
Fig. 4: Occurrence ratios of color values of an original image mainly depends on two parameters, i.e., the size of the
and its encrypted versions with different Npmt . It can be block (blksize) and the number of cluster centres (k ). We
observed that the histogram gets more flattened if the image test our scheme under a large range of parameter settings.
is encrypted with more permutations. As shown in Table 3, our scheme achieves the highest mAP
with blksize set to be 20 × 20 and k set to be 3000. With
setting of blksize = 20 × 20 and k = 3000, we compare
our scheme with the scheme in [4] that uses the secure
global histogram and the scheme in [26] based on partial
protecting steps. As shown in Fig. 6 (b), with a small blksize encryption. In addition, we implement our scheme in RGB
in a big image, the intra-block pixel permutation does not and YUV color space with this setting. As shown in Table 4,
protect the image content a lot. The image is blurred, but the the proposed scheme also performs well in YUV and RGB
content can be almost fully learned. Conversely, the block color spaces. Compared to the scheme using secure global
permutation makes a good randomization to the image with histogram [4], our scheme increases the mAP from 0.54564
a small blksize (see Fig. 6 (c)). The color value substitution to 0.64244 in HSV color model, which is a considerable
with Npmt = 1 has fully permuted the color information improvement.
but leaves much texture information revealed. The image
content can be further blurred by increasing the number 6.2.2 Accuracy with multiple permutations in color value
of permutations as show in Fig. 6 (d-f). Finally, the image encryption
content can be well protected by the combination of the In this paper, we use multiple permutations to protect the
three steps (see Fig. 6 (g-h)). color values to resist the statistic attack. However, this may

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 9

TABLE 3: The mAP of our scheme with different parameter pairs

blksize
k
15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200 Average
100 0.57095 0.57713 0.57907 0.57099 0.55027 0.50554 0.55899
500 0.62097 0.62816 0.63544 0.62718 0.60810 0.56265 0.61375
1000 0.63398 0.63161 0.62967 0.63117 0.60523 0.58453 0.61937
2000 0.63473 0.62966 0.63059 0.63280 0.61689 0.57699 0.62028
3000 0.63080 0.64244 0.63613 0.63039 0.60846 0.57851 0.62112
4000 0.62960 0.63583 0.63647 0.64002 0.62398 0.58542 0.62522
5000 0.63293 0.63715 0.63510 0.63089 0.62129 0.57146 0.62147
8000 0.63170 0.63626 0.64046 0.63771 0.62205 0.59026 0.62641
Average 0.62321 0.62728 0.62787 0.62514 0.60703 0.56942 0.61333

TABLE 4: The mAPs of two previous schemes and our scheme in different color spaces

Schemes BOEW-HSV BOEW-YUV BOEW-RGB IES [4] SSE [13] reported in [4] Partial-encryption based scheme [26]
mAP 0.64244 0.62641 0.61107 0.54564 0.49075 0.5604

degrade the retrieval accuracy. Here, we set blksize = block. It takes blknumT × O(blksize) time for the
20 × 20 and k = 3000 and test the retrieval accuracy with local histogram calculation. Table 7 shows more time
different Npmt . As shown in Fig. 7, the accuracy does not is consumed with a smaller blksize to extract local
decrease a lot when we increase Npmt up to 50. histogram from the whole Inria holiday database.
• Time consumption of local histogram clustering. In our
6.3 Efficiency implementation, the built-in k -means function in
In this section, we present the time consumptions of image MatLab is used to cluster the local histograms. The
encryption, index generation, and search operation. time cost of the k -means algorithm depends on the
predefined number of class k , the number of used
6.3.1 Time consumption of image encryption local features blknumU , and the number of itera-
In the proposed scheme, it takes O(imgsize) time for color tions t. The time complexity of k -means algorithm
value substitution, O(blknum) for block permutation, and is O(k × blknumU × t). In our experiment, we set
blknum × O(blksize) for intra-block pixel permutation to the maximal iteration times to be 300. When we set
encrypt an image. In this paper, we separately test the blksize to be 20 × 20, it generates 17,099,758 local
time consumptions of color value substitution with different histograms from Inria Holiday database. Our scheme
Npmt and time consumptions of pixel position permutation runs out of memory (64GB) when we use all of these
(including block and intra-block permutations) with differ- local histograms. Thus, in this experiment, we only
ent blksize, which are listed in Table 5 and 6. As listed in use a part of local histograms to generate the cluster
Table 5, it does not increase time consumption a lot to use centers.
multiple color value permutations, but as listed in Table 6, At first, we set a similar blknumU for different
more time is consumed with smaller blksize in the position blksize so as to figure out proper parameters blksize
permutations. The image encryption is very time consum- and k that obtain good retrieval accuracies. In Ta-
ing. However, the proposed image encryption method is ble 8, we list the total number of local histograms
much more efficient than some standard encryption meth- blknumT when different blksize are set. We also list
ods. For example, it takes 548.29s to encrypt the first image the number of used local histograms blknumU in
(3942KB) in Inria Holidays database using DES. Note that clustering. Finally, the time consumption of cluster-
the previous outsourced CBIR schemes [13], [14], [15], [17], ing with different blksize and k are given in Table
[18], [19], [21], [22], [24], [25] need to encrypt images and 8. Next, the influence of blknumU on our scheme
upload encrypted ones to the cloud server, which is also is tested when blksize is set to 20 × 20 as shown
quite time-consuming. in Table 9. Table 8 and 9 shows that more time
is consumed with larger blknumU and k . Table 9
6.3.2 Time consumption of index generation shows that the retrieval accuracy does not decrease
In the proposed scheme, feature generation includes three a lot even though a very small part (1/5000) of local
steps, local histogram calculation, clustering of local his- histograms are used for clustering.
tograms, and generation of feature vectors. The time con- • Time consumption of feature generation. After histogram
sumptions of these steps are separately tested. clustering, a normalized histogram of the encrypted
words is calculated from each image as the feature
• Time consumption of local histogram calculation. In the vector. The time complexity of feature vector calcula-
proposed scheme, the cloud server divides the im- tion is O(blknumT ×k) for the entire image database.
ages into blocks and extracts a histogram from each

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 10

TABLE 5: Time consumptions in color value substitution with different Npmt

Npmt 1 2 3 4 5 10 20 50 100 200 500

Time consumption (s) 5390.60 5478.60 5412.40 5387.60 5369.20 5370.70 5375.90 5424.60 5463.50 5643.00 5889.80

TABLE 6: Time consumptions in block and intra-block permutations with different blksize

blksize 15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200

Time consumption (s) 5437.90 3869.40 3149.80 2185.80 1949.90 1796.60

TABLE 7: Time consumptions in local histogram calculation

blksize 15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200

Time consumption (s) 618.40 551.41 467.43 383.58 350.84 332.51

TABLE 8: Time consumptions in the clustering with different blksize and k (s)

blksize
15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200
blknumT 30406710 17099758 10887924 2707247 671176 158828
blknumU 76844 171252 170763 169546 167916 158828
blknumU/blknumT 1/400 1/100 1/64 1/16 1/4 1
100 88.11 226.64 445.03 310.68 447.44 415.88
500 219.19 659.61 880.04 707.60 597.67 559.11
1000 231.34 872.96 862.16 836.25 859.65 791.93
2000 306.92 1158.16 1095.38 1119.35 1087.68 988.78
k
3000 329.95 1431.94 1421.01 1359.65 1348.08 1252.13
4000 382.25 1684.34 1573.37 1599.92 1570.11 1500.51
5000 431.61 1915.47 1904.63 1843.33 1806.84 1694.72
8000 576.66 2625.00 2591.51 2538.60 2523.75 2362.38

The time consumption of global feature generation under the proposed BOEW model. It depends on how the
for Inria Holiday dataset with different parameters intra-block pixels are processed and encrypted. For exam-
are listed in Table 10. ple, one may extract and encrypt the gradient information
from image blocks as local features. The utilization of texture
6.3.3 Time consumption of searching information may achieve better retrieval accuracy. However,
With the linear index, it needs to search on the whole index the information leakage is a problem to solve. Secondly,
to find the θ most similar images. The time complexity of our scheme is constructed on the basis of uncompressed
search is O(k × n). In our scheme, the length of feature images. Storage consumption could be a practical problem.
vector equals to the number of cluster center k . Table 11 According to [3], [4], the encrypted image can also be
lists the searching time of our schemes with different k . The compressed to reduce the storage cost despite the correla-
results are averaged from 500 queries. tion among the pixels is destroyed. We do not discuss the
image compression a lot in this paper. Finally, it could be a
6.4 Discussion meaningful work to apply the BOEW model to the retrieval
The proposed scheme is robust to parameters. It achieves of encrypted JPEG images.
a good retrieval accuracy as long as we choose blksize in
the range of 15 × 15 to 100 × 100 and k in range [500, 7 C ONCLUSIONS
8000]. In addition, our scheme can get a satisfying retrieval In this paper, a novel privacy-preserving CBIR scheme is
accuracy even though a very small part of local histograms proposed. A novel bag-of-encrypted-words (BOEW) model
are used for clustering. Compared to the previous privacy- is designed to achieve a good retrieval accuracy. As a case
preserving CBIR schemes [13], [14], [15], [17], [18], [19], [21], study, we protect the image content by color value substitu-
[22], [24], [25], our scheme outsources the index construction tion, block permutation, and intra-block pixel permutation.
to the cloud server, which further relieves the image owner’s Local histograms are calculated as local features. k -means
burden. Compared to the scheme using the global histogram algorithm is utilized to generate encrypted visual words.
[3], [4], our scheme improve the mAP from 0.56544 to The histogram of the visual words is calculated to represent
0.64244. It is a considerable improvement. the image. The similarity between images can be directly
The proposed scheme can be improved from several measured by the Manhattan distance between feature vec-
aspects. Firstly, it is possible to extract better local features tors on the cloud server side. Besides the search operation,

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 11

TABLE 9: Performance of the proposed scheme with different number of used local histogram, blksize = 20 × 20,
blknumT = 17, 099, 758

blknumU/blknumT 1/100 1/200 1/500 1/1000 1/2000 1/5000

Time consumption in clustering (s) 1381.84 407.86 136.49 68.44 39.82 20.71
mAP 0.64244 0.63902 0.6282 0.63519 0.63254 0.62967

TABLE 10: Time consumptions in the global feature generation (s)

blksize
15 × 15 20 × 20 25 × 25 50 × 50 100 × 100 200 × 200
100 301.24 109.66 67.97 23.05 14.79 11.13
500 735.27 343.93 214.13 61.94 23.32 15.15
1000 1210.27 636.99 401.51 108.95 36.44 17.85
2000 2196.60 1170.73 768.25 202.83 61.68 25.40
k
3000 3209.22 1733.15 1110.68 294.43 87.06 33.98
4000 4207.75 2299.12 1466.80 388.36 112.16 42.75
5000 5204.52 2858.70 1826.36 483.31 135.73 50.20
8000 8284.66 4546.23 2906.85 751.16 211.83 74.05

TABLE 11: Time consumptions of searching at the server side with linear index (s)

k 100 500 1000 2000 3000 4000 5000 8000

Time 0.0044 0.0087 0.0119 0.0211 0.0300 0.0381 0.0464 0.0704

the index construction in our scheme can be also outsourced [2] C. S. Lu, “Homomorphic encryption-based secure sift for privacy-
to the cloud server. preserving feature extraction,” Proceedings of SPIE The International
Society for Optical Engineering, vol. 7880, no. 2, pp. 788 005–17, 2011.
The proposed scheme can be further improved. Firstly, [3] B. Ferreira, J. Rodrigues, J. Leitão, and H. Domingos, “Privacy-
it could be a meaningful future work to design better preserving content-based image retrieval in the cloud,” in IEEE
local descriptors under our BOEW model. Secondly, how to 34th Symposium on Reliable Distributed Systems. IEEE, 2015, pp.
protect the image content under CPA model needs further 11–20.
[4] B. Ferreira, J. Rodrigues, J. Leitao, and H. Domingos, “Practical
studies. Finally, it could be interesting to apply the BOEW privacy-preserving content-based retrieval in cloud image reposi-
model to JPEG images. tories,” IEEE Transactions on Cloud Computing, vol. PP, no. 99, pp.
1–1, 2017.
[5] Y. Rui, T. S. Huang, M. Ortega, and S. Mehrotra, “Relevance feed-
ACKNOWLEDGMENTS back: a power tool for interactive content-based image retrieval,”
IEEE Transactions on Circuits and Systems for Video Technology, vol. 8,
This work is supported in part by the National Nat- no. 5, pp. 644–655, 1998.
ural Science Foundation of China under grant num- [6] Y. Liu, D. Zhang, G. Lu, and W.-Y. Ma, “A survey of content-based
image retrieval with high-level semantics,” Pattern Recognition,
bers 61672294, 61502242, 61702276, U1536206, U1405254, vol. 40, no. 1, pp. 262–282, 2007.
61772283, 61602253, 61601236, and 61572258, in part by [7] C. B. Akgül, D. L. Rubin, S. Napel, C. F. Beaulieu, H. Greenspan,
Six peak talent project of Jiangsu Province (R2016L13), in and B. Acar, “Content-based image retrieval in radiology: current
status and future directions,” Journal of Digital Imaging, vol. 24,
part by National Key R&D Program of China under grant
no. 2, pp. 208–222, 2011.
2018YFB1003205, in part by NRF-2016R1D1A1B03933294, [8] X. Zhang, W. Liu, M. Dundar, S. Badve, and S. Zhang, “Towards
in part by the Jiangsu Basic Research Programs-Natural large-scale histopathological image analysis: Hashing-based im-
Science Foundation under grant numbers BK20150925 and age retrieval,” IEEE Transactions on Medical Imaging, vol. 34, no. 2,
pp. 496–506, 2015.
BK20151530, in part by the Priority Academic Program
[9] R. Curtmola, J. Garay, S. Kamara, and R. Ostrovsky, “Searchable
Development of Jiangsu Higher Education Institutions (PA- symmetric encryption: Improved definitions and efficient con-
PD) fund, in part by the Collaborative Innovation Center structions,” Journal of Computer Security, vol. 19, no. 5, pp. 79–88,
of Atmospheric Environment and Equipment Technology 2011.
[10] J. Li, Q. Wang, C. Wang, N. Cao, K. Ren, and W. Lou, “Fuzzy
(CICAEET) fund, China. Zhihua Xia is supported by BK21+ keyword search over encrypted data in cloud computing,” in 2010
program from the Ministry of Education of Korea. Proceedings IEEE INFOCOM. IEEE, 2010, pp. 1–5.
[11] N. Cao, C. Wang, M. Li, K. Ren, and W. Lou, “Privacy-preserving
multi-keyword ranked search over encrypted cloud data,” IEEE
R EFERENCES Transactions on parallel and distributed systems, vol. 25, no. 1, pp.
222–233, 2013.
[1] J. M. Lewin, R. E. Hendrick, C. J. D’Orsi, P. K. Isaacs, L. J. [12] Z. Xia, X. Wang, X. Sun, and Q. Wang, “A secure and dynamic
Moss, A. Karellas, G. A. Sisney, C. C. Kuni, and G. R. Cutter, multi-keyword ranked search scheme over encrypted cloud data,”
“Comparison of full-field digital mammography with screen-film IEEE Transactions on Parallel and Distributed Systems, vol. 27, no. 2,
mammography for cancer detection: results of 4,945 paired exam- pp. 340–352, 2016.
inations.” Radiology, vol. 218, no. 3, pp. 873–80, 2001. [13] W. Lu, A. Swaminathan, A. L. Varna, and M. Wu, “Enabling search

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 12

0.65

0.6

mAP
0.55

0.5

0.45
(a) (b) 1 2 3 4 5 10 20
Npmt

Fig. 7: The mAPs with different Npmt

privacy-preserving image-centric social discovery,” in Distributed

Computing Systems (ICDCS), 2014 IEEE 34th International Conference
on. IEEE, 2014, pp. 198–207.
[17] Z. Xia, Y. Zhu, X. Sun, Z. Qin, and K. Ren, “Towards privacy-
preserving content-based image retrieval in cloud computing,”
(c) (d) IEEE Transactions on Cloud Computing, vol. 6, no. 1, pp. 276–286,
2018.
[18] J. Yuan, S. Yu, and L. Guo, “Seisa: Secure and efficient encrypted
image search with access control,” in 2015 IEEE Conference on
Computer Communications (INFOCOM). IEEE, 2015, pp. 2083–2091.
[19] H. Cheng, X. Zhang, J. Yu, and F. Li, “Markov process-based re-
trieval for encrypted jpeg images,” EURASIP Journal on Information
Security, vol. 2016, no. 1, pp. 1–9, 2016.
[20] L. Weng, L. Amsaleg, A. Morton, and S. Marchand-Maillet, “A
privacy-preserving framework for large-scale content-based infor-
mation retrieval,” IEEE Transactions on Information Forensics and
Security, vol. 10, no. 1, pp. 152–167, Jan 2015.
(e) (f) [21] L. Weng, L. Amsaleg, and T. Furon, “Privacy-preserving out-
sourced media search,” IEEE Transactions on Knowledge and Data
Engineering, vol. 28, no. 10, pp. 2738–2751, 2016.
[22] Z. Xia, X. Wang, L. Zhang, Z. Qin, X. Sun, and K. Ren, “A privacy-
preserving and copy-deterrence content-based image retrieval
scheme in cloud computing,” IEEE Transactions on Information
Forensics and Security, vol. 11, no. 11, pp. 2594–2608, 2016.
[23] L. Zhang, T. Jung, K. Liu, X. Y. Li, X. Ding, J. Gu, and Y. Liu, “Pic:
Enable large-scale privacy preserving content-based image search
on cloud,” IEEE Transactions on Parallel and Distributed Systems,
vol. 28, no. 11, pp. 3258–3271, Nov 2017.
[24] R. Bellafqira, G. Coatrieux, D. Bouslimi, and G. Quellec, “Content-
based image retrieval in homomorphic encryption domain,” in
(g) (h) 2015 37th Annual International Conference of the IEEE Engineering in
Medicine and Biology Society (EMBC). IEEE, 2015, pp. 2944–2947.
Fig. 6: The visual effect of encryption with blksize = [25] ——, “An end to end secure cbir over encrypted medical
20 × 20, (a) the original image (100301.jpg in Inria Holidays database,” in Engineering in Medicine and Biology Society (EMBC),
database), the size of which is 3264 × 2448, (b) with intra- 2016 IEEE 38th Annual International Conference of the. IEEE, 2016,
pp. 2537–2540.
block pixel permutation only, (c) with block permutation [26] Y. Xu, J. Gong, L. Xiong, Z. Xu, J. Wang, and
only, (d) with color value substitution only under Npmt = 1, Y. qing Shi, “A privacy-preserving content-based image
(e) with color value substitution under Npmt = 5, (f) with retrieval method in cloud environment,” Journal of Visual
color value substitution under Npmt = 20, (g) with the Communication and Image Representation, vol. 43, pp. 164 –
172, 2017. [Online]. Available: http://www.sciencedirect.com/
combination of the three steps under Npmt = 1, and (h) science/article/pii/S104732031730007X
with the combination of the three steps under Npmt = 20. [27] D. Liu, J. Shen, Z. Xia, and X. Sun, “A content-based image
retrieval scheme using an encrypted difference histogram in cloud
computing,” Information, vol. 8, no. 3, 2017. [Online]. Available:
http://www.mdpi.com/2078-2489/8/3/96
over encrypted multimedia databases,” Proceedings of SPIE The [28] T. Hoang, A. A. Yavuz, and J. G. Merchan, “A secure searchable
International Society for Optical Engineering, vol. 7254, pp. 725 418– encryption framework for privacy-critical cloud storage services,”
11, 2009. IEEE Transactions on Services Computing, 2019.
[14] W. Lu, A. L. Varna, A. Swaminathan, and M. Wu, “Secure image [29] M. Kuzu, M. S. Islam, and M. Kantarcioglu, “Efficient similarity
retrieval through feature protection,” in IEEE International Confer- search over encrypted data,” in IEEE International Conference on
ence on Acoustics, 2009, pp. 1533–1536. Data Engineering, 2012, pp. 1156–1167.
[15] W. Lu, A. L. Varna, and M. Wu, “Confidentiality-preserving image [30] F. Hahn and F. Kerschbaum, “Searchable encryption with secure
search: A comparative study between homomorphic encryption and efficient updates,” in ACM Sigsac Conference on Computer and
and distance-preserving randomization,” IEEE Access, vol. 2, pp. Communications Security, 2014, pp. 310–320.
125–141, 2014. [31] L. Weng, L. Amsaleg, A. Morton, and S. Marchand-Maillet, “A
[16] X. Yuan, X. Wang, C. Wang, A. Squicciarini, and K. Ren, “Enabling privacy-preserving framework for large-scale content-based in-

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/TSC.2019.2927215, IEEE
Transactions on Services Computing
JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2018 13

formation retrieval,” IEEE Transactions on Information Forensics & Leqi Jiang is a Ph.D. student in School of
Security, vol. 10, no. 1, pp. 152–167, 2015. Computer and Software, Nanjing University of
[32] J. Zhang, M. Marszałek, S. Lazebnik, and C. Schmid, “Local fea- Information Science & Technology. He received
tures and kernels for classification of texture and object categories: his M.S. degree in Software Engineering from
A comprehensive study,” International Journal of Computer Vision, Nanchang Hangkong University in 2016. His re-
vol. 73, no. 2, pp. 213–238, 2007. search interests include encrypted image pro-
[33] D. Nister and H. Stewenius, “Scalable recognition with a vocab- cessing, data security in cloud.
ulary tree,” in Computer vision and pattern recognition, 2006 IEEE
computer society conference on, vol. 2. Ieee, 2006, pp. 2161–2168.
[34] Z. Wu, Q. Ke, J. Sun, and H.-Y. Shum, “A multi-sample, multi-
tree approach to bag-of-words image representation for image
retrieval,” in IEEE 12th International Conference on Computer Vision.
IEEE, 2009, pp. 1992–1999.
[35] J. Wang, Y. Li, Y. Zhang, C. Wang, H. Xie, G. Chen, and X. Gao,
“Bag-of-features based medical image retrieval via multiple as-
signment and visual words weighting.” IEEE Transactions on Med-
ical Imaging, vol. 30, no. 11, pp. 1996–2011, 2011.
[36] D. G. Lowe, “Distinctive image features from scale-invariant key-
points,” International Journal of Computer Vision, vol. 60, no. 60, pp.
Dandan Liu received her BS in network engi-
91–110, 2004.
neering from Nanjing University of Information
[37] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust
Science & Technology in 2015, China. She is
features (surf),” Computer vision and image understanding, vol. 110,
currently pursuing her MS in computer science
no. 3, pp. 346–359, 2008.
and technology at the College of Computer and
[38] Z. Qin, J. Yan, K. Ren, C. W. Chen, and C. Wang, “Towards efficient
Software, in Nanjing University of Information
privacy-preserving image feature extraction in cloud computing,”
Science & Technology, China. Her research in-
in Proceedings of the 22nd ACM International Conference on Multime-
terest is encrypted image retrieval.
dia. ACM, 2014, pp. 497–506.
[39] ——, “Secsift: Secure image sift feature extraction in cloud com-
puting,” ACM Transactions on Multimedia Computing, Communica-
tions, and Applications (TOMM), vol. 12, no. 4s, p. 65, 2016.
[40] G. Chen, Y. Mao, and C. K. Chui, “A symmetric image encryption
scheme based on 3d chaotic cat maps,” Chaos, Solitons & Fractals,
vol. 21, no. 3, pp. 749–761, 2004.
[41] X. Chai, Y. Chen, and L. Broyde, “A novel chaos-based image
encryption algorithm using dna sequence operations,” Optics and
Lasers in engineering, vol. 88, pp. 197–213, 2017.
[42] Z. Qin, J. Yan, and K. Ren, “Private image computation: The case of
cloud based privacy-preserving sift,” in Computer Communications Lihua Lu received her BS in network engineer-
Workshops (INFOCOM WKSHPS), 2014 IEEE Conference on. IEEE, ing from Nanjing University of Information Sci-
2014, pp. 179–180. ence & Technology in 2016, China. She is cur-
[43] K. Krishna and M. N. Murty, “Genetic k-means algorithm,” IEEE rently pursuing her MS in computer science and
Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), technology at the College of Computer and Soft-
vol. 29, no. 3, pp. 433–439, 1999. ware, in Nanjing University of Information Sci-
[44] R. Canetti, “Universally composable security: A new paradigm for ence & Technology, China. Her research interest
cryptographic protocols,” in 42nd IEEE Symposium on Foundations is encrypted image retrieval.
of Computer Science. IEEE, 2001, pp. 136–145.
[45] A. Jolfaei, X.-W. Wu, and V. Muthukkumarasamy, “On the security
of permutation-only image encryption schemes,” IEEE Transaction-
s on Information Forensics and Security, vol. 11, no. 2, pp. 235–246,
2016.
[46] A. Clark and E. Dawson, “A parallel genetic algorithm for crypt-
analysis of the polyalphabetic substitution cipher,” Cryptologia,
vol. 21, no. 2, pp. 129–138, 1997.
[47] B. Pinkas and T. Reinman, “Oblivious ram revisited,” in Annual
Cryptology Conference. Springer, 2010, pp. 502–519.
[48] E. Shi, T.-H. H. Chan, E. Stefanov, and M. Li, “Oblivious ram
Byeungwoo Jeon received the B.S. degree
with o ((logn) 3) worst-case cost,” in International Conference on
(Magna Cum Laude) in 1985, the M.S. degree
The Theory and Application of Cryptology and Information Security.
in 1987 in electronics engineering from Seoul
Springer, 2011, pp. 197–214.
National University, Seoul, Korea, and the Ph.D.
[49] H. Jegou, M. Douze, and C. Schmid, “Hamming embedding
degree in electrical engineering from Purdue U-
and weak geometric consistency for large scale image search,”
niversity, West Lafayette, in 1992. Since Septem-
Computer Vision–ECCV 2008, pp. 304–317, 2008.
ber 1997, he has been with the faculty of the
School of Electronic and Electrical Engineer-
ing, Sungkyunkwan University, Korea, where he
is currently a full Professor. He has authored
many papers in the areas of video compression,
prepost processing, and pattern recognition. His research interests in-
Zhihua Xia received the BS degree in Hunan
clude multimedia signal processing, video compression, statistical pat-
City University, China and PhD degree in com-
tern recognition, and remote sensing.
puter science and technology from Hunan Uni-
versity, China, in 2006 and 2011, respectively.
He works as an associate professor in School
of Computer & Software, Nanjing University of
Information Science & Technology. His research
interests include digital forensic and encrypted
image processing. He is a member of the IEEE.

1939-1374 (c) 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.