Semantic-Based Image Retrieval Using Balanced

Clustering Tree

Nguyen Thi Uyen Nhi1,3 , Thanh The Van2(B) , and Thanh Manh Le1
1 Faculty of Information Technology, University of Science - Hue University, Hue, Vietnam
[email protected], [email protected]
2 Office of Scientific Research Management and Postgraduate Affairs, HCMC,
University of Food Industry, Hue, Vietnam
[email protected]
3 Faculty of Statistics and Informatics, University of Economics, the University of Danang,

Hue, Vietnam

Abstract. In this paper, we propose a model for semantic-based image retrieval

(SBIR) on the clustering balanced tree, C-Tree, and ontology to analyze the seman-
tics of an image and extract a similar set of images, in which the input is a query
image. This structure is constructed rely on separating the nodes from the leaf node
and growing towards the root to create a balanced tree. A set of similar images are
searched on the C-Tree to classify the query image based on the k-NN (k-Nearest
Neighbor) algorithm. Then, the SPARQL query is generated to query the semantics
of the image on ontology. We experimented with image datasets such as COREL
(1000 images), Wang (10,800 images), ImageCLEF (20,000 images). The results
are compared and evaluated with the relevant projects published recently on the
same datasets.

Keywords: Clustering · C-Tree · SBIR · Ontology · SPARQL

1 Introduction
According to IDC (International Data Corporation), in 2016 the world created 1,138
trillion images (700 times more than 2015). In 2017, digital data reached 17 trillion
gigabytes and it is predicted that it will reach 175 trillion gigabytes by 2025 [3]. The
problem of data mining, search and semantic analysis of images is becoming more
common and is a research topic that is interested in many people. Therefore, it’s essential
to gain the development of highly accurate image retrieval system using a structure for
mapping low-level features to high-level semantics on ontology [5, 15].
We solve two problems: (1) search for a set of similar images by low-level features
with input image; (2) convert low-level content into high-level semantics of the image
based on SPARQL query and ontology. So, we propose C-Tree structure which can store
large data on external memory and make query times faster. The leaf nodes contain the
indexes of the image; the internal node contains only the centers of child nodes and
the paths to its children. At the same time, C-Tree is grown in a balanced way from

© The Author(s), under exclusive license to Springer Nature Switzerland AG 2021

Á. Rocha et al. (Eds.): WorldCIST 2021, AISC 1366, pp. 416–427, 2021.
https://doi.org/10.1007/978-3-030-72651-5_40
 Semantic-Based Image Retrieval Using Balanced Clustering Tree 417

leaf to root. The query result on the C-Tree finds a set of images that are similar to the
content with the query image. To analyze the semantics of the image, the k-NN algorithm
[10] is proposed to classify the image as a basis for extracting high-level semantics for
that image. The SPARQL query is generated based on the classifications and query on
ontology. The result of this query is the set of descriptive semantics of the image and
the set of similar images according to the semantic classification of the input image.

2 Related Works
Many types of research focusing on the construction of multidimensional index have been
developed such as B-tree [6], R-trees [4], KD-tree [11], S-tree [12], etc. In the indexing
method, an image is represented by features vector to find the similarity between the
query image and the image in the database [7]. KD-tree and R-tree are examples of the
method of searching for images in space [19]. R*-Tree [2] has a better performance
than R-Tree. This shows that researches related to tree structure image querying are paid
much attention in recent years.
Y. Liu, D. Zang, and G. Lu [9] proposed an image retrieval method based on high-
level semantics of each region on the image. During the retrieval, a set of images whose
semantic concept matches the query is returned. The system was tested on the COREL
image dataset. V. Vijayarajan et al. [23] performed an image retrieval based on natural
language analysis to create a SPARQL query for the purpose of searching for an image
set based on the triplet language RDF. M. Jiu, et al. [8] used deep multi-layer networks
based on nonlinear activation functions for image annotation. SVM technique is applied
to classify images at the output layer in order to extract a semantic level according to
visual information for similar images based on the BoW (Bag-of-Words). This method
is evaluated the experimental results on the ImageCLEF image data set. Van T.T. and Le
MT. [12] proposed a binary signature clustering method of an image to create a clustering
graph structure based on an S-Tree for constructing content-based image retrieval. They
also evaluated the experiment the datasets including COREL, CBIR, Wang, MSRDI,
and ImageCLEF. Wang, X., & Wilkes, D.M. [14] proposed the method of fast image
retrieval based on the Quantization Tree and presented an effective way to index local
image regions. Quantization Tree has a hierarchical organization of labeled descriptions
from the image color chart. Quantization Tree combines the visual semantic content of
an image into content-based image retrieval algorithms for semantic querying of images.
The above researches mainly focus on tree data structures for efficient storage of
image indexes that are low-level vector extracted from large image datasets for content-
based image retrieval but lack of semantic analysis of the image. This limitation makes
the image query system not to meet the needs of the user, creating a “semantic gap”
between content-based queries and high-level semantics. Therefore, many studies on
semantic-based image retrieval on ontology are paid attention to.
418 N. T. U. Nhi et al.

3 Semantic-Based Image Retrieval on C-Tree

3.1 Architecture of SBIR_CT

We describe the architecture of the semantic image retrieval system on C-Tree named
SBIR_CT and its components. Figure 1 describes the architecture of a semantic image
query system on C-Tree with two specific stages as follows:

(1) Preprocessing phase: Each image from the dataset segmented and extracted features
vector; create a balanced clustering tree structure named C-Tree from the training
data samples; Build the ontology from the image dataset and WWW.
(2) Image query phase: Execute a query in a C-Tree to obtain a set of similar images and
an arrangement of similar images according to the measure; Classify image from a
set of similar image based on the k-NN algorithm; Query the semantic image from
SPARQL based on ontology.

Fig. 1. Model of semantic-based image retrieval on C-Tree

3.2 Extracting the Feature Vector

We perform extraction of low-level features, which is described in Fig. 2, including

MPEG7 color feature, shape feature, texture feature, detecting the Laplacian of Gaussian
object, object recognition based on boundary and surface with Sobel filter, enhancing
pixel intensity with Gaussian filtering.
 Semantic-Based Image Retrieval Using Balanced Clustering Tree 419

Fig. 2. An example of the low-level feature extraction of the image

3.3 Structure of C-Tree

In Fig. 3, the C-Tree consists of a root, a set of nodes I and a set of leaves L. Nodes are
connected to each other through the path of the parent-child relationship. The element
ED is stored in the leaf, and the center EC is stored in the node. The element ED =
index, f , ID, file, cla contains index is the ED position on the file, f is the vector
feature of an image, the image identifier ID, the image annotation file, and the cla are
classes of the image. The center EC = index, fc , isNextLeaf , path contains index
is the EC position on the file, fc is the center vector of the feature vectors f at a child
node that has a path linking the path to the EC, and is Next Leaf is the value that
checks whether the next subset is a leaf or not. Therefore, the C-Tree includes leaves L
contain data elements ED: L = {ED1 , ED2 , . . . , EDn }, 1 ≤ n ≤ M , in which M is the
maximum number of elements in a leaf; the nodes I are nodes with at least one child
node, containing center elements EC: I = {EC1 , EC2 , . . . , ECm },1 ≤ m ≤ N , in which
N is the maximum number of elements in the node; each element internal node-links to
its adjacent child via its path; each leaf has the same height. The rules of building C-Tree
are described as follows:

a) When the root is empty, the root is a leaf containing at most M elements ED.
b) Data elements are added to the tree according to the branch selection rule with the
closest measure until a suitable leaf is found to add; Fig. 4 describes how to add a
new element to the cluster.
c) If a node in a C-Tree is full (the number of elements of node, leaf exceeds the
threshold M , N ), the splitting based on K-means algorithm is performed; Fig. 5
shows an example of splitting a leaf on C-Tree.

3.4 Building the Ontology for Image Datasets

Figure 6 is the proposed model consists of the steps: adding image annotations and
semantic descriptions for images; extracting the resource identifiers URIs, descriptions
of image taken from WWW. Figure 7 is an example of a semantic description for the
class from the thesaurus. Figure 8 is an example of a ontology built on Protégé.
420 N. T. U. Nhi et al.

Fig. 3. Structure balanced clustering tree C-Tree

Fig. 4. Adding an element to the cluster on C-Tree

4 Search Algorithms on SBIR_CT System

4.1 Content-Based Image Retrieval on C-Tree

We extract the feature vector of the query image and find a branch with closest similar
measure. Image retrieval algorithm on the C-Tree is done as follows:
 Semantic-Based Image Retrieval Using Balanced Clustering Tree 421

Fig. 5. Splitting a node on C-Tree

Fig. 6. A model for building the ontology

Fig. 7. An example of a semantic description for FLOWER class

422 N. T. U. Nhi et al.

Fig. 8. An example of a visual ontology built on Protégé

Fig. 9. A search results for CBIR of SBIR_CT system

4.2 Query Image Classification

An input image is queried on the C-Tree to find a set of similar images by content. From
there, the k-NN algorithm is implemented to classify the query image.
 Semantic-Based Image Retrieval Using Balanced Clustering Tree 423

4.3 Image Semantic Retrieval and Analysis

Relying on the visual word vector W generated from the CIKN algorithm, the SPARQL
query is generated to execute the query on the ontology.

5 Experiment
5.1 Application
We used datasets in the experiment including COREL, WANG, ImageCLEF. The result
of this process is a sets of similar images and semantic descriptions as meta-data, URI,
concept. Figure 9 shows a result of the SBIR_CT query system.

5.2 Experimental Results

To evaluate image search efficiency, we use evaluation factors including precision, recall
in Fig. 10, Fig. 11, Fig. 12. The average search time (milliseconds) and performance
values on the datasets are summarized in Table 1, with 19.914 ms (COREL), 39.747 ms
(Wang), 44.089 ms (ImageCLEF).
Table 2 summarizes the results of classification accuracy and time on the C-Tree
of COREL, Wang, ImageCLEF. We compared the obtained performance with other
studies on the same set of image data. Table 3, Table 4, Table 5 show the results of
the comparisons between our proposed method and the methods of the studies on the
COREL, Wang, mageCLEF.
424 N. T. U. Nhi et al.

Fig. 10. Image retrieval performance on C-Tree of COREL dataset

Fig. 11. Image retrieval performance on C-Tree of WANG dataset

Fig. 12. Image retrieval performance on C-Tree of ImageCLEF dataset

 Semantic-Based Image Retrieval Using Balanced Clustering Tree 425

Table 1. Performance of SBIR_CT on image datasets

Performances COREL WANG ImageCLEF

Avg. precision 0.677655 0.607222 0.606217
Avg. recall 0.699885 0.489184 0.409401
Avg. F-measure 0.688521 0.545011 0.474718
Avg. query time (ms) 19.91437 39.74690 44.08912

Table 2. Image classification performance of SBIR_CT system on image datasets

Dataset Precision Time (ms)

Corel 0.73017535 12.18645
Wang 0.68935259 14.23075
ImageCLEF 0.69135772 15.96235

Table 3. Comparison between methods on the COREL dataset

Methods Mean average precision (MAP)

A. Huneiti, 2015 [17] 0.559
Garg, M., 2019 [18] 0.602
Bella M. I. T., 2019 [19] 0.609
Our method 0.6777

Table 4. Comparison between methods on the WANG dataset

Methods Mean average precision (MAP)

Dos Santos, 2015 [20] 0.570
R. Das, 2018 [21] 0.559
P. Chhabra, 2018 [22] 0.577
Our method 0.6072

Table 5. Comparison between methods on the ImageCLEF dataset

Methods Mean average precision (MAP)

V. Vijayarajan, 2016 0.4618
[13]
H. Cevikalp, 2017 [16] 0.4678
Z. Seymour, 2018 [23] 0.420
Our method 0.6062
426 N. T. U. Nhi et al.

6 Conclusion
In this paper, we have built a balanced clustering tree structure C-Tree and proposed
creation and querying algorithm on C-Tree to store image data. A k-NN algorithm is
used to classify images based on a set of similar images. Besides, we have built the
ontology for the image sets to store the descriptive data for image sets based on the RDF
trilogy language. The SPARQL queries are generated to semantic query on the ontology
for the input image. Since then, a semantic image query system SBIR_CT is built on
the basis of C-Tree and ontology. The average query accuracy per set of COREL, Wang,
ImageCLEF are 0.6777, 0.6072, 0.6062, showing that our proposed query system is
feasible for single-object and multi-object image datasets. In the future, we will build
a semi-automatic ontology for image sets from WWW to enhance the efficiency of
semantic-based image retrieval.

Acknowledgment. The authors would like to thank the Faculty of Information Technology, Uni-
versity of Science - Hue University for their professional advice for this study. This work has
been sponsored and funded by Ho Chi Minh City University of Food Industry under Contract No.
147/HD-DCT.

References
1. Bella, M.I.T., Vasuki, A.: An efficient image retrieval framework using fused information
feature. Comput. Electr. Eng. 75, 46–60 (2019)
2. Beckmann, N. et al.: The R*-tree: an efficient and robust access method for points and
rectangles. In: ACM SIGMOD Conference, pp. 322–331 (1990)
3. Reinsel, D., Gantz, J., Rydning, J.: The Digitization of the World: From Edge to Core
sponsored by Seagate, IDC Technical Report (2018)
4. Elmarsson, A., A comparison of different R-tree construction techniques for range queries
on neuromorphological data, School of Elec. Engi. and Comp. Sci. (2020)
5. Gombos, G., Kiss, A.: P-Spar (k) ql: SPARQL evaluation method on Spark GraphX with
parallel query plan. In: IEEE 5th International Conference, pp. 212–219 (2017)
6. Graefe, G., et al.: Modern B-tree techniques. In: IEEE International Conference, pp. 1370–
1373 (2011)
7. Jiang, W., Er, G., Dai, Q., Gu, J.: Similarity-based online feature selection in content-based
image retrieval. IEEE Trans. Image Process. 15(3), 702–712 (2006)
8. Jiu, M., Sahbi, H.: Nonlinear deep kernel learning for image annotation. IEEE Trans. Image
Process. 26(4), 1820–1832 (2017)
9. Liu, Y., Zhang, D., Lu, G.: Region-based image retrieval with high-level semantics using
decision tree learning. Pattern Recogn. 41(8), 2554–2570 (2008)
10. Ma, Y., Xie, Q., Liu, Y., Xiong, S.: A weighted KNN-based automatic image annotation
method. Neural Comput. Appl. 1–12 (2019)
11. Procopiuc, O et al.: Bkd-tree: a dynamic scalable kd-tree. In: International Symposium on
Spatial and Temporal Databases, Springer, pp. 46–65 (2003)
12. Van, T.T., Le, T.M.: Contentbased image retrieval based on binary signatures cluster graph.
Expert Syst. 35(1), e12220 (2018)
13. Vijayarajan, V., et al.: A generic framework for ontology-based information retrieval and
image retrieval in web data. Hum.-centric Comp. Infor. Sci. 6(1), 18 (2016)
 Semantic-Based Image Retrieval Using Balanced Clustering Tree 427

14. Wang, X., et al.: A fast image retrieval method based on a quantization tree. In: Machine
Learning-based Natural Scene Recognition, Springer, Berlin, pp. 195–214 (2020)
15. Zhong, B., et al.: Ontology-based semantic modeling of knowledge in construction: clas-
sification and identification of hazards implied in images. J. Constr. Eng. Manage. 146(4),
04020013 (2020)
16. Cevikalp, H., Elmas, M., Ozkan, S.: Large-scale image retrieval using transductive support
vector machines. Comp. Vision Image Underst. 173, 2–12 (2018)
17. Huneiti, A., Daoud, M.: Content-based image retrieval using SOM and DWT. J. Softw. Eng.
Appl. 8(02), 51 (2015)
18. Garg, M., Singh, H., et al.: Fuzzy-NN approach with statistical features for description and
classification of efficient image retrieval. Mod. Phys. Lett. 34(03), 1950022 (2019)
19. Bella, M.I.T., Vasuki, A.: An efficient image retrieval framework using fused information
feature. Comput. Electr. Eng. 75, 46–60 (2019)
20. dos, Santos, J.M. et al.: A signature-based bag of visual words method for image indexing
and search. Pattern Recogn. Lett. 65, 1–7 (2015)
21. Das, R., Thepade, S. et al., Novel feature extraction technique for content-based image
recognition with query classification. Int. J. Comp. Vis. Robot. 7(1–2), 123-147 (2017)
22. Chhabra, P., Garg, N.K., Kumar, M.: Content-based image retrieval system using ORB and
SIFT features. Neural Comput. Appl. 32(7), 2725–2733 (2020)
23. Seymour, Z., et al., Image annotation retrieval with text-domain label denoising. In: ACM on
International Conference on Multimedia Retrieval, pp. 240–248 (2018)