Prototyping Color-based Image Retrieval with MATLAB®

Petteri Kerminen1, Moncef Gabbouj2

1
Tampere University of Technology, Pori, Finland
2
Tampere University of Technology, Signal Processing Laboratory, Tampere, Finland

Abstract
In the MuVi-project [1] this kind of tool is under
Content-based retrieval of (image) databases has construction. It will cover the properties presented above.
become more popular than before. Algorithm develop- Research work on content-based image retrieval has been
ment for this purpose requires testing/simulation tools, done in [2 – 6]. The system, which is presented in this
but there are no suitable commercial tools on the paper, is a simulation environment, where MuVi’s color
market. content based retrieval has been developed and tested.
A simulation environment for retrieving images from
database according histogram similarities is presented 2. System development
in this paper. This environment allows the use of MATLAB is an efficient program for vector and matrix
different color spaces and numbers of bins. The data processing. It contains ready functions for matrix
algorithms are implemented with MATLAB. Each color manipulations and image visualization and allows a
system has its own m-files. program to have modular structure. Because of these facts
The phases of the software building process are pre- MATLAB has been chosen as prototyping software.
sented from system design to graphical user interface
(GUI). The functionality is described with snapshots of 2.1 System design
GUI.
Before any m-files have been written, the system design
has been done. A system design for the HSV (hue,
1. Introduction saturation and value) color system based retrieval process
is presented in Figure 1. Similar design has been done for
Nowadays there are thousands or hundreds of all used color systems.
thousands of digital images in an image database. If the
user wants to find a suitable image for his/her
purposes, he/she has to go through the database until tesths27 image_pos
the correct image has been found or use a reference
book or some “intelligent” program. Video on demand
(VoD) services also requires an intelligent search
system for end-users. VoD systems’ search methods
differ slightly from image database’s methods. hs27read dif_hsv

A reference book is a suitable option, if the images

are arranged with a useful method, for example: 1)
categories: animals, flags, etc, 2) names (requires a hs27conv
good naming technique) or 3) dates. An experienced
user can use these systems as well as textual searches
(keywords have to be inserted in a database)
efficiently. There are situations when a multi-language
system has to be used. There a language independent hsquan27
search system’s best properties can be utilized. A tool
which is based on the images’ properties can be made
language independent. These properties can be for Figure 1: Function chart for HSV color space
example color, shape, texture, spatial location of shape with 27 bins histogram.
etc.
 Tesths27 is the main function for this color system directory with an almost similar algorithm as in the case of
and this number of bins. It calls other functions the query image. The difference is that now there will be a
(hs27read, dif_hsv and image_pos) when needed. Each histogram difference calculation between the query
color system has a main function of its own and image’s and current image’s histogram. Finally the
variable number (2 – 3) of sub-functions. If there is no image_pos function will be used to put a query image and
need for color space conversion there are 2 functions, the desired number of best match images on the display.
otherwise 3 functions on the first branch of the function
chart. 2.3 Linking
It is not possible to use a program before the main function
The function call of the main function is: and sub-functions are connected to each other. The main
matches=tesths27(imagen,directory,num) function will be called from the command line or through
the graphical user interface, which will be presented later
The variable imagen specifies the query image’s name
in this paper. In both cases the function call will contain
and path. The directory is a path of the image database
the same arguments. For multi-level search purposes
and num is a desired number of retrieved images.
separate main functions have been implemented, but it is
2.2 Functions possible to utilize “normal” functions and add one
parameter, where the best matches array can be transferred
At this moment there are functions implemented for for second a stage comparison function.
four color spaces: HSV, L*a*b*, RGB and XYZ [7].
Each color space has from 2 to 4 implementations for The main function calls an image read function with the
different numbers of bins. There are altogether 14 main image’s name. The histogram will be returned to the main
functions. function. If a color space conversion is needed, the
conversion function will be called from the read function
For some color systems it is possible to make these with r, g and b –vectors. The histogram will be returned to
functions dynamic, i.e. dynamic histogram calculation. the calling function. Finally the histogram build function
Every color system / bin combination requires its own will be called with converted color vectors. This function
histograms and these can be made only with an returns a quantized histogram, which will go through all
exhaustive method (pixel by pixel). Histogram functions until it achieves the main function.
calculation takes ½ - 5 minutes per image, each
approximately 320×240 pixels, depending on the The main function calls the histogram difference function
complexity of the color space on 150 MHz Pentium. with two histogram vectors and will get a difference value
Thus it is not reasonable to let the user select a bin as a response. The difference function uses Euclidean-
number freely, especially in the case of large databases. distance calculation, but it can be easily changed to
another algorithm due to the modularity of the program. If
The functions have been named so that the names the difference is smaller than largest difference on a best
contain information of the color space used, the match table, the current result will be written over the last
purpose of the functions and the number of used bins. result on the best match table. After that the table is
Some functions, for example image_pos, have been arranged again in an ascending order of distance. When all
used by many or all main functions and these functions the images have been analyzed, the sorted best match
have not been named as described above. table, the number of desired output images, the query
image’s name, the search image’s path and the database
The main function checks, if the function call is path are transferred to the image_pos function. These
correct. If the query image’s name doesn’t contain a values can be transferred into larger components
path, the function assumes that the image is situated in (vectors/containers). Now the program works faster with
the database directory. In addition to this, the main several input arguments, because there is no need for
function checks, if the query image already has a picking up variables from a container.
histogram in the currently used database. If the
required histogram is not there, the image read (for
example hs27read) function is called. This function 2.4 Graphical user interface
also normalizes pixel values and arranges image matrix The graphical user interface (GUI) is an important part of
data to a vector format. After that stage a color space software development. The designing of the GUI have to
conversion function (if needed) is called. Finally a solve the following problems: learning time, speed of
quantization function builds the histogram with the performance, rate of errors by users, retention over time,
correct number of bins. and subjective satisfaction [9]. This software is, at the
moment, intended to be used only for testing purposes.
The histogram will then be saved into the database The most important property of this software is that the
directory. If the histogram already exists there, the results of different test queries can be seen quickly and the
three previous steps will not be executed. Now the results can be saved safely on a disk. Thus the visual
query image has been analyzed. Then the main layout is not as important as in case of a commercial
function will go through all images in the database software product.
In Figure 2 the first screen on GUI is presented. The
purposes of the buttons, menus and other components
will be presented later. If this software is developed
into a commercial product, the menu bar will be
disabled in the future and the exit and help buttons will
be added on the canvas.

Figure 4: Color system selection from a popup menu.

The second menu is disabled because a one-level
search is selected.
quantized or a more accurate histogram is used in one loop
(one color system).

Figure 2: GUI before the search image selection. In a multi-level search two different color systems /
histograms are used. During the first loop the roughly
In Figure 3 the search screen is presented just before quantized histograms are used and during the second loop
starting a search. The user is shown a search image, the more accurate histograms are utilized for the best
and in this way he/she can be sure that the search will matches from the first loop. The color system on the
be made with the correct image. second loop can be either the same as on the first loop or a
different one. For queries with one-level search the
selection of a second color system is disabled. The user
can select the number of retrieved images at the final
stage. The software can be linked to many image databases
and the user can select a database where the query will be
directed.

The user can select a search image either from the same
database where the query will be directed to (default) or
from any directory in his/her PC. The selection will be
made with the file – open dialog, which is presented in
Figure 5. The form can be cleared with the “Reset” button.
A query is executed with the “Search” button. Finally the
results of the search will appear on the screen in a separate
window, as presented in Figure 6. Earlier [8] the software
opened each image in a separate window and
evaluating/saving the results is more difficult than after the
improvement. In the top left top corner is the original
query image. Below that image the best matches are
Figure 3: GUI just before running a query.
presented in a descending order of similarity from left to
right and from top to bottom. The user can select suitable
The results of the query will be presented on the screen images for further use with the “Copy selected” or the
in the format which is presented in Figure 6. “Print selected” buttons. The “New search” button closes
this form and goes back to the original search form. The
“Search similar” button executes a new search where a
3. Using the software query histogram is composed of histograms of the selected
The first screen has already been presented in Figure 2. images. If the user has selected a larger number than 21 as
The user can choose from pop-up menus (see Figure “Number of matching images”, the best matches will be
4), if the search is made with one a color system or as shown on multiple screens. The user can browse these
a multi-level search. In a one-level search a roughly pages with the “Previous page” and “Following page”
buttons.
Figure 5: The query image selection dialog. The
language of the dialog depends on the language of the
operating system used.

Figure 6: The results of a query will be presented graphically.

4. Summary References
The color content-based retrieval requires algorithms,
[1] Alaya Cheikh F et. al. MUVIS: A System for Content-Based
which give visually correct results. Correctly working Indexing and Retrieval in Large Image Databases.
algorithms can not be chosen before simulations. The Proceedings of SPIE Storage and Retrieval for Image and
software presented in this paper is intended to be used Video Databases, San Jose, 1999,vol. 3656: 98 – 106.
for testing purposes. Some operations will be [2] Sawhney H. S., Hafner J. L., Efficient Color Histogram
implemented, if the software is developed into a Indexing for Quadratic Form Distance Functions, IEEE
commercial product. Some modifications are under Transactions on Pattern Analysis and Machine Intelligence,
construction. 1995, vol. 17 no. 7: 729 – 736.
[3] Vellaikal A. et. al. Content-Based Retrieval of Color and
Multispectral Images Using Joint Spatial-Spectral Indexing,
This software has been used as a testing platform for
SPIE vol. 2606: 232 – 243.
histogram quantization tests. The modularity of this [4] Ng V. et. al. Adaptive Histogram Indexing, SPIE vol. 2606:
program makes it possible to take new algorithms as a 202 – 211.
part of the software in a short time. MATLAB makes [5] Smith J. R., Chang S. F. Automatic Image Retrieval Using
quick prototyping possible. A possibility to save Color and Texture, 1995-6, http://www.ctr.columbia.edu/
figures (search results) directly on a disk is a ~jrsmith/html/pubs/PAMI/pami_final_1.htm.
fulfillment of the program’s requirements. After the [6] Flickner M. et. al. Query by Image and Video Content: The
results have been analyzed visually, the best algorithms QBIC System, IEEE Computer 1995 vol. 28 no. 9: 23 – 32.
will be taken as a part of the final software. [7] Jain A. K. Fundamentals of Digital Image Processing,
Prentice Hall, 1989: 66 - 71.
5. Acknowledgements [8] Kerminen P, Gabbouj M. Image Retrieval Based on Color
Matching. Proceedings of FINSIG’99, Oulun yliopistopaino,
This work has been founded by the European Union 1999: 89 – 93.
– ERDF, the Technology Development Centre Tekes, [9] Shneiderman B. Designing the User Interface, Addison
Alma Media, the Helsinki Telephone Company, Nokia Wesley Longman, 1998: 14 – 16.
Research Center, the Satakunta High Technology
Foundation and Ulla Tuominen’s Foundation.
Address for correspondence.

Petteri Kerminen
P.O. BOX 300
FIN-28101 PORI
Finland
[email protected]