Performances of mobile-agents for interactive image retrieval

David Picard, Arnaud Revel Matthieu Cord

ETIS - UMR 8051 LIP6, UPMC-Paris6
FRANCE FRANCE
[email protected], [email protected] [email protected]

Abstract query and presents the relevant results given a specific rel-
evance function to the user. The user annotates the results
In this paper, we present a system for image retrieval in order to improve the relevance function. The system then
over a network of computer based on “ant-like” mobile- presents new results according to this refinement. This user
agents. Image databases are hosted on the network, and the interaction loop is called relevance feedback, and is per-
user wants to find all the images matching a specific con- formed several times. When the refinement is satisfying,
cept (cars, flower, Italy, etc...). Usually, content based im- the best results are presented to the user as final result.
age retrieval systems (CBIR) do not consider the dispertion The best efforts made so far in order to improve the
of the data among the network. We train a SVM classifier learning process of the classifier is “active learning” and
with examples annotated by the user and then launch mobile aims at optimizing the set of training examples [4]. The ac-
agents which explore the network in order to retrieve the tive learning strategy aims at presenting to the user the un-
most relevant images. Several interactive session (launch- labeled document that will enhance the most the relevance
ing of agents then annotation of the results) are made to function [8].
improve the classifier. With the increase of interconnections between networks,
Experiments are made both to see the influence of local- the available images are distributed over thousands of small
ization of the search concept on the quality of the learning, databases. The internet shows us an example of a real con-
and to focus on the quality of the agent based solution com- text for interactive image retrieval. Classical CBIR systems
pared to a centralizing system within a fixed amount of time has not been designed to face this network-based context.
for the interaction. They consider that all the images are available in a huge
single database.
We are now introducing a new system designed for CBIR
1 Introduction. over a network. It is based on mobile agents inspired by
ants [6].These agents explore the network looking for im-
With the growth of image collections coming from per-
ages matching the user’s concept. All the processing of the
sonal or professional digitization, spread on private net-
images are made on the computer where they are hosted,
works, or on the Internet, the incredible amount of available
thus allowing a natural distribution of the computation. We
data makes it impossible to search within by hand. Usually,
have already presented preliminary description of our sys-
users searching for a category of images (let say images
tem [7, 5], we are now testing its performances.
of cars, or images of Europe) mostly use a search engine
This paper if organized as follows: the next part will give
and feed it with semantic keywords supposed to represent
an overview of our system. We will present results on the
the searched concept (as with Google). Yet, these search
quality of the retrieval depending on the distribution of the
engines rely on meta-data (annotations) or the context in
data. And then, we will evaluate the performance of our
which the image is found (the web page for example) and
system in a real-case application, and comment the results
not on the internal features of the image.
obtained in order to draw a conclusion on the advantages of
Tools have been developed, as [10], in order to use low
the distributed approach.
level features of the images (colors, textures, shapes, etc.).
In order to improve the retrieval, a learning processes has 2 Description of our strategy.
been introduced in the context of content-based image re-
trieval (CBIR) [9, 2]. A typical interactive retrieval follows Let us consider a network of computers, some of which
this scheme: first, the user gives an image or a keyword containing image databases. On each of these machines,
to begin the search process. Then, the system analyzes the our system runs a local static agent (Local Image-Database

Proceedings of the 2006 IEEE/WIC/ACM International Conference

on Web Intelligence (WI 2006 Main Conference Proceedings)(WI'06)
0-7695-2747-7/06 Authorized
$20.00 ©licensed
2006 use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:06 UTC from IEEE Xplore. Restrictions apply.
 Agent - LIDA) which indexes the local database. The index The answers of all the mobiles agents are collected into
consists in a set of features extracted from the images (25 a pool on the user’s computer until a timeout is reached.
colors and 25 textures). Then, I images are selected from the pool and proposed to
The user starts the search process on his own computer the user who labels them positively or negatively. For each
s0 connected to the network. He submits a query image label, a reinforcement signal is sent back to all the com-
which serves as an example of what he searches to a human- puters on the route to the corresponding image database in
machine interface. The features are extracted from this im- order to update the pheromone level of the computers on
age, and a similarity function is built with this example. The the pathway taken by the mobile agent to return this image
program then launches N mobile agents, which aim at find- (see algorithm). Positive labeling increases the pheromone
ing the most relevant images. These agents carry the simi- level, whereas negative labeling decreases it, according to
larity function, and explore the network. Their movements the sames rules as for the agents movements (see eq. 1 and
are ruled by the following algorithm based on the reinforce- eq. 2).
ment of marker, like ants with pheromones: As soon as images are given, the mobile agents are re-
Algo 1: Mobile-agent behaviour launched with the latest similarity function in order to ex-
I NITIALIZATION : sc ← s0 , prof ← O plore the network and bring new examples to be learned.
Do Moreover, given the path reinforcement process, the agents
If sc = sLIDA // A local agent is found! tend to prefer moving to the sites containing images labeled
Go back to the nest positively more frequently. An “autocatalytic” process is
Else // Look forward thus resulting from this loop. In the image retrieval con-
P hj text, the searched concept is small in terms of number of
Go to site sj with probability Pk +ξ
images compared to the great amount of available data, and
l=1 P hl
with k = |succ(sc )| the training set often contains a lot of uninformative im-
prof ← prof + 1 ages, biasing the learning process. This database selection
End If improves the learning process by increasing the number of
While sLIDA not found or prof < θ informative images added to the training set. Once a “suf-
ficient” number of images has been learned, mobile agents
With sj being the computers in the network, sc the cur- are launched a very last time in order to return the most rel-
rent site explored by the agent and succ(sc ) the sites di- evant images. Results are then ranked and presented to the
rectly following sc . P hk is the pheromone level on the site user.
sk . prof is the current depth, and θ is a threshold corre-
sponding to a maximum exploration depth. The active learning strategy. As many agents can reach
If the agent arrives on a computer containing a image the same LIDA at the same time with the same classifier,
database, it sends a request containing the relevance func- an active strategy which does not select the same set of im-
tion to the LIDA. The LIDA builds an active strategy with ages must be built. The reference SV Mactive strategy from
this function, and sends a set of selected images to the mo- Tong [8] was adapted in order to fit our context, as described
bile agent. The mobile agent comes back to the user’s com- in [5]. This strategy aims at reducing the uncertainty by se-
puter and proposes them to the interface program. At each lecting the closest images to the hyperplane’s boundary. Let
movement, agents modify the markers P hk on the host k D be the local database containing the feature vectors x of
they are moving to. the images. All the images in the database (represented by
When searching for images, they decrease the level of their feature vector x) are ranked given their distance to the
pheromones in order to simulate evaporation and enable the classifier’s hyperplane. We associate a Gaussian probability
discovery of new paths : P (x) to each image based on this ranking r(x):

Decrease : P hk ← (1 − α) · P hk (1) P (x) ∝ e− 2 (

1 r(x) 2
σ ) (3)
They use an increasing rule on their way back to the user in To obtain the discrete probability, P (x) is normalized
order to reinforce the paths leading to computers where it is over the local database D:
known that images can be found : 1 r(x) 2
e− 2 ( σ )
P (x) = X (4)
1 r(xi ) 2
Increase : P hk ← P hk + β (2) e− 2 ( σ )
xi ∈D
Such reinforcement rules inspired by the colonies of ants
are known to find the shortest path to the sites containing I images are randomly sampled thanks to the probability
relevant information ([3, 1]). P (x) over the xi ∈ D. σ was computed so that the set of

Proceedings of the 2006 IEEE/WIC/ACM International Conference

on Web Intelligence (WI 2006 Main Conference Proceedings)(WI'06)
0-7695-2747-7/06 Authorized
$20.00 ©licensed
2006 use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:06 UTC from IEEE Xplore. Restrictions apply.
 the I nearest images from the hyperplane have a probability
of p:
I
X
p= P (xi ) (5)
r(xi )=1

p is the only parameter we tune, and represents the ex-

ploration in the local database D. This allows any image in
the database to be selected, even if the images are far from
the hyperplane. As I images are selected, if p = 1, the re-
turned images are exactly the I nearest images, which is the
SV Mactive strategy.

3 Results.
Effects of thematization of databases. We made a real
and functionnal implementation of this system using the
JADE1 framework. In order to see the influence of the path
reinforcement on the learning process, the system has been Figure 1. Mean Average Precision against the
tested on a simplificated setup. This setup is as follows: our localization of the search concept. The gain
test database is an excerpt of the Corel database which con- in quality when the concept is localized is be-
tains 6,000 images categorized in 50 concepts of 50 to 300 tween 5% and 10%.
images. The database is split into two subsets hosted on
separated computers A and B. Agents were able to move
the existings systems are not designed for a use over a net-
either to A or to B. The localization of the searched concept
work, we have built a very simple system to be compared
varied from 50% on A (equally distributed) to 100% on A
to, which we call “centralizing system”. At the begining
(entirely on A). When the category is equally distributed
of each retrieval session, it gathers all the images available
over the network, the path reinforcement algorithm cannot
on the network into a local database, and then performs the
give the advantage to one of the bases. On the contrary, if
classical interactive CBIR scheme. The user presents an
the category is entirely on A, this algorithm forgets the path
image representing the concept he is looking for. The sys-
to B quickly. Eight mobile-agents were launched, each of
tem analyzes the image by extracting its features and build
them carrying back two images. We made relevance feed-
a classifier based on SVM. 5 images are selected from the
back rounds until 100 images were annotated.
local database given the strategy SV Mactive [8]. The user
The figure 1 shows the Mean Average Precision (M AP )
labels them (1 relevant, -1 irrelevant), and a new SVM clas-
obtained. The M AP indicated an overall quality of the in-
sifier is computed, leading to another round of active learn-
teractive retrieval system, and is computed as follow : Given
ing, exactly as for the distributed approach. When the time
a ranking of the bests images, We call precision P the num-
out is reached, the best results are presented to the user.
ber of relevant images retrieved over the total number of
Let us introduce now the setup used to compare this cen-
retrieved images, and recall R the number of relevant im-
tralizing strategy to our system. We have a network of 5
ages retrieved over the total number of relevant images. The
computers S0 to S4 . The user is working on the machine
Average Precision Ap is the value of P each time R in-
S0 . The other computers are hosting a image database from
creases. The M AP is the mean of all Ap through the final
{A, B, C, D} with the following pairs : {S1 , A}, {S2 , B},
ranking. A perfect retrieval will lead to a M AP of 1. For
{S3 , C}, {S4 , D}. S0 is connected to each of the other
very easy categories, the path learning has little influence
computers. Given the results of previous experiments, the
on the concept learning. The same appears with difficult
concentration of the searched category is set to 90% on A.
categories. For the categories in between, a well localiza-
The query is presented to the system, and then, the retrieval
tion allows us to gain up to 10% of quality, as seen in figure
aims at finding the most relevant images of {A, B, C, D}
1. If the searched category is uniformly distributed over the
within a specific maximum interaction time (M IT ).
network, the reinforcement algorithm has no effect on the
A Gaussian distribution centered on 1s and a σ of 100ms
learning.
is used to model the time taken by the user to annotate an
image. At the end of this session, we will compute the mean
Overall performances. In these experiments, we want to average precision (M AP ). We made the experiment for
compare our system to classical CBIR strategies. But since several values of M IT in order to catch the behavior of the
1 http://jade.tilab.it two systems. Each image in the category was taken as the

Proceedings of the 2006 IEEE/WIC/ACM International Conference

on Web Intelligence (WI 2006 Main Conference Proceedings)(WI'06)
0-7695-2747-7/06 Authorized
$20.00 ©licensed
2006 use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:06 UTC from IEEE Xplore. Restrictions apply.
 input query resulting in a retrieval session. At the beginning The advantages of the distributed solution are a natu-
of each retrieval session, we reseted the pheromone levels ral parallelization of the computation and the reduced use
on all the computers, in order to avoid correlation with pre- of bandwith by avoiding the transfer of huge data. We
vious searches. We then computed the result for a category made experiments on a generalist database with an “user
as the mean of all sessions. oriented” setup in order to evaluate our system within a
In figure 2, the Mean Average Precision M AP accord- fixed amount of time, and compare it to a simple central-
ing to the M IT is shown. The results are for the category izing system. We showed that our system has slightly the
city, which contains 147 images of various photographs avantage in terms of quality of retrieval.
from cities around the world. The M AP increases with We are now looking to catch the semantic of the search
the M IT , and the results are slightly in favor of our mobile into the markers, in order to re-use the markers from previ-
agents strategy. if we compare the M AP to the one of the ous searches.
figure 1, the performances are not as good as expected. This
may be due to the less amount of labels available (about 40 References
for 75s and 70 for 120s) than for the previous test (100 la-
bels). The standard deviation for the M AP is very small [1] E. Bonabeau, M. Dorigo, and G. Theraulaz. The social in-
(under 1%), and thus, we can be very confident into this sect paradigm for optimization and control. Nature, 406:39–
result. 42, 2000.
[2] E. Chang, B. T. Li, G. Wu, and K. Goh. Statistical learning
for effective visual information retrieval. In IEEE Interna-
tional Conference on Image Processing, Barcelona, Septem-
ber 2003.
[3] J. Deneubourg, G. Theraulaz, and R. Beckers. Swarm-made
architectures. In F. Varela and P. Bourgine, editors, Pro-
ceedings of the first european conference on artificial intel-
ligence, pages 123–133. MIT Press, 1992.
[4] P. Gosselin and M. Cord. A comparison of active classi-
fication methods for content-based image retrieval. In In-
ternational Workshop on Computer Vision meets Databases
(CVDB), ACM Sigmod, pages 51–58, Paris, France, June
2004.
[5] D. Picard, M. Cord, and A. Revel. Cbir in distributed
databases using a multi-agent system. In IEEE International
Conference on Image Processing (ICIP’06), Atlanta, GA,
USA, October 2006.
[6] A. Revel. From robots to web-agents: Building cognitive
software agents for web-information retrieval by taking in-
spiration from experience in robotics. In ACM International
conference on Web Intelligence, Université Technologique
Figure 2. Mean Average Precision for the cat- de Compiègne, Compiègne, France, Septembre 2005.
egory city for the centralizing strategy and our [7] A. Revel, D. Picard, and M. Cord. Ant-like mobile agents for
mobile agents approach. content-based image retrieval in distributed databases. In In
proc. workshop Machine Learning Techniques for Process-
ing Multimedia Content, Bonn, Germany, 2005.
[8] S. Tong and E. Chang. Support vector machine active learn-
4 Conclusion. ing for image retrieval. In ACM Multimedia, 2001.
[9] N. Vasconcelos and M. Kunt. Content-based retrieval from
In this article, we presented a distributed CBIR system image databases: current solutions and future directions.
which uses mobile agents with an ant-like behavior in order In International Conference in Image Processing (ICIP’01),
to find the relevant images over the network. The agents volume 3, pages 6–9, Thessaloniki, Greece, October 2001.
[10] J. Wang, J. Li, and G. Wiederhold. SIMPLIcity: Semantics-
movements are influenced by markers on the computers
Sensitive Integrated Matching for Picture LIbraries. IEEE
of the network, which update depends on the information Transactions on pattern analysis and machine intelligence,
found by previous agents and on user’s feedback. We test 23(9), September 2001.
our system in the case of a variation of the localization of
the search concept. We showed that if the search concept
is well localized, our system offers a gain between 5% and
10%.

Proceedings of the 2006 IEEE/WIC/ACM International Conference

on Web Intelligence (WI 2006 Main Conference Proceedings)(WI'06)
0-7695-2747-7/06 Authorized
$20.00 ©licensed
2006 use limited to: REVA UNIVERSITY. Downloaded on June 28,2023 at 07:53:06 UTC from IEEE Xplore. Restrictions apply.